hoelee/curl-impersonate
1
0
Fork 0
mirror of https://github.com/lwthiker/curl-impersonate.git synced 2026-09-05 16:40:29 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #45 from lwthiker/makefile

Browse Source 
Native build system using Makefile
This commit is contained in:
lwthiker
2022-04-12 22:33:57 +03:00
committed by GitHub
parent 319139e045 842415f119
commit 7717c220d1
9 changed files with 457 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/build-and-test.yml → .github/workflows/build-and-test-docker.yml
+2 -2
View File
@@ -1,4 +1,4 @@
name: Build and test
name: Docker images
on:
push:
@@ -10,7 +10,7 @@ on:
jobs:
build-and-test:
name: Build curl-impersonate and run the tests
name: Build curl-impersonate Docker images and run the tests
runs-on: ubuntu-latest
services:
# A local registry is needed to push the build images to because we use
.github/workflows/build-and-test-make.yml
+66
View File
@@ -0,0 +1,66 @@
name: Build and test
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
build-and-test:
name: Build curl-impersonate and run the tests
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-20.04]
steps:
- name: Install Ubuntu dependencies
if: matrix.os == 'ubuntu-20.04'
run: |
sudo apt install build-essential pkg-config cmake ninja-build curl autoconf automake libtool
# Firefox version dependencies
sudo apt install python3-pip python-is-python3
pip install gyp-next
# Chrome version dependencies
sudo apt install golang-go
# Needed to compile 'minicurl'
sudo apt install libcurl4-openssl-dev
# More dependencies for the tests
sudo apt install tcpdump nghttp2-server libnss3
- name: Check out the repo
uses: actions/checkout@v2
- name: Install dependencies for the tests script
run: |
# Install globally so that we can run 'pytest' with 'sudo'
sudo pip install -r tests/requirements.txt
- name: Run configure script
run: |
autoconf
mkdir ${{ runner.temp }}/install
./configure --prefix=${{ runner.temp }}/install
- name: Build the Chrome version of curl-impersonate
run: |
make chrome-build
make chrome-install
- name: Build the Firefox version of curl-impersonate
run: |
make firefox-build
make firefox-install
- name: Prepare the tests
run: |
# Compile 'minicurl' which is used by the tests
gcc -Wall -Werror -o ${{ runner.temp }}/install/bin/minicurl tests/minicurl.c `curl-config --libs`
- name: Run the tests
run: |
cd tests
# sudo is needed for capturing packets
sudo pytest . --log-cli-level DEBUG --install-dir ${{ runner.temp}}/install
INSTALL.md
+92
View File
@@ -0,0 +1,92 @@
# Building and installing curl-impersonate
This guide shows how to compile and install curl-impersonate and libcurl-impersonate from source.
The build process takes care of downloading dependencies, patching them, compiling them and finally compiling curl itself with the needed patches.
There are currently two build options depending on your use case:
* Native build using Makefile
* Docker container build
There are two versions of `curl-impersonate` for technical reasons. The **chrome** version is used to impersonate Chrome, Edge and Safari. The **firefox** version is used to impersonate Firefox.
## Native build
Currently tested for Linux only. Work on porting to Mac is in progress.
Install dependencies for building all the components:
```
sudo apt install build-essential pkg-config cmake ninja-build curl autoconf automake libtool
# For the Firefox version only
sudo apt install python3-pip python-is-python3
pip install gyp-next
# For the Chrome vesion only
sudo apt install golang-go
```
Clone this repository:
```
git clone https://github.com/lwthiker/curl-impersonate.git
cd curl-impersonate
```
Generate the configure script:
```
autoconf
```
Configure and compile:
```
mkdir build && cd build
../configure
# Build and install the Firefox version
make firefox-build
make firefox-install
# Build and install the Chrome version
make chrome-build
make chrome-install
# Optionally remove all the build files
cd ../ && rm -Rf build
```
This will install curl-impersonate, libcurl-impersonate and the wrapper scripts to `/usr/local`. To change the installation path, pass `--prefix=/path/to/install/` to the `configure` script.
After installation you can run the wrapper scripts, e.g.:
```
curl_ff98 https://www.wikipedia.org
curl_chrome99 https://www.wikipedia.org
```
or run directly with you own flags:
```
curl-impersonate-ff https://www.wikipedia.org
curl-impersonate-chrome https://www.wikipedia.org
```
### Static compilation
To compile curl-impersonate statically with libcurl-impersonate, pass `--enable-static` to the `configure` script.
## Docker build
The Docker build is a bit more reproducible and serves as the reference implementation. It creates a Debian-based Docker image with the binaries.
### Chrome version
[`chrome/Dockerfile`](chrome/Dockerfile) is a debian-based Dockerfile that will build curl with all the necessary modifications and patches. Build it like the following:
```
docker build -t curl-impersonate-chrome chrome/
```
The resulting image contains a `/build/out` directory with the following:
* `curl-impersonate-chrome`, `curl-impersonate` - The curl binary that can impersonate Chrome/Edge/Safari. It is compiled statically against libcurl, BoringSSL, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04.
* `curl_chrome98`, `curl_chrome99`, `...` - Wrapper scripts that launch `curl-impersonate` with all the needed flags.
* `libcurl-impersonate-chrome.so`, `libcurl-impersonate.so` - libcurl compiled with impersonation support. See [libcurl-impersonate](#libcurl-impersonate) below for more details.
You can use them inside the docker, copy them out using `docker cp` or use them in a multi-stage docker build.
### Firefox version
Build with:
```
docker build -t curl-impersonate-ff firefox/
```
The resulting image contains a `/build/out` directory with the following:
* `curl-impersonate-ff`, `curl-impersonate` - The curl binary that can impersonate Firefox. It is compiled statically against libcurl, nss, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04.
* `curl_ff91esr`, `curl_ff95`, `...` - Wrapper scripts that launch `curl-impersonate` with all the needed flags.
* `libcurl-impersonate-ff.so`, `libcurl-impersonate.so` - libcurl compiled with impersonation support. See [libcurl-impersonate](#libcurl-impersonate) below for more details.
If you use it outside the container, install the following dependency:
* `sudo apt install libnss3`. Even though nss is statically compiled into `curl-impersonate`, it is still necessary to install libnss3 because curl dynamically loads `libnssckbi.so`, a file containing Mozilla's list of trusted root certificates. Alternatively, use `curl -k` to disable certificate verification.
Makefile.in
+213
View File
@@ -0,0 +1,213 @@
# Makefile to build curl-impersonate
# Some Makefile tricks were taken from https://tech.davis-hansson.com/p/make/
SHELL := bash
.ONESHELL:
.SHELLFLAGS := -eu -o pipefail -c
.DELETE_ON_ERROR:
MAKEFLAGS += --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules
BROTLI_VERSION := 1.0.9
NSS_VERSION := nss-3.75
NSS_URL := https://ftp.mozilla.org/pub/security/nss/releases/NSS_3_75_RTM/src/nss-3.75-with-nspr-4.32.tar.gz
BORING_SSL_COMMIT := 3a667d10e94186fd503966f5638e134fe9fb4080
NGHTTP2_VERSION := nghttp2-1.46.0
NGHTTP2_URL := https://github.com/nghttp2/nghttp2/releases/download/v1.46.0/nghttp2-1.46.0.tar.bz2
CURL_VERSION := curl-7.81.0
brotli_install_dir := $(abspath brotli-$(BROTLI_VERSION)/out/installed)
brotli_static_libs := $(brotli_install_dir)/lib/libbrotlicommon-static.a $(brotli_install_dir)/lib/libbrotlidec-static.a
nss_install_dir := $(abspath $(NSS_VERSION)/dist/Release)
nss_static_libs := $(nss_install_dir)/lib/libnss_static.a
boringssl_install_dir := $(abspath boringssl/build)
boringssl_static_libs := $(boringssl_install_dir)/lib/libssl.a $(boringssl_install_dir)/lib/libcrypto.a
nghttp2_install_dir := $(abspath $(NGHTTP2_VERSION)/installed)
nghttp2_static_libs := $(nghttp2_install_dir)/lib/libnghttp2.a
# Dependencies needed to compile the Firefox version
firefox_libs := $(brotli_static_libs) $(nss_static_libs) $(nghttp2_static_libs)
# Dependencies needed to compile the Chrome version
chrome_libs := $(brotli_static_libs) $(boringssl_static_libs) $(nghttp2_static_libs)
# To be set by the configure script
prefix = @prefix@
exec_prefix = @exec_prefix@
srcdir = @abs_srcdir@
# Auto-generate Makefile help.
# Borrowed from https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html
help: ## Show this help message
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
.PHONY: help
.DEFAULT_GOAL := help
firefox-build: $(CURL_VERSION)/.firefox ## Build the Firefox version of curl-impersonate
cd $(CURL_VERSION)
# Don't pass this Makefile's MAKEFLAGS
make MAKEFLAGS=
.PHONY: firefox-build
firefox-install: ## Install the Firefox version of curl-impersonate after build
cd $(CURL_VERSION)
make install-exec MAKEFLAGS=
# Wrapper scripts for the Firefox version (e.g. 'curl_ff98')
cp -f $(srcdir)/firefox/curl_ff* @bindir@
.PHONY: firefox-install
firefox-uninstall: ## Uninstall the Firefox version of curl-impersonate after 'make install'
cd $(CURL_VERSION)
make uninstall MAKEFLAGS=
rm -Rf @bindir@/curl_ff*
.PHONY: firefox-uninstall
chrome-build: $(CURL_VERSION)/.chrome ## Build the Chrome version of curl-impersonate
cd $(CURL_VERSION)
# Don't pass this Makefile's MAKEFLAGS
make MAKEFLAGS=
.PHONY: chrome-build
chrome-install: ## Install the Chrome version of curl-impersonate after build
cd $(CURL_VERSION)
make install-exec MAKEFLAGS=
# Wrapper scripts for the Chrome version (e.g. 'curl_chrome99')
cp -f $(srcdir)/chrome/curl_chrome* $(srcdir)/chrome/curl_edge* $(srcdir)/chrome/curl_safari* @bindir@
.PHONY: chrome-install
chrome-uninstall: ## Uninstall the Chrome version of curl-impersonate after 'make install'
cd $(CURL_VERSION)
make uninstall MAKEFLAGS=
rm -Rf @bindir@/curl_chrome* @bindir@/curl_edge* @bindir@/curl_safari*
.PHONY: chrome-uninstall
clean: ## Remove build artifacts
rm -Rf brotli-$(BROTLI_VERSION).tar.gz brotli-$(BROTLI_VERSION)
rm -Rf $(NSS_VERSION).tar.gz $(NSS_VERSION)
rm -Rf boringssl.zip boringssl
rm -Rf $(NGHTTP2_VERSION).tar.bz2 $(NGHTTP2_VERSION)
rm -Rf $(CURL_VERSION).tar.xz $(CURL_VERSION)
brotli-$(BROTLI_VERSION).tar.gz:
curl -L "https://github.com/google/brotli/archive/refs/tags/v${BROTLI_VERSION}.tar.gz" \
-o "brotli-${BROTLI_VERSION}.tar.gz"
$(brotli_static_libs): brotli-$(BROTLI_VERSION).tar.gz
tar xf brotli-$(BROTLI_VERSION).tar.gz
cd brotli-$(BROTLI_VERSION)
mkdir -p out
cd out
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=./installed ..
cmake --build . --config Release --target install
$(NSS_VERSION).tar.gz:
curl -L -o $(NSS_VERSION).tar.gz $(NSS_URL)
$(nss_static_libs): $(NSS_VERSION).tar.gz
tar xf $(NSS_VERSION).tar.gz
cd $(NSS_VERSION)/nss
./build.sh -o --disable-tests --static
boringssl.zip:
curl -L https://github.com/google/boringssl/archive/$(BORING_SSL_COMMIT).zip \
-o boringssl.zip
# Patch boringssl and use a dummy '.patched' file to mark it patched
boringssl/.patched: $(srcdir)/chrome/patches/boringssl-*.patch
unzip -o boringssl.zip
mv boringssl-$(BORING_SSL_COMMIT) boringssl
cd boringssl/
for p in $^; do patch -p1 < $$p; done
touch .patched
$(boringssl_static_libs): boringssl.zip boringssl/.patched
mkdir -p $(boringssl_install_dir)
cd $(boringssl_install_dir)
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=on -GNinja ..
ninja
# Fix the directory structure so that curl can compile against it.
# See https://everything.curl.dev/source/build/tls/boringssl
mkdir -p lib
ln -sf ../crypto/libcrypto.a lib/libcrypto.a
ln -sf ../ssl/libssl.a lib/libssl.a
cp -Rf ../include .
$(NGHTTP2_VERSION).tar.bz2:
curl -L $(NGHTTP2_URL) -o $(NGHTTP2_VERSION).tar.bz2
# Patch nghttp2 and use a dummy '.patched' file to mark it patched
$(NGHTTP2_VERSION)/.patched: $(srcdir)/firefox/patches/libnghttp2-*.patch
rm -Rf $(NGHTTP2_VERSION)
tar -xf $(NGHTTP2_VERSION).tar.bz2
cd $(NGHTTP2_VERSION)
for p in $^; do patch -p1 < $$p; done
# Re-generate the configure script
autoreconf -i && automake && autoconf
touch .patched
$(nghttp2_static_libs): $(NGHTTP2_VERSION).tar.bz2 $(NGHTTP2_VERSION)/.patched
cd $(NGHTTP2_VERSION)
./configure --prefix=$(nghttp2_install_dir) --with-pic
make MAKEFLAGS=
make install MAKEFLAGS=
$(CURL_VERSION).tar.xz:
curl -L "https://curl.se/download/$(CURL_VERSION).tar.xz" \
-o "$(CURL_VERSION).tar.xz"
# Apply the "Firefox version" patches and mark using a dummy file
$(CURL_VERSION)/.patched-ff: $(srcdir)/firefox/patches/curl-*.patch
rm -Rf $(CURL_VERSION)
tar -xf $(CURL_VERSION).tar.xz
cd $(CURL_VERSION)
for p in $^; do patch -p1 < $$p; done
# Re-generate the configure script
autoreconf -fi
touch .patched-ff
rm -f .patched-chrome
# Apply the "Chorme version" patches and mark using a dummy file
$(CURL_VERSION)/.patched-chrome: $(srcdir)/chrome/patches/curl-*.patch
rm -Rf $(CURL_VERSION)
tar -xf $(CURL_VERSION).tar.xz
cd $(CURL_VERSION)
for p in $^; do patch -p1 < $$p; done
# Re-generate the configure script
autoreconf -fi
touch .patched-chrome
rm -f .patched-ff
# This is a small hack that flags that curl was patched and configured in the "firefox" version
$(CURL_VERSION)/.firefox: $(firefox_libs) $(CURL_VERSION).tar.xz $(CURL_VERSION)/.patched-ff
cd $(CURL_VERSION)
./configure @curl_configure_options@ \
--prefix=@prefix@ \
--with-nghttp2=$(nghttp2_install_dir) \
--with-brotli=$(brotli_install_dir) \
--with-nss=$(nss_install_dir) \
USE_CURL_SSLKEYLOGFILE=true \
CFLAGS="-I$(nss_install_dir)/../public/nss -I$(nss_install_dir)/include/nspr"
# Remove possible leftovers from a previous compilation
make clean MAKEFLAGS=
touch .firefox
# Remove the Chrome flag if it exists
rm -f .chrome
# This is a small hack that flags that curl was patched and configured in the "chrome" version
$(CURL_VERSION)/.chrome: $(chrome_libs) $(CURL_VERSION).tar.xz $(CURL_VERSION)/.patched-chrome
cd $(CURL_VERSION)
./configure @curl_configure_options@ \
--prefix=@prefix@ \
--with-nghttp2=$(nghttp2_install_dir) \
--with-brotli=$(brotli_install_dir) \
--with-openssl=$(boringssl_install_dir) \
USE_CURL_SSLKEYLOGFILE=true \
LIBS="-pthread" \
CFLAGS="-I$(boringssl_install_dir)"
# Remove possible leftovers from a previous compilation
make clean MAKEFLAGS=
touch .chrome
# Remove the Firefox flag if it exists
rm -f .firefox
README.md
+6 -29
View File
@@ -1,4 +1,6 @@
# curl-impersonate &middot; ![workflow](https://github.com/lwthiker/curl-impersonate/actions/workflows/build-and-test.yml/badge.svg)
# curl-impersonate
![workflow](https://github.com/lwthiker/curl-impersonate/actions/workflows/build-and-test-docker.yml/badge.svg)
![workflow](https://github.com/lwthiker/curl-impersonate/actions/workflows/build-and-test-make.yml/badge.svg)
A special compilation of [curl](https://github.com/curl/curl) that makes it impersonate real browsers. It can impersonate the four major browsers: Chrome, Edge, Safari & Firefox. This curl binary is able to perform a TLS handshake that is identical to that of a real browser.
@@ -49,6 +51,9 @@ See [Advanced usage](#Advanced-usage) for more options.
## Installation
There are two versions of `curl-impersonate` for technical reasons. The **chrome** version is used to impersonate Chrome, Edge and Safari. The **firefox** version is used to impersonate Firefox.
### Building from source
See [INSTALL.md](INSTALL.md).
### Docker images
Docker images based on Alpine Linux with `curl-impersonate` compiled and ready to use are available on [Docker Hub](https://hub.docker.com/r/lwthiker/curl-impersonate). The images contain the binary and all the wrapper scripts. Use like the following:
```bash
@@ -65,34 +70,6 @@ docker run --rm lwthiker/curl-impersonate:0.3-chrome curl_chrome99 https://www.w
AUR packages are available to Arch users: [curl-impersonate-chrome](https://aur.archlinux.org/packages/curl-impersonate-chrome), [curl-impersonate-firefox](https://aur.archlinux.org/packages/curl-impersonate-firefox).
## Building from source
As mentioned there are two separate build systems. The **chrome** build is used to impersonate Chrome, Edge and Safari. The **firefox** build is used to impersonate Firefox.
### Chrome build
[`chrome/Dockerfile`](chrome/Dockerfile) is a debian-based Dockerfile that will build curl with all the necessary modifications and patches. Build it like the following:
```
docker build -t curl-impersonate-chrome chrome/
```
The resulting image contains a `/build/out` directory with the following:
* `curl-impersonate-chrome`, `curl-impersonate` - The curl binary that can impersonate Chrome/Edge/Safari. It is compiled statically against libcurl, BoringSSL, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04.
* `curl_chrome98`, `curl_chrome99`, `...` - Wrapper scripts that launch `curl-impersonate` with all the needed flags.
* `libcurl-impersonate-chrome.so`, `libcurl-impersonate.so` - libcurl compiled with impersonation support. See [libcurl-impersonate](#libcurl-impersonate) below for more details.
You can use them inside the docker, copy them out using `docker cp` or use them in a multi-stage docker build.
### Firefox build
Build with:
```
docker build -t curl-impersonate-ff firefox/
```
The resulting image contains a `/build/out` directory with the following:
* `curl-impersonate-ff`, `curl-impersonate` - The curl binary that can impersonate Firefox. It is compiled statically against libcurl, nss, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04.
* `curl_ff91esr`, `curl_ff95`, `...` - Wrapper scripts that launch `curl-impersonate` with all the needed flags.
* `libcurl-impersonate-ff.so`, `libcurl-impersonate.so` - libcurl compiled with impersonation support. See [libcurl-impersonate](#libcurl-impersonate) below for more details.
If you use it outside the container, install the following dependency:
* `sudo apt install libnss3`. Even though nss is statically compiled into `curl-impersonate`, it is still necessary to install libnss3 because curl dynamically loads `libnssckbi.so`, a file containing Mozilla's list of trusted root certificates. Alternatively, use `curl -k` to disable certificate verification.
## Advanced usage
### libcurl-impersonate
`libcurl-impersonate.so` is libcurl compiled with the same changes as the command line `curl-impersonate`.
configure.ac
+11
View File
@@ -0,0 +1,11 @@
AC_INIT([curl-impersonate], [0.3.2], [[email protected]])
AC_ARG_ENABLE([static],
[AS_HELP_STRING([--enable-static],
[Build curl-impersonate statically with libcurl-impersonate])],
[AC_SUBST([curl_configure_options], ["--enable-static --disable-shared"])],
[])
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
tests/Dockerfile
+8 -4
View File
@@ -19,14 +19,18 @@ RUN pip install --upgrade pip && \
RUN mkdir /tests/firefox /tests/chrome
# Copy the built binaries from both containers
COPY --from=ff /build/out/* /tests/firefox/
COPY --from=chrome /build/out/* /tests/chrome/
COPY --from=ff /build/out/curl-impersonate-ff /tests/install/bin/
COPY --from=ff /build/out/curl_* /tests/install/bin/
COPY --from=ff /build/out/libcurl-impersonate* /tests/install/lib/
COPY --from=chrome /build/out/curl-impersonate-chrome /tests/install/bin/
COPY --from=chrome /build/out/curl_* /tests/install/bin/
COPY --from=chrome /build/out/libcurl-impersonate* /tests/install/lib/
COPY . .
# Compile 'minicurl' which is used for testing libcurl-impersonate.
# 'minicurl' is compiled against the "regular" libcurl.
# libcurl-impersonate will replace it at runtime via LD_PRELOAD.
RUN gcc -Wall -Werror -o minicurl minicurl.c `curl-config --libs`
RUN gcc -Wall -Werror -o /tests/install/bin/minicurl minicurl.c `curl-config --libs`
ENTRYPOINT ["pytest"]
ENTRYPOINT ["pytest", "--install-dir", "/tests/install"]
tests/conftest.py
+4
View File
@@ -0,0 +1,4 @@
def pytest_addoption(parser):
# Where to find curl-impersonate's binaries
parser.addoption("--install-dir", action="store", default="/usr/local")
parser.addoption("--capture-interface", action="store", default="eth0")
tests/test_impersonate.py
+55 -34
View File
@@ -126,103 +126,105 @@ class TestImpersonation:
# List of binaries and their expected signatures
CURL_BINARIES_AND_SIGNATURES = [
# Test wrapper scripts
("chrome/curl_chrome98", None, "chrome_98.0.4758.102_win10"),
("chrome/curl_chrome99", None, "chrome_99.0.4844.51_win10"),
("chrome/curl_chrome99_android", None, "chrome_99.0.4844.73_android12-pixel6"),
("chrome/curl_edge98", None, "edge_98.0.1108.62_win10"),
("chrome/curl_edge99", None, "edge_99.0.1150.30_win10"),
("chrome/curl_safari15_3", None, "safari_15.3_macos11.6.4"),
("firefox/curl_ff91esr", None, "firefox_91.6.0esr_win10"),
("firefox/curl_ff95", None, "firefox_95.0.2_win10"),
("firefox/curl_ff98", None, "firefox_98.0_win10"),
("curl_chrome98", None, None, "chrome_98.0.4758.102_win10"),
("curl_chrome99", None, None, "chrome_99.0.4844.51_win10"),
("curl_chrome99_android", None, None, "chrome_99.0.4844.73_android12-pixel6"),
("curl_edge98", None, None, "edge_98.0.1108.62_win10"),
("curl_edge99", None, None, "edge_99.0.1150.30_win10"),
("curl_safari15_3", None, None, "safari_15.3_macos11.6.4"),
("curl_ff91esr", None, None, "firefox_91.6.0esr_win10"),
("curl_ff95", None, None, "firefox_95.0.2_win10"),
("curl_ff98", None, None, "firefox_98.0_win10"),
# Test libcurl-impersonate by loading it with LD_PRELOAD to an app
# linked against the regular libcurl and setting the
# CURL_IMPERSONATE env var.
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./chrome/libcurl-impersonate.so",
"CURL_IMPERSONATE": "chrome98"
},
"libcurl-impersonate-chrome.so",
"chrome_98.0.4758.102_win10"
),
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./chrome/libcurl-impersonate.so",
"CURL_IMPERSONATE": "chrome99"
},
"libcurl-impersonate-chrome.so",
"chrome_99.0.4844.51_win10"
),
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./chrome/libcurl-impersonate.so",
"CURL_IMPERSONATE": "chrome99_android"
},
"libcurl-impersonate-chrome.so",
"chrome_99.0.4844.73_android12-pixel6"
),
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./chrome/libcurl-impersonate.so",
"CURL_IMPERSONATE": "edge98"
},
"libcurl-impersonate-chrome.so",
"edge_98.0.1108.62_win10"
),
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./chrome/libcurl-impersonate.so",
"CURL_IMPERSONATE": "edge99"
},
"libcurl-impersonate-chrome.so",
"edge_99.0.1150.30_win10"
),
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./chrome/libcurl-impersonate.so",
"CURL_IMPERSONATE": "safari15_3"
},
"libcurl-impersonate-chrome.so",
"safari_15.3_macos11.6.4"
),
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./firefox/libcurl-impersonate.so",
"CURL_IMPERSONATE": "ff91esr"
},
"libcurl-impersonate-ff.so",
"firefox_91.6.0esr_win10"
),
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./firefox/libcurl-impersonate.so",
"CURL_IMPERSONATE": "ff95"
},
"libcurl-impersonate-ff.so",
"firefox_95.0.2_win10"
),
(
"./minicurl",
"minicurl",
{
"LD_PRELOAD": "./firefox/libcurl-impersonate.so",
"CURL_IMPERSONATE": "ff98"
},
"libcurl-impersonate-ff.so",
"firefox_98.0_win10"
)
]
@pytest.fixture
def tcpdump(self):
def tcpdump(self, pytestconfig):
"""Initialize a sniffer to capture curl's traffic."""
interface = pytestconfig.getoption("capture_interface")
logging.debug(
f"Running tcpdump on interface {self.TCPDUMP_CAPTURE_INTERFACE}"
f"Running tcpdump on interface {interface}"
)
p = subprocess.Popen([
"tcpdump", "-n",
"-i", self.TCPDUMP_CAPTURE_INTERFACE,
"-i", interface,
"-s", "0",
"-w", "-",
"-U", # Important, makes tcpdump unbuffered
@@ -255,7 +257,7 @@ class TestImpersonation:
def _run_curl(self, curl_binary, env_vars, extra_args, url):
env = os.environ.copy()
if env_vars:
env |= env_vars
env.update(env_vars)
logging.debug(f"Launching '{curl_binary}' to {url}")
if env_vars:
@@ -282,7 +284,7 @@ class TestImpersonation:
"""
for ts, buf in dpkt.pcap.Reader(io.BytesIO(pcap)):
eth = dpkt.ethernet.Ethernet(buf)
if not isinstance(eth.data, dpkt.ip.IP):
if not isinstance(eth.data, dpkt.ip.IP) and not isinstance(eth.data, dpkt.ip6.IP6):
continue
ip = eth.data
if not isinstance(ip.data, dpkt.tcp.TCP):
@@ -341,13 +343,15 @@ class TestImpersonation:
return pseudo_headers, headers
@pytest.mark.parametrize(
"curl_binary, env_vars, expected_signature",
"curl_binary, env_vars, ld_preload, expected_signature",
CURL_BINARIES_AND_SIGNATURES
)
def test_tls_client_hello(self,
pytestconfig,
tcpdump,
curl_binary,
env_vars,
ld_preload,
browser_signatures,
expected_signature):
"""
@@ -358,6 +362,14 @@ class TestImpersonation:
extracts the Client Hello packet from the capture and compares its
signature with the expected one defined in the YAML database.
"""
curl_binary = os.path.join(
pytestconfig.getoption("install_dir"), "bin", curl_binary
)
if ld_preload:
env_vars["LD_PRELOAD"] = os.path.join(
pytestconfig.getoption("install_dir"), "lib", ld_preload
)
ret = self._run_curl(curl_binary,
env_vars=env_vars,
extra_args=None,
@@ -396,15 +408,24 @@ class TestImpersonation:
assert equals, msg
@pytest.mark.parametrize(
"curl_binary, env_vars, expected_signature",
"curl_binary, env_vars, ld_preload, expected_signature",
CURL_BINARIES_AND_SIGNATURES
)
def test_http2_headers(self,
pytestconfig,
nghttpd,
curl_binary,
env_vars,
ld_preload,
browser_signatures,
expected_signature):
curl_binary = os.path.join(
pytestconfig.getoption("install_dir"), "bin", curl_binary
)
if ld_preload:
env_vars["LD_PRELOAD"] = os.path.join(
pytestconfig.getoption("install_dir"), "lib", ld_preload
)
ret = self._run_curl(curl_binary,
env_vars=env_vars,
extra_args=["-k"],