Add a few commands to the Dockerfile to check that 'curl-impersonate'
was compiled correctly: Check that it has brotli, http2 and tls support,
and check that the dependencies were compiled statically.
These are basic checks which are useful when modifying the Dockerfile:
Sometimes even small modifications cause curl to be compiled
incorrectly but without failing the build.
Previously '-l:nghttp2.a' was used to specify static linking with
nghttp2 and to stop the linker from linking dynamically with
libnghttp2.so. This way of linking is not supported on macOS. Instead,
add '--disable-shared' to prevent libnghttp2.so from even being
compiled. This way the linker will find the static library only and link
against it.
Set CURLOPT_ACCEPT_ENCODING to an empty string in
curl_easy_impersonate() to enable decompression of encoded responses
using all built-in compressions. This is similar to adding
'--compressed' in the command line curl and is necessary since
curl_easy_impersonate() adds the 'Accept-Encoding' header which may
cause the server to respond with compressed content.
On some systems (Ubuntu), after doing 'make firefox-install' or 'make chrome-install', the linker can't find libcurl-impersonate.
Instruct users to run 'ldconfig' manually after installation to refresh the linker's cache, so it finds libcurl-impersonate in /usr/local/
Add a GitHub action to compile and test native builds using the new
Makefile-based build system. Currently runs on Ubuntu and in the future
will run similarly on MacOS.
Locate curl-impersonate and libcurl-impersonate in a directory which is
configurable from the command line instead of looking for them in the
current directory. '--install-dir' is passed to pytest, where a 'bin'
and 'lib' directories are expected with (lib)curl-impersonate.
Rename the Actions file for the Docker build to allow the creation of
addition Action for native builds.
Add INSTALL.md with explanation about using the new Makefile-based build
system and the Docker-based build system, and remove the same section
from README.md.
When doing 'make firefox-install' or 'make chrome-install' in our own
Makefile, only install curl's binaries (curl & libcurl) and do not
install the man pages and headers. This will prevent collision of these
files with any installed curl man pages or headers. In the future it
might be possible to rename the man pages or headers as well to allow
them live side-by-side with the vanilla curl.
Use autoconf's automatic variable 'abs_srcdir' to find the path to the
patch files and wrapper scripts, instead of relying on the Makefile's
directory. This allows doing out-of-tree builds, i.e. 'mkdir build && cd
build && ../configure && make'.
Use a 'configure' script generated by autoconf to configure whether a
static build is desired, and where to install the final curl-impersonate
binaries. Add installation targets to the Makefile.
Add a Makefile to orchestrate native builds of curl-impersonate without
Docker. The Makefile does not compile any program by itself but is just
used to build all the dependencies and then build curl itself with the
needed patches. It has a very similar flow to the Dockerfile.
Change the binary names of curl and libcurl as part of the curl build
process by patching curl's build scripts. When running 'make' in the
patched curl directory the resulting binaries will be already named
'curl-impersonate-ff' and 'curl-impersonate-chrome' (and the same for
libcurl), thus saving the need for manually renaming them after the
compilation. This also enables running 'make install' with curl's own
Makefiles in order to install curl-impersonate.
Rename the binary files of curl-impersonate so that the Firefox and
Chrome versions can co-exist on the same system. The Firefox version is
now named 'curl-impersonate-ff' and 'libcurl-impersonate-ff' and the
Chrome version is named 'curl-impersonate-chrome' and
'libcurl-impersonate-chrome'. The wrapper scripts look for these names
as well. Symbolic names with the old names are still created inside the
Docker images to keep compatibility as much as possible.
Add support for impersonating Chrome 99 on Android 12 (Pixel 6 was
chosen as an arbitrary Android phone to impersonate). The TLS signature
for Chrome on Android is identical to Chrome on Windows. The difference
is in a few HTTP headers ('sec-ch-ua-mobile', 'sec-ch-ua-platform' and
'user-agent').
Add support for impersonating Firefox 98 released a few days ago. It has
the same TLS signature as Firefox 95 so the adaptation includes changing
the user-agent only. Upgrade the NSS version used to 3.75, even though
it's not strictly necessary.
Add 'apk add nss' for the final alpine image of the Firefox version of
curl-impersonate. This is needed for curl to find the list of root
certificates.
Add Dockerfiles for building curl-impersonate on Alpine Linux.
The Dockerfile template file was modified to support Alpine Linux,
mainly by changing the dependency installation from 'apt' to 'apk'.
The resulting alpine images are small (~11mb) and will be uploaded to
Docker hub.
Since the firefox and chrome builds are similar except for the TLS
library used, it makes sense to unify their Dockerfiles. This commit
introduces a template Dockerfile from which both the build Dockerfiles
are generated using the simple 'mustache' template system.
The GitHub Action workflow uses a cache to speed up the Docker builds.
However, it seems that it only partially works, saving only the Docker
layers for the Firefox build. This is an attempt to solve it by using
two separate cache paths, one for each build.
Add impersonation support for Chrome 99 and Edge 99 which were just released.
Their TLS signature is identical to the previous versions (98).
The only difference is in the user agents.
curl patch from
ca13947f00
