Add support for impersonating Chrome 110. Chrome 110 comes with TLS
extension permutation enabled by default. We mimic this behavior in libcurl with
the new CURLOPT_SSL_PERMUTE_EXTENSIONS option, which enables the
corresponding flag in BoringSSL.
---------
Co-authored-by: Johann Saunier <saunier.johann@gmail.com>
Chrome 107 was recently released. The only difference in signature from
previous Chrome versions is that an HTTP2_NO_SERVER_PUSH setting is sent
in the HTTP2 SETTINGS frame.
When impersonating with libcurl, the built-in user agent used for
impersonation was overriding and useragent the user had set via
CURLOPT_USERAGENT. The fix takes care to specifically handle user agent
set with CURLOPT_USERAGENT and use it instead of the built-in one when
it is supplied.
Minimize the size of the resulting Docker image size by using
multi-stage build and copying the resulting binaries into a minimal
Debian system. This was done with the Alpine Docker images up until now
but not with the Debian images.
Let the user disable the built-in list of HTTP headers when using
libcurl-impersonate, either directly or when replacing it at runtime
with LD_PRELOAD. This is intended to give the user more precise control
over the content and order of the HTTP headers.
To support this, the curl_easy_impersonate() now has an added argument
that can be set to 0, in which case the built-in list of HTTP headers
used by libcurl-impersonate will not be automatically sent. Instead,
the user is expected to supply all the headers by themselves using the
standard CURLOPT_HTTPHEADER libcurl option.
When using LD_PRELOAD to inject libcurl-impersonate, one can disable
the built-in headers by setting the CURL_IMPERSONATE_HEADERS
environment variable to "no".
Cherry pick commit b9b6148c45a00d675d5bb261bf4cbb45468ad807 from
upstream curl to fix a bug in curl 7.84.0 which causes failures with the
'TE: Trailers' header.
HTTP/2 includes various settings pertaining to stream priorities. Chrome
and Firefox handle them differently, and this behavior was not mimicked
in curl-impersonate well. With this commit, the stream settings set
by curl-impersonate are identical to the real browsers.
* With Chrome, the default stream weight is 256 and the "exclusive bit"
is set to ON.
* With Firefox, a complex tree of stream dependencies is created
by default using PRIORITY frames. This behavior is now mimicked by
curl-impersonate.
curl_easy_reset() may be used by an application to reset the options on
a curl handle. If an app has the CURL_IMPERSONATE env var defined, then
the impersonation options are automatically set in curl_easy_init() but
will be cleared in a call to curl_easy_reset(). The desired behavior is
for the impersonation options to be retained (as they are "transparent"
to the user), which this commit takes care of.
Note that this only has an effect when libcurl-impersonate is loaded and
the CURL_IMPERSONATE env var is set. Otherwise the regular behavior of
resetting all the handle options is retained.
Test that the unique TLS signature of curl-impersonate is preserved
after a call to curl_easy_reset() when libcurl-impersonate is loaded.
For this purpose change the 'minicurl' testing util to support multiple
URLs and launch it with 2 different URLs when testing the TLS signature.
.. and for Edge 101 as well. The TLS fingerprint is identical to
previous versions. The HTTP headers have the usual differences in the
user agents. One important change though is in the way the HTTP2
SETTINGS frame is formed. Up until Chrome 98, there was an additional
randomly-generated setting in the frame. This seems to have been removed
since. Therefore it was removed from curl-impersonate as well, and
support for Chrome/Edge 98 was deprecated, since supporting both
signatures requires a lot of work.
When reusing a curl handle on which the 'Host' header was explicitly
set, the previously-set header was being kept in use for following
requests.
The issue was in curl-impersonate's merging of user-supplied headers
with its own list of browser headers. The call to
Curl_http_merge_headers() which takes care of this had been placed after
the handling of the host header, which caused the previous one to be
used.
* Set curl's HTTP/2 window size to match Chrome and Firefox. This
affects the "Window Size Increment" parameter in the WINDOW_UPDATE
HTTP/2 frame sent out by curl, which was different than the one
sent by Chrome or Firefox.
* Set curl's HTTP/2 SETTINGS frame to match Firefox.
Add a few commands to the Dockerfile to check that 'curl-impersonate'
was compiled correctly: Check that it has brotli, http2 and tls support,
and check that the dependencies were compiled statically.
These are basic checks which are useful when modifying the Dockerfile:
Sometimes even small modifications cause curl to be compiled
incorrectly but without failing the build.
Previously '-l:nghttp2.a' was used to specify static linking with
nghttp2 and to stop the linker from linking dynamically with
libnghttp2.so. This way of linking is not supported on macOS. Instead,
add '--disable-shared' to prevent libnghttp2.so from even being
compiled. This way the linker will find the static library only and link
against it.
Set CURLOPT_ACCEPT_ENCODING to an empty string in
curl_easy_impersonate() to enable decompression of encoded responses
using all built-in compressions. This is similar to adding
'--compressed' in the command line curl and is necessary since
curl_easy_impersonate() adds the 'Accept-Encoding' header which may
cause the server to respond with compressed content.
Change the binary names of curl and libcurl as part of the curl build
process by patching curl's build scripts. When running 'make' in the
patched curl directory the resulting binaries will be already named
'curl-impersonate-ff' and 'curl-impersonate-chrome' (and the same for
libcurl), thus saving the need for manually renaming them after the
compilation. This also enables running 'make install' with curl's own
Makefiles in order to install curl-impersonate.
Rename the binary files of curl-impersonate so that the Firefox and
Chrome versions can co-exist on the same system. The Firefox version is
now named 'curl-impersonate-ff' and 'libcurl-impersonate-ff' and the
Chrome version is named 'curl-impersonate-chrome' and
'libcurl-impersonate-chrome'. The wrapper scripts look for these names
as well. Symbolic names with the old names are still created inside the
Docker images to keep compatibility as much as possible.
Add support for impersonating Chrome 99 on Android 12 (Pixel 6 was
chosen as an arbitrary Android phone to impersonate). The TLS signature
for Chrome on Android is identical to Chrome on Windows. The difference
is in a few HTTP headers ('sec-ch-ua-mobile', 'sec-ch-ua-platform' and
'user-agent').
Add Dockerfiles for building curl-impersonate on Alpine Linux.
The Dockerfile template file was modified to support Alpine Linux,
mainly by changing the dependency installation from 'apt' to 'apk'.
The resulting alpine images are small (~11mb) and will be uploaded to
Docker hub.
Since the firefox and chrome builds are similar except for the TLS
library used, it makes sense to unify their Dockerfiles. This commit
introduces a template Dockerfile from which both the build Dockerfiles
are generated using the simple 'mustache' template system.
Add impersonation support for Chrome 99 and Edge 99 which were just released.
Their TLS signature is identical to the previous versions (98).
The only difference is in the user agents.
curl patch from
ca13947f00
Add layer caching to speed up the CI builds. In addition, optimize the
Dockerfiles by moving all the dependency installations (which are
unlikely to change much) to the initial layers.
Safari can now be imperonsated with libcurl using
curl_easy_impersonate() with the "safari15_3" target or by setting the
env var CURL_IMPERSONATE to "safari15_3".
curl patch from
0340cd8b3e
Add a new wrapper script that impersonates Safari 15.3 on MacOS 11.6.4
("Big Sur"). The wrapper script uses command line arguments
previously added to "curl-impersonate" for that purpose:
* --curves
* --signature-hashes
* --no-tls-session-ticket
* --http2-pseudo-headers-order
Add the ability to control the order of the HTTP/2 pseudo-headers. Each
browser uses a different order for the ":method", ":authority",
":scheme" and ":path" pseudo-headers. It is therefore desirable to be
able to control it. The CURLOPT_HTTP2_PSEUDO_HEADERS_ORDER libcurl
option and '--http2-pseudo-headers-order' command line option now allow
doing that.
Patch from
dd4b76241e
A previous commit has enabled the TLS session ticket extension (by
removing SSL_OP_NO_TICKET) because Chrome uses it. This commit makes it
configurable via the CURLOPT_SSL_ENABLE_TICKET libcurl option or the
'--tls-session-ticker' command line flag. The goal is to impersonate
Safari which, as of version 15.3, does not use TLS session tickets.
This commit adds:
* Support for configuring the TLS certificate compression algorithms the
client is willing to receive via the CURLOPT_SSL_CERT_COMPRESSION option or
the '--cert-compression' command line flag.
* Support for decompressing zlib-compressed certificates in addition to
brotli.
Previously brotli decompression only was available and it was hardcoded
into the binary.
This commit makes some of the TLS options that are used for
impersonation configurable via libcurl options and command line flags to
curl-impersonate. The goal is to give more flexibility in configuring
the TLS extensions instead of hardcoding everything into the binary.
This will enable using the same binary for impersonating different
browsers (e.g. Safari).
The following options are now present:
* CURLOPT_SSL_EC_CURVES and the '--curves' flag are now usable. These
were present in the upstream curl but only for OpenSSL builds. This
commit also enables them for BoringSSL. They configure TLS extension
'supported_groups' (no. 10).
* CURLOPT_SSL_ENABLE_NPN and the '--no-npn' flags are usable. These were
present in the upstream curl but were disabled in a previous commit by
commenting out the relevant code (as Chrome disables NPN). They now work
and the wrapper scripts use the '--no-npn' flag.
* CURLOPT_SSL_ENABLE_ALPS and the '--alps' flag were added. These
control the ALPS TLS extension that Chrome uses.
* CURLOPT_SSL_SIG_HASH_ALGS and the '--signature-hashes' option were
added. These control the clien't list of supported signature & hash
algorithms, i.e. TLS extension 'signature_algorithms' (no. 13).
48415a4b00 added impersonation
capabilities to libcurl in the Chrome build. This adds the same
capabilities to the Firefox build as well.
curl-impersonate.patch generated from
b30b245b72
