Add layer caching to speed up the CI builds. In addition, optimize the
Dockerfiles by moving all the dependency installations (which are
unlikely to change much) to the initial layers.
The tests container copies files from the build containers. In order for
this to work in the GitHub actions workflow, the build images need to be
uploaded to a local Docker registry first.
Add tests to verify that the HTTP headers and HTTP/2 pseudo-headers
generated by curl-impersonate match the expected ones from the browser.
The test uses a local nghttpd HTTP/2 server instance with a self-signed
certificate.
Add a new wrapper script that impersonates Safari 15.3 on MacOS 11.6.4
("Big Sur"). The wrapper script uses command line arguments
previously added to "curl-impersonate" for that purpose:
* --curves
* --signature-hashes
* --no-tls-session-ticket
* --http2-pseudo-headers-order
Add the ability to control the order of the HTTP/2 pseudo-headers. Each
browser uses a different order for the ":method", ":authority",
":scheme" and ":path" pseudo-headers. It is therefore desirable to be
able to control it. The CURLOPT_HTTP2_PSEUDO_HEADERS_ORDER libcurl
option and '--http2-pseudo-headers-order' command line option now allow
doing that.
Patch from
https://github.com/lwthiker/curl/commit/dd4b76241ed2d099ed7938d5c7db56b86cbaf710
A previous commit has enabled the TLS session ticket extension (by
removing SSL_OP_NO_TICKET) because Chrome uses it. This commit makes it
configurable via the CURLOPT_SSL_ENABLE_TICKET libcurl option or the
'--tls-session-ticker' command line flag. The goal is to impersonate
Safari which, as of version 15.3, does not use TLS session tickets.
This commit adds:
* Support for configuring the TLS certificate compression algorithms the
client is willing to receive via the CURLOPT_SSL_CERT_COMPRESSION option or
the '--cert-compression' command line flag.
* Support for decompressing zlib-compressed certificates in addition to
brotli.
Previously brotli decompression only was available and it was hardcoded
into the binary.
This commit makes some of the TLS options that are used for
impersonation configurable via libcurl options and command line flags to
curl-impersonate. The goal is to give more flexibility in configuring
the TLS extensions instead of hardcoding everything into the binary.
This will enable using the same binary for impersonating different
browsers (e.g. Safari).
The following options are now present:
* CURLOPT_SSL_EC_CURVES and the '--curves' flag are now usable. These
were present in the upstream curl but only for OpenSSL builds. This
commit also enables them for BoringSSL. They configure TLS extension
'supported_groups' (no. 10).
* CURLOPT_SSL_ENABLE_NPN and the '--no-npn' flags are usable. These were
present in the upstream curl but were disabled in a previous commit by
commenting out the relevant code (as Chrome disables NPN). They now work
and the wrapper scripts use the '--no-npn' flag.
* CURLOPT_SSL_ENABLE_ALPS and the '--alps' flag were added. These
control the ALPS TLS extension that Chrome uses.
* CURLOPT_SSL_SIG_HASH_ALGS and the '--signature-hashes' option were
added. These control the clien't list of supported signature & hash
algorithms, i.e. TLS extension 'signature_algorithms' (no. 13).
Test that libcurl-impersonate produces the desired TLS signature when
the CURL_IMPERSONATE env var is set. A small C program called "minicurl"
is linked to libcurl, and libcurl-impersonate is loaded at runtime with
LD_PRELOAD.
BoringSSL removed some old and weak cipehrs from OpenSSL. It appears as
though Safari still uses some of them.
The included patch restores them, so that using them in the "--ciphers"
option to curl will add them to the client's list of supported ciphers.
These ciphers may not actually work if the server chooses to use them,
because the "real" code to handle them is missing. But since they are
considered weak it is unlikely to happen.
Microsoft Edge is, since 2019, based on Chromium. It has a completely
identical TLS signature to Chrome's. The only difference is in the HTTP
headers. Adding support for impersonating Edge was therefore extremely
simple.
Patch generated from https://github.com/lwthiker/curl/commit/e8cd43c8ebd2624b3c635ec96b165c28f675cdf2
Add curl_easy_impersonate() API function that sets the needed options
on the curl 'easy' handle. It sets the various TLS options needed for
impersonation and the HTTP headers that the browser sends by default.
In addition, libcurl will check for the environment variable
CURL_IMPERSONATE when curl_easy_init() is called, and if it exists it
will call curl_easy_impersonate() internally. This theoretically allows
replacing an existing libcurl by setting the LD_LIBRARY_PATH and
CURL_IMPERSONATE env vars, without having to recompile the app.
In addition to the statically linked curl-impersonate binary, compile
libcurl.so for dynamic linking as well. The output file is saved at
/build/out/libcurl-impersonate.so.
Also strip the output binaries to reduce their size.
For now, support is for the Chrome build only.
Consolidate all of curl's patch files into a single file (one for each
browser). The modified curl source code is now maintained in Git (at
https://github.com/lwthiker/curl) and it is easier to just
generate the needed patch with 'git --diff'.
Add automated testing infrastructure with tests verifying that
`curl-impersonate` has the same TLS signature as that of the
impersonated browser.
Each wrapper script (e.g. curl_chrome98) is launched to wikipedia.org
while a packet capture is running in the background. The Client Hello
is extracted from the capture, parsed and then compared to the known
browser signature.
The known signatures are stored in a YAML database.
After inspecting an encrypted TLS session, I noticed that Firefox
sends the HTTP2 pseudo headers in the order :method, :path, :authority,
:scheme while curl was sending them with :authority being the last one.
This patch fixes the order of HTTP2 headers in curl to match Firefox.
lwthiker