Test that libcurl-impersonate produces the desired TLS signature when
the CURL_IMPERSONATE env var is set. A small C program called "minicurl"
is linked to libcurl, and libcurl-impersonate is loaded at runtime with
LD_PRELOAD.
BoringSSL removed some old and weak cipehrs from OpenSSL. It appears as
though Safari still uses some of them.
The included patch restores them, so that using them in the "--ciphers"
option to curl will add them to the client's list of supported ciphers.
These ciphers may not actually work if the server chooses to use them,
because the "real" code to handle them is missing. But since they are
considered weak it is unlikely to happen.
Microsoft Edge is, since 2019, based on Chromium. It has a completely
identical TLS signature to Chrome's. The only difference is in the HTTP
headers. Adding support for impersonating Edge was therefore extremely
simple.
Patch generated from e8cd43c8eb
Add curl_easy_impersonate() API function that sets the needed options
on the curl 'easy' handle. It sets the various TLS options needed for
impersonation and the HTTP headers that the browser sends by default.
In addition, libcurl will check for the environment variable
CURL_IMPERSONATE when curl_easy_init() is called, and if it exists it
will call curl_easy_impersonate() internally. This theoretically allows
replacing an existing libcurl by setting the LD_LIBRARY_PATH and
CURL_IMPERSONATE env vars, without having to recompile the app.
In addition to the statically linked curl-impersonate binary, compile
libcurl.so for dynamic linking as well. The output file is saved at
/build/out/libcurl-impersonate.so.
Also strip the output binaries to reduce their size.
For now, support is for the Chrome build only.
Consolidate all of curl's patch files into a single file (one for each
browser). The modified curl source code is now maintained in Git (at
https://github.com/lwthiker/curl) and it is easier to just
generate the needed patch with 'git --diff'.
Add automated testing infrastructure with tests verifying that
`curl-impersonate` has the same TLS signature as that of the
impersonated browser.
Each wrapper script (e.g. curl_chrome98) is launched to wikipedia.org
while a packet capture is running in the background. The Client Hello
is extracted from the capture, parsed and then compared to the known
browser signature.
The known signatures are stored in a YAML database.
After inspecting an encrypted TLS session, I noticed that Firefox
sends the HTTP2 pseudo headers in the order :method, :path, :authority,
:scheme while curl was sending them with :authority being the last one.
This patch fixes the order of HTTP2 headers in curl to match Firefox.
In preparation for merging the support for Chrome impersonation, move
all build files to the 'firefox' folder. The two builds will live
separately as they are rather different (using two different SSL
libraries for instance).
Building curl-impersonate with Chrome impersonation requires a different
compilation and a different set of patches which may conflict with the
Firefox impersonation. Therefore we will treat them a separate builds.
All chrome related files were moved to the chrome/ directory.
Because we send the 'Accept-Encoding: gzip, deflate, br' header we
may receive compressed content (gzip or brotli). Previously this was
handled manually in the wrapper script, however curl has a '--compressed'
flag exactly for that purpose. Add it by default to the wrapper scripts.
* Change the default HTTP/2 settings sent by Curl to match the ones that
Chrome sends. They appear in the SETTINGS message in the beginning of
the HTTP/2 handshake, which can be seen only after decrypting the TLS
traffic.
* Change the order of the HTTP/2 pseudo-headers ":authority", ":scheme"
and ":path" in the HEADERS message. Curl sent them in a different order
than Chrome.
* Enable TLS extension 35 (session_ticket) which Curl turned off.
* Set the signature algorithms to match Chrome's.
* Fix the headers to exactly match Chrome.
It seems that the Client Hello message is now identical to Chrome.
However, it is still getting rejected.
* Headers and ciphers are aligned to Chrome 98 (Windows, non-incognito)
* GREASE enabled because chrome uses it as well
* TLS extensions 27, 5, 18 enabled.
As suggested on the Hacker News thread
(https://news.ycombinator.com/item?id=30378562), add support for Firefox
Extended Support Release.
The required changes were adding one more cipher to the
ciphers list and changing the user agent. Apart from that the TLS
fingerprint is identical to Firefox 95 which was already supported.
