curl_easy_reset() may be used by an application to reset the options on
a curl handle. If an app has the CURL_IMPERSONATE env var defined, then
the impersonation options are automatically set in curl_easy_init() but
will be cleared in a call to curl_easy_reset(). The desired behavior is
for the impersonation options to be retained (as they are "transparent"
to the user), which this commit takes care of.
Note that this only has an effect when libcurl-impersonate is loaded and
the CURL_IMPERSONATE env var is set. Otherwise the regular behavior of
resetting all the handle options is retained.
Test that the unique TLS signature of curl-impersonate is preserved
after a call to curl_easy_reset() when libcurl-impersonate is loaded.
For this purpose change the 'minicurl' testing util to support multiple
URLs and launch it with 2 different URLs when testing the TLS signature.
.. and for Edge 101 as well. The TLS fingerprint is identical to
previous versions. The HTTP headers have the usual differences in the
user agents. One important change though is in the way the HTTP2
SETTINGS frame is formed. Up until Chrome 98, there was an additional
randomly-generated setting in the frame. This seems to have been removed
since. Therefore it was removed from curl-impersonate as well, and
support for Chrome/Edge 98 was deprecated, since supporting both
signatures requires a lot of work.
Add support for impersonating Firefox 100. The TLS signature is identical to
previous versions of Firefox.
In addition, upgrade NSS (Firefox's TLS library) to version 3.77 used by
Firefox 100. This is not strictly necessary as the previous version used
works just fine, but it's better keep up with the newest version.
Seems like the tests may fail because we don't wait for nghttpd (the
HTTP2 server used for testing) to actually start listening.
This commit uses asyncio to launch nghttpd and read its stdout until it
starts listening.
Injecting libcurl-impersonate with LD_PRELOAD is supported on
Linux only. On Mac there is DYLD_INSERT_LIBRARIES but it
reuqires more work to be fully functional.
Set CURLOPT_ACCEPT_ENCODING to an empty string in
curl_easy_impersonate() to enable decompression of encoded responses
using all built-in compressions. This is similar to adding
'--compressed' in the command line curl and is necessary since
curl_easy_impersonate() adds the 'Accept-Encoding' header which may
cause the server to respond with compressed content.
Locate curl-impersonate and libcurl-impersonate in a directory which is
configurable from the command line instead of looking for them in the
current directory. '--install-dir' is passed to pytest, where a 'bin'
and 'lib' directories are expected with (lib)curl-impersonate.
Rename the Actions file for the Docker build to allow the creation of
addition Action for native builds.
Add support for impersonating Chrome 99 on Android 12 (Pixel 6 was
chosen as an arbitrary Android phone to impersonate). The TLS signature
for Chrome on Android is identical to Chrome on Windows. The difference
is in a few HTTP headers ('sec-ch-ua-mobile', 'sec-ch-ua-platform' and
'user-agent').
Add support for impersonating Firefox 98 released a few days ago. It has
the same TLS signature as Firefox 95 so the adaptation includes changing
the user-agent only. Upgrade the NSS version used to 3.75, even though
it's not strictly necessary.
Add impersonation support for Chrome 99 and Edge 99 which were just released.
Their TLS signature is identical to the previous versions (98).
The only difference is in the user agents.
curl patch from
ca13947f00
Add tests to verify that the HTTP headers and HTTP/2 pseudo-headers
generated by curl-impersonate match the expected ones from the browser.
The test uses a local nghttpd HTTP/2 server instance with a self-signed
certificate.
Safari can now be imperonsated with libcurl using
curl_easy_impersonate() with the "safari15_3" target or by setting the
env var CURL_IMPERSONATE to "safari15_3".
curl patch from
0340cd8b3e
Add a new wrapper script that impersonates Safari 15.3 on MacOS 11.6.4
("Big Sur"). The wrapper script uses command line arguments
previously added to "curl-impersonate" for that purpose:
* --curves
* --signature-hashes
* --no-tls-session-ticket
* --http2-pseudo-headers-order
48415a4b00 added impersonation
capabilities to libcurl in the Chrome build. This adds the same
capabilities to the Firefox build as well.
curl-impersonate.patch generated from
b30b245b72
Test that libcurl-impersonate produces the desired TLS signature when
the CURL_IMPERSONATE env var is set. A small C program called "minicurl"
is linked to libcurl, and libcurl-impersonate is loaded at runtime with
LD_PRELOAD.
Microsoft Edge is, since 2019, based on Chromium. It has a completely
identical TLS signature to Chrome's. The only difference is in the HTTP
headers. Adding support for impersonating Edge was therefore extremely
simple.
Add automated testing infrastructure with tests verifying that
`curl-impersonate` has the same TLS signature as that of the
impersonated browser.
Each wrapper script (e.g. curl_chrome98) is launched to wikipedia.org
while a packet capture is running in the background. The Client Hello
is extracted from the capture, parsed and then compared to the known
browser signature.
The known signatures are stored in a YAML database.
