Cherry pick commit b9b6148c45a00d675d5bb261bf4cbb45468ad807 from
upstream curl to fix a bug in curl 7.84.0 which causes failures with the
'TE: Trailers' header.
Add support for cross-compiling curl-impersonate.
Cross compiling can now be done using the '--host' flag to the configure
script. This will make sure that all sub-components are cross-compiled.
In addition, compiling for a different system requires explicitly
specifying multiple paths used by curl (e.g. for certificates). These
options were added to the configure script as well.
The build and test CI workflow will now attempt to cross-compile
curl-impersonate to ARM64 (aarch64), and upload this binary to the
GitHub release page.
Add the 'configure' script generated by autoconf and its dependencies to
the repository to save the users from having to run 'autoconf' manually.
HTTP/2 includes various settings pertaining to stream priorities. Chrome
and Firefox handle them differently, and this behavior was not mimicked
in curl-impersonate well. With this commit, the stream settings set
by curl-impersonate are identical to the real browsers.
* With Chrome, the default stream weight is 256 and the "exclusive bit"
is set to ON.
* With Firefox, a complex tree of stream dependencies is created
by default using PRIORITY frames. This behavior is now mimicked by
curl-impersonate.
curl_easy_reset() may be used by an application to reset the options on
a curl handle. If an app has the CURL_IMPERSONATE env var defined, then
the impersonation options are automatically set in curl_easy_init() but
will be cleared in a call to curl_easy_reset(). The desired behavior is
for the impersonation options to be retained (as they are "transparent"
to the user), which this commit takes care of.
Note that this only has an effect when libcurl-impersonate is loaded and
the CURL_IMPERSONATE env var is set. Otherwise the regular behavior of
resetting all the handle options is retained.
Test that the unique TLS signature of curl-impersonate is preserved
after a call to curl_easy_reset() when libcurl-impersonate is loaded.
For this purpose change the 'minicurl' testing util to support multiple
URLs and launch it with 2 different URLs when testing the TLS signature.
Add support for impersonating Firefox 100. The TLS signature is identical to
previous versions of Firefox.
In addition, upgrade NSS (Firefox's TLS library) to version 3.77 used by
Firefox 100. This is not strictly necessary as the previous version used
works just fine, but it's better keep up with the newest version.
When reusing a curl handle on which the 'Host' header was explicitly
set, the previously-set header was being kept in use for following
requests.
The issue was in curl-impersonate's merging of user-supplied headers
with its own list of browser headers. The call to
Curl_http_merge_headers() which takes care of this had been placed after
the handling of the host header, which caused the previous one to be
used.
* Set curl's HTTP/2 window size to match Chrome and Firefox. This
affects the "Window Size Increment" parameter in the WINDOW_UPDATE
HTTP/2 frame sent out by curl, which was different than the one
sent by Chrome or Firefox.
* Set curl's HTTP/2 SETTINGS frame to match Firefox.
libnssckbi is loaded at runtime by NSS. On some systems it is located in
a non-standard location that dlopen() can't find. For example, in Ubuntu
it may be in /usr/lib/x86_64-linux-gnu and on Mac M1 in
/opt/homebrew/nss. This becomes a problem when you static link NSS.
Search for libnssckbi in the configure script and add the relevant path
using '-rpath' linker flag. In addition, drop the previous hack for
Ubuntu that searched libnssckbi in a hardcoded location.
This is an attmept to link with NSS statically on macOS (both Intel and
Apple M1).
Statically linking with NSS is a total mess and completely
undocumented. There are 20+ .a files to link with, and their linking
order matters. The main reference for this commit is a Mozilla Rust code
responsible for statically linking NSS:
b2690fd2e4/components/support/rc_crypto/nss/nss_b uild_common/src/lib.rs#L94
Unfortunately, even with that in hand, a lot of hacking is needed to
make it all work.
Set CURLOPT_ACCEPT_ENCODING to an empty string in
curl_easy_impersonate() to enable decompression of encoded responses
using all built-in compressions. This is similar to adding
'--compressed' in the command line curl and is necessary since
curl_easy_impersonate() adds the 'Accept-Encoding' header which may
cause the server to respond with compressed content.
Change the binary names of curl and libcurl as part of the curl build
process by patching curl's build scripts. When running 'make' in the
patched curl directory the resulting binaries will be already named
'curl-impersonate-ff' and 'curl-impersonate-chrome' (and the same for
libcurl), thus saving the need for manually renaming them after the
compilation. This also enables running 'make install' with curl's own
Makefiles in order to install curl-impersonate.
Add support for impersonating Firefox 98 released a few days ago. It has
the same TLS signature as Firefox 95 so the adaptation includes changing
the user-agent only. Upgrade the NSS version used to 3.75, even though
it's not strictly necessary.
48415a4b00 added impersonation
capabilities to libcurl in the Chrome build. This adds the same
capabilities to the Firefox build as well.
curl-impersonate.patch generated from
b30b245b72
Consolidate all of curl's patch files into a single file (one for each
browser). The modified curl source code is now maintained in Git (at
https://github.com/lwthiker/curl) and it is easier to just
generate the needed patch with 'git --diff'.
