When reusing a curl handle on which the 'Host' header was explicitly
set, the previously-set header was being kept in use for following
requests.
The issue was in curl-impersonate's merging of user-supplied headers
with its own list of browser headers. The call to
Curl_http_merge_headers() which takes care of this had been placed after
the handling of the host header, which caused the previous one to be
used.
* Set curl's HTTP/2 window size to match Chrome and Firefox. This
affects the "Window Size Increment" parameter in the WINDOW_UPDATE
HTTP/2 frame sent out by curl, which was different than the one
sent by Chrome or Firefox.
* Set curl's HTTP/2 SETTINGS frame to match Firefox.
libnssckbi is loaded at runtime by NSS. On some systems it is located in
a non-standard location that dlopen() can't find. For example, in Ubuntu
it may be in /usr/lib/x86_64-linux-gnu and on Mac M1 in
/opt/homebrew/nss. This becomes a problem when you static link NSS.
Search for libnssckbi in the configure script and add the relevant path
using '-rpath' linker flag. In addition, drop the previous hack for
Ubuntu that searched libnssckbi in a hardcoded location.
Seems like the tests may fail because we don't wait for nghttpd (the
HTTP2 server used for testing) to actually start listening.
This commit uses asyncio to launch nghttpd and read its stdout until it
starts listening.
Injecting libcurl-impersonate with LD_PRELOAD is supported on
Linux only. On Mac there is DYLD_INSERT_LIBRARIES but it
reuqires more work to be fully functional.
This is an attmept to link with NSS statically on macOS (both Intel and
Apple M1).
Statically linking with NSS is a total mess and completely
undocumented. There are 20+ .a files to link with, and their linking
order matters. The main reference for this commit is a Mozilla Rust code
responsible for statically linking NSS:
b2690fd2e4/components/support/rc_crypto/nss/nss_b uild_common/src/lib.rs#L94
Unfortunately, even with that in hand, a lot of hacking is needed to
make it all work.
Add a few commands to the Dockerfile to check that 'curl-impersonate'
was compiled correctly: Check that it has brotli, http2 and tls support,
and check that the dependencies were compiled statically.
These are basic checks which are useful when modifying the Dockerfile:
Sometimes even small modifications cause curl to be compiled
incorrectly but without failing the build.
Previously '-l:nghttp2.a' was used to specify static linking with
nghttp2 and to stop the linker from linking dynamically with
libnghttp2.so. This way of linking is not supported on macOS. Instead,
add '--disable-shared' to prevent libnghttp2.so from even being
compiled. This way the linker will find the static library only and link
against it.
Set CURLOPT_ACCEPT_ENCODING to an empty string in
curl_easy_impersonate() to enable decompression of encoded responses
using all built-in compressions. This is similar to adding
'--compressed' in the command line curl and is necessary since
curl_easy_impersonate() adds the 'Accept-Encoding' header which may
cause the server to respond with compressed content.
On some systems (Ubuntu), after doing 'make firefox-install' or 'make chrome-install', the linker can't find libcurl-impersonate.
Instruct users to run 'ldconfig' manually after installation to refresh the linker's cache, so it finds libcurl-impersonate in /usr/local/
Add a GitHub action to compile and test native builds using the new
Makefile-based build system. Currently runs on Ubuntu and in the future
will run similarly on MacOS.
Locate curl-impersonate and libcurl-impersonate in a directory which is
configurable from the command line instead of looking for them in the
current directory. '--install-dir' is passed to pytest, where a 'bin'
and 'lib' directories are expected with (lib)curl-impersonate.
Rename the Actions file for the Docker build to allow the creation of
addition Action for native builds.
Add INSTALL.md with explanation about using the new Makefile-based build
system and the Docker-based build system, and remove the same section
from README.md.
When doing 'make firefox-install' or 'make chrome-install' in our own
Makefile, only install curl's binaries (curl & libcurl) and do not
install the man pages and headers. This will prevent collision of these
files with any installed curl man pages or headers. In the future it
might be possible to rename the man pages or headers as well to allow
them live side-by-side with the vanilla curl.
Use autoconf's automatic variable 'abs_srcdir' to find the path to the
patch files and wrapper scripts, instead of relying on the Makefile's
directory. This allows doing out-of-tree builds, i.e. 'mkdir build && cd
build && ../configure && make'.
Use a 'configure' script generated by autoconf to configure whether a
static build is desired, and where to install the final curl-impersonate
binaries. Add installation targets to the Makefile.
