ce95990620
The GitHub Action workflow uses a cache to speed up the Docker builds. However, it seems that it only partially works, saving only the Docker layers for the Firefox build. This is an attempt to solve it by using two separate cache paths, one for each build.
curl-impersonate · 
A special compilation of curl that makes it impersonate real browsers. Currently supports Chrome, Edge & Firefox. This curl binary is able to perform a TLS handshake that is identical to that of a real browser.
Why?
When you use an HTTP client with a TLS website, it first performs a TLS handshake. The first message of that handshake is called Client Hello. The Client Hello message that curl produces differs drastically from that of a real browser. Compare the following Wireshark capture. Left is a regular curl, right is Firefox.
Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block it. With the modified curl in this repository, the Client Hello message looks exactly like Chrome's or Firefox's. This tricks TLS fingerprinters to think that it is a real browser that is accessing them.
How?
The modifications that were needed to make this work:
- Compiling curl with nss, the TLS library that Firefox uses, instead of OpenSSL. For the Chrome version, compiling with BoringSSL.
- Modifying the way curl configures various TLS extensions and SSL options.
- Running curl with some non-default flags, specifically
--http2,--ciphers, and some-Hheaders.
The resulting curl looks, from a network perspective, identical to various browser versions. Compare: (left is curl-impersonate, right is Firefox):
Read the full description in the blog post: part a, part b.
Installation
This repository maintains two separate build systems, one for the Chrome version and one for the Firefox version.
Chrome version
chrome/Dockerfile is a Dockerfile that will build curl with all the necessary modifications for impersonating Chrome. Build it like the following:
docker build -t curl-impersonate-chrome chrome/
The resulting image contains:
/build/out/curl-impersonate- The curl binary that can impersonate Chrome. It is compiled statically against libcurl, BoringSSL, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04./build/out/curl_chrome98- A wrapper script that launchescurl-impersonatewith the needed headers and ciphers to impersonate Chrome 98./build/out/curl_edge98- Same but with Edge 98 (which is based on Chromium)./build/out/libcurl-impersonate.so- libcurl compiled with impersonation support. See Usage below for more details.
You can use them inside the docker, copy them out using docker cp or use them in a multi-stage docker build. If you use it outside this container:
- Install dependencies:
sudo apt install libbrotli1
Firefox version
Build with:
docker build -t curl-impersonate-ff firefox/
The resulting image contains:
/build/out/curl-impersonate- The curl binary that can impersonate Firefox. It is compiled statically against libcurl, nss, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04./build/out/curl_ff91esr- A wrapper script that launchescurl-impersonatewith the needed headers and ciphers to impersonate Firefox 91 ESR (Extended Support Release)./build/out/curl_ff95- Same but with Firefox 95./build/out/libcurl-impersonate.so- libcurl compiled with impersonation support. See Usage below for more details.
If you use it outside this container:
- Install dependencies:
sudo apt install libbrotli1 - Install libnss3:
sudo apt install libnss3. Even though nss is statically compiled intocurl-impersonate, it is still necessary to install libnss3 because curl dynamically loadslibnssckbi.so, a file containing Mozilla's list of trusted root certificates. Alternatively, usecurl -kto disable certificate verification.
Distro packages
AUR packages are available to Arch users: curl-impersonate-chrome, curl-impersonate-firefox.
Usage
It is recommended to use the wrapper scripts curl_chrome98 or curl_ff91esr, as they add all the correct headers and flags. For example:
curl_chrome98 https://www.google.com
You can add command line flags and they will be passed on to curl. However, some flags change curl's TLS signature which may cause it to be detected.
libcurl-impersonate
libcurl-impersonate.so is libcurl compiled with the same changes as the command line curl-impersonate.
It has an additional API function:
CURLcode curl_easy_impersonate(struct Curl_easy *data, const char *target);
You can call it with the target names, e.g. "chrome98", and it will internally set all the options and headers that are otherwise set by the wrapper scripts. Specifically it sets:
CURLOPT_HTTP_VERSIONCURLOPT_SSLVERSION,CURLOPT_SSL_CIPHER_LIST,CURLOPT_SSL_EC_CURVES,CURLOPT_SSL_ENABLE_NPN,CURLOPT_SSL_ENABLE_ALPNCURLOPT_HTTPBASEHEADER(non-standard HTTP option created for this project).CURLOPT_SSL_ENABLE_ALPS,CURLOPT_SSL_SIG_HASH_ALGS,CURLOPT_SSL_CERT_COMPRESSION,CURLOPT_SSL_ENABLE_TICKET(non-standard TLS options created for this project).
Note that if you call curl_easy_setopt() later with one of the above it will override the options set by curl_easy_impersonate().
Using CURL_IMPERSONATE env var
Experimental: If your application uses libcurl already, you can replace the existing library at runtime with LD_PRELOAD. You can then set the CURL_IMPERSONATE env var. For example:
LD_PRELOAD=/path/to/libcurl-impersonate.so CURL_IMPERSONATE=chrome98 my_app
The CURL_IMPERSONATE env var will cause curl_easy_impersonate() to be called automatically for any new curl handle created by curl_easy_init().
Note that the above will NOT WORK for curl itself because the curl tool overrides the TLS settings. Use the wrapper scripts instead.
Contents
This repository contains two main folders:
- chrome - Scripts and patches for building the Chrome version of
curl-impersonate. - firefox - Scripts and patches for building the Firefox version of
curl-impersonate.
The layout is similar for both. For example, the Firefox directory contains:
- Dockerfile - Used to build
curl-impersonatewith all dependencies. - curl_ff91esr, curl_ff95 - Wrapper scripts that launch
curl-impersonatewith the correct flags. - curl-impersonate.patch - The main patch that makes curl use the same TLS extensions as Firefox. Also makes curl compile statically with libnghttp2 and libnss.
- libnghttp2-pc.patch - Patch to make libnghttp2 compile statically.
Other files of interest:
- tests/signatures.yaml - YAML database of known browser signatures that can be impersonated.
Contributing
If you'd like to help, please check out the open issues. You can open a pull request with your changes.
This repository contains the build process for curl-impersonate. The actual patches to curl are maintained in a separate repository forked from the upstream curl. The changes are maintained in the impersonate-firefox and impersonate-chrome branches.