curl-impersonate ·

A special compilation of curl that makes it impersonate real browsers. It can impersonate the four major browsers: Chrome, Edge, Safari & Firefox. This curl binary is able to perform a TLS handshake that is identical to that of a real browser.

Why?

When you use an HTTP client with a TLS website, it first performs a TLS handshake. The first message of that handshake is called Client Hello. The Client Hello message that curl produces differs drastically from that of a real browser. Compare the following Wireshark capture. Left is a regular curl, right is Firefox.

Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block it. With the modified curl in this repository, the Client Hello message looks exactly like that of a real browser. This tricks TLS fingerprinters to think that it is a real browser that is accessing them.

How?

The modifications that were needed to make this work:

• Compiling curl with nss, the TLS library that Firefox uses, instead of OpenSSL. For the Chrome version, compiling with BoringSSL.
• Modifying the way curl configures various TLS extensions and SSL options.
• Adding support for new TLS extensions.
• Running curl with some non-default flags, for example --ciphers, --curves and some -H headers.

The resulting curl looks, from a network perspective, identical to a real browser. Compare: (left is curl-impersonate, right is Firefox):

Read the full description in the blog post: part a, part b.

Supported browsers

The following browsers can be impersonated.

Browser	Version	Build	OS	Target name	Wrapper script
	98	98.0.4758.102	Windows 10	chrome98	curl_chrome98
	99	99.0.4844.51	Windows 10	chrome99	curl_chrome99
	98	98.0.1108.62	Windows 10	edge98	curl_edge98
	99	99.0.1150.30	Windows 10	edge99	curl_edge99
	91 ESR	91.6.0esr	Windows 10	ff91esr	curl_ff91esr
	95	95.0.2	Windows 10	ff95	curl_ff95
	15.3	16612.4.9.1.8	MacOS Big Sur	safari15_3	curl_safari15_3

Basic usage

For each supported browser there is a wrapper script that launches curl-impersonate with all the needed headers and flags. For example:

curl_chrome99 https://www.wikipedia.org

You can add command line flags and they will be passed on to curl. However, some flags change curl's TLS signature which may cause it to be detected.

See Advanced usage for more options.

Installation

This repository maintains two separate build systems for technical reasons. The chrome build is used to impersonate Chrome, Edge and Safari. The firefox build is used to impersonate Firefox.

Chrome build

chrome/Dockerfile is a Dockerfile that will build curl with all the necessary modifications and patches. Build it like the following:

docker build -t curl-impersonate-chrome chrome/

The resulting image contains:

• /build/out/curl-impersonate - The curl binary that can impersonate Chrome/Edge/Safari. It is compiled statically against libcurl, BoringSSL, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04.
• /build/out/curl_chrome98, /build/out/curl_chrome99, ... - Wrapper scripts that launch curl-impersonate with all the needed flags.
• /build/out/libcurl-impersonate.so - libcurl compiled with impersonation support. See libcurl-impersonate below for more details.

You can use them inside the docker, copy them out using docker cp or use them in a multi-stage docker build.

Firefox build

Build with:

docker build -t curl-impersonate-ff firefox/

The resulting image contains:

• /build/out/curl-impersonate - The curl binary that can impersonate Firefox. It is compiled statically against libcurl, nss, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04.
• /build/out/curl_ff91esr, /build/out/curl_ff95 - Wrapper scripts that launch curl-impersonate with all the needed flags.
• /build/out/libcurl-impersonate.so - libcurl compiled with impersonation support. See libcurl-impersonate below for more details.

If you use it outside the container, install the following dependency:

• sudo apt install libnss3. Even though nss is statically compiled into curl-impersonate, it is still necessary to install libnss3 because curl dynamically loads libnssckbi.so, a file containing Mozilla's list of trusted root certificates. Alternatively, use curl -k to disable certificate verification.

Distro packages

AUR packages are available to Arch users: curl-impersonate-chrome, curl-impersonate-firefox.

Advanced usage

libcurl-impersonate

libcurl-impersonate.so is libcurl compiled with the same changes as the command line curl-impersonate. It has an additional API function:

CURLcode curl_easy_impersonate(struct Curl_easy *data, const char *target);

You can call it with the target names, e.g. chrome98, and it will internally set all the options and headers that are otherwise set by the wrapper scripts. Specifically it sets:

• CURLOPT_HTTP_VERSION
• CURLOPT_SSLVERSION, CURLOPT_SSL_CIPHER_LIST, CURLOPT_SSL_EC_CURVES, CURLOPT_SSL_ENABLE_NPN, CURLOPT_SSL_ENABLE_ALPN
• CURLOPT_HTTPBASEHEADER, CURLOPT_HTTP2_PSEUDO_HEADERS_ORDER (non-standard HTTP options created for this project).
• CURLOPT_SSL_ENABLE_ALPS, CURLOPT_SSL_SIG_HASH_ALGS, CURLOPT_SSL_CERT_COMPRESSION, CURLOPT_SSL_ENABLE_TICKET (non-standard TLS options created for this project).

Note that if you call curl_easy_setopt() later with one of the above it will override the options set by curl_easy_impersonate().

Using CURL_IMPERSONATE env var

Experimental: If your application uses libcurl already, you can replace the existing library at runtime with LD_PRELOAD. You can then set the CURL_IMPERSONATE env var. For example:

LD_PRELOAD=/path/to/libcurl-impersonate.so CURL_IMPERSONATE=chrome98 my_app

The CURL_IMPERSONATE env var will cause curl_easy_impersonate() to be called automatically for any new curl handle created by curl_easy_init().

Note that the above will NOT WORK for curl itself because the curl tool overrides the TLS settings. Use the wrapper scripts instead.

Contents

This repository contains two main folders:

• chrome - Scripts and patches for building the Chrome version of curl-impersonate.
• firefox - Scripts and patches for building the Firefox version of curl-impersonate.

The layout is similar for both. For example, the Firefox directory contains:

• Dockerfile - Used to build curl-impersonate with all dependencies.
• curl_ff91esr, curl_ff95 - Wrapper scripts that launch curl-impersonate with the correct flags.
• curl-impersonate.patch - The main patch that makes curl use the same TLS extensions as Firefox. Also makes curl compile statically with libnghttp2 and libnss.
• libnghttp2-pc.patch - Patch to make libnghttp2 compile statically.

Other files of interest:

• tests/signatures.yaml - YAML database of known browser signatures that can be impersonated.

Contributing

If you'd like to help, please check out the open issues. You can open a pull request with your changes.

This repository contains the build process for curl-impersonate. The actual patches to curl are maintained in a separate repository forked from the upstream curl. The changes are maintained in the impersonate-firefox and impersonate-chrome branches.