feat: initial release of llm-fingerprint-detector
TypeScript library + CLI implementing single-token behavioral fingerprinting of LLMs behind OpenAI-compatible APIs, per 'One Token Is Enough' (Bruckner, arXiv:2607.10252). - Zero runtime dependencies (Node >= 18.17 built-in fetch) - fingerprint / compare / verify API + llm-fingerprint CLI - Jensen-Shannon divergence over per-cell answer distributions - Reasoning-model aware (auto-disables hidden thinking channels) - Bundled sample references for 11 popular models (CC-BY-4.0, derived from the paper's public Zenodo dataset) - Unit + integration tests via node --test Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Deng-m1
@@ -0,0 +1,253 @@
|
||||
# llm-fingerprint-detector
|
||||
|
||||
[](LICENSE)
|
||||
[](package.json)
|
||||
[](package.json)
|
||||
[](https://arxiv.org/abs/2607.10252)
|
||||
|
||||
**Is that API really serving the model it claims?** Fingerprint and verify any OpenAI-compatible LLM endpoint from single-token output distributions — no logits, no weights, no privileged access. Just ~100–400 cheap one-word completions. Catches **model substitution**, silent quantization and server-side prompt injection by API resellers, gateways and aggregators.
|
||||
|
||||
An independent, open-source TypeScript implementation of:
|
||||
|
||||
> Tomáš Bruckner, **"One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions"**, [arXiv:2607.10252](https://arxiv.org/abs/2607.10252).
|
||||
> Dataset: [DOI 10.5281/zenodo.21278557](https://doi.org/10.5281/zenodo.21278557) (CC-BY-4.0) · Paper code: [DOI 10.5281/zenodo.21278793](https://doi.org/10.5281/zenodo.21278793) (MIT)
|
||||
|
||||
This package is **not affiliated with the paper's author** — it is a from-scratch implementation of the published method, engineered as a reusable library + CLI. If you use the method in research, cite the paper.
|
||||
|
||||
- **Zero runtime dependencies** — Node ≥ 18 built-in `fetch`, nothing else
|
||||
- **Library + CLI** — embed it, or run `llm-fingerprint verify` in CI
|
||||
- **Reasoning-model aware** — auto-detects how to disable hidden "thinking" (OpenRouter / Zhipu / OpenAI-style), with a graceful fallback
|
||||
- **Filter-resistant probes** — every probe is a plain semantic question drawn from a paraphrase pool; there is no magic string a gateway can special-case
|
||||
- **Bundled sample references** for 11 popular models, derived from the paper's public dataset
|
||||
|
||||
> Prefer a no-install web version? Try the free online checker: **[tosea.ai/free-tools/llm-api-fingerprint-checker](https://tosea.ai/free-tools/llm-api-fingerprint-checker)** — same method, runs entirely in your browser.
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
LLMs answer "*Name a random number between 1 and 100*" with model-specific, surprisingly stable biases (GPT-family models love 42 and 73; other families prefer 57, 37, 7…). The paper's key result: the **empirical distribution of one-word answers** across a small battery of such tasks is a reliable *behavioral fingerprint* — stable across time, load and providers for the same model, and clearly different between models.
|
||||
|
||||
```
|
||||
probe battery (task × language cells) collect at temperature 1.0
|
||||
┌──────────────────────────────┐ ┌─────────────────────────┐
|
||||
│ random number 1-100 (en/zh) │ │ "42" ×19 "73" ×4 ... │
|
||||
│ random color / letter / city │ ──25×──▶ │ per-cell answer │
|
||||
│ coin flip / animal / fav-num │ │ distributions │
|
||||
└──────────────────────────────┘ └───────────┬─────────────┘
|
||||
▼
|
||||
reference fingerprint ──── mean per-cell Jensen-Shannon
|
||||
(trusted endpoint) divergence (base 2) ──▶ verdict
|
||||
```
|
||||
|
||||
1. **Probe** — ask one-word questions (random numbers, colors, letters, coin flips…) in English and Chinese, `temperature=1.0`, `max_tokens=16`, a fixed minimal system prompt, hidden reasoning disabled. Requests are shuffled and paraphrased per call.
|
||||
2. **Normalize** — NFC, punctuation/emoji stripping, case folding, first word, digit unification (`seven`/`七`/`٧`/`７` → `7`), color and coin-word canonicalization; answers are classified valid / invalid / refusal / empty.
|
||||
3. **Compare** — per-cell Jensen-Shannon divergence (base 2, range 0–1 bit), averaged over cells where both sides have ≥10 valid samples.
|
||||
4. **Verdict** — three bands calibrated against the paper's baselines (see [Interpreting results](#interpreting-results)).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install llm-fingerprint-detector # library + `llm-fingerprint` CLI
|
||||
# or run the CLI ad hoc:
|
||||
npx llm-fingerprint-detector --help
|
||||
```
|
||||
|
||||
Requires Node ≥ 18.17 (built-in `fetch`). The core library is runtime-agnostic and also works in browsers/edge runtimes; only the CLI and the bundled-reference loader touch the filesystem.
|
||||
|
||||
## Quick start — CLI
|
||||
|
||||
```bash
|
||||
# 1. Fingerprint the endpoint you trust (key read from OPENAI_API_KEY)
|
||||
export OPENAI_API_KEY=sk-...
|
||||
llm-fingerprint fingerprint \
|
||||
--base-url https://api.openai.com/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--out reference.gpt-4o-mini.json
|
||||
|
||||
# 2. Verify the endpoint you don't
|
||||
export LLM_FINGERPRINT_API_KEY=sk-... # key for the endpoint under test
|
||||
llm-fingerprint verify \
|
||||
--base-url https://cheap-llm-reseller.example.com/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--reference reference.gpt-4o-mini.json
|
||||
```
|
||||
|
||||
```
|
||||
Verdict: MISMATCH — behavior differs from the reference
|
||||
Mean JSD: 0.481 over 8 comparable cell(s)
|
||||
|
||||
Interpretation scale (paper baselines, arXiv:2607.10252):
|
||||
same model ≈ 0.14 · same model, other provider ≈ 0.227 · different model ≈ 0.463
|
||||
thresholds: match ≤ 0.25 < uncertain ≤ 0.35 < mismatch
|
||||
|
||||
Per-cell JSD (most divergent first):
|
||||
random-number-1-100:en 0.712 (24 vs 25 valid)
|
||||
...
|
||||
```
|
||||
|
||||
Exit codes are CI-friendly: `0` match · `2` mismatch · `3` uncertain · `4` insufficient · `1` error. API keys are only ever read from environment variables (`--api-key-env NAME`, defaulting to `LLM_FINGERPRINT_API_KEY` then `OPENAI_API_KEY`) and are never logged.
|
||||
|
||||
More: `llm-fingerprint --help`, [`examples/cli-examples.sh`](examples/cli-examples.sh).
|
||||
|
||||
## Quick start — library
|
||||
|
||||
```ts
|
||||
import { fingerprint, compare, verify } from 'llm-fingerprint-detector'
|
||||
|
||||
// Collect a fingerprint
|
||||
const run = await fingerprint(
|
||||
{ baseUrl: 'https://api.openai.com/v1', model: 'gpt-4o-mini', apiKey: process.env.OPENAI_API_KEY },
|
||||
{ cells: 8, samplesPerCell: 25, onProgress: (e) => console.log(e.done, '/', e.total) },
|
||||
)
|
||||
console.log(run.fingerprint) // JSON-serializable artifact
|
||||
console.log(run.splitHalfJsd) // self-consistency (≈0.14 is normal)
|
||||
|
||||
// Verify another endpoint against it
|
||||
const result = await verify(
|
||||
{ baseUrl: 'https://suspect.example.com/v1', model: 'gpt-4o-mini', apiKey: process.env.SUSPECT_KEY },
|
||||
run.fingerprint,
|
||||
)
|
||||
console.log(result.verdict, result.meanJsd) // 'match' | 'uncertain' | 'mismatch' | 'insufficient'
|
||||
|
||||
// Or compare two saved fingerprints offline
|
||||
const distance = compare(fingerprintA, fingerprintB)
|
||||
```
|
||||
|
||||
Bundled sample references (Node only):
|
||||
|
||||
```ts
|
||||
import { listBundledReferences, loadBundledReference } from 'llm-fingerprint-detector/references'
|
||||
|
||||
const reference = loadBundledReference('openai/gpt-4o-mini')
|
||||
const result = await verify(suspectEndpoint, reference)
|
||||
```
|
||||
|
||||
Runnable examples: [`examples/01-fingerprint-endpoint.mjs`](examples/01-fingerprint-endpoint.mjs), [`examples/02-verify-endpoint.mjs`](examples/02-verify-endpoint.mjs).
|
||||
|
||||
## Tutorial: "Is my cheap API really GPT-4o / Claude / DeepSeek?"
|
||||
|
||||
You bought API access from a reseller/aggregator at half price. Are you getting the real model, a cheaper substitute, or a quantized clone? Ten minutes:
|
||||
|
||||
1. **Collect a trusted reference.** Fingerprint the *official* API (or any endpoint you fully trust) for the model in question:
|
||||
|
||||
```bash
|
||||
llm-fingerprint fingerprint --base-url https://api.openai.com/v1 \
|
||||
--model gpt-4o-mini --api-key-env OFFICIAL_KEY --out ref.json
|
||||
```
|
||||
|
||||
No official access? Start with a bundled sample (`llm-fingerprint references`) — good enough for a first signal, with the caveats below.
|
||||
|
||||
2. **Verify the suspect endpoint** with the *same* model id:
|
||||
|
||||
```bash
|
||||
llm-fingerprint verify --base-url https://reseller.example.com/v1 \
|
||||
--model gpt-4o-mini --api-key-env RESELLER_KEY --reference ref.json
|
||||
```
|
||||
|
||||
3. **Read the verdict.**
|
||||
- `match` — the endpoint's one-token behavior is statistically consistent with your reference. That is strong (not absolute) evidence it's the same model.
|
||||
- `mismatch` — the behavior is as far from the reference as *different* models typically are. Common causes, in practice: a substituted cheaper model, a heavily quantized deployment, or an injected system prompt.
|
||||
- `uncertain` — the gray zone. Raise `--samples` (e.g. 40), use `--preset strict` (all 16 cells), or refresh your reference — models drift when providers ship updates.
|
||||
- Also watch the warnings: a high **split-half JSD** means the endpoint disagrees *with itself* between the first and second half of your own run — a classic sign of an aggregator rotating several backends.
|
||||
|
||||
4. **Re-run before you conclude anything.** Two independent mismatch runs on different days are a much stronger signal than one.
|
||||
|
||||
## Interpreting results
|
||||
|
||||
Distances are mean Jensen-Shannon divergence (base 2), so 0 = identical behavior, 1 = disjoint answer sets. Paper baselines:
|
||||
|
||||
| meanJsd | reading |
|
||||
|---|---|
|
||||
| ≈ 0.14 | same model, same endpoint (sampling noise floor) |
|
||||
| ≈ 0.227 | same model, different provider (median) |
|
||||
| **≤ 0.25** | → verdict **match** |
|
||||
| 0.25 – 0.35 | → verdict **uncertain** |
|
||||
| **> 0.35** | → verdict **mismatch** |
|
||||
| ≈ 0.463 | different models (median) |
|
||||
|
||||
**Error rates (paper):** distinguishing same-model vs different-model pairs achieves an equal error rate of ≈ **10.6% with 8 cells** and ≈ **7.3% with 40 cells**. A single run is *evidence*, not proof — treat verdicts accordingly.
|
||||
|
||||
### Limitations you must know
|
||||
|
||||
- **Fingerprints drift.** Providers silently update models; a 3-month-old reference can legitimately mismatch today's deployment. Always check `collectedAt`, refresh references regularly.
|
||||
- **You need a trusted reference.** The method compares two endpoints; it cannot conjure ground truth. If your reference is wrong, your verdict is wrong.
|
||||
- **The system prompt must be identical on both sides.** Swapping only the system prompt shifts fingerprints by JSD ≈ 0.44–0.46 — the magnitude of a model swap. This tool pins the same minimal system prompt on both sides automatically; if an endpoint *injects* its own server-side prompt, that will (correctly) surface as divergence.
|
||||
- **Reasoning fallback lowers confidence.** When hidden reasoning can't be disabled, the run is flagged `postReasoning` — distributions shift measurably in that channel.
|
||||
- **Quantization/serving stack changes** the same weights can move distances into the uncertain band.
|
||||
- **A mismatch is a statistical observation, not an accusation.** Do not treat any single verdict as proof of fraud by a provider; investigate, re-run, and compare notes before drawing conclusions.
|
||||
|
||||
## Bundled sample references
|
||||
|
||||
`data/reference-fingerprints.sample.json` ships fingerprints of 11 popular models (GPT-4o, GPT-4o-mini, GPT-4.1-mini, Claude Sonnet 4.5, Gemini 2.5 Flash, DeepSeek-chat, Llama-3.1-8B, Qwen3-30B-A3B, Mistral Small 3.2, GLM-4.5, Kimi K2), derived from the paper's public dataset:
|
||||
|
||||
> Bruckner, T. (2026). *Single-token output distributions as behavioral fingerprints of large language models* [Data set]. Zenodo. [https://doi.org/10.5281/zenodo.21278557](https://doi.org/10.5281/zenodo.21278557) — CC-BY-4.0. Counts reconstructed from the published per-cell distributions and re-normalized with this package's normalizer (see [`scripts/build-sample-references.mjs`](scripts/build-sample-references.mjs)).
|
||||
|
||||
Those samples were collected by the paper's harness (via OpenRouter) under the paper's prompt protocol — close to, but not identical to, this package's battery. `compare()` flags such pairs with `protocolMismatch: true` and the verdict should be read as *indicative*. For anything that matters, collect your own reference:
|
||||
|
||||
```bash
|
||||
llm-fingerprint fingerprint --base-url <trusted-url> --model <id> --out my-reference.json
|
||||
```
|
||||
|
||||
To rebuild or extend the bundled samples from the Zenodo dataset, download the dataset, then:
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
node scripts/build-sample-references.mjs path/to/distributions.json --models openai/gpt-4o,another/model
|
||||
```
|
||||
|
||||
## API surface (TypeScript, fully typed)
|
||||
|
||||
| export | what it does |
|
||||
|---|---|
|
||||
| `fingerprint(endpoint, options?)` | probe an endpoint → `FingerprintRun` (fingerprint, adapter, split-half, warnings) |
|
||||
| `compare(a, b)` | two fingerprints → `ComparisonResult` (meanJsd, per-cell JSD, verdict, baselines) |
|
||||
| `verify(endpoint, reference, options?)` | fingerprint + compare in one call → `VerifyResult` |
|
||||
| `normalizeAnswer(raw, domain)` | the full normalization pipeline (pure, unit-tested) |
|
||||
| `jensenShannonDivergence(p, q)` | base-2 JSD over count maps |
|
||||
| `splitHalfJsd(samplesByCell)` | endpoint self-consistency check |
|
||||
| `detectReasoningAdapter(endpoint)` | which reasoning-disable field the endpoint accepts |
|
||||
| `PROBE_TASKS`, `CELL_PRIORITY_ORDER`, `SYSTEM_PROMPTS` | the battery itself |
|
||||
| `llm-fingerprint-detector/references` | bundled sample loader (Node only) |
|
||||
|
||||
All options (`cells`, `samplesPerCell`, `concurrency`, `timeoutMs`, `maxRetries`, `signal`, `onProgress`, …) are documented in [`src/types.ts`](src/types.ts).
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run build # tsc → dist/
|
||||
npm test # builds, then runs node --test against the built output
|
||||
```
|
||||
|
||||
No test framework, no bundler — TypeScript and the Node built-in test runner only.
|
||||
|
||||
## Contributing
|
||||
|
||||
Issues and PRs are welcome. Especially valuable:
|
||||
|
||||
- **More languages** in the probe battery (the paper also used Arabic and Russian) — requires matching normalizer support, see `src/normalizer.ts`
|
||||
- **Reference fingerprints** for more models/providers, collected with this tool's protocol (`one-token/v1`) and a documented date/channel
|
||||
- **Threshold calibration data**: pairs of same-model / different-model runs to sharpen the match/mismatch cut points
|
||||
|
||||
Please keep changes dependency-free and covered by `node --test` tests.
|
||||
|
||||
## Citation & license
|
||||
|
||||
Method: cite the paper —
|
||||
|
||||
```bibtex
|
||||
@article{bruckner2026onetoken,
|
||||
title = {One Token Is Enough: Fingerprinting and Verifying Large Language Models
|
||||
from Single-Token Output Distributions},
|
||||
author = {Bruckner, Tom{\'a}{\v s}},
|
||||
journal = {arXiv preprint arXiv:2607.10252},
|
||||
year = {2026}
|
||||
}
|
||||
```
|
||||
|
||||
This package: [MIT](LICENSE). Bundled sample data: CC-BY-4.0, © Tomáš Bruckner (see above).
|
||||
|
||||
---
|
||||
|
||||
*Built and maintained by [Tosea.ai](https://tosea.ai). Want the zero-setup version? → [LLM API Fingerprint Checker](https://tosea.ai/free-tools/llm-api-fingerprint-checker) (runs in your browser, your key never leaves it).*
|
||||
Reference in New Issue
Block a user