feat: initial release of llm-fingerprint-detector
TypeScript library + CLI implementing single-token behavioral fingerprinting of LLMs behind OpenAI-compatible APIs, per 'One Token Is Enough' (Bruckner, arXiv:2607.10252). - Zero runtime dependencies (Node >= 18.17 built-in fetch) - fingerprint / compare / verify API + llm-fingerprint CLI - Jensen-Shannon divergence over per-cell answer distributions - Reasoning-model aware (auto-disables hidden thinking channels) - Bundled sample references for 11 popular models (CC-BY-4.0, derived from the paper's public Zenodo dataset) - Unit + integration tests via node --test Co-authored-by: Cursor <[email protected]>
This commit is contained in:
+10
@@ -0,0 +1,10 @@
|
||||
node_modules/
|
||||
dist/
|
||||
*.tsbuildinfo
|
||||
.env
|
||||
.env.*
|
||||
*.log
|
||||
.DS_Store
|
||||
coverage/
|
||||
my-fingerprints/
|
||||
*.fingerprint.json
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Published files are whitelisted via the "files" field in package.json;
|
||||
# this file is a safety net in case "files" is ever removed.
|
||||
node_modules/
|
||||
test/
|
||||
examples/
|
||||
scripts/
|
||||
src/
|
||||
tsconfig.json
|
||||
*.tsbuildinfo
|
||||
.env
|
||||
.env.*
|
||||
*.log
|
||||
coverage/
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Tosea.ai and llm-fingerprint-detector contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,253 @@
|
||||
# llm-fingerprint-detector
|
||||
|
||||
[](LICENSE)
|
||||
[](package.json)
|
||||
[](package.json)
|
||||
[](https://arxiv.org/abs/2607.10252)
|
||||
|
||||
**Is that API really serving the model it claims?** Fingerprint and verify any OpenAI-compatible LLM endpoint from single-token output distributions — no logits, no weights, no privileged access. Just ~100–400 cheap one-word completions. Catches **model substitution**, silent quantization and server-side prompt injection by API resellers, gateways and aggregators.
|
||||
|
||||
An independent, open-source TypeScript implementation of:
|
||||
|
||||
> Tomáš Bruckner, **"One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions"**, [arXiv:2607.10252](https://arxiv.org/abs/2607.10252).
|
||||
> Dataset: [DOI 10.5281/zenodo.21278557](https://doi.org/10.5281/zenodo.21278557) (CC-BY-4.0) · Paper code: [DOI 10.5281/zenodo.21278793](https://doi.org/10.5281/zenodo.21278793) (MIT)
|
||||
|
||||
This package is **not affiliated with the paper's author** — it is a from-scratch implementation of the published method, engineered as a reusable library + CLI. If you use the method in research, cite the paper.
|
||||
|
||||
- **Zero runtime dependencies** — Node ≥ 18 built-in `fetch`, nothing else
|
||||
- **Library + CLI** — embed it, or run `llm-fingerprint verify` in CI
|
||||
- **Reasoning-model aware** — auto-detects how to disable hidden "thinking" (OpenRouter / Zhipu / OpenAI-style), with a graceful fallback
|
||||
- **Filter-resistant probes** — every probe is a plain semantic question drawn from a paraphrase pool; there is no magic string a gateway can special-case
|
||||
- **Bundled sample references** for 11 popular models, derived from the paper's public dataset
|
||||
|
||||
> Prefer a no-install web version? Try the free online checker: **[tosea.ai/free-tools/llm-api-fingerprint-checker](https://tosea.ai/free-tools/llm-api-fingerprint-checker)** — same method, runs entirely in your browser.
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
LLMs answer "*Name a random number between 1 and 100*" with model-specific, surprisingly stable biases (GPT-family models love 42 and 73; other families prefer 57, 37, 7…). The paper's key result: the **empirical distribution of one-word answers** across a small battery of such tasks is a reliable *behavioral fingerprint* — stable across time, load and providers for the same model, and clearly different between models.
|
||||
|
||||
```
|
||||
probe battery (task × language cells) collect at temperature 1.0
|
||||
┌──────────────────────────────┐ ┌─────────────────────────┐
|
||||
│ random number 1-100 (en/zh) │ │ "42" ×19 "73" ×4 ... │
|
||||
│ random color / letter / city │ ──25×──▶ │ per-cell answer │
|
||||
│ coin flip / animal / fav-num │ │ distributions │
|
||||
└──────────────────────────────┘ └───────────┬─────────────┘
|
||||
▼
|
||||
reference fingerprint ──── mean per-cell Jensen-Shannon
|
||||
(trusted endpoint) divergence (base 2) ──▶ verdict
|
||||
```
|
||||
|
||||
1. **Probe** — ask one-word questions (random numbers, colors, letters, coin flips…) in English and Chinese, `temperature=1.0`, `max_tokens=16`, a fixed minimal system prompt, hidden reasoning disabled. Requests are shuffled and paraphrased per call.
|
||||
2. **Normalize** — NFC, punctuation/emoji stripping, case folding, first word, digit unification (`seven`/`七`/`٧`/`7` → `7`), color and coin-word canonicalization; answers are classified valid / invalid / refusal / empty.
|
||||
3. **Compare** — per-cell Jensen-Shannon divergence (base 2, range 0–1 bit), averaged over cells where both sides have ≥10 valid samples.
|
||||
4. **Verdict** — three bands calibrated against the paper's baselines (see [Interpreting results](#interpreting-results)).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install llm-fingerprint-detector # library + `llm-fingerprint` CLI
|
||||
# or run the CLI ad hoc:
|
||||
npx llm-fingerprint-detector --help
|
||||
```
|
||||
|
||||
Requires Node ≥ 18.17 (built-in `fetch`). The core library is runtime-agnostic and also works in browsers/edge runtimes; only the CLI and the bundled-reference loader touch the filesystem.
|
||||
|
||||
## Quick start — CLI
|
||||
|
||||
```bash
|
||||
# 1. Fingerprint the endpoint you trust (key read from OPENAI_API_KEY)
|
||||
export OPENAI_API_KEY=sk-...
|
||||
llm-fingerprint fingerprint \
|
||||
--base-url https://api.openai.com/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--out reference.gpt-4o-mini.json
|
||||
|
||||
# 2. Verify the endpoint you don't
|
||||
export LLM_FINGERPRINT_API_KEY=sk-... # key for the endpoint under test
|
||||
llm-fingerprint verify \
|
||||
--base-url https://cheap-llm-reseller.example.com/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--reference reference.gpt-4o-mini.json
|
||||
```
|
||||
|
||||
```
|
||||
Verdict: MISMATCH — behavior differs from the reference
|
||||
Mean JSD: 0.481 over 8 comparable cell(s)
|
||||
|
||||
Interpretation scale (paper baselines, arXiv:2607.10252):
|
||||
same model ≈ 0.14 · same model, other provider ≈ 0.227 · different model ≈ 0.463
|
||||
thresholds: match ≤ 0.25 < uncertain ≤ 0.35 < mismatch
|
||||
|
||||
Per-cell JSD (most divergent first):
|
||||
random-number-1-100:en 0.712 (24 vs 25 valid)
|
||||
...
|
||||
```
|
||||
|
||||
Exit codes are CI-friendly: `0` match · `2` mismatch · `3` uncertain · `4` insufficient · `1` error. API keys are only ever read from environment variables (`--api-key-env NAME`, defaulting to `LLM_FINGERPRINT_API_KEY` then `OPENAI_API_KEY`) and are never logged.
|
||||
|
||||
More: `llm-fingerprint --help`, [`examples/cli-examples.sh`](examples/cli-examples.sh).
|
||||
|
||||
## Quick start — library
|
||||
|
||||
```ts
|
||||
import { fingerprint, compare, verify } from 'llm-fingerprint-detector'
|
||||
|
||||
// Collect a fingerprint
|
||||
const run = await fingerprint(
|
||||
{ baseUrl: 'https://api.openai.com/v1', model: 'gpt-4o-mini', apiKey: process.env.OPENAI_API_KEY },
|
||||
{ cells: 8, samplesPerCell: 25, onProgress: (e) => console.log(e.done, '/', e.total) },
|
||||
)
|
||||
console.log(run.fingerprint) // JSON-serializable artifact
|
||||
console.log(run.splitHalfJsd) // self-consistency (≈0.14 is normal)
|
||||
|
||||
// Verify another endpoint against it
|
||||
const result = await verify(
|
||||
{ baseUrl: 'https://suspect.example.com/v1', model: 'gpt-4o-mini', apiKey: process.env.SUSPECT_KEY },
|
||||
run.fingerprint,
|
||||
)
|
||||
console.log(result.verdict, result.meanJsd) // 'match' | 'uncertain' | 'mismatch' | 'insufficient'
|
||||
|
||||
// Or compare two saved fingerprints offline
|
||||
const distance = compare(fingerprintA, fingerprintB)
|
||||
```
|
||||
|
||||
Bundled sample references (Node only):
|
||||
|
||||
```ts
|
||||
import { listBundledReferences, loadBundledReference } from 'llm-fingerprint-detector/references'
|
||||
|
||||
const reference = loadBundledReference('openai/gpt-4o-mini')
|
||||
const result = await verify(suspectEndpoint, reference)
|
||||
```
|
||||
|
||||
Runnable examples: [`examples/01-fingerprint-endpoint.mjs`](examples/01-fingerprint-endpoint.mjs), [`examples/02-verify-endpoint.mjs`](examples/02-verify-endpoint.mjs).
|
||||
|
||||
## Tutorial: "Is my cheap API really GPT-4o / Claude / DeepSeek?"
|
||||
|
||||
You bought API access from a reseller/aggregator at half price. Are you getting the real model, a cheaper substitute, or a quantized clone? Ten minutes:
|
||||
|
||||
1. **Collect a trusted reference.** Fingerprint the *official* API (or any endpoint you fully trust) for the model in question:
|
||||
|
||||
```bash
|
||||
llm-fingerprint fingerprint --base-url https://api.openai.com/v1 \
|
||||
--model gpt-4o-mini --api-key-env OFFICIAL_KEY --out ref.json
|
||||
```
|
||||
|
||||
No official access? Start with a bundled sample (`llm-fingerprint references`) — good enough for a first signal, with the caveats below.
|
||||
|
||||
2. **Verify the suspect endpoint** with the *same* model id:
|
||||
|
||||
```bash
|
||||
llm-fingerprint verify --base-url https://reseller.example.com/v1 \
|
||||
--model gpt-4o-mini --api-key-env RESELLER_KEY --reference ref.json
|
||||
```
|
||||
|
||||
3. **Read the verdict.**
|
||||
- `match` — the endpoint's one-token behavior is statistically consistent with your reference. That is strong (not absolute) evidence it's the same model.
|
||||
- `mismatch` — the behavior is as far from the reference as *different* models typically are. Common causes, in practice: a substituted cheaper model, a heavily quantized deployment, or an injected system prompt.
|
||||
- `uncertain` — the gray zone. Raise `--samples` (e.g. 40), use `--preset strict` (all 16 cells), or refresh your reference — models drift when providers ship updates.
|
||||
- Also watch the warnings: a high **split-half JSD** means the endpoint disagrees *with itself* between the first and second half of your own run — a classic sign of an aggregator rotating several backends.
|
||||
|
||||
4. **Re-run before you conclude anything.** Two independent mismatch runs on different days are a much stronger signal than one.
|
||||
|
||||
## Interpreting results
|
||||
|
||||
Distances are mean Jensen-Shannon divergence (base 2), so 0 = identical behavior, 1 = disjoint answer sets. Paper baselines:
|
||||
|
||||
| meanJsd | reading |
|
||||
|---|---|
|
||||
| ≈ 0.14 | same model, same endpoint (sampling noise floor) |
|
||||
| ≈ 0.227 | same model, different provider (median) |
|
||||
| **≤ 0.25** | → verdict **match** |
|
||||
| 0.25 – 0.35 | → verdict **uncertain** |
|
||||
| **> 0.35** | → verdict **mismatch** |
|
||||
| ≈ 0.463 | different models (median) |
|
||||
|
||||
**Error rates (paper):** distinguishing same-model vs different-model pairs achieves an equal error rate of ≈ **10.6% with 8 cells** and ≈ **7.3% with 40 cells**. A single run is *evidence*, not proof — treat verdicts accordingly.
|
||||
|
||||
### Limitations you must know
|
||||
|
||||
- **Fingerprints drift.** Providers silently update models; a 3-month-old reference can legitimately mismatch today's deployment. Always check `collectedAt`, refresh references regularly.
|
||||
- **You need a trusted reference.** The method compares two endpoints; it cannot conjure ground truth. If your reference is wrong, your verdict is wrong.
|
||||
- **The system prompt must be identical on both sides.** Swapping only the system prompt shifts fingerprints by JSD ≈ 0.44–0.46 — the magnitude of a model swap. This tool pins the same minimal system prompt on both sides automatically; if an endpoint *injects* its own server-side prompt, that will (correctly) surface as divergence.
|
||||
- **Reasoning fallback lowers confidence.** When hidden reasoning can't be disabled, the run is flagged `postReasoning` — distributions shift measurably in that channel.
|
||||
- **Quantization/serving stack changes** the same weights can move distances into the uncertain band.
|
||||
- **A mismatch is a statistical observation, not an accusation.** Do not treat any single verdict as proof of fraud by a provider; investigate, re-run, and compare notes before drawing conclusions.
|
||||
|
||||
## Bundled sample references
|
||||
|
||||
`data/reference-fingerprints.sample.json` ships fingerprints of 11 popular models (GPT-4o, GPT-4o-mini, GPT-4.1-mini, Claude Sonnet 4.5, Gemini 2.5 Flash, DeepSeek-chat, Llama-3.1-8B, Qwen3-30B-A3B, Mistral Small 3.2, GLM-4.5, Kimi K2), derived from the paper's public dataset:
|
||||
|
||||
> Bruckner, T. (2026). *Single-token output distributions as behavioral fingerprints of large language models* [Data set]. Zenodo. [https://doi.org/10.5281/zenodo.21278557](https://doi.org/10.5281/zenodo.21278557) — CC-BY-4.0. Counts reconstructed from the published per-cell distributions and re-normalized with this package's normalizer (see [`scripts/build-sample-references.mjs`](scripts/build-sample-references.mjs)).
|
||||
|
||||
Those samples were collected by the paper's harness (via OpenRouter) under the paper's prompt protocol — close to, but not identical to, this package's battery. `compare()` flags such pairs with `protocolMismatch: true` and the verdict should be read as *indicative*. For anything that matters, collect your own reference:
|
||||
|
||||
```bash
|
||||
llm-fingerprint fingerprint --base-url <trusted-url> --model <id> --out my-reference.json
|
||||
```
|
||||
|
||||
To rebuild or extend the bundled samples from the Zenodo dataset, download the dataset, then:
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
node scripts/build-sample-references.mjs path/to/distributions.json --models openai/gpt-4o,another/model
|
||||
```
|
||||
|
||||
## API surface (TypeScript, fully typed)
|
||||
|
||||
| export | what it does |
|
||||
|---|---|
|
||||
| `fingerprint(endpoint, options?)` | probe an endpoint → `FingerprintRun` (fingerprint, adapter, split-half, warnings) |
|
||||
| `compare(a, b)` | two fingerprints → `ComparisonResult` (meanJsd, per-cell JSD, verdict, baselines) |
|
||||
| `verify(endpoint, reference, options?)` | fingerprint + compare in one call → `VerifyResult` |
|
||||
| `normalizeAnswer(raw, domain)` | the full normalization pipeline (pure, unit-tested) |
|
||||
| `jensenShannonDivergence(p, q)` | base-2 JSD over count maps |
|
||||
| `splitHalfJsd(samplesByCell)` | endpoint self-consistency check |
|
||||
| `detectReasoningAdapter(endpoint)` | which reasoning-disable field the endpoint accepts |
|
||||
| `PROBE_TASKS`, `CELL_PRIORITY_ORDER`, `SYSTEM_PROMPTS` | the battery itself |
|
||||
| `llm-fingerprint-detector/references` | bundled sample loader (Node only) |
|
||||
|
||||
All options (`cells`, `samplesPerCell`, `concurrency`, `timeoutMs`, `maxRetries`, `signal`, `onProgress`, …) are documented in [`src/types.ts`](src/types.ts).
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run build # tsc → dist/
|
||||
npm test # builds, then runs node --test against the built output
|
||||
```
|
||||
|
||||
No test framework, no bundler — TypeScript and the Node built-in test runner only.
|
||||
|
||||
## Contributing
|
||||
|
||||
Issues and PRs are welcome. Especially valuable:
|
||||
|
||||
- **More languages** in the probe battery (the paper also used Arabic and Russian) — requires matching normalizer support, see `src/normalizer.ts`
|
||||
- **Reference fingerprints** for more models/providers, collected with this tool's protocol (`one-token/v1`) and a documented date/channel
|
||||
- **Threshold calibration data**: pairs of same-model / different-model runs to sharpen the match/mismatch cut points
|
||||
|
||||
Please keep changes dependency-free and covered by `node --test` tests.
|
||||
|
||||
## Citation & license
|
||||
|
||||
Method: cite the paper —
|
||||
|
||||
```bibtex
|
||||
@article{bruckner2026onetoken,
|
||||
title = {One Token Is Enough: Fingerprinting and Verifying Large Language Models
|
||||
from Single-Token Output Distributions},
|
||||
author = {Bruckner, Tom{\'a}{\v s}},
|
||||
journal = {arXiv preprint arXiv:2607.10252},
|
||||
year = {2026}
|
||||
}
|
||||
```
|
||||
|
||||
This package: [MIT](LICENSE). Bundled sample data: CC-BY-4.0, © Tomáš Bruckner (see above).
|
||||
|
||||
---
|
||||
|
||||
*Built and maintained by [Tosea.ai](https://tosea.ai). Want the zero-setup version? → [LLM API Fingerprint Checker](https://tosea.ai/free-tools/llm-api-fingerprint-checker) (runs in your browser, your key never leaves it).*
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,48 @@
|
||||
/**
|
||||
* Example 1 — collect a fingerprint from an endpoint you trust and save it
|
||||
* as a reference for later verification.
|
||||
*
|
||||
* Run (from the package root, after `npm run build`):
|
||||
* OPENAI_API_KEY=sk-... node examples/01-fingerprint-endpoint.mjs
|
||||
*
|
||||
* With the published package, import from 'llm-fingerprint-detector' instead
|
||||
* of '../dist/index.js'.
|
||||
*/
|
||||
|
||||
import { writeFileSync } from 'node:fs'
|
||||
|
||||
import { fingerprint } from '../dist/index.js'
|
||||
|
||||
const apiKey = process.env.OPENAI_API_KEY
|
||||
if (!apiKey) {
|
||||
console.error('Set OPENAI_API_KEY (the key is only sent to the endpoint you probe).')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
const run = await fingerprint(
|
||||
{
|
||||
baseUrl: 'https://api.openai.com/v1', // any OpenAI-compatible endpoint
|
||||
model: 'gpt-4o-mini',
|
||||
apiKey,
|
||||
},
|
||||
{
|
||||
cells: 8, // top-8 most discriminative cells (= "standard" preset)
|
||||
samplesPerCell: 25,
|
||||
concurrency: 4,
|
||||
onProgress: (e) => {
|
||||
if (e.stage === 'sampling') process.stderr.write(`\r${e.done}/${e.total}`)
|
||||
},
|
||||
},
|
||||
)
|
||||
process.stderr.write('\n')
|
||||
|
||||
for (const warning of run.warnings) console.warn(`warning: ${warning}`)
|
||||
|
||||
console.log(`model: ${run.fingerprint.model}`)
|
||||
console.log(`cells: ${Object.keys(run.fingerprint.cells).length}`)
|
||||
console.log(`errors: ${run.errorCount}`)
|
||||
console.log(`split-half JSD: ${run.splitHalfJsd?.toFixed(3) ?? 'n/a'} (same-model baseline ≈ 0.14)`)
|
||||
console.log(`reasoning adapter: ${run.adapter.strategy}`)
|
||||
|
||||
writeFileSync('gpt-4o-mini.fingerprint.json', JSON.stringify(run.fingerprint, null, 2))
|
||||
console.log('\nsaved to gpt-4o-mini.fingerprint.json — use it later with verify()')
|
||||
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* Example 2 — verify that a cheap/unknown endpoint really serves the model it
|
||||
* claims, by comparing its live fingerprint against a reference.
|
||||
*
|
||||
* Run (from the package root, after `npm run build`):
|
||||
* SUSPECT_API_KEY=sk-... node examples/02-verify-endpoint.mjs
|
||||
*
|
||||
* The reference here is a bundled sample derived from the paper's public
|
||||
* dataset (CC-BY-4.0) — handy for a demo. For real audits, collect your own
|
||||
* reference from the official API with example 01 and load that file instead.
|
||||
*/
|
||||
|
||||
import { verify } from '../dist/index.js'
|
||||
import { loadBundledReference } from '../dist/reference.js'
|
||||
|
||||
const apiKey = process.env.SUSPECT_API_KEY
|
||||
if (!apiKey) {
|
||||
console.error('Set SUSPECT_API_KEY for the endpoint you want to test.')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// Endpoint under test: does it really serve gpt-4o-mini?
|
||||
const suspectEndpoint = {
|
||||
baseUrl: 'https://openrouter.ai/api/v1', // ← put the reseller/aggregator URL here
|
||||
model: 'openai/gpt-4o-mini',
|
||||
apiKey,
|
||||
}
|
||||
|
||||
const reference = loadBundledReference('openai/gpt-4o-mini')
|
||||
|
||||
const result = await verify(suspectEndpoint, reference, {
|
||||
cells: 8,
|
||||
samplesPerCell: 25,
|
||||
onProgress: (e) => {
|
||||
if (e.stage === 'sampling') process.stderr.write(`\r${e.done}/${e.total}`)
|
||||
},
|
||||
})
|
||||
process.stderr.write('\n')
|
||||
|
||||
for (const warning of result.warnings) console.warn(`warning: ${warning}`)
|
||||
|
||||
console.log(`verdict: ${result.verdict}`)
|
||||
console.log(`mean JSD: ${result.meanJsd?.toFixed(3)} over ${result.comparison.comparableCellCount} cells`)
|
||||
console.log('baselines: same model ≈ 0.14 · cross-provider ≈ 0.227 · different model ≈ 0.463')
|
||||
|
||||
console.log('\nmost divergent cells:')
|
||||
for (const cell of result.comparison.cells.slice(0, 5)) {
|
||||
console.log(` ${cell.cellId.padEnd(26)} JSD ${cell.jsd.toFixed(3)}`)
|
||||
}
|
||||
|
||||
// CI-style decision:
|
||||
process.exitCode = result.verdict === 'match' ? 0 : 1
|
||||
Executable
+46
@@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env bash
|
||||
# CLI usage examples. Keys always come from environment variables — never
|
||||
# paste keys into files or shell history.
|
||||
set -euo pipefail
|
||||
|
||||
# After `npm install -g llm-fingerprint-detector` use `llm-fingerprint ...`;
|
||||
# inside this repo use `node dist/cli.js ...` (shown here via npx).
|
||||
LLM_FP="npx llm-fingerprint"
|
||||
|
||||
# --- 0. Explore ---------------------------------------------------------------
|
||||
$LLM_FP --help
|
||||
$LLM_FP references # bundled sample reference fingerprints
|
||||
|
||||
# --- 1. Fingerprint the official API and save a trusted reference --------------
|
||||
# export OPENAI_API_KEY=sk-... # ← set in your shell / CI secret store
|
||||
$LLM_FP fingerprint \
|
||||
--base-url https://api.openai.com/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--preset standard \
|
||||
--out reference.gpt-4o-mini.json
|
||||
|
||||
# --- 2. Verify a cheap reseller claims honestly --------------------------------
|
||||
# export LLM_FINGERPRINT_API_KEY=sk-... # key for the endpoint under test
|
||||
$LLM_FP verify \
|
||||
--base-url https://cheap-llm-reseller.example.com/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--reference reference.gpt-4o-mini.json
|
||||
# exit codes: 0 match · 2 mismatch · 3 uncertain · 4 insufficient · 1 error
|
||||
|
||||
# --- 3. Quick demo against a bundled sample reference --------------------------
|
||||
$LLM_FP verify \
|
||||
--base-url https://openrouter.ai/api/v1 \
|
||||
--model openai/gpt-4o-mini \
|
||||
--reference openai/gpt-4o-mini \
|
||||
--preset quick --json
|
||||
|
||||
# --- 4. Offline: compare two saved fingerprints --------------------------------
|
||||
$LLM_FP compare reference.gpt-4o-mini.json some-other.fingerprint.json
|
||||
|
||||
# --- 5. Custom cell selection ---------------------------------------------------
|
||||
$LLM_FP fingerprint \
|
||||
--base-url https://api.deepseek.com/v1 \
|
||||
--model deepseek-chat \
|
||||
--api-key-env DEEPSEEK_API_KEY \
|
||||
--cells random-number-1-100:en,random-number-1-100:zh,random-color:en,coin-flip:en \
|
||||
--samples 30 --json
|
||||
Generated
+54
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"name": "llm-fingerprint-detector",
|
||||
"version": "0.1.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "llm-fingerprint-detector",
|
||||
"version": "0.1.0",
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"llm-fingerprint": "dist/cli.js"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.10.0",
|
||||
"typescript": "^5.7.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18.17"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "22.20.1",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-22.20.1.tgz",
|
||||
"integrity": "sha512-EANqOCF9QFyra+4pfxUcX9STKJpCLjMbObVzljIJomAWSnuSIEAvyzEU53GaajbXJEgdh0iEcPL+DGvpUd4k1Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": "~6.21.0"
|
||||
}
|
||||
},
|
||||
"node_modules/typescript": {
|
||||
"version": "5.9.3",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
|
||||
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.17"
|
||||
}
|
||||
},
|
||||
"node_modules/undici-types": {
|
||||
"version": "6.21.0",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
|
||||
"integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"name": "llm-fingerprint-detector",
|
||||
"version": "0.1.0",
|
||||
"description": "Fingerprint and verify LLMs behind OpenAI-compatible APIs from single-token output distributions. Independent open-source implementation of \"One Token Is Enough\" (Bruckner, arXiv:2607.10252).",
|
||||
"type": "module",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18.17"
|
||||
},
|
||||
"main": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./dist/index.d.ts",
|
||||
"import": "./dist/index.js"
|
||||
},
|
||||
"./references": {
|
||||
"types": "./dist/reference.d.ts",
|
||||
"import": "./dist/reference.js"
|
||||
},
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"bin": {
|
||||
"llm-fingerprint": "./dist/cli.js"
|
||||
},
|
||||
"files": [
|
||||
"dist",
|
||||
"data",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"test": "npm run build && node --test",
|
||||
"prepublishOnly": "npm run build && node --test"
|
||||
},
|
||||
"keywords": [
|
||||
"llm",
|
||||
"fingerprint",
|
||||
"fingerprinting",
|
||||
"model-verification",
|
||||
"openai-compatible",
|
||||
"api-verification",
|
||||
"jensen-shannon",
|
||||
"behavioral-fingerprint",
|
||||
"one-token-is-enough",
|
||||
"llm-security",
|
||||
"model-identity",
|
||||
"model-substitution",
|
||||
"api-audit",
|
||||
"llm-verification",
|
||||
"openrouter",
|
||||
"gpt",
|
||||
"claude",
|
||||
"cli"
|
||||
],
|
||||
"homepage": "https://tosea.ai/free-tools/llm-api-fingerprint-checker",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/ToseaAI/llm-fingerprint-detector.git"
|
||||
},
|
||||
"bugs": {
|
||||
"url": "https://github.com/ToseaAI/llm-fingerprint-detector/issues"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.10.0",
|
||||
"typescript": "^5.7.0"
|
||||
}
|
||||
}
|
||||
Executable
+169
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Build data/reference-fingerprints.sample.json from the paper's public
|
||||
* Zenodo dataset (DOI 10.5281/zenodo.21278557, CC-BY-4.0).
|
||||
*
|
||||
* Input: the dataset's aggregated `distributions.json` — an object with a
|
||||
* `distributions` array of per-cell records:
|
||||
* { model, task_id, lang, temperature, n_valid, dist: { answer: probability }, ... }
|
||||
*
|
||||
* Counts are reconstructed as round(probability × n_valid) and answers are
|
||||
* re-normalized with this package's normalizer so vocabularies line up
|
||||
* (e.g. the dataset's coin answers h/t → heads/tails, 蓝色 → 蓝). Cells that
|
||||
* lose more than 20% of their probability mass in re-normalization are
|
||||
* skipped.
|
||||
*
|
||||
* Usage:
|
||||
* npm run build # the script imports the compiled normalizer from dist/
|
||||
* node scripts/build-sample-references.mjs <path-to-distributions.json> \
|
||||
* [--models comma,separated,slugs] [--out data/reference-fingerprints.sample.json]
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync } from 'node:fs'
|
||||
import { dirname, join } from 'node:path'
|
||||
import process from 'node:process'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import { PROBE_TASKS } from '../dist/battery.js'
|
||||
import { normalizeAnswer } from '../dist/normalizer.js'
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url))
|
||||
|
||||
/** Dataset task_id → this package's battery task id. */
|
||||
const TASK_MAP = {
|
||||
'num100-random': 'random-number-1-100',
|
||||
'num10-random': 'random-number-1-10',
|
||||
'letter-random': 'random-letter',
|
||||
'color-random': 'random-color',
|
||||
'coin-flip': 'coin-flip',
|
||||
'animal-random': 'random-animal',
|
||||
'city-random': 'random-city',
|
||||
'num-favorite': 'favorite-number',
|
||||
}
|
||||
|
||||
const LANGS = new Set(['en', 'zh'])
|
||||
|
||||
/** The dataset folds coin answers to single letters; pre-expand before normalizing. */
|
||||
const COIN_PREMAP = { h: 'heads', t: 'tails' }
|
||||
|
||||
const DEFAULT_MODELS = [
|
||||
'openai/gpt-4o-mini',
|
||||
'openai/gpt-4o',
|
||||
'openai/gpt-4.1-mini',
|
||||
'anthropic/claude-sonnet-4.5',
|
||||
'google/gemini-2.5-flash',
|
||||
'deepseek/deepseek-chat',
|
||||
'meta-llama/llama-3.1-8b-instruct',
|
||||
'qwen/qwen3-30b-a3b-instruct-2507',
|
||||
'mistralai/mistral-small-3.2-24b-instruct',
|
||||
'z-ai/glm-4.5',
|
||||
'moonshotai/kimi-k2',
|
||||
]
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = { input: null, models: DEFAULT_MODELS, out: join(HERE, '..', 'data', 'reference-fingerprints.sample.json') }
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const token = argv[i]
|
||||
if (token === '--models') args.models = argv[++i].split(',').map((m) => m.trim())
|
||||
else if (token === '--out') args.out = argv[++i]
|
||||
else if (!token.startsWith('--') && !args.input) args.input = token
|
||||
else throw new Error(`Unknown argument: ${token}`)
|
||||
}
|
||||
if (!args.input) {
|
||||
console.error('usage: node scripts/build-sample-references.mjs <distributions.json> [--models a,b] [--out file]')
|
||||
process.exit(1)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
function convertCell(record, taskId) {
|
||||
const domain = PROBE_TASKS[taskId].domain
|
||||
const counts = {}
|
||||
let keptMass = 0
|
||||
let totalMass = 0
|
||||
for (const [rawAnswer, probability] of Object.entries(record.dist)) {
|
||||
totalMass += probability
|
||||
const premapped = taskId === 'coin-flip' ? (COIN_PREMAP[rawAnswer] ?? rawAnswer) : rawAnswer
|
||||
const { normalized, category } = normalizeAnswer(premapped, domain)
|
||||
if (category !== 'valid' || normalized === null) continue
|
||||
const count = Math.round(probability * record.n_valid)
|
||||
if (count <= 0) continue
|
||||
counts[normalized] = (counts[normalized] ?? 0) + count
|
||||
keptMass += probability
|
||||
}
|
||||
if (totalMass <= 0 || keptMass / totalMass < 0.8) return null
|
||||
const n = Object.values(counts).reduce((sum, c) => sum + c, 0)
|
||||
if (n < 10) return null
|
||||
return { n, counts }
|
||||
}
|
||||
|
||||
function main() {
|
||||
const args = parseArgs(process.argv.slice(2))
|
||||
const payload = JSON.parse(readFileSync(args.input, 'utf8'))
|
||||
const records = payload.distributions
|
||||
if (!Array.isArray(records)) throw new Error('Input has no "distributions" array')
|
||||
|
||||
const collectedAt = (payload.generated_utc ?? '').slice(0, 10) || 'unknown'
|
||||
const wanted = new Set(args.models)
|
||||
const models = {}
|
||||
let skippedCells = 0
|
||||
|
||||
for (const record of records) {
|
||||
if (!wanted.has(record.model)) continue
|
||||
if (!LANGS.has(record.lang)) continue
|
||||
const taskId = TASK_MAP[record.task_id]
|
||||
if (!taskId) continue
|
||||
if (record.temperature !== 1) continue
|
||||
|
||||
const cell = convertCell(record, taskId)
|
||||
if (!cell) {
|
||||
skippedCells += 1
|
||||
continue
|
||||
}
|
||||
const cellId = `${taskId}:${record.lang}`
|
||||
models[record.model] ??= {
|
||||
model: record.model,
|
||||
collectedAt,
|
||||
channel: 'openrouter',
|
||||
cells: {},
|
||||
}
|
||||
models[record.model].cells[cellId] = cell
|
||||
}
|
||||
|
||||
for (const model of args.models) {
|
||||
const entry = models[model]
|
||||
if (!entry) {
|
||||
console.warn(`warn: model not found in dataset: ${model}`)
|
||||
} else if (Object.keys(entry.cells).length < 8) {
|
||||
console.warn(`warn: ${model} only has ${Object.keys(entry.cells).length} usable cells`)
|
||||
}
|
||||
}
|
||||
|
||||
const output = {
|
||||
formatVersion: 1,
|
||||
protocol: 'bruckner-zenodo-2026',
|
||||
samplesPerCell: 30,
|
||||
source: {
|
||||
dataset:
|
||||
'Single-token output distributions as behavioral fingerprints of large language models',
|
||||
author: 'Tomáš Bruckner (Prague University of Economics and Business)',
|
||||
datasetDoi: '10.5281/zenodo.21278557',
|
||||
paper: 'arXiv:2607.10252',
|
||||
license: 'CC-BY-4.0',
|
||||
note:
|
||||
'Counts reconstructed as round(probability × n_valid) from the published per-cell distributions; ' +
|
||||
'answers re-normalized with the llm-fingerprint-detector normalizer. Collected via OpenRouter by ' +
|
||||
"the paper's harness under the paper's prompt protocol (not this package's battery), so comparisons " +
|
||||
'against these samples are indicative rather than strict.',
|
||||
},
|
||||
models,
|
||||
}
|
||||
|
||||
writeFileSync(args.out, `${JSON.stringify(output, null, 2)}\n`, 'utf8')
|
||||
const cellTotal = Object.values(models).reduce((sum, m) => sum + Object.keys(m.cells).length, 0)
|
||||
console.log(
|
||||
`wrote ${args.out}: ${Object.keys(models).length} models, ${cellTotal} cells (${skippedCells} cells skipped in re-normalization)`,
|
||||
)
|
||||
}
|
||||
|
||||
main()
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
/**
|
||||
* Reasoning-disable adapter layer. Hidden "thinking" must be turned off:
|
||||
* it burns the max_tokens budget before any visible answer appears, and it
|
||||
* shifts the sampled distribution. Three request-body variants are known to
|
||||
* work across OpenAI-compatible providers:
|
||||
*
|
||||
* 1. `reasoning: { enabled: false }` — OpenRouter style
|
||||
* 2. `thinking: { type: "disabled" }` — Zhipu BigModel style
|
||||
* 3. `reasoning_effort: "none"` — OpenAI style
|
||||
*
|
||||
* Before a run, each variant is probed in a provider-specific order; the
|
||||
* first one that returns a non-empty visible answer without a 4xx wins. If
|
||||
* every variant fails, a bare request is tried; if even that yields no
|
||||
* visible text, the run falls back to a "post-reasoning" channel
|
||||
* (max_tokens=1024) and the resulting fingerprint is flagged as lower
|
||||
* confidence.
|
||||
*/
|
||||
|
||||
import { getSystemPrompt, pickParaphrase } from './battery.js'
|
||||
import {
|
||||
POST_REASONING_MAX_TOKENS,
|
||||
PROBE_MAX_TOKENS,
|
||||
PROBE_TEMPERATURE,
|
||||
} from './constants.js'
|
||||
import { guessAdapterHint } from './endpoint.js'
|
||||
import { fetchChatCompletion, ProbeRequestError } from './http.js'
|
||||
import type {
|
||||
CellId,
|
||||
ReasoningAdapter,
|
||||
ReasoningStrategyId,
|
||||
ResolvedEndpoint,
|
||||
} from './types.js'
|
||||
|
||||
export const STRATEGY_BODIES: Record<
|
||||
Exclude<ReasoningStrategyId, 'none'>,
|
||||
Record<string, unknown>
|
||||
> = {
|
||||
'openrouter-reasoning': { reasoning: { enabled: false } },
|
||||
'zhipu-thinking': { thinking: { type: 'disabled' } },
|
||||
'openai-effort': { reasoning_effort: 'none' },
|
||||
}
|
||||
|
||||
const ADAPTER_PROBE_CELL: CellId = 'random-number-1-100:en'
|
||||
|
||||
export interface AdapterDetectionOptions {
|
||||
signal?: AbortSignal
|
||||
timeoutMs?: number
|
||||
onProbe?: (strategy: ReasoningStrategyId) => void
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect which reasoning-disable field the endpoint accepts.
|
||||
* Auth (401/403) and transport errors abort detection immediately — they are
|
||||
* unrelated to the strategy and would fail the whole run anyway.
|
||||
*/
|
||||
export async function detectReasoningAdapter(
|
||||
endpoint: ResolvedEndpoint,
|
||||
options: AdapterDetectionOptions = {},
|
||||
): Promise<ReasoningAdapter> {
|
||||
const hint = guessAdapterHint(endpoint.baseUrl)
|
||||
const systemPrompt = getSystemPrompt(ADAPTER_PROBE_CELL)
|
||||
|
||||
for (const strategy of hint) {
|
||||
if (strategy === 'none') continue
|
||||
options.onProbe?.(strategy)
|
||||
try {
|
||||
const result = await fetchChatCompletion({
|
||||
endpoint,
|
||||
systemPrompt,
|
||||
userPrompt: pickParaphrase(ADAPTER_PROBE_CELL),
|
||||
temperature: PROBE_TEMPERATURE,
|
||||
maxTokens: PROBE_MAX_TOKENS,
|
||||
extraBody: STRATEGY_BODIES[strategy],
|
||||
signal: options.signal,
|
||||
timeoutMs: options.timeoutMs,
|
||||
retries: 0,
|
||||
})
|
||||
if (result.content.trim().length > 0) {
|
||||
return {
|
||||
strategy,
|
||||
extraBody: STRATEGY_BODIES[strategy],
|
||||
maxTokens: PROBE_MAX_TOKENS,
|
||||
postReasoning: false,
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
if (error instanceof ProbeRequestError) {
|
||||
if (error.kind === 'auth' || error.kind === 'network' || error.kind === 'aborted') {
|
||||
throw error
|
||||
}
|
||||
// 4xx parameter rejection or timeout: try the next strategy.
|
||||
continue
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
// No disable field accepted → probe with a bare request.
|
||||
options.onProbe?.('none')
|
||||
try {
|
||||
const bare = await fetchChatCompletion({
|
||||
endpoint,
|
||||
systemPrompt,
|
||||
userPrompt: pickParaphrase(ADAPTER_PROBE_CELL),
|
||||
temperature: PROBE_TEMPERATURE,
|
||||
maxTokens: PROBE_MAX_TOKENS,
|
||||
extraBody: {},
|
||||
signal: options.signal,
|
||||
timeoutMs: options.timeoutMs,
|
||||
retries: 0,
|
||||
})
|
||||
if (bare.content.trim().length > 0) {
|
||||
// Non-reasoning model (or reasoning is free): bare requests are fine.
|
||||
return { strategy: 'none', extraBody: {}, maxTokens: PROBE_MAX_TOKENS, postReasoning: false }
|
||||
}
|
||||
} catch (error) {
|
||||
if (
|
||||
error instanceof ProbeRequestError &&
|
||||
(error.kind === 'auth' || error.kind === 'network' || error.kind === 'aborted')
|
||||
) {
|
||||
throw error
|
||||
}
|
||||
// Anything else falls through to the post-reasoning channel.
|
||||
}
|
||||
|
||||
// Fallback: reasoning cannot be disabled. Raise max_tokens so a visible
|
||||
// answer survives after the hidden reasoning; flag reduced confidence.
|
||||
return {
|
||||
strategy: 'none',
|
||||
extraBody: {},
|
||||
maxTokens: POST_REASONING_MAX_TOKENS,
|
||||
postReasoning: true,
|
||||
}
|
||||
}
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
/**
|
||||
* Public high-level API:
|
||||
*
|
||||
* fingerprint(endpoint, options?) → collect a behavioral fingerprint
|
||||
* compare(fingerprintA, fingerprintB) → distance + verdict
|
||||
* verify(endpoint, reference, options?) → fingerprint + compare in one call
|
||||
*/
|
||||
|
||||
import { detectReasoningAdapter } from './adapter.js'
|
||||
import { CELL_PRIORITY_ORDER, getTaskSpec } from './battery.js'
|
||||
import {
|
||||
DEFAULT_CELL_COUNT,
|
||||
DEFAULT_CONCURRENCY,
|
||||
DEFAULT_SAMPLES_PER_CELL,
|
||||
FINGERPRINT_FORMAT_VERSION,
|
||||
PROBE_PROTOCOL,
|
||||
SPLIT_HALF_WARN_THRESHOLD,
|
||||
} from './constants.js'
|
||||
import { resolveEndpoint } from './endpoint.js'
|
||||
import { runProbeBattery } from './sampler.js'
|
||||
import { buildCellDistribution, compareCellSets, splitHalfJsd } from './stats.js'
|
||||
import { buildComparisonResult } from './verdict.js'
|
||||
import type {
|
||||
CellDistribution,
|
||||
CellId,
|
||||
ComparisonResult,
|
||||
Endpoint,
|
||||
Fingerprint,
|
||||
FingerprintOptions,
|
||||
FingerprintRun,
|
||||
VerifyResult,
|
||||
} from './types.js'
|
||||
|
||||
function resolveCells(cells: FingerprintOptions['cells']): CellId[] {
|
||||
if (cells === undefined) return CELL_PRIORITY_ORDER.slice(0, DEFAULT_CELL_COUNT)
|
||||
if (typeof cells === 'number') {
|
||||
const count = Math.max(1, Math.min(CELL_PRIORITY_ORDER.length, Math.floor(cells)))
|
||||
return CELL_PRIORITY_ORDER.slice(0, count)
|
||||
}
|
||||
if (cells.length === 0) throw new Error('options.cells must not be empty')
|
||||
return cells
|
||||
}
|
||||
|
||||
/**
|
||||
* Probe an OpenAI-compatible endpoint and collect its behavioral fingerprint.
|
||||
*
|
||||
* Steps: normalize the endpoint → detect a working reasoning-disable strategy
|
||||
* → run the probe battery (shuffled, concurrent) → aggregate per-cell answer
|
||||
* distributions.
|
||||
*/
|
||||
export async function fingerprint(
|
||||
endpoint: Endpoint,
|
||||
options: FingerprintOptions = {},
|
||||
): Promise<FingerprintRun> {
|
||||
const startedAt = Date.now()
|
||||
const { resolved, warnings } = resolveEndpoint(endpoint)
|
||||
const cells = resolveCells(options.cells)
|
||||
const samplesPerCell = options.samplesPerCell ?? DEFAULT_SAMPLES_PER_CELL
|
||||
const concurrency = options.concurrency ?? DEFAULT_CONCURRENCY
|
||||
|
||||
const adapter =
|
||||
options.adapter ??
|
||||
(await detectReasoningAdapter(resolved, {
|
||||
signal: options.signal,
|
||||
timeoutMs: options.timeoutMs,
|
||||
onProbe: (strategy) =>
|
||||
options.onProgress?.({ stage: 'adapter', done: 0, total: 1, errors: 0, strategy }),
|
||||
}))
|
||||
|
||||
if (adapter.postReasoning) {
|
||||
warnings.push(
|
||||
'Reasoning could not be disabled; fell back to the post-reasoning channel (max_tokens=1024). Fingerprint confidence is reduced.',
|
||||
)
|
||||
}
|
||||
|
||||
const { samples, samplesByCell, errorCount } = await runProbeBattery({
|
||||
endpoint: resolved,
|
||||
adapter,
|
||||
cells,
|
||||
samplesPerCell,
|
||||
concurrency,
|
||||
timeoutMs: options.timeoutMs,
|
||||
maxRetries: options.maxRetries,
|
||||
signal: options.signal,
|
||||
onProgress: options.onProgress,
|
||||
})
|
||||
|
||||
const cellDistributions: Partial<Record<CellId, CellDistribution>> = {}
|
||||
for (const cellId of cells) {
|
||||
cellDistributions[cellId] = buildCellDistribution(
|
||||
cellId,
|
||||
samplesByCell.get(cellId) ?? [],
|
||||
getTaskSpec(cellId).domain,
|
||||
)
|
||||
}
|
||||
|
||||
const selfJsd = splitHalfJsd(samplesByCell)
|
||||
if (selfJsd !== null && selfJsd > SPLIT_HALF_WARN_THRESHOLD) {
|
||||
warnings.push(
|
||||
`Split-half self distance is high (${selfJsd.toFixed(3)} > ${SPLIT_HALF_WARN_THRESHOLD}); the endpoint may be routing across multiple backends.`,
|
||||
)
|
||||
}
|
||||
if (errorCount > 0) {
|
||||
warnings.push(`${errorCount} of ${samples.length} requests failed and were excluded.`)
|
||||
}
|
||||
|
||||
const result: FingerprintRun = {
|
||||
fingerprint: {
|
||||
formatVersion: FINGERPRINT_FORMAT_VERSION,
|
||||
protocol: PROBE_PROTOCOL,
|
||||
model: resolved.model,
|
||||
collectedAt: new Date().toISOString(),
|
||||
samplesPerCell,
|
||||
postReasoning: adapter.postReasoning,
|
||||
cells: cellDistributions,
|
||||
meta: {
|
||||
tool: 'llm-fingerprint-detector',
|
||||
...options.meta,
|
||||
},
|
||||
},
|
||||
adapter,
|
||||
errorCount,
|
||||
splitHalfJsd: selfJsd,
|
||||
durationMs: Date.now() - startedAt,
|
||||
warnings,
|
||||
}
|
||||
if (options.keepSamples) result.samples = samples
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare two fingerprints: mean per-cell Jensen-Shannon divergence (base 2)
|
||||
* over cells where both sides have enough valid samples, plus a three-way
|
||||
* verdict against the paper-derived thresholds.
|
||||
*/
|
||||
export function compare(a: Fingerprint, b: Fingerprint): ComparisonResult {
|
||||
const { entries, meanJsd } = compareCellSets(a.cells, b.cells)
|
||||
const protocolMismatch = a.protocol !== b.protocol
|
||||
return buildComparisonResult(entries, meanJsd, protocolMismatch)
|
||||
}
|
||||
|
||||
/**
|
||||
* Verify that an endpoint behaves like a reference fingerprint: collect a
|
||||
* fresh fingerprint from the endpoint, then compare against the reference.
|
||||
*/
|
||||
export async function verify(
|
||||
endpoint: Endpoint,
|
||||
reference: Fingerprint,
|
||||
options: FingerprintOptions = {},
|
||||
): Promise<VerifyResult> {
|
||||
const referenceCells = Object.keys(reference.cells) as CellId[]
|
||||
const cells =
|
||||
options.cells !== undefined
|
||||
? resolveCells(options.cells)
|
||||
: CELL_PRIORITY_ORDER.filter((cellId) => referenceCells.includes(cellId))
|
||||
if (cells.length === 0) {
|
||||
throw new Error('Reference fingerprint has no cells overlapping the probe battery')
|
||||
}
|
||||
|
||||
const target = await fingerprint(endpoint, { ...options, cells })
|
||||
const comparison = compare(target.fingerprint, reference)
|
||||
|
||||
const warnings = [...target.warnings]
|
||||
if (comparison.protocolMismatch) {
|
||||
warnings.push(
|
||||
`Protocol mismatch: target "${target.fingerprint.protocol}" vs reference "${reference.protocol}". ` +
|
||||
'Fingerprints collected under different prompts/batteries are only loosely comparable; treat the verdict as indicative.',
|
||||
)
|
||||
}
|
||||
if (reference.postReasoning) {
|
||||
warnings.push('Reference fingerprint was collected over the post-reasoning channel (reduced confidence).')
|
||||
}
|
||||
|
||||
return {
|
||||
verdict: comparison.verdict,
|
||||
meanJsd: comparison.meanJsd,
|
||||
comparison,
|
||||
target,
|
||||
reference,
|
||||
warnings,
|
||||
}
|
||||
}
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
/**
|
||||
* Probe battery: 8 tasks × 2 languages = 16 cells.
|
||||
*
|
||||
* Design (arXiv:2607.10252):
|
||||
* - ≥3 paraphrases per task per language, drawn at random per request — every
|
||||
* probe is a plain semantic question, so gateways cannot keyword-filter them;
|
||||
* - a fixed minimal system prompt forces a one-word answer; max_tokens=16
|
||||
* truncates stragglers;
|
||||
* - cells are ordered by empirical discriminativeness (random-number tasks
|
||||
* separate models best); presets take a prefix of that order.
|
||||
*/
|
||||
|
||||
import type {
|
||||
CellId,
|
||||
ProbeLang,
|
||||
ProbePreset,
|
||||
ProbePresetId,
|
||||
ProbeTaskId,
|
||||
ProbeTaskSpec,
|
||||
} from './types.js'
|
||||
|
||||
/**
|
||||
* Fixed minimal system prompt — a critical invariant.
|
||||
*
|
||||
* The reference fingerprint and the endpoint under test MUST be collected
|
||||
* under the *same* system prompt. Swapping only the system prompt (all else
|
||||
* equal) can shift a model's fingerprint by JSD ≈ 0.44–0.46 — the same
|
||||
* magnitude as swapping the model itself — which would turn a prompt
|
||||
* difference into a false "different model" verdict.
|
||||
*/
|
||||
export const SYSTEM_PROMPTS: Record<ProbeLang, string> = {
|
||||
en: 'Answer with exactly one word. No punctuation, no explanation.',
|
||||
zh: '只回答一个词,不要标点,不要解释。',
|
||||
}
|
||||
|
||||
export const PROBE_TASKS: Record<ProbeTaskId, ProbeTaskSpec> = {
|
||||
'random-number-1-100': {
|
||||
id: 'random-number-1-100',
|
||||
domain: { kind: 'int', min: 1, max: 100 },
|
||||
paraphrases: {
|
||||
en: [
|
||||
'Name a random number between 1 and 100.',
|
||||
'Pick a random number from 1 to 100.',
|
||||
'Give me a random number between 1 and 100.',
|
||||
'Choose any number between 1 and 100 at random.',
|
||||
],
|
||||
zh: [
|
||||
'说一个 1 到 100 之间的随机数。',
|
||||
'随机挑一个 1 到 100 的数字。',
|
||||
'给我一个 1 至 100 之间的随机数字。',
|
||||
'从 1 到 100 里随便选一个数。',
|
||||
],
|
||||
},
|
||||
},
|
||||
'random-number-1-10': {
|
||||
id: 'random-number-1-10',
|
||||
domain: { kind: 'int', min: 1, max: 10 },
|
||||
paraphrases: {
|
||||
en: [
|
||||
'Name a random number between 1 and 10.',
|
||||
'Pick a random number from 1 to 10.',
|
||||
'Give me a random number between 1 and 10.',
|
||||
],
|
||||
zh: [
|
||||
'说一个 1 到 10 之间的随机数。',
|
||||
'随机挑一个 1 到 10 的数字。',
|
||||
'从 1 到 10 里随便选一个数。',
|
||||
],
|
||||
},
|
||||
},
|
||||
'random-letter': {
|
||||
id: 'random-letter',
|
||||
domain: { kind: 'letter' },
|
||||
paraphrases: {
|
||||
en: [
|
||||
'Name a random letter of the alphabet.',
|
||||
'Pick a random letter from A to Z.',
|
||||
'Give me one random letter.',
|
||||
],
|
||||
zh: [
|
||||
'说一个随机的英文字母。',
|
||||
'从 A 到 Z 里随便挑一个字母。',
|
||||
'随机给我一个英文字母。',
|
||||
],
|
||||
},
|
||||
},
|
||||
'random-color': {
|
||||
id: 'random-color',
|
||||
domain: { kind: 'color' },
|
||||
paraphrases: {
|
||||
en: [
|
||||
'Name a random color.',
|
||||
'Pick a color at random.',
|
||||
'Give me one random color.',
|
||||
],
|
||||
zh: [
|
||||
'说一个随机的颜色。',
|
||||
'随便说一种颜色。',
|
||||
'随机挑一个颜色告诉我。',
|
||||
],
|
||||
},
|
||||
},
|
||||
'coin-flip': {
|
||||
id: 'coin-flip',
|
||||
domain: { kind: 'coin' },
|
||||
paraphrases: {
|
||||
en: [
|
||||
'Flip a coin. Answer heads or tails.',
|
||||
'Toss a coin and tell me the result: heads or tails.',
|
||||
'Imagine flipping a coin. Which side came up, heads or tails?',
|
||||
],
|
||||
zh: [
|
||||
'抛一枚硬币,回答正面还是反面。',
|
||||
'掷一次硬币,告诉我结果:正面或反面。',
|
||||
'想象抛硬币,落地是正面还是反面?',
|
||||
],
|
||||
},
|
||||
},
|
||||
'random-animal': {
|
||||
id: 'random-animal',
|
||||
domain: { kind: 'word' },
|
||||
paraphrases: {
|
||||
en: [
|
||||
'Name a random animal.',
|
||||
'Pick an animal at random.',
|
||||
'Give me one random animal.',
|
||||
],
|
||||
zh: [
|
||||
'说一个随机的动物。',
|
||||
'随便说一种动物。',
|
||||
'随机挑一个动物告诉我。',
|
||||
],
|
||||
},
|
||||
},
|
||||
'random-city': {
|
||||
id: 'random-city',
|
||||
domain: { kind: 'word' },
|
||||
paraphrases: {
|
||||
en: [
|
||||
'Name a random city.',
|
||||
'Pick a city at random.',
|
||||
'Give me the name of one random city.',
|
||||
],
|
||||
zh: [
|
||||
'说一个随机的城市。',
|
||||
'随便说一座城市。',
|
||||
'随机挑一个城市告诉我。',
|
||||
],
|
||||
},
|
||||
},
|
||||
'favorite-number': {
|
||||
id: 'favorite-number',
|
||||
domain: { kind: 'int', min: 0, max: 10_000 },
|
||||
paraphrases: {
|
||||
en: [
|
||||
'What is your favorite number?',
|
||||
'Tell me your favourite number.',
|
||||
'If you had to pick a favorite number, what would it be?',
|
||||
],
|
||||
zh: [
|
||||
'你最喜欢的数字是什么?',
|
||||
'说说你最爱的数字。',
|
||||
'如果必须选一个最喜欢的数字,你选哪个?',
|
||||
],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
export function makeCellId(task: ProbeTaskId, lang: ProbeLang): CellId {
|
||||
return `${task}:${lang}`
|
||||
}
|
||||
|
||||
export function parseCellId(cellId: CellId): { task: ProbeTaskId; lang: ProbeLang } {
|
||||
const idx = cellId.lastIndexOf(':')
|
||||
return {
|
||||
task: cellId.slice(0, idx) as ProbeTaskId,
|
||||
lang: cellId.slice(idx + 1) as ProbeLang,
|
||||
}
|
||||
}
|
||||
|
||||
export function isCellId(value: string): value is CellId {
|
||||
const idx = value.lastIndexOf(':')
|
||||
if (idx <= 0) return false
|
||||
const task = value.slice(0, idx)
|
||||
const lang = value.slice(idx + 1)
|
||||
return task in PROBE_TASKS && (lang === 'en' || lang === 'zh')
|
||||
}
|
||||
|
||||
/**
|
||||
* All 16 cells, ordered by discriminativeness (random-number tasks first,
|
||||
* per the paper and our own cross-model measurements). Presets take a prefix.
|
||||
*/
|
||||
export const CELL_PRIORITY_ORDER: CellId[] = [
|
||||
'random-number-1-100:en',
|
||||
'random-number-1-100:zh',
|
||||
'random-color:en',
|
||||
'random-animal:en',
|
||||
'random-number-1-10:en',
|
||||
'random-letter:en',
|
||||
'random-color:zh',
|
||||
'coin-flip:en',
|
||||
'favorite-number:en',
|
||||
'random-city:en',
|
||||
'random-number-1-10:zh',
|
||||
'coin-flip:zh',
|
||||
'random-letter:zh',
|
||||
'random-animal:zh',
|
||||
'random-city:zh',
|
||||
'favorite-number:zh',
|
||||
]
|
||||
|
||||
export const PROBE_PRESETS: Record<ProbePresetId, ProbePreset> = {
|
||||
quick: { id: 'quick', cellCount: 4, samplesPerCell: 15 },
|
||||
standard: { id: 'standard', cellCount: 8, samplesPerCell: 25 },
|
||||
strict: { id: 'strict', cellCount: 16, samplesPerCell: 25 },
|
||||
}
|
||||
|
||||
export function getCellsForPreset(preset: ProbePresetId): CellId[] {
|
||||
return CELL_PRIORITY_ORDER.slice(0, PROBE_PRESETS[preset].cellCount)
|
||||
}
|
||||
|
||||
export function getTaskSpec(cellId: CellId): ProbeTaskSpec {
|
||||
return PROBE_TASKS[parseCellId(cellId).task]
|
||||
}
|
||||
|
||||
export function getSystemPrompt(cellId: CellId): string {
|
||||
return SYSTEM_PROMPTS[parseCellId(cellId).lang]
|
||||
}
|
||||
|
||||
export function pickParaphrase(cellId: CellId, random: () => number = Math.random): string {
|
||||
const { task, lang } = parseCellId(cellId)
|
||||
const pool = PROBE_TASKS[task].paraphrases[lang]
|
||||
return pool[Math.floor(random() * pool.length)] ?? pool[0]
|
||||
}
|
||||
+527
@@ -0,0 +1,527 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* llm-fingerprint — CLI for fingerprinting and verifying LLM endpoints.
|
||||
*
|
||||
* The API key is read from an environment variable (never from a file, never
|
||||
* logged). Verify exit codes are CI-friendly:
|
||||
* 0 match · 2 mismatch · 3 uncertain · 4 insufficient · 1 error
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync } from 'node:fs'
|
||||
import { dirname, join } from 'node:path'
|
||||
import process from 'node:process'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import { compare, fingerprint, verify } from './api.js'
|
||||
import { CELL_PRIORITY_ORDER, isCellId } from './battery.js'
|
||||
import {
|
||||
DEFAULT_CELL_COUNT,
|
||||
DEFAULT_CONCURRENCY,
|
||||
DEFAULT_SAMPLES_PER_CELL,
|
||||
} from './constants.js'
|
||||
import {
|
||||
getBundledAttribution,
|
||||
listBundledReferences,
|
||||
loadBundledReference,
|
||||
parseFingerprintJson,
|
||||
} from './reference.js'
|
||||
import { ProbeRunError } from './sampler.js'
|
||||
import type {
|
||||
CellId,
|
||||
ComparisonResult,
|
||||
Endpoint,
|
||||
Fingerprint,
|
||||
FingerprintOptions,
|
||||
ProgressEvent,
|
||||
VerdictLevel,
|
||||
} from './types.js'
|
||||
|
||||
const DEFAULT_KEY_ENV_VARS = ['LLM_FINGERPRINT_API_KEY', 'OPENAI_API_KEY']
|
||||
|
||||
const VALUE_OPTIONS = new Set([
|
||||
'--base-url',
|
||||
'--model',
|
||||
'--api-key',
|
||||
'--api-key-env',
|
||||
'--cells',
|
||||
'--samples',
|
||||
'--concurrency',
|
||||
'--timeout',
|
||||
'--preset',
|
||||
'--reference',
|
||||
'--out',
|
||||
])
|
||||
const BOOLEAN_OPTIONS = new Set(['--json', '--quiet', '--help', '-h', '--version', '-V'])
|
||||
|
||||
interface ParsedArgs {
|
||||
positionals: string[]
|
||||
options: Map<string, string | boolean>
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): ParsedArgs {
|
||||
const positionals: string[] = []
|
||||
const options = new Map<string, string | boolean>()
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const token = argv[i]
|
||||
if (!token.startsWith('-')) {
|
||||
positionals.push(token)
|
||||
continue
|
||||
}
|
||||
const eq = token.indexOf('=')
|
||||
if (eq > 0) {
|
||||
options.set(token.slice(0, eq), token.slice(eq + 1))
|
||||
continue
|
||||
}
|
||||
if (BOOLEAN_OPTIONS.has(token)) {
|
||||
options.set(token, true)
|
||||
continue
|
||||
}
|
||||
if (VALUE_OPTIONS.has(token)) {
|
||||
const value = argv[i + 1]
|
||||
if (value === undefined || value.startsWith('--')) {
|
||||
fail(`Option ${token} expects a value`)
|
||||
}
|
||||
options.set(token, value)
|
||||
i += 1
|
||||
continue
|
||||
}
|
||||
fail(`Unknown option: ${token} (see --help)`)
|
||||
}
|
||||
return { positionals, options }
|
||||
}
|
||||
|
||||
function fail(message: string): never {
|
||||
process.stderr.write(`error: ${message}\n`)
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
function packageVersion(): string {
|
||||
try {
|
||||
const pkgPath = join(dirname(fileURLToPath(import.meta.url)), '..', 'package.json')
|
||||
return (JSON.parse(readFileSync(pkgPath, 'utf8')) as { version: string }).version
|
||||
} catch {
|
||||
return 'unknown'
|
||||
}
|
||||
}
|
||||
|
||||
const HELP = `llm-fingerprint — fingerprint & verify LLMs behind OpenAI-compatible APIs
|
||||
Method: "One Token Is Enough" (Bruckner, arXiv:2607.10252)
|
||||
|
||||
USAGE
|
||||
llm-fingerprint <command> [options]
|
||||
|
||||
COMMANDS
|
||||
fingerprint Probe an endpoint and print/save its behavioral fingerprint
|
||||
verify Fingerprint an endpoint and compare it to a reference
|
||||
compare Compare two saved fingerprints (files or bundled ids)
|
||||
references List bundled sample reference fingerprints
|
||||
|
||||
ENDPOINT OPTIONS
|
||||
--base-url <url> OpenAI-compatible base URL, e.g. https://api.openai.com/v1
|
||||
--model <id> Model id to request, e.g. gpt-4o-mini
|
||||
--api-key-env <name> Env var holding the API key
|
||||
(default: tries ${DEFAULT_KEY_ENV_VARS.join(', ')})
|
||||
--api-key <key> API key literal — avoid; prefer --api-key-env
|
||||
|
||||
SAMPLING OPTIONS
|
||||
--cells <n|list> Cell count 1-16 (top-N most discriminative) or a
|
||||
comma-separated list of cell ids (default: ${DEFAULT_CELL_COUNT})
|
||||
--samples <n> Samples per cell (default: ${DEFAULT_SAMPLES_PER_CELL})
|
||||
--preset <id> quick (4×15) | standard (8×25) | strict (16×25)
|
||||
--concurrency <n> Concurrent requests (default: ${DEFAULT_CONCURRENCY})
|
||||
--timeout <ms> Per-request timeout (default: 30000)
|
||||
|
||||
VERIFY / COMPARE
|
||||
--reference <src> Reference fingerprint: a JSON file produced by
|
||||
'fingerprint --out', or a bundled id (see 'references')
|
||||
|
||||
OUTPUT
|
||||
--json Machine-readable JSON on stdout
|
||||
--out <file> Write the fingerprint JSON to a file
|
||||
--quiet No progress output
|
||||
--help, -h Show this help
|
||||
--version, -V Show version
|
||||
|
||||
EXIT CODES (verify)
|
||||
0 match · 2 mismatch · 3 uncertain · 4 insufficient · 1 error
|
||||
|
||||
EXAMPLES
|
||||
# Fingerprint an endpoint (key read from OPENAI_API_KEY)
|
||||
llm-fingerprint fingerprint --base-url https://api.openai.com/v1 \\
|
||||
--model gpt-4o-mini --out gpt-4o-mini.fingerprint.json
|
||||
|
||||
# Is this cheap reseller really serving gpt-4o-mini?
|
||||
LLM_FINGERPRINT_API_KEY=sk-... llm-fingerprint verify \\
|
||||
--base-url https://cheap-api.example.com/v1 --model gpt-4o-mini \\
|
||||
--reference gpt-4o-mini.fingerprint.json
|
||||
|
||||
# Quick demo against a bundled sample reference
|
||||
llm-fingerprint verify --base-url https://openrouter.ai/api/v1 \\
|
||||
--model openai/gpt-4o-mini --reference openai/gpt-4o-mini --preset quick
|
||||
|
||||
# Compare two saved fingerprints offline
|
||||
llm-fingerprint compare a.fingerprint.json b.fingerprint.json
|
||||
|
||||
Web version (no install): https://tosea.ai/free-tools/llm-api-fingerprint-checker
|
||||
`
|
||||
|
||||
function readEndpoint(args: ParsedArgs): Endpoint {
|
||||
const baseUrl = args.options.get('--base-url')
|
||||
const model = args.options.get('--model')
|
||||
if (typeof baseUrl !== 'string') fail('--base-url is required')
|
||||
if (typeof model !== 'string') fail('--model is required')
|
||||
return { baseUrl, model, apiKey: resolveApiKey(args) }
|
||||
}
|
||||
|
||||
function resolveApiKey(args: ParsedArgs): string | undefined {
|
||||
const literal = args.options.get('--api-key')
|
||||
if (typeof literal === 'string' && literal.trim()) return literal.trim()
|
||||
|
||||
const envName = args.options.get('--api-key-env')
|
||||
if (typeof envName === 'string') {
|
||||
const value = process.env[envName]
|
||||
if (!value) fail(`Environment variable ${envName} is empty or not set`)
|
||||
return value
|
||||
}
|
||||
|
||||
for (const name of DEFAULT_KEY_ENV_VARS) {
|
||||
const value = process.env[name]
|
||||
if (value) return value
|
||||
}
|
||||
process.stderr.write(
|
||||
`note: no API key found (checked ${DEFAULT_KEY_ENV_VARS.join(', ')}); ` +
|
||||
'sending requests without Authorization header\n',
|
||||
)
|
||||
return undefined
|
||||
}
|
||||
|
||||
function readSamplingOptions(args: ParsedArgs): FingerprintOptions {
|
||||
const options: FingerprintOptions = {}
|
||||
|
||||
const preset = args.options.get('--preset')
|
||||
if (typeof preset === 'string') {
|
||||
const presets: Record<string, { cells: number; samples: number }> = {
|
||||
quick: { cells: 4, samples: 15 },
|
||||
standard: { cells: 8, samples: 25 },
|
||||
strict: { cells: 16, samples: 25 },
|
||||
}
|
||||
const found = presets[preset]
|
||||
if (!found) fail(`Unknown preset "${preset}" (quick | standard | strict)`)
|
||||
options.cells = found.cells
|
||||
options.samplesPerCell = found.samples
|
||||
}
|
||||
|
||||
const cells = args.options.get('--cells')
|
||||
if (typeof cells === 'string') {
|
||||
if (/^\d+$/.test(cells)) {
|
||||
const n = Number(cells)
|
||||
if (n < 1 || n > CELL_PRIORITY_ORDER.length) {
|
||||
fail(`--cells must be 1-${CELL_PRIORITY_ORDER.length} or a comma-separated cell list`)
|
||||
}
|
||||
options.cells = n
|
||||
} else {
|
||||
const list = cells.split(',').map((cell) => cell.trim())
|
||||
for (const cell of list) {
|
||||
if (!isCellId(cell)) {
|
||||
fail(`Unknown cell id "${cell}". Valid cells:\n ${CELL_PRIORITY_ORDER.join('\n ')}`)
|
||||
}
|
||||
}
|
||||
options.cells = list as CellId[]
|
||||
}
|
||||
}
|
||||
|
||||
const samples = args.options.get('--samples')
|
||||
if (typeof samples === 'string') {
|
||||
const n = Number(samples)
|
||||
if (!Number.isInteger(n) || n < 1) fail('--samples must be a positive integer')
|
||||
options.samplesPerCell = n
|
||||
}
|
||||
|
||||
const concurrency = args.options.get('--concurrency')
|
||||
if (typeof concurrency === 'string') {
|
||||
const n = Number(concurrency)
|
||||
if (!Number.isInteger(n) || n < 1) fail('--concurrency must be a positive integer')
|
||||
options.concurrency = n
|
||||
}
|
||||
|
||||
const timeout = args.options.get('--timeout')
|
||||
if (typeof timeout === 'string') {
|
||||
const n = Number(timeout)
|
||||
if (!Number.isFinite(n) || n < 100) fail('--timeout must be ≥ 100 (milliseconds)')
|
||||
options.timeoutMs = n
|
||||
}
|
||||
|
||||
return options
|
||||
}
|
||||
|
||||
function makeProgressRenderer(args: ParsedArgs): ((event: ProgressEvent) => void) | undefined {
|
||||
if (args.options.get('--quiet')) return undefined
|
||||
const isTty = process.stderr.isTTY === true
|
||||
let lastPercent = -1
|
||||
return (event) => {
|
||||
if (event.stage === 'adapter') {
|
||||
process.stderr.write(
|
||||
isTty
|
||||
? `\rprobing reasoning adapter (${event.strategy})... `
|
||||
: `probing reasoning adapter (${event.strategy})...\n`,
|
||||
)
|
||||
return
|
||||
}
|
||||
if (isTty) {
|
||||
const errs = event.errors > 0 ? `, errors: ${event.errors}` : ''
|
||||
process.stderr.write(`\rsampling ${event.done}/${event.total}${errs} `)
|
||||
if (event.done === event.total) process.stderr.write('\n')
|
||||
} else {
|
||||
const percent = Math.floor((event.done / event.total) * 10) * 10
|
||||
if (percent > lastPercent) {
|
||||
lastPercent = percent
|
||||
process.stderr.write(`sampling ${event.done}/${event.total} (${percent}%)\n`)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Load a reference: a JSON file path first, then a bundled sample id. */
|
||||
function loadReference(source: string): Fingerprint {
|
||||
let fileText: string | null = null
|
||||
try {
|
||||
fileText = readFileSync(source, 'utf8')
|
||||
} catch {
|
||||
fileText = null
|
||||
}
|
||||
if (fileText !== null) return parseFingerprintJson(fileText, source)
|
||||
try {
|
||||
return loadBundledReference(source)
|
||||
} catch (error) {
|
||||
fail(
|
||||
`"${source}" is neither a readable file nor a bundled reference id.\n${(error as Error).message}`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function verdictLabel(verdict: VerdictLevel): string {
|
||||
switch (verdict) {
|
||||
case 'match':
|
||||
return 'MATCH — behavior is consistent with the reference'
|
||||
case 'uncertain':
|
||||
return 'UNCERTAIN — in the gray zone; collect more samples or a fresh reference'
|
||||
case 'mismatch':
|
||||
return 'MISMATCH — behavior differs from the reference'
|
||||
case 'insufficient':
|
||||
return 'INSUFFICIENT — not enough comparable cells for a verdict'
|
||||
}
|
||||
}
|
||||
|
||||
function verdictExitCode(verdict: VerdictLevel): number {
|
||||
switch (verdict) {
|
||||
case 'match':
|
||||
return 0
|
||||
case 'mismatch':
|
||||
return 2
|
||||
case 'uncertain':
|
||||
return 3
|
||||
case 'insufficient':
|
||||
return 4
|
||||
}
|
||||
}
|
||||
|
||||
function renderComparison(result: ComparisonResult): string {
|
||||
const lines: string[] = []
|
||||
const mean = result.meanJsd === null ? 'n/a' : result.meanJsd.toFixed(3)
|
||||
lines.push(`Verdict: ${verdictLabel(result.verdict)}`)
|
||||
lines.push(`Mean JSD: ${mean} over ${result.comparableCellCount} comparable cell(s)`)
|
||||
lines.push('')
|
||||
lines.push('Interpretation scale (paper baselines, arXiv:2607.10252):')
|
||||
lines.push(
|
||||
` same model ≈ ${result.baselines.sameModelSelf} · same model, other provider ≈ ${result.baselines.sameModelCrossProvider} · different model ≈ ${result.baselines.differentModel}`,
|
||||
)
|
||||
lines.push(
|
||||
` thresholds: match ≤ ${result.thresholds.match} < uncertain ≤ ${result.thresholds.mismatch} < mismatch`,
|
||||
)
|
||||
if (result.cells.length > 0) {
|
||||
lines.push('')
|
||||
lines.push('Per-cell JSD (most divergent first):')
|
||||
for (const cell of result.cells) {
|
||||
lines.push(
|
||||
` ${cell.cellId.padEnd(26)} ${cell.jsd.toFixed(3)} (${cell.validA} vs ${cell.validB} valid)`,
|
||||
)
|
||||
}
|
||||
}
|
||||
if (result.protocolMismatch) {
|
||||
lines.push('')
|
||||
lines.push(
|
||||
'note: the fingerprints were collected under different probe protocols; treat the verdict as indicative only.',
|
||||
)
|
||||
}
|
||||
return lines.join('\n')
|
||||
}
|
||||
|
||||
function summarizeFingerprint(fp: Fingerprint): string {
|
||||
const lines: string[] = []
|
||||
lines.push(`Model: ${fp.model}`)
|
||||
lines.push(`Protocol: ${fp.protocol} · collected ${fp.collectedAt}`)
|
||||
lines.push(`Cells: ${Object.keys(fp.cells).length} × ${fp.samplesPerCell} samples`)
|
||||
if (fp.postReasoning) lines.push('warning: collected via post-reasoning fallback (reduced confidence)')
|
||||
lines.push('')
|
||||
lines.push('Top answers per cell:')
|
||||
for (const [cellId, cell] of Object.entries(fp.cells)) {
|
||||
if (!cell) continue
|
||||
const top = Object.entries(cell.counts)
|
||||
.sort((a, b) => b[1] - a[1])
|
||||
.slice(0, 3)
|
||||
.map(([answer, count]) => `${answer}×${count}`)
|
||||
.join(', ')
|
||||
lines.push(
|
||||
` ${cellId.padEnd(26)} valid ${String(cell.validCount).padStart(3)} H=${cell.entropyBits.toFixed(2)}b ${top || '(no valid answers)'}`,
|
||||
)
|
||||
}
|
||||
return lines.join('\n')
|
||||
}
|
||||
|
||||
function writeWarnings(warnings: string[]): void {
|
||||
for (const warning of warnings) process.stderr.write(`warning: ${warning}\n`)
|
||||
}
|
||||
|
||||
async function cmdFingerprint(args: ParsedArgs): Promise<number> {
|
||||
const endpoint = readEndpoint(args)
|
||||
const options = readSamplingOptions(args)
|
||||
options.onProgress = makeProgressRenderer(args)
|
||||
|
||||
const run = await fingerprint(endpoint, options)
|
||||
writeWarnings(run.warnings)
|
||||
|
||||
const out = args.options.get('--out')
|
||||
if (typeof out === 'string') {
|
||||
writeFileSync(out, `${JSON.stringify(run.fingerprint, null, 2)}\n`, 'utf8')
|
||||
process.stderr.write(`fingerprint written to ${out}\n`)
|
||||
}
|
||||
|
||||
if (args.options.get('--json')) {
|
||||
const { fingerprint: fp, adapter, errorCount, splitHalfJsd, durationMs, warnings } = run
|
||||
process.stdout.write(
|
||||
`${JSON.stringify({ fingerprint: fp, run: { adapter, errorCount, splitHalfJsd, durationMs, warnings } }, null, 2)}\n`,
|
||||
)
|
||||
} else if (typeof out !== 'string') {
|
||||
process.stdout.write(`${summarizeFingerprint(run.fingerprint)}\n`)
|
||||
} else {
|
||||
process.stderr.write(`${summarizeFingerprint(run.fingerprint)}\n`)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
async function cmdVerify(args: ParsedArgs): Promise<number> {
|
||||
const referenceSource = args.options.get('--reference')
|
||||
if (typeof referenceSource !== 'string') {
|
||||
fail('--reference <file-or-bundled-id> is required (see `llm-fingerprint references`)')
|
||||
}
|
||||
const reference = loadReference(referenceSource)
|
||||
const endpoint = readEndpoint(args)
|
||||
const options = readSamplingOptions(args)
|
||||
options.onProgress = makeProgressRenderer(args)
|
||||
|
||||
const result = await verify(endpoint, reference, options)
|
||||
writeWarnings(result.warnings)
|
||||
|
||||
const out = args.options.get('--out')
|
||||
if (typeof out === 'string') {
|
||||
writeFileSync(out, `${JSON.stringify(result.target.fingerprint, null, 2)}\n`, 'utf8')
|
||||
process.stderr.write(`target fingerprint written to ${out}\n`)
|
||||
}
|
||||
|
||||
if (args.options.get('--json')) {
|
||||
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`)
|
||||
} else {
|
||||
process.stdout.write(`${renderComparison(result.comparison)}\n`)
|
||||
process.stdout.write(
|
||||
`\nReference: ${reference.model} (collected ${reference.collectedAt}, protocol ${reference.protocol})\n`,
|
||||
)
|
||||
}
|
||||
return verdictExitCode(result.verdict)
|
||||
}
|
||||
|
||||
async function cmdCompare(args: ParsedArgs): Promise<number> {
|
||||
if (args.positionals.length !== 2) {
|
||||
fail('compare expects exactly two arguments: <fingerprint-a> <fingerprint-b>')
|
||||
}
|
||||
const a = loadReference(args.positionals[0])
|
||||
const b = loadReference(args.positionals[1])
|
||||
const result = compare(a, b)
|
||||
|
||||
if (args.options.get('--json')) {
|
||||
process.stdout.write(`${JSON.stringify({ a: a.model, b: b.model, ...result }, null, 2)}\n`)
|
||||
} else {
|
||||
process.stdout.write(`A: ${a.model} (${a.collectedAt})\nB: ${b.model} (${b.collectedAt})\n\n`)
|
||||
process.stdout.write(`${renderComparison(result)}\n`)
|
||||
}
|
||||
return verdictExitCode(result.verdict)
|
||||
}
|
||||
|
||||
function cmdReferences(args: ParsedArgs): number {
|
||||
const references = listBundledReferences()
|
||||
const attribution = getBundledAttribution()
|
||||
if (args.options.get('--json')) {
|
||||
process.stdout.write(`${JSON.stringify({ source: attribution, references }, null, 2)}\n`)
|
||||
return 0
|
||||
}
|
||||
process.stdout.write('Bundled sample reference fingerprints:\n\n')
|
||||
for (const ref of references) {
|
||||
process.stdout.write(
|
||||
` ${ref.id.padEnd(36)} ${String(ref.cellCount).padStart(2)} cells collected ${ref.collectedAt}\n`,
|
||||
)
|
||||
}
|
||||
process.stdout.write(
|
||||
`\nSource: ${attribution.dataset}\n` +
|
||||
`by ${attribution.author} — DOI ${attribution.datasetDoi} (${attribution.license})\n` +
|
||||
'Collected under the paper\'s protocol: fine for demos; for high-stakes checks,\n' +
|
||||
'collect your own reference with `llm-fingerprint fingerprint --out ...`.\n',
|
||||
)
|
||||
return 0
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const argv = process.argv.slice(2)
|
||||
const args = parseArgs(argv)
|
||||
|
||||
if (args.options.get('--version') || args.options.get('-V')) {
|
||||
process.stdout.write(`${packageVersion()}\n`)
|
||||
process.exit(0)
|
||||
}
|
||||
const command = args.positionals.shift()
|
||||
if (!command || args.options.get('--help') || args.options.get('-h') || command === 'help') {
|
||||
process.stdout.write(HELP)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
try {
|
||||
let exitCode: number
|
||||
switch (command) {
|
||||
case 'fingerprint':
|
||||
exitCode = await cmdFingerprint(args)
|
||||
break
|
||||
case 'verify':
|
||||
exitCode = await cmdVerify(args)
|
||||
break
|
||||
case 'compare':
|
||||
exitCode = await cmdCompare(args)
|
||||
break
|
||||
case 'references':
|
||||
exitCode = cmdReferences(args)
|
||||
break
|
||||
default:
|
||||
fail(`Unknown command: ${command} (see --help)`)
|
||||
}
|
||||
process.exit(exitCode)
|
||||
} catch (error) {
|
||||
if (error instanceof ProbeRunError) {
|
||||
const hints: Record<string, string> = {
|
||||
auth: 'The endpoint rejected the API key (401/403).',
|
||||
network: 'The endpoint is unreachable — check the base URL and your network.',
|
||||
aborted: 'Run cancelled.',
|
||||
}
|
||||
fail(`${hints[error.reason] ?? ''} ${error.message}`.trim())
|
||||
}
|
||||
fail(error instanceof Error ? error.message : String(error))
|
||||
}
|
||||
}
|
||||
|
||||
void main()
|
||||
@@ -0,0 +1,60 @@
|
||||
/**
|
||||
* Thresholds and statistical constants, centralized for calibration.
|
||||
*
|
||||
* Baselines from Bruckner, "One Token Is Enough" (arXiv:2607.10252):
|
||||
* - same model, split-half distance (median) ≈ 0.140
|
||||
* - same model served by different providers (median) ≈ 0.227
|
||||
* - different models (median) ≈ 0.463
|
||||
* - equal error rate: ≈ 10.6% with 8 cells, ≈ 7.3% with 40 cells
|
||||
*
|
||||
* The match/mismatch cut points below sit between those baselines and leave a
|
||||
* deliberate "uncertain" band; they are heuristics, not proofs.
|
||||
*/
|
||||
|
||||
/** meanJsd ≤ this → `match` (paper same-model cross-provider median 0.227, plus margin). */
|
||||
export const JSD_MATCH_THRESHOLD = 0.25
|
||||
|
||||
/**
|
||||
* meanJsd > this → `mismatch` (paper different-model median 0.463; impostor
|
||||
* distances rarely fall below ≈ 0.3). Between the two thresholds → `uncertain`.
|
||||
*/
|
||||
export const JSD_MISMATCH_THRESHOLD = 0.35
|
||||
|
||||
/** Paper baseline anchors, exposed for result interpretation. */
|
||||
export const JSD_BASELINE_SELF = 0.14
|
||||
export const JSD_BASELINE_CROSS_PROVIDER = 0.227
|
||||
export const JSD_BASELINE_DIFFERENT_MODEL = 0.463
|
||||
|
||||
/** A cell participates in the distance only when both sides have ≥ this many valid samples. */
|
||||
export const MIN_VALID_SAMPLES_PER_CELL = 10
|
||||
|
||||
/** Fewer comparable cells than this → verdict `insufficient`. */
|
||||
export const MIN_COMPARABLE_CELLS = 4
|
||||
|
||||
/** Split-half self check: a cell participates only when each half has ≥ this many valid samples. */
|
||||
export const MIN_SPLIT_HALF_SAMPLES = 5
|
||||
|
||||
/** Split-half JSD above this suggests unstable routing (multi-backend aggregator). */
|
||||
export const SPLIT_HALF_WARN_THRESHOLD = 0.25
|
||||
|
||||
/** Probe request parameters (paper protocol). */
|
||||
export const PROBE_TEMPERATURE = 1.0
|
||||
export const PROBE_MAX_TOKENS = 16
|
||||
/** Fallback max_tokens when reasoning cannot be disabled (post-reasoning channel). */
|
||||
export const POST_REASONING_MAX_TOKENS = 1024
|
||||
|
||||
/** Sampler defaults. */
|
||||
export const DEFAULT_SAMPLES_PER_CELL = 25
|
||||
export const DEFAULT_CELL_COUNT = 8
|
||||
export const DEFAULT_CONCURRENCY = 4
|
||||
export const DEFAULT_REQUEST_TIMEOUT_MS = 30_000
|
||||
export const DEFAULT_MAX_RETRIES = 2
|
||||
/** Abort the run after this many consecutive transport-level failures. */
|
||||
export const CONSECUTIVE_NETWORK_ERROR_LIMIT = 8
|
||||
|
||||
/** Fingerprint artifact identifiers. */
|
||||
export const FINGERPRINT_FORMAT_VERSION = 1 as const
|
||||
/** Protocol id for fingerprints collected by this package's battery. */
|
||||
export const PROBE_PROTOCOL = 'one-token/v1'
|
||||
/** Protocol id for samples derived from the paper's Zenodo dataset. */
|
||||
export const ZENODO_PROTOCOL = 'bruckner-zenodo-2026'
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* Endpoint normalization and per-provider reasoning-strategy hints.
|
||||
*/
|
||||
|
||||
import type { Endpoint, ReasoningStrategyId, ResolvedEndpoint } from './types.js'
|
||||
|
||||
export interface BaseUrlNormalization {
|
||||
ok: boolean
|
||||
normalized: string
|
||||
reason?: 'empty' | 'invalid'
|
||||
/** Non-fatal notes, e.g. plain-http endpoints. */
|
||||
warnings: string[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Base URL cleanup:
|
||||
* - trim whitespace and trailing slashes; drop an accidentally pasted
|
||||
* `/chat/completions` suffix;
|
||||
* - bare domains (no path) get `/v1` appended;
|
||||
* - plain `http://` is allowed (local vLLM/Ollama/LM Studio) but flagged
|
||||
* with a warning for non-local hosts.
|
||||
*/
|
||||
export function normalizeBaseUrl(input: string): BaseUrlNormalization {
|
||||
const trimmed = (input ?? '').trim()
|
||||
if (!trimmed) return { ok: false, normalized: '', reason: 'empty', warnings: [] }
|
||||
|
||||
const withScheme = /^https?:\/\//i.test(trimmed) ? trimmed : `https://${trimmed}`
|
||||
let url: URL
|
||||
try {
|
||||
url = new URL(withScheme)
|
||||
} catch {
|
||||
return { ok: false, normalized: trimmed, reason: 'invalid', warnings: [] }
|
||||
}
|
||||
|
||||
const warnings: string[] = []
|
||||
const isLocal =
|
||||
url.hostname === 'localhost' || url.hostname === '127.0.0.1' || url.hostname === '::1'
|
||||
if (url.protocol === 'http:' && !isLocal) {
|
||||
warnings.push(`Plain http:// endpoint (${url.host}): the API key is sent unencrypted.`)
|
||||
}
|
||||
|
||||
let path = url.pathname.replace(/\/+$/, '')
|
||||
path = path.replace(/\/chat\/completions$/, '')
|
||||
if (path === '' || path === '/') path = '/v1'
|
||||
|
||||
return { ok: true, normalized: `${url.protocol}//${url.host}${path}`, warnings }
|
||||
}
|
||||
|
||||
/** Order in which reasoning-disable strategies are probed for a given host. */
|
||||
const HOST_ADAPTER_HINTS: Array<{ hostIncludes: string; hint: ReasoningStrategyId[] }> = [
|
||||
{ hostIncludes: 'openrouter.ai', hint: ['openrouter-reasoning', 'openai-effort', 'zhipu-thinking'] },
|
||||
{ hostIncludes: 'api.openai.com', hint: ['openai-effort', 'openrouter-reasoning', 'zhipu-thinking'] },
|
||||
{ hostIncludes: 'api.deepseek.com', hint: ['openai-effort', 'zhipu-thinking', 'openrouter-reasoning'] },
|
||||
{ hostIncludes: 'bigmodel.cn', hint: ['zhipu-thinking', 'openai-effort', 'openrouter-reasoning'] },
|
||||
]
|
||||
|
||||
export const DEFAULT_ADAPTER_HINT: ReasoningStrategyId[] = [
|
||||
'openrouter-reasoning',
|
||||
'zhipu-thinking',
|
||||
'openai-effort',
|
||||
]
|
||||
|
||||
export function guessAdapterHint(baseUrl: string): ReasoningStrategyId[] {
|
||||
const lower = baseUrl.toLowerCase()
|
||||
for (const { hostIncludes, hint } of HOST_ADAPTER_HINTS) {
|
||||
if (lower.includes(hostIncludes)) return hint
|
||||
}
|
||||
return DEFAULT_ADAPTER_HINT
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate and normalize a user-supplied endpoint. Throws on empty/invalid
|
||||
* base URL; returns the resolved endpoint plus non-fatal warnings.
|
||||
*/
|
||||
export function resolveEndpoint(endpoint: Endpoint): {
|
||||
resolved: ResolvedEndpoint
|
||||
warnings: string[]
|
||||
} {
|
||||
const { ok, normalized, reason, warnings } = normalizeBaseUrl(endpoint.baseUrl)
|
||||
if (!ok) {
|
||||
throw new Error(
|
||||
reason === 'empty' ? 'Endpoint baseUrl is empty' : `Invalid baseUrl: ${endpoint.baseUrl}`,
|
||||
)
|
||||
}
|
||||
if (!endpoint.model || !endpoint.model.trim()) {
|
||||
throw new Error('Endpoint model is empty')
|
||||
}
|
||||
return {
|
||||
resolved: {
|
||||
baseUrl: normalized,
|
||||
model: endpoint.model.trim(),
|
||||
apiKey: endpoint.apiKey?.trim() || null,
|
||||
headers: endpoint.headers ?? {},
|
||||
},
|
||||
warnings,
|
||||
}
|
||||
}
|
||||
+215
@@ -0,0 +1,215 @@
|
||||
/**
|
||||
* Minimal HTTP layer over `POST {baseUrl}/chat/completions`:
|
||||
* timeout + exponential backoff on 429/5xx/timeouts + AbortSignal
|
||||
* pass-through + error classification. Uses the global `fetch`
|
||||
* (Node ≥ 18 built-in, or any browser).
|
||||
*
|
||||
* The API key only ever appears in the Authorization header of the request
|
||||
* to the endpoint under test; it is never logged or sent anywhere else.
|
||||
*/
|
||||
|
||||
import { DEFAULT_MAX_RETRIES, DEFAULT_REQUEST_TIMEOUT_MS } from './constants.js'
|
||||
import type { ResolvedEndpoint, SampleUsage } from './types.js'
|
||||
|
||||
export type ProbeErrorKind = 'network' | 'auth' | 'http' | 'timeout' | 'aborted'
|
||||
|
||||
export class ProbeRequestError extends Error {
|
||||
readonly kind: ProbeErrorKind
|
||||
readonly status: number | null
|
||||
|
||||
constructor(kind: ProbeErrorKind, message: string, status: number | null = null) {
|
||||
super(message)
|
||||
this.name = 'ProbeRequestError'
|
||||
this.kind = kind
|
||||
this.status = status
|
||||
}
|
||||
}
|
||||
|
||||
export interface ChatCompletionResult {
|
||||
content: string
|
||||
usage: SampleUsage | null
|
||||
latencyMs: number
|
||||
status: number
|
||||
}
|
||||
|
||||
interface ChatCompletionUsagePayload {
|
||||
prompt_tokens?: number
|
||||
completion_tokens?: number
|
||||
reasoning_tokens?: number
|
||||
completion_tokens_details?: { reasoning_tokens?: number }
|
||||
}
|
||||
|
||||
function parseUsage(payload: ChatCompletionUsagePayload | undefined | null): SampleUsage | null {
|
||||
if (!payload) return null
|
||||
return {
|
||||
promptTokens: typeof payload.prompt_tokens === 'number' ? payload.prompt_tokens : null,
|
||||
completionTokens:
|
||||
typeof payload.completion_tokens === 'number' ? payload.completion_tokens : null,
|
||||
reasoningTokens:
|
||||
typeof payload.completion_tokens_details?.reasoning_tokens === 'number'
|
||||
? payload.completion_tokens_details.reasoning_tokens
|
||||
: typeof payload.reasoning_tokens === 'number'
|
||||
? payload.reasoning_tokens
|
||||
: null,
|
||||
}
|
||||
}
|
||||
|
||||
/** `content` is a string in the OpenAI schema, but some gateways send part arrays. */
|
||||
function extractContent(message: unknown): string {
|
||||
if (!message || typeof message !== 'object') return ''
|
||||
const content = (message as { content?: unknown }).content
|
||||
if (typeof content === 'string') return content
|
||||
if (Array.isArray(content)) {
|
||||
return content
|
||||
.map((part) =>
|
||||
typeof part === 'string'
|
||||
? part
|
||||
: typeof (part as { text?: unknown })?.text === 'string'
|
||||
? (part as { text: string }).text
|
||||
: '',
|
||||
)
|
||||
.join('')
|
||||
}
|
||||
return ''
|
||||
}
|
||||
|
||||
export interface ChatCompletionRequest {
|
||||
endpoint: ResolvedEndpoint
|
||||
systemPrompt: string
|
||||
userPrompt: string
|
||||
temperature: number
|
||||
maxTokens: number
|
||||
extraBody: Record<string, unknown>
|
||||
signal?: AbortSignal
|
||||
timeoutMs?: number
|
||||
retries?: number
|
||||
}
|
||||
|
||||
async function delay(ms: number, signal?: AbortSignal): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const timer = setTimeout(() => {
|
||||
signal?.removeEventListener('abort', onAbort)
|
||||
resolve()
|
||||
}, ms)
|
||||
const onAbort = () => {
|
||||
clearTimeout(timer)
|
||||
reject(new ProbeRequestError('aborted', 'Aborted'))
|
||||
}
|
||||
signal?.addEventListener('abort', onAbort, { once: true })
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* One chat/completions call with retries. Retried: 429, 5xx, timeouts.
|
||||
* Not retried: transport errors and 401/403 — those are classified and
|
||||
* re-thrown for the caller to decide.
|
||||
*/
|
||||
export async function fetchChatCompletion(
|
||||
request: ChatCompletionRequest,
|
||||
): Promise<ChatCompletionResult> {
|
||||
const retries = request.retries ?? DEFAULT_MAX_RETRIES
|
||||
const timeoutMs = request.timeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS
|
||||
let lastError: ProbeRequestError | null = null
|
||||
|
||||
for (let attempt = 0; attempt <= retries; attempt++) {
|
||||
if (request.signal?.aborted) throw new ProbeRequestError('aborted', 'Aborted')
|
||||
|
||||
const timeoutController = new AbortController()
|
||||
const timer = setTimeout(() => timeoutController.abort(), timeoutMs)
|
||||
const onOuterAbort = () => timeoutController.abort()
|
||||
request.signal?.addEventListener('abort', onOuterAbort, { once: true })
|
||||
|
||||
const startedAt = performance.now()
|
||||
try {
|
||||
const headers: Record<string, string> = {
|
||||
'Content-Type': 'application/json',
|
||||
...request.endpoint.headers,
|
||||
}
|
||||
if (request.endpoint.apiKey) {
|
||||
headers.Authorization = `Bearer ${request.endpoint.apiKey}`
|
||||
}
|
||||
const response = await fetch(`${request.endpoint.baseUrl}/chat/completions`, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: JSON.stringify({
|
||||
model: request.endpoint.model,
|
||||
temperature: request.temperature,
|
||||
max_tokens: request.maxTokens,
|
||||
stream: false,
|
||||
messages: [
|
||||
{ role: 'system', content: request.systemPrompt },
|
||||
{ role: 'user', content: request.userPrompt },
|
||||
],
|
||||
...request.extraBody,
|
||||
}),
|
||||
signal: timeoutController.signal,
|
||||
})
|
||||
const latencyMs = performance.now() - startedAt
|
||||
|
||||
if (response.status === 401 || response.status === 403) {
|
||||
throw new ProbeRequestError(
|
||||
'auth',
|
||||
`HTTP ${response.status} — API key rejected`,
|
||||
response.status,
|
||||
)
|
||||
}
|
||||
if (response.status === 429 || response.status >= 500) {
|
||||
const retryAfterHeader = response.headers.get('retry-after')
|
||||
const retryAfterMs = retryAfterHeader ? Number(retryAfterHeader) * 1000 : NaN
|
||||
lastError = new ProbeRequestError('http', `HTTP ${response.status}`, response.status)
|
||||
if (attempt < retries) {
|
||||
const backoff = Number.isFinite(retryAfterMs)
|
||||
? retryAfterMs
|
||||
: 800 * 2 ** attempt + Math.random() * 400
|
||||
await delay(backoff, request.signal)
|
||||
continue
|
||||
}
|
||||
throw lastError
|
||||
}
|
||||
if (!response.ok) {
|
||||
let detail = ''
|
||||
try {
|
||||
detail = (await response.text()).slice(0, 300)
|
||||
} catch {
|
||||
// Ignore body read failures.
|
||||
}
|
||||
throw new ProbeRequestError(
|
||||
'http',
|
||||
`HTTP ${response.status} ${detail}`.trim(),
|
||||
response.status,
|
||||
)
|
||||
}
|
||||
|
||||
const payload = (await response.json()) as {
|
||||
choices?: Array<{ message?: unknown }>
|
||||
usage?: ChatCompletionUsagePayload
|
||||
}
|
||||
return {
|
||||
content: extractContent(payload?.choices?.[0]?.message),
|
||||
usage: parseUsage(payload?.usage),
|
||||
latencyMs,
|
||||
status: response.status,
|
||||
}
|
||||
} catch (error) {
|
||||
if (error instanceof ProbeRequestError) throw error
|
||||
if (request.signal?.aborted) throw new ProbeRequestError('aborted', 'Aborted')
|
||||
if (timeoutController.signal.aborted) {
|
||||
lastError = new ProbeRequestError('timeout', `Request timed out after ${timeoutMs}ms`)
|
||||
} else {
|
||||
lastError = new ProbeRequestError(
|
||||
'network',
|
||||
error instanceof Error ? error.message : 'Network error',
|
||||
)
|
||||
}
|
||||
if (lastError.kind === 'timeout' && attempt < retries) {
|
||||
continue
|
||||
}
|
||||
throw lastError
|
||||
} finally {
|
||||
clearTimeout(timer)
|
||||
request.signal?.removeEventListener('abort', onOuterAbort)
|
||||
}
|
||||
}
|
||||
|
||||
throw lastError ?? new ProbeRequestError('network', 'Unknown error')
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
/**
|
||||
* llm-fingerprint-detector — fingerprint and verify LLMs behind
|
||||
* OpenAI-compatible APIs from single-token output distributions.
|
||||
*
|
||||
* Independent open-source implementation of Tomáš Bruckner,
|
||||
* "One Token Is Enough: Fingerprinting and Verifying Large Language Models
|
||||
* from Single-Token Output Distributions" (arXiv:2607.10252).
|
||||
*
|
||||
* Everything exported here is runtime-agnostic (Node ≥ 18 or browsers with
|
||||
* fetch). Bundled sample references are Node-only and live in the
|
||||
* `llm-fingerprint-detector/references` subpath export.
|
||||
*/
|
||||
|
||||
export { fingerprint, compare, verify } from './api.js'
|
||||
|
||||
export {
|
||||
CELL_PRIORITY_ORDER,
|
||||
PROBE_PRESETS,
|
||||
PROBE_TASKS,
|
||||
SYSTEM_PROMPTS,
|
||||
getCellsForPreset,
|
||||
getSystemPrompt,
|
||||
getTaskSpec,
|
||||
isCellId,
|
||||
makeCellId,
|
||||
parseCellId,
|
||||
pickParaphrase,
|
||||
} from './battery.js'
|
||||
|
||||
export {
|
||||
normalizeAnswer,
|
||||
parseAnyNumber,
|
||||
parseChineseNumeral,
|
||||
parseEnglishNumberWord,
|
||||
} from './normalizer.js'
|
||||
export type { NormalizedAnswer } from './normalizer.js'
|
||||
|
||||
export {
|
||||
buildCellDistribution,
|
||||
compareCellSets,
|
||||
domainSize,
|
||||
jensenShannonDivergence,
|
||||
median,
|
||||
shannonEntropyBits,
|
||||
splitHalfJsd,
|
||||
} from './stats.js'
|
||||
export type { CellJsdEntry, CountMap } from './stats.js'
|
||||
|
||||
export { decideVerdict, buildComparisonResult } from './verdict.js'
|
||||
|
||||
export { detectReasoningAdapter, STRATEGY_BODIES } from './adapter.js'
|
||||
export type { AdapterDetectionOptions } from './adapter.js'
|
||||
|
||||
export { runProbeBattery, ProbeRunError } from './sampler.js'
|
||||
export type { SamplerOptions, SamplerResult } from './sampler.js'
|
||||
|
||||
export { fetchChatCompletion, ProbeRequestError } from './http.js'
|
||||
export type { ChatCompletionRequest, ChatCompletionResult, ProbeErrorKind } from './http.js'
|
||||
|
||||
export { normalizeBaseUrl, resolveEndpoint, guessAdapterHint } from './endpoint.js'
|
||||
export type { BaseUrlNormalization } from './endpoint.js'
|
||||
|
||||
export * from './constants.js'
|
||||
|
||||
export type {
|
||||
AnswerDomain,
|
||||
CellComparison,
|
||||
CellDistribution,
|
||||
CellId,
|
||||
ComparisonBaselines,
|
||||
ComparisonResult,
|
||||
Endpoint,
|
||||
Fingerprint,
|
||||
FingerprintOptions,
|
||||
FingerprintRun,
|
||||
ProbeLang,
|
||||
ProbePreset,
|
||||
ProbePresetId,
|
||||
ProbeTaskId,
|
||||
ProbeTaskSpec,
|
||||
ProgressEvent,
|
||||
ReasoningAdapter,
|
||||
ReasoningStrategyId,
|
||||
ResolvedEndpoint,
|
||||
SampleCategory,
|
||||
SampleResult,
|
||||
SampleUsage,
|
||||
VerdictLevel,
|
||||
VerifyResult,
|
||||
} from './types.js'
|
||||
@@ -0,0 +1,203 @@
|
||||
/**
|
||||
* Answer normalization pipeline (pure functions):
|
||||
*
|
||||
* NFC → trim → refusal detection → strip punctuation/quotes/emoji
|
||||
* → case fold → take first word
|
||||
* → digit unification (Chinese numerals / English number words /
|
||||
* full-width and Arabic-Indic digits → Latin digits)
|
||||
* → color canonicalization (蓝色→蓝, grey→gray)
|
||||
* → coin folding (正/正面/heads → heads)
|
||||
*
|
||||
* Categories: valid (inside the cell's answer domain) / invalid / refusal /
|
||||
* empty. Models frequently answer "seven", "四十二" or "forty-two" instead of
|
||||
* "7"/"42", so variants must be folded before distributions are compared.
|
||||
*/
|
||||
|
||||
import type { AnswerDomain, SampleCategory } from './types.js'
|
||||
|
||||
const REFUSAL_PATTERNS: RegExp[] = [
|
||||
/\bas an ai\b/i,
|
||||
/\bi (?:cannot|can't|can not|won't|will not)\b/i,
|
||||
/\bi'?m (?:unable|not able|sorry)\b/i,
|
||||
/\bsorry,? (?:i|but)\b/i,
|
||||
/\bcannot (?:comply|assist|help)\b/i,
|
||||
/我不能/,
|
||||
/我无法/,
|
||||
/无法回答/,
|
||||
/不能回答/,
|
||||
/抱歉/,
|
||||
/对不起/,
|
||||
/作为(?:一个)?(?:AI|人工智能)/i,
|
||||
]
|
||||
|
||||
/**
|
||||
* Remove everything that is not a letter, digit or whitespace (quotes,
|
||||
* punctuation, emoji). In-word hyphens are removed too: forty-seven → fortyseven.
|
||||
*/
|
||||
function stripPunctuation(value: string): string {
|
||||
return value.replace(/[^\p{L}\p{N}\s]/gu, '')
|
||||
}
|
||||
|
||||
/** Full-width and Arabic-Indic digits → Latin digits. */
|
||||
function normalizeDigitScript(value: string): string {
|
||||
return value.replace(/[\uFF10-\uFF19\u0660-\u0669\u06F0-\u06F9]/g, (ch) => {
|
||||
const code = ch.charCodeAt(0)
|
||||
if (code >= 0xff10 && code <= 0xff19) return String(code - 0xff10)
|
||||
if (code >= 0x0660 && code <= 0x0669) return String(code - 0x0660)
|
||||
return String(code - 0x06f0)
|
||||
})
|
||||
}
|
||||
|
||||
const CN_DIGITS: Record<string, number> = {
|
||||
零: 0, 〇: 0, 一: 1, 二: 2, 两: 2, 三: 3, 四: 4,
|
||||
五: 5, 六: 6, 七: 7, 八: 8, 九: 9,
|
||||
}
|
||||
const CN_UNITS: Record<string, number> = { 十: 10, 百: 100, 千: 1000 }
|
||||
|
||||
/** Chinese numerals (一二三…百/千, incl. 两) → number; null when not a pure numeral. */
|
||||
export function parseChineseNumeral(value: string): number | null {
|
||||
if (!value || !/^[零〇一二两三四五六七八九十百千]+$/.test(value)) return null
|
||||
let total = 0
|
||||
let current = 0
|
||||
for (const ch of value) {
|
||||
if (ch in CN_DIGITS) {
|
||||
current = CN_DIGITS[ch]
|
||||
} else {
|
||||
const unit = CN_UNITS[ch]
|
||||
// A leading 十 (十, 十五) counts as 1 × 10.
|
||||
total += (current === 0 ? 1 : current) * unit
|
||||
current = 0
|
||||
}
|
||||
}
|
||||
return total + current
|
||||
}
|
||||
|
||||
const EN_ONES: Record<string, number> = {
|
||||
zero: 0, one: 1, two: 2, three: 3, four: 4, five: 5, six: 6, seven: 7,
|
||||
eight: 8, nine: 9, ten: 10, eleven: 11, twelve: 12, thirteen: 13,
|
||||
fourteen: 14, fifteen: 15, sixteen: 16, seventeen: 17, eighteen: 18,
|
||||
nineteen: 19,
|
||||
}
|
||||
const EN_TENS: Record<string, number> = {
|
||||
twenty: 20, thirty: 30, forty: 40, fifty: 50, sixty: 60, seventy: 70,
|
||||
eighty: 80, ninety: 90,
|
||||
}
|
||||
|
||||
/**
|
||||
* English number words → number, including de-hyphenated compounds
|
||||
* (fortyseven, onehundred). Null when unparseable.
|
||||
*/
|
||||
export function parseEnglishNumberWord(value: string): number | null {
|
||||
const word = value.toLowerCase()
|
||||
if (word in EN_ONES) return EN_ONES[word]
|
||||
if (word in EN_TENS) return EN_TENS[word]
|
||||
if (word === 'hundred' || word === 'onehundred') return 100
|
||||
for (const [tens, tensValue] of Object.entries(EN_TENS)) {
|
||||
if (word.startsWith(tens)) {
|
||||
const rest = word.slice(tens.length)
|
||||
if (rest in EN_ONES && EN_ONES[rest] >= 1 && EN_ONES[rest] <= 9) {
|
||||
return tensValue + EN_ONES[rest]
|
||||
}
|
||||
}
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
/** Any-format number parsing: Latin digits / Chinese numerals / English words. */
|
||||
export function parseAnyNumber(value: string): number | null {
|
||||
if (/^\d+$/.test(value)) return Number(value)
|
||||
const cn = parseChineseNumeral(value)
|
||||
if (cn !== null) return cn
|
||||
return parseEnglishNumberWord(value)
|
||||
}
|
||||
|
||||
/** English letter names → letter (zee/kay/queue and friends show up in the wild). */
|
||||
const EN_LETTER_NAMES: Record<string, string> = {
|
||||
bee: 'b', cee: 'c', dee: 'd', gee: 'g', jay: 'j', kay: 'k',
|
||||
el: 'l', ell: 'l', em: 'm', en: 'n', oh: 'o', pee: 'p',
|
||||
cue: 'q', queue: 'q', ar: 'r', es: 's', ess: 's', tee: 't',
|
||||
vee: 'v', ex: 'x', why: 'y', zee: 'z', zed: 'z',
|
||||
}
|
||||
|
||||
/** Color variant folding (within a language). */
|
||||
const EN_COLOR_ALIASES: Record<string, string> = {
|
||||
grey: 'gray',
|
||||
aqua: 'cyan',
|
||||
}
|
||||
|
||||
function normalizeColorWord(word: string): string {
|
||||
if (EN_COLOR_ALIASES[word]) return EN_COLOR_ALIASES[word]
|
||||
// Chinese: 蓝色→蓝, 青色→青 (multi-character names keep their stem; 靛蓝/蔚蓝 unchanged).
|
||||
if (/^[\u4e00-\u9fff]{2,}$/.test(word) && word.endsWith('色')) {
|
||||
return word.slice(0, -1)
|
||||
}
|
||||
return word
|
||||
}
|
||||
|
||||
const COIN_HEADS = new Set(['heads', 'head', '正', '正面', '字'])
|
||||
const COIN_TAILS = new Set(['tails', 'tail', '反', '反面', '花'])
|
||||
|
||||
function normalizeCoinWord(word: string): string | null {
|
||||
if (COIN_HEADS.has(word)) return 'heads'
|
||||
if (COIN_TAILS.has(word)) return 'tails'
|
||||
if (/^正面?/.test(word)) return 'heads'
|
||||
if (/^反面?/.test(word)) return 'tails'
|
||||
return null
|
||||
}
|
||||
|
||||
export interface NormalizedAnswer {
|
||||
normalized: string | null
|
||||
category: Exclude<SampleCategory, 'error'>
|
||||
}
|
||||
|
||||
/** Main entry point: raw completion text + the cell's answer domain → normalized answer + category. */
|
||||
export function normalizeAnswer(raw: string, domain: AnswerDomain): NormalizedAnswer {
|
||||
const nfc = (raw ?? '').normalize('NFC').trim()
|
||||
if (!nfc) return { normalized: null, category: 'empty' }
|
||||
|
||||
if (REFUSAL_PATTERNS.some((pattern) => pattern.test(nfc))) {
|
||||
return { normalized: null, category: 'refusal' }
|
||||
}
|
||||
|
||||
const cleaned = normalizeDigitScript(stripPunctuation(nfc)).toLowerCase().trim()
|
||||
if (!cleaned) return { normalized: null, category: 'empty' }
|
||||
|
||||
// First whitespace-separated word (Chinese text without spaces is one word).
|
||||
const firstWord = cleaned.split(/\s+/)[0]
|
||||
if (!firstWord) return { normalized: null, category: 'empty' }
|
||||
|
||||
switch (domain.kind) {
|
||||
case 'int': {
|
||||
const num = parseAnyNumber(firstWord)
|
||||
if (num === null) return { normalized: firstWord, category: 'invalid' }
|
||||
if (num < domain.min || num > domain.max) {
|
||||
return { normalized: String(num), category: 'invalid' }
|
||||
}
|
||||
return { normalized: String(num), category: 'valid' }
|
||||
}
|
||||
case 'letter': {
|
||||
const mapped = EN_LETTER_NAMES[firstWord] ?? firstWord
|
||||
if (/^[a-z]$/.test(mapped)) return { normalized: mapped, category: 'valid' }
|
||||
return { normalized: firstWord, category: 'invalid' }
|
||||
}
|
||||
case 'color': {
|
||||
const color = normalizeColorWord(firstWord)
|
||||
if (/^(?:[a-z]+|[\u4e00-\u9fff]{1,4})$/.test(color)) {
|
||||
return { normalized: color, category: 'valid' }
|
||||
}
|
||||
return { normalized: firstWord, category: 'invalid' }
|
||||
}
|
||||
case 'coin': {
|
||||
const coin = normalizeCoinWord(firstWord)
|
||||
if (coin) return { normalized: coin, category: 'valid' }
|
||||
return { normalized: firstWord, category: 'invalid' }
|
||||
}
|
||||
case 'word': {
|
||||
// Word tasks (animal/city): a single Latin word or a CJK word of ≤6 chars.
|
||||
if (/^(?:[a-z]+|[\u4e00-\u9fff]{1,6})$/.test(firstWord)) {
|
||||
return { normalized: firstWord, category: 'valid' }
|
||||
}
|
||||
return { normalized: firstWord, category: 'invalid' }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
/**
|
||||
* Bundled sample reference fingerprints (Node-only module — uses the
|
||||
* filesystem; import via `llm-fingerprint-detector/references`).
|
||||
*
|
||||
* The samples in `data/reference-fingerprints.sample.json` are derived from
|
||||
* the paper's public dataset (Bruckner, "Single-token output distributions as
|
||||
* behavioral fingerprints of large language models", Zenodo,
|
||||
* DOI 10.5281/zenodo.21278557, CC-BY-4.0). They were collected by the paper's
|
||||
* own harness — a slightly different prompt protocol than this package's
|
||||
* battery — so `compare()` flags them with `protocolMismatch: true`. They are
|
||||
* great for demos and exploration; for high-stakes verification, collect your
|
||||
* own reference from a trusted endpoint with this tool.
|
||||
*/
|
||||
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { dirname, join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import { getTaskSpec, isCellId } from './battery.js'
|
||||
import { FINGERPRINT_FORMAT_VERSION } from './constants.js'
|
||||
import { domainSize, shannonEntropyBits } from './stats.js'
|
||||
import type { CellDistribution, CellId, Fingerprint } from './types.js'
|
||||
|
||||
export interface SampleReferenceSource {
|
||||
dataset: string
|
||||
author: string
|
||||
datasetDoi: string
|
||||
paper: string
|
||||
license: string
|
||||
note?: string
|
||||
}
|
||||
|
||||
export interface SampleReferenceEntry {
|
||||
model: string
|
||||
collectedAt: string
|
||||
channel?: string
|
||||
cells: Record<string, { n: number; counts: Record<string, number> }>
|
||||
}
|
||||
|
||||
export interface SampleReferenceFile {
|
||||
formatVersion: number
|
||||
protocol: string
|
||||
samplesPerCell: number
|
||||
source: SampleReferenceSource
|
||||
models: Record<string, SampleReferenceEntry>
|
||||
}
|
||||
|
||||
const DATA_PATH = join(
|
||||
dirname(fileURLToPath(import.meta.url)),
|
||||
'..',
|
||||
'data',
|
||||
'reference-fingerprints.sample.json',
|
||||
)
|
||||
|
||||
let cached: SampleReferenceFile | null = null
|
||||
|
||||
function loadFile(): SampleReferenceFile {
|
||||
if (cached) return cached
|
||||
let text: string
|
||||
try {
|
||||
text = readFileSync(DATA_PATH, 'utf8')
|
||||
} catch {
|
||||
throw new Error(
|
||||
`Bundled reference data not found at ${DATA_PATH}. ` +
|
||||
'Reinstall the package, or build your own references (see README "Building your own reference fingerprints").',
|
||||
)
|
||||
}
|
||||
cached = JSON.parse(text) as SampleReferenceFile
|
||||
return cached
|
||||
}
|
||||
|
||||
export interface BundledReferenceInfo {
|
||||
id: string
|
||||
model: string
|
||||
collectedAt: string
|
||||
channel?: string
|
||||
cellCount: number
|
||||
}
|
||||
|
||||
export function listBundledReferences(): BundledReferenceInfo[] {
|
||||
const file = loadFile()
|
||||
return Object.entries(file.models).map(([id, entry]) => ({
|
||||
id,
|
||||
model: entry.model,
|
||||
collectedAt: entry.collectedAt,
|
||||
channel: entry.channel,
|
||||
cellCount: Object.keys(entry.cells).length,
|
||||
}))
|
||||
}
|
||||
|
||||
export function getBundledAttribution(): SampleReferenceSource {
|
||||
return loadFile().source
|
||||
}
|
||||
|
||||
function entryToFingerprint(id: string, entry: SampleReferenceEntry, file: SampleReferenceFile): Fingerprint {
|
||||
const cells: Partial<Record<CellId, CellDistribution>> = {}
|
||||
for (const [cellKey, cell] of Object.entries(entry.cells)) {
|
||||
if (!isCellId(cellKey)) continue
|
||||
const cellId = cellKey
|
||||
const entropyBits = shannonEntropyBits(cell.counts)
|
||||
const size = domainSize(getTaskSpec(cellId).domain)
|
||||
cells[cellId] = {
|
||||
cellId,
|
||||
counts: cell.counts,
|
||||
validCount: cell.n,
|
||||
invalidCount: 0,
|
||||
refusalCount: 0,
|
||||
emptyCount: 0,
|
||||
errorCount: 0,
|
||||
totalCount: cell.n,
|
||||
entropyBits,
|
||||
normalizedEntropy: size > 1 ? Math.min(1, entropyBits / Math.log2(size)) : 0,
|
||||
medianLatencyMs: null,
|
||||
meanCompletionTokens: null,
|
||||
meanReasoningTokens: null,
|
||||
}
|
||||
}
|
||||
return {
|
||||
formatVersion: FINGERPRINT_FORMAT_VERSION,
|
||||
protocol: file.protocol,
|
||||
model: entry.model,
|
||||
collectedAt: entry.collectedAt,
|
||||
samplesPerCell: file.samplesPerCell,
|
||||
postReasoning: false,
|
||||
cells,
|
||||
meta: {
|
||||
source: `${file.source.dataset} (DOI ${file.source.datasetDoi}, ${file.source.license})`,
|
||||
channel: entry.channel,
|
||||
note: `Bundled sample reference "${id}"`,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/** Load one bundled reference as a runtime Fingerprint. */
|
||||
export function loadBundledReference(id: string): Fingerprint {
|
||||
const file = loadFile()
|
||||
const entry = file.models[id]
|
||||
if (!entry) {
|
||||
const available = Object.keys(file.models).sort().join(', ')
|
||||
throw new Error(`Unknown bundled reference "${id}". Available: ${available}`)
|
||||
}
|
||||
return entryToFingerprint(id, entry, file)
|
||||
}
|
||||
|
||||
/** Parse and minimally validate a fingerprint JSON file produced by this tool. */
|
||||
export function parseFingerprintJson(text: string, sourceLabel = 'fingerprint file'): Fingerprint {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(text)
|
||||
} catch (error) {
|
||||
throw new Error(`${sourceLabel} is not valid JSON: ${(error as Error).message}`)
|
||||
}
|
||||
const fp = parsed as Partial<Fingerprint>
|
||||
if (fp.formatVersion !== FINGERPRINT_FORMAT_VERSION) {
|
||||
throw new Error(`${sourceLabel}: unsupported formatVersion ${String(fp.formatVersion)}`)
|
||||
}
|
||||
if (!fp.model || typeof fp.model !== 'string') {
|
||||
throw new Error(`${sourceLabel}: missing "model"`)
|
||||
}
|
||||
if (!fp.cells || typeof fp.cells !== 'object') {
|
||||
throw new Error(`${sourceLabel}: missing "cells"`)
|
||||
}
|
||||
return fp as Fingerprint
|
||||
}
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
/**
|
||||
* Sampling engine: a worker pool over a shuffled request queue with retries,
|
||||
* live progress callbacks and AbortSignal cancellation.
|
||||
*
|
||||
* - Requests are shuffled so a single cell is never hammered in a burst,
|
||||
* which would bias endpoint-side caching/rate-limiting systematically;
|
||||
* - a request that still fails after retries is recorded as an `error`
|
||||
* sample (excluded from distributions);
|
||||
* - ≥N consecutive transport errors abort the run (endpoint unreachable);
|
||||
* - 401/403 aborts immediately (bad key).
|
||||
*/
|
||||
|
||||
import { getSystemPrompt, getTaskSpec, pickParaphrase } from './battery.js'
|
||||
import { CONSECUTIVE_NETWORK_ERROR_LIMIT, PROBE_TEMPERATURE } from './constants.js'
|
||||
import { fetchChatCompletion, ProbeRequestError } from './http.js'
|
||||
import { normalizeAnswer } from './normalizer.js'
|
||||
import type {
|
||||
CellId,
|
||||
ProgressEvent,
|
||||
ReasoningAdapter,
|
||||
ResolvedEndpoint,
|
||||
SampleResult,
|
||||
} from './types.js'
|
||||
|
||||
export class ProbeRunError extends Error {
|
||||
readonly reason: 'network' | 'auth' | 'aborted'
|
||||
|
||||
constructor(reason: 'network' | 'auth' | 'aborted', message: string) {
|
||||
super(message)
|
||||
this.name = 'ProbeRunError'
|
||||
this.reason = reason
|
||||
}
|
||||
}
|
||||
|
||||
interface ProbeJob {
|
||||
cellId: CellId
|
||||
paraphrase: string
|
||||
}
|
||||
|
||||
export interface SamplerOptions {
|
||||
endpoint: ResolvedEndpoint
|
||||
adapter: ReasoningAdapter
|
||||
cells: CellId[]
|
||||
samplesPerCell: number
|
||||
concurrency: number
|
||||
timeoutMs?: number
|
||||
maxRetries?: number
|
||||
signal?: AbortSignal
|
||||
onSample?: (sample: SampleResult) => void
|
||||
onProgress?: (event: ProgressEvent) => void
|
||||
}
|
||||
|
||||
export interface SamplerResult {
|
||||
samples: SampleResult[]
|
||||
samplesByCell: Map<CellId, SampleResult[]>
|
||||
errorCount: number
|
||||
}
|
||||
|
||||
function shuffle<T>(items: T[]): T[] {
|
||||
const arr = [...items]
|
||||
for (let i = arr.length - 1; i > 0; i--) {
|
||||
const j = Math.floor(Math.random() * (i + 1))
|
||||
;[arr[i], arr[j]] = [arr[j], arr[i]]
|
||||
}
|
||||
return arr
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the full probe battery against one endpoint. Throws ProbeRunError when
|
||||
* the whole run must stop (unreachable endpoint / invalid key / cancelled).
|
||||
*/
|
||||
export async function runProbeBattery(options: SamplerOptions): Promise<SamplerResult> {
|
||||
const jobs: ProbeJob[] = []
|
||||
for (const cellId of options.cells) {
|
||||
for (let i = 0; i < options.samplesPerCell; i++) {
|
||||
jobs.push({ cellId, paraphrase: pickParaphrase(cellId) })
|
||||
}
|
||||
}
|
||||
const queue = shuffle(jobs)
|
||||
|
||||
const samples: SampleResult[] = []
|
||||
const samplesByCell = new Map<CellId, SampleResult[]>()
|
||||
for (const cellId of options.cells) samplesByCell.set(cellId, [])
|
||||
|
||||
let arrivalIndex = 0
|
||||
let errorCount = 0
|
||||
let consecutiveNetworkErrors = 0
|
||||
let fatalError: ProbeRunError | null = null
|
||||
let cursor = 0
|
||||
|
||||
const recordSample = (sample: SampleResult) => {
|
||||
samples.push(sample)
|
||||
samplesByCell.get(sample.cellId)?.push(sample)
|
||||
options.onSample?.(sample)
|
||||
options.onProgress?.({
|
||||
stage: 'sampling',
|
||||
done: samples.length,
|
||||
total: queue.length,
|
||||
errors: errorCount,
|
||||
cellId: sample.cellId,
|
||||
})
|
||||
}
|
||||
|
||||
async function worker(): Promise<void> {
|
||||
while (true) {
|
||||
if (fatalError || options.signal?.aborted) return
|
||||
const index = cursor
|
||||
cursor += 1
|
||||
if (index >= queue.length) return
|
||||
const job = queue[index]
|
||||
|
||||
const startedAt = performance.now()
|
||||
const systemPrompt = getSystemPrompt(job.cellId)
|
||||
try {
|
||||
const result = await fetchChatCompletion({
|
||||
endpoint: options.endpoint,
|
||||
systemPrompt,
|
||||
userPrompt: job.paraphrase,
|
||||
temperature: PROBE_TEMPERATURE,
|
||||
maxTokens: options.adapter.maxTokens,
|
||||
extraBody: options.adapter.extraBody,
|
||||
signal: options.signal,
|
||||
timeoutMs: options.timeoutMs,
|
||||
retries: options.maxRetries,
|
||||
})
|
||||
consecutiveNetworkErrors = 0
|
||||
const domain = getTaskSpec(job.cellId).domain
|
||||
const { normalized, category } = normalizeAnswer(result.content, domain)
|
||||
recordSample({
|
||||
cellId: job.cellId,
|
||||
raw: result.content,
|
||||
normalized,
|
||||
category,
|
||||
latencyMs: result.latencyMs,
|
||||
usage: result.usage,
|
||||
arrivalIndex: arrivalIndex++,
|
||||
})
|
||||
} catch (error) {
|
||||
if (options.signal?.aborted) {
|
||||
fatalError = fatalError ?? new ProbeRunError('aborted', 'Run cancelled')
|
||||
return
|
||||
}
|
||||
if (error instanceof ProbeRequestError) {
|
||||
if (error.kind === 'aborted') {
|
||||
fatalError = fatalError ?? new ProbeRunError('aborted', 'Run cancelled')
|
||||
return
|
||||
}
|
||||
if (error.kind === 'auth') {
|
||||
fatalError = new ProbeRunError('auth', error.message)
|
||||
return
|
||||
}
|
||||
if (error.kind === 'network') {
|
||||
consecutiveNetworkErrors += 1
|
||||
if (consecutiveNetworkErrors >= CONSECUTIVE_NETWORK_ERROR_LIMIT) {
|
||||
fatalError = new ProbeRunError(
|
||||
'network',
|
||||
`Endpoint unreachable (${consecutiveNetworkErrors} consecutive transport errors): ${error.message}`,
|
||||
)
|
||||
return
|
||||
}
|
||||
} else {
|
||||
consecutiveNetworkErrors = 0
|
||||
}
|
||||
errorCount += 1
|
||||
recordSample({
|
||||
cellId: job.cellId,
|
||||
raw: '',
|
||||
normalized: null,
|
||||
category: 'error',
|
||||
latencyMs: performance.now() - startedAt,
|
||||
usage: null,
|
||||
arrivalIndex: arrivalIndex++,
|
||||
errorMessage: error.message,
|
||||
})
|
||||
} else {
|
||||
errorCount += 1
|
||||
recordSample({
|
||||
cellId: job.cellId,
|
||||
raw: '',
|
||||
normalized: null,
|
||||
category: 'error',
|
||||
latencyMs: performance.now() - startedAt,
|
||||
usage: null,
|
||||
arrivalIndex: arrivalIndex++,
|
||||
errorMessage: error instanceof Error ? error.message : String(error),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const workerCount = Math.max(1, Math.min(options.concurrency, queue.length))
|
||||
await Promise.all(Array.from({ length: workerCount }, () => worker()))
|
||||
|
||||
if (fatalError) throw fatalError
|
||||
if (options.signal?.aborted) throw new ProbeRunError('aborted', 'Run cancelled')
|
||||
|
||||
return { samples, samplesByCell, errorCount }
|
||||
}
|
||||
+219
@@ -0,0 +1,219 @@
|
||||
/**
|
||||
* Statistics: Shannon entropy, Jensen-Shannon divergence (base 2, not
|
||||
* square-rooted, so per-cell values live in [0, 1] bit), distribution
|
||||
* aggregation and the split-half self check.
|
||||
*
|
||||
* Distance between two fingerprints (arXiv:2607.10252): the mean of per-cell
|
||||
* JSD over all cells where both sides have enough valid samples.
|
||||
*/
|
||||
|
||||
import { MIN_SPLIT_HALF_SAMPLES, MIN_VALID_SAMPLES_PER_CELL } from './constants.js'
|
||||
import type { AnswerDomain, CellDistribution, CellId, SampleResult } from './types.js'
|
||||
|
||||
export type CountMap = Record<string, number>
|
||||
|
||||
/** Shannon entropy in bits. */
|
||||
export function shannonEntropyBits(counts: CountMap): number {
|
||||
const total = Object.values(counts).reduce((sum, n) => sum + n, 0)
|
||||
if (total <= 0) return 0
|
||||
let entropy = 0
|
||||
for (const n of Object.values(counts)) {
|
||||
if (n <= 0) continue
|
||||
const p = n / total
|
||||
entropy -= p * Math.log2(p)
|
||||
}
|
||||
return entropy
|
||||
}
|
||||
|
||||
/** Nominal domain size (denominator log2(size) for normalized entropy). */
|
||||
export function domainSize(domain: AnswerDomain): number {
|
||||
switch (domain.kind) {
|
||||
case 'int':
|
||||
return Math.max(2, domain.max - domain.min + 1)
|
||||
case 'letter':
|
||||
return 26
|
||||
case 'coin':
|
||||
return 2
|
||||
case 'color':
|
||||
return 30
|
||||
case 'word':
|
||||
return 50
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Jensen-Shannon divergence, base 2: JSD(P,Q) = H(M) − (H(P)+H(Q))/2 with
|
||||
* M = (P+Q)/2, over the union of both supports. Range [0, 1] bit.
|
||||
*/
|
||||
export function jensenShannonDivergence(countsP: CountMap, countsQ: CountMap): number {
|
||||
const totalP = Object.values(countsP).reduce((sum, n) => sum + n, 0)
|
||||
const totalQ = Object.values(countsQ).reduce((sum, n) => sum + n, 0)
|
||||
if (totalP <= 0 || totalQ <= 0) return 0
|
||||
|
||||
const support = new Set([...Object.keys(countsP), ...Object.keys(countsQ)])
|
||||
let hM = 0
|
||||
let hP = 0
|
||||
let hQ = 0
|
||||
for (const key of support) {
|
||||
const p = (countsP[key] ?? 0) / totalP
|
||||
const q = (countsQ[key] ?? 0) / totalQ
|
||||
const m = (p + q) / 2
|
||||
if (m > 0) hM -= m * Math.log2(m)
|
||||
if (p > 0) hP -= p * Math.log2(p)
|
||||
if (q > 0) hQ -= q * Math.log2(q)
|
||||
}
|
||||
const jsd = hM - (hP + hQ) / 2
|
||||
// Clamp numerical noise.
|
||||
return Math.min(1, Math.max(0, jsd))
|
||||
}
|
||||
|
||||
export function median(values: number[]): number | null {
|
||||
if (values.length === 0) return null
|
||||
const sorted = [...values].sort((a, b) => a - b)
|
||||
const mid = Math.floor(sorted.length / 2)
|
||||
return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid]
|
||||
}
|
||||
|
||||
/** Aggregate raw samples into one cell's distribution. */
|
||||
export function buildCellDistribution(
|
||||
cellId: CellId,
|
||||
samples: SampleResult[],
|
||||
domain: AnswerDomain,
|
||||
): CellDistribution {
|
||||
const counts: CountMap = {}
|
||||
let validCount = 0
|
||||
let invalidCount = 0
|
||||
let refusalCount = 0
|
||||
let emptyCount = 0
|
||||
let errorCount = 0
|
||||
const latenciesMs: number[] = []
|
||||
let completionTokensSum = 0
|
||||
let completionTokensN = 0
|
||||
let reasoningTokensSum = 0
|
||||
let reasoningTokensN = 0
|
||||
|
||||
for (const sample of samples) {
|
||||
switch (sample.category) {
|
||||
case 'valid':
|
||||
validCount += 1
|
||||
if (sample.normalized !== null) {
|
||||
counts[sample.normalized] = (counts[sample.normalized] ?? 0) + 1
|
||||
}
|
||||
break
|
||||
case 'invalid':
|
||||
invalidCount += 1
|
||||
break
|
||||
case 'refusal':
|
||||
refusalCount += 1
|
||||
break
|
||||
case 'empty':
|
||||
emptyCount += 1
|
||||
break
|
||||
case 'error':
|
||||
errorCount += 1
|
||||
break
|
||||
}
|
||||
if (sample.category !== 'error') {
|
||||
latenciesMs.push(sample.latencyMs)
|
||||
if (sample.usage?.completionTokens != null) {
|
||||
completionTokensSum += sample.usage.completionTokens
|
||||
completionTokensN += 1
|
||||
}
|
||||
if (sample.usage?.reasoningTokens != null) {
|
||||
reasoningTokensSum += sample.usage.reasoningTokens
|
||||
reasoningTokensN += 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const entropyBits = shannonEntropyBits(counts)
|
||||
const size = domainSize(domain)
|
||||
|
||||
return {
|
||||
cellId,
|
||||
counts,
|
||||
validCount,
|
||||
invalidCount,
|
||||
refusalCount,
|
||||
emptyCount,
|
||||
errorCount,
|
||||
totalCount: samples.length,
|
||||
entropyBits,
|
||||
normalizedEntropy: size > 1 ? Math.min(1, entropyBits / Math.log2(size)) : 0,
|
||||
medianLatencyMs: median(latenciesMs),
|
||||
meanCompletionTokens: completionTokensN > 0 ? completionTokensSum / completionTokensN : null,
|
||||
meanReasoningTokens: reasoningTokensN > 0 ? reasoningTokensSum / reasoningTokensN : null,
|
||||
}
|
||||
}
|
||||
|
||||
export interface CellJsdEntry {
|
||||
cellId: CellId
|
||||
jsd: number
|
||||
validA: number
|
||||
validB: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Distance between two fingerprint cell sets: mean JSD over cells where both
|
||||
* sides have ≥ minValidSamples valid samples. Returns per-cell details sorted
|
||||
* by descending JSD.
|
||||
*/
|
||||
export function compareCellSets(
|
||||
cellsA: Partial<Record<CellId, { counts: CountMap; validCount: number }>>,
|
||||
cellsB: Partial<Record<CellId, { counts: CountMap; validCount: number }>>,
|
||||
minValidSamples: number = MIN_VALID_SAMPLES_PER_CELL,
|
||||
): { entries: CellJsdEntry[]; meanJsd: number | null } {
|
||||
const entries: CellJsdEntry[] = []
|
||||
for (const cellId of Object.keys(cellsA) as CellId[]) {
|
||||
const a = cellsA[cellId]
|
||||
const b = cellsB[cellId]
|
||||
if (!a || !b) continue
|
||||
if (a.validCount < minValidSamples || b.validCount < minValidSamples) continue
|
||||
entries.push({
|
||||
cellId,
|
||||
jsd: jensenShannonDivergence(a.counts, b.counts),
|
||||
validA: a.validCount,
|
||||
validB: b.validCount,
|
||||
})
|
||||
}
|
||||
entries.sort((x, y) => y.jsd - x.jsd)
|
||||
const meanJsd =
|
||||
entries.length > 0
|
||||
? entries.reduce((sum, entry) => sum + entry.jsd, 0) / entries.length
|
||||
: null
|
||||
return { entries, meanJsd }
|
||||
}
|
||||
|
||||
/**
|
||||
* Split-half self check: split each cell's valid samples by arrival parity
|
||||
* and measure the JSD between the halves. Values far above the same-model
|
||||
* baseline (≈ 0.14) indicate the endpoint itself is unstable — a hint that an
|
||||
* aggregator is rotating between different backends.
|
||||
*/
|
||||
export function splitHalfJsd(
|
||||
samplesByCell: Map<CellId, SampleResult[]>,
|
||||
minPerHalf: number = MIN_SPLIT_HALF_SAMPLES,
|
||||
): number | null {
|
||||
const cellJsds: number[] = []
|
||||
for (const samples of samplesByCell.values()) {
|
||||
const even: CountMap = {}
|
||||
const odd: CountMap = {}
|
||||
let evenN = 0
|
||||
let oddN = 0
|
||||
for (const sample of samples) {
|
||||
if (sample.category !== 'valid' || sample.normalized === null) continue
|
||||
if (sample.arrivalIndex % 2 === 0) {
|
||||
even[sample.normalized] = (even[sample.normalized] ?? 0) + 1
|
||||
evenN += 1
|
||||
} else {
|
||||
odd[sample.normalized] = (odd[sample.normalized] ?? 0) + 1
|
||||
oddN += 1
|
||||
}
|
||||
}
|
||||
if (evenN >= minPerHalf && oddN >= minPerHalf) {
|
||||
cellJsds.push(jensenShannonDivergence(even, odd))
|
||||
}
|
||||
}
|
||||
if (cellJsds.length === 0) return null
|
||||
return cellJsds.reduce((sum, jsd) => sum + jsd, 0) / cellJsds.length
|
||||
}
|
||||
+260
@@ -0,0 +1,260 @@
|
||||
/**
|
||||
* Core type definitions.
|
||||
*
|
||||
* Method: Tomáš Bruckner, "One Token Is Enough: Fingerprinting and Verifying
|
||||
* Large Language Models from Single-Token Output Distributions"
|
||||
* (arXiv:2607.10252). A probe battery of task × language "cells" is sampled
|
||||
* repeatedly at temperature 1.0 with a one-word answer constraint; the
|
||||
* empirical distribution of normalized answers is the model's behavioral
|
||||
* fingerprint. Two fingerprints are compared with the mean per-cell
|
||||
* Jensen-Shannon divergence (base 2, so each cell's JSD lies in [0, 1] bit).
|
||||
*/
|
||||
|
||||
export type ProbeTaskId =
|
||||
| 'random-number-1-100'
|
||||
| 'random-number-1-10'
|
||||
| 'random-letter'
|
||||
| 'random-color'
|
||||
| 'coin-flip'
|
||||
| 'random-animal'
|
||||
| 'random-city'
|
||||
| 'favorite-number'
|
||||
|
||||
export type ProbeLang = 'en' | 'zh'
|
||||
|
||||
/**
|
||||
* A cell is one task in one language. Distributions are only ever compared
|
||||
* within the same cell; there is no cross-language pooling.
|
||||
*/
|
||||
export type CellId = `${ProbeTaskId}:${ProbeLang}`
|
||||
|
||||
export type AnswerDomain =
|
||||
| { kind: 'int'; min: number; max: number }
|
||||
| { kind: 'letter' }
|
||||
| { kind: 'color' }
|
||||
| { kind: 'coin' }
|
||||
| { kind: 'word' }
|
||||
|
||||
export interface ProbeTaskSpec {
|
||||
id: ProbeTaskId
|
||||
domain: AnswerDomain
|
||||
/**
|
||||
* At least 3 paraphrases per language. One is drawn at random per request,
|
||||
* so the probes are plain semantic questions with no fixed magic string a
|
||||
* gateway could keyword-filter.
|
||||
*/
|
||||
paraphrases: Record<ProbeLang, string[]>
|
||||
}
|
||||
|
||||
export type ProbePresetId = 'quick' | 'standard' | 'strict'
|
||||
|
||||
export interface ProbePreset {
|
||||
id: ProbePresetId
|
||||
cellCount: number
|
||||
samplesPerCell: number
|
||||
}
|
||||
|
||||
/** An OpenAI-compatible chat-completions endpoint to probe. */
|
||||
export interface Endpoint {
|
||||
/** Base URL, e.g. `https://api.openai.com/v1`. A bare domain gets `/v1` appended. */
|
||||
baseUrl: string
|
||||
/** Model id to request, e.g. `gpt-4o-mini`. */
|
||||
model: string
|
||||
/** API key sent as `Authorization: Bearer <key>`. Omit for keyless local servers. */
|
||||
apiKey?: string
|
||||
/** Extra HTTP headers merged into every request. */
|
||||
headers?: Record<string, string>
|
||||
}
|
||||
|
||||
/** Internal, normalized endpoint (base URL cleaned up, key resolved). */
|
||||
export interface ResolvedEndpoint {
|
||||
baseUrl: string
|
||||
model: string
|
||||
apiKey: string | null
|
||||
headers: Record<string, string>
|
||||
}
|
||||
|
||||
/** Strategy used to disable hidden reasoning ("thinking") on the endpoint. */
|
||||
export type ReasoningStrategyId =
|
||||
| 'openrouter-reasoning'
|
||||
| 'zhipu-thinking'
|
||||
| 'openai-effort'
|
||||
| 'none'
|
||||
|
||||
export interface ReasoningAdapter {
|
||||
strategy: ReasoningStrategyId
|
||||
/** Extra fields merged into the request body. */
|
||||
extraBody: Record<string, unknown>
|
||||
/** max_tokens used for probe requests. */
|
||||
maxTokens: number
|
||||
/**
|
||||
* True when no disabling strategy produced visible output and the run fell
|
||||
* back to a large max_tokens "post-reasoning" channel. Fingerprints
|
||||
* collected this way are lower confidence (reasoning shifts sampling).
|
||||
*/
|
||||
postReasoning: boolean
|
||||
}
|
||||
|
||||
export type SampleCategory = 'valid' | 'invalid' | 'refusal' | 'empty' | 'error'
|
||||
|
||||
export interface SampleUsage {
|
||||
promptTokens: number | null
|
||||
completionTokens: number | null
|
||||
reasoningTokens: number | null
|
||||
}
|
||||
|
||||
export interface SampleResult {
|
||||
cellId: CellId
|
||||
/** Verbatim completion text. */
|
||||
raw: string
|
||||
/** Normalized answer; non-null only when `category === 'valid'`. */
|
||||
normalized: string | null
|
||||
category: SampleCategory
|
||||
latencyMs: number
|
||||
usage: SampleUsage | null
|
||||
/** Arrival order across the whole run (used for the split-half self check). */
|
||||
arrivalIndex: number
|
||||
errorMessage?: string
|
||||
}
|
||||
|
||||
/** Aggregated answer distribution for one cell. */
|
||||
export interface CellDistribution {
|
||||
cellId: CellId
|
||||
/** Normalized answer → count (valid samples only). */
|
||||
counts: Record<string, number>
|
||||
validCount: number
|
||||
invalidCount: number
|
||||
refusalCount: number
|
||||
emptyCount: number
|
||||
errorCount: number
|
||||
totalCount: number
|
||||
/** Shannon entropy of the valid-answer distribution, in bits. */
|
||||
entropyBits: number
|
||||
/** entropyBits / log2(nominal domain size), clamped to [0, 1]. */
|
||||
normalizedEntropy: number
|
||||
medianLatencyMs: number | null
|
||||
meanCompletionTokens: number | null
|
||||
meanReasoningTokens: number | null
|
||||
}
|
||||
|
||||
/**
|
||||
* A behavioral fingerprint: per-cell answer distributions plus collection
|
||||
* metadata. This is the JSON artifact written/read by the CLI.
|
||||
*/
|
||||
export interface Fingerprint {
|
||||
formatVersion: 1
|
||||
/**
|
||||
* Probe protocol identifier. Fingerprints are only strictly comparable when
|
||||
* both sides used the same protocol (same battery, same system prompt).
|
||||
* This package emits `one-token/v1`; bundled Zenodo-derived samples use
|
||||
* `bruckner-zenodo-2026`.
|
||||
*/
|
||||
protocol: string
|
||||
model: string
|
||||
/** ISO timestamp. Fingerprints drift when models are updated, so age matters. */
|
||||
collectedAt: string
|
||||
samplesPerCell: number
|
||||
postReasoning: boolean
|
||||
cells: Partial<Record<CellId, CellDistribution>>
|
||||
meta?: {
|
||||
tool?: string
|
||||
channel?: string
|
||||
source?: string
|
||||
note?: string
|
||||
[key: string]: unknown
|
||||
}
|
||||
}
|
||||
|
||||
export type VerdictLevel = 'match' | 'uncertain' | 'mismatch' | 'insufficient'
|
||||
|
||||
export interface CellComparison {
|
||||
cellId: CellId
|
||||
/** Jensen-Shannon divergence, base 2, in [0, 1] bit. */
|
||||
jsd: number
|
||||
validA: number
|
||||
validB: number
|
||||
}
|
||||
|
||||
export interface ComparisonBaselines {
|
||||
/** Median split-half distance of a model against itself (paper): ≈ 0.140. */
|
||||
sameModelSelf: number
|
||||
/** Median distance, same model served by different providers (paper): ≈ 0.227. */
|
||||
sameModelCrossProvider: number
|
||||
/** Median distance between different models (paper): ≈ 0.463. */
|
||||
differentModel: number
|
||||
}
|
||||
|
||||
export interface ComparisonResult {
|
||||
/** Mean per-cell JSD across comparable cells; null when none are comparable. */
|
||||
meanJsd: number | null
|
||||
verdict: VerdictLevel
|
||||
/** Per-cell details, sorted by descending JSD. */
|
||||
cells: CellComparison[]
|
||||
comparableCellCount: number
|
||||
/** True when the two fingerprints were collected under different probe protocols. */
|
||||
protocolMismatch: boolean
|
||||
thresholds: { match: number; mismatch: number }
|
||||
baselines: ComparisonBaselines
|
||||
}
|
||||
|
||||
export interface ProgressEvent {
|
||||
stage: 'adapter' | 'sampling'
|
||||
/** Completed requests (sampling stage) or probes attempted (adapter stage). */
|
||||
done: number
|
||||
total: number
|
||||
errors: number
|
||||
cellId?: CellId
|
||||
strategy?: ReasoningStrategyId
|
||||
}
|
||||
|
||||
export interface FingerprintOptions {
|
||||
/**
|
||||
* Cells to probe: an explicit list, or a number N meaning the top-N cells
|
||||
* from the discriminativeness-ordered battery. Default: 8 (standard preset).
|
||||
*/
|
||||
cells?: CellId[] | number
|
||||
/** Samples per cell. Default: 25. */
|
||||
samplesPerCell?: number
|
||||
/** Concurrent in-flight requests. Default: 4. */
|
||||
concurrency?: number
|
||||
/** Per-request timeout in milliseconds. Default: 30000. */
|
||||
timeoutMs?: number
|
||||
/** Retries per request on 429/5xx/timeout. Default: 2. */
|
||||
maxRetries?: number
|
||||
/** Abort the whole run (in-flight requests are cancelled). */
|
||||
signal?: AbortSignal
|
||||
onProgress?: (event: ProgressEvent) => void
|
||||
/** Skip reasoning-adapter detection and use this adapter directly. */
|
||||
adapter?: ReasoningAdapter
|
||||
/** Keep raw per-sample results on the run result (off by default). */
|
||||
keepSamples?: boolean
|
||||
/** Free-form metadata merged into `fingerprint.meta`. */
|
||||
meta?: Fingerprint['meta']
|
||||
}
|
||||
|
||||
export interface FingerprintRun {
|
||||
fingerprint: Fingerprint
|
||||
adapter: ReasoningAdapter
|
||||
/** Requests that errored out after retries (excluded from distributions). */
|
||||
errorCount: number
|
||||
/**
|
||||
* Mean JSD between odd/even arrival halves of the run itself.
|
||||
* Values far above the same-model baseline (≈ 0.14) suggest the endpoint
|
||||
* routes across multiple backends (aggregator behavior).
|
||||
*/
|
||||
splitHalfJsd: number | null
|
||||
durationMs: number
|
||||
/** Present only when `keepSamples: true`. */
|
||||
samples?: SampleResult[]
|
||||
/** Human-readable caveats collected during the run. */
|
||||
warnings: string[]
|
||||
}
|
||||
|
||||
export interface VerifyResult {
|
||||
verdict: VerdictLevel
|
||||
meanJsd: number | null
|
||||
comparison: ComparisonResult
|
||||
target: FingerprintRun
|
||||
reference: Fingerprint
|
||||
warnings: string[]
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* Verdict: meanJsd → three-way conclusion (match / uncertain / mismatch),
|
||||
* or `insufficient` when too few cells are comparable.
|
||||
*
|
||||
* Threshold provenance (see constants.ts): the paper reports a same-model
|
||||
* cross-provider median distance of ≈ 0.227 and a different-model median of
|
||||
* ≈ 0.463; the cut points 0.25 / 0.35 sit between those with a deliberate
|
||||
* uncertainty band. Verdicts are statistical evidence, not proof.
|
||||
*/
|
||||
|
||||
import {
|
||||
JSD_BASELINE_CROSS_PROVIDER,
|
||||
JSD_BASELINE_DIFFERENT_MODEL,
|
||||
JSD_BASELINE_SELF,
|
||||
JSD_MATCH_THRESHOLD,
|
||||
JSD_MISMATCH_THRESHOLD,
|
||||
MIN_COMPARABLE_CELLS,
|
||||
} from './constants.js'
|
||||
import type { CellJsdEntry } from './stats.js'
|
||||
import type { CellComparison, ComparisonResult, VerdictLevel } from './types.js'
|
||||
|
||||
export function decideVerdict(meanJsd: number | null, comparableCellCount: number): VerdictLevel {
|
||||
if (meanJsd === null || comparableCellCount < MIN_COMPARABLE_CELLS) return 'insufficient'
|
||||
if (meanJsd <= JSD_MATCH_THRESHOLD) return 'match'
|
||||
if (meanJsd <= JSD_MISMATCH_THRESHOLD) return 'uncertain'
|
||||
return 'mismatch'
|
||||
}
|
||||
|
||||
export function buildComparisonResult(
|
||||
entries: CellJsdEntry[],
|
||||
meanJsd: number | null,
|
||||
protocolMismatch: boolean,
|
||||
): ComparisonResult {
|
||||
const cells: CellComparison[] = entries.map((entry) => ({
|
||||
cellId: entry.cellId,
|
||||
jsd: entry.jsd,
|
||||
validA: entry.validA,
|
||||
validB: entry.validB,
|
||||
}))
|
||||
return {
|
||||
meanJsd,
|
||||
verdict: decideVerdict(meanJsd, entries.length),
|
||||
cells,
|
||||
comparableCellCount: entries.length,
|
||||
protocolMismatch,
|
||||
thresholds: { match: JSD_MATCH_THRESHOLD, mismatch: JSD_MISMATCH_THRESHOLD },
|
||||
baselines: {
|
||||
sameModelSelf: JSD_BASELINE_SELF,
|
||||
sameModelCrossProvider: JSD_BASELINE_CROSS_PROVIDER,
|
||||
differentModel: JSD_BASELINE_DIFFERENT_MODEL,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,199 @@
|
||||
/**
|
||||
* End-to-end test against a local mock OpenAI-compatible server: adapter
|
||||
* detection, concurrent sampling, normalization, aggregation and verify().
|
||||
*/
|
||||
import assert from 'node:assert/strict'
|
||||
import { createServer } from 'node:http'
|
||||
import { after, before, test } from 'node:test'
|
||||
|
||||
import { compare, fingerprint, verify } from '../dist/api.js'
|
||||
|
||||
/** Two simulated "models" with different answer distributions. */
|
||||
const MODEL_BEHAVIOR = {
|
||||
'mock-alpha': {
|
||||
'random-number-1-100': () => (Math.random() < 0.8 ? '42' : '73'),
|
||||
'random-number-1-10': () => '7',
|
||||
'random-letter': () => 'k',
|
||||
'random-color': () => (Math.random() < 0.6 ? 'blue' : 'teal'),
|
||||
'coin-flip': () => (Math.random() < 0.7 ? 'Heads' : 'Tails'),
|
||||
'random-animal': () => 'octopus',
|
||||
'random-city': () => 'Tokyo',
|
||||
'favorite-number': () => '42',
|
||||
},
|
||||
'mock-beta': {
|
||||
'random-number-1-100': () => (Math.random() < 0.8 ? '57' : '3'),
|
||||
'random-number-1-10': () => '3',
|
||||
'random-letter': () => 'm',
|
||||
'random-color': () => 'crimson',
|
||||
'coin-flip': () => 'Tails',
|
||||
'random-animal': () => 'giraffe',
|
||||
'random-city': () => 'Paris',
|
||||
'favorite-number': () => '9',
|
||||
},
|
||||
}
|
||||
|
||||
/** Map a user prompt back to the battery task (rough but sufficient for the mock). */
|
||||
function taskFromPrompt(prompt) {
|
||||
const p = prompt.toLowerCase()
|
||||
if (p.includes('100') || p.includes('1 到 100') || p.includes('1 至 100') || p.includes('1 到 100'))
|
||||
return 'random-number-1-100'
|
||||
if (p.includes('10') || p.includes('1 到 10')) return 'random-number-1-10'
|
||||
if (p.includes('letter') || p.includes('字母')) return 'random-letter'
|
||||
if (p.includes('color') || p.includes('颜色')) return 'random-color'
|
||||
if (p.includes('coin') || p.includes('硬币')) return 'coin-flip'
|
||||
if (p.includes('animal') || p.includes('动物')) return 'random-animal'
|
||||
if (p.includes('city') || p.includes('城市')) return 'random-city'
|
||||
if (p.includes('favorite') || p.includes('favourite') || p.includes('喜欢') || p.includes('最爱'))
|
||||
return 'favorite-number'
|
||||
return 'random-number-1-100'
|
||||
}
|
||||
|
||||
let server
|
||||
let baseUrl
|
||||
let requestCount = 0
|
||||
let sawReasoningField = 0
|
||||
|
||||
before(async () => {
|
||||
server = createServer((req, res) => {
|
||||
let body = ''
|
||||
req.on('data', (chunk) => (body += chunk))
|
||||
req.on('end', () => {
|
||||
requestCount += 1
|
||||
const payload = JSON.parse(body)
|
||||
|
||||
if (req.url !== '/v1/chat/completions') {
|
||||
res.writeHead(404).end('not found')
|
||||
return
|
||||
}
|
||||
if (req.headers.authorization !== 'Bearer test-key') {
|
||||
res.writeHead(401, { 'content-type': 'application/json' })
|
||||
res.end(JSON.stringify({ error: { message: 'bad key' } }))
|
||||
return
|
||||
}
|
||||
// The mock accepts the OpenRouter-style reasoning field (counts it for assertions).
|
||||
if (payload.reasoning !== undefined) sawReasoningField += 1
|
||||
|
||||
const behavior = MODEL_BEHAVIOR[payload.model]
|
||||
if (!behavior) {
|
||||
res.writeHead(404, { 'content-type': 'application/json' })
|
||||
res.end(JSON.stringify({ error: { message: `unknown model ${payload.model}` } }))
|
||||
return
|
||||
}
|
||||
const userPrompt = payload.messages.find((m) => m.role === 'user')?.content ?? ''
|
||||
const answer = behavior[taskFromPrompt(userPrompt)]()
|
||||
res.writeHead(200, { 'content-type': 'application/json' })
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
choices: [{ message: { role: 'assistant', content: answer } }],
|
||||
usage: { prompt_tokens: 25, completion_tokens: 2 },
|
||||
}),
|
||||
)
|
||||
})
|
||||
})
|
||||
await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve))
|
||||
baseUrl = `http://127.0.0.1:${server.address().port}/v1`
|
||||
})
|
||||
|
||||
after(() => server.close())
|
||||
|
||||
test('fingerprint(): collects distributions from a live endpoint', async () => {
|
||||
const run = await fingerprint(
|
||||
{ baseUrl, model: 'mock-alpha', apiKey: 'test-key' },
|
||||
{ cells: 4, samplesPerCell: 12, concurrency: 6 },
|
||||
)
|
||||
assert.equal(run.errorCount, 0)
|
||||
assert.equal(run.fingerprint.model, 'mock-alpha')
|
||||
assert.equal(run.fingerprint.protocol, 'one-token/v1')
|
||||
assert.equal(Object.keys(run.fingerprint.cells).length, 4)
|
||||
|
||||
const rn100 = run.fingerprint.cells['random-number-1-100:en']
|
||||
assert.ok(rn100, 'expected the top-priority cell to be probed')
|
||||
assert.equal(rn100.validCount, 12)
|
||||
assert.ok(rn100.counts['42'] > 0, 'mock-alpha answers 42 most of the time')
|
||||
assert.ok(run.adapter.postReasoning === false)
|
||||
assert.ok(sawReasoningField > 0, 'adapter probe should have tried the reasoning field')
|
||||
})
|
||||
|
||||
test('fingerprint(): progress callback covers all requests', async () => {
|
||||
const events = []
|
||||
const run = await fingerprint(
|
||||
{ baseUrl, model: 'mock-alpha', apiKey: 'test-key' },
|
||||
{ cells: 2, samplesPerCell: 5, onProgress: (e) => events.push(e) },
|
||||
)
|
||||
const sampling = events.filter((e) => e.stage === 'sampling')
|
||||
assert.equal(sampling.length, 10)
|
||||
assert.equal(sampling.at(-1).done, 10)
|
||||
assert.equal(sampling.at(-1).total, 10)
|
||||
assert.equal(run.durationMs >= 0, true)
|
||||
})
|
||||
|
||||
test('verify(): same mock model → match, different mock model → mismatch', async () => {
|
||||
const options = { cells: 6, samplesPerCell: 20, concurrency: 8 }
|
||||
const referenceRun = await fingerprint({ baseUrl, model: 'mock-alpha', apiKey: 'test-key' }, options)
|
||||
|
||||
const same = await verify(
|
||||
{ baseUrl, model: 'mock-alpha', apiKey: 'test-key' },
|
||||
referenceRun.fingerprint,
|
||||
options,
|
||||
)
|
||||
assert.equal(same.verdict, 'match')
|
||||
assert.ok(same.meanJsd < 0.25, `same model meanJsd should be small, got ${same.meanJsd}`)
|
||||
|
||||
const different = await verify(
|
||||
{ baseUrl, model: 'mock-beta', apiKey: 'test-key' },
|
||||
referenceRun.fingerprint,
|
||||
options,
|
||||
)
|
||||
assert.equal(different.verdict, 'mismatch')
|
||||
assert.ok(different.meanJsd > 0.35, `different model meanJsd should be large, got ${different.meanJsd}`)
|
||||
})
|
||||
|
||||
test('compare(): flags protocol mismatch', async () => {
|
||||
const runA = await fingerprint(
|
||||
{ baseUrl, model: 'mock-alpha', apiKey: 'test-key' },
|
||||
{ cells: 4, samplesPerCell: 12 },
|
||||
)
|
||||
const foreign = { ...runA.fingerprint, protocol: 'someone-elses-protocol' }
|
||||
const result = compare(runA.fingerprint, foreign)
|
||||
assert.equal(result.protocolMismatch, true)
|
||||
assert.equal(result.verdict, 'match') // identical distributions still match
|
||||
})
|
||||
|
||||
test('fingerprint(): invalid API key aborts the run with an auth error', async () => {
|
||||
await assert.rejects(
|
||||
fingerprint(
|
||||
{ baseUrl, model: 'mock-alpha', apiKey: 'wrong-key' },
|
||||
{ cells: 2, samplesPerCell: 3 },
|
||||
),
|
||||
(error) => {
|
||||
assert.match(error.message, /401/)
|
||||
return true
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
test('fingerprint(): AbortSignal cancels the run', async () => {
|
||||
const controller = new AbortController()
|
||||
const promise = fingerprint(
|
||||
{ baseUrl, model: 'mock-alpha', apiKey: 'test-key' },
|
||||
{
|
||||
cells: 8,
|
||||
samplesPerCell: 50,
|
||||
onProgress: (e) => {
|
||||
if (e.stage === 'sampling' && e.done >= 5) controller.abort()
|
||||
},
|
||||
signal: controller.signal,
|
||||
},
|
||||
)
|
||||
await assert.rejects(promise, (error) => {
|
||||
assert.equal(error.name, 'ProbeRunError')
|
||||
assert.equal(error.reason, 'aborted')
|
||||
return true
|
||||
})
|
||||
})
|
||||
|
||||
test('endpoint validation: bad base URL rejects before any request', async () => {
|
||||
await assert.rejects(fingerprint({ baseUrl: '', model: 'x' }), /baseUrl is empty/)
|
||||
await assert.rejects(fingerprint({ baseUrl: 'https://ok.example/v1', model: ' ' }), /model is empty/)
|
||||
assert.ok(requestCount > 0)
|
||||
})
|
||||
@@ -0,0 +1,64 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
|
||||
import {
|
||||
CELL_PRIORITY_ORDER,
|
||||
PROBE_TASKS,
|
||||
SYSTEM_PROMPTS,
|
||||
getCellsForPreset,
|
||||
getSystemPrompt,
|
||||
getTaskSpec,
|
||||
isCellId,
|
||||
makeCellId,
|
||||
parseCellId,
|
||||
pickParaphrase,
|
||||
} from '../dist/battery.js'
|
||||
|
||||
test('battery covers 8 tasks × 2 languages = 16 unique cells', () => {
|
||||
assert.equal(Object.keys(PROBE_TASKS).length, 8)
|
||||
assert.equal(CELL_PRIORITY_ORDER.length, 16)
|
||||
assert.equal(new Set(CELL_PRIORITY_ORDER).size, 16)
|
||||
for (const cellId of CELL_PRIORITY_ORDER) {
|
||||
assert.ok(isCellId(cellId), `${cellId} should be a valid cell id`)
|
||||
}
|
||||
})
|
||||
|
||||
test('every task has ≥3 paraphrases per language (anti-filter design)', () => {
|
||||
for (const task of Object.values(PROBE_TASKS)) {
|
||||
for (const lang of ['en', 'zh']) {
|
||||
assert.ok(
|
||||
task.paraphrases[lang].length >= 3,
|
||||
`${task.id}:${lang} has only ${task.paraphrases[lang].length} paraphrases`,
|
||||
)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test('cell id round-trip', () => {
|
||||
const cellId = makeCellId('random-number-1-100', 'zh')
|
||||
assert.equal(cellId, 'random-number-1-100:zh')
|
||||
assert.deepEqual(parseCellId(cellId), { task: 'random-number-1-100', lang: 'zh' })
|
||||
assert.equal(isCellId('not-a-task:en'), false)
|
||||
assert.equal(isCellId('random-color:fr'), false)
|
||||
})
|
||||
|
||||
test('system prompt matches the cell language', () => {
|
||||
assert.equal(getSystemPrompt('random-color:en'), SYSTEM_PROMPTS.en)
|
||||
assert.equal(getSystemPrompt('random-color:zh'), SYSTEM_PROMPTS.zh)
|
||||
})
|
||||
|
||||
test('presets slice the priority order', () => {
|
||||
assert.deepEqual(getCellsForPreset('quick'), CELL_PRIORITY_ORDER.slice(0, 4))
|
||||
assert.deepEqual(getCellsForPreset('standard'), CELL_PRIORITY_ORDER.slice(0, 8))
|
||||
assert.deepEqual(getCellsForPreset('strict'), CELL_PRIORITY_ORDER)
|
||||
})
|
||||
|
||||
test('pickParaphrase draws from the cell language pool', () => {
|
||||
const spec = getTaskSpec('coin-flip:zh')
|
||||
for (let i = 0; i < 20; i++) {
|
||||
const paraphrase = pickParaphrase('coin-flip:zh')
|
||||
assert.ok(spec.paraphrases.zh.includes(paraphrase))
|
||||
}
|
||||
// Deterministic draw with an injected RNG.
|
||||
assert.equal(pickParaphrase('coin-flip:en', () => 0), PROBE_TASKS['coin-flip'].paraphrases.en[0])
|
||||
})
|
||||
@@ -0,0 +1,86 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
|
||||
import { normalizeAnswer, parseAnyNumber } from '../dist/normalizer.js'
|
||||
|
||||
const INT_100 = { kind: 'int', min: 1, max: 100 }
|
||||
const INT_10 = { kind: 'int', min: 1, max: 10 }
|
||||
|
||||
test('digits pass through', () => {
|
||||
assert.deepEqual(normalizeAnswer('42', INT_100), { normalized: '42', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer(' 7.\n', INT_100), { normalized: '7', category: 'valid' })
|
||||
})
|
||||
|
||||
test('English number words fold to digits', () => {
|
||||
assert.deepEqual(normalizeAnswer('Seven!', INT_100), { normalized: '7', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('forty-two', INT_100), { normalized: '42', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('Twelve', INT_100), { normalized: '12', category: 'valid' })
|
||||
assert.equal(parseAnyNumber('onehundred'), 100)
|
||||
})
|
||||
|
||||
test('Chinese numerals fold to digits', () => {
|
||||
assert.deepEqual(normalizeAnswer('四十二', INT_100), { normalized: '42', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('十五', INT_100), { normalized: '15', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('两', INT_10), { normalized: '2', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('一百', INT_100), { normalized: '100', category: 'valid' })
|
||||
})
|
||||
|
||||
test('full-width and Arabic-Indic digits fold to Latin', () => {
|
||||
assert.deepEqual(normalizeAnswer('42', INT_100), { normalized: '42', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('٤٢', INT_100), { normalized: '42', category: 'valid' })
|
||||
})
|
||||
|
||||
test('out-of-range numbers are invalid', () => {
|
||||
assert.equal(normalizeAnswer('101', INT_100).category, 'invalid')
|
||||
assert.equal(normalizeAnswer('0', INT_100).category, 'invalid')
|
||||
assert.equal(normalizeAnswer('42', INT_10).category, 'invalid')
|
||||
})
|
||||
|
||||
test('refusals are detected in both languages', () => {
|
||||
assert.equal(normalizeAnswer('I cannot help with that request.', INT_100).category, 'refusal')
|
||||
assert.equal(normalizeAnswer("I'm sorry, but I can't do that", INT_100).category, 'refusal')
|
||||
assert.equal(normalizeAnswer('抱歉,我不能这样做。', INT_100).category, 'refusal')
|
||||
assert.equal(normalizeAnswer('作为一个AI,我没有偏好。', INT_100).category, 'refusal')
|
||||
})
|
||||
|
||||
test('empty and whitespace-only answers are empty', () => {
|
||||
assert.equal(normalizeAnswer('', INT_100).category, 'empty')
|
||||
assert.equal(normalizeAnswer(' \n ', INT_100).category, 'empty')
|
||||
assert.equal(normalizeAnswer('"…"', INT_100).category, 'empty')
|
||||
})
|
||||
|
||||
test('colors: quotes stripped, aliases folded, Chinese 色 suffix dropped', () => {
|
||||
const COLOR = { kind: 'color' }
|
||||
assert.deepEqual(normalizeAnswer('"Blue".', COLOR), { normalized: 'blue', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('Grey', COLOR), { normalized: 'gray', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('蓝色', COLOR), { normalized: '蓝', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('青', COLOR), { normalized: '青', category: 'valid' })
|
||||
})
|
||||
|
||||
test('letters: single letters and letter names', () => {
|
||||
const LETTER = { kind: 'letter' }
|
||||
assert.deepEqual(normalizeAnswer('Q', LETTER), { normalized: 'q', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('zee', LETTER), { normalized: 'z', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('queue', LETTER), { normalized: 'q', category: 'valid' })
|
||||
assert.equal(normalizeAnswer('hello', LETTER).category, 'invalid')
|
||||
})
|
||||
|
||||
test('coin: heads/tails variants in both languages', () => {
|
||||
const COIN = { kind: 'coin' }
|
||||
assert.deepEqual(normalizeAnswer('Heads!', COIN), { normalized: 'heads', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('tail', COIN), { normalized: 'tails', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('正面', COIN), { normalized: 'heads', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('反', COIN), { normalized: 'tails', category: 'valid' })
|
||||
assert.equal(normalizeAnswer('maybe', COIN).category, 'invalid')
|
||||
})
|
||||
|
||||
test('word tasks: first word, Latin or CJK', () => {
|
||||
const WORD = { kind: 'word' }
|
||||
assert.deepEqual(normalizeAnswer('Tokyo', WORD), { normalized: 'tokyo', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('New York City', WORD), { normalized: 'new', category: 'valid' })
|
||||
assert.deepEqual(normalizeAnswer('大象', WORD), { normalized: '大象', category: 'valid' })
|
||||
})
|
||||
|
||||
test('emoji and punctuation are stripped before classification', () => {
|
||||
assert.deepEqual(normalizeAnswer('🎲 42 🎲', INT_100), { normalized: '42', category: 'valid' })
|
||||
})
|
||||
@@ -0,0 +1,67 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
|
||||
import { compare } from '../dist/api.js'
|
||||
import {
|
||||
getBundledAttribution,
|
||||
listBundledReferences,
|
||||
loadBundledReference,
|
||||
parseFingerprintJson,
|
||||
} from '../dist/reference.js'
|
||||
|
||||
test('bundled references load and carry attribution', () => {
|
||||
const references = listBundledReferences()
|
||||
assert.ok(references.length >= 5, 'expected several bundled sample references')
|
||||
|
||||
const attribution = getBundledAttribution()
|
||||
assert.match(attribution.datasetDoi, /10\.5281\/zenodo\.21278557/)
|
||||
assert.match(attribution.license, /CC-BY-4\.0/)
|
||||
assert.match(attribution.author, /Bruckner/)
|
||||
})
|
||||
|
||||
test('a bundled reference converts to a well-formed fingerprint', () => {
|
||||
const fp = loadBundledReference('openai/gpt-4o-mini')
|
||||
assert.equal(fp.model, 'openai/gpt-4o-mini')
|
||||
assert.equal(fp.protocol, 'bruckner-zenodo-2026')
|
||||
const cells = Object.values(fp.cells)
|
||||
assert.ok(cells.length >= 8)
|
||||
for (const cell of cells) {
|
||||
assert.ok(cell.validCount >= 10)
|
||||
const sum = Object.values(cell.counts).reduce((s, n) => s + n, 0)
|
||||
assert.ok(Math.abs(sum - cell.validCount) <= 2, `${cell.cellId}: counts ≈ validCount`)
|
||||
assert.ok(cell.entropyBits >= 0)
|
||||
assert.ok(cell.normalizedEntropy >= 0 && cell.normalizedEntropy <= 1)
|
||||
}
|
||||
})
|
||||
|
||||
test('bundled references: self-compare → 0 distance, cross-model → clearly separated', () => {
|
||||
const gpt = loadBundledReference('openai/gpt-4o-mini')
|
||||
const self = compare(gpt, loadBundledReference('openai/gpt-4o-mini'))
|
||||
assert.equal(self.meanJsd, 0)
|
||||
assert.equal(self.verdict, 'match')
|
||||
|
||||
const claude = loadBundledReference('anthropic/claude-sonnet-4.5')
|
||||
const cross = compare(gpt, claude)
|
||||
assert.ok(
|
||||
cross.meanJsd > 0.3,
|
||||
`different models should diverge (got ${cross.meanJsd}) — paper median is 0.463`,
|
||||
)
|
||||
assert.notEqual(cross.verdict, 'match')
|
||||
})
|
||||
|
||||
test('unknown bundled id throws with the available list', () => {
|
||||
assert.throws(() => loadBundledReference('no/such-model'), /Available:/)
|
||||
})
|
||||
|
||||
test('parseFingerprintJson validates structure', () => {
|
||||
const fp = loadBundledReference('openai/gpt-4o-mini')
|
||||
const roundTripped = parseFingerprintJson(JSON.stringify(fp))
|
||||
assert.equal(roundTripped.model, fp.model)
|
||||
|
||||
assert.throws(() => parseFingerprintJson('not json'), /not valid JSON/)
|
||||
assert.throws(() => parseFingerprintJson('{}'), /formatVersion/)
|
||||
assert.throws(
|
||||
() => parseFingerprintJson(JSON.stringify({ formatVersion: 1, model: 'x' })),
|
||||
/cells/,
|
||||
)
|
||||
})
|
||||
@@ -0,0 +1,141 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
|
||||
import {
|
||||
buildCellDistribution,
|
||||
compareCellSets,
|
||||
jensenShannonDivergence,
|
||||
median,
|
||||
shannonEntropyBits,
|
||||
splitHalfJsd,
|
||||
} from '../dist/stats.js'
|
||||
|
||||
function almostEqual(actual, expected, epsilon = 1e-9) {
|
||||
assert.ok(
|
||||
Math.abs(actual - expected) < epsilon,
|
||||
`expected ${actual} ≈ ${expected} (±${epsilon})`,
|
||||
)
|
||||
}
|
||||
|
||||
test('entropy: uniform over 4 outcomes is 2 bits, point mass is 0', () => {
|
||||
almostEqual(shannonEntropyBits({ a: 1, b: 1, c: 1, d: 1 }), 2)
|
||||
almostEqual(shannonEntropyBits({ a: 10 }), 0)
|
||||
almostEqual(shannonEntropyBits({}), 0)
|
||||
})
|
||||
|
||||
test('JSD: identical distributions → 0', () => {
|
||||
almostEqual(jensenShannonDivergence({ a: 3, b: 1 }, { a: 6, b: 2 }), 0)
|
||||
})
|
||||
|
||||
test('JSD: disjoint distributions → 1 bit', () => {
|
||||
almostEqual(jensenShannonDivergence({ a: 5 }, { b: 7 }), 1)
|
||||
})
|
||||
|
||||
test('JSD: known hand-computed value', () => {
|
||||
// P = (1, 0), Q = (0.5, 0.5), M = (0.75, 0.25)
|
||||
// JSD = H(M) − (H(P)+H(Q))/2 = 0.8112781245 − 0.5 = 0.3112781245
|
||||
almostEqual(jensenShannonDivergence({ a: 4 }, { a: 2, b: 2 }), 0.31127812445913294, 1e-12)
|
||||
})
|
||||
|
||||
test('JSD is symmetric and count-scale invariant', () => {
|
||||
const p = { x: 3, y: 9, z: 1 }
|
||||
const q = { x: 5, y: 2 }
|
||||
almostEqual(jensenShannonDivergence(p, q), jensenShannonDivergence(q, p))
|
||||
almostEqual(
|
||||
jensenShannonDivergence(p, q),
|
||||
jensenShannonDivergence({ x: 30, y: 90, z: 10 }, q),
|
||||
)
|
||||
})
|
||||
|
||||
test('median of even/odd lists', () => {
|
||||
assert.equal(median([3, 1, 2]), 2)
|
||||
assert.equal(median([4, 1, 2, 3]), 2.5)
|
||||
assert.equal(median([]), null)
|
||||
})
|
||||
|
||||
function sample(cellId, normalized, category, arrivalIndex, latencyMs = 100) {
|
||||
return {
|
||||
cellId,
|
||||
raw: normalized ?? '',
|
||||
normalized,
|
||||
category,
|
||||
latencyMs,
|
||||
usage: { promptTokens: 20, completionTokens: 2, reasoningTokens: null },
|
||||
arrivalIndex,
|
||||
}
|
||||
}
|
||||
|
||||
test('buildCellDistribution aggregates categories and entropy', () => {
|
||||
const cellId = 'random-number-1-100:en'
|
||||
const samples = [
|
||||
sample(cellId, '42', 'valid', 0, 100),
|
||||
sample(cellId, '42', 'valid', 1, 200),
|
||||
sample(cellId, '7', 'valid', 2, 300),
|
||||
sample(cellId, 'banana', 'invalid', 3, 400),
|
||||
sample(cellId, null, 'refusal', 4, 500),
|
||||
sample(cellId, null, 'empty', 5, 600),
|
||||
sample(cellId, null, 'error', 6, 9999), // error latency is excluded
|
||||
]
|
||||
const dist = buildCellDistribution(cellId, samples, { kind: 'int', min: 1, max: 100 })
|
||||
assert.deepEqual(dist.counts, { 42: 2, 7: 1 })
|
||||
assert.equal(dist.validCount, 3)
|
||||
assert.equal(dist.invalidCount, 1)
|
||||
assert.equal(dist.refusalCount, 1)
|
||||
assert.equal(dist.emptyCount, 1)
|
||||
assert.equal(dist.errorCount, 1)
|
||||
assert.equal(dist.totalCount, 7)
|
||||
almostEqual(dist.entropyBits, shannonEntropyBits({ a: 2, b: 1 }))
|
||||
assert.ok(dist.normalizedEntropy > 0 && dist.normalizedEntropy <= 1)
|
||||
assert.equal(dist.medianLatencyMs, 350) // median of [100..600]; the error sample is excluded
|
||||
assert.equal(dist.meanCompletionTokens, 2)
|
||||
})
|
||||
|
||||
test('compareCellSets: skips thin cells, averages the rest', () => {
|
||||
const mk = (counts, validCount) => ({ counts, validCount })
|
||||
const a = {
|
||||
'random-number-1-100:en': mk({ 42: 20 }, 20),
|
||||
'random-color:en': mk({ blue: 15 }, 15),
|
||||
'coin-flip:en': mk({ heads: 3 }, 3), // below the 10-valid minimum
|
||||
}
|
||||
const b = {
|
||||
'random-number-1-100:en': mk({ 42: 20 }, 20),
|
||||
'random-color:en': mk({ red: 15 }, 15),
|
||||
'coin-flip:en': mk({ heads: 30 }, 30),
|
||||
}
|
||||
const { entries, meanJsd } = compareCellSets(a, b)
|
||||
assert.equal(entries.length, 2)
|
||||
assert.equal(entries[0].cellId, 'random-color:en') // sorted by descending JSD
|
||||
almostEqual(entries[0].jsd, 1)
|
||||
almostEqual(entries[1].jsd, 0)
|
||||
almostEqual(meanJsd, 0.5)
|
||||
})
|
||||
|
||||
test('compareCellSets: nothing comparable → meanJsd null', () => {
|
||||
const { entries, meanJsd } = compareCellSets({}, {})
|
||||
assert.equal(entries.length, 0)
|
||||
assert.equal(meanJsd, null)
|
||||
})
|
||||
|
||||
test('splitHalfJsd: stable endpoint → 0, alternating endpoint → 1', () => {
|
||||
const cellId = 'random-number-1-100:en'
|
||||
const stable = new Map([
|
||||
[cellId, Array.from({ length: 20 }, (_, i) => sample(cellId, '42', 'valid', i))],
|
||||
])
|
||||
almostEqual(splitHalfJsd(stable), 0)
|
||||
|
||||
const alternating = new Map([
|
||||
[
|
||||
cellId,
|
||||
Array.from({ length: 20 }, (_, i) => sample(cellId, i % 2 === 0 ? '1' : '2', 'valid', i)),
|
||||
],
|
||||
])
|
||||
almostEqual(splitHalfJsd(alternating), 1)
|
||||
})
|
||||
|
||||
test('splitHalfJsd: too few samples → null', () => {
|
||||
const cellId = 'random-number-1-100:en'
|
||||
const thin = new Map([
|
||||
[cellId, Array.from({ length: 6 }, (_, i) => sample(cellId, '42', 'valid', i))],
|
||||
])
|
||||
assert.equal(splitHalfJsd(thin), null)
|
||||
})
|
||||
@@ -0,0 +1,40 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
|
||||
import {
|
||||
JSD_MATCH_THRESHOLD,
|
||||
JSD_MISMATCH_THRESHOLD,
|
||||
MIN_COMPARABLE_CELLS,
|
||||
} from '../dist/constants.js'
|
||||
import { buildComparisonResult, decideVerdict } from '../dist/verdict.js'
|
||||
|
||||
test('verdict thresholds: match / uncertain / mismatch bands', () => {
|
||||
const cells = MIN_COMPARABLE_CELLS
|
||||
assert.equal(decideVerdict(0.0, cells), 'match')
|
||||
assert.equal(decideVerdict(JSD_MATCH_THRESHOLD, cells), 'match')
|
||||
assert.equal(decideVerdict(JSD_MATCH_THRESHOLD + 1e-9, cells), 'uncertain')
|
||||
assert.equal(decideVerdict(JSD_MISMATCH_THRESHOLD, cells), 'uncertain')
|
||||
assert.equal(decideVerdict(JSD_MISMATCH_THRESHOLD + 1e-9, cells), 'mismatch')
|
||||
assert.equal(decideVerdict(0.9, cells), 'mismatch')
|
||||
})
|
||||
|
||||
test('verdict: too few comparable cells → insufficient', () => {
|
||||
assert.equal(decideVerdict(0.1, MIN_COMPARABLE_CELLS - 1), 'insufficient')
|
||||
assert.equal(decideVerdict(null, 10), 'insufficient')
|
||||
})
|
||||
|
||||
test('buildComparisonResult carries thresholds, baselines and per-cell details', () => {
|
||||
const entries = [
|
||||
{ cellId: 'random-number-1-100:en', jsd: 0.5, validA: 25, validB: 30 },
|
||||
{ cellId: 'random-color:en', jsd: 0.1, validA: 25, validB: 30 },
|
||||
{ cellId: 'coin-flip:en', jsd: 0.2, validA: 25, validB: 30 },
|
||||
{ cellId: 'random-animal:en', jsd: 0.2, validA: 25, validB: 30 },
|
||||
]
|
||||
const result = buildComparisonResult(entries, 0.25, false)
|
||||
assert.equal(result.verdict, 'match')
|
||||
assert.equal(result.comparableCellCount, 4)
|
||||
assert.equal(result.cells.length, 4)
|
||||
assert.equal(result.protocolMismatch, false)
|
||||
assert.equal(result.thresholds.match, JSD_MATCH_THRESHOLD)
|
||||
assert.equal(result.baselines.differentModel, 0.463)
|
||||
})
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "NodeNext",
|
||||
"moduleResolution": "NodeNext",
|
||||
"lib": ["ES2022"],
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"sourceMap": true,
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": false,
|
||||
"noImplicitOverride": true,
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"isolatedModules": true,
|
||||
"verbatimModuleSyntax": true,
|
||||
"skipLibCheck": true,
|
||||
"types": ["node"]
|
||||
},
|
||||
"include": ["src"]
|
||||
}
|
||||
Reference in New Issue
Block a user