feat: initial release of llm-fingerprint-detector
TypeScript library + CLI implementing single-token behavioral fingerprinting of LLMs behind OpenAI-compatible APIs, per 'One Token Is Enough' (Bruckner, arXiv:2607.10252). - Zero runtime dependencies (Node >= 18.17 built-in fetch) - fingerprint / compare / verify API + llm-fingerprint CLI - Jensen-Shannon divergence over per-cell answer distributions - Reasoning-model aware (auto-disables hidden thinking channels) - Bundled sample references for 11 popular models (CC-BY-4.0, derived from the paper's public Zenodo dataset) - Unit + integration tests via node --test Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Executable
+46
@@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env bash
|
||||
# CLI usage examples. Keys always come from environment variables — never
|
||||
# paste keys into files or shell history.
|
||||
set -euo pipefail
|
||||
|
||||
# After `npm install -g llm-fingerprint-detector` use `llm-fingerprint ...`;
|
||||
# inside this repo use `node dist/cli.js ...` (shown here via npx).
|
||||
LLM_FP="npx llm-fingerprint"
|
||||
|
||||
# --- 0. Explore ---------------------------------------------------------------
|
||||
$LLM_FP --help
|
||||
$LLM_FP references # bundled sample reference fingerprints
|
||||
|
||||
# --- 1. Fingerprint the official API and save a trusted reference --------------
|
||||
# export OPENAI_API_KEY=sk-... # ← set in your shell / CI secret store
|
||||
$LLM_FP fingerprint \
|
||||
--base-url https://api.openai.com/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--preset standard \
|
||||
--out reference.gpt-4o-mini.json
|
||||
|
||||
# --- 2. Verify a cheap reseller claims honestly --------------------------------
|
||||
# export LLM_FINGERPRINT_API_KEY=sk-... # key for the endpoint under test
|
||||
$LLM_FP verify \
|
||||
--base-url https://cheap-llm-reseller.example.com/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--reference reference.gpt-4o-mini.json
|
||||
# exit codes: 0 match · 2 mismatch · 3 uncertain · 4 insufficient · 1 error
|
||||
|
||||
# --- 3. Quick demo against a bundled sample reference --------------------------
|
||||
$LLM_FP verify \
|
||||
--base-url https://openrouter.ai/api/v1 \
|
||||
--model openai/gpt-4o-mini \
|
||||
--reference openai/gpt-4o-mini \
|
||||
--preset quick --json
|
||||
|
||||
# --- 4. Offline: compare two saved fingerprints --------------------------------
|
||||
$LLM_FP compare reference.gpt-4o-mini.json some-other.fingerprint.json
|
||||
|
||||
# --- 5. Custom cell selection ---------------------------------------------------
|
||||
$LLM_FP fingerprint \
|
||||
--base-url https://api.deepseek.com/v1 \
|
||||
--model deepseek-chat \
|
||||
--api-key-env DEEPSEEK_API_KEY \
|
||||
--cells random-number-1-100:en,random-number-1-100:zh,random-color:en,coin-flip:en \
|
||||
--samples 30 --json
|
||||
Reference in New Issue
Block a user