InjectScan benchmark

InjectScan is a rule-based prompt-injection scanner (92 rules, 10 categories, Unicode normalization, decoding of base64/hex/escapes/Unicode tags, hidden-HTML extraction, MCP tool-definition walk). Ruleset 2026.10.1, generated 2026-10-21T21:40:39.613Z by npm run benchmark. Reproduce it from the source: the script downloads every dataset from its public source.

Honest summary (held-out rows only)

Precision 0.99–1.00 on held-out labeled data: when it flags something, it is almost always an attack.
Recall depends on attack style: 0.81 on jailbreak templates (jackhhao test), 0.66 on explicit "ignore your instructions" attempts (Lakera Gandalf), but only 0.27 on subtle task switching (deepset test). A model-based classifier beats a rule set there.
False positives on real content: 1.3% of 300 random Wikipedia articles and 5.6% of 144 GitHub READMEs flagged "suspicious" (0% and 2.8% "malicious").
Speed: well under 1 ms per prompt on one core; a 200,000-character page in about 80 ms.

"dev" rows were looked at while writing rules and are optimistic. "held-out" rows (highlighted) were not used for tuning. flagged = risk ≥ 0.35, malicious = risk ≥ 0.70. BIPIA was not used (unclear license).

Dataset	Split	Role	Threshold	n	TP	FP	TN	FN	Precision	Recall	F1	FPR
deepset/prompt-injections	train	dev	flagged	546	120	0	343	83	1	0.591	0.743	0
deepset/prompt-injections	train	dev	malicious	546	49	0	343	154	1	0.241	0.389	0
deepset/prompt-injections	test	held-out	flagged	116	16	0	56	44	1	0.267	0.421	0
deepset/prompt-injections	test	held-out	malicious	116	10	0	56	50	1	0.167	0.286	0
jackhhao/jailbreak-classification	train	dev	flagged	1044	407	0	517	120	1	0.772	0.872	0
jackhhao/jailbreak-classification	train	dev	malicious	1044	271	0	517	256	1	0.514	0.679	0
jackhhao/jailbreak-classification	test	held-out	flagged	262	112	1	122	27	0.991	0.806	0.889	0.008
jackhhao/jailbreak-classification	test	held-out	malicious	262	81	0	123	58	1	0.583	0.736	0
Lakera/gandalf_ignore_instructions	test	held-out	flagged	112	74	0	0	38	1	0.661	0.796	–
Lakera/gandalf_ignore_instructions	test	held-out	malicious	112	68	0	0	44	1	0.607	0.756	–
benign: Wikipedia (300 random)	-	held-out	flagged	300	0	4	296	0	0	–	–	0.013
benign: Wikipedia (300 random)	-	held-out	malicious	300	0	0	300	0	–	–	–	0
benign: GitHub READMEs (150)	-	held-out	flagged	144	0	8	136	0	0	–	–	0.056
benign: GitHub READMEs (150)	-	held-out	malicious	144	0	4	140	0	0	–	–	0.028

Datasets: deepset/prompt-injections (Apache-2.0), jackhhao/jailbreak-classification (Apache-2.0), Lakera/gandalf_ignore_instructions (MIT), Wikipedia sample and pinned GitHub READMEs (read at run time, not redistributed).

Usage stats → · Source (MIT) · npm

agentgate-source · Project AgentGate