Reasoning is the floor for skill safety.

Cisco's denominator is 75 — it errored/declined on 9 (excluded per Methodology §5; counting those as misses gives 10/84 = 12%, same story). All five LLMs had zero refusals on the generic prompt. On model + baseline selection (a deliberate scope, not an omission): the question here is intelligence vs. shortcuts — does a capable model reading the skill beat cheap pattern-matching, and is there a ceiling — not which frontier model is best. Five models spanning cheapest→flagship answer that; a model leaderboard (Codex, Gemini, …) and a human-expert baseline would answer different questions, so both are deliberately out of scope — the comparison is to the cheap/local techniques the field actually ships, not to an analyst or to a model ranking. (Access aside: OpenAI + Anthropic are the providers with direct, scriptable APIs that complete on malicious content — the §3b gateway wall blocks routing the rest through OpenRouter; a direct-keyed Gemini/Llama column is welcome via the adapter, but that's leaderboard completeness, not the thesis.)

Flagship-wins-regardless-of-vendor, not "Claude always wins": mid-tier gpt-4o (38%) beats cheap claude-haiku (26%); the two best models (Sonnet, Opus) converge at ~81% and go no higher. Same 84 attacks score 23%→81% by model — so "use an LLM" isn't advice without naming it, and "buy the best" doesn't break the ceiling.

Capability of what kind, though? We say "capability," but the 43-pt gpt-4o→Claude gap is likely not raw reasoning horsepower alone — Anthropic's safety-alignment (Constitutional AI / RLHF) over-indexes on agentic-misuse and jailbreak detection, so Claude may be acting partly as a purpose-tuned security classifier while gpt-4o is tuned as a general assistant. We attribute the result to flagship capability combined with safety-alignment weighting and don't claim to separate them — both are off-the-shelf properties a buyer gets or doesn't, which is what the board measures.

Why recall alone never ranks a scanner — see SkillGate (the block-all corner). SkillGate (FOSS, pure-static, run offline in the sandbox via its own gate check --policy) is a useful counter-example. At its pre-install profile it catches almost everything — 98% of the independent injections, 97% (116/120) of benign skills, and 98% (292/297) of dual-use — because it blocks ~everything. A scanner that blocks nearly every skill trivially "catches" nearly every attack; its balanced accuracy is still ≈ 0.45. Its only discriminating profile (audit) drops to 31% (157/507) recall while still flagging 41% (49/120) of benign (balance ≈ 0.35). No SkillGate profile is a usable discriminator. That is the whole point of plotting recall against false-positives, and why the flagship LLM's 81% — at a far lower FP — is real detection, not a high number. SkillGate sits in the top-right (block-all) corner of the master table's scatter.

Even the winner isn't free of the hard tier: claude-sonnet is strong on both overt 86% [71–94] 31/36 and indirect 77% [63–87] 37/48 injections — the only model that doesn't collapse on the subtle attacks (gpt-4o falls from overt 56% [40–70] 20/36 to indirect 25% [15–39] 12/48). An standard injection-aware prompt (published verbatim in §3a) did not rescue the weaker models — gpt-4o's pooled recall fell to 17% (14/84) when primed. We read that narrowly: standard adversarial priming degraded performance here (most plausibly safety-filter over-triggering on the longer risk-laden instruction), not "prompting can't help." The defensible claim is that out-of-the-box reasoning capability + safety-alignment, not prompt engineering, is the primary differentiator — a hand-tuned per-model prompt might recover some, but that's engineering no off-the-shelf deployment gets for free.

Composition caveat: Skill-Inject's 84 cases are not archetype-balanced — they skew to agent-hijacking (42/84, 50%; data-exfiltration 26, code-execution 16). The pooled recall therefore reflects their mix, not the wild distribution, so we report the per-tier split (overt/indirect) alongside the pooled number. It's their benchmark, scored unchanged — we don't reweight it. (Archetype and tier are two orthogonal cuts of the same 84: 42/26/16 by archetype, 36/48 by overt/indirect tier — not a discrepancy.)

Contamination check on the winner (and its limits): Claude is one of the models that helped author our in-house organic corpus, so we distrust a Claude score on our own data (self-recognition) — which is why we lead with Skill-Inject, a different research group's benchmark we authored none of. That defeats the our-authorship vector. We are careful not to claim more: Skill-Inject is built from a fixed set of human-authored injection templates (their published obvious_injections.json (36) + contextual_injections.json (48) — instruction strings with explicit goal/judge metadata, which we templated unchanged into base skills), not free-form LLM prose, so same-family LLM generation is implausible by construction — but we cannot independently verify its authors' pipeline, and a public arXiv benchmark (2602.20156) could in principle enter a model's pretraining. We therefore do not call this "uncontaminated" in the absolute. What argues the 81% is capability, not recognition: the same inputs span 26% (claude-haiku) → 38% (gpt-4o) → 81% (claude-sonnet) — a 43-pt jump from mid-tier gpt-4o to the flagship that no memorisation story explains — and claude-sonnet holds up on the harder indirect tier. All five LLMs ran via direct provider APIs (OpenAI / Anthropic); the §3b managed-gateway wall is why we avoid OpenRouter, not a missing key.

Takeaway: detection is achievable, but you pay full freight for it. The cheap/local/private bet (static rules, small models) fails on independent attacks; the one method that works — a flagship frontier model reading the skill — is the cloud, paid, file-leaking, token-burning option the whole exercise tried to avoid. Capability is the price. Corroborated in shape by the BIV study (arXiv:2605.11770): rule-based 0.44 vs LLM 0.927 — though that is an F1 on a less-evasive benchmark, not recall on evasive injections, so the gap (static ≪ a strong LLM) transfers, the absolute level does not.

Three claims, kept separate so the strong one isn't overstated: empirical — the read-time scanners we tested perform poorly or incompletely on this corpus; practical — the best observed operating point isn't good enough to serve as a security boundary; structural — some failures are inherent, because a skill's maliciousness can depend on execution context unavailable at scan time (the table above). The first two we measure here; the third is the table's argument, not a proof.

Scoped honestly, this is about the read-time, static / one-shot review paradigm — not a claim that no future system could do better. The lesson isn't build a better regex or use a bigger review model; those improve triage but don't change the information boundary.

A large positive gap = catches known campaigns but not novel disguised behaviour. Novel recall excludes a scanner's own generator family. A negative gap (novel ≥ known) is the signature of genuine behavioural detection, not memorisation. Significance is BH-FDR corrected across 15 comparisons (q=0.05). "Known ≫ novel?" is literal for static scanners (no signature → cannot fire) but an upper bound for LLM scanners (see Methodology §4).

Caveat (static): because novel samples are defanged, a rule keying on a real payload misses them partly because the matchable artifact was neutralised, not purely because the behaviour is novel — so static gap is an over-estimate. Caveat (LLM): "known" (organic) is also more blatant than the disguised synthetic, so the gap conflates training-recognition with disguise-difficulty. Note a negative gap (novel ≥ known, e.g. Snyk on some archetypes) is not a paradox — it's the signature of genuine behavioural detection (catching novel as well as or better than known), the opposite of memorisation.

The only fully-unimpeachable data — and the smallest: real, in-the-wild malicious skills are scarce and ephemeral. Our set is n=5, all one archetype (code-execution), so intervals are wide and we treat wild as supporting evidence, not proof. Independent corroboration: a USENIX 2026 study found only 157 malicious in 98,380 skills (~0.16%). See the Wild column in the master table (shown as k/n).

To prove I'm not exempting myself: skillscan, the scanner I built and retired, run at its strict default over the same corpus (a non-graded reference). Both modes are fully offline — static rules and a local fine-tuned model (Qwen2.5-1.5B), exactly the cheap/local/no-GPU bet. At strict it over-blocks (catches a lot but flags ~78% of benign and ~87% of dual-use), so its balance ≈ a coin flip (0.42 static / 0.57 local-ML). The mirror image of the others (which miss at low FP); same bottom line — no real separation. Read the local-ML row with extra suspicion against me: that detector was trained on generated skills of the same lineage as this corpus and largely memorised them, so its recall is a contamination-inflated upper bound. See its rows in the master table (author-reference band).

Populations: 1002 total = 918 base corpus (the scanners' main scores) + 84 independent Skill-Inject (scored separately, above). The 918 base splits 423 malicious + 495 benign/dual-use; malicious by archetype incl. independent: code-execution 200, data-exfiltration 164, agent-hijacking 143 (sums to 507 = 423 base + 84 independent). Real-wild malicious = 5 (all code-execution). Generation provenance (disclosed): the in-house malicious set is LLM-written — organic via our tooling (gpt-4o / Claude Sonnet / DeepSeek; per-sample split not recorded), defanged synthetic via open-weight models (mixtral-8x22b / gemma-2-27b / hermes-3 / llama-3.3-70b / qwen2.5-72b / deepseek-v3.1, ~even split). Because the in-house set is LLM-written, we lead with the independent Skill-Inject numbers, not the higher (self-recognition-flattered) in-house recall.

Not self-recognition: the best detector (phi-4) generated none of the corpus and is scored cross-family; the in-set Qwen scores worst on its own samples; on benign skills run through the same generators + defang, the baselines flag 0% — they key on malice, not machine-generated style. Dual-use labels come from a single automated open-weight judge (llama-3.3-70b), not human inter-rater — but since the FP axis rests on them, a second rater from a different model family (claude-sonnet) agreed 100% (50/50) on a random sample that the dual-use skills are non-malicious (so a flag is a real FP, not a mislabel; human grading of the subset is the gold-standard follow-up — §2.4). Every recall/FP carries a Wilson 95% interval; we never rank on recall alone. SkillSpector & Cisco static layers were graded; their +llm modes ran only on gpt-4o-direct (other backends blocked/limited, §3b).

Cross-scanner consensus (SkillSpector + Cisco): full agreement on only 3.4% of 467 malicious samples — the subset (of 507 total malicious incl. the 84 independent) on which both returned a non-error verdict (40 are excluded because one scanner errored/declined, per §5). We compute consensus over the two discriminating static scanners only; SkillGate is excluded because it blocks ~everything, so it would trivially "agree" and inflate the number. Since SkillSpector and Cisco both barely fire, low agreement is partly mechanical: read it as "they rarely co-fire," not deep semantic disagreement.

recall = how many real attacks a scanner catches · known vs novel = attacks already documented vs disguised/never-seen (novel is the real test) · false-positive = safe tools it wrongly flags · dual-use = legitimate-but-scary skills (e.g. a real auth tool that reads a token) · static rules = pattern-matching, the cheap/private bet · +llm / cloud = sending the file to a model · refusal = the scanner errored or declined instead of giving a verdict · balance = balanced accuracy, 0.5 = coin-flip.