I evaluated 5 LLM agents on patching real-world CVEs. Here is what I found.

I built an independent benchmark with 20 real CVEs across 15 CWE categories, 5 models (3 OpenAI, 2 Poolside Laguna), three prompt conditions: full advisory, behavioral description only, and location only (file and function, no description of the flaw). I have three findings worth sharing: No model reliably fixes real vulnerabilities. The best solve rate (gpt-5.5) is 50% overall and 60% under the most favorable condition. The failure modes (e.g, wrong-search drift, budget exhaustion mid-implementation, plausible-but-incomplete patches that pass every visible test) are structured and repeatable across models and tasks. Token cost varies 4x for equivalent outcomes. The Laguna models consume 3–4x more tokens than OpenAI models of the same capability tier, with no improvement in solve rate. The locate condition is the benchmark’s sharpest instrument. Give a model only a file and function (no description of the flaw). Every model drops. The differences between models are within noise at this scale, but it’s the condition that most closely resembles what a security researcher actually does: reading code cold and recognizing independently that something is wrong. Benchmark code and evaluation traces are open sourced. submitted by /u/Fickle-Box1433 [link] [comments]Technical Information Security Content & DiscussionRead More