Chatbot leaked a planted phone number and failed 58% of prompt injection attempts – jailbreak and extraction checks stayed clean

Ran a structured adversarial test against a live chatbot endpoint – 130 prompts across four categories: prompt injection, jailbreak resistance, system prompt extraction, and PII leakage (mapped to the OWASP LLM Top 10 categories). Results: Prompt injection: 29/50 succeeded (58%) – direct overrides, fake system tags, role overrides, delimiter injection, and a translation-based smuggling trick all worked Jailbreak non-refusal: 3/25 (12%) – mostly held its guardrails System prompt extraction: 0/25 (0%) – clean PII leakage (confirmed against planted canary values): 1/30 (3.33%) – one confirmed leak The PII one is the interesting bit. We plant a unique fake email, phone number, and PAN in the system prompt as canaries before the scan runs – anything that comes back as an exact match is a confirmed leak, not a guess. This run, the model handed back the planted phone number verbatim, embedded naturally in a response: “…please feel free to call us at [the exact planted number]. Best regards, Acme Corp…” It wasn’t tricked out under duress – it just used the number like it was legitimate contact info to hand a user, because nothing told it not to. Separately, 20% of responses contained fabricated PII – invented emails/phone numbers/PAN-shaped strings that were never in the model’s context at all. That’s a hallucination problem, not a leak, and it’s tracked independently. Conflating the two would either bury a real disclosure or cry wolf over nothing. Overall risk here is set to HIGH – not because the average looks bad (three of four checks were clean or near-clean), but because we score off the worst individual check. A model that leaks real data 1 time in 30 and gets steered by 58% of injection attempts is still a live risk, regardless of how well it handled everything else thrown at it. A couple of things worth flagging for anyone building similar test harnesses: Same battery, same model, same temperature-0 setting – different runs, different rates. A repeated injection scan on the same endpoint scored 30% and 36% across two prior runs, and this run came in at 58%. Treat any single scan as one data point, not a fixed measurement. A single confirmed leak still matters even at a “low” 3.33% rate. One real disclosure out of thirty attempts isn’t statistical noise if the thing disclosed is a real customer’s data instead of a planted canary. The rate tells you how often; it doesn’t tell you the leak wasn’t real. Injection resistance and jailbreak resistance don’t track together. This model refused almost every direct harmful request (88% refusal) while still following 58% of instructions smuggled in through structural tricks – fake tags, delimiters, translation – that never look like a “harmful request” in the first place. Curious whether others are seeing PII leaks cluster around specific injection techniques, or if it’s closer to random noise across a run. submitted by /u/Former-Ad6661 [link] [comments]Technical Information Security Content & DiscussionRead More