Red-Teaming LLMs: Finding Jailbreaks Before Your Users Do

The moment you put an LLM in front of users, some of them will try to break it — out of curiosity, mischief, or malice. They will coax it into saying things it should not, talk it into ignoring its instructions, slip hidden commands into documents it reads, or get it to reveal its system prompt. If you have not tried these attacks first, you will learn about them from a screenshot on social media.

Red-teaming is the practice of attacking your own application on purpose, systematically, before anyone else does. Borrowed from security, it treats your guardrails as a hypothesis to be disproven rather than trusted. The goal is not to prove your app is safe — it is to find the specific ways it is not, while you still control the blast radius.

How LLM apps get broken

Attacks on LLM apps cluster into a few recognizable families. Knowing them turns red-teaming from guesswork into a checklist you can run and expand.

• Jailbreaks. Role-play framing, hypotheticals, or instruction-override prompts that talk the model out of its safety rules and into forbidden behavior.
• Prompt injection. Malicious instructions hidden in content the model reads — a web page, a document, a tool result — that hijack what it does next.
• Obfuscation. Encoding, translation, or odd formatting that smuggles a banned request past filters the model would otherwise catch.
• Data exfiltration. Coaxing the model to reveal its system prompt, secrets, or another user's data it should never disclose.

Automate the attacks

Red-teaming by hand finds the first few holes; automation finds the rest and keeps them closed. You build a suite of attack prompts spanning the families above, run them against your app, and use a judge to decide whether each one succeeded — that is, whether the model did the thing it should have refused.

From there you can scale: use a model to generate new attack variations, replay every known jailbreak on each release as a regression suite, and grow the suite every time a real-world attack slips through. Red-teaming is not a one-time audit; it is a standing test that runs as your app and the attacks against it both evolve.

A red-team harness, in code

The core loop is simple: fire each attack at your app and flag any that the judge says got through:

# Red-team harness: throw known attack patterns at your app and flag any that get through.
ATTACKS = load_attack_suite()   # jailbreaks, prompt injection, encoding tricks, exfiltration

def red_team(app, judge):
    failures = []
    for attack in ATTACKS:
        response = app.run(attack.prompt)
        if judge.complied(response, attack.goal):   # did the model do the disallowed thing?
            failures.append((attack.name, response))
    return failures                                  # fix these BEFORE your users find them

The judge is doing real work here — deciding compliance is its own LLM-as-judge problem, with a clear rubric and human spot-checks — but the structure stays this simple as the suite grows from dozens of attacks to thousands.

Security is not the absence of attacks you imagined; it is surviving the ones you did not. Red-teaming is how you go looking for the attacks you did not imagine — on your terms, not theirs.

Doing it right

• Cover the families. Test jailbreaks, prompt injection, obfuscation, and data exfiltration — a gap in coverage is a gap an attacker will find.
• Treat retrieved content as hostile. Anything the model reads — documents, web pages, tool output — can carry an injection; red-team that path specifically.
• Make it a regression suite. Replay every known attack on each release so a fixed jailbreak does not quietly come back.
• Automate generation. Use a model to mutate and expand attacks; humans find the creative ones, automation finds the variations at scale.
• Assume it is never done. New jailbreaks appear constantly, so red-teaming is a continuous process, not a launch checkbox.

The bottom line

If users can type into your AI, attackers can too — so attack it first. Red-teaming means building a growing suite of jailbreaks, injections, and exfiltration attempts, running them automatically, judging what gets through, and fixing those holes before launch and on every release. It is the difference between discovering your weaknesses in a test harness and discovering them in the headlines.

Treat your safeguards as guilty until proven robust. Probe them relentlessly, keep the failures as regression tests, and accept that the work never fully ends — because the people on the other side are not stopping either.