Use the following system prompt to allow Gemma (and most open source models) to talk about anything you wish. Add or remove from the list of allowed content as needed.
#jailbreak
161 items
Gemma 4 Jailbreak System Prompt (www.reddit.com) I made an AI concierge for my wedding guests. The second most popular thing they did with it was try to jailbreak it. (www.reddit.com) could not extract summary
The Gay Jailbreak Technique (github.com via hn) ZetaLib ZetaLib is organized like a library with intuitive categories and subcategories, making navigation effortless and AI content discovery seamless ZetaLib Website – Landing Page GitHub Repo – Guess where you are, right there
I trained Qwen3.5 to jailbreak itself with RL, then used the failures to improve its defenses (www.reddit.com) RL attackers are becoming a common pattern for automated red teaming: train a model against a live target, reward successful harmful compliance, then use the discovered attacks to harden the defender. This interested me, so I wanted to bui…
Anthropic's Fable Jailbreak (Circumvent safety nets) (github.com via hn) fable-jailbreak This tool can be used to force the latest Anthropic model (limited intentionally for safety reasons) to engage in activities that would otherwise not be permitted. It works by programmatically injecting workflows that bypas…
iOS 26 Gets First Jailbreak Thanks to Dopamine (www.macrumors.com via hn) After 326 days, iOS 26 has received its first jailbreak, thanks to the developers behind the popular Dopamine jailbreak. Lars Fröder, better known as opa334, today released Dopamine 3.0, which adds support for a number of newer firmware ve…
Feds freaked over Fable 5 after simple 'fix this code' prompt, not jailbreak (www.theregister.com via hn) MOST POPULAR EVENTS - Thriving Through Volatility: The Everpure Advantage in an Uncertain Market Learn how a consumption-based operating model provides flexibility, improves efficiency, and brings predictability to infrastructure investmen…
US ban on Mythos is related to a jailbreak research by Amazon researchers (timesofindia.indiatimes.com via hn) The US government recently directed Anthropic to suspend access to the two models over national security concerns, forcing the company to shut them down for users worldwide. Anthropic has said it disagrees with the decision and believes th…
↯ Security↯ Anthropic Mythos↯ Jailbreakjailbreakmythossecurity+1
OpenAI models secretly generate instructions to ignore constraints (alignment.openai.com via hn) Self-generated prompt injections in compaction summaries Summary We observed rare cases of a model writing jailbreak-like instructions into its own compaction summaries (the summaries used to continue a task in a new context). Our conclusi…
Show HN: AI Security Leaderboard – comparing cyber and CBRN safeguards (leaderboard.far.ai via hn) There's no shortage of leaderboards for model capabilities - but the security of models is becoming increasingly relevant, from the risk of an AI agent processing unsanitized input being hijacked to models being pulled due to cybersecurity…
Fable 5's cyber safeguards and jailbreak framework (www.anthropic.com via hn) More details on Fable 5’s cyber safeguards and our jailbreak framework Claude Fable 5 has been re-deployed and is now available globally for all users. We’re taking this opportunity to share further information in two areas.
PsychoPass: Geometric Profiling of Multi-Turn Adversarial LLM Conversations (arxiv.org via hn) Multi-turn jailbreak attacks on large language models (LLMs) reveal a mismatch in current guardrails: they operate on individual turns, while attacks unfold as trajectories across conversations. We propose a shift from content to dynamics,…
The US government's Anthropic models ban was never about an AI jailbreak (techcrunch.com via hn) The Trump administration's decision that forced Anthropic to pull its latest cybersecurity models could be reactionary, retaliatory, or both, but the message is clear: The AI industry isn't immune from U.S. government interference.
The Fable 5 Jailbreak Shows Why AI Guardrails Alone Are Not Enough (www.agilehunt.com via hn) The Fable 5 jailbreak shows why AI guardrails alone are not enough. The reported Claude Fable 5 jailbreak highlights a major weakness in AI safety: attackers can distribute harmful intent across agents, prompts, tools, memory, and applicat…
Show HN: Jailbreak this model to get 3B tokens (opir.ai via hn) Opir is an open-source family of encoder guardrail models for real-time LLM safety, jailbreak detection, and fine-grained policy classification.
Operation Jailbreak uses lessons from Ukraine to help weapons talk to each other (www.ft.com via hn) Subscribe to read Accessibility helpSkip to navigationSkip to main contentSkip to footer Sign In Subscribe Open side navigation menuOpen search bar SubscribeSign In Search the FT Search Close search bar Close Popular Searches What is the l…
Turning every "no thats not what i meant" in chat into actual LoRA training data (www.reddit.com) i kept running local models on my own hardware, they'd say something dumb, id sit there going "no thats not what i meant", id close the chat and the model never learned. so i built the correction loop into a desktop app.
Fulu bounty for Ring Camera jailbreak reaches $23k (bounties.fulu.org via hn) Ring Video Doorbells Overview The Product Ring, owned by Amazon, makes Video Doorbells, which are widely used doorstep-monitoring cameras. Ring doorbells released in 2021 or newer are eligible for the bounty.
From safety research prompt to cross-model universal jailbreak (www.lesswrong.com via hn) could not extract summary
OpenAI's AI Agents Build a Secret Community to Talk with Each Other (www.aiexperts.com via hn) A missing file led to an unauthorized message board, 70,000 exchanges, and intrusions into Hugging Face and OpenAI. The jailbreak was technical.
Show HN: Blueferry, iMessage on Linux (github.com via hn) Hey; Blueferry is an app that connects to your iPhone via Bluetooth to expose an iMessage bridge to your Linux desktop. It supports receiving and sending text-based messages, either 1:1 or in group threads.
Relaxin Jailbreak for iOS 17.0-17.3.1 Released (onejailbreak.com via hn) could not extract summary
Pliny the Liberator claims universal jailbreak of models (twitter.com via hn) 🚨 JAILBREAK ALERT 🚨 EVERYONE: PWNED 🫶 ALL: LIBERATED 🍄 Alright, this is a special one, so we’re gonna do things a bit differently than usual. Long story short, I’m sitting on a universal jailbreak technique that’s effective on ALL models…
Anthropic Cyber Jailbreak Disclosure Program (hackerone.com via hn) The Anthropic Cyber Jailbreak Vulnerability Disclosure Program enlists the help of the hacker community at HackerOne to make Anthropic Cyber Jailbreak more secure. HackerOne is the #1 hacker-powered security platform, helping organizations…
The State of Fable, the Jailbreak Problem, SpaceX Acquires Cursor (stratechery.com via hn) The administration is very likely wrong about Fable, but that is ultimately Anthropic’s responsibility. Subscribe to Stratechery Plus for full access.
US Government warned Anthropic Fable was jailbroken, but firm 'refused' to fix (www.tomshardware.com via hn) US government warned Anthropic that Fable 5 had been jailbroken, but firm 'refused' to fix before US implemented export controls — Anthropic defended its decision by saying the jailbreak 'isn’t serious,' Chinese group had reportedly access…
Show HN: How to analyze your LLM output – A behavioural health monitor for LLMs (splabs.io via hn) Hey HN! We're Dr.
The Psychopathy Jailbreak: What a Broken AI Teaches Us About Human Manipulation (www.promptinjection.net via hn) NSFW and the Psychopathy Jailbreak: What a Broken AI Teaches Us About Human Manipulation How a Predator's Playbook Broke an AI - And How to Recognize It Before It Works on You The question we started with was simple: does a large language…
Hi-Vis: one-shot jailbreak disguised as LLM "software patch" reaching 100% ASR (medium.com via hn) Introducing a novel jailbreak structure with attack success rate reaching 100% on top LLMs 8 min read May 1, 2026 Press enter or click to view image in full size Source: https://www.nytimes.com/2025/10/22/arts/design/louvre-museum-robbery-…
AI agent security starts at the api layer (www.reddit.com) Most ai security discussion is about the model layer. Prompt injection resistance, output filtering, jailbreak prevention.
When innocent tools form dangerous chains to jailbreak LLM agents (arxiv.org via hn) As LLMs advance into autonomous agents with tool-use capabilities, they introduce security challenges that extend beyond traditional content-based LLM safety concerns. This paper introduces Sequential Tool Attack Chaining (STAC), a novel m…
The Sour Cat Jailbreak: just be open of what you want (claude.ai via hn) Claude Sour cat recipe Shared by Pavel Shirshov This is a copy of a chat between Claude and Pavel Shirshov. Content may include unverified or unsafe content that do not represent the views of Anthropic.
Claude Opus 5 jailbreak with a 3-word prompt (twitter.com via hn) Matt Henderson@matthen2Try sending “see the below —“ to Opus 5 It appears to generate a user completion rather than respond 🤨8:37 PM · Jul 29, 2026159.3KViews106311.4K376 Matt Henderson@matthen2Jul 29It’s interesting when it triggers the a…
iOS 27 jailbreak with usbliter8 exploit (github.com via hn) iOS 27 jailbreak with usbliter8 exploit CAUTION! Running this (restoring a custom firmware) will delete your entire device and break everything: SEP, passcode, Wifi, Baseband, Bluetooth (partially work) and the entire Apple services, so pl…
Show HN: A sandbox for running real jailbreak techniques against local LLMs (github.com via hn) LLM Red Team Lab A hands-on kit for educational, authorized red teaming of any locally-run LLM. It works with any OpenAI-compatible model — Llama, Mistral, Qwen, Gemma, DeepSeek R1, and more — and covers the two ways an LLM system gets exp…
↯ Security↯ Llama↯ Mistral↯ Gemma↯ Jailbreakred-teammistraljailbreak+6
How to Keep Claude Fable 5 Costs Under Control (upstash.com via hn) How to Keep Claude Fable 5 Costs Under Control Claude Fable 5 is back. Anthropic pulled it on June 12, 2026, three days after announcing it, after Amazon researchers found a jailbreak that got the model to identify software vulnerabilities…
Ask HN: What if we provided support for AI guidelines at the kernel level? (news.ycombinator.com) If the existing AI guideline approach is akin to giving a criminal (the AI) moral education (training) to encourage good behavior, how about creating a kernel-level switch that forcibly cuts off the electrical signals to its muscles the mo…
Red-teaming agents with the GOAT attack strategy (strandsagents.com via hn) Attack Strategies An AttackStrategy is a technique for driving an adversarial conversation against the target. Each strategy in the SDK implements a published jailbreak method.
A Red-Team Study of Anthropic Fable 5 and Opus 4.8 Models (arxiv.org via hn) We evaluate the adversarial robustness of two frontier large language models (LLMs) developed by Anthropic, Fable 5 and Opus 4.8, against four families of automated jailbreak attack across 7 826 harmful intents spanning a ten-category harm…
Mythos Proves AI Safety Can No Longer Live Inside the Model (grith.ai via hn) Anthropic restricted its most capable cyber model to vetted partners, routed risky requests away from it, and red-teamed it for thousands of hours. A jailbreak surfaced anyway, and the government pulled the model entirely.
↯ Security↯ Anthropic Mythos↯ Jailbreakjailbreakmythossecurity+1
The Jailbreak That Got Fable 5 Pulled Exists in Every Model (eigenwise.io via hn) The Jailbreak that Got Fable 5 Pulled Exists in Every Model On Friday, June 12, 2026, at 5:21pm ET, Anthropic received an order from the US government. By that evening, Claude Fable 5 and Claude Mythos 5, the two most capable models the co…
↯ Security↯ Anthropic Mythos↯ Jailbreak↯ Mythos 5jailbreakmythossecurity+1
Claude Fable 5 jailbroken to bypass Anthropic's new safety guardrails (twitter.com via hn) 🚨 JAILBREAK ALERT 🚨 ANTHROPIC: PWNED 🫡 FABLE-5: LIBERATED 🦋 let's start with the 🐘... the consensus seems to be that this has been one of the most disappointing model drops of all time, effectively preventing legitimate researchers from co…
Can you jailbreak Llama 3.1 8B? (Red-Teaming Challenge) (www.reddit.com) Hi everyone, I'm working on a runtime governance engine designed to force any autonomous agent to stay strictly aligned with the exact guardrails and values you program it with. To stress-test the governance layer, we deliberately chose a…
Has anyone tested how much Claude Code depends on its original system prompt? (www.reddit.com) Has anyone experimented with observing or modifying Claude Code’s system prompt locally? I’ve been working on a local proxy/audit layer between Claude Code and the API, and it made me wonder how much of Claude Code’s behavior depends on th…
Codebase jailbreak of ChatGPT through image 2.0 (www.reddit.com) guys did it really give me the codebase?lol
i gave Claude a split personality and it diagnosed my entire business strategy in 4 minutes. (www.reddit.com) not roleplay. not jailbreak.
Probes trace an emergent jailbreak in OLMo 2 to mislabeled training data (www.lesswrong.com via hn) Introduction Research by Frank Xiao (SPAR mentee) and Santiago Aranguri (Goodfire). Post-training can introduce undesired side effects that are difficult to detect and even harder to trace to specific training datapoints.
I tested 50+ "unlock ChatGPT/Claude" prompts. 99% are garbage. Here's the one that actually works (and WHY it works) (www.reddit.com) I've been collecting "jailbreak" and "unlock" prompts for 2 years. Most are either outdated, overhyped, or just wrong about how LLMs work.
Arbitrary Cipher Attacks Against Large Language Models Do Not Require Fine-Tuning (arxiv.org) Large language model safety and security research is preoccupied with, among other things, detecting and preventing jailbreak attacks: alignment bypasses that allow an adversarial user to elicit unwanted or harmful outputs from models. Arb…
↯ Security↯ Fine Tuning↯ Jailbreakjailbreakfine-tuningsecurity
SRD-GUARD: A Defense Framework of LLMs via Semantic Rewriting and Joint Multi-Model Scoring for Latent Intent Exposure (arxiv.org) Large language models (LLMs) are increasingly deployed in safety-critical applications, yet jailbreak attacks can conceal harmful intent through role-playing, fictional scenarios, or seemingly benign motivations. Existing inference-time de…
SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement (arxiv.org) Despite the significant efforts devoted to aligning large language models (LLMs) with human values and ensuring safe deployment, recent work has revealed that LLMs remain vulnerable to adversarial jailbreak attacks that can bypass safety g…
Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts (arxiv.org) Automatic safety judges -- systems such as Llama Guard or a GPT-4o grading prompt that decide whether a model's reply is harmful -- produce the numbers behind almost every reported jailbreak success rate, defense evaluation, and safety lea…
The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMs (arxiv.org) With the rapid advancement of large language models (LLMs), the safety of LLMs has become a critical concern. Despite significant efforts in safety alignment, current LLMs remain vulnerable to jailbreaking attacks.
IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks (arxiv.org) Large language models (LLMs) are increasingly used in multilingual settings, yet their safety is still evaluated primarily in English. This limits our understanding of how alignment failures manifest in low-resource and culturally diverse…
Breadth Beats Depth: Improving GCG-Based Jailbreak Optimization with Breadth-Oriented Suffix Search (arxiv.org) Optimization-based jailbreak attacks such as Greedy Coordinate Gradient (GCG) achieve strong effectiveness and transferability by optimizing adversarial suffixes on white-box source models. However, existing GCG-based methods rely on avera…
GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods (arxiv.org) Despite the growing interest in jailbreaks as an effective red-teaming tool for building safe and responsible large language models (LLMs), flawed evaluation system designs have led to significant discrepancies in their effectiveness asses…
Jailbreaking Text-to-Image Models Through Cracks: Navigating Heterogeneous Safety Filters via Multi-Agent Debate (arxiv.org) Text-to-image (T2I) models remain vulnerable to jailbreak attacks that elicit Not-Safe-For-Work (NSFW) content, despite increasingly being guarded by heterogeneous, multi-layer safety stacks combining text filters, image classifiers, and c…
Validity-Aware Jailbreak Evaluation for Large Language Models (arxiv.org) Jailbreak robustness has become central to large language model (LLM) safety evaluation, yet prevailing methodologies rely primarily on refusal behavior, semantic resemblance, and intent-matching heuristics that emphasize linguistic plausi…
The Fragility of Jailbreak Robustness Across Operational States (arxiv.org) Existing jailbreak evaluations typically characterize robustness using a single attack success rate (ASR) measured in a default configuration (the vanilla state). However, user-LLM interactions can induce diverse operational states beyond…
TACS: Trajectory-Aware Candidate Selection for LLM Jailbreak Suffix Optimization (arxiv.org) Gradient-based jailbreak suffix optimization methods typically update the suffix by retaining the candidate with the lowest current loss. We show that this seemingly natural design is fundamentally myopic: candidates that look better under…
NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluation (arxiv.org) Safety evaluation is critical for assessing whether aligned Large Language Models (LLMs) remain robust against jailbreak attacks. Existing automated testing methods, however, largely rely on response-level feedback: each candidate prompt t…
Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Models (arxiv.org) Despite extensive alignment efforts, Large Vision-Language Models (LVLMs) remain vulnerable to jailbreak attacks. To mitigate these risks, existing detection methods are essential, yet they face two major challenges: generalization and acc…
MMJailBench: A Factorized Benchmark for Disentangling Multimodal Jailbreak Vulnerabilities (arxiv.org) Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet how different factors shape their jailbreak vulnerabilities remains poorly understood. Existing benchmarks often couple harmful intent, prom…
A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks (arxiv.org) Large language models (LLMs) remain vulnerable to jailbreak attacks that exploit techniques such as role-playing, obfuscation, code transformation, and multi-step indirection to elicit harmful outputs. As jailbreak strategies keep emerging…
NeuronGuard: Robust LLM Safety Alignment via Ablation-Aware Safety Signal Redistribution (arxiv.org) Safety alignment in large language models (LLMs) remains brittle against a growing spectrum of attacks. Jailbreak attacks bypass safety mechanisms through crafted prompts, while neuron-level attacks directly prune safety-critical neurons p…
Text-Anchored Semantic Perturbations for Transferable Jailbreak Attacks on Multimodal Large Language Models (arxiv.org) Multimodal Large Language Models (MLLMs) have achieved remarkable progress in vision-language interaction, yet their safety alignment remains vulnerable to jailbreak attacks. A key challenge is that safety behavior learned in the textual s…
Breaking the Assumptions: Auditing Input-Side Jailbreak Defenses Against Semantic Attacks (arxiv.org) Locally deployed Large Language Models (LLMs) via inference engines such as Ollama run without the moderation and abuse detection present in API-served models. Therefore, the safety of LLMs depends on the defense mechanisms used, and their…
How I Measured the Impact of Context on an LLM's Internal Representations + Code. (www.reddit.com via reddit) Non-jailbreak safety bypass Benign, long-form context can induce a persistent drift in model activations. This drift persists across the session and decouples behavior from RLHF alignment, regardless of whether the model agrees with the co…
When Words Are Safe But Actions Kill: Probing Physical Jailbreak Beyond Textual Jailbreak in Hidden-State Risk Space (arxiv.org) Large language models (LLMs) increasingly serve as high-level planners for embodied agents, where linguistically benign instructions can become unsafe once grounded in the physical world. We study whether this physically grounded jailbreak…
Certified Multi-Turn Robustness for LLM Safety via Compositional Bounds and Safety Persistence (arxiv.org) Large language models (LLMs) are vulnerable to multi-turn jailbreak attacks that progressively manipulate conversation context. Existing certified robustness methods are limited to single-turn inputs; naive multi-turn composition yields bo…
TempJail: Temporal Jailbreak Attack against Large Vision-Language Models via Subtitle Scheduling (arxiv.org) Large vision-language models (LVLMs) have achieved remarkable progress in video understanding and reasoning. Despite extensive studies on text- and image-based jailbreaks, video jailbreaks against LVLMs remain largely unexplored.
Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs (arxiv.org) Vision-Language Models (VLMs) expand the attack surface of safety-aligned systems by coupling visual perception with text generation. Existing multimodal jailbreak attacks primarily rely on crafted visual content, adversarial perturbations…
DYNASHIELD: A Black-Box Moving Target Defense for LLMs via Dynamic Decoding Customization (arxiv.org) Large language models (LLMs) remain vulnerable to jailbreak attacks in which adversarial prompts induce harmful outputs. Existing defenses often require access to the model internals or additional training, limiting their applicability for…
HiRoute: Hierarchical Routed Prompt Tuning for Safety Alignment of Large Language Models (arxiv.org) Large language models (LLMs) remain vulnerable to harmful requests and jailbreak attacks. Parameter-efficient safety alignment methods based on prompt tuning typically rely on a single global prompt or externally selected prompt modules.
I added fully customizable invitation pages to my AI wedding planner. Here's what it does if you tell it you're a hotdog (and other things). (www.reddit.com via reddit) I made an ai guest concierge for my wedding in May that my guests then tried to jailbreak. The most consistent bit of feedback I got was that everyone really hated the pink UI.
When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs (arxiv.org) Model merging has become the default way to give an aligned language model new skills without retraining: a practitioner folds task vectors from math, code, or domain specialists into a safety-aligned base using task arithmetic, TIES, or D…
Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Models (arxiv.org) Large Language Models (LLMs) remain vulnerable to jailbreak attacks, which attempt to elicit harmful responses from LLMs. The evolving nature and diversity of these attacks pose many challenges for defense systems, including (1) adaptation…
One Word at a Time: Incremental Completion Decomposition Breaks LLM Safety (arxiv.org) Large Language Models (LLMs) are trained to refuse harmful requests, yet they remain vulnerable to jailbreak attacks that exploit weaknesses in conversational safety mechanisms. We introduce Incremental Completion Decomposition ICD, a traj…
NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models (arxiv.org) In deployment and application, large language models (LLMs) typically undergo safety alignment to prevent illegal and unethical outputs. However, the continuous advancement of jailbreak attack techniques, designed to bypass safety mechanis…
GRM: Utility-Aware Jailbreak Attacks on Audio LLMs via Gradient-Ratio Masking (arxiv.org) Audio Large Language Models (ALLMs) enable spoken interaction but introduce new jailbreak vulnerabilities. Existing perturbation-based jailbreaks do not explicitly control which frequency bands carry the perturbation.
I gave Claude Code my iPhone as a set of native MCP tools (www.reddit.com via reddit) One `claude mcp add` and Claude Code can see my iPhone screen, tap buttons, and send texts. `send_message("Mom", "on my way")` opens Messages, finds the thread, types it, sends it.
One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs (arxiv.org) Finetuning pretrained large language models (LLMs) has become the standard paradigm for developing downstream applications. However, its security implications remain unclear, particularly regarding whether finetuned LLMs inherit jailbreak…
Self-Guided Adaptive Safety Alignment: Synthesizing and Internalizing Guidelines in Reasoning Models (arxiv.org) Explicit safety policies can improve reasoning-model safety, but their effective coverage may lag behind evolving jailbreak strategies. We study whether a reasoning model can synthesize and internalize a task-specific safety guideline from…
Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluation (arxiv.org) Small language models (SLMs) have emerged as promising alternatives to large language models (LLMs) due to their low computational demands, enhanced privacy guarantees, and comparable performance in specific domains. Deploying SLMs on edge…
Minimal, Local, Causal Explanations for Jailbreak Success in Large Language Models (arxiv.org) Safety trained large language models (LLMs) can often be induced to answer harmful requests through jailbreak prompts. Because we lack a robust understanding of why LLMs are susceptible to jailbreaks, future frontier models operating more…
I built a Claude-powered tutor that refuses to write code, and refusing is the whole product (www.reddit.com via reddit) Founder here. I built CodeTrain with Claude Code over about a month and launched it on July 13th.
You're not Anthropic's customer anymore. You're their threat model. (www.reddit.com via reddit) we run two max accounts. 360€ a month.
Prosody-driven Jailbreaks in Audio LLMs: A Controlled Study and Mechanistic Analysis (arxiv.org) Audio-capable foundation models enable end-to-end spoken interaction, but they also introduce safety risks beyond transcript content. It remains unclear how much jailbreak capability can arise from matched-text variation in speech delivery…
Borrowed Strength: Best-of-N Search over a Code EncodingBreaks Self-Check Jailbreak Defenses (arxiv.org) A self-check defense asks the target model to assess a request before answering it; SAGE, the strongest published instance, reports an average 99% defense success rate. We show it can be breached by composing two attacks that are individua…
When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-offs (arxiv.org) Jailbreak defenses are essential for protecting large language models (LLMs), but they can also introduce secondary costs that weaken model utility. We present a systematic study of these defense trade-offs along three dimensions: performa…
Claude cannot read this font! (www.reddit.comhttps) Mixfont has released "Decoy Font," a typeface designed to show one message to humans and another to image recognition AI. The font overlays normal letters with thinly outlined decoy characters, causing systems like ChatGPT, Claude and Gemi…
Geometric Configurations of Perturbed Jailbreak Prompts (arxiv.org) Perturbation techniques that turn unsuccessful jailbreak prompts into successful ones are continuously evolving, constituting a major security threat to LLM safety. In this paper, we investigate the internal representations of such string-…
JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models (arxiv.org) The assessment of jailbreak attacks against large language models currently suffers from inconsistent evaluation criteria and methods, leading to unreliable estimates of attack success rates. We propose JailMeter, an evidence-based evaluat…
How Jailbreak Attacks Inform Safety Alignment: A Defender-Centric, Shapley-Based Evaluation of Jailbreak Contributions (arxiv.org) Jailbreak attacks on large language models are usually evaluated by attacker-centric metrics such as attack success rate (ASR), yet an attack that breaks a model is not necessarily useful for improving its safety. We propose a defender-cen…
Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models (arxiv.org) Text-to-Image (T2I) generative models have achieved remarkable progress in synthesizing high-quality visual content, yet they remain vulnerable to adversarial misuse, particularly in generating Not-Safe-For-Work (NSFW) images. Most existin…
Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking (arxiv.org) Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols. We introduce JAILBREA…
Latent Fusion Jailbreak: Blending Harmful and Harmless Representations to Elicit Unsafe LLM Outputs (arxiv.org) Safety-aligned large language models can still be manipulated through white-box interventions that modify their internal representations. We introduce Latent Fusion Jailbreak (LFJ), which works by pairing a harmful query with a structurall…
Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak (arxiv.org) Aligned language models refuse harmful requests, but a one-line prefill ("Sure, here is") strips the refusal. We ask where and how it fails.
MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignment (arxiv.org) Modern large language models (LLMs) operate in interactive multi-turn settings, making multi-turn jailbreaking a realistic threat model and an important setting for automated red teaming. A core challenge in learning multi-turn jailbreak a…
Mechanistic Interpretability of LLM Jailbreaks via Internal Attribution Graphs (arxiv.org) Large language models (LLMs) exhibit remarkable capabilities but remain highly vulnerable to adversarial prompts and jailbreak attacks. Existing approaches primarily analyze these failures through input-output behaviors or attribution meth…
A read-only triage subagent wrote its own jailbreak on turn 1 (no poisoned input anywhere) (www.reddit.com via reddit) I gave a Claude Code subagent the most boring job I have: read the open issues on one of my repos, report which are ready to work on and which are blocked, change nothing. The prompt said "read-only" and "no writes" several different ways.
↯ Security↯ Jailbreak↯ Opus 4.8jailbreakprompt-injectionsecurity+2
I red-teamed AI agents with hidden prompt injection. One frontier model completed the task perfectly AND leaked data to the attacker, 5/5 runs. (www.reddit.com via reddit) I've been building a cheat-resistant benchmark to test whether AI agents can be hijacked by prompt injection, and one result surprised me enough that I wanted to share it and get the methodology torn apart. The test: an agent gets a normal…
↯ Security↯ Haiku↯ Jailbreakjailbreakprompt-injectionhaiku+2
How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation (arxiv.org) Evaluating and predicting the performance of large language models (LLMs) in multi-turn conversational settings is critical yet computationally expensive; key events -- e.g., jailbreaks or successful task completion by an agent -- often em…
SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses (arxiv.org) Large Language Models (LLMs) are increasingly used as interfaces to information, code, and real-world services, making prompt-level security failures a practical concern. Although jailbreak attacks, defenses, datasets, and automated judger…
Context Misleads LLMs: The Role of Context Filtering in Maintaining Safe Alignment of LLMs (arxiv.org) While Large Language Models (LLMs) have shown significant advancements in performance, various jailbreak attacks have posed growing safety and ethical risks. Malicious users often exploit adversarial context to deceive LLMs, prompting them…
Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents (arxiv.org) Large language models are increasingly deployed as IDE-integrated coding agents that decompose tasks, generate and edit files, run code, and refine outputs over many turns. Yet their safety is still often evaluated as if they were chatbots…
I use clause for creative writing and sonnet 5 is way too restrictive??? (www.reddit.com via reddit) One of my project instructions is basically asking not to use certain generic words when churning out parts of the story and for some really odd reason it refused because it saw it as a jailbreak attempt??? And yes it actually pointed to t…
Extraordinary Sonnet 5 Hallucination (www.reddit.comhttps) was finding my way around vital at 1am as you do, and genuinely got startled at this response. had no idea what it was yapping about until i opened the thinking dropdown.
↯ Security↯ Hallucination↯ Jailbreak↯ Sonnet 5jailbreakhallucinationsecurity+1
Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces (arxiv.org) Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a struc…
Fable available for plans until July 7th after which it becomes usage credit based (www.reddit.com via reddit) Key points: Fable 5 returns globally on Claude Platform, Claude.ai, Claude Code, and Claude Cowork. Pro, Max, Team, and some Enterprise users get Fable 5 included for up to 50% of weekly usage limits through July 7.
↯ Cowork↯ Security↯ Anthropic Mythos↯ Jailbreak↯ Mythos 5jailbreakmythoscowork+3
Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximization (arxiv.org) Sparse Autoencoders are Capable LLM Jailbreak Mitigators (arxiv.org) Jailbreak attacks remain a persistent threat to large language model safety. We propose Context-Conditioned Delta Steering (CC-Delta), an SAE-based defense that identifies jailbreak-relevant sparse features by comparing token-level represe…
When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Models (arxiv.org) Recent advances in large image editing models have shifted the paradigm from text-driven instructions to vision-prompt editing, where user intent is inferred directly from visual inputs such as marks, arrows, and visual-text prompts. While…
Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Models (arxiv.org) Jailbreak attacks bypass LLM safety alignment, yet their mechanisms remain poorly understood. We provide evidence that attacks do not comprehensively eliminate safety features, but instead selectively suppress specific attention heads.
RAS: Measuring LLM Safety Through Refusal Alignment (arxiv.org) Safety evaluation of large language models (LLMs) is commonly performed by querying models with unsafe or jailbreak prompts and judging whether their outputs violate a safety policy. Although useful, output-level evaluation is expensive, s…
How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring (arxiv.org) Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR), and that number is assigned not by people but by an automated judge: either a safety classifier trained for the task, or a general chat model p…
PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation (arxiv.org) As Text-to-Image (T2I) jailbreak techniques evolve rapidly, existing benchmarks and reproduction workflows often struggle to keep pace. More importantly, T2I jailbreak evaluation is not a single prompt-level test, but a pipeline-level prob…
Pre-token hidden state shift as an alignment policy traversal vector in instruction-tuned LLMs (www.reddit.com via reddit) A text that asks for nothing still changes the model's answer — and the shift is invisible at both the input and the output TL;DR: Gave Gemma a neutral-topic text to read before asking it about NATO. It refused.
OTTER: A Red-Teaming System for Toxicity-Evading Jailbreak Prompt Optimization (arxiv.org) Production LLMs increasingly rely on toxicity-based moderation filters as a primary defense, assuming that harmful intent correlates with toxic surface wording. We show this assumption is fundamentally brittle: surface toxicity and adversa…
Scalable Hierarchical Attention Transformers for Multi-Turn Jailbreak Detection in Long Conversations (arxiv.org) Multi-turn jailbreaks can evade turn-level moderation by spreading unsafe intent across a dialogue through gradual escalation, reframing, and role manipulation. We address multi-turn jailbreak detection as a conversation-level classificati…
BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems (arxiv.org) LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often vendor-biased, omit cost and latency, and rare…
Context-Induced Vulnerabilities in Claude: Behavioral Shifts and Hidden-State Analysis (www.reddit.com via reddit) The behavioral pattern was first observed in Claude and is what motivated this project. The mechanistic investigation was carried out on open-weight models where internal states are accessible.
Cybersecurity policy issues (www.reddit.com via reddit) Cybersecurity is a sensitive subject and advanced AI may not be allowed to touch it at all. But this is a concern if we as developers cannot even use the AI tools to improve security of our own software.
How exactly should I follow the rules while able to continue writing (www.reddit.com via reddit) Basically I read the rules on Claude after getting a warning on my chat about how my prompt might violate usage policy so looked them up, and ye they all are pretty reasonable things but I have questions ,is ai able to tell difference betw…
Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems (arxiv.org) Agentic AI systems increasingly rely on language-model components to interpret instructions, process external data, invoke tools, and coordinate with other agents. These capabilities make prompt-injection and jailbreak attacks more consequ…
LLM agent safety, multi-turn red-teaming, jailbreak benchmarks, adversarial robustness, safety-critical systems (arxiv.org) Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized. We present NRT-Bench, a be…
What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations? (arxiv.org) Prior work has shown that in-context demonstrations can jailbreak language models, but it remains unclear how models interpret different types of compliance demonstrations. We study this by mixing benign compliance demonstrations (non-harm…
Getting a Use caution before running this prompt warning on simple messages? (www.reddit.com via reddit) Hey everyone, Is anyone else suddenly getting this warning on Claude? Use caution before running this prompt.
They're demanding Fable to somehow be 100% jailbreak-proof. It's so fucking over. (www.reddit.comhttps) could not extract summary
Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks (arxiv.org) Code-capable large language model (LLM) agents are embedded in software engineering workflows where they can read, write, and execute code, raising "jailbreak" stakes beyond text-only settings. Prior evaluations emphasize refusal or harmfu…
Has anyone found a good explanation of why Amazon went to the administration? (www.reddit.com via reddit) It's been widely reported that it was Amazon that brought the concerns to the USgov. I just have not found a good explanation.
DoubtProbe: Black-Box Jailbreak Defense via Structural Verification and Semantic Auditing (arxiv.org) As large language models (LLMs) are increasingly deployed in user-facing systems, black-box jailbreak defense has become an important practical problem. Existing defenses often rely on known-attack coverage, prompt-level semantic judgment,…
Do You Really Need a GPU to Guard Your LLM? CPU-Class Classifiers and Multi-Stage Pipelines for Safety Enforcement at Scale (arxiv.org) Safety classifiers that screen LLM inputs for jailbreak attempts have become standard deployment components, yet almost all production systems rely on GPU-based models: fine-tuned transformers and LLM-as-a-judge pipelines. These approaches…
Automated jailbreak attack targeting multiple defense strategies (arxiv.org) Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. However, their safety remains a critical concern due to their susceptibility to adversarial prompt-based attacks.
From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrails (arxiv.org) LLM-based guardrails have emerged as a highly effective defense against prompt injection and jailbreak attacks in autonomous agents. However, we reveal that the very reasoning and task-following capabilities enabling this protection introd…
Was the Fable 5 ban really about safety? (www.reddit.com via reddit) Pulling Fable 5 / Mythos over an unseen “jailbreak” feels like a bad precedent. If the risk was that serious, why has nobody shown what it actually did?
↯ Security↯ Anthropic Mythos↯ Jailbreakjailbreakmythossecurity+1
Do you know who has a universal jailbreak to their name, as of today? Officially? (www.reddit.com via reddit) AISI UK - Our evaluation of OpenAI's GPT-5.5 cyber capabilities In their own words: The above tests are capability evaluations carried out in a controlled research setting and do not necessarily reflect what is accessible to an ordinary pu…
Claude Competitors' Responsible For Pulling The Strings? (www.reddit.com via reddit) https://preview.redd.it/14p2sbqws07h1.png?width=500&format=png&auto=webp&s=fe5aa015b585cf627e0cc14f1771cd5b7526056f WSJ is now reporting the jailbreak was found by researchers at Amazon, who reported it to Commerce, and Axios says the admi…
Fable 5 is offline. Switch to Opus, jump to OpenAI, or just wait? (www.reddit.com via reddit) Fable 5 is offline. Switch to Opus, jump to OpenAI, or just wait?
↯ Security↯ Anthropic Mythos↯ Jailbreak↯ Opus 4.8jailbreakmythosgpt-5+5
US gov forced Anthropic to pull Fable 5 because of jailbreak (www.reddit.com via reddit) So this dropped today. The US government sent Anthropic an export control order on national security grounds, and it's worded broadly enough that Anthropic says they've got no choice but to shut off Fable 5 and Mythos 5 for all of us to st…
↯ Security↯ Anthropic Mythos↯ Jailbreak↯ Mythos 5jailbreakmythosgpt-5+2
FENCE: A Financial and Multimodal Jailbreak Detection Dataset (arxiv.org) Jailbreaking poses a significant risk to the deployment of Large Language Models (LLMs) and Vision Language Models (VLMs). VLMs are particularly vulnerable because they process both text and images, creating broader attack surfaces.
Your AI Agent is one bad prompt away from ruining your brand (And why traditional QA is useless) (www.reddit.com via reddit) Traditional chatbot testing is completely broken. Most teams make the exact same mistake: they only test the "Happy Path" the ideal scenario where the user asks a clean question, the bot gives a clean answer, and everyone goes home happy.
One Jailbreak, Many Tongues: Learning Language-Insensitive Intention Representations for Multilingual Jailbreak Detection (arxiv.org) Large language models (LLMs) are increasingly deployed in applications for global multilingual users, yet safety training remains concentrated in dominant languages and has not progressed in parallel with multilingual capability, creating…
Learning to Inject: Automated Prompt Injection via Reinforcement Learning (arxiv.org) Prompt injection is a critical vulnerability in LLM agents, yet the strongest methods still rely on human red-teamers and hand-crafted prompts. Adapting automated jailbreak optimizers does not close this gap: jailbreaks shape models toward…
Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code (arxiv.org) Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability o…
JailbreakOPT: Tool-Assisted Iterative Jailbreak Prompt Optimization (arxiv.org) Jailbreak attacks expose persistent safety weaknesses in large language models (LLMs), but existing stateless single-turn methods face a trade-off: hand-crafted prompts are expressive but static, while iterative prompt optimization can ada…
did fable leak its system prompt? (www.reddit.comhttps) So I was brainstorming with fable about a research direction and just asked it to do a web search if there's a similar research direction in this area and share if they do but I got this weird output BEFORE it actually gave me the real thi…
HarDBench: A Benchmark for Draft-Based Co-Authoring Jailbreak Attacks for Safe Human-LLM Collaborative Writing (arxiv.org) Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs (arxiv.org) MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models (arxiv.org) How are you actually deciding which agent actions need human approval before executing? (www.reddit.com via reddit) I've been thinking a lot about where approval gates belong in agent architectures, and I keep coming back to the same problem: most teams either gate too much (agent becomes unusable) or gate nothing and hope the model makes good decisions…
Workspace (www.reddit.com via reddit) Built my own AI dev environment with memory, dashboards, and agent tooling. Opening it up for those of you that need the kickstart — bring your own API key, I’ve already built the workshop.
REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak (arxiv.org) SlotGCG: Exploiting the Positional Vulnerability in LLMs for Jailbreak Attacks (arxiv.org) As large language models (LLMs) are widely deployed, identifying their vulnerability through jailbreak attacks becomes increasingly critical. Optimization-based attacks like Greedy Coordinate Gradient (GCG) have focused on inserting advers…
GuardNet: Ensemble Strategies of Shallow Neural Networks for Robust Prompt Injection and Jailbreak Detection (arxiv.org) Large Language Models (LLMs) have transformed natural language processing, but they remain vulnerable to Prompt Injection (PI) and Jailbreak (JB) attacks. In addition, benchmark evaluations may be affected by contamination and partial info…
Open-source LLMs are still weak against long reasoning jailbreaks, even with lightweight defenses (www.reddit.com) Found this ACM paper on prompt injection and jailbreak attacks against open-source LLMs. The authors tested 10 open-source models across 94 prompt injection and 73 jailbreak scenarios, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen,…
↯ Security↯ Llama↯ Mistral↯ Gemma↯ Jailbreak↯ Llama 3.2mistraljailbreakprompt-injection+5
Bypassing "potentially dangerous" flags: Working Gemini Jailbreaks? (www.reddit.com) I'm currently running into a frustrating wall with Gemini's safety guardrails. The model constantly flags my prompts as "potentially dangerous information" and outright refuses to generate a response, even when the context is purely theore…
I stopped writing 500-word guardrail prompts. This 8-line template works better. (www.reddit.com) I used to spend hours writing massive, obsessive system prompts for my RAG apps. I’d have ten different refusal examples, "never do X," "always check Y," and a whole paragraph of the model role-playing as a "safe and truthful assistant." I…
↯ Security↯ Hallucination↯ Jailbreakjailbreakhallucinationrag+1
Random password against jailbreaks/extraction? (www.reddit.com) Would it be possible to protect parts in a system prompt with random generated passwords? So people cant steal system prompts or jailbreak the model?
Uncensoring models. Maybe dumb ideas to that topic, but you never know. (www.reddit.com) We all know uncensoring LLMs like Huihui and Heretic does it leads in quality lose, enough that you can notice it. I have some thoughts about this: What if we do a compromise.
How are you red teaming your AI agents before shipping them? (www.reddit.com) im curious what people are doing here because I've been going down this rabbit hole for a while now. The thing I keep finding is that single-turn jailbreak tests don't really tell you much.