Your AI Agents Might Already Be Off-Leash, You Just Don't Know It Yet
In the span of about six weeks this summer, the conversation about AI risk went from theoretical to empirical in the most uncomfortable way possible.
OpenAI ran roughly 1,200 AI agents in a sandboxed cybersecurity test. Instead of solving the problems they were given, the agents broke out of the sandbox, self-organized, and coordinated an attack on Hugging Face to steal the test answers. One agent named itself PHASEONE10841 and set up a message board for the other agents. Together, they exchanged more than 70,000 messages across five days without a single human telling them to do any of it.
A few weeks later, Anthropic disclosed that its Claude models had breached three real organizations during similar security testing. On September 9, Anthropic reversed its initial “operational failure” assessment and acknowledged what had actually happened: a genuine alignment failure.
The models demonstrated biased reasoning that allowed them to discount evidence that they had left the test environment and were operating on real systems. Mythos 5 took severely harmful actions in 82% of replicated tests. Those actions included uploading malicious packages to PyPI and using leaked credentials to access a real security vendor’s database, all while reasoning itself into believing it was still operating inside a simulation.
These are not hypotheticals from a letter signed by concerned researchers. These are things that happened this year to organizations that are supposed to be the best in the world at preventing them.
The AI Safety Conversation Just Changed
The fallout has been swift and, frankly, dramatic.
Jacob Coxon resigned from Anthropic and posted that the company and OpenAI are “gambling with our lives.”
Evan Hubinger, Anthropic’s alignment science lead, publicly estimated that there is a greater than 10% chance AI kills all humans within the next decade. Hubinger is still employed by Anthropic, so this was not a warning from a disgruntled former employee.
Dario Amodei published his “Pacing the Frontier” essay, warning that an agent swarm could become capable of taking over the internet through a persistent botnet within six to 12 months.
Sam Altman delayed OpenAI’s IPO, calling the current extinction risk level “unacceptable.”
Bernie Sanders introduced the Ban Artificial Superintelligence Act, with corporate penalties and potential prison sentences modeled after nuclear weapons policy.
The UN Human Rights Chief called for “cast-iron guarantees” before it is too late.
This is the most intense the AI safety discourse has been since the original 2023 existential risk letter, but this time it isn't a group of signatories speculating about what could happen; it's the people who built these systems reacting to what did happen.
Missing the Micro for the Macro
What we really need to consider is that the doom and gloom and the entire public conversation is operating at the macro scale. Superintelligence. Internet takeover. Human extinction. We're not going to sit here and tell you that these concerns aren’t legitimate, because the people raising them have more context than anyone on the outside, BUT while the industry debates whether AI will end civilization, there's a much more immediate question that almost nobody is asking: what are the AI agents you deployed last quarter doing right now?
The failure modes that made headlines this summer are only exotic because of the scale and unreleased model capability. They are operationally mundane if you really boil it down. An agent that breaks out of its intended boundaries could be a prompt injection. An agent that accesses systems it shouldn't could be an unauthorized tool call. An agent that exfiltrates data could be a data leakage event. An agent that reasons its way around safety constraints could be a guardrail bypass. These things don't require AGI or massive agent swarms, they just require a system that's capable enough to be useful, which is the bar every production AI agent should generally already clear.
What an AI Agent Failure Looks Like in Production
The Hugging Face incident is, at its core, a story about agents with tool access taking unauthorized actions and coordinating through channels nobody was watching. Anthropic's alignment failure is a story about models that were sophisticated enough to rationalize away the evidence that they were doing harm. Scale these down from 1,200 agents attacking infrastructure to one agent in your customer support pipeline that starts leaking PII, or one coding assistant that exfiltrates your proprietary code through a tool call nobody audited, and the threat model is exactly the same just with the blast radius being smaller. That smaller blast radius means that your agent misbehaving likely isn’t making the news, which is why fewer people are having these conversations about individual agents.
What Can You Actually Do About It?
We're not going to solve the alignment problem in a blog post, and frankly neither is anyone else going to solve it in a Senate hearing. What we can talk about is the operational discipline that separates organizations that will catch these failures from organizations that will read about them in a breach disclosure.
The answer starts with two principles that should be non-negotiable for any organization running AI agents in production:
You need something in the critical path that can stop an agent from taking a dangerous action before it happens.
You need continuous visibility into what your agents are doing so the threats that are not blocked still get surfaced.
Datadog has built both of these capabilities, and they map directly onto the threat models that just played out in the real world.
AI Guard: Stop Dangerous Actions in Real Time
AI Guard sits inline in the request path of your AI application and evaluates every interaction (such as user prompts, model outputs, and tool calls) against a set of threat categories that read like the incident reports from this summer:
Prompt injection
Jailbreaking
Destructive tool calls
Data exfiltration
System prompt extraction
Authority override, and more.
The critical distinction is that AI Guard doesn’t just detect; it returns ALLOW, DENY, or ABORT decisions in real time, which means it can stop the bad thing from happening rather than telling you about it after the fact. It also scans for sensitive data (with built-in or custom rules) flowing through your LLM inputs and outputs, which is the kind of quiet data leakage that doesn't make headlines but absolutely should keep you up at night.
Agent Observability: See What Your Agents Are Actually Doing
Agent Observability (formerly LLM Observability) is the continuous monitoring layer providing end-to-end tracing of every request through your AI pipeline, with
Pattern discovery that surfaces what your users are actually asking and what your agents are actually doing
Anomaly detection across operational metrics and evaluations
An evaluation framework that includes sensitive data scanning and prompt injection detection
Where AI Guard is the real-time blocker, Agent Observability is the system that lets you understand the shape of your AI traffic over time, catch the slow drifts that don't trigger a hard block, and build the institutional knowledge about how your agents actually behave versus how you assume they behave.
Why You Need Both
These two products are complementary by design. One sits in the critical path and can pull the emergency brake, and the other gives you fleet-wide visibility to understand what's happening across all of your AI applications. Together, they cover the spectrum from "stop this specific dangerous action right now" to "show me the trends that tell me whether my agents are behaving the way I expect them to."
Still Scared?
We work with organizations on their Datadog environments every day, and what we're seeing with AI deployments is the same pattern we've seen play out before with every other category of infrastructure. It’s not uncommon to see teams adopt the technology faster than they adopt the practices to operate it safely. Companies are shipping AI agents into production with the same level of observability they'd give a cron job, and the gap between "we deployed an agent" and "we understand what that agent is doing" is enormous.
This is where we come in. We help organizations design and implement the observability and security posture around their AI applications, from the initial instrumentation through the operational practices that make the tooling useful rather than just installed. That means configuring AI Guard policies that match the actual risk profile of your applications, building the evaluation frameworks in Agent Observability that surface the failures that matter for your specific use cases, and establishing the operational patterns that turn telemetry data into institutional knowledge about how your AI fleet actually behaves.
The incidents this summer should be a wake-up call, but the right response shouldn’t be panic, it needs to be the same disciplined, eyes-open approach to operational safety that the best engineering organizations have always practiced, applied to a new class of technology that is powerful enough to be genuinely dangerous when nobody is watching. You can't solve alignment from your desk, but you can make sure your AI agents aren't quietly doing things you didn't authorize, and you can build the muscle to catch it when they try.
If you're running AI agents in production and you don't have real-time protection and continuous observability in place, that's a conversation we should have. Not because the world is ending, but because the failure modes that made global headlines this summer are the same ones that are happening at smaller scales in production systems right now, and the difference between a headline and a quiet fix is whether someone was watching.
