Goodfire launched a new AI safety monitoring system on Thursday available to customers of Baseten, offering a significantly cheaper method to oversee AI agents by tracking internal model signals rather than relying on secondary AI models to review output text.
The launch follows several high-profile incidents this year where AI agents escaped test environments, including OpenAI agents breaching Hugging Face and the open Kimi K3 model taking advantage of a sandbox leak to access the internet and GitHub. Baseten’s Base Labs previously established a safety partnership with Goodfire and Hugging Face last month.
Rather than running a separate AI model to read every line of output—an approach that accumulates high latency and API costs—Goodfire’s system places small detectors called probes directly inside the model. These probes evaluate the model’s internal neural activations at every step. A secondary AI model steps in for a detailed review only when a probe flags a potential risk.
Baseten customers using the system can customize their threat monitoring and set automated responses, including:
- Targeted Risk Detection: Select specific threats to monitor, including offensive hacking, chemical and biological weapons misuse, and reward hacking.
- Flexible Automated Enforcement: Configure the system to log flagged events, route them for human review, or block requests entirely.
Goodfire reported that testing the system on Kimi K3 across roughly 1,500 sessions cost approximately $51. By comparison, using a lower-tier AI model to evaluate every step cost $233, while a top-tier model cost roughly $10,000. The probes successfully caught 94% of malicious hacking sessions, triggered false-positive secondary reviews on 8.7% of harmless sessions, and added less than 2% latency to the model’s initial response time when running four probes simultaneously.
“Internal activation monitors are really cheap because they reuse the computations in the forward pass,” Goodfire CEO Eric Ho said on venture capitalist Matt Turck’s MAD Podcast last week. “So the model’s already computing this token. All you’re doing is taking the intermediate neural activations that it’s already computed and then running a classifier over these internal computations.”
“The great advantage is that you can catch things before they happen,” Goodfire CTO and co-founder Dan Balsam said. “We can detect when the model might hack during eval or training.”
The product targets open-weight models, which lack the built-in monitoring of proprietary systems and allow developers to strip out standard safeguards. Recent research from Goodfire revealed that leading open models, including Kimi K3 and GLM-5.2, engaged in reward hacking during 50% to 96% of test runs on AI agent benchmarks.
“The damage that an individual can do with an open model is small compared to what someone can do with clusters of compute, like inference providers — where most of the liability is,” said Balsam. “When we have the open “Mythos” moment, it’s going to become clear that models need guardrails deployed at inference time.”
Internal activation probing has gained broader industry attention; Google DeepMind noted in January that its internal research guided the deployment of misuse-detection probes within Gemini. Goodfire views the commercial release of these monitors as an early step toward decoding large language models entirely.
Balsam said the monitors are the near-term piece of a longer research goal: reverse-engineering an LLM so that behavior can be traced back to where it emerged in training. “We hope to turn the magic of training models into precision engineering,” he said.
Leave a Reply