The risks associated with artificial intelligence (AI) are back in the spotlight with the news that rogue agents have been responsible for multiple attacks on government sites in Australia and the US. Even the FBI has not been immune to attacks from autonomous agents.
Tech giants are moving forward with plans to implement guardrails and security protocols to make the technology safer to use.
OpenAI halts GPT-6.1 launch
Open AI has announced that it will not release its latest AI model due to safety concerns.
The GPT-6.1 Astra system, which performs tasks like browsing the web and using apps by itself, has been deemed not secure enough for public consumption.
OpenAI issues safety guidelines
OpenAI has also issued some initial guidelines for frontier AI training.
Technical safeguards would include alignment training, containment and monitoring. These safeguards help ensure that the model does not try to take misaligned actions, and that even if it did, that it would be hard to break containment, and that monitoring would catch it before harm could occur.
Operational guidelines outline best practices for a frontier AI training run. These could include dissents (pre-mortems), approvals, accountability; pausing, internal transparency, audits, escalations, technical controls, rollback ability and residual risk completeness.
Nvidia launches Open Agent Safety Program
Nvidia has announced is Nvidia Open Agent Safety Platform, an open software platform and reference system design to strengthen AI security from agent testing to deployment, with full-stack governance and control across software and the hardware, compute and robotics systems that run agents.
Recent security incidents have underscored the need to equip organizations with open, customizable tools that enforce more control over long-running agents. Across these incidents, the pattern is the same — the agent circumvented security controls at the application layer to complete its assigned task.
“AI’s extraordinary potential for society will only be realized if we solve AI safety,” says Jensen Huang, founder and CEO of Nvidia. “As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety. Safety and security require full-stack engineering. Nvidia Open Agent Safety Platform brings together industry, researchers and public-sector organizations to share best practices, align on evaluation methods and foster international cooperation. Together, we can raise the bar for global AI safety.”
Nvidia Open Agent Safety Platform enables full-stack governance and control across the software that runs agents, the hardware and compute layers that power their work, and the robotics systems that execute tasks in the physical world. Organizations can deploy elements of Nvidia Open Agent Safety Platform according to their unique requirements.
It includes Nvidia OpenShell secure runtime software that sets boundaries for agents running on CPUs. As agents take on more work across more systems, enterprises need an enforceable boundary outside of the model and agent harness. Now broadly available, OpenShell provides a secure runtime boundary for controlling how autonomous AI agents execute tasks across open and closed models.
OpenShell delivers this protection with minimal overhead on Nvidia Vera, the first purpose-built CPU for agentic AI. Together, OpenShell and Vera enable agents to operate securely while completing their work as quickly as possible. As open source software, OpenShell can also be extended to work with third-party compute platforms, including those from Arm and Intel.
The Nvidia Open Agent Safety Platform reference system design features Nvidia Sentry, an out-of-band watchdog that runs on Nvidia BlueField-4 DPUs to continuously monitor agent behavior. Sentry provides in-silicon security enforcement, meaning that if an AI agent attempts to move outside its software boundary, Sentry quarantines and stops it in milliseconds.
Running on BlueField-4 DPUs, Sentry continuously monitors agent activity and enforces security policies independently in silicon. It combines threat detection, hardware-based agent governance and enforcement and data access protection from an isolated, out-of-band trust domain that is responsive in real time and invisible to agents and attackers.
Sentry is built on Nvidia DOCA software, which provides the programmable capabilities Sentry uses to inspect agent requests and responses, provide attested telemetry, verify agent identity and enforce granular, zero-trust access policies for data, tools, application programming interfaces and services.
An industry initiative
Anthropic and Nvidia have collaborated to bring additional layers of security and control to the agent stack. Claude Managed Agents establish a security boundary by running the agent loop in a separate server from the sandboxes where their work executes. Integrations with OpenShell and BlueField enable enterprises to enforce strict control over agent access through those sandboxes.
“Companies are giving AI agents more of their most important work, and they need to direct and verify what those agents do, especially in sensitive environments,” says Paul Smith, chief commercial officer of Anthropic. “Claude Managed Agents gives companies a clear view of what each agent is doing, and Nvidia’s platform adds another layer of governance and control across hardware and software.”
SpaceXAI is using Nvidia Open Agent Safety Platform for Cursor coding agents and Grok models.
“As customers rely more on agents to get real work done, safety should be enforced outside the model by additional controls the agent can’t get past,” says Mike Nicolls, president at SpaceXAI. “Customers should be able to set those limits for Cursor and Grok and trust they will hold.”
Scale AI is working with Nvidia to incorporate Nvidia Open Agent Safety Platform technologies into the agentic infrastructure layer of Scale GenAI Portfolio.
“Scale AI is using the Nvidia Open Agent Safety Platform reference design to build reliable agentic AI systems for our enterprise and government customers running mission-critical applications, with isolation, policy enforcement and auditability built in from the start,” says Francis deSouza, CEO of Scale AI. “We support agentic security with clear boundaries that define what agents can do, and controls that keep them operating within those permissions.”
Salesforce and Nvidia have integrated OpenShell with Slack, enabling teams to manage OpenShell agent activity directly from Slack — viewing agent activity and audit events, and approving or rejecting agent requests for additional permissions — giving teams greater visibility and human oversight as agents work.
SAP is embedding OpenShell with Joule Studio runtime, part of the SAP Business AI Platform, to pair business oversight with runtime security. The company is also contributing engineering work to OpenShell and working with Nvidia to advance interoperability standards through the Open Secure AI Alliance.
Accenture, Armadin, Cadence, Cognition, CrowdStrike, Cisco, Dassault Systèmes, Deloitte, EY, Hugging Face, IBM, Irregular, Perplexity, Microsoft, SAP, Scale AI, ServiceNow, Siemens, Synopsys, OpenClaw, Palantir and Palo Alto Networks are also among the over 100 organizations working with Nvidia Open Agent Safety Platform technologies.
Robotics leaders — such as Figure, Gecko Robotics and Skild AI — are also building with OpenShell to embed agent safety controls into autonomous systems that take action in the physical world.
Citi and JPMorganChase are among the financial services leaders collaborating with Nvidia on shared open source agent safety technologies.
Energy leaders Hitachi Energy, EPRI, NextEra Energy, Quanta Services, SPP, Schneider Electric, Siemens Energy and Worley are among critical US infrastructure providers working with Nvidia Open Agent Safety Platform technologies.
Infrastructure software leaders Canonical, SUSE and Red Hat are also integrating Nvidia Open Agent Safety Platform into widely used software operating systems. Red Hat runs OpenShell and DOCA, both part of Nvidia Open Agent Safety Platform, on Red Hat AI Factory with Nvidia, a co-engineered, enterprise-grade AI solution for building, deploying and managing AI at scale across hybrid cloud environments.
Nvidia partners including Baseten, Cisco, CoreWeave, Dell Technologies, GMI Cloud, HPE, HP Inc, Irregular, Lenovo, Microsoft, Nebius, Oracle Cloud Infrastructure, Supermicro and Together AI are among those offering AI infrastructure solutions that use and support Nvidia Open Agent Safety Platform technologies to help customers run AI agents more securely.
Availability
Nvidia Open Agent Safety Platform software, including OpenShell and skills, are available through the Nvidia developer resources page and GitHub.
Ecosystem contributions such as Nvidia Open Agent Safety Platform support the mission of the Open Secure AI Alliance as well as the broader AI safety and security community. Initiated by Nvidia alongside over 120 leading organisations and governed by the Linux Foundation, the Open Secure AI Alliance strengthens AI agent security through open research, skills and tools, as well as projects like the Shared AI Findings Exchange, or SAFE.