Client autonomy without guardrails is an SRE nightmare
Joao Freitas is Managing Director and Vice President of Artificial Intelligence and Automation Engineering at PagerDuty
As the use of AI in large organizations continues to evolve, leaders are increasingly seeking the next development that will drive significant ROI. The latest wave of this ongoing trend is the adoption of artificial intelligence agents. However, as with any new technology, organizations must ensure they adopt AI agents in a responsible manner that allows them to facilitate speed and security.
More than half of organizations have already deployed AI agents to some degree, and more organizations are expected to follow suit in the next two years. But many early adopters are now reevaluating their approach. Four in 10 technology leaders regret not creating A stronger foundation for governance From the beginning, suggesting that they have embraced AI quickly, but with margin for improving policies, rules and best practices designed to ensure the responsible, ethical and legal development and use of AI.
As the adoption of artificial intelligence acceleratesOrganizations must find the right balance between their exposure risks and implement guardrails to ensure the safe use of AI.
Where do AI agents create potential risks?
There are three main areas to consider for safer AI adoption.
The first is Shadow AIwhen employees use unauthorized AI tools without explicit permission, bypassing approved tools and processes. IT must create the necessary processes for experimentation and innovation to deliver more efficient ways of working with AI. While shadow AI has been around for as long as the AI tools themselves have been around, the autonomy of the AI agent makes it easier for unauthorized tools to operate outside IT, potentially creating new security risks.
Second, organizations must close gaps in AI ownership and accountability to prepare for incidents or processes that go wrong. The power of AI agents lies in their autonomy. However, if agents behave in unexpected ways, teams must be able to determine who is responsible for addressing any issues.
The third risk arises when there is a lack of explanation for the actions taken by AI agents. AI agents are goal-orientedBut how they achieve their goals may be unclear. AI agents must have an explainable logic behind their actions so that engineers can track actions that might cause problems in existing systems and undo them if necessary.
Although none of these risks should delay adoption, they will help organizations better ensure their security.
The Three Guidelines for Responsible AI Agent Certification
Once organizations identify the risks that AI agents can pose, they must implement guidelines and guardrails to ensure safe use. By following these three steps, organizations can reduce these risks.
1: Make human oversight the default
AI continues to develop at a rapid pace. However, we still need human oversight when AI agents are given the ability to act, make decisions, and pursue a goal that may impact key systems. A human should be aware of the loop by default, especially for business-critical use cases and systems. Teams using AI must understand what actions they might take and where they might need to intervene. Start conservatively, and over time, increase the level of agency given to AI agents.
At the same time, operations teams, engineers, and security professionals must understand the role they play in overseeing the workflow of AI agents. Each agent must be assigned a specific human owner for clearly defined oversight and accountability. Organizations should also allow any human to report or override an AI agent’s behavior when an action has a negative outcome.
When considering tasks for AI agents, organizations must understand that while traditional automation is good at handling repetitive, rule-based processes with structured data inputs, AI agents can handle more complex tasks and adapt to new information in a more autonomous way. This makes it an attractive solution for all types of tasks. But as AI agents are deployed, organizations must control the actions the agents can take, especially in the early stages of a project. Therefore, teams working with AI agents must have approval pathways in place for high-impact actions to ensure the agent’s scope does not expand beyond anticipated use cases, reducing risk to the broader system.
2: Bread in security
Introduction of new tools should not expose the system to new security risks.
Organizations should consider proxy platforms that comply with high security standards and are validated by enterprise-level certifications such as SOC2, FedRAMP, or equivalent. Furthermore, AI agents should not be allowed free rein to control enterprise systems. At a minimum, the permissions and security scope of the AI agent should be aligned with the owner’s scope, and any tools added to the agent should not allow extended permissions. Limiting the AI agent’s access to the system based on its role will also ensure a smooth deployment process. Keeping complete records of every action an AI agent takes can help engineers understand what happened in the event of an incident and track the problem.
3: Make the output interpretable
Using AI in an organization should never be a black box. The reason behind an action must be explained so that any engineer trying to access it can understand the context the agent used in making the decision and access the implications that led to those actions.
IThe nputs and outputs of each action must be recorded and accessible. This will help organizations create a consistent overview of the logic behind the AI agent’s actions, providing significant value if anything goes wrong.
Security underscores the success of AI agents
AI agents provide a huge opportunity for organizations to accelerate and improve their existing processes. However, if they do not prioritize security and strong governance, they may expose themselves to new risks.
As AI agents become more common, organizations must ensure they have systems in place to measure how they are performing and the ability to take action when problems occur.
Read more from our website Guest writers. Or consider submitting a post of your own! See our Guidelines here.
