Feature
Guide
Is AI Making Your Organization More Productive or More Dependent?
Key Takeaways
- AI can automate decisions, but accountability always remains with the organization.
- Clear ownership across IT, business, legal and leadership is essential to managing AI risk.
- Agentic AI requires continuous, real-time governance rather than periodic oversight.
- Human oversight remains the final safeguard for AI-driven decisions and outcomes.
South Africa’s government recently withdrew a proposed national AI policy after concerns emerged that portions of the document contained fictitious or potentially AI-generated sources, resulting in the suspension of two officials.
It's a headline we've heard before — and likely one we'll continue to confront as AI moves further into operational workflows.
Yet when AI produces inaccurate information or contributes to a costly business decision, responsibility doesn't remain with the technology itself. It lands somewhere inside the organization. The question is where?
Who owns the consequences when AI systems make mistakes?
Who Actually Owns AI Risk?
“Accountability rests with the organization deploying the AI system. AI can’t own risk."
- Kathy Lange
Research Director, AI, Data and Automation Software, IDC
One of the most persistent misconceptions surrounding enterprise AI is that responsibility can somehow be delegated alongside automation. Kathy Lange, research director for AI, data and automation software at IDC, argues the opposite is true. “Accountability rests with the organization deploying the AI system. AI can’t own risk."
It's the organization that chooses the models, determines how those systems integrate into business processes and decides which decisions AI will influence.
Those choices create ownership regardless of whether an issue originates from flawed data, a hallucinated response or a poorly designed workflow. The challenge is that accountability often becomes muddled once AI enters production.
According to Lange, IT teams frequently end up carrying the burden because they implemented the technology, even when the underlying business decisions are owned elsewhere.
In practice, responsibility is distributed across multiple groups:
- IT teams deploy, maintain and secure AI systems
- Business units determine how AI outputs are used in operational decisions
- Legal and compliance teams oversee regulatory obligations
- Risk management teams evaluate potential exposure
- Executives ultimately own financial, operational and reputational outcomes
The more AI becomes embedded across the enterprise, the harder it becomes to determine where accountability begins and ends.
Related Article: Governance That Enables Iteration: Operating Models for Enterprise AI at Scale
Why AI Governance Still Leaves Accountability Gaps
Many organizations have already established AI governance committees, responsible AI frameworks and internal policies, but accountability gaps continue to emerge.
“The problem is that there are a lot of people involved in AI governance, but no one person is typically ‘in charge,’” Lange said.
That fragmentation becomes especially problematic when AI systems cross organizational boundaries. A customer-facing AI application, for example, may involve technology teams, legal departments, compliance specialists, procurement teams, security professionals and business stakeholders simultaneously.
Mark Moccia, vice president and research director at Forrester, said he believes organizations should focus less on governance structures in the abstract and more on defining ownership around specific decisions.
“The responsibility falls in two areas; the governance control owner of that decision and the owner of that agent build,” he explained. “An agent without a governance control is a bad design.”
Organizations making progress tend to have several characteristics in common:
- Clear decision rights defining ownership of AI-enabled processes
- Formal escalation procedures when issues occur
- Documented controls tied to regulatory requirements
- Regular testing and evidence collection to validate compliance
- Defined accountability for both the technology and the business outcome
Without those structures, AI failures can quickly turn into organizational disputes about who should have been responsible in the first place.
Agentic AI Makes Accountability More Complex
The accountability challenge becomes significantly more complicated as organizations adopt agentic AI systems.
Traditional generative AI tools typically provide information, draft content or generate recommendations, but agentic systems go further. They can initiate actions, execute workflows and interact with business systems with increasing levels of autonomy, and this changes the nature of oversight.
“Static governance doesn’t work for agentic systems,” said Lange.
Many existing governance models were designed around periodic reviews, approval checkpoints and retrospective audits. Agentic AI decisions may be made in real time, often across multiple systems and business processes.
Organizations need governance capabilities that are built directly into execution rather than layered on afterward, according to Lange. These include:
- Dynamic, risk-based oversight
- Continuous monitoring of AI activity
- Strong identity and access controls
- Exception-based triggers for human intervention
- Kill switches and rollback mechanisms when risks emerge
Humans Still Own the Final Decision
“Humans are still your most important asset."
- Mark Moccia
VP & Research Director, Forrester
For all the discussion about autonomous systems, experts continue to emphasize the importance of human accountability. As Moccia put it, “Humans are still your most important asset."
While organizations often focus on how AI can reduce manual effort or accelerate decision-making, far less attention is paid to preserving ownership of outcomes.
Human-in-the-loop controls are frequently described as a safety mechanism, but they are equally important as an accountability mechanism. Someone must approve recommendations, authorize deployments and decide how much autonomy an AI system receives. That also means someone must determine what happens when things go wrong.
The organizations closing accountability gaps are increasingly formalizing those responsibilities rather than assuming they will emerge naturally.
As Lange noted, many enterprises are moving toward centralized AI leadership, formal inventories of AI systems, automated governance platforms and risk-tiering processes that assign ownership before deployment occurs.
“Enterprises that are closing accountability gaps are the ones that have moved governance from a committee activity to a process activity,” she said.
Related Article: How to Form an AI Council: Lessons From Those Who've Done It Right
Accountability Must Be Operational — Not Aspirational
The South Africa episode likely offers a preview of a broader challenge that enterprises will face as AI becomes more deeply integrated into business operations.
Organizations can debate model performance, explainability, bias and regulation — important conversations — but the practical reality is that every AI deployment eventually becomes a question of ownership.
When an AI-generated recommendation influences a hiring decision, a procurement approval, a customer interaction or a regulatory filing, responsibility does not disappear into the technology stack. It remains with the people and organizations that chose to deploy the system in the first place.
“Governance can’t just be aspirational,” said Lange. “It needs to be operational.”