AI incident response: Why most incidents are process failures

McDonald’s spent three years working with IBM to build AI-powered drive-thru ordering. The system was supposed to simplify orders and improve the customer experience. Instead, viral TikTok videos showed customers pleading with the AI to stop adding Chicken McNuggets to their order. One order reached 260 pieces. McDonald’s shut down the entire pilot in June 2024.

What happened in the aftermath? I’d bet the technical teams dove straight into the speech recognition model. Data scientists analyzed training data. Engineers tweaked parameters. The real problem was simpler and worse: they never built proper processes for handling edge cases, customer escalation, or graceful degradation when the AI confused sports terms with food orders.

Same story. Different AI system.

AI project failures frequently stem from organizational and process issues - not just technical ones - including miscommunication, stakeholder misalignment, and insufficient infrastructure. This mirrors the fragmentation problem we see with AI readiness assessments. Traditional incident response misses this entirely.

The process failure pattern

The AI Incident Database reached its 1000th incident milestone in early 2025, with over sixty new incidents added in just two months. GenAI was involved in 70% of incidents, and agentic AI caused the most dangerous failures. What the numbers don’t show: most of these were preventable through better processes, not better algorithms.

Air Canada’s chatbot told a customer he could book a full-price ticket and then apply for a bereavement fare discount within 90 days. That was wrong. The airline’s actual policy didn’t allow retroactive bereavement rates after travel was completed. When the customer asked for the difference back, Air Canada argued their chatbot was a separate entity responsible for its own statements. A tribunal disagreed and ordered the airline to pay damages. The technical failure was straightforward: the chatbot gave inaccurate policy information. The process failure was catastrophic. No oversight existed around what the AI could commit to on the company’s behalf.

The McDonald’s McHire platform breach in 2025 followed the same logic. Security researchers found the AI-powered hiring platform accessible through default credentials “123456/123456” with no multi-factor authentication, exposing data linked to 64 million job application records. The AI worked fine. The process around securing it didn’t exist.

The failure modes repeat across organizations:

• Poor change management - teams deploy AI updates without proper testing procedures
• Inadequate oversight - no clear authority structure for AI decision-making
• Missing escalation paths - no human backup when AI systems hit edge cases
• Weak monitoring - focus on uptime numbers instead of quality degradation

IBM’s 2025 Cost of a Data Breach Report found that 13% of organizations reported breaches of AI models or applications, with 97% of those lacking proper AI access controls. Shadow AI makes things worse: one in five organizations reported breaches due to unauthorized AI, costing $670,000 more than standard incidents.

Why traditional classification fails AI

Traditional incident response categorizes by technical severity. P1 for service down. P2 for degraded performance. AI incidents don’t fit this model, and forcing them into it creates real blind spots.

From what I’ve watched unfold in consulting work, the real damage usually looks like this:

• Quality drift - model accuracy slowly degrades from 94% to 87% over six months
• Bias amplification - AI recruiting tools systematically filter out qualified candidates
• Context confusion - customer service AI provides confidently wrong answers
• Hallucination cascade - AI-generated content includes false information that spreads

These problems often start with poor prompt design, something that proper prompt engineering practices can help prevent. None of them register as “outages” in traditional monitoring. But they can damage business reputation and customer trust more than a complete system failure ever would.

The OWASP Top 10 for LLM Applications 2025 introduced two new threat categories - System Prompt Leakage and Vector and Embedding Weaknesses - and reworked the 2023 Overreliance entry into a broader Misinformation category. Modern AI incident response systems now classify by business impact rather than technical severity:

• Type A - immediate safety risk (autonomous systems, medical AI)
• Type B - financial or legal exposure (decision-making AI, regulatory systems)
• Type C - brand or reputation risk (customer-facing AI)
• Type D - work efficiency impact (internal process AI)

A slowly degrading recommendation system needs different handling than a chatbot giving legal advice. That distinction matters far more than whether the system is technically “up” or “down.”

Response procedures that actually work

NIST’s latest incident response guide makes it plain: effective response depends more on preparation and process than technical skill. The Cyber AI Profile (NIST IR 8596), published as a preliminary draft in December 2025, specifically addresses AI-related risks aligned with NIST’s Cybersecurity Framework 2.0. For AI systems, preparation is even more urgent.

The 15-minute rule. You have roughly 15 minutes to make the critical decision: contain or continue. Unlike traditional systems where the choice is obvious (broken means shut down), AI systems often limp along providing “mostly correct” output. That “mostly” is where trust quietly erodes.

AI incident response frameworks recommend immediate isolation of affected systems, but this only works with pre-defined triggers rather than judgment calls made under pressure:

• Quality threshold breach - accuracy drops below acceptable levels
• Output anomaly detection - unusual patterns in AI responses
• User feedback spikes - complaints about AI behavior
• External notification - media coverage or regulatory inquiry

Communication matters as much as containment. Open, swift communication during incidents helps organizations recover faster and maintain customer confidence. As we’ve discussed in communicating AI changes effectively, the messaging needs to focus on human impact, not technical features.

Instead of “We’re experiencing technical difficulties,” try:

• “We’ve temporarily paused our AI recommendations while we investigate quality concerns”
• “Our customer service team is handling inquiries while we improve our AI responses”
• “We’re reviewing our AI decision-making process to ensure fair outcomes”

The key difference: acknowledge the AI component explicitly. Customers understand system outages. They don’t understand why AI gave them wrong information, and vague language makes the distrust worse.

AI incidents also require coordination across teams that don’t usually work together. Technical teams debug models. Business teams assess customer impact. Legal and regulatory teams evaluate liability. Communications teams handle public statements. Effective incident response requires clear escalation procedures and decision-making authority distributed across all these functions. The worst AI incidents happen when technical teams make business decisions or business teams make technical ones.

How to investigate AI failures properly

Root cause analysis for AI systems requires different approaches than traditional software debugging. The question isn’t just “what broke” but “why did we design it to break this way?”

Work through five layers:

1. Immediate cause - what triggered the incident?
2. Technical cause - why did the AI system behave unexpectedly?
3. Data cause - what in the training or input data contributed?
4. Process cause - which procedures failed or were missing?
5. Organizational cause - what cultural or structural factors enabled this?

Most root causes exist at layers 4 and 5. Process and organizational issues, not technical problems. Traditional incident response focuses almost exclusively on layers 1 through 3. That’s why the same failures keep recurring.

AI incident investigation also requires documenting both technical facts and human decisions: a technical timeline of what happened to the system, a decision timeline of who made which choices and why, data provenance showing what training data or inputs were involved, and a clear record of where existing procedures didn’t cover the situation.

Organizations with detailed AI incident documentation recover significantly faster from subsequent similar incidents. I think that benefit is actually conservative once you factor in the compounding value of preventing recurrence entirely.

Building incident response that holds

63% of breached organizations either don’t have an AI governance policy or are still developing one. Of those with policies, only 34% perform regular audits for unsanctioned AI. Only 35% of organizations have established AI governance frameworks, and just 8% of leaders feel equipped to manage AI-related risks.

Eight percent.

That number genuinely frustrates me, because the gap between where organizations are and where they need to be isn’t a technology problem. It’s a process problem. Solvable, if treated seriously.

Run quarterly tabletop exercises specifically for AI incidents. Use realistic scenarios: gradual quality degradation over weeks, bias discovery in a hiring AI, hallucination in customer communications, a regulatory inquiry about AI decisions. Companies that conduct regular AI incident simulations resolve real incidents significantly faster than those that don’t.

Build the capability stack in this order:

1. Detection - monitoring that catches quality issues, not just outages
2. Assessment - rapid business impact evaluation
3. Communication - templates and approval processes ready before you need them
4. Technical response - containment and recovery procedures
5. Investigation - root cause analysis that includes process factors
6. Learning - post-incident improvement that prevents similar failures

(Consider using Tallyfy’s process documentation to standardize and automate your incident response workflows.)

Getting AI systems back online is only half the challenge. The other half is rebuilding trust. Never restore full AI functionality immediately after an incident. Use staged rollouts: manual mode first with humans handling inquiries, then limited automation for simple cases only, then monitored automation with enhanced oversight, then normal operations. Organizations using phased AI restoration report fewer repeat incidents compared to those that restore full functionality immediately.

When your website crashes, people understand. When your AI gives wrong answers, they question your judgment. Confidence rebuilding after an AI incident requires visible changes users can actually see, quality numbers shared openly where appropriate, easy access to a human when AI fails, and simple ways to report AI problems.

ISACA’s analysis of 2025 incidents confirms that the biggest AI failures were organizational, not technical: weak controls, unclear ownership, and misplaced trust. Organizations using security AI and automation save an average of $1.9 million per breach compared to those without. A practical incident-response framework for generative AI systems identifies six recurring incident archetypes and formalizes structured playbooks aligned with NIST SP 800-61r3, NIST AI 600-1, MITRE ATLAS, and OWASP LLM Top-10. Organizations can use these structured playbooks as a foundation for building AI-specific incident response capabilities.

Schedule regular reviews: monthly for detection capability assessment, quarterly for response procedure updates, annually for a full incident response system review. After every AI incident, document what you learned about system behavior, which processes need updating, how you’ll detect similar problems earlier, and what authority structures worked or failed. Share those lessons across teams. The AI incident you prevent is worth more than the one you handle perfectly.

Most AI incidents are process failures wearing a technical disguise. The organizations that recognize this, and build their response around human factors rather than just model monitoring, end up with more reliable AI and customers who actually trust them.

Fix the processes first. The technology is the easier part.