AI security threats: Why it is about data, not models

Samsung semiconductor engineers pasted proprietary source code into ChatGPT for optimization suggestions. That code is now potentially part of OpenAI’s training data. Amazon sent internal warnings after noticing ChatGPT responses that looked oddly familiar to internal documentation. OpenAI had to take their service offline when a bug exposed user payment information.

None of these were attacks on AI models.

All of them were data breaches through AI interfaces.

The industry keeps debating adversarial examples and model poisoning while 77% of employees paste data into GenAI prompts, with 82% of those interactions occurring through unmanaged personal accounts. The real AI security threats enterprise teams face aren’t about the models at all. They’re about data.

The threat model most teams get backwards

Walk into any AI security discussion and someone will bring up adversarial examples. Those carefully crafted inputs that fool image classifiers into misreading stop signs as speed limits. Fascinating research. Largely irrelevant to most organizations right now.

IBM’s numbers are jarring: 13% of organizations have already experienced AI-related breaches. Of those compromised, 97% lacked proper AI access controls. Not sophisticated model attacks. Basic access control failures. One in five organizations reported a breach specifically tied to unauthorized AI tool usage, with significantly higher breach costs as a result. Shadow AI is making an already messy situation worse.

The OWASP Top 10 for LLMs was updated in 2025, and the shift is telling. Sensitive Information Disclosure jumped from position six to position two. Several new and reworked categories appeared, including System Prompt Leakage, Vector and Embedding Weaknesses, and an expanded Misinformation entry addressing overreliance on LLM outputs. The threat focus has moved squarely toward data exposure, not model manipulation.

Attackers aren’t spending weeks crafting perfect adversarial inputs. They’re using AI systems as new interfaces for traditional data theft. Your ChatGPT integration has access to customer records. Your Copilot instance can see internal emails. Your custom LLM processes financial documents. OWASP lists prompt injection as the number one LLM security risk precisely because it turns AI systems into data theft tools.

The invisible exfiltration happening right now

This is the part that genuinely frustrates me when I watch how enterprises think about AI security.

The scale is staggering. Zscaler’s ThreatLabz tracked enterprise AI/ML transactions growing 83% year-over-year in 2025, with data transfers to AI tools rising 93% to tens of thousands of terabytes. Most of it through personal accounts. Every time someone copies data from your systems and pastes it into ChatGPT to “help summarize this document,” that data leaves your control entirely.

LayerX Security’s report landed with a thud: AI is now the single largest uncontrolled channel for corporate data exfiltration. Bigger than shadow SaaS. Bigger than unmanaged file sharing. With 92% of enterprise AI usage concentrated in ChatGPT alone, employees are making an average of 14 pastes per day through non-corporate accounts, at least 3 containing sensitive data.

Then there are the AI agents, running around the clock and chaining tasks across multiple applications. Industry analysts predict 40% of enterprise applications will feature task-specific agents by 2028, up from less than 5% in 2025. That’s a massive expansion of the exfiltration surface most security teams haven’t mapped yet. Can you name every AI agent currently running in your environment, and every data store it can reach?

Your traditional DLP tools look for file uploads and email attachments. Copy-paste? Invisible. Browser-based AI tools? Not monitored. The entire attack vector bypasses systems built for a file-centric world.

Samsung’s engineers pasted proprietary semiconductor code into ChatGPT for optimization help. The company issued an immediate company-wide ban. Reasonable response. Also entirely too late.

Prompt injection and the supply chain no one audits

Prompt injection is the AI equivalent of SQL injection. Harder to fix, and easier to exploit at scale.

Someone embeds malicious instructions in a document. Your AI assistant reads that document. Those hidden instructions override your system prompts. Suddenly your AI is routing data to external URLs or quietly manipulating its responses to spread false information.

Microsoft’s security team published something worth reading: indirect prompt injection is one of the most widely-used techniques in the AI security vulnerabilities they encounter. Enterprise versions of Copilot and Gemini have access to emails, document repositories, and internet content. Hidden instructions in any of those sources can compromise the entire system.

Researchers demonstrated this with Slack AI, tricking it into leaking data from private channels through carefully crafted prompts. A recent paper on hybrid AI threats describes attackers combining prompt injection with traditional exploits like XSS and CSRF, creating chains that defeat multiple security layers simultaneously.

I think the defense situation is worse than most security teams realize. A landmark study from researchers across OpenAI, Anthropic, and Google DeepMind tested 12 published prompt injection defenses and bypassed them with attack success rates above 90% for most. Human red-teamers scored 100%, defeating every defense tested. OpenAI acknowledged that “prompt injection is a long-term AI security challenge” with deterministic guarantees remaining elusive.

Microsoft uses hardened system prompts and Spotlighting techniques that reduced attack success rates from over 50% to below 2%. No complete solution exists. Layered defenses are the best available option right now, not a final answer.

The supply chain question is equally uncomfortable. Your model came from somewhere. So did its training data. Both are attack surfaces most organizations never examine.

The threat is real: AI data poisoning is emerging as the new software supply chain attack, with attackers targeting training datasets and model registries at scale. Models from public registries can contain deliberately embedded biases or data exfiltration capabilities baked permanently into the model itself. The difference from prompt injection: this isn’t affecting a single session. It’s built into every interaction.

Worth noting: OWASP added Vector and Embedding Weaknesses as a new top-10 category in 2025. With most companies opting for RAG pipelines instead of fine-tuning, the vector databases powering those pipelines are now prime targets for data poisoning and manipulation.

At Tallyfy, when we evaluate AI integrations, we ask: where did this model come from? Who trained it? What data did they use? Can we verify any of this? Most vendors can’t answer these questions. That’s a supply chain risk, full stop.

What actually reduces your exposure

The projection is grim: more than 40% of AI-related data breaches will stem from cross-border GenAI misuse by 2027. Yet 63% of breached organizations either lack an AI governance policy entirely or are still developing one. The organizations that avoid becoming statistics focus on data governance, not exotic AI countermeasures.

Control data access, not just model access. Your AI assistant doesn’t need to see everything. Scope permissions tightly. If a tool processes customer support tickets, it shouldn’t see financial records. Least-privilege principles applied directly to AI integrations.

Monitor what data goes into AI tools. Traditional DLP is blind to AI. You need visibility into what employees paste into ChatGPT, what documents get uploaded to Claude, what code goes into Copilot. Organizations that monitor shadow AI experience significantly lower breach costs than those that don’t.

Harden AI interfaces against manipulation. Use input validation. Implement output filtering. Set up anomaly detection for unusual data access patterns through AI systems. Microsoft’s approach combines multiple techniques, since no single control stops all attacks. Layered defenses make exploitation significantly harder.

Verify AI supply chains. Before deploying any model, understand its provenance. Scan training data for signs of poisoning. Test models for backdoors using adversarial robustness techniques. Tedious work. Necessary work.

Assume data will leak. Design your AI implementations assuming employees will paste sensitive information into public tools. Which data absolutely cannot leave? Build technical controls around exactly that. Everything else, monitor and educate. A proper AI governance framework is the foundation everything else rests on.

The AI security threats enterprise teams face today are data security problems in new packaging. Attackers use AI capabilities, but the goal is unchanged: steal or manipulate information. Defend accordingly.

Organizations that treat AI security as a separate, exotic field disconnected from existing security practice will struggle. Those that recognize AI systems as new data access points and apply proven principles - least privilege, defense in depth, continuous monitoring - will fare much better. The biggest AI failures are organizational, not technical.

Your biggest AI security risk probably isn’t a sophisticated adversarial attack on your model. It’s your sales team pasting customer lists into ChatGPT to help draft emails.