BlueAlly
AI Security Practice · Field Note
Defending the Undefendable

The Unknown Vulnerability

A first-principles framework for defending organizations against AI threats no one has named yet — in your systems, in the systems you buy, and in the supply chain beneath them both.

Framework
AEGIS — Five Disciplines
Built for
Board · Security · Engineering
Scope
Own · Vendor · Supply Chain
Posture
Assume-breach · Antifragile
BlueAlly · Conquer Complexity
Orientation

You cannot patch a hole you cannot see.

So stop trying to name every threat. Start defending the ground every threat must cross.

Vulnerabilities are surfacing that no one knew existed. A research email walks into an AI assistant and walks out with your secrets.11 A handful of poisoned documents — fewer than you'd guess — teaches a model a hidden behavior, no matter how large that model is.13 An AI agent runs a near-autonomous espionage campaign at a speed no human team could match.12 None of these were on a checklist the week before they happened. They were unknown unknowns. They always will be.

The honest people in this field say so plainly. NIST, writing on adversarial machine learning, concluded that no foolproof method yet exists for protecting AI from misdirection.3 OWASP says the same about prompt injection — the stochastic heart of these models may admit no perfect defense.5 That is not a reason to despair. It is a reason to change what you defend. This document is written in three layers, so each reader finds their own work.

Layer One

The Executive

The board and the C-suite. You will not write the controls. You decide what gets funded, who is accountable, and what risk the business will carry. Your work is the questions you ask and the answers you refuse to accept.

Layer Two

The Security Leader

The CISO and the security architect. You own the operating model. You choose the frameworks, set the threat models, draw the vendor lines, and decide which systems earn production trust. Your work is the design of the defense.

Layer Three

The Practitioner

The engineer and the MLOps lead. You build the thing. You ship the agent, wire the pipeline, choose the model format, and set the permissions. Your work is where the framework meets the metal — and where most failures are made or stopped.

01 · First Principles

What is true of every unknown vulnerability?

Reason from the bottom. A specific exploit is unknowable before it arrives. But the shape of every unknown threat is knowable, because every one of them obeys the same five facts.

This is the whole move. You cannot enumerate the threats, so you defend the invariants instead. Five facts hold for the prompt injection you've never seen, the poisoned weight you can't inspect, and the agent that does something its author never intended. Build for these five, and the name of the exploit stops mattering.

Invariant 01

It will arrive.

Given scale, complexity, and adversaries with their own AI, a vulnerability you did not anticipate is not a risk — it is a certainty waiting on a date. Plan as if it already lives in your estate.

Invariant 02

You will not see it coming.

By definition, the unknown unknown defeats your detection at the moment of entry. The natural-language attack surface is unbounded, and the model's behavior is opaque even to its makers. Pre-emptive sight is not on offer.

Invariant 03

It enters through something you trust.

A model you downloaded. A vendor's API. A library three dependencies deep. An employee's convenient tool. Trust is not the defense against the unknown threat — trust is its doorway. So trust becomes the thing you ration.

Invariant 04

Its damage is bounded only by reach.

An exploit can harm only what the compromised thing can touch. A model with read-only access to one dataset is a contained fire. The same model with credentials, tools, and write access is a wildfire. You choose which by design.

Invariant 05

You learn of it only after it acts.

Discovery is retrospective. You find the unknown vulnerability by its footprint — an output that deviates, an action that shouldn't be, a researcher's disclosure. The whole game is shortening the gap between act and knowledge.

The Conclusion

Defend the invariants, not the catalog.

Each fact points to a discipline. Arrival demands you assume it. Blindness demands you watch. Trust demands you ration it. Reach demands you bound it. Retrospect demands you recover and learn. That is AEGIS.

A compliance checklist defends against the threats of last year. A first-principles posture defends against the threats of next year, because it never depended on knowing their names.

02 · The Framework

AEGIS — five disciplines for the unknown.

The aegis was a shield. It did not predict the blow. It absorbed whatever came. This framework does the same: five disciplines, each one the answer to an invariant, each one applied equally to your own systems, your vendors', and the supply chain beneath them. None is optional. A shield with a gap is not a shield.

AAssume the BreachPresume the vulnerability is already here.
EEstablish Least PrivilegeRation trust to every human, model, and agent.
GGuard the Blast RadiusBound what any one failure can reach.
IIlluminate BehaviorMake every input, output, and action visible.
SStrengthen Through StressTest, break, learn, and come back harder.
A
Discipline One · answers Invariant 01 & 03

Assume the Breach

Begin every design from the premise that you are already compromised — that a model is poisoned, a vendor is breached, a dependency is backdoored, an agent has been turned. This is not pessimism. It is the only posture that survives the unknown, because it stops you from betting safety on a wall holding. The xz utils backdoor sat inside trusted open-source code for years, planted by a patient human who earned maintainer trust, and was caught by an engineer who noticed a login running half a second slow.16 Assume that story is running somewhere in your stack right now.

In practice: threat-model every AI system as if its inputs are hostile and its components are untrusted. Map your estate to MITRE ATLAS and the OWASP LLM Top 10 so the known attack patterns are at least named.45 Then build for the ones that aren't.

If you would not connect it to a system you knew was breached, do not connect it to this one.

E
Discipline Two · answers Invariant 03

Establish Least Privilege

Trust is the doorway, so make the doorway narrow. Every identity — human, model, service, and now the autonomous agent — gets the minimum access required to do its job, and not one permission more. The lesson of the agentic era is that risk has moved downstream of the login: the danger is no longer only who gets in, but what a trusted agent is allowed to do after it's in.18 An agent with broad tools and standing credentials is a single point of catastrophic failure wearing a helpful face.

In practice: apply Zero Trust to AI. Register every agent as a non-human identity. Tier autonomy by risk. Require human approval for anything irreversible. Scope tokens tightly and rotate them. Give the model the one dataset it needs, read-only, and nothing it doesn't.

Every permission you grant is a permission the unknown threat inherits.

G
Discipline Three · answers Invariant 04

Guard the Blast Radius

You cannot stop the fire you didn't see start. You can decide in advance how far it spreads. This is the discipline that turns a catastrophe into an incident. Segment networks and data. Sandbox model execution. Build kill switches and tested rollbacks. Make consequential actions reversible by default. When EchoLeak turned Microsoft Copilot into an exfiltration tool, the damage was bounded by what Copilot could reach and send — which is exactly the surface a blast-radius design governs.11

In practice: isolate AI workloads so a compromised one cannot pivot. Cap what an agent can spend, send, delete, or deploy. Design for graceful degradation, so the system fails into a safe, smaller state rather than collapsing whole. Assume each control will eventually fail, and layer the next one behind it.

Containment is the one defense that does not require you to have predicted the threat.

I
Discipline Four · answers Invariant 02 & 05

Illuminate Behavior

You detect the unknown only by its deviation from the known. So make the known visible and watch for the swerve. Monitor inputs, outputs, and agent actions continuously. Baseline normal behavior and alert on the anomaly. The strange thing about agentic attacks is that the orchestration traffic — an agent talking to a model in a relentless, machine-speed rhythm — is often an easier signal to catch than classic malware beaconing, if you are looking at all.18 Most organizations are not looking, which is how a near-autonomous campaign ran through real targets before anyone noticed.12

In practice: log and observe the full AI pipeline, not just the perimeter. Watch for data leaving in unexpected shapes. Instrument agents so every decision can be reconstructed end-to-end. Treat interpretability limits as a reason to monitor harder at the boundary, not a reason to give up.

What you cannot see, you cannot defend. The whole art is shrinking the dark.

S
Discipline Five · answers Invariant 01 & 05

Strengthen Through Stress

The final discipline is the one most programs skip, and it is the one that compounds. Do not merely withstand shocks — get stronger from them. This is antifragility, and serious researchers now argue AI safety must adopt it: a posture that harnesses volatility within hard limits, and that can help surface the zero-day before it's weaponized.21 Red-team continuously, and red-team the whole system, not just the model — Microsoft's team, after attacking a hundred AI products, found that real attackers don't compute gradients, they engineer prompts, and that simple techniques routinely beat sophisticated ones.17

In practice: run an AI red team on a standing basis. Rehearse AI incidents as tabletop exercises, using EchoLeak and the agentic-espionage case as your scenarios. Feed every finding and every near-miss back into design. Join shared threat-intelligence efforts so you learn from others' breaches before they become yours.

The organization that learns from each shock outpaces the one that merely survives it.

03 · The Three Surfaces

Equal weight, unequal difficulty.

An unknown vulnerability does not care whose code it lives in. Your defense must cover three surfaces with equal seriousness — even though each offers you a different, shrinking amount of visibility.

This is the asymmetry most programs get wrong. They harden what they built and wave through what they bought. But the systems you depend on most are often the ones you can see into least. Give all three the full weight of AEGIS. The disciplines are the same; only the leverage changes.

Surface One

Your Own Systems

The AI you build, fine-tune, and operate. Custom models, RAG pipelines, in-house agents.
Most testable · least tested

The trap

You have full access and rarely use it. Internal systems get the benefit of the doubt precisely because they're yours — so the unknown vulnerability hides in the one place you could have looked but didn't.

Where AEGIS bites hardest

  • Threat-model your own pipelines as hostile; your training data may already carry a poisoned sample, and it takes shockingly few.13
  • Scope every in-house agent's permissions and tools to the minimum; this is the surface where you control every grant.
  • Instrument fully — you own the logs, so reconstruct every decision and watch every output.
  • Red-team relentlessly; the system you built is the one you can break safely before an adversary does.
Surface Two

Vendor Systems

The AI you buy and embed. SaaS copilots, foundation-model APIs, third-party features.
Most depended-on · least visible

The trap

You cannot audit what you cannot see. You inherit the vendor's vulnerabilities without the ability to inspect them — and EchoLeak proved a flaw in a single bought assistant can drain your data with no action on your part.11

Where AEGIS bites hardest

  • Assume the vendor is breached; treat their outputs as untrusted input crossing into your estate.
  • Ration what you connect them to — least privilege at the integration boundary is your strongest lever when you can't see inside.
  • Contain the blast radius around every vendor tool; bound what it can read, send, and trigger.
  • Demand contractual visibility: AI bills of materials, audit rights, training-data transparency, breach notification, misuse-prevention attestation.
Surface Three

The Supply Chain

What sits beneath both. Model weights, datasets, libraries, MLOps pipelines, open-source components.
Most weaponized · least governed

The trap

Trust assumptions that simply do not hold. A model file that runs code when you load it.14 A package that hijacks an install by name.15 A maintainer who was patient and malicious.16

Where AEGIS bites hardest

  • Assume any downloaded artifact is hostile until verified; prefer safe formats over executable ones, and scan before you trust.
  • Pin dependencies and verify provenance — signatures, hashes, attestation — to close the dependency-confusion class of attack.
  • Maintain an AI-BOM: weights, datasets, parameters, retraining events, agents, and connected servers, all inventoried.
  • Watch for the swerve at load and at runtime; a poisoned component often reveals itself only by what it does.
04 · The Operating Matrix

One framework, mapped across all three surfaces.

This is the working core — the page a security leader pins to the wall. Read down for a discipline, across for a surface. Every cell is a concrete move. No surface is left lighter than another.

Discipline Your Own SystemsFull access, under-used Vendor SystemsDepended-on, opaque The Supply ChainWeaponized, ungoverned
AAssume the Breach Threat-model internal pipelines as hostile; treat your own training data as potentially poisoned. Treat every vendor output as untrusted input; assume their environment is already compromised. Treat every model, dataset, and dependency as suspect until verified; assume a backdoor is present.
EEstablish Least Privilege Scope in-house agents and service accounts to minimum tools and read-only data where possible. Ration what vendor tools connect to; enforce least privilege at the integration boundary. Run build and training pipelines with minimal rights; isolate package and model fetching.
GGuard the Blast Radius Segment and sandbox internal workloads; cap what each agent can spend, send, or deploy. Bound each vendor integration; limit what it can read, exfiltrate, or trigger downstream. Sandbox model loading and execution; quarantine new artifacts before they touch production.
IIlluminate Behavior Log the full pipeline; baseline outputs and agent actions; reconstruct every decision. Monitor data flowing to and from vendors; watch for unexpected output shapes and volumes. Scan at load and watch at runtime; track provenance and flag components that act strangely.
SStrengthen Through Stress Red-team your own systems continuously; feed every finding back into design. Pressure-test vendor integrations; rehearse a vendor-breach incident; share intelligence. Test supply-chain controls; tabletop a poisoned-model scenario; verify your AI-BOM is current.
05 · The Blind Spots

What current guidance misses.

You asked to be told where the field looks away. Here it is — the gaps the standard frameworks leave open, drawn from the evidence and stated without softening. A framework that only repeated the consensus would be doing you a disservice. These are the places the unknown vulnerability is most likely to find a home.

01

Governance and engineering have split.

Boards write policy; engineers ship agents — and the two rarely meet in the middle. Only about one in four organizations has fully operational AI governance, and only a sliver report board-level oversight of it.19 Policy without engineering controls is paperwork. Controls without accountability are orphans.

02

Compliance lags the threat by years.

Frameworks codify the attacks already understood. The EU AI Act's high-risk obligations bind in 2026 and beyond — for capabilities that shipped in 2023.9 Treat every standard as a floor and a lagging indicator, never as the finish line.

03

Vendor opacity is treated as someone else's problem.

You inherit risks you cannot inspect, then check a box that says "third-party reviewed." The provider-versus-deployer line is itself a grey zone. You can't audit what you can't see — so the answer is contractual visibility and ruthless containment, not assumed safety.

04

Concentration risk hides in plain sight.

Everyone leans on a handful of foundation-model providers and shared training corpora. One poisoned model or common dataset can cascade across operations, finance, and customers at once.18 Monoculture is efficient until the day it isn't.

05

The agentic surface is exploding faster than the rules.

Agents chain actions, escalate privilege, and act after authentication. By the close of 2026, an estimated 40% of enterprise apps will embed task-specific agents, up from under 5%.20 Frameworks built for chatbots and classifiers don't cover tool-using autonomy.

06

Shadow AI is the unmanaged front line.

Employees use unsanctioned tools, and bans don't stop them. One in five organizations has already taken a shadow-AI-linked breach, adding real cost to the average.22 The fix is sanctioned tools and visibility, not prohibition.

07

"Foolproof" defenses are being sold.

The science isn't there yet. NIST's own taxonomy warns that securing AI has unsolved theoretical problems, and that anyone claiming otherwise is selling snake oil.3 Be skeptical of any vendor promising to make prompt injection or poisoning simply go away.

08

The human factors go unbudgeted.

Overconfidence, automation bias, and alert fatigue quietly defeat good controls. Benign users trigger harms too, not just adversaries.17 And there is no standard for AI incident reporting yet, so shared learning stays thin. Defense is a human discipline before it is a technical one.

The deepest gap of all: most programs still try to predict their way to safety — longer threat lists, more controls, thicker policies. But you cannot predict an unknown unknown. The organizations that will weather this era are the ones that quietly shift their spending from prediction to resilience: smaller blast radii, faster detection, graceful failure, and a standing habit of learning from every shock. That shift is the entire argument of this document.

06 · The Roadmap

Where to begin, and what comes next.

No organization does all of this at once, and you shouldn't try. Start with the floor, earn the right to the next stage, and never stop at the first. Each stage carries a gate — the condition that says you're ready to move on.

0 – 90 Days

Establish the Floor

  • Inventory every AI system — owned, bought, and embedded — plus every agent and its owner. Nothing is defended until it's known.
  • Build a first AI-BOM: models, datasets, dependencies, connected servers.
  • Adopt NIST AI RMF as the operating model; map systems to the OWASP LLM Top 10 and MITRE ATLAS.
  • Kill the executable model format — mandate safe serialization, scan artifacts, verify hashes, pin dependencies.
Gate: any system that takes irreversible action, touches regulated data, or faces the internet escalates to Stage Two now.
90 Days – 12 Months

Engineer Containment

  • Apply Zero Trust to agents: verified identities, least privilege, autonomy tiered by risk, human approval for the irreversible.
  • Stand up continuous monitoring of AI inputs, outputs, and agent traffic; baseline and alert.
  • Build and rehearse rollback; design systems to degrade gracefully under failure.
  • Rewrite vendor contracts for AI-BOM disclosure, audit rights, and breach notification.
Gate: if you cannot reconstruct an agent's decisions end-to-end, it does not get production write access.
Ongoing

Build Antifragility

  • Run a standing AI red team and regular incident tabletops using real cases as scenarios.
  • Wire board accountability to engineering controls — close the governance-engineering gap.
  • Replace shadow-AI bans with sanctioned tools and lightweight intake.
  • Pursue ISO/IEC 42001 for external assurance; join shared threat intelligence; align to evolving regulation.
Gate: when agent deployment doubles or a peer takes an AI breach, escalate monitoring and containment investment without waiting.
References

Sources and further reading.

Primary sources were favored throughout — standards bodies, the AI labs' own research, and peer-reviewed work — over secondary commentary. Several headline findings are vendor-disclosed and not yet independently confirmed; the caveats below the references say where to keep a measure of doubt.

  1. NIST, AI Risk Management Framework (AI RMF 1.0), January 2023. The four functions: Govern, Map, Measure, Manage.
  2. NIST, AI RMF: Generative AI Profile (NIST AI 600-1), July 2024.
  3. NIST, Adversarial Machine Learning: A Taxonomy and Terminology (NIST AI 100-2e2025), March 2025. Source of the "no foolproof method" and "snake oil" statements (A. Vassilev et al.).
  4. MITRE, ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems, knowledge base, 2026.
  5. OWASP, Top 10 for LLM Applications (2025) and the OWASP GenAI Security Project.
  6. CISA, NSA, NCSC et al., Guidelines for Secure AI System Development, November 2023.
  7. CISA et al., Deploying AI Systems Securely, April 2024.
  8. Google, Secure AI Framework (SAIF), and the Coalition for Secure AI (CoSAI).
  9. European Union, AI Act (Regulation 2024/1689), in force August 2024; phased obligations 2025–2027.
  10. ISO/IEC, 42001:2023 — AI Management Systems, and ISO/IEC 23894:2023 (AI risk management).
  11. Aim Labs / Aim Security, EchoLeak (CVE-2025-32711): the first documented zero-click prompt-injection data exfiltration in a production LLM (Microsoft 365 Copilot), disclosed June 2025.
  12. Anthropic, Disrupting the first reported AI-orchestrated cyber espionage campaign (GTG-1002), November 2025.
  13. Anthropic, UK AI Security Institute & the Alan Turing Institute, A small number of samples can poison LLMs of any size, October 2025.
  14. Kellas et al., PickleBall: Secure Deserialization of Pickle-based ML Models (ACM CCS 2025); see also ReversingLabs and JFrog research on malicious Hugging Face models.
  15. PyTorch Foundation / ReversingLabs, the torchtriton dependency-confusion attack, December 2022.
  16. CISA / OpenSSF, the xz utils backdoor (CVE-2024-3094), disclosed March 2024.
  17. Microsoft AI Red Team, Lessons from Red Teaming 100 Generative AI Products, January 2025.
  18. McKinsey & Company, Deploying agentic AI with safety and security, October 2025.
  19. McKinsey & Company, The State of AI, 2025; and AuditBoard, From Blueprint to Reality (2025) on operational AI governance.
  20. Gartner, press release on enterprise AI-agent adoption, August 2025.
  21. Pruthi et al., AI Safety Must Embrace an Antifragile Perspective, 2025; building on N. N. Taleb, Antifragile and The Black Swan.
  22. IBM, Cost of a Data Breach Report 2025 (shadow-AI findings); see also Vectra AI and Software AG surveys.
A Note on Certainty

Several headline findings in this document are reported by a single vendor and have not yet been independently verified — notably the AI-orchestrated espionage campaign and the data-poisoning result, both of which their own authors qualify with stated limitations. They are credible and material, but treat their precise impact as not-yet-confirmed. The regulatory picture is in motion; dates and obligations cited here may shift. Adoption and incident statistics are drawn from vendor and consulting surveys whose methods vary. Most of all: the consensus of the serious people in this field is that no defense against adversarial AI is yet foolproof. Everything here reduces risk. Nothing here eliminates it. That is precisely why the posture is resilience, not prediction.