I presented this work at DEFCON 32 (video, slides), HOPE XV, BSides Las Vegas, and to the NVIDIA security team.
An earlier version of this was presented as a UCSC LSD Seminar Talk and constituted my undergraduate thesis under Professor Wenke Lee.
Fun Fact: This was quoted in Songs on the Security of Networks.
Here's an annotated version of my slides taken and cleaned up from a transcript.
Hello, I’m Suha! I’m really excited to be here today. Let’s get started. I want to talk to you about ML security. Specifically, I want to talk about this new class of exploits I identified called incubated ML exploits which combines backdoors and input-handling bugs. Don’t worry if you don’t know much about ML or ML security, I’ll explain the important stuff as we go along.
So who am I and why am I even talking to you today? I’m an engineer at Trail of Bits where I focus on AI/ML security. I’ve been in this field for a few years now. I graduated from Georgia Tech. And I’m originally from Queens. Outside of work, I like Brazilian Jiu-Jitsu, trying new restaurants, making things, and an obscure card game called Cubirds.
So it’s becoming pretty clear that with ML and AI popping up everywhere, people are figuring out how to trick these things based on how these models work. Maybe you’ve seen someone use prompt injection to convince a chatbot to give them a refund. Or maybe you’ve seen this story of protestors tricking self-driving cars with traffic cones. Notice the fact that this trick is rooted in an understanding of the training data for these models.
So how can we actually construct our own useful exploits against ML systems?
Let’s play a game of pretend real quick. You’re a college student and you really want the prize money for a robotics competition, so (naturally) you decide to sabotage another team. (Side note: I don’t recommend this or condone this behavior of course). Anyway, the competition requires teams to build a tiny autonomous vehicle that uses a specific pretrained model and stops at stop signs.
You find out that some of these stop signs have stickers on them and you find some flaws with how they’ve stored and distributed the model. Which, by the way, isn’t out of the question! ML artifacts are shared widely often without any substantial trust mechanisms. So you decide to grab that file and inject a model backdoor in it using a file format RCE of some kind and put it back.
Then you sit back on the day of the competition and watch as your competitor’s vehicle absolutely plows through and ignores every stop sign with a sticker on it. What you just did is execute an incubated ML exploit, which is what my talk is all about.
Obviously, the stakes of this story is just a lost competition. But attacking a real autonomous vehicle is a hallmark of model backdoor research as you can see with the image on the left. So I’ll let you use your imagination to raise the stakes.
First, I’m going to tell you about this framework I use to bridge the gap between model and systems security because we can’t continue to treat models as standalone objects. Next, I’ll tell you about these input-handling bugs I found in model serialization and connect them to backdoors. I'll do that by taking a page out of this subfield called LangSec. So effectively, I’ll go through a bunch of examples of incubated ML exploits and use LangSec to organize them.
But first I need to explain some stuff. What even is a model vulnerability or a ML backdoor?
Super briefly, you can think of ML models like these squishy, flexible sequences of linear algebra operations that are trained on tons and tons of data. There’s a popular saying: 'all models are wrong, but some are useful.' It's just saying that these models aren't perfect. There are many different ways that models can mess up or get tripped up by something that may be unexpected to us. And that’s the basis of such model vulnerabilities. While popular examples of model vulnerabilities include model inversion and membership inference, we're zooming in on one specific type: backdoors.
The precise definition is that a backdoor attack allows a malicious actor to force an ML model to produce specific outputs given specific inputs. There’s a couple of things that I think makes backdoors really interesting to study. You can use them as primitives for other model vulnerabilities like membership inference, you can identify pre-existing “natural” backdoors in models, and there’s pretty strong evidence that suggests that this is an inherent threat.
While there’s a lot of awesome research on ML model attacks, they can actually be pretty hard to exploit in the real world— with some exceptions of course. While there are multiple reasons for it, to me, one reason that really sticks out to me is the gap between research and the real-world.
For the most part, many attacks and attack frameworks and tools restrict their analysis to this formulation: An ML model receives an input and produces an output. But this isn’t an accurate representation of what an ML system actually looks like. There's so much more going on in practice.
Here is a software architecture diagram for a ML system reviewed by ToB recently. This is a system that uses the AskAstro tool for RAG. I’ve circled where the model actually is in this photo. Do you see what I mean? We need to look at all of this holistically! There’s a large and evolving landscape of tools being used in and for ML systems. And that brings me to the exploit framework.
The title of my talk obviously references an incubated ML exploit, but there’s a larger category of exploits called hybrid ML exploit that are important to think about first.
Specifically, a hybrid ML exploit chains a system security issue with a model vulnerability. This can go in either direction: a model vulnerability can expose a system security issue or a system security issue can be used to exploit a model vulnerability.
The big issue with ML security is that model security and systems security are treated separately. But what I need you to understand is that if you only know model security, you’re missing a big piece, and if you only cover systems security, you’re still missing a big piece. And you can’t treat these two processes completely independently. You’re then entirely ignoring the potential for hybrid ML exploits.
This is an emergent property. Your model is embedded in a system and it's going to interact with all of the different system components in new and exploitable ways. You’ll notice that there’s a lot of screenshots of paper titles on the slides. That’s because there have been specific instances of hybrid ML exploits in the literature and in practice. They’re just not called that explicitly.
Exploitable software gadgets have been used for backdoors; the Summoning Demons paper at the top chained model evasion with memory corruption; and the learned systems security paper next to includes an example of a poisoning attack that causes exponential memory blowup in an index structure. But the ML security literature, frameworks, and tools that I’m familiar with are largely limited to just that: specific instances or implications. What I’m trying to do here is treat this interaction explicitly and systematically. And that’s why I made this framework.
So one kind of system security issue is an input-handling bug. And one kind of model vulnerability is a model backdoor.
Put that together and that’s how we get an incubated ML exploit, which is a type of hybrid ML exploit where an attacker uses an input-handling bug to inject a backdoor. I made this diagram to make this distinction between the two pretty clear.
Here it is again. We’ll leave the framework here for now. We did end up going into a more formal model of exploitation, namely a schema for incubated ML exploits. But we’ll return to this notion later.
To backdoor a pre-existing model, the attacker should be able to change the parameters of the model or the architecture. At the level of abstraction we’re dealing with, we can put input and component manipulation on the side for now. How this actually plays out can vary a lot.
For example, sometimes, the attacker can have control over an element of the training process. If they do, they could sneak in some manipulated data to mess with the model's parameters, which is usually called data poisoning. Or, they might go a step further and fiddle with the model's source code to change the architecture.
Now, before we dive into the exploit PoCs, I need to explain a few things about input-handling bugs.
An ML model is stored as a file. To process these models, you need parsers! Parsing these files into objects and back is deserialization and serialization.
But wait! Quoting Ange Albertini here, “…a file has no intrinsic meaning. The meaning of a file—its type, its validity, its contents—can be different for each parser or interpreter.” This is the reason we can make cool potentially malicious file artifacts like polyglot files and ambiguous files, which I’ll talk more about later.
I’m focusing very specifically on bugs when you parse ML model files. There’s also cool bugs that arise in other parts of the pipeline, but I’m picking ML model files for several reasons. The first reason is the most important: it’s fun. But, more seriously, the security of ML file formats has become increasingly important.
ML has a culture of sharing these artifacts without sufficient validation. Real malicious models have been found on HuggingFace Hub for example. There’s aso tons of ML file formats out. I’ve tried to list and organize in the repository listed in the middle, but what’s important to takeaway is that there’s a large set of possibilities for exploits as well as just fun hacks with these formats. There’s already a lot of great work on this as shown in the slide.
File format tricks are within the realm of LangSec, but this field actually thinks more abstractly about inputs as a general class. LangSec applies formal language theory to systems security. It focuses on exploring input-handling bugs or parser problems as a big root cause for security issues. After all, lots of impactful vulnerabilities like Heartbleed and Android Master Key have been parser bugs.
While I like formal language theory, this isn’t theoretical computer science 101, so just know that fundamentally what LangSec is saying is 'Hey, let's treat all input as a specific language and then make our code just capable enough to understand that language properly.'
Our work is centered around a specific taxonomy of input-handling bugs. Here are all the different bug classes. There are 8 different types listed here. These categories are actually not completely distinct from each other. The one you choose comes from a root cause analysis.
With the exception of 1, I’m going to show you multiple examples of each in ML tools and use them to construct an ML backdoor.
So, once again, in order to show that input-handling bugs are a vector, I identified ML model serialization issues across the different bug classes to construct ML backdoors.
Now we can dive into the most fun part: the exploits. Specifically, for the sake of time, I’m going to focus a bit more on the useful gadgets that actually arise in these situations.
These are some characters that play important roles in the ML ecosystem to help us understand the impact.
-
Alice distributes models. She takes open-source LLMs and fine-tunes them. The models she distributes are what everyone else in our story is going to use.
-
Bob is a front-line user who directly uses Alice's models in his own life, maybe through a chat interface.
-
Then there's Dave, an engineer who integrates these models into products.
-
Frank is the end user, relying on Dave's products in his daily life, often unaware of the ML models working behind the scenes.
-
And Chuck? He's the attacker, looking to exploit vulnerabilities in the models and disrupt the work of Alice, Bob, and Dave.
Our focus here will mainly be on Bob and Dave and how they’re impacted. I’ll show exploits involving the file formats associated with pickle, PyTorch, TorchScript, ONNX, and Safetensors.
The first category is called non-minimalist input-handling code. It sounds a little fancy but all it means is that the code used to check and parse the inputs is too complex, so an attacker can potentially grab the necessary gadgets for their exploits.
Here’s a common case. Pickling is a serialization method that allows you to save arbitrary objects that is very common in the ML ecosystem. Recently, my coworker Boyan Milanov led the development of Sleepy Pickle, which is an incubated ML exploit. What it does is that it chains a pickle RCE with model backdoors. So on the right you can see an LLM that has been backdoored to phish users. There’s also examples of an LLM being backdoored to spread misinformation and even steal user data in the blog post. What’s cool about this one is that it can happen on-the-fly, so there's far more room and possibilities for an attacker than just uploading a malicious model.
So what do I mean by pickle RCE? Python pickles are compiled programs that run in a unique virtual machine called a “Pickle Machine”, or “PM” for short. The PM interprets the pickle file’s sequence of opcodes to construct an arbitrarily complex Python object. But it has two opcodes, GLOBAL and REDUCE, that can execute arbitrary code outside of the PM, which makes it possible to construct malicious pickle data. The underlying reason here is that the PM is more complex than something that’s only parsing ML models should actually be.
Way back in 2021, we released Fickling. This project was led by Evan Sultanik. To our knowledge, Fickling was the first pickle security tool tailored for ML use cases. It’s a decompiler, static analyzer, and bytecode rewriter for the Python pickle module. It can help you detect, analyze, or create malicious pickle files. Fickling is safe to run on potentially malicious files because it symbolically executes code using its own implementation of the Pickle Machine (PM).
I also added a PyTorch module to it relatively recently, so that you can statically analyze and inject code into PyTorch files. Anyway, pickles are clearly an issue for Bob. If Alice distributes models as pickle files or PyTorch files, that makes it easier for Chuck to inject a backdoor using a pickle RCE.
Now onto the next class. This term just means you shouldn’t try to correct invalid input, just reject it altogether. Sometimes it’s called the anti-robustness principle.
To mitigate the issues with pickling, many developers write restricted unpicklers. Restricted unpicklers are subclasses of Unpickler that enforce an allow-list or a block-list through the find_class method. But these are easily bypassable.
There’s this methodology called Pain Pickle that demonstrates how to automatically bypass restricted unpicklers, enabling arbitrary code execution and backdoor attacks. They identified 8 different types of unpicklers and 3 strategies that work against the vast majority of these unpicklers.
Much like pickle was a problem for Bob, restricted unpickling bypasses pose a problem for Dave if he relies on these in some fashion.
Now, we can talk about parser differentials!
This happens when different parsers in a system read the same input but interpret it differently. So when two parsers interpret the same file in different ways, that file is known as an ambiguous file. This is a very common exploit technique that’s especially good for bypasses. It means you can create an ML model file that is benign for one system or system component but backdoored for another.
There are some big implications for ML system exploitation here that we’ll talk a bit more about later. But note that whether or not this is impactful all depends on your system, so this is where threat modeling comes in handy.
We actually were able to create two differential PoCs with TorchScript. So TorchScript is this popular format to store ML models in for a bunch of reasons related to performance and portability. But you can make a parser differential using it and chain it to an architectural backdoor. This is because you can turn a PyTorch model into a TorchScript one through tracing or scripting, and tracing does not incorporate dynamic control flow. So all you have to do is represent the malicious components for the backdoor through dynamic control flow.
The second example was found during the YOLO audit. Last year, my team and I audited a popular open-source codebase for computer vision called YOLOv7. They released standard versions of their model and TorchScripted versions for deployment. We noticed many cases where tracing didn't capture the model accurately. After serialization and deserialization, key information was lost, and the usual PyTorch warnings didn't show up. To spot this differential, we used the TorchScript automatic trace checker, torch.fx, and the TorchScript IR. With what we found, we created an input that made the two versions of the model act differently – essentially, a backdoor attack.
Once again, this is a problem for Bob. He’s getting a fundamentally different model than the one Alice trained. This breaks any pre-existing premises.
We identified a parser differential with safetensors. Safetensors is another file format for ML models developed in response to the insecurity of pickling. Last year, I was on an audit of the safetensors library where we identified the inclusion of JSON in the file format as a source of parser differentials.
JSON is known to be underspecified– there’s lots of exploits especially in the web security world leveraging this. The thing is the reference safetensors implementation uses the serde parser, which is strict and rejects duplicate keys.
But a lot of external tools use the Python built-in JSON parser, which doesn’t. So you can use a duplicate key for the offsets to append backdoored weights and create manipulated safetensors files. These files will be rejected by the reference implementation but accepted by external parsers. It has to be a weights-based backdoor because weights are stored separately here.
There’s some more details and caveats regarding exploitability but just know that the safetensors parser differential is more impactful for Dave! He needs to make sure all of the parsers in his product agree. If his tool uses a more permissive safetensors parser than the reference implementation, it would accept manipulated safetensors files that actually carry backdoored models.
One big part of my research was analyzing previous works and noticing trends. I don’t want to get too into the weeds as I’d like to save formalisms for the accompanying materials. But one thing that became clear is from parser differentials, we get model differentials: instances where the same model is interpreted differently. As expected, the attack is dependent on the supply chain component and lifecycle stage. So in an ML system, you can preprocess inputs or you can apply model transformations before you deploy a model.
Some studies have exploited parser differentials right at the preprocessing stage – things like image-scaling or how Unicode is parsed. Those attacks change the weights. There have also been backdoor attacks that take advantage of differences during model transformations like compilation and quantization. Those attacks usually change the architecture. I think it’s very possible that most transformations that can be encoded in a loss function can result in an exploitable backdoor. Moving forward.
Next up is shotgun parsing. This is just what happens when you don’t fully and properly check your input before beginning to process it.
So, let’s talk about polyglot files, which are files that can be validly interpreted as two or more different formats. Polyglots files have been utilized to distribute malware, bypass code signing checks, and enable other malicious behaviors.
With regards to ML model serialization, these polyglot files can be placed in model hubs to confuse downstream consumers. But more importantly, two different ML pipelines could interpret the same file as two different models. So you can smuggle a backdoored model with a benign one.
During our audit of the safetensors library, we were able to make multiple polyglots. This includes ZIP, PDF, TFRecords, Keras native file formats, and later on PyTorch MAR. The safetensors audit report itself is a PDF/ZIP polyglot and the ZIP file contains all the safetensors polyglots made during the audit. You can slap on a weights based backdoored model in one of these formats to a benign model in a safetensors file. When you open it with safetensors, everything seems fine. But load it up with PyTorch MAR or some other system, and boom, there's your backdoor.
This is a real problem for folks like Dave who depend on these models because now you've got malicious models sneaking in with benign ones. The reason this is possible is because of a missing check. Specifically, the program did not check whether the start and end offsets correspond with the tensor size, so attackers can append arbitrary data to the file. That can be combined with the ability to change the header size, which expands the number of polyglots. This issue has been fixed in safetensors, however.
Our next category is incomplete protocol specification. Just think of it as underspecification. While there are multiple examples of this in the literature, for the sake of time, let’s focus on PyTorch polyglots.
Many are unaware that PyTorch actually supports multiple file formats. Some are deprecated but still supported by external parsers. One key issue with the file format is that it lacks consistent versioning. That means you can create polyglots of files that can be validly interpreted as different types of PyTorch file formats. You can also create ambiguous files. So add three files to polymock PyTorch v1.3 and TorchScript v1.4.
Another issue is the reliance on both ZIP and Pickle. Pickle is a streaming file format that ends once it reaches the STOP opcode. Any data after that is fair game. Most ZIP parsers do not enforce the magic at the start like the PyTorch MAR. So you can append a ZIP to a file to create a ZIP/Pickle polyglot, which gives you some good PyTorch polyglots. Fickling now has a polyglot module, so you can differentiate, identify, and create polyglots for the different PyTorch file formats.
Now onto the next class. This one just means that your inputs should be simple and well-defined so you can check it thoroughly.
Take ONNX. It's a Protobuf-based way to store ML models. Adelin Travers discovered a neat hack for ONNX that he packaged into a tool called LobotoMI. So ML runtimes and frameworks often let you add custom operators to a model on the fly, and the language used for ONNXRuntime custom operations was complex.
Now, even though the ONNX protobuf specification officially disallows side effects, arbitrary code can actually be encapsulated in a custom operation in the ONNXRuntime. You can use this to launch an architectural backdoor attack on a model saved in the ONNX file format. Just like pickle, this isn’t good news for Bob.
Alright, to recap. Bob, our direct consumer, was affected by our exploits using the pickle, ONNX, and TorchScript formats for ML models. Dave, on the other hand, was affected by PyTorch, Safetensors, and restricted unpickling issues.
What a lot of people miss is how important and complex the ML stack is. The model you choose changes the technologies in the stack, so when I’m assessing a system or doing some kind of vulnerability research, I’m always trying to think about what layer of the ML stack I’m dealing with. The layers listed are hardware, infrastructure, low-level, compiler, high-level, framework, model, and knowledge.
I told you about a whole bunch of exploits just now. Of the ones I told you about, the restricted unpickler, ONNXRuntime, and pickle PoCs are issues that are exposed and impactful at the framework level. The TorchScript differential corresponds to the compiler level, and the safetensors and pytorch polyglot issues are impactful at the infrastructure level. This is just a starting point.
There’s going to be exploits up and down this stack that impact systems with ML in it. If you wanna get into attacking ML systems, this is a solid place to begin. So are you really good at breaking hardware? Go take a look at a TPU. Do you happen to know a lot about distributed systems security? Go write some hybrid ML exploits at the infrastructure level.
I made this schema for incubated ML exploits. This is just one piece of the formal model of exploitation I worked on. I’ll talk about this at a very high level just to shed some light on the terrain here.
If you want to pull off an incubated ML exploit, you need write primities for the weights or architecture. The PoCs point to some additional capabilities. You probably want to read primitives as well. But also with the safetensors parser differential, you saw that access to the metadata can really help an attacker. Specifically, read and write primitives for the metadata of a model file format can enable both kinds of backdoors.
There’s a lot of utility in exploiting model transformations and model differentials; with that, you can construct exploits at different stages of the pipeline with existing procedures. Differentials are broadly useful; they’re pretty localized to their stage. With ONNX, it became obvious that you can maliciously use custom operators in serialization formats and potentially in places like compiler dialects. I’ll release more details on this in accompanying materials.
I do want to make more explicit recommendations. Apologies for the busy slide. Models should be checked for integrity and their metadata should be well-parsed. We want good trust mechanisms and validation. We want to minimize complexity, so avoid custom operators and separate weight and architecture storage. Follow best practices for file formats like having versions, checksums, and magic signatures as well as enforcing a signature at offset 0. Invest in robust specifications, avoid overloading fields, and avoid corrections for invalid input.
I’m hoping we can see hybrid ML exploits and incubated ML exploits addressed by more frameworks and tools. I’d love to see this framework evolve and be applied to specific ML tools and contexts. I wanna see it use more bug classes and more model vulnerabilities. I’d like to see people investigate exploit persistence, reliability, mitigations, and defenses more. I also think there’s a lot more interesting work to be done here as well as with ML differentials, file formats, specification improvements, and even reverse engineering.
Before we finish, what helps me to identify and make progress on ML security problems is understanding these two root causes. First, we’re building all these new systems for ML: hardware, programming languages, compilers, file formats. There are conferences dedicated just to designing ML infrastructure. That means these new systems introduce new attack surfaces.
Also it’s becoming increasingly clear that the ML stack and supply chain has not been subject to sufficient review. That’s why we see pickles everywhere. Second, simply placing an ML model into a program introduces all these new vulnerabilities that stem from how the model interacts with different components. ML is not a quick add on but something that can fundamentally change your system’s security posture.
I hope you leave this talk knowing that we need to concurrently and holistically think about system security and model security. I recommend checking out the full audit reports for safetensors and YOLO as well as the blog posts on Fickling and the file formats repo. I’ll post more details in accompanying materials, and we’re going to be releasing a paper on this topic as well. You can find my contact information on my website or just send me a message on Twitter. Thanks everyone for listening! Do you have any questions?
References
Appendix
I’m not going to spend too much time on the ins and outs of backdoor algorithms. But let me detail one kind of attack to give you some intuition.
Developed by Hong et al., the handcrafted backdoor attack directly edits the parameters of a model to inject a backdoor. To understand how this attack works, consider a building. Not all walls in a building are load-bearing. Someone can remove some walls without affecting the structural integrity of the building. Imagine someone knows which walls they can remove without the building collapsing and they use that knowledge to create a hidden path.
That’s essentially what is happening with the backdoor attack. Not all of the neurons in a neural network impact its functionality. The attack manipulates these less impactful neurons to inject a pathway between the input and the output. This pathway only lights up when a certain trigger is in the data, making the model spit out whatever the attacker wants.
Why do this? While there are multiple advantages, if you’re familiar with the ML ecosystem, it becomes quite clear that it’s very plausible for this kind of attack to occur. We’ll return to this idea again, but ML models are regularly shared on model hubs in insecure file formats such as pickles. A culture of sharing ML models without proper validation makes these attacks achievable.
