Disclaimer: The views expressed here are my own and do not represent those of any current or former employer or affiliated organization.
As an auditor, I would get very excited when target documentation said, explicitly or implicitly, “We assume X is not possible.” That assumption presented an opportunity. Security is all about understanding, and our industry is built on false or incorrect assumptions about both attackers and underlying systems. In my opinion, we should be using AI to build tools that help us expose these assumptions faster. However, that’s not limited to AI bugfinding tools! We can use AI for program analysis as well (very carefully of course).
Python program analysis has been a particular interest of mine for quite some time, and Geoffrey Litt’s “Enough AI copilots! We need AI HUDs” inspired to build a mini-world/HUD to visualize Pynsy, a very cool Python program analysis tool that instruments Python bytecode as it runs and lets you inspect executed instructions and their operands. This walks me through the actual tensor shape inference so I can absorb the execution events, constraint solving, and symbolic shape annotations, even helping me build an intuition about how it handles hyperparameter perturbation. The focus of this post is not the HUD, but what it means to me, however.
Years ago, I read Scott Nicholson’s “A RECIPE for Meaningful Gamification,” for my Educational Technology class. This was my first formal introduction to the idea that our ultimate goal should actually be to build intrinsic motivation, not replace it. This is how I felt when playing with the Pynsy HUD. I felt almost like I was playing The Witness, exploring a brand new world. That’s what I want to see us build more of!
In his AI Engineer’s World Fair talk, Litt argues that we review PRs to understand the system well enough to generate new ideas. I’d go further: we also read them to cultivate taste and build intuition.
Now, taste has been the subject of much recent writing (including “Why We Like Things” by Celine Nguyen). While it’s a fuzzy, often ill-defined, and controversial notion, it gets at something many of us intuitively believe (perhaps this something is much better represented by a more precise word, but that makes it lose its generality and is not the focus of this post).
In the world of security, there is taste in mitigations, defenses, architectures, and exploits. Many offensive security researchers will tell you exactly how and why finding more bugs is not the same thing as finding better bugs.
Experts sometimes claim to be able to tell when experience is lacking based on their taste is embodied in conversations. You can tell when someone has trained a model by how they talk about model behavior. You can tell when someone has exploited a buffer overflow. I built a lot of that intuition by manually looking at training data and metrics, or by reverse engineering and staring at hex dumps. It felt useless at the time, but it was all useful. There’s tacit knowledge encoded in that work.
Our attempts at legibility can erode the nuance of what attackers can and cannot do in practice. Defenders often ignore exploit chaining and reliability for lack of tools, frameworks, and processes. An image downscaling attack may seem hard unless an attacker has already invested in the profiling datasets needed; then it becomes reliable and easy. A similar pattern likely holds true for floating point attacks and emulators. How many attacks are being made easier not because AI is doing the attacking, but because AI helps people build tools that once took too much work to build?
Grant Sanderson recently imagined AI agents continually working on formal mathematics problems for years on end, noting that this domain may have a notion of grindability that is lacking in similar fields that would allow them to continuously work at a set of problems and still produce useful information over time. Meanwhile, this analogue does not work for software engineering: the slop will simply beget more slop.
Offensive security is somewhat similar. After all, an exploit is a proof of vulnerability (plus a kill chain). I would expect agents in this scenario to also produce notions of leaky abstractions that could prove exploitable, even if the exploits themselves are not foolproof. One could argue that making an exploit reliable or deciding which vulnerability is worth pursuing requires taste a verifier cannot capture. One could argue all we’re doing now in AI discourse is searching for the gaps our verifiable domains don’t cover. This likely means that we’re having to rethink our systems that were designed with legibility in mind, having to surface the illegible tasks that actually made our systems run.
I want AI to help us build better analyzers, expose false assumptions, and make the hard work of understanding systems easier to do. We should proactively include exercises that build taste, because taste is hard to measure. AI can help us do harder things, not just faster things.
Celine Nguyen posited that disagreements are a feature of our community and how we build taste, so taste will remain a human connection. I believe this is true. I wonder if we can proactively include the construction of mini-worlds in our life to build this intuition, so that we can preserve taste and work against the forces of slop.
For a while, I was struggling on how to use AI to help me to do research. AI is quite good at fooling you and good research is all about not fooling yourself. Constrained tasks somewhat work; I haven’t had great success with Autoresearch/RSI-style techniques personally. The greatest research insights often come from the most random of places with the most unexpected of paths to arrive at a particular conclusion. Perhaps this is what we need to do to bring back these moments of insight and discovery that make research so critical: build mini-worlds.
Many people are already doing this. I'd love to see even more. But more than that, I think organizations should proactively invest in building these tools. Yes, these projects won't have metrics tied to it, but they will pay off.
