But did you even READ the policy?

If you work in a company, almost any company, you have obviously read all the security policies. At least, you have signed off that you have read all the policies, because that’s part of the required annual security training. “Sure, yeah, I read all of them. I know where they are anyway. I think.”

There are two reasons why every annual mandatory security training requires you to read the policies. First, you’re actually, you know, supposed to follow them. It’s probably hard to do that if you don’t read them first. Second, so the company can fire you if you fail to follow them. Then again, I haven’t read the Revised Code of Washington, and I have to follow that too, so what’s the point?

When you sign off agree to the “…up to and including termination” part at the bottom of most policies. In our legal system this isn’t necessary because of the common law principle of ignorantia juris non excusat—Ignorance of the law is no excuse. This principle is fundamental to the system and in policies we use sign-off as a substitute, but it comes to the same thing: humans are responsible for their actions even if they don’t know what the rules are. With the advent of AI, however, this whole principle breaks down.

The Policy Problem with AI

With the advent of AI, particularly agentic AI and coding agents that take 5GL as an input and produces actions, it is no longer the human being that needs to comply with your policies; it’s the AI. The human will ask for something relatively innocuous, such as “Download data from the data lake, extract data on <something sensitive>, produce a summary report for my monthly business review, and put it somewhere I can get to it.” The AI writes an agent that does that, and then takes the report and uploads it to a public cloud service, where it doesn’t belong. The human didn’t ask where the data was going, nor tell the AI.

The problem is that our policies haven’t kept up with the technology we are using. That is not really new though. Policies are the single most under-resourced security function in nearly every company. They’re “paper tigers” and “only for compliance” and nobody actually reads them. It has mostly worked because humans with a certain amount of judgment will sometimes (often) act in the general direction the policies tell us to go.

However, LLMs do not have judgment. They simply perform very sophisticated pattern matching. They may pass a Turing test, but they are not intelligent in the sense described in Searle’s Chinese Room paper. The result is actions that are no longer orchestrated directly by humans with judgment. The actions are described in a 5GL that is sufficiently abstracted from the actual action that the interpretation of the 4GL and its conversion into actionable computer code is where the policy violation happens. This problem will have dire consequences at a velocity we won’t be able to keep up with if we don’t address it.  Data will leak and be discovered by other agents and AI based tools before we even know it is out there.Ineffective Approaches

There are three possible solutions to address this. We could attempt to govern the 5GL: “Staff members must not request AI tools to upload data to places it doesn’t belong,” and about a million other similar and largely ineffective statements. That approach would ignore the fact that the human doesn’t decide how to execute the request, the AI does, and that the humans wouldn’t remember all the things they can’t do even if they were to actually read the policies.

The second alternative is to require humans to inspect all actions before they take place, to review the 3GL and 4GL the tool produces. I already talked in another paper about why that won’t work. To summarize that argument, humans actually have a terrible track record of approving the wrong things. We did not evolve to understand whether an action an AI tool is recommending is safe or not. To most people, that question will look like “Do you want me to do something? [Yes|Ask me again until I say yes]”. Furthermore, the velocity of actions is too high for humans to process, which is the entire point of AI - to speed up our work. If we have to decide if every action is safe, it won’t speed up, it will slow down. Finally, the human likely won’t know how to even understand the action, and even if they do, it will take too long. The action may involve a thousand or more lines of code. Asking a human that doesn’t understand code to approve it is meaningless.

Machine-Readable Policies

The only option, then, is to change how policies are written and consumed. It is no longer the human staff member that needs to enforce the policy. It’s the AI. The policies must be written in a machine readable form that is sufficiently prescriptive to guide the AI to take the right action. 

Policies today are usually written in a form such as “All secrets must be stored securely.” The interpretation of “securely” is up to human judgment, or the Software Security team to judge. We have resisted encoding things like “All secrets must be stored in the Aegis Key Vault”, because the product may be called Aegis today, but after the next re-org it might be renamed to Fafnir. However, at the velocity of AI, we are taking the humans who (should) know what it is called today out of the loop. If we don’t actually name the product in the machine readable policy that AI will probably just write a new key vault every time one is needed, or pull down some random GitHub repo that looks semi legitimate. That might meet the exact letter of the policy, but certainly not the spirit of it. It also turns out that computers, and AI is just a computer, are better at remembering what the product is called today anyway, so the policy could say “the place that is currently approved for X as noted in Y.”

Second, the policies must actually be available to, in fact, required input into, the AI tools. The tools must enforce the policies, and they can’t do that unless they actually can read them. Today, many of our policies are hidden away, and we may even have policies that say policies are not to be distributed to external tools, like third party LLMs. That has to change.

If you have not already started resourcing your Security Policy Team to meet this challenge, it’s probably time to start doing so. AI isn’t coming, it’s here. Your people will not use it, they are. All of InfoSec must operate at the velocity of AI and it can’t wait.