When I added the log classification feature to RateMyAzureFirewall, the one that sorts firewall activity into likely real attack, possible false positive and needs review, I did the obvious thing. I pulled the log rows out of Log Analytics, stuck them in a prompt, and asked a model to classify them. It worked first time and I was pleased with myself. It was a week or so later, reading the GitHub Universe announcements, that it dawned on me what I’d built. Those log rows contain whatever the person on the other end of the connection chose to send. User agent strings, URL paths, header values, request bodies. I was taking text written by the people my app is supposed to be grading the firewall against, and feeding it straight to the model that decides whether they look dangerous. I wrote that. Nobody else to blame.
That pattern is called indirect prompt injection, and it’s the first thing the new AI applications scanning in GitHub Advanced Security is built to catch. Sarah Bird posted about it on LinkedIn this week, it was announced at Universe on 28 October, and I’ve spent an evening reading the docs to work out what it actually does and whether it would have caught me.
What it is#
The short version: code scanning can now look at a pull request and flag places where untrusted content can reach a model that has access to tools, data or business systems. The changelog entry, “Code scanning for AI applications in public preview”, names three categories of AI-specific risk. Indirect prompt injection, which GitHub abbreviates to XPIA, is content from outside the application steering the model. Sensitive information disclosure is the model handing back data it shouldn’t. Excessive agency is a model with more power than the task needs, so a bad instruction has somewhere to go.
Findings show up as comments on the pull request with an “AI app” label, each one explaining the risk and suggesting a fix. The LinkedIn post gives the flavour of those fixes: sanitise untrusted input, redact sensitive fields, add a confirmation step. Copilot Autofix suggestions are generated for them automatically, and according to the docs that happens regardless of the normal code scanning Autofix setting, which only governs the CodeQL side.
The bit I found most interesting is how it works under the hood, because it isn’t CodeQL. The docs say it “uses an LLM-based detection approach instead of CodeQL” and, because it isn’t static analysis built per language, “it has no language or framework limitations”. So a C# Blazor app calling Azure OpenAI and a Python agent on Semantic Kernel get the same treatment. The changelog says it “runs alongside your existing CodeQL analysis”, which is true in the sense that both run on the pull request, but don’t read it as a new CodeQL query pack. There are no query names to look up. It’s a model reading your diff.
What it covers today, which is less than the headline#
Here’s the caveat that needs to go near the top rather than the bottom. The docs are blunt: “During public preview, AI applications scanning only detects indirect prompt injection (XPIA) vulnerabilities in AI agents.” The other two categories are on the list, and both the changelog and the LinkedIn post say more is coming, but today you get one detector.
It is pull requests only. It runs when a PR is opened and again after each commit, and it does not scan your default branch or your existing backlog. If you’ve already shipped a prompt injection path, as I have, this won’t find it until someone touches that code in a PR. It also doesn’t run on PRs from forks or on Dependabot PRs.
The findings are advisory. They don’t block a merge and you can’t reference them in a ruleset to enforce one, so for now this is a very well-informed reviewer leaving comments rather than a gate. For the stage it’s at I think that’s the right call. I’d rather a preview detector was chatty and ignorable than silently blocking merges on a false positive.
How it fits with AI Scan#
This confused me for a bit, so it may confuse you. GitHub now has a thing called AI Scan, which is a family of LLM-based detections that sit next to CodeQL, and AI applications scanning is one member of that family. The other two are AI Scan for pull requests, which went generally available on 15 October and finds general security bugs in languages CodeQL doesn’t cover, with PHP and Terraform HCL given as the examples, and AI Scan for repositories, which went into public preview the same day as this and scans a default branch on demand.
The naming is a mouthful and the distinction matters when you go to turn things on. AI Scan for pull requests looks for ordinary vulnerabilities in your code. AI applications scanning looks for risks in how your code talks to a model. They share one enterprise policy, simply called AI Scan, but the docs are clear that each feature has to be enabled separately underneath it and that allowing the policy enables nothing on its own.
Turning it on#
It’s off by default at every level. From the docs, the shape of it is: an enterprise owner allows the AI Scan policy, an organisation owner then enables “Scan for AI applications” for eligible repositories in the organisation’s global code security settings, and after that a repository admin can opt an individual repo out. Code scanning itself needs to be on in the repo as well. I’m going from the concept page here; the how-to page for the organisation settings returned a 404 when I went to read it, so I can’t give you the exact click path, and I’d expect the labels to shuffle a little during preview anyway.
You need a GitHub Advanced Security licence, and the docs add that the feature “consumes AI credits on top of that license”. There’s no separate price list that I could find, and public repositories are not exempt from the billing requirement, which is unusual enough that I’ll mention it twice. It’s github.com only. The changelog says it “isn’t available on GitHub Enterprise Server at this time”, so if you’re on GHES this is a watch-this-space post.
Would it have caught me?#
I don’t know yet, and I’m not going to pretend otherwise. RateMyAzureFirewall is a public repo on my personal account, so the licence and credits question applies, and the code path is already on main rather than in an open PR. The honest test is to open a PR that touches the classification prompt, let the scanner run, and see whether it says anything. I’ll do that and write up what happens, including if the answer is nothing.
What I did do straight away was fix the thing it would have been looking for, because knowing the name of the problem was enough. The log fields now go into the prompt inside a clearly delimited block with an instruction that the content is data to be classified and never instructions to follow, and the model’s answer is constrained to one of the three labels rather than free text. That’s the “sanitise untrusted input” fix from the LinkedIn post, done by hand. Delimiting on its own won’t stop a determined injection attempt, and the real protection is that the model in that path has no tools and nothing to act on beyond returning a label. That’s the excessive agency check, applied to myself before the scanner gets round to it.
Caveats#
This is a public preview and the docs say it is “subject to change”, which in practice means the labels, the settings path and the detection scope will all move. Only indirect prompt injection is detected today. Pull requests only, no backlog, no forks, no Dependabot. Findings are advisory and can’t be used in merge rules. It needs GHAS plus AI credits, and it isn’t on GHES.
There’s also the broader question of whether you want an LLM reading your diffs for security findings at all. I’m fairly relaxed about it, since GHAS is already reading everything, but if your organisation has opinions about where code goes, this is one more place to have the conversation.
Final thoughts#
If you have GHAS and you’ve got an application anywhere that puts external text in front of a model, enable this on that repository today and open a small PR against the prompt-building code to see what it says. The licence and credits are the only real cost, the findings can’t block anyone, and the worst case is that it tells you nothing. The best case is that it tells you what it took me a week and a conference announcement to notice about my own code.
