Streaming support
The AI firewall should not buffer the response from the LLM but rather support streaming of the response from the LLM.
As you can see in the below screenshot, the output of the prompt “Describe the TLS handshake process in 400 words”(left side) is streaming in frames(right side).
Zooming in – each frame is a genuine streaming chunk
AI firewall should be able to show below metrics :
| Measurement | What it is |
|---|---|
| TTFT | Time to first token. What the user waits before the answer starts moving. |
| TTFB | Time to first byte off the wire, just ahead of TTFT. |
| Total | Start to last token, the whole response. |
| Frames | How many chunks the response arrived in. Spread out means streaming, bunched at the end means buffered. |
| Model time | The LLM’s own generation time. Reported by the AI gateway. |
| AI Gateway time | The AI gateway’s overhead. Reported by the gateway. |
| Firewall processing* | TTFB − (model + gateway) The firewall’s slice. Inspection plus the network path to its edge, not CPU alone. |
*Please note :
When the AI firewall blocks a prompt, it never reaches the AI gateway or the model – so all the time taken is by the AI firewall itself.
When the AI firewall does not block a prompt, the prompt goes through the AI gateway and then model and response comes back through same channel back.
An industry standard AI gateway includes the response headers noting the time taken by the upstream LLM and the AI gateway itself. Those response headers can be used by the AI firewall to calculate total time by the firewall.
Inspect ahead, cut before release. For the responses, It should catch a violation mid-stream and kills it before the bad part reaches the screen.
Both directions, separately:
Input and output are two inspections. Should be able to configure both or either,
Detection Engine
SLM for the first pass
A small language model should screen every request first. It’s fast and cheap enough to sit inline without wrecking latency.
Teach the SLM the domain specific constructs
The small language model should be trained on domain-specific constructs so it doesn’t throw false positives on legit prompts. For example, “Foley” reads like a person’s surname, but in a clinical note it’s a catheter. The model should understand the context, not just the word. And if it still flags a false positive, that’s where the LLM judge catches it.
Open term sources to train on:
* **Healthcare:** SNOMED CT and RxNorm, via the NLM’s UMLS.
* **Finance:** FIBO, from the EDM Council.
LLM judge to confirm
When the small model flags something which could be a false positive, it should escalate that one request to a bigger LLM judge to confirm. That way you only pay for the judge on the suspicious slice, not on every call.
Custom detectors
You should be able to add your own detectors, not just rely on the built-in ones. Sooner or later you’ll need to catch something the vendor never shipped. For example, quicly build a regex to detect something or exclude something without changing the SLM.
Configure the sensitivity of detection of SLM :
Should be able to configure the sensitivity of SLM to avoid false positives.
Named threat classes:
The firewall should name what it actually detects, so nobody waves it off as “AI stuff” – prompt injection, jailbreaks, PII and data exfiltration, tool and agent abuse.
Deployment
API key based access per agent
Inline to block. The firewall should be able to sit inline, right in the request path, so it can actually stop a bad prompt or response before it goes through.
Out-of-band / API based option.
It should also support an out-of-band mode, where it inspects a copy of the traffic off to the side. This adds no latency and won’t take your traffic down if it fails, but it can only alert, it can’t block.
https://guptaashish.com/2026/06/22/out-of-band-ai-inspection-the-missing-piece-in-cloudflares-ai-security-stack/
No forced public egress
For client data, the traffic should never leave to the public internet. The firewall should run inside your own cloud, or connect over a private link.
Fail-open vs fail-closed
You should be able to choose what happens when the firewall is unreachable. Does the traffic pass through, or does it get blocked?
Multi-AZ and multi-region
The firewall should run across multiple availability zones and regions.
Security and Governance
Metadata-only logging by default
The firewall should log only the metadata by default and keep the prompt and response content out, because that content can carry client PII. You turn on the full content only when you have a reason to.
Fast retrieval into SOAR
When a detection fires, you should be able to pull that prompt and response straight into your SOAR workflow, so the analyst can see exactly what triggered it and investigate right away.
Analysis time per request
The firewall should log how long it took to inspect each request, so you can prove you are inside your latency budget and catch it when it starts to drift.
Every detection mapped to a framework
A detection should never read as just “blocked.” When the firewall catches a prompt injection, it should tag it as OWASP LLM01. A jailbreak, a data exfiltration attempt, a tool-abuse attempt, each one should carry its label from MITRE ATLAS, the OWASP LLM Top 10, or the NIST AI RMF. That is what lets you report coverage to risk and compliance, and it is what makes the whole thing auditable.
