Edge LLMs that never block the detection path

If you put a language model behind an intrusion detector, there are two latencies in the system and only one of them is allowed to matter.

The first is generation: the model reading an alert and writing a paragraph a human can act on. On a small quantized model running locally, that is around eight and a half seconds per alert. The second is dispatch: constructing the alert record and handing it to a queue. That is 16.6 microseconds at p99.

They differ by about six orders of magnitude, and everything about whether the design is defensible comes down to keeping them separate — in the architecture, and then in how you describe it.

Why the model is local at all

The obvious implementation is an API call. It is also the one you cannot ship, because the deployment target is a sensor that may be air-gapped, may sit on a network where exfiltrating traffic descriptions to a third party is precisely the thing being defended against, and may be somewhere with no reliable link at all.

So the model runs on the device. TinyLlama 1.1B at 4-bit, about 0.77 GB — which fits alongside the detection model on hardware that has no business running a language model, and is the reason the generation number is eight seconds rather than eight hundred milliseconds.

The design is a queue, and that is the whole trick

Detection runs at batch size one and is measured in microseconds. If explanation is on that path, the detector inherits the generator’s latency and the system is finished.

So the detection path does exactly one thing when it fires: it constructs a compact alert record and pushes it into a bounded ring buffer. It never waits, never allocates against the model, and never observes whether generation succeeded. A separate consumer drains the buffer at whatever rate the GPU allows. If the buffer fills, alerts are dropped from the explanation stream and detection continues untouched — which is the correct failure mode, since a missed explanation is an inconvenience and a missed detection is the thing you built the system to prevent.

The buffer also does the arithmetic that makes this viable at all. A burst of tens of thousands of DDoS alerts is not tens of thousands of generation calls; near-identical alerts aggregate, so a volume on the order of twenty-five thousand collapses to about ten calls. Explanation is per kind of thing happening, not per packet.

What I measure, and what I refuse to measure

The 16.6 µs p99 figure covers alert construction and queue dispatch. That is the number the detection path actually pays, and it is the number I report.

It is not generation time. It is not end-to-end time to a human-readable explanation. And it is emphatically not evidence that the explanations are any good — under the explainability checks, the free-form text quality is weak, and the project is not titled or described as validated LLM-based explainability. What has been demonstrated is a dispatch mechanism with a measured cost and a design that provably keeps the generator off the critical path. What has not been demonstrated is that the generator is worth listening to.

I am reasonably confident that stating it that way costs the project something in how impressive it sounds. It is also the only version of the sentence I can defend, and a claim you cannot defend in a review is a liability with a delayed fuse.

The general shape

This generalises past intrusion detection. Any system that wants a language model attached to a latency-critical path has the same two numbers and the same temptation to quote the small one while implying the big one.

The rules that keep it honest are short. Put the model behind a bounded queue, never a call. Make queue-full degrade the optional feature rather than the essential one. Measure and report the cost the critical path actually pays, and state plainly what that number does not include. Then — separately — be honest about whether the thing coming out the other end is any good, because the fact that it arrived cheaply says nothing at all about that.