Skip to content
Work Writing About Contact

Work / Slipstream

active 2026

Slipstream

An LLM inference engine built from scratch, where the research sits in the scheduler — an output-length-predictive policy that decides which sequences get to occupy a full KV cache.

  • LLM
  • Inference
  • Scheduling
  • Systems
  • Triton
Code

Figures

1 figure · conditions stated on every plot

Fig. 01 schematic

Schematic roofline diagram with no measured points on it. The horizontal axis is arithmetic intensity in FLOP per byte on a base-10 logarithmic scale, from 1 to 10,000; during decode it can also be read as the number of concurrent sequences, because arithmetic intensity is roughly equal to batch size. The vertical axis is attainable performance as a fraction of peak and carries no measured numbers. The curve rises with intensity and then flattens at the compute ceiling; the corner, the ridge point, is at about 153 FLOP per byte on an A100. Everything to the left of that corner is the bandwidth-bound region, which is where decode runs: the write-up states the GPU stays bandwidth-starved until something like 150 sequences run concurrently. The curve is the roofline model, not a measurement.

schematic — roofline model · no measured points

1 10 100 1,000 10,000

arithmetic intensity — FLOP/byte · log₁₀ scale

≈ concurrent sequences during decode · arithmetic intensity ≈ batch size

  • roofline model — min(peak, bandwidth × intensity)
  • compute ceiling — peak FLOP/s, unitless here
  • ridge point ≈ 153 FLOP/byte · A100
  • bandwidth-bound — where decode runs
  • Schematic. The curve is the roofline definition itself — attainable performance is min(peak, bandwidth × arithmetic intensity) — not a measurement. No achieved FLOP/s figures are published for this project, so none are plotted and the y axis carries no numbers.
  • "During decode, arithmetic intensity is roughly equal to batch size. On an A100 — ridge point around 153 FLOP/byte — the GPU stays bandwidth-starved until something like 150 sequences run concurrently." — the project write-up.
  • The only measured figures this project publishes are on a laptop RTX 3050 running Qwen2.5-0.5B at n ≈ 3, which the repository itself calls a sketch. None of them appear on this figure.
Roofline model around the published A100 ridge point · schematic 2026

The thesis

Serving an LLM looks like a compute problem and behaves like a memory-management problem.

During decode, arithmetic intensity is roughly equal to batch size. On an A100 — ridge point around 153 FLOP/byte — the GPU stays bandwidth-starved until something like 150 sequences run concurrently. What stops you reaching that batch size is the KV cache, and naive allocation wastes most of the memory that determines it. Fix the memory system and throughput follows. Then the bottleneck moves again, to the policy deciding which requests are admitted and which are evicted — and that is where this project’s actual contribution lives.

Horizon

Production engines schedule first-come-first-served and are blind to how long a request will run. Horizon predicts remaining output length online and schedules for goodput — the share of requests that stay under their latency target for their whole lifetime — rather than raw tokens per second. Non-clairvoyant scheduling under hard KV-cache constraints is an open problem, not a solved one, which is why it is worth building rather than importing.

The engine underneath it is written from scratch so the scheduler has something honest to sit on: PagedAttention, continuous batching, chunked prefill, prefix caching, custom Triton kernels, speculative decoding, tensor parallelism, and prefill/decode disaggregation.

Where it actually stands

Phase 4, and everything measured so far is on development hardware — a laptop RTX 3050 running Qwen2.5-0.5B, n≈3 seeds. Those runs are a sketch, not a result table, and the repository says so in its first paragraph.

What the sketch shows is that the policy does the thing it was designed to do. Under a KV-bound burst, FCFS over-admits and then preempts repeatedly; Horizon refuses the overflow and preempts nothing, which keeps its inter-token-latency tail flat instead of letting it blow out. It wins on smooth goodput on every seed tried. It also loses on raw throughput, and misses its fairness target on most seeds — both of which are real and neither of which is hidden. An ablation is the most encouraging part: a mean-only length predictor keeps a small fraction of requests smooth where the full feature set keeps all of them, which suggests the prediction is carrying the result rather than decorating it.

None of that is the claim the project is aiming at. The targets are written against A100-class hardware and a real chat workload, and until those runs exist there is no headline number here.

Companion work

qgemm-mx is the same thesis one layer down. Both start from the decode path being bandwidth-bound; qgemm-mx attacks that at the kernel level, Slipstream at the system level, where allocation and scheduling policy set the achievable batch size. A fast block-scaled FP4 GEMM is exactly the kind of kernel that plugs into a quantized decode path.

Stack

Python · PyTorch · Triton · CUDA · PagedAttention · speculative decoding · W4A16 quantization