What do you do when APM cannot trace the TCP library that more than 200 of your services share? A real-money gaming operator had this problem. The platform handles more than 180M transactions a day, with peaks of 600,000 requests per second on more than 30,000 VMs. It took 10 to 15 minutes to detect business problems.

The shared library was built on Netty, with a binary protocol and asynchronous event loops. Commercial APM agents could not follow requests through it. Nobody had changed it in five years, because changes were risky. Our team made the following changes: • They added OpenTelemetry client and server spans inside the shared library. • They carried W3C trace context through the binary protocol. One framework change gave end-to-end traces for every service. • Used AI to map the Netty pipeline and the event-loop hand-offs before the code change. Then they benchmarked the change on Java 8. • They replaced poll-based checks with four streaming metrics: spins, bets, wins and revenue.

Understanding FP8 and NVFP4 for AI Inference , and why does it matter ? What do FP8 and NVFP4 actually mean, and why do they matter for AI inference? FP stands for floating point, a way to represent numbers. The number after FP tells you how many bits are used to represent each value. FP8 uses 8 bits. FP4 uses 4 bits. NVFP4 is NVIDIA's 4-bit floating-point format, designed for efficient AI computation on Blackwell GPUs. Let's see what this means for memory and inference.

Why does using fewer bits matter? First, memory capacity. Smaller weights leave more GPU memory available for the KV cache, which stores information about the tokens processed in each conversation. That can allow more concurrent requests. Second, data movement. Reading fewer bytes of weights can improve inference throughput, depending on the workload and hardware. But the GPU must support the format efficiently. NVIDIA Blackwell GPUs support native FP4 Tensor Core operations. Hopper GPUs, including H100 and H200, support FP8 but lack native FP4 Tensor Core support.

Understanding total and active parameters Four open-weight models were announced in the first week of October. Each model card shows two numbers. Together they decide the hardware that you need to run the model yourself.

The two large models arrive later this month: Mistral Large 4 (1.05T total, 52B active) and Reflection's Beam (501B total, 23B active). For these, the number format decides the GPU: • FP8, 1 byte for each parameter: Mistral Large 4 needs about 1.05 TB. Eight H200s (1,128 GB) leave only about 78 GB for users. • NVFP4, about half a byte: about 590 GB. Eight B200s (1,440 GB) leave about 850 GB.

Understanding what a desktop AI machine can serve NVIDIA's DGX Spark has 128 GB of memory, and NVIDIA says it runs models up to 200B parameters. Quiz: how many tokens per second does a 70B model make on it? The answer is in this thread, part 4 of our inference series.

For each token, the model reads all of its weights. So the speed for one user is the bandwidth divided by the size of the weights: • 70B model in NVFP4, about 40 GB: about 7 tokens/s. That is the quiz answer. • Llama 3.1 8B in FP8, about 8 GB: about 34 tokens/s • Kolibri-1, a mixture-of-experts (MoE) model: about 79 tokens/s An MoE model uses only a few of its parameters for each token. Kolibri-1 has 78B, but only about 3.5B are active, so it reads less memory.

If models releases are bothering you, here's something - 1/3 Sonnet 5.5 (28 Sep), GPT-6.1 Sol (29 Sep) and Gemini 4 Argon (30 Sep, gated access) launched around the same price: $2/M input tokens and $10/M output. Same list price doesn't mean the same cost per task. Interested? let's dive in

If you're routing work across providers, run the same labelled task set through each model. Record: - Input and output tokens - Cache hits - Pass rate against your acceptance criteria - Cost per successful task - Latency at the percentile that matters to your users