Skip to content
Posts en inglés. Usá el traductor del navegador para leerlos en tu idioma.
Featured

How to Measure LLM Performance Under Burst Traffic with Goodput

Yammbo
· 4 min read
Copyright: CC BY-NC-SA 4.0
goodput inference latency load testing llm traffic spikes ai performance metrics
How to Measure LLM Performance Under Burst Traffic with Goodput

Large Language Models (LLMs) are at the core of many modern applications, but their performance can degrade significantly under sudden surges in user demand. A viral event, a product launch, or a sudden wave of agent traffic can multiply requests to your AI application within seconds. While your service might report high requests per second (RPS), users could still face long waits or timeouts. This tutorial guides you through a practical approach to load-testing LLM inference, focusing on 'goodput' – the actual useful capacity your service delivers within a defined deadline – to truly understand how your model performs when overloaded.

Grasping the Dynamics of LLM Inference Under Load

Understanding how LLMs process requests is crucial for effective load testing. An LLM request typically involves two distinct phases: prefill and decode.

  • Prefill Phase: This is where the entire input prompt is processed at once. Longer prompts require more prefill time.
  • Decode Phase: Following prefill, the model generates output one token at a time. The length of the generated answer directly impacts decode time.

Initially, during a traffic burst, batching can improve hardware utilization. By allowing multiple requests to share each GPU step, throughput can rise. However, there's a useful capacity threshold. Once work arrives faster than the GPU can complete it, queues begin to grow. Requests age in the queue, deadlines expire, and client retries can further exacerbate the load, adding to the original demand. Simply tracking raw request counts won't reveal this bottleneck; you need deeper metrics to understand when your service is truly overwhelmed.

Defining Key Metrics for Burst Traffic Analysis

To accurately measure your LLM service's performance under burst conditions, you need to track more than just basic requests per second. Here are the critical metrics:

  • Offered RPS: This is the rate of incoming requests your load generator intends to send per second. It represents the theoretical demand.
  • Sent RPS: This is the actual rate of requests your load generator manages to send per second. It can be lower than Offered RPS if the generator itself becomes a bottleneck.
  • Successful RPS: This measures the number of requests completed each second with a full, valid response from your LLM service. This indicates the service's raw processing capability.
  • Goodput: This is the most crucial metric for user experience. Goodput represents the number of successful requests completed each second that also meet a predefined per-request deadline. A service might complete many requests, but if they are all too slow for your application's requirements, they aren't truly useful.

For example, if your load generator plans to send 100 RPS (Offered RPS) but only sends 90 (Sent RPS), and your service completes 60 requests per second with valid responses (Successful RPS), but only 45 of those finish before your application's deadline, then your goodput is 45 requests per second. Always report goodput alongside offered RPS, sent RPS, failures, and latency percentiles to get a complete picture. A service that quickly rejects requests might appear fast but serves very few users effectively.

Preparing Your Inference Environment for Testing

The type of inference deployment you are testing significantly impacts what you can observe and control during a burst test. Consider these common deployment models:

  1. Managed Serverless Inference: In this setup, a cloud provider manages the underlying infrastructure. You typically interact with an API (e.g., an OpenAI-compatible endpoint). Your control is primarily over client behavior (admission control, deadlines, bounded retries). You can observe client-side timings, status codes, and quota headers. A burst test here measures the capacity available to your account, model, and workload at the time of the test, within the provider's shared resource limits.
  2. Dedicated Cloud Inference Instances: Here, you provision specific GPU plans or accelerator counts. You have more control over the model deployment and endpoint access. You can observe client timings and deployment-specific status. A burst test helps determine the maximum capacity of your chosen dedicated configuration.
  3. Self-Managed Inference (e.g., on Virtual Machines or Kubernetes): This offers the highest level of control. You manage the inference engine, batching strategies, concurrency settings, and scaling policies. You can integrate detailed engine metrics, traces, and observe scaling events. Burst testing in this environment allows you to pinpoint where queues form, understand the impact of specific configuration changes, and fine-tune your resource allocation.

Before testing, clearly identify your deployment type and understand its inherent limitations and observability features. This will guide your test design and interpretation of results.

Designing and Implementing Your Load Test

A well-designed load test should simulate realistic burst conditions. An