The network path behind an AI response

Where the time goes

A user presses send and waits. The application records a slow response. That observation tells an operator very little about the cause. The request may have spent time establishing a connection, crossing a congested network, waiting for a worker, or processing a long prompt. Each delay calls for a different intervention.

My work at Gcore covers connectivity and traffic steering for distributed services, including AI inference. I approach performance by following the request from the user to the serving system and back. Before changing a route, I want to know which part of that trip is slow.

Follow the actual route

The endpoint a client connects to may be a gateway rather than the machine running the model. A short round trip to that gateway does not establish a short path to the GPU. The gateway may forward the request to another region, and the model may call a retrieval service somewhere else.

Measure those legs separately. Record the serving region, connection setup time, gateway-to-backend delay, queue time and model processing time where instrumentation allows. Trace identifiers should survive the handoffs. Otherwise, the network team and the application team can each produce a healthy chart while the user is still waiting.

A simple routing trade-off

Consider two hypothetical deployments of the same model. Region A has 12 milliseconds of network round-trip delay and a 180-millisecond queue. Region B has 40 milliseconds of round-trip delay and a 20-millisecond queue. Holding everything else equal, B saves 132 milliseconds before accounting for model processing: 60 milliseconds rather than 192. These are illustrative figures, not production measurements.

That comparison changes as soon as traffic moves. Sending enough requests to B increases its queue. A useful routing policy needs current load signals, limits on how much traffic it moves and a way to recover when those signals become stale.

The worker matters too

Selecting a region is only one decision. Inside it, the serving system still needs to assign work. A worker with reusable prompt state may avoid computation that another worker would repeat. That benefit has to be weighed against load and the cost of reaching it.

Gcore’s published Global Inference Routing design considers cache overlap, network latency and live load together. It is a useful example of why routing for inference needs information beyond network reachability. BGP does not carry a model’s queue depth or cache state.

Measure what the user receives

For a streaming response, measure time to first token at the client and the gaps between subsequent tokens. Also record completion time, errors and timeouts. A quick first token followed by long pauses is a different failure from a slow start followed by steady output.

Compare equivalent workloads: the same model, similar prompt and output lengths, comparable concurrency and known cache conditions. Inspect the slow end of the distribution as well as the median. Break the results down by access network and serving region; a global average can conceal a poor route affecting one provider.

A routing change should come with a specific prediction, a limited initial traffic share and a rollback threshold. If the predicted improvement does not appear at the client, follow the trace again. Moving packets faster only helps when those packets were part of the delay.