DISTRIBUTED INFERENCE
Inference latency has more than one address.
The nearest GPU is one input to a routing decision. Queueing, model readiness and serving behavior also matter.
Follow a request all the way through
For an interactive AI application, the experience includes reaching the service, waiting for work to begin and receiving the generated response. Network latency is part of that experience. So are the serving system and the workload. A shorter path cannot, by itself, guarantee a faster answer.
Separate the decisions
Choosing a region and choosing a worker inside a deployment operate at different levels. Gcore’s explanation of NVIDIA Dynamo describes separating prefill and decode and routing with awareness of the KV cache. Those serving decisions complement the connectivity that brings a request to the infrastructure in the first place.
Measure the experience you promise
For a streaming application, time to first token and the pace of subsequent output tell different stories. Compare like-for-like requests: prompt size, output length, model and concurrency can change the result. Include failures in the picture. A fast response from the successful subset does not explain what happened to the requests that timed out.
Keep the network in the conversation
A hypothetical nearby region with a long queue can lose to a more distant region with available capacity. That does not make proximity unimportant. It makes the decision conditional. The engineering challenge is to understand where delay comes from, which layer can influence it, and what happens when conditions change. Inference at scale needs both a dependable network and a serving system that uses compute effectively.