Field Guide: Eliminating p99 Tail Latency in Edge APIs

Incident Analysis & Diagnostic Methodology

During peak traffic events, while median (p50) latency may hover around a crisp 18ms, the 99th percentile (p99) often balloons past 600ms. In high-concurrency systems, a 1% failure or delay rate affects thousands of users per minute.

Tactical Resolution Strategy

1. HTTP/3 & 0-RTT Connection Resumption

Enabling 0-RTT connection handshakes avoids renegotiating cryptographic keys when repeat clients make new requests:

// Edge handler configuration with connection reuse
const originResponse = await fetch(originUrl, {
  cf: {
    cacheTtl: 300,
    cacheEverything: true,
    mirroredRequest: false
  }
});

2. Dual-Query Hedging Pattern

If an upstream database replica fails to respond within 45ms, immediately fire a duplicate query to a secondary regional replica and return whichever responds first:

async function fetchWithHedging(url: string, timeoutMs: number = 45): Promise<Response> {
  const controller = new AbortController();
  const primaryPromise = fetch(url, { signal: controller.signal });
  
  const timerPromise = new Promise((resolve) => setTimeout(resolve, timeoutMs));
  const winner = await Promise.race([primaryPromise, timerPromise]);

  if (winner instanceof Response) {
    return winner;
  }

  // Backup hedge request fired
  const hedgePromise = fetch(`${url}?backup=true`);
  return Promise.race([primaryPromise, hedgePromise]);
}

Takeaway Summary

By combining connection pre-warming, HTTP/3 QUIC transport, and request hedging, our production p99 latency dropped from 620ms down to 48ms.