Intro

HTTP (Hypertext Transfer Protocol) is a stateless request-response protocol that carries nearly all web and API traffic in modern systems. You reach for it every time a client talks to a server over the network — browser page loads, REST APIs, webhooks, health checks, file downloads. Understanding HTTP mechanics — connection lifecycle, caching semantics, status codes, and TLS — directly affects the latency, reliability, and security of every service you build. In .NET, HTTP is the transport underneath HttpClient, ASP.NET Core, and gRPC (over HTTP/2), making it foundational knowledge for senior .NET work.

Request-Response Lifecycle

An HTTP/1.1 exchange follows this sequence:

sequenceDiagram
  participant Client
  participant Server

  Client->>Server: TCP handshake SYN SYN-ACK ACK
  Client->>Server: TLS handshake if HTTPS
  Client->>Server: HTTP request method path headers body
  Server->>Client: HTTP response status headers body
  Note over Client,Server: Connection stays open via keep-alive
  Client->>Server: Next request reuses connection

Persistent Connections

HTTP/1.0 opened a new TCP connection per request — catastrophically expensive. HTTP/1.1 made persistent connections the default: the same TCP socket is reused for multiple request-response cycles. The Connection: keep-alive header controls idle timeout and max requests. Idle connections still consume server-side file descriptors and memory, so servers close them after a configurable timeout.

Head-of-Line Blocking

HTTP/1.1 processes requests sequentially on each connection: the response to request #1 must complete before request #2 begins. If request #1 triggers a slow database query, requests #2 and #3 wait behind it. Browsers work around this by opening up to 6 parallel TCP connections per origin, but this wastes resources and still hits limits. HTTP 2 solves this with stream multiplexing.

Pipelining was an HTTP/1.1 attempt to fix HOL blocking by sending multiple requests without waiting for responses. It failed in practice because responses still had to arrive in order, and buggy intermediary proxies mishandled pipelined responses. Browsers disabled it by default.

HTTPS and TLS

HTTPS is HTTP over TLS. It provides confidentiality (encryption), integrity (tamper detection), and server authentication via certificates. Plain HTTP has no protection against eavesdropping or manipulation.

TLS 1.2 Handshake (2 RTTs)

Client → Server: ClientHello (supported ciphers, random)
Server → Client: ServerHello, Certificate, ServerHelloDone
Client → Server: ClientKeyExchange, ChangeCipherSpec, Finished
Server → Client: ChangeCipherSpec, Finished
→ Application data begins (2 RTTs after TCP handshake)

The client validates the certificate chain: leaf cert → intermediate CA(s) → trusted root CA. Each cert is verified for signature validity, expiry, revocation status (OCSP or CRL), and hostname match against the SAN (Subject Alternative Name) field.

TLS 1.3 (1 RTT)

TLS 1.3 (RFC 8446) reduced the handshake to 1 RTT and eliminated weak primitives:

• All cipher suites provide forward secrecy — ephemeral Diffie-Hellman only, no static RSA key exchange
• Handshake messages are encrypted — the certificate is not visible to network observers
• 0-RTT resumption for repeat connections, but replay-vulnerable — safe only for idempotent requests
• Removed RC4, 3DES, CBC mode, and RSA key exchange entirely

.NET TLS Guidance

Use SslProtocols.None to defer TLS version selection to the OS — never hardcode SslProtocols.Tls12. The legacy SslProtocols.Default maps to SSL3 + TLS 1.0, which is insecure. ServicePointManager.SecurityProtocol is process-global and affects all HttpClient instances — a footgun in shared libraries that silently downgrades TLS for the entire process.

Methods and Semantics

Safe means the request does not change server state. Idempotent means repeating the same request produces the same resulting state — not the same response, but the same server-side outcome. Clients can safely auto-retry idempotent methods on ambiguous network failure, which is why PUT and DELETE are preferable over POST when the operation allows it.

Status Codes

Commonly Misused Codes

Caching

HTTP caching prevents redundant data transfer by storing responses and reusing them for matching requests.

Cache-Control

Cache-Control: max-age=3600, must-revalidate, public

Key directives:

• max-age=N — fresh for N seconds; no revalidation until expired
• no-cache — store the response but always revalidate before serving (does NOT mean "don't cache")
• no-store — never store the response anywhere (for sensitive data like banking pages)
• private — only browser cache, not shared or CDN caches
• public — storable in shared caches (required when Authorization header is present and caching is intended)
• immutable — content will never change; skip revalidation even on reload (use with cache-busted URLs like app.abc123.js)

Heuristic caching trap: if no Cache-Control is set, RFC 9111 allows caches to apply heuristic freshness (~10% of Last-Modified age). Always set explicit Cache-Control headers — especially on API responses.

Conditional Requests and ETag

# Server sends ETag with original response
ETag: "33a64df5"

# Client revalidates later
If-None-Match: "33a64df5"

# Server responds 304 if unchanged — no body transferred
HTTP/1.1 304 Not Modified

ETags are preferred over Last-Modified because Last-Modified has only 1-second granularity and distributed servers struggle to synchronize file timestamps. If-None-Match takes precedence over If-Modified-Since when both are present (RFC 9110).

Vary Header

Vary: Accept-Encoding tells caches to key on both URL and the listed request headers. Without it, a CDN might serve a gzip-compressed response to a client that only accepts brotli. Vary: User-Agent is a cache-killing anti-pattern — thousands of User-Agent variants effectively disable caching.

Pitfalls

1) HttpClient Socket Exhaustion in .NET

• What goes wrong: creating a new HttpClient per request exhausts sockets. Dispose() does not immediately release the underlying socket — the OS holds it in TIME_WAIT state (~4 minutes on Windows). Under load, ephemeral ports run out → SocketException.
• Why it happens: HttpClient wraps a connection pool in SocketsHttpHandler. Disposing the client tears down the pool, but the OS keeps TCP sockets in TIME_WAIT for reliable connection termination.
• Mitigation: use IHttpClientFactory (default handler lifetime: 2 minutes, handles both pooling and DNS refresh) or a singleton HttpClient with SocketsHttpHandler.PooledConnectionLifetime set to force periodic connection recycling.

2) DNS Staleness with Singleton HttpClient

• What goes wrong: a singleton HttpClient resolves DNS once and caches the IP for the lifetime of the connection. In Kubernetes where pod IPs change on redeploy, requests route to terminated pods.
• Why it happens: the connection pool reuses established TCP connections without re-resolving DNS. There is no mechanism to honor DNS TTL at the connection level.
• Mitigation: set PooledConnectionLifetime = TimeSpan.FromMinutes(2) on SocketsHttpHandler to force connection recycling and DNS re-resolution. IHttpClientFactory does this automatically via handler lifetime rotation.

3) Heuristic Caching Surprises

• What goes wrong: API responses without explicit Cache-Control headers get cached by intermediate proxies using heuristic rules, serving stale data to clients.
• Why it happens: RFC 9111 permits caches to infer freshness from Last-Modified when no explicit freshness directive is present.
• Mitigation: always set explicit Cache-Control headers on API responses. Use no-store for sensitive data, no-cache when revalidation is acceptable, or a specific max-age when staleness is tolerable.

Tradeoffs

Decision rule: HTTP/1.1 for maximum compatibility and simplicity. Upgrade to HTTP/2 when multiplexing and header compression matter (most modern services benefit). HTTP/3 for latency-sensitive, loss-prone networks like mobile clients and global CDN edges.

Questions

Links

• RFC 9110 — HTTP Semantics
• RFC 9111 — HTTP Caching
• RFC 8446 — TLS 1.3
• MDN — HTTP Caching
• MDN — Connection Management in HTTP/1.x
• Microsoft Learn — Guidelines for Using HttpClient
• Microsoft Learn — IHttpClientFactory
• You're Using HttpClient Wrong (ASP.NET Monsters)