Kubernetes in Production: The Definitive Day-2 Architecture Checklist

Running Kubernetes in staging is straightforward; running it in production without 3 AM wake-up calls requires systematic hardening. Most cluster outages stem from mundane configuration gaps: missing resource requests, unbounded namespace consumption, unbudgeted pod disruptions, or unmonitored etcd latency.

This field guide outlines the battle-tested Day-2 architecture checklist implemented across tier-1 infrastructure teams.

1. Pod Disruption Budgets (PDB) & Graceful Termination

Every stateless service running multiple replicas must define a PodDisruptionBudget. Without a PDB, routine cluster upgrades (such as worker node AMI rolling replacements) can drain nodes simultaneously, causing downtime.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-gateway-pdb
spec:
  minAvailable: 75%
  selector:
    matchLabels:
      app.kubernetes.io/name: api-gateway

Additionally, align your application's preStop hook and terminationGracePeriodSeconds. Ingress controllers and kube-proxy take between 3 to 10 seconds to detach an endpoint from the routing table after pod deletion begins.

2. Resource Requests, Limits & Quality of Service (QoS)

Never deploy a pod without explicit requests and limits. If requests are omitted, the Kubernetes scheduler treats CPU and memory demands as zero, leading to node oversubscription and aggressive kernel OOM (Out Of Memory) kills.

• Guaranteed QoS: requests.cpu == limits.cpu and requests.memory == limits.memory. Reserve this QoS class for stateful databases, latency-critical RPC services, and cache tiers.
• Burstable QoS: requests.cpu < limits.cpu. CPU is compressible; the Linux CFS bandwidth controller will throttle cycles rather than kill the process. Memory is non-compressible: if you breach limits.memory, the pod terminates immediately with exit code 137.

3. Network Policies and Namespace Isolation

By default, Kubernetes clusters operate on a flat network where any pod can communicate with any other pod across namespaces. Implement default-deny ingress and egress policies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Whitelisting must explicitly authorize traffic between the frontend gateway and downstream services, blocking lateral movement during vulnerability exploits.