How Pod Scaling Actually Works

What is the autoscaler actually doing?

It looks complicated but it is one proportion:

desired_pods = current_pods × (current_value / target_value)

# example: 4 pods, average CPU 90%, target 60%
# desired = 4 × (90 / 60) = 6 pods

That is all. The whole question is what you chose as "current_value". And most teams choose CPU and stop thinking about it.

Why is CPU usually the wrong metric?

Look at the life of a typical web request: 3 ms parsing JSON, 80 ms waiting on the database, 2 ms serialising. The application spends 95% of its time waiting, and waiting costs no CPU.

So the system can be on its knees while CPU shows 25%. The autoscaler concludes everything is fine and does nothing.

# with KEDA: one worker per 100 pending messages
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  scaleTargetRef:
    name: notification-worker
  minReplicaCount: 2
  maxReplicaCount: 40
  triggers:
    - type: rabbitmq
      metadata:
        queueName: notifications
        queueLength: "100"

requests and limits: the two most misconfigured fields

Confuse these two and scaling is computed wrongly from the start.

resources:
  requests:
    cpu: 100m        # ← it actually uses 400m
  limits:
    cpu: 1000m

# Say the HPA target is 70%. The maths uses the request:
# 400m / 100m = 400% utilisation
# desired = 4 × (400 / 70) ≈ 23 pods
#
# The application is perfectly comfortable. The autoscaler panicked.

Practical rule: the request should be slightly above real usage at normal load. Not guessed, measured. Look at the 90th percentile of real usage over a week and write that.

About CPU limits

A process hitting its CPU limit is not killed, it is throttled. That is one of the hardest latency sources to diagnose: no errors, clean logs, and the CPU graph even looks below the limit — while requests stall.

The reason is that throttling is applied over very short windows. Even if your average is below the limit, short bursts get you stopped for milliseconds at a time. The only way to see it is to look at the throttling metric directly:

container_cpu_cfs_throttled_seconds_total

# If this keeps climbing, your CPU limit is too low.
# A common answer for latency-sensitive services is to set
# no CPU limit at all (a request alone is enough).

The real trap: the bottleneck underneath

Back to the opening story. Pods went from 4 to 24. Each pod opened a pool of 20 connections to the database.

4 pods  × 20 connections =  80  ✓ (limit 200)
24 pods × 20 connections = 480  ✗ (limit 200)

Result: the database refuses new connections.
The application throws connection timeouts.
Health checks fail, pods are restarted.
Restarted pods ask for 20 connections again...

That is a vicious circle, and scaling accelerates it. Had the system never scaled, it would have been slow but alive.

Three defences:

1. Size the pool against pod count. pool_per_pod × max_pods ≤ database_limit × 0.8. If you target 24 pods, the pool should be 6, not 20.
2. Use a connection pooler. A layer like PgBouncer reduces hundreds of application connections to tens of real ones. It is the one structural change that makes scaling possible.
3. Do not pick maxReplicas arbitrarily. That number should be derived from what the weakest link below can take. The autoscaler's upper bound is a seatbelt; "let us put 100 to be safe" is taking the belt off.

The slow-starting application problem

We hit this on a Java service: a new pod comes up, reports itself ready, starts taking traffic — and answers very slowly for the first 30 seconds (JIT warm-up, cache filling, connections being established).

Result: the new pod raises average latency, the autoscaler reads that as insufficient capacity and starts another pod, which also starts slowly... This is called flapping and it looks like a sawtooth on a graph.

# 1) Do not take traffic until warm
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  initialDelaySeconds: 20
  periodSeconds: 5

# 2) Do not mistake a slow start for death (breaks the restart loop)
startupProbe:
  httpGet: { path: /health, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5

# 3) Give the autoscaler time to cool down
behavior:
  scaleUp:
    stabilizationWindowSeconds: 60
  scaleDown:
    stabilizationWindowSeconds: 300

The third is especially important: scaling down should be slower than scaling up. If traffic fluctuates, a system that shrinks quickly will reopen the same pods two minutes later and pay the cold-start cost again and again.

Losing work while scaling down

The least discussed part of scaling posts. When a pod is terminated it may still hold half-finished work.

When is vertical growth the better answer?

Horizontal is not the answer to everything. Adding pods will not help when:

• A single request is slow. An 8-second query takes 8 seconds with 40 pods too.
• The app holds a large in-memory cache. Every pod keeps its own copy; the waste multiplies.
• There is a per-connection or per-instance licence cost. Replica count goes straight onto the invoice.

In those cases you need to fix the query, the cache or the algorithm first. Scaling is a capacity solution, not a performance solution.

Checklist AI agent task list

Conclusion

Autoscaling set up well means sleeping at night; set up badly it is a mechanism that accelerates a collapse. The difference is not the number of pods but which number you watch and whether you protected what is underneath.

In the incident above, the fix was fewer pods — we went from 24 down to 8, put a connection pooler in front of the database, and changed the metric from CPU to concurrent requests. The system got faster on fewer resources.