Dynamic Kubernetes Cluster Scaling on Application and Infrastructure Levels

Why Kubernetes Scaling Really Matters

Real‑world traffic is unpredictable:

• Day/night traffic patterns
• Marketing spikes
• Batch jobs & cron workloads
• Sudden failures or rollouts

If your Kubernetes cluster doesn’t scale automatically, you either:

• Pay for idle nodes
• Face pod failures during traffic spikes
• Manually scale infrastructure (slow & risky)

Dynamic scaling ensures:

• Performance under load
• Cost efficiency
• Zero manual intervention

Kubernetes scaling happens at two layers:

1. Application (Pods)
2. Infrastructure (Nodes)

Scaling Kubernetes on Infrastructure Level 

• A Kubernetes cluster typically consists of a master (or a couple of them) and multiple nodes where application pods are scheduled. The maths is quite simple here: the more applications you run in your cluster, the more resources (nodes) you need. Say, you have a microservices application consisting of 3 services, each started as an individual pod which requests 1GiB of RAM. 

• It means that you will need a 4 GiB node (K8s components and OS will require some RAM too). What if you need additional RAM in case of high load, potential memory leaks or if you deploy more services to the cluster? Correct, you either need a larger node or add an additional node to the cluster. 

• Usually in both cases, you will pay for the exact amount of resources that come with a VM (i.e. you will pay for, say, 3GiB of RAM even if half of it is unused). That’s not the case with Infrastructure though.

Vertical Scaling of Kubernetes Nodes

• Let’s get back to maths. If the application roughly needs 3GiB of RAM, and there’s not much going on in the cluster, you need just one node. Sure thing, having some extra free RAM is always a good idea, so a 5GiB node makes a lot of sense.

• Again, not with Infrastructure. What you can do is request a 3GiB node and have a 2GiB in stash. When your application (K8s pod) starts consuming more (which is configured on K8s side too) or you simply deploy more pods (as in the chart below), those 2 extra GiB become immediately available, and you start paying for those resources only when they are used.

• As a result, you can do some simple math and figure out the best cluster topology: say, 3 nodes, with 4 GiB of reserved RAM and 3GiB of dynamic resources.

               

Horizontal AutoScaling of Kubernetes Nodes

• Having one huge node in a Kubernetes cluster is not a good idea since all deployments will be affected in case of an outage or any other major incident. 
• Having several nodes in a stand-by mode is not cost efficient. Is it possible that Kubernetes adds a node when it needs it? Yes, a Kubernetes cluster in Infrastructure can be configured with horizontal node auto-scaling. 
• New nodes will be added to a cluster when RAM, CPU, I/O or Disk usage reaches certain levels. Needless to say, you get billed for additional resources only when they are used. Newly added nodes will be created according to current topology, i.e. existing vertical scaling configurations will be applied. 
• The system will scale down as soon as resource consumption gets back to expected levels. Your Kubernetes cluster will not starve, yet you will not pay for unused resources.

Prerequisites

• Metrics Server installed

kubectl apply -f 
https://github.com/kubernetes-sigs/metrics-server/releases
/latest/download/components.yaml

Example: CPU‑based HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Scaling Kubernetes on Application Level 

• Kubernetes has its own horizontal pod auto-scalers (HPA). In simple words, HPA will replicate chosen deployments based on utilization of CPU. If CPU consumption of all pods grows more than, say, 70%, HPA will schedule more pods, and when CPU consumption gets back to normal, deployment is scaled back to the original number of replicas.

• Why is this cool and how does it work with automatic horizontal scaling of Kubernetes nodes? Say, you have one node with a couple of running pods. All of a sudden, a particular service in the pod starts getting a lot of requests and performing some CPU costly operations. RAM utilization does not grow, and as a result at this point there is no mechanism to scale the application which will soon become unresponsive. 

• Kubernetes HPA will scale up pods, and an internal K8s load balancer will redirect requests to healthy pods. Those new pods will require more resources, and this is where Infrastructure horizontal and vertical scaling comes into play. New pods will be either placed on the same node and utilize dynamic RAM, or a new node will be added (in case there’s not enough resources on existing ones).

• On top of that, you may set resource caps on Kubernetes pods. For example, if you know for sure that a particular service should not consume more than 1GiB, and there’s a memory leak if it does, you instruct Kubernetes to kill the pod when RAM utilization reaches 1GiB. A new pod will start automatically. This gives you control over resources your Kubernetes deployments utilize.

Cluster Autoscaler Improvement
Custom gRPC Expander
The most significant improvement we made to Cluster Autoscaler was to provide a new method for determining node groups to scale. Internally, Cluster Autoscaler maintains a list of node groups which map to different candidates for scaling, and it filters out node groups that do not satisfy pod scheduling requirements by running a scheduling simulation against the current set of Pending (unschedulable) pods. 
If there are any Pending (unschedulable) pods, Cluster Autoscaler attempts to scale the cluster to accommodate these pods. Any node groups that satisfy all pod requirements are passed to a component called the Expander.

The Expander is responsible for further filtering the node groups based on operational requirements. Cluster Autoscaler has a number of different built-in expander options, each with different logic. 
For example, the default is the random expander, which selects from available options uniformly at random. Another option,and the one that Airbnb has historically used, is the priority expander, which chooses which node group to expand based on a user-specified tiered priority list.
Event-Driven Autoscaling with KEDA
KEDA (Kubernetes Event-Driven Autoscaling) allows Kubernetes workloads to scale based on external events, not just CPU or memory.
Why KEDA is Needed (Real Problem)
Traditional scaling (HPA) fails in cases like:
Scenario
Problem with HPA
Kafka consumers.         
  CPU may be low, but lag is high     
Job queues
  Work piling up but no CPU spike
Async workers
   No traffic → no scaling trigge

Key Advantages
Scale based on actual work
Supports external systems
Can scale to zero (cost savings)Ideal for async microservices
Example: KEDA ScaledObject 
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: queue-scaler
spec:
  scaleTargetRef:
    name: worker-app
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
  - type: rabbitmq
    metadata:
      queueName: orders
      host: amqp://guest:guest@rabbitmq:5672/
      queueLength: "50"
Modern Kubernetes Scaling: Karpenter 
Why Karpenter Is Better
Scales in seconds
No node groups required
Dynamic instance selection
Massive cost savings
Key Idea
Instead of scaling node groups, Karpenter:
Directly provisions right-sized nodes
Based on actual pod requirementsWhy Karpenter is Powerful
Faster node provisioning
No need for pre-defined node groups
Better bin-packing (cost optimization)
Works well with spot instances
Karpenter Architecture
Karpenter Architecture
Pod becomes Pending
Karpenter selects best EC2 instance
Launches node
Pod schedules immediately
Node is terminated when unused
Example: Karpenter NodePool
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general-purpose
spec:
  limits:
    cpu: "200"
  disruption:
    consolidationPolicy: WhenUnderutilized
  template:
    spec:
      requirements:
      - key: node.kubernetes.io/instance-type
        operator: In
        values: ["c7i.large", "m7i.large"]
Example: EC2NodeClass
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: KarpenterNodeRole
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: my-cluster
✅ Best for microservices, batch jobs, CI/CD, event‑driven traffic
✅ Excellent bin‑packing → lower cost

Cost Optimization: Real Impact

Setup
Monthly Cost
Notes
    Static Nodes.          
₹95,000.            
Huge idle time.       
Cluster Autoscaler
₹72,000
Slow recovery
Karpenter
₹48,000
Fast + efficient

Cluster Autoscaler vs Karpenter

Feature
HPA
Cluster Autoscaler
Karpenter
KEDA
Scales
Pods
Nodes
Nodes
Pods
Trigger
CPU / Memory.  
Pending pods
Scheduling needs.  
Events / external metrics     
Speed
Fast
Medium
Fastest
Fast
Scale to zero
❌ No
❌ No
❌ No
✅ Yes
Best for
Web apps
Infra scaling
Dynamic infra   

Step-by-Step Flow
User traffic increases or events begin flowing
Application demand rises due to increased requests, background jobs, or event streams.
Application load increases
Load is reflected through CPU usage, request rate, queue length, or consumer lag, depending on the workload type.
Horizontal Pod Autoscaler scales pods
HPA detects the increased metrics and scales the number of pod replicas to handle the additional load.
Insufficient node capacity detected
If existing nodes cannot accommodate the new pods, those pods enter a Pending state.
Infrastructure autoscaler provisions nodes
Cluster Autoscaler or Karpenter detects scheduling pressure and provisions new nodes that meet the workload requirements.
Pods are scheduled on newly provisioned nodes
Once nodes join the cluster, pending pods are successfully scheduled and start processing traffic.
System stabilizes under load
Application performance normalizes as capacity aligns with demand.
Load decreases
When traffic or event volume drops, resource utilization falls below scaling thresholds.
Scale‑down occurs
HPA reduces pod replicas
Cluster Autoscaler or Karpenter removes underutilized nodes safely
Real-World Challenges

1. Cold Start Problem
Issue:
New pods and nodes require time to initialize, which can cause latency spikes during sudden traffic increases.
Mitigation:
Maintain buffer capacity using minimum replicas or node pools
Pre‑warm nodes using scheduled scaling or paused workloads
2. Metrics Delay (HPA Lag)
Issue:
Horizontal Pod Autoscaler reacts only after metrics reflect increased load, leading to delayed scaling.
Mitigation:
Use custom or external metrics (for example, queue depth or request rate)
Apply predictive or scheduled scaling for known traffic patterns
3. Over‑Scaling and Cost Impact
Issue:
Aggressive scaling thresholds may result in unnecessary nodes, increasing infrastructure costs.
Mitigation:
Fine‑tune HPA and autoscaling thresholds
Use Karpenter for efficient instance selection and bin‑packing
4. Pending Pods Due to Resource Mismatch
Issue:
Pods remain in a pending state because requested resources do not match available node capacity.
Mitigation:
Define accurate CPU and memory requests
Allow multiple instance types to improve scheduling flexibility
5. Event Spikes (Queue Flooding)
Issue:
Sudden spikes in event volume can overwhelm consumers before scaling occurs.
Mitigation:
Use KEDA for event‑driven autoscaling
Apply rate limiting and back‑pressure mechanisms at the consumer level
Production Best Practices

1. Combine All Scaling Layers
Relying on a single autoscaling mechanism is insufficient for production workloads. Effective scaling requires coordination across multiple layers.
Recommended approach:
HPA: Real‑time pod scaling based on resource or custom metrics
KEDA: Event‑driven pod scaling for queues, streams, and asynchronous workloads
Cluster Autoscaler or Karpenter: Infrastructure scaling to ensure adequate node capacity
This layered strategy ensures fast reaction to demand while maintaining infrastructure efficiency.
2. Always Define Resource Requests and Limits
Every production workload must have explicit CPU and memory requests and limits.
Without proper resource definitions:
Kubernetes scheduling decisions become unreliable
Autoscalers cannot make accurate scaling decisions
Pods may over‑consume resources, impacting cluster stability
Defining requests and limits is foundational for predictable scaling behavior.
3. Use Custom and Business‑Relevant Metrics
Default CPU and memory metrics are often insufficient for real‑world workloads.
Examples of effective custom metrics:
HTTP request rate
Queue length or consumer lag
Request latency or processing time
Custom metrics allow scaling decisions to align with actual workload pressure rather than infrastructure signals alone.
4. Maintain Slight Overprovisioning
Running at strict minimum capacity increases the risk of service disruption during sudden load changes.
Best practice:
Maintain a small buffer of unused capacity
Configure minimum replicas or baseline node pools
Avoid scaling from zero for latency‑sensitive services
This approach reduces cold‑start impact and improves system resilience