Kubernetes Memory Limits: How I Fixed Production Crashes in 2 Hours


Yesterday, I spent 2 hours debugging why our production Kubernetes cluster kept crashing every few minutes. The pods were being evicted, the logs were cryptic, and our team was panicking. Then I realized: we forgot to set memory limits properly. Hereโ€™s exactly what I did to fix it.

The Problem: OOMKilled PodsYour Kubernetes cluster starts killing pods randomly. You check the logs and see: OOMKilled (Out of Memory). The issue? No memory limits or requests set on the deployment.

Without memory limits, Kubernetes doesnโ€™t know when to stop allowing a container to consume resources. Eventually, the node runs out of memory, and Kubernetes kills whichever pods it thinks are least important.

The Solution: Set Memory Limits and Requests

Step 1: Check Current Pod Memory Usage

First, see what your pods are actually consuming:kubectl top pods -n your-namespaceThis shows you exactly how much memory each pod is using right now. Note the valuesโ€”this is critical for setting realistic limits.

Step 2: Update Your Deployment with Memory Limits

Edit your deployment YAML and add resources section:apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
containers:
– name: app
image: my-app:latest
resources:
requests:
memory: “256Mi”
limits:
memory: “512Mi”What these mean:

  • requests: Kubernetes reserves this amount for your pod
  • limits: Pod will be killed if it exceeds this

Set limit 2x the request for safety margin.

Step 3: Apply and Monitorkubectl apply -f deployment.yaml
kubectl get pods -w # Watch for restartsWatch your cluster for 10-15 minutes. If pods are stable, youโ€™ve fixed it!

Key Takeaways

  • Always set memory limits and requests
  • Run kubectl top to find real usage
  • Set limits 1.5-2x your actual usage
  • Monitor after deployment for OOMKilled errors
  • Use vertical-pod-autoscaler for long-term optimization

Have you faced OOMKilled errors? Share your fix in the comments!

,

Leave a Reply

Discover more from inboryn

Subscribe now to keep reading and get access to the full archive.

Continue reading