Yesterday, I spent 2 hours debugging why our production Kubernetes cluster kept crashing every few minutes. The pods were being evicted, the logs were cryptic, and our team was panicking. Then I realized: we forgot to set memory limits properly. Hereโs exactly what I did to fix it.
The Problem: OOMKilled PodsYour Kubernetes cluster starts killing pods randomly. You check the logs and see: OOMKilled (Out of Memory). The issue? No memory limits or requests set on the deployment.
Without memory limits, Kubernetes doesnโt know when to stop allowing a container to consume resources. Eventually, the node runs out of memory, and Kubernetes kills whichever pods it thinks are least important.
The Solution: Set Memory Limits and Requests
Step 1: Check Current Pod Memory Usage
First, see what your pods are actually consuming:kubectl top pods -n your-namespaceThis shows you exactly how much memory each pod is using right now. Note the valuesโthis is critical for setting realistic limits.
Step 2: Update Your Deployment with Memory Limits
Edit your deployment YAML and add resources section:apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
containers:
– name: app
image: my-app:latest
resources:
requests:
memory: “256Mi”
limits:
memory: “512Mi”What these mean:
requests: Kubernetes reserves this amount for your podlimits: Pod will be killed if it exceeds this
Set limit 2x the request for safety margin.
Step 3: Apply and Monitorkubectl apply -f deployment.yaml
kubectl get pods -w # Watch for restartsWatch your cluster for 10-15 minutes. If pods are stable, youโve fixed it!
Key Takeaways
- Always set memory limits and requests
- Run
kubectl topto find real usage - Set limits 1.5-2x your actual usage
- Monitor after deployment for OOMKilled errors
- Use vertical-pod-autoscaler for long-term optimization
Have you faced OOMKilled errors? Share your fix in the comments!