Problem Description
By default,
- Containers run with unbounded compute resources on a Kubernetes cluster.
- If pod/ containers gone rouge and can monopolize resources, it can kill the entire cluster.
Solutions:
Use k8s features like Resource Quotas, LimitRange, Request/Limits to restrict this behavior. In this trashcan article, I am going through official blog to spike LimitRange feature.
Official page mentioned 4 types of resources,
- Limiting Container compute resources
- Limiting Pod compute resources
- Limiting Storage resources
- Limits/Requests Ratio
I am going to try all 4.
Limiting container compute resources
- Create a namespace limitrange-demo using the following kubectl command:
kubectl create namespace limitrange-demo
- Create a configuration file (limitrange.yaml) for limit range objects
apiVersion: v1
kind: LimitRange
metadata:
name: limit-mem-cpu-per-container
spec:
limits:
- max:
cpu: "800m"
memory: "1Gi"
min:
cpu: "100m"
memory: "99Mi"
default:
cpu: "700m"
memory: "900Mi"
defaultRequest:
cpu: "110m"
memory: "111Mi"
type: Container
- Now apply this limit range configuration file
kubectl apply -f limitrange.yaml -n limitrange-demo
Output:
limitrange/limit-mem-cpu-per-container created
** You can validate if the object is properly created using the following command
kubectl describe -f limitrange.yaml -n limitrange-demo
Output:
Name: limit-mem-cpu-per-container
Namespace: limitrange-demo
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container memory 99Mi 5Gi 111Mi 900Mi -
Container cpu 100m 8 110m 700m -
# Or using following
kubectl describe limitrange/limit-mem-cpu-per-container -n limitrange-demo
Output:
Name: limit-mem-cpu-per-container
Namespace: limitrange-demo
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu 100m 8 110m 700m -
Container memory 99Mi 5Gi 111Mi 900Mi -
- Example manifests to test this feature.
Create a manifest file busybox1.yaml with the following content.
apiVersion: v1
kind: Pod
metadata:
name: busybox1
spec:
containers:
- name: busybox-cnt01
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt01; sleep 10;done"]
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "200Mi"
cpu: "500m"
- name: busybox-cnt02
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt02; sleep 10;done"]
resources:
requests:
memory: "100Mi"
cpu: "100m"
- name: busybox-cnt03
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt03; sleep 10;done"]
resources:
limits:
memory: "200Mi"
cpu: "500m"
- name: busybox-cnt04
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt04; sleep 10;done"]
Now send this manifest to k8s api.
kubectl apply -f busybox1.yaml -n limitrange-demo
Let’s validate
kubectl describe -f busybox1.yaml -n limitrange-demo
Name: busybox1
Namespace: limitrange-demo
Priority: 0
Node: ip-69-76-24-106.us-west-2.compute.internal/69.76.24.106
Start Time: Tue, 25 Feb 2020 14:38:54 -0700
Labels: <none>
Annotations: cni.projectcalico.org/podIP: 100.96.5.82/32
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"busybox1","namespace":"limitrange-demo"},"spec":{"containers":[{"args...
kubernetes.io/limit-ranger:
LimitRanger plugin set: cpu, memory limit for container busybox-cnt02; cpu, memory request for container busybox-cnt04; cpu, memory limit ...
Status: Running
IP: 100.96.5.82
IPs: <none>
Containers:
busybox-cnt01:
Container ID: docker://afcd9d2ec8e5568fde8d9d9c155266bd1d685a36d8bc79ba6677a119ba6f09b5
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
/bin/sh
Args:
-c
while true; do echo hello from cnt01; sleep 10;done
State: Running
Started: Tue, 25 Feb 2020 14:38:57 -0700
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 200Mi
Requests:
cpu: 100m
memory: 100Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mnfxb (ro)
busybox-cnt02:
Container ID: docker://a9bacf09475944c447a021f299ac384087d5af46f139b70ce14ee595098bbcf8
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
/bin/sh
Args:
-c
while true; do echo hello from cnt02; sleep 10;done
State: Running
Started: Tue, 25 Feb 2020 14:38:59 -0700
Ready: True
Restart Count: 0
Limits:
cpu: 700m
memory: 900Mi
Requests:
cpu: 100m
memory: 100Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mnfxb (ro)
busybox-cnt03:
Container ID: docker://c8a2520d632e672a5b343f2dc680c178f2612515e70531cc1be0125e9a9c1fd3
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
/bin/sh
Args:
-c
while true; do echo hello from cnt03; sleep 10;done
State: Running
Started: Tue, 25 Feb 2020 14:39:00 -0700
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 200Mi
Requests:
cpu: 500m
memory: 200Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mnfxb (ro)
busybox-cnt04:
Container ID: docker://c70901422ba7739e1ef1820d917ebf6471d8efb998de904287e7afdb24e9c909
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
/bin/sh
Args:
-c
while true; do echo hello from cnt04; sleep 10;done
State: Running
Started: Tue, 25 Feb 2020 14:39:02 -0700
Ready: True
Restart Count: 0
Limits:
cpu: 700m
memory: 900Mi
Requests:
cpu: 110m
memory: 111Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mnfxb (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-mnfxb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mnfxb
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 117s kubelet, ip-69-76-24-106.us-west-2.compute.internal pulling image "busybox"
Normal Pulled 114s kubelet, ip-69-76-24-106.us-west-2.compute.internal Successfully pulled image "busybox"
Normal Created 114s kubelet, ip-69-76-24-106.us-west-2.compute.internal Created container
Normal Started 114s kubelet, ip-69-76-24-106.us-west-2.compute.internal Started container
Normal Pulling 114s kubelet, ip-69-76-24-106.us-west-2.compute.internal pulling image "busybox"
Normal Pulled 113s kubelet, ip-69-76-24-106.us-west-2.compute.internal Successfully pulled image "busybox"
Normal Created 113s kubelet, ip-69-76-24-106.us-west-2.compute.internal Created container
Normal Started 112s kubelet, ip-69-76-24-106.us-west-2.compute.internal Started container
Normal Pulling 112s kubelet, ip-69-76-24-106.us-west-2.compute.internal pulling image "busybox"
Normal Pulled 111s kubelet, ip-69-76-24-106.us-west-2.compute.internal Successfully pulled image "busybox"
Normal Created 111s kubelet, ip-69-76-24-106.us-west-2.compute.internal Created container
Normal Started 111s kubelet, ip-69-76-24-106.us-west-2.compute.internal Started container
Normal Pulling 111s kubelet, ip-69-76-24-106.us-west-2.compute.internal pulling image "busybox"
Normal Pulled 109s kubelet, ip-69-76-24-106.us-west-2.compute.internal Successfully pulled image "busybox"
Normal Created 109s kubelet, ip-69-76-24-106.us-west-2.compute.internal Created container
Normal Started 109s kubelet, ip-69-76-24-106.us-west-2.compute.internal Started container
Normal Scheduled 40s default-scheduler Successfully assigned limitrange-demo/busybox1 to ip-69-76-24-106.us-west-2.compute.internal
If you look at the above output, any containers which are missing limits/request is allocated a default value which is mentioned in LimitRange object. If limits are missing value Kubernetes will allocate value for limits and if requests are missing value it will assign value for requests. If both values are missing it will assign a default value for both. If all values are mentioned in manifest it will do nothing as long as manifest respecting hard max and min limits.
Container 1
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[0].resources"
Output:
{
"limits": {
"cpu": "500m",
"memory": "200Mi"
},
"requests": {
"cpu": "100m",
"memory": "100Mi"
}
}
Container 2
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[1].resources"
Output:
{
"limits": {
"cpu": "700m",
"memory": "900Mi"
},
"requests": {
"cpu": "100m",
"memory": "100Mi"
}
}
Container 3
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[2].resources"
Output:
{
"limits": {
"cpu": "500m",
"memory": "200Mi"
},
"requests": {
"cpu": "500m",
"memory": "200Mi"
}
}
Container 4
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[3].resources"
Output:
{
"limits": {
"cpu": "700m",
"memory": "900Mi"
},
"requests": {
"cpu": "110m",
"memory": "111Mi"
}
}
