Problem Description

As a rule,

Containers run on a Kubernetes cluster with unlimited computing resources.
If a pod/container goes rogue and monopolizes resources, it has the potential to destroy the entire cluster.

Solutions:

To limit this behavior, use k8s features such as Resource Quotas, LimitRange, and Request/Limits. I’m going through the official blog in this trashcan article to spike the LimitRange feature.

Official page mentioned 4 types of resources,

Limiting Container compute resources
Limiting Pod compute resources
Limiting Storage resources
Limits/Requests Ratio

I am going to try all 4.

Limiting container compute resources

Create a namespace limitrange-demo using the following kubectl command:

kubectl create namespace limitrange-demo

Create a configuration file (limitrange.yaml) for limit range objects

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-mem-cpu-per-container
spec:
  limits:
  - max:
      cpu: "800m"
      memory: "1Gi"
    min:
      cpu: "100m"
      memory: "99Mi"
    default:
      cpu: "700m"
      memory: "900Mi"
    defaultRequest:
      cpu: "110m"
      memory: "111Mi"
    type: Container

Now apply this limit range configuration file

kubectl apply -f limitrange.yaml -n limitrange-demo
Output:
limitrange/limit-mem-cpu-per-container created

** You can validate if the object is properly created using the following command

kubectl describe -f limitrange.yaml  -n limitrange-demo
Output:
Name:       limit-mem-cpu-per-container
Namespace:  limitrange-demo
Type        Resource  Min   Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---   ---  ---------------  -------------  -----------------------
Container   memory    99Mi  5Gi  111Mi            900Mi          -
Container   cpu       100m  8    110m             700m           -
# Or using following
kubectl describe limitrange/limit-mem-cpu-per-container -n limitrange-demo
Output:
Name:       limit-mem-cpu-per-container
Namespace:  limitrange-demo
Type        Resource  Min   Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---   ---  ---------------  -------------  -----------------------
Container   cpu       100m  8    110m             700m           -
Container   memory    99Mi  5Gi  111Mi            900Mi          -

Example manifests to test this feature.

Create a manifest file busybox1.yaml with the following content.

apiVersion: v1
kind: Pod
metadata:
  name: busybox1
spec:
  containers:
  - name: busybox-cnt01
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello from cnt01; sleep 10;done"]
    resources:
      requests:
        memory: "100Mi"
        cpu: "100m"
      limits:
        memory: "200Mi"
        cpu: "500m"
  - name: busybox-cnt02
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello from cnt02; sleep 10;done"]
    resources:
      requests:
        memory: "100Mi"
        cpu: "100m"
  - name: busybox-cnt03
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello from cnt03; sleep 10;done"]
    resources:
      limits:
        memory: "200Mi"
        cpu: "500m"
  - name: busybox-cnt04
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello from cnt04; sleep 10;done"]

Now send this manifest to k8s api.

kubectl apply -f busybox1.yaml -n limitrange-demo

Let’s validate

kubectl describe -f busybox1.yaml -n limitrange-demo
Name:         busybox1
Namespace:    limitrange-demo
Priority:     0
Node:         ip-69-76-24-106.us-west-2.compute.internal/69.76.24.106
Start Time:   Tue, 25 Feb 2020 14:38:54 -0700
Labels:       <none>
Annotations:  cni.projectcalico.org/podIP: 100.96.5.82/32
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"busybox1","namespace":"limitrange-demo"},"spec":{"containers":[{"args...
              kubernetes.io/limit-ranger:
                LimitRanger plugin set: cpu, memory limit for container busybox-cnt02; cpu, memory request for container busybox-cnt04; cpu, memory limit ...
Status:       Running
IP:           100.96.5.82
IPs:          <none>
Containers:
  busybox-cnt01:
    Container ID:  docker://afcd9d2ec8e5568fde8d9d9c155266bd1d685a36d8bc79ba6677a119ba6f09b5
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      while true; do echo hello from cnt01; sleep 10;done
    State:          Running
      Started:      Tue, 25 Feb 2020 14:38:57 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  200Mi
    Requests:
      cpu:        100m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mnfxb (ro)
  busybox-cnt02:
    Container ID:  docker://a9bacf09475944c447a021f299ac384087d5af46f139b70ce14ee595098bbcf8
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      while true; do echo hello from cnt02; sleep 10;done
    State:          Running
      Started:      Tue, 25 Feb 2020 14:38:59 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     700m
      memory:  900Mi
    Requests:
      cpu:        100m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mnfxb (ro)
  busybox-cnt03:
    Container ID:  docker://c8a2520d632e672a5b343f2dc680c178f2612515e70531cc1be0125e9a9c1fd3
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      while true; do echo hello from cnt03; sleep 10;done
    State:          Running
      Started:      Tue, 25 Feb 2020 14:39:00 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  200Mi
    Requests:
      cpu:        500m
      memory:     200Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mnfxb (ro)
  busybox-cnt04:
    Container ID:  docker://c70901422ba7739e1ef1820d917ebf6471d8efb998de904287e7afdb24e9c909
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      while true; do echo hello from cnt04; sleep 10;done
    State:          Running
      Started:      Tue, 25 Feb 2020 14:39:02 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     700m
      memory:  900Mi
    Requests:
      cpu:        110m
      memory:     111Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mnfxb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-mnfxb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mnfxb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                 Message
  ----    ------     ----  ----                                                 -------
  Normal  Pulling    117s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  pulling image "busybox"
  Normal  Pulled     114s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Successfully pulled image "busybox"
  Normal  Created    114s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Created container
  Normal  Started    114s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Started container
  Normal  Pulling    114s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  pulling image "busybox"
  Normal  Pulled     113s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Successfully pulled image "busybox"
  Normal  Created    113s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Created container
  Normal  Started    112s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Started container
  Normal  Pulling    112s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  pulling image "busybox"
  Normal  Pulled     111s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Successfully pulled image "busybox"
  Normal  Created    111s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Created container
  Normal  Started    111s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Started container
  Normal  Pulling    111s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  pulling image "busybox"
  Normal  Pulled     109s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Successfully pulled image "busybox"
  Normal  Created    109s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Created container
  Normal  Started    109s  kubelet, ip-69-76-24-106.us-west-2.compute.internal  Started container
  Normal  Scheduled  40s   default-scheduler                                    Successfully assigned limitrange-demo/busybox1 to ip-69-76-24-106.us-west-2.compute.internal

If you look at the above output, any containers which are missing limits/request is allocated a default value which is mentioned in LimitRange object. If limits are missing value Kubernetes will allocate value for limits and if requests are missing value it will assign value for requests. If both values are missing it will assign a default value for both. If all values are mentioned in manifest it will do nothing as long as manifest respecting hard max and min limits.

Container 1

kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[0].resources"
Output:
{
  "limits": {
    "cpu": "500m",
    "memory": "200Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "100Mi"
  }
}

Container 2

kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[1].resources"
Output:
{
  "limits": {
    "cpu": "700m",
    "memory": "900Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "100Mi"
  }
}

Container 3

kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[2].resources"
Output:
{
  "limits": {
    "cpu": "500m",
    "memory": "200Mi"
  },
  "requests": {
    "cpu": "500m",
    "memory": "200Mi"
  }
}

Container 4

kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[3].resources"
Output:
{
  "limits": {
    "cpu": "700m",
    "memory": "900Mi"
  },
  "requests": {
    "cpu": "110m",
    "memory": "111Mi"
  }
}