Anyone who doesn’t know what is NTP (Network Time Protocol), directly from Wikipedia,
“The Network Time Protocol is a networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks.”
In this blog, I am using OpenNTPD which is a FREE implementation of NTP. It provides
- the ability to sync the local clock to remote NTP servers
- can act as NTP server itself, redistributing the local clock.
Problem Description
I was curious if it is possible to run NTPD using openntpd in the Kubernetes cluster. Found a handy blog. In this blog, the author is showing how you can run NTP in a container. So I am going to follow this blog to create a build process for NTPD docker image and use that image to run NTPD in the Kubernetes cluster.
Solutions
Test with Docker container
First, create a Dockerfile using the following content.
FROM alpine RUN apk update RUN apk add openntpd ENTRYPOINT ["ntpd"]
Now you can build docker image using the following command
docker build -t goglides/ntpd .
Let’s test it using the following
docker run goglides/ntpd -d Output: ntpd: can't set priority: Permission denied reset adjtime failed: Operation not permitted creating new /var/db/ntpd.drift adjtimex failed: Operation not permitted ntp engine ready constraint request to 172.217.9.196 constraint request to 2607:f8b0:4004:806::2004 tls connect failed: 2607:f8b0:4004:806::2004 (www.google.com): connect: Address not available no constraint reply from 2607:f8b0:4004:806::2004 received in time, next query 900s tls connect failed: 172.217.9.196 (www.google.com): ssl verify memory setup failure no constraint reply from 172.217.9.196 received in time, next query 900s
Apparently, our container is not working as expected, if you look at following errors,
ntpd: can't set priority: Permission denied reset adjtime failed: Operation not permitted creating new /var/db/ntpd.drift adjtimex failed: Operation not permitted
which is basically saying not enough permission is because docker is not able to run binary, we can fix this by adding --privileged flag as follows
docker run --privileged goglides/ntpd -d Output: creating new /var/db/ntpd.drift ntp engine ready constraint request to 2607:f8b0:4004:806::2004 constraint request to 172.217.9.196 tls connect failed: 2607:f8b0:4004:806::2004 (www.google.com): connect: Address not available no constraint reply from 2607:f8b0:4004:806::2004 received in time, next query 900s tls connect failed: 172.217.9.196 (www.google.com): ssl verify memory setup failure no constraint reply from 172.217.9.196 received in time, next query 900s
And for SSL issue I keep seeing it, I tried adding following in Dockerfile which didn’t fix the issue,
RUN apk add ca-certificates RUN update-ca-certificates
There is an open bug in the official repo, https://gitlab.alpinelinux.org/alpine/aports/issues/9635 which mentioning why we are seeing this issue.
The main cause is mentioned below (copy/pasted from the ticket)
This happens because of the chroot(2) call in ntpd(8) child. ntpd(8) calls open(“/tmp/libtlscompat***”, …) after chrooting to /var/empty and fails because there is no /var/empty/tmp. Doing mkdir -m 1777 /var/empty/tmp helps, but I don’t know if this is the right approach. UPD 1: There is a way to specify the directory where ntpd(8) should chroot with ./configure … –with-privsep-path=, so it would make sense to create /var/lib/openntpd with tmp inside owned by ntp:ntp. UPD 2: Actually, there is no call to open(“/tmp/libtls***”, …) on OpenBSD. I guess that this is standalone-only thing. UPD 3: The offending function SSL_CTX_load_verify_mem resides here. Calling mkstemp(3) from there happens after chrooting and ntpd(8) does not expect that since vanilla libressl does all this in memory:
/* From libressl-2.9.1 */
int
SSL_CTX_load_verify_mem(SSL_CTX *ctx, void *buf, int len)
{
return (X509_STORE_load_mem(ctx->cert_store, buf, len));
}
UPD 4: OpenNTPD does not use –with-privsep-path=in code, it just for hints. Instead it chroots to the home directory of a running user, so it’s necessary to setup a dedicated user (e.g. openntpd) with a home directory pointing to e.g. /var/lib/openntpd.
So solution is to add mkdir -m 1777 /var/empty/tmp as workaround for now. My final Dockerfile looks like this.
FROM alpine:3.11.3 RUN apk update RUN apk add openntpd RUN mkdir -m 1777 /var/empty/tmp ENTRYPOINT ["ntpd"]
Now I am seeing following output
docker run --privileged goglides/ntpd -d Output: creating new /var/db/ntpd.drift ntp engine ready constraint request to 172.217.9.196 constraint request to 2607:f8b0:4004:806::2004 tls connect failed: 2607:f8b0:4004:806::2004 (www.google.com): connect: Address not available no constraint reply from 2607:f8b0:4004:806::2004 received in time, next query 900s constraint reply from 172.217.9.196: offset -1.288009 reply from 66.228.58.20: offset -0.292780 delay 0.079884, next query 8s reply from 72.87.88.202: offset -0.299418 delay 0.088550, next query 6s reply from 138.68.201.49: offset -0.288903 delay 0.137101, next query 9s
The next step is to provide a configuration file. I am simply going to create /tmp/ntpd.conf conf file with servers pool.ntp.org content and mount that to the container. For detail configuration options, please visit ntpd.conf man page
docker run -v /tmp/ntpd.conf:/etc/ntpd.conf --privileged goglides/ntpd -d -f /etc/ntpd.conf Output: creating new /var/db/ntpd.drift ntp engine ready reply from 69.89.207.99: offset -0.356652 delay 0.122182, next query 33s reply from 96.8.121.205: offset -0.356918 delay 0.153844, next query 30s reply from 138.236.128.112: offset -0.346649 delay 0.108738, next query 30s reply from 96.8.121.205: offset -0.359319 delay 0.153514, next query 34s reply from 204.11.201.10: offset -0.359962 delay 0.152213, next query 32s reply from 69.89.207.99: offset -0.362657 delay 0.121926, next query 32s adjusting local clock by 0.064548s
Alright our test is complete let’s run this container in the background with following
docker run --name ntpd \
-v /tmp/ntpd.conf:/etc/ntpd.conf \
--restart=always -d --privileged \
goglides/ntpd -d -s -f /etc/ntpd.conf
The -s means “Try to set the time immediately at startup.” You can check if the process is running or not using following,
docker ps Output: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 035cf608e489 goglides/ntpd "ntpd -d -s -f /etc/…" About a minute ago Up About a minute ntpd
and logs
docker logs ntpd Output: creating new /var/db/ntpd.drift adjtimex adjusted frequency by 0.000000ppm ntp engine ready reply from 45.55.217.50: offset -0.395447 delay 0.139999, next query 7s reply from 50.205.244.112: offset -0.392487 delay 0.139775, next query 6s set local clock to Tue Mar 3 20:18:48 UTC 2020 (offset -0.395447s) reply from 216.240.36.24: negative delay -0.247795s, next query 3217s reply from 23.129.64.159: negative delay -0.245678s, next query 3019s reply from 50.205.244.112: offset 0.024988 delay 0.082805, next query 5s reply from 45.55.217.50: offset 0.019650 delay 0.093784, next query 7s reply from 45.55.217.50: offset 0.009099 delay 0.085599, next query 9s peer 45.55.217.50 now valid
Deploy to Kubernetes cluster
Since we validate NTP is working in a container, I should able to deploy this in the Kubernetes cluster easily. For this, I have to modify Dockerfile little bit to address some of the challenges.
- Run container in Foreground We can easily achieve this by changing our docker entrypoint, by adding -d flag. As per man page,
-dmeans do not daemonize. If this option is specified, ntpd will run in the foreground and log to stderr. - Supply configuration file I am thinking about supplying configuration using configmap. I have to modify Dockerfile to access file config using
-fflag. After that create configmap with actual config, mount the volume and use that custom configuration.
For this created a custom entrypoint.sh to handle some logic.
#! /bin/sh
if [ -z "${NTP_CONF_FILE}" ]
then
NTP_CONF_FILE="/etc/ntpd.conf"
fi
ntpd -v -d -s -f ${NTP_CONF_FILE}
and modified Dockerfile to consume this.
FROM alpine:3.11.3 RUN apk update RUN apk add openntpd RUN mkdir -m 1777 /var/empty/tmp ADD ./entrypoint.sh ./entrypoint.sh RUN chmod 755 ./entrypoint.sh ENTRYPOINT ["./entrypoint.sh"]
After that, I created a Kubernetes DaemonSet object to use this docker image. Save following content to ds-ntpd.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: ntpd-config
data:
ntpd.conf: |
servers pool.ntp.org
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ntpd
labels:
k8s-app: ntpd
created-by: tech.goglides.com
spec:
selector:
matchLabels:
name: ntpd
template:
metadata:
labels:
name: ntpd
spec:
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: ntp-sync
image: goglides/ntpd
imagePullPolicy: Never
resources:
limits:
memory: 20Mi
cpu: 20m
requests:
cpu: 10m
memory: 10Mi
securityContext:
privileged: true
env:
- name: NTP_CONF_FILE
value: /app/ntpd.conf
volumeMounts:
- name: ntpd-config
mountPath: /app/
volumes:
- name: ntpd-config
configMap:
name: ntpd-config
Now finally apply this manifest to Kubernetes using the following,
kubectl apply -f ds-ntpd.yaml Output: configmap/ntpd-config created daemonset.apps/ntpd created
You can verify if pods are running or not using following,
kubectl get pods Output: NAME READY STATUS RESTARTS AGE ntpd-b7csn 1/1 Running 0 68s
And check logs,
kubectl logs ntpd-b7csn Output: creating new /var/db/ntpd.drift adjtimex adjusted frequency by 0.000000ppm ntp engine ready reply from 51.158.147.92: offset -0.715358 delay 0.126846, next query 6s reply from 176.9.40.131: offset -0.722062 delay 0.148438, next query 9s reply from 5.186.65.2: offset -0.716915 delay 0.148267, next query 7s reply from 212.25.15.128: offset -0.703432 delay 0.160469, next query 8s set local clock to Tue Mar 3 23:09:38 UTC 2020 (offset -0.715358s)
