It has been a long while since I have posted, mostly due to work having fun enough projects that my lab became a sort of second job. That isn’t to say I wasn’t tinkering, just that most of my time was going to the Kubernetes equivalent of weed pulling. Provided that moving off of VMware to Kubevirt counts as weed pulling. I don’t have the energy to document that journey just yet.
But then a bolt of inspiration happened after his kids wrecked a portion of our majestic mountain castle:
“If only you could change the password automatically”
In fairness to the kids “Dave” and “Normol the Red”… They only led the stone golem to the base. After a couple of play sessions of finger pointing and comic book guy style bans (there is nothing funnier than a kid building a village just to ban the adults and his brother), I decided to get to work. This sounds like a job for someone with more time than sense…
Building on lloesche’s excellent work
None of this would be possible without standing on the shoulder’s of the giant that is lloesche using his excellent Valheim server container. Seriously, star his repo and buy him a coffee and a puppy.
Prerequisites
Besides the obvious Kubernetes cluster, we are going to need a few things for our server to work correctly:
First, we are going to need a place to store persistent data. I currently the local-path-provisioner from the SUSE Rancher folks. This simply creates a local directory for every new PVC/PV that is created. You are welcome to use anything here.
Second, we need a load balancer (or configure a nodeport if you would like). My lab uses Metallb.
Deploy Valheim to Kubernetes
I don’t plan to show all of the YAML required to get this going, so instead, let’s download the repo from github.com:
git clone git@github.com:ccrow42/valheim-k8s-server.git cd valheim-k8s-server
We can apply these files in order to deploy the Valheim server container:
ccrow--MacBookPro18:deploy ccrow$ ls deploy 01-namespace.yaml 02-valheim-pvc.yaml 03-valheim-deployment.yaml 04-valheim-service.yaml k apply -f deploy/.
This will apply all of the manifests required to get Valheim running.
You may wish to change storageClass in the 03-valheim-pvc.yaml file:
... spec: storageClassName: local-path ...
and the service configuration in 04-valheim-service.yaml if you are not using a load balancer:
...
spec:
ports:
- name: gameport
nodePort: 30742
port: 2456
protocol: UDP
- name: queryport
nodePort: 32422
port: 2457
protocol: UDP
selector:
app: valheim-server
type: LoadBalancer
...The type can easily be changed to nodePort. Take note of the ports required for valheim to operate.
We now need to create a secret to make this work correctly. The secret reference can be found in the 04-valheim-deployment.yaml file (we don’t need to modify anything, but it is important to know what the name and key is for our secret:
...
spec:
template:
spec:
containers:
- name: valheim-server
env:
valueFrom:
secretKeyRef:
name: valheim-pass
key: SERVER_PASSNow let’s create the secret. Don’t worry about what you set the secret to as the entire point of this article is to be able to cycle the password automatically:
k create secret generic -n valheim valheim-pass --from-literal=SERVER_PASS=lumberjack
Now let’s check on our service and port information so we know how to configure our firewall:
ccrow--MacBookPro18:deploy ccrow$ k get svc -n valheim NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE valheim-server LoadBalancer 10.43.14.176 172.42.3.32 2456:30742/UDP,2457:32422/UDP 166d
We would of course forward 2456-2457 UDP on our firewall to 172.42.3.32. We can stop now if all we want to do is build a Valheim server on Kubernetes, but you’re here for the shenanigans.
Rotating the Valheim server password
How to rotate this password was a hotly debated topic. Should I use a gitops pipeline to update a sealed secret and make sure ArgoCD bounces the pod? This seems like a lot of work to persist secrets in a git repo (I’m not a fan of this, even with sealed secrets). Besides, our deployment shouldn’t be hard-coding a password in the first place. Although the real reason is I suck at writing gitlab actions.
Should I use an external secrets provider? This is probably the correct way and tell it to cycle the password. This did bring up the question about how to get the Valheim pod to notice the change and restart (I suspect that this should be a sidecar). Either way, I don’t have an external secrets provider configured… yet.
At the end of the day, any solution is going to require some custom scripting so that I can notify people of the password change. I decided to go with a Kubernetes CronJob and a custom image to do the work.
I decided to use a simple list of dictionary words for the password. I also decided to use discord for notifications to a private channel.
It is funny how often things devolve to bash…
Building a custom image
Our custom image is going to have a couple of helpful tools installed to do a password rotation. I was also lazy and baked the word list into the image itself. I have also added a little script to notify Discord of the password change. It is a generic function, and will pull the Discord webhook URL from a secret. If you haven’t generated a Discord webhook yet, see these instructions.
Let’s take a look at a couple more files in our repo:
#!/usr/bin/env bash
set -ex
MESSAGE="$*"
# Safely encode message and send
curl -k -X POST -H "Content-Type: application/json" \
-d "$(jq -nc --arg content "$MESSAGE" '{content: $content}')" \
"$DISCORD_WEBHOOK"This file will be included in our image. Let’s take a look at the Dockerfile:
FROM debian:bookworm-slim
RUN apt-get update && \
apt-get install -y --no-install-recommends bash jq curl python3 ffmpeg gettext-base && \
apt-get clean && rm -rf /var/lib/apt/lists/*
COPY words.txt .
COPY notify_discord.sh /usr/local/bin
COPY update_youtube_channels.sh /usr/local/bin
RUN chmod +x /usr/local/bin/notify_discord.sh
RUN curl -k -LO "https://dl.k8s.io/release/$(curl -k -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
RUN mv kubectl /usr/local/bin
RUN curl -k -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
RUN chmod a+rx /usr/local/bin/yt-dlp
RUN chmod +x /usr/local/bin/kubectl
# Set bash as default shell
SHELL ["/bin/bash", "-c"]We can now build and push our image:
docker build -t registry.lan.ccrow.org/debian-custom:latest debian-custom/. docker push registry.lan.ccrow.org/debian-custom:latest
Of course you will need to change the location where you are storing your image. My registry is not public.
Finally, let’s create our webhook secret:
k -n valheim create secret generic discord-password-url --from-literal=DISCORD_WEBHOOK=https://discord.com/api/webhooks/aVeryLongStringofThings
Be sure to use the webhook URL you created earlier.
Creating the CronJob
Our cronjob is really the glue that makes this whole thing work. It starts by updating the password in the secret we created earlier. It then restarts the Valheim deployment (I don’t bother to check if folks are in the server. If you are still playing at 4am than this your hint to go to bed). Lastly, it posts the password to the discord channel using the URL we stored in the above secret.
Because we are messing with some Kubernetes objects from our container, we need to create a service account the the proper permissions. Take a moment to review the configuration in the valheim-password-rotation/service-account.yaml file and apply it:
k apply -f valheim-password-rotation/service-account.yaml
Now let’s take a look at the CronJob itself:
apiVersion: batch/v1
kind: CronJob
metadata:
name: valheim-password-rotate
namespace: valheim
spec:
schedule: "0 4 * * *"
timeZone: "US/Pacific"
jobTemplate:
spec:
template:
spec:
serviceAccountName: valheim-password-rotator
restartPolicy: OnFailure
containers:
- name: rotate-password
image: registry.lan.ccrow.org/debian-custom:latest
imagePullPolicy: Always
env:
- name: SECRET_NAME
value: valheim-pass
- name: DISCORD_WEBHOOK
valueFrom:
secretKeyRef:
name: discord-password-url
key: DISCORD_WEBHOOK
- name: SECRET_KEY
value: SERVER_PASS
- name: DEPLOYMENT_NAME
value: valheim-server
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
command:
- /bin/bash
- -c
- |
set -ex
PASSWORD=$(shuf -n 1 words.txt | tr -d '\r\n')
echo "New password: $PASSWORD"
kubectl get secret "$SECRET_NAME" -n "$NAMESPACE" -o json | \
jq --arg pw "$(echo -n "$PASSWORD" | base64)" \
'.data[$ENV.SECRET_KEY] = $pw' | \
kubectl apply -f -
notify_discord.sh "new valheim server password is $PASSWORD"
kubectl rollout restart deployment/"$DEPLOYMENT_NAME" -n "$NAMESPACE"
We pass a number of configuration variables, which if you have been following this guide you shouldn’t need to change.
Be sure to update the image on line 17.
Line 40 is where the magic happens using the shuf command. This logic would be easy to update if you would like a different password policy.
Once you are satisfied, let’s apply the manifests and run a test job:
k apply -f valheim-password-rotation/rotate-password-cron.yaml kubectl create job --from=cronjob/valheim-password-rotate valheim-password-rotate-test -n valheim sleep 60 kubectl logs job/valheim-password-rotate-test -n valheim -f
Wrapping up
If all went well, you should see a discord notification trigger:
I swear I did not plan that password…
This was a fun little project that I could wrap my walnut around given the arrival of our new baby (which also explains the numerous spelling and grammar mistakes).
One more fun idea… Don’t give your kids the new password until they used the previous password in a sentence:
And there you have it. May your fires be warm and your homes unmolested by children!
