Building a K3s cluster on Raspberry PIs

2024-05-13

This collection of articles aims to describe my journey into building a personal Kubernetes cluster at home.

I took a lot of inspiration from the Small homelab K8s cluster on Proxmox VE series for both the structure and some of the content, but declined for my own needs and choices. That is a very high quality read.

Table of contents

  1. Introduction - this document
    1. Cluster hardware and operating system
    2. About the network
  2. Setup Git to store K3s cluster setup and configuration
  3. Single node K3s cluster
  4. K3s server setup on dietpi and client configuration
  5. The first app on K3s, Home Assistant
  6. SSL certificates with ZeroSSL
  7. Local DNS with Pi-hole
  8. Exposing arbitrary TCP and UDP ports through ingress NGINX
  9. Accessing services with Cloudflare Zero Trust
  10. Reliable K3s storage with Longhorn
  11. K3s load balancing with kube-vip
  12. Monitoring with Prometheus and Grafana
    1. Root config
    2. Kube State Metrics
    3. Prometheus Node Exporter
    4. Prometheus Server
    5. Grafana
  13. Accessing K3s hosted services through a VPN with Wireguard
  14. Adding WebDAV for quick access to service data
  15. Fixing wrong decisions
    1. Disable Traefik after setting up the cluster
    2. Adding Longhorn after setting up the cluster
    3. Adding kube-vip after setting up the cluster
    4. Migrating K3s datastore from Sqlite to etcd
    5. Handling hardware setup with Ansible
  16. Apps
    1. Organize documents with Paperless-ngx
    2. Keeping personal finance organized with XXX
  17. Useful snippets
    1. Upload data to a Longhorn drive
  18. Meta
    1. How is this guide published?
  19. Build log

Introduction

Why self-hosting?

It’s tough to trust big companies with our data these days1. I’ve been wanting to take control of my digital stuff for a while now. Sure, some things are tricky2 or pricey3 4 to self-host, but with a bit of effort, you can often get a setup that’s just as good or even better than those SaaS options.

My objective is not to ditch everything that is in the cloud, rather choosing what makes sense to self host and what doesn’t. I don’t think I will ever host a clone of Google Maps, but will most probably find an alternative to hosting my personal documents on Dropbox.

Last but not least, being myself a software engineer, this is also a good way to learn new things hands-on, because I learn best through hands-on experience.

Why a Kubernetes cluster?

Reliability is key because I’m the only one at home who can manage this setup. It needs to just work without me constantly tinkering.

So why Kubernetes5? I worked a tiny bit with it, and what I like is the way how it can automatically recover from failures (ie. a node going down). Eventually, after a very long setup, it hopefully will just run. Of course it will need to be maintained, but that is something one needs to take into account when self-hosting in general.

Why running Kubernetes on a Raspberry Pi cluster?

One big goal is to make the cluster super reliable, which means I need a few machines working together. I decided not to host very resource intensive services on the cluster for a few reasons:

  1. Since the cluster is right next to my desk, it has to be quiet. That’s why I chose the Raspberry Pi 4B over the Raspberry Pi 5;
  2. I’d prefer to not use too much physical space;
  3. Power consumption is a thing;
  4. I chose to go with Raspberry Pi over other SBCs because of the community size.

Once everything’s up and running, I’ll play around with more demanding tasks like local AIs or media streaming.

Isn’t Kubernetes too heavy for a Raspberry Pi?

Yep, Kubernetes can be heavy for a Raspberry Pi. So, I chose a super lightweight OS6 and a minimal Kubernetes distribution7. I’ll dive into the details in the next chapters.

Main rules for this project

I tried to give myself a few rules for building this projects so that I won’t be stuck in the design phase forever.

  1. Build incrementally: start with the basics and add features as you go. Don’t over-engineer;
  2. Make it reliable: the cluster should run smoothly without me needing to step in, unless something really goes wrong (and if that happens, the cluster will be the least of my worries);
  3. Solve real problems: build something that’s genuinely useful;
  4. Keep it secure: no one wants hackers snooping around or accessing my security cams;
  5. Keep it offline: I’d love for some core services to work even if the internet’s down or there’s a power outage;
  6. Document everything: my memory isn’t great, so documenting helps me avoid wasting hours on the same issues over and over;
  7. Learn something: I’ll definitely pick up new skills, and documenting everything helps it stick;
  8. Make it replicable: things will break (they already have), so I need to be able to rebuild the cluster quickly.

  1. Slack has been using data from your chats to train its machine learning models: https://www.engadget.com/yuck-slack-has-been-scanning-your-messages-to-train-its-ai-models-181918245.html.↩︎

  2. Self-hosting a password manager as Bitwarden requires a high level of security, it is most probably safer to just use the hosted version.↩︎

  3. See Llama2 Memory Requirements to have an idea.↩︎

  4. Self-hosting media can be very expensive as it requires a lot of fast storage, replication, and good hardwer for encoding/decoding.↩︎

  5. From kubernetes.io: Kubernetes, also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications.↩︎

  6. From dietpi.com: DietPi is extremely lightweight at its core, with features of low process/memory footprint and DietPi-RAMlog installed by default, to get the maximum performance from your device.↩︎

  7. From k3s.io: The certified Kubernetes distribution built for IoT & Edge computing.↩︎