aviary

my apartment's server configuration, because deterministic is cool!

philosophy

the biggest thing i learned using k8s for personal things, is that not everything has to follow the cool enterprise-y trends, like GitOps, fully config-file based systems, or overkill performance optimizations.

this setup strives to be, in order of priority:

1. lean; easy to learn, understand, & reason about
2. low-maintenance & fast-moving
3. resilient against faults & outages
4. cost/resource-effective
5. deterministic & declarative

tech stack

• arch linux
  because it's so minimal (essentially all software on there is software you've installed), it massively reduces room for error – without adding maintenance overhead (e.g. nix) or sacrificing on features / ecosystem (talos, alpine); arch also makes it easy to take advantage of modern kernel features (e.g. eBPF/XDP)
  • chrony for time
  • systemd-networkd because it's simple and plays nicely with other things
  • systemd-resolved (stub) to cache and hence speed up DNS queries
  • btrfs is just a good filesystem
  • linux-zen kernel out of personal preference
  • no bootloader (efi entry straight to vmlinuz) to reduce moving parts
• k0s
  wanted to try something new after using k3s in prod; also, the declarative setup using k0sctl is really nice
• helmfile
  used to use flux, but maintaining state and reconciling felt incredibly fragile (and slow!); also, helmfile lets me use well-supported tooling like kubectl and helm alongside it
• sops
  does simple git-commitable secrets

networking:

• calico as cni
  comes bundled with k0s, "just works" out of the box & has better docs than cilium (this is a big thing); maybe once cilium investigates my issue more i'll switch back, but it's been 5 months at the time of writing
  • ipip/bird mode; i am intimately familiar with bird
• udm as core/ToR router
  it's what i have at home, speaks bgp, & is debian so quick fixes are pretty straightforward
• nginx ingress
  supported by any piece of k8s software, very performant, very flexible, and something i am very familiar with
• metallb for lb service ipam
  it was designed for this exact setup

storage:

• longhorn as csi
  it just works, has a nice ui, and doesn't make questionable assumptions about disk speed distribution (looking it you, ceph)

setup

• follow the documentation in hosts/ to setup nodes and network
• bin/ contains scripts to handle lifecycle tasks; bin/install will install k0s from the config files in cluster/
• helmfile.d/ contains the applications running on the cluster, as helm charts using helmfile
• secrets-example/ need copied to secrets/, filled in and encrypted with sops encrypt

tl;dr

1. set up hosts
2. double check k0sctl.yaml
3. set up secrets using sops
4. bin/install
5. helmfile sync