This repository describes how to run your own Kubernetes cluster locally (and probably behind NAT) but still expose it over a public IP address using just a small cloud machine.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Dimitris Karakasilis adc55fed64
Add link to blog post
4 weeks ago
chart Initial commit 1 month ago
image Initial commit 1 month ago
.gitignore Initial commit 1 month ago
README.md Add link to blog post 4 weeks ago
high-level.excalidraw Initial commit 1 month ago
high-level.svg Initial commit 1 month ago
woodpecker-helm-values.yaml Initial commit 1 month ago

README.md

Self hosted CI

Original blog post about this repository here: https://dimitris.karakasilis.me/2022/12/26/self-hosted-ci.html

This repository helps you run Woodpecker CI on a local cluster and expose it publicly on your desired domain.

Use cases

  • Projects that need more resources than you can afford on the cloud
  • Projects that need CI only every now and then, so there is no need to maintain a CI system 24/7
  • Projects hosted on code hosting services that don't have integrated CI

Why

Because:

  • cloud resources are expensive
  • local resources are cheap
  • always-on, zero-downtime CI is not always required
  • CI doesn't have to be a pet

Requirements

CI should be publicly accessible because:

  • Code hosting services (Gitea/Codeberg, Gitlab, Github) use webhooks that need to be able to reach your CI server.
  • Commit status should point to some URL that is accessible so that the users can see the results.

How

Instead of putting the whole CI server (whatever CI solution that may be), we setup a "proxy" (let's call it the "cloud box") which is the only publicly accessible machine. The "cloud box" only needs to have ssh server running.

We use an image that creates an SSH tunnel to the "cloud box" and then runs an nginx reverse proxy to forward all traffic to the traefik load balancer within a Kubernetes cluster.

This way, all requests landing on the "cloud box", will be redirected through the ssh tunnel to the Pod where we run the nginx reverse proxy. Eventually the requests will reach the traefik load balancer and thus every ingress in the cluster.

This setup allows us to run the CI server even behind NAT (on you workstation for example). The "cloud box" is not running the CI, nor any CI jobs whatsoever, so it only needs resources to run ssh. The smallest machine you can find online probably has more resources than needed.

On our cluster we just expose the traefik load balancer to the public, from within the cluster itself. No need to start the cluster in any specific way (forward host ports or anything).

The following describes the above in a more visual way:

architecture image

Here are the steps to get this running:

  1. Get a public IP "cloud box" EC2, DigitalOcean droplet, whatever suits you best. Just get the smallest (and cheapest) and you are good to go. Make sure you have SSH access to it.

  2. Setup your domain Create an A Record on your nameserver to point your desired domain to the IP address of the box above.

  3. Create a Kubernetes cluster Decide how many resources you are going to need and create a Kubernetes cluster that has enough resource for your needs. If you are not sure, or just playing around, a k3d cluster should be good to get you started:

k3d cluster create woodpecker-ci

Don't worry too much if this is going to be enough. You can replace it later without having to repeat any of the previous steps.

For now, this repository assumes there is a traefik ingress provider running on the cluster.

Warning: What we are going to do, will essentially expose 2 ports of some Pod to the public. Depending on what you expose from your cluster using ingresses, and what software you run (e.g. application frame works), you may be vulnerable to attacks. An attacker gaining shell access to your Pod, could also gain access to your host if for example you run k3d as privileged container. Also, if you run the Woodpecker agents on the same cluster, make sure you don't allow arbitrary Pull Requests to change the pipelines without approval. This makes it extremely easy for anyone to run arbitrary code on your cluster. Make sure you understand the risks and that you take precautions and follow security best practices. In any case, do this at your own risk!

NOTE: An probably more secure alternative would be to create the cluster as a VM. That way you can isolate it from the host machine better. A very easy way to create a cluster using VMs is Kairos. In the future this project may provide a kairos config file that will automatically deploy the provided helm chart. Until then, you can see how this works for metallb in the docs: https://kairos.io/docs/examples/metallb/

  1. Create a values.yaml file for the helm chart we are going to deploy:
  # The email to be used with the lets-encrypt issuer
  letsencrypt_email: ""

  jumpbox:
    # The IP address or URL of the publicly acessible cloud box
    # This is the machine to which you should point your domain to.
    url: "<the ip address of the cloud box>"
    user: <the user that has ssh access to the box>
    ssh_key: <the private ssh key base64 encoded>
    ssh_key_pub: <the public ssh key base64 encoded>

  # Generate a key with:
  woodpecker_agent_secret: <use `openssl rand -hex 32` to generate one>

  # Woodpecker subchart values
  # https://codeberg.org/Codeberg-CI/woodpecker/raw/branch/master/charts/woodpecker-server/values.yaml
  woodpecker-server:
    replicaCount: 1

    image:
      registry: docker.io
      repository: woodpeckerci/woodpecker-server
      pullPolicy: Always
      # Overrides the image tag whose default is the chart appVersion.
      tag: ""

    env:
      WOODPECKER_ADMIN: "your codeberg username"
      WOODPECKER_HOST: "http://<the domain pointing to your cloud box>"
      WOODPECKER_GITHUB: false
      WOODPECKER_GITEA: true
      WOODPECKER_GITEA_URL: "https://codeberg.org"
      WOODPECKER_GITEA_CLIENT: "<create an oauth2 application on codeberg: https://codeberg.org/user/settings/applications>"
      WOODPECKER_GITEA_SECRET: "<create an oauth2 application on codeberg: https://codeberg.org/user/settings/applications>"

    extraSecretNamesForEnvFrom:
    - woodpecker-secret

    persistentVolume:
      enabled: true
      size: 10Gi
      mountPath: "/var/lib/woodpecker"
      storageClass: "local-path" # Use your cluster's storage class here

    serviceAccount:
      # Specifies whether a service account should be created
      create: true

    ingress:
      enabled: true
      annotations:
        traefik.ingress.kubernetes.io/router.entrypoints: websecure
        traefik.ingress.kubernetes.io/router.tls: "true"

      hosts:
        - host: <the domain pointing to your cloud box>
          paths:
          - path: /
            backend:
              serviceName: chart-example.local
              servicePort: 80

      tls:
        - secretName: woodpecker-tls
          hosts:
            - <the domain pointing to your cloud box>
  1. Deploy cert-manager

    This chart will use let's encrypt (production) to issue a certificate for the CI server. To do so, cert-manager should already be installed. You can install cert-manager on your cluster following the docs:

    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml
    
  2. Deploy the helm chart:

    Wait until cert-manager components are up and running and then install this chart:

    helm upgrade --install -n woodpecker --create-namespace woodpecker chart/ -f my-values.yaml