Signed-off-by: Dimitris Karakasilis <email@example.com>
|4 weeks ago|
|chart||1 month ago|
|image||1 month ago|
|.gitignore||1 month ago|
|README.md||4 weeks ago|
|high-level.excalidraw||1 month ago|
|high-level.svg||1 month ago|
|woodpecker-helm-values.yaml||1 month ago|
Self hosted CI
Original blog post about this repository here: https://dimitris.karakasilis.me/2022/12/26/self-hosted-ci.html
This repository helps you run Woodpecker CI on a local cluster and expose it publicly on your desired domain.
- Projects that need more resources than you can afford on the cloud
- Projects that need CI only every now and then, so there is no need to maintain a CI system 24/7
- Projects hosted on code hosting services that don't have integrated CI
- cloud resources are expensive
- local resources are cheap
- always-on, zero-downtime CI is not always required
- CI doesn't have to be a pet
CI should be publicly accessible because:
- Code hosting services (Gitea/Codeberg, Gitlab, Github) use webhooks that need to be able to reach your CI server.
- Commit status should point to some URL that is accessible so that the users can see the results.
Instead of putting the whole CI server (whatever CI solution that may be), we setup a "proxy" (let's call it the "cloud box") which is the only publicly accessible machine. The "cloud box" only needs to have ssh server running.
We use an image that creates an SSH tunnel to the "cloud box" and then runs an nginx reverse proxy to forward all traffic to the traefik load balancer within a Kubernetes cluster.
This way, all requests landing on the "cloud box", will be redirected through the ssh tunnel to the Pod where we run the nginx reverse proxy. Eventually the requests will reach the traefik load balancer and thus every ingress in the cluster.
This setup allows us to run the CI server even behind NAT (on you workstation for example). The "cloud box" is not running the CI, nor any CI jobs whatsoever, so it only needs resources to run ssh. The smallest machine you can find online probably has more resources than needed.
On our cluster we just expose the traefik load balancer to the public, from within the cluster itself. No need to start the cluster in any specific way (forward host ports or anything).
The following describes the above in a more visual way:
Here are the steps to get this running:
Get a public IP "cloud box" EC2, DigitalOcean droplet, whatever suits you best. Just get the smallest (and cheapest) and you are good to go. Make sure you have SSH access to it.
Setup your domain Create an A Record on your nameserver to point your desired domain to the IP address of the box above.
Create a Kubernetes cluster Decide how many resources you are going to need and create a Kubernetes cluster that has enough resource for your needs. If you are not sure, or just playing around, a k3d cluster should be good to get you started:
k3d cluster create woodpecker-ci
Don't worry too much if this is going to be enough. You can replace it later without having to repeat any of the previous steps.
For now, this repository assumes there is a traefik ingress provider running on the cluster.
Warning: What we are going to do, will essentially expose 2 ports of some Pod to the public. Depending on what you expose from your cluster using ingresses, and what software you run (e.g. application frame works), you may be vulnerable to attacks. An attacker gaining shell access to your Pod, could also gain access to your host if for example you run k3d as privileged container. Also, if you run the Woodpecker agents on the same cluster, make sure you don't allow arbitrary Pull Requests to change the pipelines without approval. This makes it extremely easy for anyone to run arbitrary code on your cluster. Make sure you understand the risks and that you take precautions and follow security best practices. In any case, do this at your own risk!
NOTE: An probably more secure alternative would be to create the cluster as a VM. That way you can isolate it from the host machine better. A very easy way to create a cluster using VMs is Kairos. In the future this project may provide a kairos config file that will automatically deploy the provided helm chart. Until then, you can see how this works for metallb in the docs: https://kairos.io/docs/examples/metallb/
- Create a values.yaml file for the helm chart we are going to deploy:
# The email to be used with the lets-encrypt issuer letsencrypt_email: "" jumpbox: # The IP address or URL of the publicly acessible cloud box # This is the machine to which you should point your domain to. url: "<the ip address of the cloud box>" user: <the user that has ssh access to the box> ssh_key: <the private ssh key base64 encoded> ssh_key_pub: <the public ssh key base64 encoded> # Generate a key with: woodpecker_agent_secret: <use `openssl rand -hex 32` to generate one> # Woodpecker subchart values # https://codeberg.org/Codeberg-CI/woodpecker/raw/branch/master/charts/woodpecker-server/values.yaml woodpecker-server: replicaCount: 1 image: registry: docker.io repository: woodpeckerci/woodpecker-server pullPolicy: Always # Overrides the image tag whose default is the chart appVersion. tag: "" env: WOODPECKER_ADMIN: "your codeberg username" WOODPECKER_HOST: "http://<the domain pointing to your cloud box>" WOODPECKER_GITHUB: false WOODPECKER_GITEA: true WOODPECKER_GITEA_URL: "https://codeberg.org" WOODPECKER_GITEA_CLIENT: "<create an oauth2 application on codeberg: https://codeberg.org/user/settings/applications>" WOODPECKER_GITEA_SECRET: "<create an oauth2 application on codeberg: https://codeberg.org/user/settings/applications>" extraSecretNamesForEnvFrom: - woodpecker-secret persistentVolume: enabled: true size: 10Gi mountPath: "/var/lib/woodpecker" storageClass: "local-path" # Use your cluster's storage class here serviceAccount: # Specifies whether a service account should be created create: true ingress: enabled: true annotations: traefik.ingress.kubernetes.io/router.entrypoints: websecure traefik.ingress.kubernetes.io/router.tls: "true" hosts: - host: <the domain pointing to your cloud box> paths: - path: / backend: serviceName: chart-example.local servicePort: 80 tls: - secretName: woodpecker-tls hosts: - <the domain pointing to your cloud box>
This chart will use let's encrypt (production) to issue a certificate for the CI server. To do so, cert-manager should already be installed. You can install cert-manager on your cluster following the docs:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml
Deploy the helm chart:
Wait until cert-manager components are up and running and then install this chart:
helm upgrade --install -n woodpecker --create-namespace woodpecker chart/ -f my-values.yaml