|
2 months ago | |
---|---|---|
deploy | 2 months ago | |
kube_downscaler | 2 months ago | |
tests | 7 months ago | |
unsupported/helm-chart | 7 months ago | |
.flake8 | 1 year ago | |
.gitignore | 1 year ago | |
.pre-commit-config.yaml | 2 months ago | |
.travis.yml | 12 months ago | |
Dockerfile | 1 year ago | |
LICENSE | 3 years ago | |
Makefile | 1 year ago | |
README.md | 2 months ago | |
poetry.lock | 2 months ago | |
pyproject.toml | 2 months ago |
Scale down / "pause" Kubernetes workload (Deployments, StatefulSets, and/or HorizontalPodAutoscalers and CronJobs too !) during non-work hours.
Table of Contents generated with DocToc
Deployments are interchangeable by statefulsets/horizontalpodautoscalers for this whole guide unless explicitly stated otherwise.
It will scale down the deployment's replicas if all of the following conditions are met:
current time is not part of the "uptime" schedule or current time is part of the "downtime" schedule. The schedules are evaluated in the following order:
: - downscaler/downscale-period
or downscaler/downtime
annotation on the deployment/stateful set
- downscaler/upscale-period
or downscaler/uptime
annotation on the deployment/stateful set
- downscaler/downscale-period
or downscaler/downtime
annotation on the deployment/stateful set's namespace
- downscaler/upscale-period
or downscaler/uptime
annotation on the deployment/stateful set's namespace
- --upscale-period
or --default-uptime
CLI argument
- --downscale-period
or --default-downtime
CLI argument
- UPSCALE_PERIOD
or DEFAULT_UPTIME
environment variable
- DOWNSCALE_PERIOD
or DEFAULT_DOWNTIME
environment
variable
the deployment's namespace is not part of the exclusion list:
: - If you provide an exclusion list, it will be used in place of the default (which includes only kube-system).
the deployment's name is not part of the exclusion list
the deployment is not marked for exclusion (annotation
downscaler/exclude: "true"
or
downscaler/exclude-until: "2024-04-05"
)
there are no active pods that force the whole cluster into uptime
(annotation downscaler/force-uptime: "true"
)
The deployment by default will be scaled down to zero replicas. This can
be configured with a deployment or its namespace's annotation of
downscaler/downtime-replicas
(e.g.
downscaler/downtime-replicas: "1"
) or via CLI with
--downtime-replicas
. In case of HorizontalPodAutoscalers, the
minReplicas
field cannot be set to zero and thus
downscaler/downtime-replicas
should be at least 1
.
Regarding CronJobs
, their state will be defined to suspend: true
as you might imagine.
Example use cases:
downscaler/uptime
(or downscaler/downtime
)
annotation. This might be useful for internal tooling frontends
which are only needed during work time.You need to combine the downscaler with an elastic cluster autoscaler to actually save cloud costs. The official cluster autoscaler and the kube-aws-autoscaler were tested to work fine with the downscaler.
Deploy the downscaler into your cluster via (also works with kind or Minikube):
$ kubectl apply -f deploy/
In case you are deploying kube-downscaler
to another namespace than
default
, for example if your context is pointing to my-namespace
.
Make sure you change the deploy/rbac.yaml
Service Account
configuration namespace: default
to the destination namespace
my-namespace
, instead of default
.
The example configuration uses the --dry-run
as a safety flag to
prevent downscaling --- remove it to enable the downscaler, e.g. by
editing the deployment:
$ kubectl edit deploy kube-downscaler
The example deployment manifests come with a configured uptime
(deploy/config.yaml
sets it to "Mon-Fri 07:30-20:30 CET"), you can
overwrite this per namespace or deployment, e.g.:
$ kubectl run nginx --image=nginx
$ kubectl annotate deploy nginx 'downscaler/uptime=Mon-Fri 09:00-17:00 America/Buenos_Aires'
Note that the default grace period of 15 minutes applies to the new nginx deployment, i.e. if the current time is not within Mon-Fri 9-17 (Buenos Aires timezone), it will downscale not immediately, but after 15 minutes. The downscaler will eventually log something like:
INFO: Scaling down Deployment default/nginx from 1 to 0 replicas (uptime: Mon-Fri 09:00-17:00 America/Buenos_Aires, downtime: never)
Note that in cases where a HorizontalPodAutoscaler (HPA) is used along with Deployments, consider the following:
minReplicas
of 0 on HPA is not allowed. Setting Deployment
replicas to 0 essentially disables the HPA. In such a case, the HPA
will emit events like "failed to get memory utilization: unable to
get metrics for resource memory: no metrics returned from resource
metrics API" as there is no Pod to retrieve metrics from.minReplicas
during downtime if there is no/low traffic. If
the Deployment is annotated instead of the HPA, it leads to a race
condition where kube-downscaler scales down the Deployment and HPA
upscales it as its minReplicas
is higher.To enable Downscaler on HPA with [--downtime-replicas=1]{.title-ref}, ensure to add the following annotations to Deployment and HPA.
$ kubectl annotate deploy nginx 'downscaler/exclude=true'
$ kubectl annotate hpa nginx 'downscaler/downtime-replicas=1'
$ kubectl annotate hpa nginx 'downscaler/uptime=Mon-Fri 09:00-17:00 America/Buenos_Aires'
The downscaler is configured via command line args, environment variables and/or Kubernetes annotations.
Time definitions (e.g. DEFAULT_UPTIME
) accept a comma separated list
of specifications, e.g. the following configuration would downscale all
deployments for non-work hours:
DEFAULT_UPTIME="Mon-Fri 07:30-20:30 Europe/Berlin"
To only downscale during the weekend and Friday after 20:00:
DEFAULT_DOWNTIME="Sat-Sun 00:00-24:00 CET,Fri-Fri 20:00-24:00 CET'
Each time specification can be in one of two formats:
<WEEKDAY-FROM>-<WEEKDAY-TO-INCLUSIVE> <HH>:<MM>-<HH>:<MM> <TIMEZONE>
.
The timezone value can be any Olson
timezone, e.g.
"US/Eastern", "PST" or "UTC".<TIME_FROM>-<TIME_TO>
where each <TIME>
is an ISO 8601 date and time of the format
<YYYY>-<MM>-<DD>T<HH>:<MM>:<SS>[+-]<TZHH>:<TZMM>
.Instead of strict uptimes or downtimes, you can chose time periods for upscaling or downscaling. The time definitions are the same. In this case, the upscale or downscale happens only on time periods, rest of times will be ignored.
If upscale or downscale periods are configured, uptime and downtime will
be ignored. This means that some options are mutually exclusive, e.g.
you can either use --downscale-period
or --default-downtime
, but not
both.
This definition will downscale your cluster between 19:00 and 20:00. If you upscale your cluster manually, it won't be scaled down until next day 19:00-20:00.
DOWNSCALE_PERIOD="Mon-Sun 19:00-20:00 Europe/Berlin"
Available command line options:
--dry-run
Dry run mode: do not change anything, just print what would be done
--debug
Debug mode: print more information
--once
Run loop only once and exit
--interval
Loop interval (default: 30s)
--namespace
Restrict the downscaler to work only in a single namespace (default:
all namespaces). This is mainly useful for deployment scenarios
where the deployer of kube-downscaler only has access to a given
namespace (instead of cluster access). If used simultaneously with
--exclude-namespaces
, none is applied.
--include-resources
Downscale resources of this kind as comma separated list. [deployments, statefulsets, stacks, horizontalpodautoscalers] (default: deployments)
--grace-period
Grace period in seconds for new deployments before scaling them down (default: 15min). The grace period counts from time of creation of the deployment, i.e. updated deployments will immediately be scaled down regardless of the grace period.
--upscale-period
Alternative logic to scale up only in given period of time (default:
never), can also be configured via environment variable
UPSCALE_PERIOD
or via the annotation downscaler/upscale-period
on each deployment
--downscale-period
Alternative logic to scale down only in given period of time
(default: never), can also be configured via environment variable
DOWNSCALE_PERIOD
or via the annotation
downscaler/downscale-period
on each deployment
--default-uptime
Default time range to scale up for (default: always), can also be
configured via environment variable DEFAULT_UPTIME
or via the
annotation downscaler/uptime
on each deployment
--default-downtime
Default time range to scale down for (default: never), can also be
configured via environment variable DEFAULT_DOWNTIME
or via the
annotation downscaler/downtime
on each deployment
--exclude-namespaces
Exclude namespaces from downscaling (list of regex patterns,
default: kube-system), can also be configured via environment
variable EXCLUDE_NAMESPACES
. If used simultaneously with
--namespace
, none is applied.
--exclude-deployments
Exclude specific deployments/statefulsets/cronjobs from downscaling
(default: kube-downscaler, downscaler), can also be configured via
environment variable EXCLUDE_DEPLOYMENTS
. Despite its name, this
option will match the name of any included resource type
(Deployment, StatefulSet, CronJob, ..).
--downtime-replicas
Default value of replicas to downscale to, the annotation
downscaler/downtime-replicas
takes precedence over this value.
--deployment-time-annotation
Optional: name of the annotation that would be used instead of the
creation timestamp of the resource. This option should be used if
you want the resources to be kept scaled up during a grace period
(--grace-period
) after a deployment. The format of the
annotation's timestamp value must be exactly the same as for
Kubernetes' creationTimestamp
: %Y-%m-%dT%H:%M:%SZ
. Recommended:
set this annotation by your deployment tooling automatically.
DEFAULT_UPTIME
, DEFAULT_DOWNTIME
, FORCE_UPTIME
and exclusion can
also be configured using Namespace annotations. Where configured these
values supersede the other global default values.
apiVersion: v1
kind: Namespace
metadata:
name: foo
labels:
name: foo
annotations:
downscaler/uptime: Mon-Sun 07:30-18:00 CET
The following annotations are supported on the Namespace level:
downscaler/upscale-period
downscaler/downscale-period
downscaler/uptime
: set "uptime" for all resources in this
namespacedownscaler/downtime
: set "downtime" for all resources in this
namespacedownscaler/force-uptime
: force scaling up all resources in this
namespace - can be true
/false
or a perioddownscaler/exclude
: set to true
to exclude all resources in the
namespacedownscaler/exclude-until
: temporarily exclude all resources in the
namespace until the given timestampdownscaler/downtime-replicas
: overwrite the default target
replicas to scale down to (default: zero)Easiest way to contribute is to provide feedback! We would love to hear what you like and what you think is missing. Create an issue or ping try_except_ on Twitter.
PRs are welcome.
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.