My website/blog hosted on Codeberg pages.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

227 lines
18 KiB

<!DOCTYPE html>
<html xmlns="">
<title>Understanding AWS IAM Roles for Kubernetes ServiceAccounts</title>
<meta name="owner" content="Michael Zeevi">
<meta name="designer" content="Michael Zeevi">
<meta name="author" content="Michael Zeevi">
<meta name="copyright" content="Michael Zeevi">
<meta name="date" content="2021-06-26">
<meta name="revised" content="2021-06-26">
<meta name="keywords" content="devops, aws, eks, kubernetes, irsa, oidc, jwt">
<meta name="generator" content="pandoc"> <!-- template by Michael Zeevi -->
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta charset="utf-8">
<link rel="stylesheet" href="res/styles.css">
<link rel="stylesheet" href="" integrity="sha384-B4dIYHKNBt8Bc12p+WXckhzcICo0wtJAoU8YZTY5qE0Id1GSseTk6S+L3BlXeVIU" crossorigin="anonymous">
<header class="wrapper">
<h1 id="site-title"><a href="index.html"></a></h1>
<ul id="site-menu">
<li><a href="index.html">Home</a></li>
<li><a href="blog.html">Tech Blog</a></li>
<li><a href="links.html">Links</a></li>
<li><a href="photography.html">Photography</a></li>
<div id="colorscheme">
<a id="colorscheme-toggle">
<i class="fas fa-fw fa-moon"></i><i class="fas fa-fw fa-toggle-on" id="colorscheme-toggle-switch"></i><i class="fas fa-fw fa-sun"></i>
<script src="res/colorscheme.js"></script>
<hr class="hidden-on-normal-displays">
<h1 id="content-title">Understanding AWS IAM Roles for Kubernetes ServiceAccounts</h1>
<p class="content-header author">Michael Zeevi</p>
<p class="content-header date"><time datetime="2021-06-26">2021-06-26</time></p>
<h2 id="intro">Intro</h2>
<p>Modern cloud and microservice based applications often reside in Kubernetes, running on cloud infrastructure such as AWS’ EKS. Such applications commonly harness additional cloud resources and services such as S3, RDS, SQS, etc.; in order to do so in a secure manner (preserving the <em>least privilege</em> security principle) one must only grant access (for said cloud resources) to the appropriate microservices (i.e. their pods).</p>
<p>Both Kubernetes and AWS have their permission management systems - RBAC and IAM (respectively), which are both well tailored for access management within their own realms. However, the above case demands both, since our principal microservice is a Kubernetes resource (such as a Deployment), whilst the resource to be accessed is on AWS (such as an S3 Bucket or SQS queue).</p>
<p>In this post we will explore and understand how to utilize both Kubernetes’ RBAC and AWS’ IAM permission management systems in such cases, forming a hybridized solution called <em>IAM Roles for ServiceAccounts</em> (IRSA).</p>
<h2 id="the-concept-of-irsa">The concept of IRSA</h2>
<p>The solution to our case requires bridging between both permissions systems in two places:</p>
<li><p>In Kubernetes the ServiceAccount (which provides an identity for processes runinng in Pods) must be appropriately annotated with the Amazon resource name (ARN) of the IAM Role to be associated with it. Thus when a Pod assigned with this ServiceAccount makes a request to assume an IAM Role, then its secret token is also transmitted. This token contains a payload specifying the ServiceAccount’s <em>name</em> and <em>namespace</em>, along with other various metadata and is signed by the Kubernetes API Server. One can consider this as the method of <em>authentication</em> (proving ones identity).</p></li>
<li><p>In the AWS IAM service we create an <em>Identity provider</em>, which must be configured with the Kubernetes API Server’s thumbprint. Thus when a request to assume an IAM Role is received, then it can be validated since the token is signed by a trusted body (the Kubernetes API Server). This forms a trust relationship between the two systems, allowing AWS IAM to <em>authorize</em> (approve) the role assumption request.</p></li>
<p>This closes a circle between Kubernetes and AWS.</p>
<h2 id="practical-implementation-guide">Practical implementation guide</h2>
<p>For this guide we will assume a workload (Pods, Deployment...) running in an AWS EKS Kubernetes cluster requires access to certain AWS resources...</p>
<p>Note: If you deployed Kuberenetes manually then you will need to <a href="">enable the OpenID Connect (OIDC) plugin for your Kubernetes API Server</a>.</p>
<h3 id="in-aws">In AWS</h3>
<ol type="1">
<li><p>Get the cluster’s <strong>OIDC provider URL</strong>. In EKS it can be found in the web console under the cluster’s <em>Details</em> tab (or it can be retrieved via the AWS CLI with the command: <code>aws eks describe-cluster --name $YOUR_CLUSTER --output text --query "cluster.identity.oidc.issuer"</code>).</p>
<p>The value should look similar to this (with a different Id at the end):</p>
<li><p>Create an <strong>IAM Identity provider</strong> of type <em>OpenID Connect</em>, place the value from the previous stage in the <em>Provider URL</em> field and set the <em>Audience</em> to <code></code>.</p>
<p>Click <em>Get thumbprint</em> and then click <em>Add provider</em> (at the bottom).</p>
<li><p>If using Terraform with <a href="">the official EKS module</a>, then just set the module’s input variable <code>enable_irsa = true</code>.</p></li>
<li><p>If using Terraform without the module, then add:</p>
<pre><code>data &quot;tls_certificate&quot; &quot;cluster&quot; {
url = aws_eks_cluster.your_cluster.identity[0].oidc[0].issuer
resource &quot;aws_iam_openid_connect_provider&quot; &quot;this&quot; {
client_id_list = [&quot;;]
thumbprint_list = [data.tls_certificate.cluster.certificates.0.sha1_fingerprint]
url = aws_eks_cluster.your_cluster.identity[0].oidc[0].issuer
<li><p>Create an <strong>IAM Role</strong> with a trusted entity of type <em>Web identity</em> (instead of the default <em>AWS service</em> type), under <em>Identity provider</em> select the Identity provider we created in the previous stage and under <em>Audience</em> select -once again- <code></code>.</p>
<p>After this any required AWS IAM Policies can be attached normally to the IAM Role.</p></li>
<h3 id="in-kubernetes">In Kubernetes</h3>
<ol start="4" type="1">
<li><p>Create a <strong>Kubernetes ServiceAccount</strong>, and <strong>annotate</strong> it with the ARN of the IAM Role from the previous stage:</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
name: can-do-stuff-on-aws
namespace: testing
annotations: arn:aws:iam::YOUR_ACCOUNT_NUMBER:role/can-do-stuff-on-aws</code></pre>
<li>The ServiceAccount name [<code></code>] doesn’t have to be the same as the IAM Role name [<code>metadata.annotations.eks....</code>] (but it can help remembering).</li>
<li><em>No</em> Kubernetes Role is <em>bound</em> to the ServiceAccount!</li>
<li><p><strong>Assign</strong> the ServiceAccount to a Pod (or workload, such as a Deployment):</p>
<pre><code>apiVersion: v1
kind: Pod
name: my-app
namespace: testing
serviceAccountName: can-do-stuff-on-aws
- name: my-app
image: nginx:alpine</code></pre></li>
<p>Once these resources have all been provisioned in the cluster and cloud, any application running in a Pod assigned with the ServiceAccount will have the AWS access rights defined in the IAM policies attached to the IAM Role!</p>
<h2 id="deep-dive-examining-the-trust-relationship">Deep-dive: Examining the trust relationship</h2>
<p>Let’s take a closer look at all the components at play and see exactly how the circle of trust is achieved and how they fit together.</p>
<img src="res/irsa/diagram.png" alt="" /><figcaption>IRSA trust relationship diagram</figcaption>
<h3 id="the-assume-role-policy">The Assume role policy</h3>
<p>Once the IAM Role exists, then under its <em>Trust relationships</em> tab, clicking on <em>Edit trust relationship</em> will show its <strong><em>Assume role policy</em></strong> (not to be confused with <em>IAM policy</em>!).</p>
<p>It should look similar to this:</p>
&quot;Version&quot;: &quot;2012-10-17&quot;
&quot;Statement&quot;: [
&quot;Effect&quot;: &quot;Allow&quot;
&quot;Principal&quot;: {
&quot;Federated&quot;: &quot;arn:aws:iam::YOUR_ACCOUNT_NUMBER:oidc-provider/;
&quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;
&quot;Condition&quot;: {
&quot;StringEquals&quot;: {
&quot;;: &quot;system:serviceaccount:testing:can-do-stuff-on-aws&quot;
<p>What this policy enforces is that this IAM Role can only be <strong>assumed</strong> via the AWS’ Secure Token Service when requested by a Web Identity [see <code>"Action"</code>] - specifically <em>our</em> OIDC Identity provider [see <code>"Principal"</code>], and - most importantly - only under the condition that the token’s payload string references the authorized subject (“<em>sub</em>”) - i.e. the <strong>Kubernetes ServiceAccount</strong> [see <code>"StringEquals"</code>, under <code>"Condition"</code>].</p>
<h3 id="the-serviceaccount">The ServiceAccount</h3>
<p>When a Kubernetes ServiceAccount is created, it automatically has a special <strong>Kubernetes Secret</strong> (of type <code></code>) created for it; it can be seen referenced under <em>Mountable secrets</em> when the ServiceAccount is described:</p>
<pre><code>kubectl -n testing describe serviceaccount can-do-stuff-on-aws</code></pre>
<p>Which returns:</p>
<pre><code>Name: can-do-stuff-on-aws
Namespace: testing
Labels: &lt;none&gt;
Annotations: arn:aws:iam::YOUR_ACCOUNT_NUMBER:role/can-do-stuff-on-aws
Image pull secrets: &lt;none&gt;
Mountable secrets: can-do-stuff-on-aws-token-5r5xg
Tokens: can-do-stuff-on-aws-token-5r5xg
Events: &lt;none&gt;</code></pre>
<p>Note: Your secret’s name will have a different random suffix than my <code>-5r5xg</code>.</p>
<h3 id="the-secret">The Secret</h3>
<p>If we describe the Kubernetes Secret itself, by running:</p>
<pre><code>kubectl -n testing describe secret can-do-stuff-on-aws-token-5r5xg</code></pre>
<p>It returns:</p>
<pre><code>Name: can-do-stuff-on-aws-token-5r5xg
Namespace: testing
Labels: &lt;none&gt;
Annotations: can-do-stuff-on-aws 59cf0215-e56c-4534-a889-3c8a6c1ada3d
ca.crt: 1066 bytes
namespace: 7 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjJvNTQwbDBodFpWMUlqX2ktOEZPQ1NJaWZuMENESTZPam53MzVmZUxoR1UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ0ZXN0aW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNhbi1kby1zdHVmZi1vbi1hd3MtdG9rZW4tNXI1eGciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2FuLWRvLXN0dWZmLW9uLWF3cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjU5Y2YwMjE1LWU1NmMtNDUzNC1hODg5LTNjOGE2YzFhZGEzZCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp0ZXN0aW5nOmNhbi1kby1zdHVmZi1vbi1hd3MifQ.Pyf4jdNNQnIH3NO2x2RIrSuecRXlzAFV3c9Ed4kK4OV2sI49RJRQI_A3rEDh-QanKJBdt0BY98G_30QWokmCfwuMbJunb7o2qUKHu4qHkcYYUgxFpGFMNZnMFmZ1hOqSOWX7b6pcfGJtH40nvw7U4FSsKAkON3lI5eQmu2e5hSIgqJgHhNhFmSpRCxdbBSBOOPcHeONQQLuKZ2ogHA6DZ1udJYjIaDMFiSiCngjwAJCccK3r75W5-DQ8jXv5J8peW-UnLNz8A3dUzc9kbzVzg2-_Uc698cnkDjH1yuE7KS8OWSqjqogIN1spuhcc7J6qmO9iBDZGsOcgzyrBiet7TQ</code></pre>
<p>Then under <em>Data</em> we can find the actual <em><strong>token</strong></em>.</p>
<h3 id="the-token">The Token</h3>
<p>Kubernetes ServiceAccounts’ secret tokens use a standard format called <a href="">JSON Web Token (JWT)</a> (which is <em>not</em> native to Kubernetes).</p>
<p>Using a <a target="_blank" href="" title="decode our token">JWT debugger to decode it</a>, one can see it’s composed of three parts (each separately base64 encoded) - a <code>HEADER</code>, <code>PAYLOAD</code> and <code>VERIFY SIGNATURE</code> - which are concatenated with periods.</p>
<p>The part that is of most interest is the <code>PAYLOAD</code>, which most importantly contains the <code>"sub"</code> (subject) field:</p>
&quot;iss&quot;: &quot;kubernetes/serviceaccount&quot;,
&quot;;: &quot;testing&quot;,
&quot;;: &quot;can-do-stuff-on-aws-token-5r5xg&quot;,
&quot;;: &quot;can-do-stuff-on-aws&quot;,
&quot;;: &quot;59cf0215-e56c-4534-a889-3c8a6c1ada3d&quot;,
&quot;sub&quot;: &quot;system:serviceaccount:testing:can-do-stuff-on-aws&quot;
<p>Note: The <code>"sub"</code> field’s value exactly matches the <code>"StringEquals"</code> condition from the <strong>Assume role polcy</strong> examined earlier.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In this post we learned about the problem which IRSA offers to solve, whilst understanding the requirements of a trust relationship between the AWS and Kubernetes permissions management systems, and we went over all the steps (across <em>both</em> platforms) to implementing the solution.</p>
<p>Following that, we went under the hood to examine the components of the <em>Assume role policy</em> that every IAM Role has, and - finally - explored JWT tokens which are used by Kubernetes ServiceAccounts, seeing what their <em>Payload</em> contains and how they relate to IAM Roles’ <em>Assume role policy</em>.</p>
<h2 id="sources-additional-info">Sources &amp; additional info</h2>
<li><a target="_blank" href="">AWS documentation for IRSA</a></li>
<li><a target="_blank" href="">JWT official introduction</a></li>
<li><a target="_blank" href="">Kubernetes documentation - Authenticating to the Kubernetes API server</a></li>
<hr class="hidden-on-normal-displays">
<ul id="social">
<li><a rel="author" target="_blank" href="" title="Mastodon:" ><i class="fab fa-fw fa-mastodon" ></i><div class="hidden-on-tiny-displays"> Mastodon</div></a></li>
<li><a rel="author" target="_blank" href="" title="Pixelfed:"><i class="fas fa-fw fa-camera" ></i><div class="hidden-on-tiny-displays"> Pixelfed</div></a></li>
<li><a rel="author" target="_blank" href="" title="Codeberg: maze" ><i class="fas fa-fw fa-code-branch"></i><div class="hidden-on-tiny-displays"> Codeberg</div></a></li>
<li><a rel="author" target="_blank" href="" title="E-mail: Michael Zeevi" ><i class="fas fa-fw fa-envelope" ></i><div class="hidden-on-tiny-displays"> E-mail</div></a></li>
<li><a rel="author" target="_blank" href="res/pgp.asc" title="PGP: Public key block" ><i class="fas fa-fw fa-fingerprint"></i><div class="hidden-on-tiny-displays"> PGP</div></a></li>
<li><a rel="author" target="_blank" href="" title="Keyoxide"><i class="fas fa-fw fa-key" ></i><div class="hidden-on-tiny-displays"> Keyoxide</div></a></li>
<p id="license" xmlns:cc="" xmlns:dct="">
<a href="index.html" property="dct:title" rel="cc:attributionURL"></a> by
<a target="_blank" href="" property="cc:attributionName" rel="cc:attributionURL dct:creator">Michael Zeevi</a> is licensed under
<a target="_blank" href="" rel="license noopener noreferrer">CC BY-SA 4.0</a>