Home

Assuming IAM Roles Using Service Accounts on Amazon EKS

Colin J. Ihrig

EKS has a nice feature called IAM Roles for Service Accounts (IRSA) that allows Kubernetes service accounts to assume AWS IAM roles using annotations. This is nice because it allows you to avoid long-term credentials like access keys in your applications. Unfortunately, like many things on AWS, IRSA can be a bit tedious to configure properly.

In this post we will create an S3 bucket and configure IRSA such that a Kubernetes pod has full access to the bucket. Note that portions of this post are adapted from the AWS documentation.

Create a Kubernetes service account

First, create a service account to test with using the following commands:

$ cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default
EOF
$ kubectl apply -f my-service-account.yaml

Create the S3 bucket

Create an S3 bucket and add a file to it using the AWS CLI:

aws s3api create-bucket --bucket my-test-bucket --region us-east-1
echo Hello | aws s3 cp - s3://my-test-bucket/my-test-file.txt

Try to access the bucket from Kubernetes

Deploy a Kubernetes pod containing the AWS CLI. We will use this pod to test our access to the S3 bucket:

$ cat >my-pod.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: awscli
  namespace: default
spec:
  serviceAccountName: my-service-account
  containers:
  - name: awscli
    image: amazon/aws-cli:latest
    command:
      - "/bin/bash"
      - "-c"
      - "sleep infinity"
EOF
$ kubectl apply -f my-pod.yaml

Once the pod is running, attempt to list the contents of the bucket. You should get a permission error:

$ kubectl exec -it awscli -n default -- aws s3 ls my-test-bucket

Create an OIDC provider

To use IRSA, the EKS cluster needs an OIDC provider associated with its OIDC issuer URL. You can follow the official guide here, but it essentially boils down to the following.

Check if you already have an OIDC provider associated with the cluster:

$ oidc_id=$(aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
$ aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4

If there is output, then you already have an OIDC provider for the cluster. If there is NOT output, then create one using the following command. Note that this command uses eksctl instead of the AWS CLI. I'm not sure how to do it using the AWS CLI.

$ eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve

Configure IRSA

Now it's time to begin configuring IRSA. First, create an IAM policy that grants access to your S3 bucket and its contents:

$ cat >my-test-policy.json <<EOF
{
  "Version":"2012-10-17",
  "Statement": [
    {
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my-test-bucket",
        "arn:aws:s3:::my-test-bucket/*"
      ]
    }
  ]
}
EOF
$ aws iam create-policy --policy-name my-test-policy --policy-document file://my-test-policy.json

Next, create the role that the service account will assume. This role needs to define a trust relationship with the EKS cluster's OIDC provider that we configured earlier.

$ account_id=$(aws sts get-caller-identity --query "Account" --output text)
$ oidc_provider=$(aws eks describe-cluster --name my-cluster --region us-east-1 --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
$ cat >trust-relationship.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$account_id:oidc-provider/$oidc_provider"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$oidc_provider:aud": "sts.amazonaws.com",
          "$oidc_provider:sub": "system:serviceaccount:default:my-service-account"
        }
      }
    }
  ]
}
EOF
$ aws iam create-role --role-name my-test-role --assume-role-policy-document file://trust-relationship.json --description "my-irsa-role"

Attach the policy we created to the role:

$ aws iam attach-role-policy --role-name my-test-role --policy-arn=arn:aws:iam::$account_id:policy/my-test-policy

Add the eks.amazonaws.com/role-arn annotation to the service account. This is where the IRSA magic happens:

$ kubectl annotate serviceaccount -n default my-service-account eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-test-role

Validate the configuration

If everything was configured properly, you should be able to successfully run this command from the beginning of this blog post:

$ kubectl exec -it awscli -n default -- aws s3 ls my-test-bucket

If you still get a permission error, here are a few things you can try to debug the issue:

  • kubectl describe pod awscli -n default should show that the pod is running under the service account that we created.
  • kubectl describe serviceaccount my-service-account -n default should show the eks.amazonaws.com/role-arn annotation, and it should refer to the correct IAM role ARN.
  • aws iam list-attached-role-policies --role-name my-test-role --query AttachedPolicies[].PolicyArn --output text should show the IAM policy that we created.
  • aws iam get-policy --policy-arn arn:aws:iam::$account_id:policy/my-test-policy should find the policy that we created.
  • aws iam get-policy-version --policy-arn arn:aws:iam::$account_id:policy/my-test-policy --version-id v1 should show the JSON body of the policy that we created (assuming you didn't create any other policy versions). Make sure that the policy looks correct, and grants permissions on both arn:aws:s3:::my-test-bucket and arn:aws:s3:::my-test-bucket/*.
  • aws iam get-role --role-name my-test-role --query Role.AssumeRolePolicyDocument should match the trust policy that we created. Verify that the various ARNs and other identifiers look correct.