Federating Workloads in Google Kubernetes Engine to Amazon Web Services

Problem: We want to run pods in GKE and have them access resources in AWS

When running a pod in AWS Elastic Kubernetes Service (EKS) you can easily use IAM Roles for Service Accounts (IRSA) to authenticate your workloads to AWS services. But how can you do this when running outside of EKS?

As a personal preference, I'd like to set this up without adding additional helper scripts or additional deployments. The simplicity of IRSA when compared to previous solutions used in EKS made it extremely attractive. I want to get a simple solution in GKE as well.

Background

IRSA uses OpenID Connect (OIDC) to authenticate to AWS IAM. The OIDC tokens are already used inside of Kubernetes, IRSA reuses the same tokens with a federated trust policy in AWS IAM. This federation is quite flexible, Github Actions also uses OIDC tokens and they can be federated with AWS IAM. Google Kubernetes Engine (GKE), like EKS, also offers a trusted OIDC token issuer that can be federated with AWS IAM.

What is federation anyway? Federation is a way to trust the credentials issued by one system in another system. Just like each country issues their own passports and they can be used to prove identity in another country. Federation does not control authorization. Each system has its own rules to determine what permissions a federated identity has. Again, each country decides what permissions are given to enter or work within a country based on what type of passport is presented.

Getting started

Just like using IRSA in EKS, there are a few prerequisites before you can get started:

• Make sure your application uses a supported version of the AWS SDK
• If you are using the Java SDK, make sure that the STS component is included

Creating the components

1. Create the GKE cluster. You'll need the OIDC issuer URL which looks like this: https://container.googleapis.com/v1/projects/${your-project-name}/locations/${your-cluster-location}/clusters/${your-cluster-name}

2. Create a manifest for a Kubernetes Service Account. You don't need to deploy it yet, but you do need to know the Name of the Service Account and the Namespace that it will be deployed in.

3. Create a manifest for a pod (or deployment, statefulset, etc that will create a pod):

   • Attach the service account
   • project the service account token into a volume attached to the pod. Make a note of the audience used in the projected token. You can use any value here, but sts.amazonaws.com is a standard choice.
   • Add an environment variable AWS_WEB_IDENTITY_TOKEN_FILE set to the path to the token you just added.

   Again, the pod doesn't have to be deployed yet.

4. Create an Identity Provider in AWS IAM. You will need the Issuer URL from your GKE cluster, and the audience value from your token.

5. Create an AWS IAM Role trust policy, filling in the values you've collected so far:

   {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "Federated": "arn:aws:iam::${your-aws-account-id}:oidc-provider/${issuer-url-without-https://}"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                  "StringEquals": {
                      "${issuer-url-without-https://}:sub": "system:serviceaccount:${kubernetes-namespace}:${kubernetes-service-account-name}"
                  }
              }
          }
      ]
   }
   

6. Create a AWS IAM Role, or update an existing role, using the trust policy.

7. Update the pod manifest you created earlier: add an environment variable AWS_ROLE_ARN set to the ARN of the role you just created or updated.

8. Deploy the service account and pod manifests.

The pod spec will ultimately look something like this:

---
apiVersion: v1
kind: Pod
metadata:
  name: ${your-pod-name}
  namespace: ${your-namespace}
spec:
  containers:
  - image: ${your-image}
    name: ${your-container-name}
    env:
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        value: /var/run/secrets/tokens/token
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::${your-aws-account-id}:role/${role-name}
    volumeMounts:
    - mountPath: /var/run/secrets/ksa_token
      name: ksa_token
  serviceAccountName: ${your-service-account-name}
  volumes:
  - name: ksa_token
    projected:
      sources:
      - serviceAccountToken:
          path: ksa_token
          expirationSeconds: 86400
          audience: sts.amazonaws.com

The Assume Role with Web Identity credential provider is part of the default set of credential providers, so your app should not require any additional configuration.

This solution should not be limited to GKE. Any Kubernetes cluster where the service account token issuer can be verified by AWS IAM should be able to federate with AWS.

Other solutions:

This solution is pretty similar, but uses a webhook to automatically set the required environment variables and mount the token on your pods. This is the same kind of automation that exists in a EKS cluster.

Another software solution Janus, described here, uses a script executed by the exec credential provider. It's able to work on GCP workloads outside of EKS, but requires that you have a python script and the boto3 library installed.

The solution you choose will depend on what level of automation and portability you require.