Workload Identity and AKS Application Lifecycle Management Best Practices

We will be creating a simple AKS cluster with Workload Identity and AKS Application Lifecycle Management (ALM) enabled using GitOps and FluxCD. We will then deploy a simple application that uses Workload Identity to mount a secret from Azure Key Vault into the application and Use GitOps and Flux to implement IaaC to manage the application lifecycle.

Introduction and Setup

Lets enable the features to use Workload Identity (currently in preview)

Enable Following Features, providers and CLI extensions:

az extension add --name aks-preview

az provider register --namespace Microsoft.ContainerService
az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
az feature register --namespace "Microsoft.ContainerService" --name "EnableOIDCIssuerPreview" 

Now lets set the environment variables

export USER_ASSIGNED_IDENTITY_NAME="myIdentity"
export KEYVAULT_NAME="mssp-kv"
export KEYVAULT_SECRET_NAME="my-secret"
export RESOURCE_GROUP="rg-aks-cluster"
export CLUSTER_NAME="aks"
export LOCATION="eastus"


# environment variables for the Federated Identity
export SUBSCRIPTION="{your subscription ID}"
# user assigned identity name
export UAID="fic-test-ua"
# federated identity name
export FICID="fic-test-fic-name"

Create Infrastructure

Lets create the Resource Group and the AKS Cluster and get the AKS OIDC Issuer

# Create the Resource Group
az group create --name $RESOURCE_GROUP --location $LOCATION

# Create the AKS Cluster
az aks create -g $RESOURCE_GROUP -n $CLUSTER_NAME --node-count 1 --enable-oidc-issuer --enable-workload-identity --generate-ssh-keys

# Get the AKS OIDC Issuer from the AKS Cluster
export AKS_OIDC_ISSUER="$(az aks show -n $CLUSTER_NAME -g $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -otsv)"

Setup Identity and Key Vault Access

Environment variables for the Kubernetes service account & federated identity credential

# native Kubernetes service account and setting environment variables
export SERVICE_ACCOUNT_NAMESPACE="demo-app"
export SERVICE_ACCOUNT_NAME="workload-identity-sa"
export SERVICE_ACCOUNT_ISSUER="$(az aks show --resource-group ${RESOURCE_GROUP} --name ${CLUSTER_NAME} --query "oidcIssuerProfile.issuerUrl" -otsv)"

# Create the User Assigned Managed Identity
az identity create --name $UAID --resource-group $RESOURCE_GROUP --location $LOCATION --subscription $SUBSCRIPTION

# export the client id of the user assigned identity
export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${UAID}" --query 'clientId' -otsv)"

# creating the federated identity credential using the native k8s service account (1:1 mapping)
az identity federated-credential create --name $FICID --identity-name $UAID --resource-group $RESOURCE_GROUP --issuer $AKS_OIDC_ISSUER --subject system:serviceaccount:$SERVICE_ACCOUNT_NAMESPACE:$SERVICE_ACCOUNT_NAME

Now we update the Kubernertes Manifests for Creating Service Account in Repo to use the Workload Identity or simply create the Service Account through CLI as Below:

az aks get-credentials -n $CLUSTER_NAME -g $RESOURCE_GROUP


cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: ${USER_ASSIGNED_CLIENT_ID}
  labels:
    azure.workload.identity/use: "true"
  name: ${SERVICE_ACCOUNT_NAME}
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
EOF

Now Let's Create the Key Vault and add a secret and grant the user assigned identity access to the secret:

Create an Azure Key Vault, add secret and grab the URL:

```sh
# create a keyvault
az keyvault create --resource-group "${RESOURCE_GROUP}" \
   --location "${LOCATION}" \
   --name "${KEYVAULT_NAME}"

# Create a secret:
az keyvault secret set --vault-name "${KEYVAULT_NAME}" \
   --name "${KEYVAULT_SECRET_NAME}" \
   --value "Hello!"

# Get the Key Vault URL
export KEYVAULT_URL="$(az keyvault show -g ${RESOURCE_GROUP} -n ${KEYVAULT_NAME} --query properties.vaultUri -o tsv)"

# Grant the user assigned identity access to the secret:
az keyvault set-policy --name $KEYVAULT_NAME --secret-permissions get --spn $USER_ASSIGNED_CLIENT_ID

Deploying the application Putting everything together

Now let's deploy an application that uses the secret from the key vault. It can be done by the adding the manifest files in the corresponding namespace folder for GitOps to reconcile the state of the cluster with the state of the repo:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: quick-start
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: ${SERVICE_ACCOUNT_NAME}
  containers:
    - image: ghcr.io/azure/azure-workload-identity/msal-go
      name: oidc
      env:
      - name: KEYVAULT_URL
        value: ${KEYVAULT_URL}
      - name: SECRET_NAME
        value: ${KEYVAULT_SECRET_NAME}
  nodeSelector:
    kubernetes.io/os: linux
EOF

Testing the application access to the secret

For Testing that everything works simply run this command:

kubectl logs pod/quick-start -n ${SERVICE_ACCOUNT_NAMESPACE}

should result in the following output:

.... main.go:30] "successfully got secret" secret="Hello!"

Authenticate Azure deployment workflow by using workload identities

For Deploying infrastructure and other Azure Resources using GitHub Actions we can use the following steps:

# Environment variables
export applicationRegistrationName=GitHub_Action_Test

Create an Azure Active Directory application

# Create an Azure Active Directory application and grab the Object ID
export applicationRegistrationObjectId="$(az ad app create --display-name $applicationRegistrationName --query 'id' -otsv)"

Federated credentials

When an identity needs to communicate with Azure, it signs in to Azure AD. By itself, an application registration doesn't allow a workflow or application to sign in to Azure. You need to assign some credentials first. Federated credentials are one type of application credential. Unlike most credentials, federated credentials don't require you to manage any secrets like passwords or keys.

When you create a federated credential for a deployment workflow, you effectively tell Azure AD and GitHub to trust each other. This trust is called a federation.

The steps involved in the sign-in process are:

When your workflow needs to communicate with Azure, GitHub securely contacts Azure AD to request an access token. GitHub provides information about the GitHub organization (my-github-user), the repository (my-repo), and the branch that the workflow is running on (main). It also includes your tenant ID within Azure AD, the application ID of the workflow identity's application registration, and the Azure subscription ID that your workflow wants to deploy to.

Azure AD validates the application ID and checks whether a federated credential exists within the application for the GitHub organization, repository, and branch.

After Azure AD determines that the request is valid, it issues an access token. Your workflow uses the access token when it communicates with Azure Resource Manager.

Create a federated credential policy file:

{
  "name": "MyFederatedCredential",
  "issuer": "https://token.actions.githubusercontent.com",
  "subject": "repo:my-github-user/my-repo:ref:refs/heads/main",
  "audiences": [
    "api://AzureADTokenExchange"
  ]
}

Use the policy file to create a federated credential:

az ad app federated-credential create \
  --id $applicationRegistrationObjectId \
  --parameters @policy.json

Grant a workload identity access to Azure

Lets create a service principal for the application registration:

# grab the client id of the application registration

export applicationRegistrationClientId="$(az ad app show --id $applicationRegistrationObjectId --query 'appId' -otsv)"

# create a service principal for the application registration
az ad sp create --id $applicationRegistrationClientId

# assign the service principal the contributor role on the resource group
az role assignment create --role contributor --assignee $applicationRegistrationClientId --scope /subscriptions/$SUBSCRIPTION/resourceGroups/$RESOURCE_GROUP

Grant the workflow access to the application registration

# get the tenant id of the Azure AD tenant
export tenantId="$(az account show --query 'tenantId' -otsv)"

Now we save Subscription ID, Tenant ID, Client ID and Client Secret as GitHub Secrets and last but not least we create GitHub Action workflow as below:

name: MyWorkflow

on: [workflow_dispatch]

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
      with:
        path: repo
    - uses: azure/login@v1
      with:
        client-id: ${{ secrets.AZURE_CLIENT_ID }}
        tenant-id: ${{ secrets.AZURE_TENANT_ID }}
        subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
    - uses: azure/arm-deploy@v1
      with:
        resourceGroupName: ToyWebsite
        template: ./deploy/main.bicep

GitOps with FluxCD

Please checkout in detailed instruction on how to setup GitOps with FluxCD