TL;DR
Crossplane is a solution you should consider when your infrastructure is to serve your k8s applications.
Story
I have been using Terraform for a very long time. It is simple to use with a very huge community support. However every solution has its shortcomings. Terraform is not really a nice solution when you have to work with k8s.
My AWS infrastructure is closely related to my K8S applications. Maybe I have to upload objects to S3, or store my data in RDS. If you are using AWS EKS, you should be very familiar with IRSA (IAM roles for service accounts) feature, it grants your service accounts with AWS permissions. Then you pods can reference these service accounts and interact with AWS APIs.
So IAM roles are AWS stuffs, and service accounts are K8S stuffs. You have to create both IAM roles and service accounts using some methods.
I can use Terraform to create both IAM roles and service accounts but it is very operational unfriendly. K8s deployments not just only have service accounts need to be applied. How about using Terraform to deploy all AWS infrastructure and k8s manifests? IMO, DO NOT DO IT. Argo CD, Flux CD or any GitOps tools are much better than Terraform.
Never solve a small problem by bringing another big trouble.
Now back to my question, how I should integrate IAM roles with service accounts? How about doing the opposite, use k8s to provision IAM roles?
And then Crossplane comes into my eyes. Simply speaking, Crossplane is a k8s style of Terraform.
Tutorial
Now I am going to show you how to use Crossplane.
Prerequisite
- You need to have an admin privilege in your AWS account.
- A k8s cluster
1. Define your variables
Specify any value you like.
EKS_CLUSTER_NAME=""
AWS_REGION=""
AWS_IAM_ROLE_NAME="${EKS_CLUSTER_NAME}-crossplane-controller"
2. Install Crossplane using helm charts
Everyone loves helm chart :)
helm repo add crossplane https://charts.crossplane.io/master/
helm install --create-namespace --namespace crossplane-system crossplane crossplane/crossplane --version 1.9.0-rc.0.9.g243f1f47
3. Install AWS provider
Crossplane can provision infrastructure in many platforms. Let say if you want to deploy to Azure, you can install Azure provider.
cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
name: aws-config
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${AWS_IAM_ROLE_NAME}
spec:
podSecurityContext:
fsGroup: 2000
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws
spec:
package: crossplane/provider-aws:v0.27.0
controllerConfigRef:
name: aws-config
EOF
4. Create IAM role for IRSA
Again it is a Chicken or the egg problem. Crossplane controller needs to have permissions to provision AWS resources. So we need to manually provision a IAM role for once.
SERVICE_ACCOUNT_NAME=$(kubectl get providers.pkg.crossplane.io provider-aws -o jsonpath="{.status.currentRevision}")
OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
read -r -d '' TRUST_RELATIONSHIP <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${OIDC_PROVIDER}:sub": "system:serviceaccount:crossplane-system:${SERVICE_ACCOUNT_NAME}"
}
}
}
]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json
aws iam create-role \
--role-name "${IAM_ROLE_NAME}" \
--assume-role-policy-document file://trust.json \
--description "IAM role for Crossplane provider-aws"
aws iam attach-role-policy --role-name "${IAM_ROLE_NAME}" --policy-arn=arn:aws:iam::aws:policy/AdministratorAccess
rm trust.json
cat <<EOF | kubectl apply -f -
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
name: aws-provider
spec:
credentials:
source: InjectedIdentity
EOF
6. Try it out
Let try to deploy something.
apiVersion: iam.aws.crossplane.io/v1beta1
kind: Role
metadata:
name: crossplane-sample-role
annotations:
spec:
deletionPolicy: Delete
forProvider:
description: "A role created by Crossplane"
assumeRolePolicyDocument: |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
providerConfigRef:
name: aws-provider
---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: Policy
metadata:
name: crossplane-sample-policy
spec:
deletionPolicy: Delete
forProvider:
name: crossplane-sample-policy
description: A policy created by Crossplane
document: |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster"
],
"Resource": "*"
}
]
}
providerConfigRef:
name: aws-provider
---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: RolePolicyAttachment
metadata:
name: crossplane-sample-role-policy-attachment
spec:
deletionPolicy: Delete
forProvider:
roleNameRef:
name: crossplane-sample-role
policyArnRef:
name: crossplane-sample-policy
providerConfigRef:
name: aws-provider
6. Cleanup
Always a good practice to do cleanup.
helm uninstall crossplane --namespace crossplane-system
kubectl delete ns crossplane-system
kubectl get crd -o name | grep crossplane.io | xargs kubectl delete
aws iam detach-role-policy --role-name ${AWS_IAM_ROLE_NAME} --policy-arn=arn:aws:iam::aws:policy/AdministratorAccess
aws iam delete-role --role-name "${AWS_IAM_ROLE_NAME}"
Discussion
Imagine you need to deploy an application that needs an ALB, RDS. Now you can package all the things using k8s manifests. No more Terraform is involved and much less management overheads.
Take a step forward, you can use Crossplane to replace Terraform to provision any infrastructure. Crossplane allows you to write Crossplane version Terraform modules, which are called Configurations.
So do I still recommend people to use Terraform? Yes I do. Developers might not really comfortable with k8s while writing Terraform is like writing a simple program to them. And Terraform has a large community supporting, new features will be supported quicker and bug fixes will be sooner.
Conclusion
In short, if you are going to deploy some infrastructure that are only for your k8s deployments, e.g. load balancers, IAM Roles for IRSA. Consider Crossplane.
If you are going to deploy some shared infrastructure or it is not relevant to your k8s deployments, e.g. Network, EC2 bastions. Use Terraform.
Top comments (5)
Have you heard of Upbound? They created Crossplane and are doing very cool things to commercialize and bring Crossplane to the masses.
Yes. If you are going to use Crossplane massively, I also suggest you to explore these commercial solutions.
Thanks for sharing. In your example Crossplane is running in EKS cluster with name EKS_CLUSTER_NAME, right ? I cannot use this example if I run Crossplane cluster on Azure AKS or GCP GKE ?
Yes and no.
My tutorial is based on AWS EKS and IRSA is a EKS feature. So you can't simply follow my tutorial in Azure AKS or GCP GKE.
However it is possible that you use Azure or GCP K8S to provision AWS infrastructure. You need to provide a static AWS credential instead of using IRSA feature, which you can checkout Crossplane documentation for details.
Yes this I know. Are you aware about similar features as IRSA in Azure and GCP ?
I know that you can use eg Hashicorp Vault as mentioned in XP documentation but I like IRSA a lot. It is very simple solution comparing to quite advanced vault.