DEV Community

Cover image for Implementing GitOps with Argo CD, GitHub, and Azure Kubernetes Service
Ivan Porta
Ivan Porta

Posted on • Originally published at Medium

Implementing GitOps with Argo CD, GitHub, and Azure Kubernetes Service

#azure #devops #kubernetes #gitops

Many organizations have already shifted to DevOps as part of their digital transformation strategy. This promotes an environment where collaboration and shared responsibility foster a more cross-functional setting, necessitating processes that enable developers to perform more IT operations-related tasks. In this article, I will explain what GitOps is and demonstrate its application using ArgoCD, GitHub, and Azure Kubernetes Service (AKS).

What’s GitOps?

GitOps is a developer-centric framework for operational practices that is declarative and based on the version control system Git, a term coined by Weaveworks in 2017. It facilitates the automation of infrastructure provisioning and software deployment. Developers store infrastructure as code (IaC) files, configuration files, and application codes in Git repositories, which represent the desired state of their infrastructure. When a commit is pushed to a monitored branch in the source control, it triggers an update that aligns the cloud infrastructure or application to match the new state.

How GitOps works?

The GitOps workflow involves the following steps:

  1. The developer begins a new task by creating a branch from the main production branch.
  2. The developer writes configuration files that define the infrastructure and resources needed.
  3. Once the development is completed, a pull request (PR) is created for team review and approval.
  4. Upon approval, the code is merged into the production branch, which the GitOps tool monitors. The tool detects discrepancies between the current deployed state and the repository state, raising an alert for manual synchronization or automatically updating the infrastructure.

There are many tools available for adopting GitOps, such as Azure DevOps Pipelines, GitHub Actions, and CircleCI. Some tools, like Azure DevOps, CircleCI, and Jenkins, have a broader scope, while others are more Kubernetes-centric. For this demonstration, I will be using ArgoCD.

What’s Argo CD?

ArgoCD is an open-source GitOps tool that is installed within a Kubernetes cluster. It offers features like RBAC, multi-cluster deployment, and automatic synchronization with Git repositories. ArgoCD streamlines continuous delivery workflows by keeping applications in sync with their desired state as defined in Git. Its web interface and CLI are designed for accessibility, promoting collaboration between developers and operations teams and enhancing reliability for Kubernetes applications.

Initial setup

Before deploying ArgoCD, we must provision an AKS cluster. Below is a Terraform configuration for setting up two AKS clusters — one for ArgoCD and another for deploying applications. Save this configuration in a file named main.tf.

provider "azurerm" {
  features = {}
}
resource "azurerm_resource_group" "aks_rg" {
  name     = "rg-training-eus-01"
  location = "East US"
}
resource "azurerm_virtual_network" "aks_vnet" {
  name                = "vnet-training-argocd-eus-01"
  address_space       = ["10.0.0.0/8"]
  resource_group_name = azurerm_resource_group.aks_rg.name
  location            = azurerm_resource_group.aks_rg.location
}
resource "azurerm_kubernetes_cluster" "aks_argocd" {
  name                = "aks-training-dev"
  resource_group_name = azurerm_resource_group.aks_rg.name
  location            = azurerm_resource_group.aks_rg.location
  dns_prefix          = "argocdcluster"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_DS2_v2"
  }

  identity {
    type = "SystemAssigned"
  }
}
resource "azurerm_kubernetes_cluster" "aks_apps" {
  name                = "aks-training-dev-02"
  resource_group_name = azurerm_resource_group.aks_rg.name
  location            = azurerm_resource_group.aks_rg.location
  dns_prefix          = "appscluster"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_DS2_v2"
  }

  identity {
    type = "SystemAssigned"
  }
}
resource "azurerm_container_registry" "acr" {
  name                     = "acrtrainingeus01"
  resource_group_name      = azurerm_resource_group.aks_rg.name
  location                 = azurerm_resource_group.aks_rg.location
  sku                      = "Standard"
  admin_enabled            = false
}
resource "azurerm_role_assignment" "acr_pull" {
  scope                = azurerm_container_registry.acr.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.aks_argocd.kubelet_identity[0].object_id
}
resource "azurerm_role_assignment" "acr_pull_second" {
  scope                = azurerm_container_registry.acr.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.aks_apps.kubelet_identity[0].object_id
}
Enter fullscreen mode Exit fullscreen mode

To initialize the Terraform configuration, use the following command:

$ terraform init
Enter fullscreen mode Exit fullscreen mode

Deploy the configuration with:

$ terraform apply — auto-approve
Enter fullscreen mode Exit fullscreen mode

With the AKS cluster in place, you can proceed to install ArgoCD.

Install Argo CD

To install ArgoCD, first retrieve your cluster credentials using the Azure CLI:

$ $ az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
$ az aks get-credentials --resource-group rg-training-dev --name aks-training-dev
Enter fullscreen mode Exit fullscreen mode

The installation of Argo CD is straightforward and can be accomplished by deploying the official manifest file from GitHub at the following URL: https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml. It's best practice to deploy it in a separate namespace. You can create a new namespace and apply the manifest file with these commands:

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
Enter fullscreen mode Exit fullscreen mode

The above commands will create three Custom Resource Definitions (CRDs) for Applications, ApplicationSets, and AppProjects, as well as multiple roles, ClusterRoles with their respective bindings, and two Secrets. These Secrets are treated as opaque blobs, which Kubernetes does not decode or process specifically. The commands also create multiple ConfigMaps and Deployments.

Login to Argo CD

By default, Argo CD’s API server is not exposed to the internet. You can set up an Ingress Controller or expose the API server by changing the service type to LoadBalancer with the following command:

kubectl patch svc argocd-server -n argocd — type=’json’ -p=”[{‘op’:’replace’, ‘path’:’/spec/type’, ‘value’:’LoadBalancer’}]”
Enter fullscreen mode Exit fullscreen mode

(Optional) If you want to connect to Argo CD directly from your localhost, you can use the port-forward method to create a streaming connection from your computer to the Argo CD service running on AKS:

kubectl port-forward svc/argocd-server -n argocd 8080:443
Enter fullscreen mode Exit fullscreen mode

However, you will need to keep this connection open in your terminal or command line instance.

Image description

Alternatively, obtain the external IP address of the load balancer to access the Argo CD UI directly with:

$ kubectl get svc argocd-server -n argocd
Enter fullscreen mode Exit fullscreen mode

Then browse to the LoadBalancer’s external IP address.

Image description

The default login credentials for Argo CD use the admin username and an auto-generated password, which can be retrieved as follows:

For PowerShell:

$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | ForEach-Object { [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($_)) };
Enter fullscreen mode Exit fullscreen mode

For Bash:

$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
Enter fullscreen mode Exit fullscreen mode

Adding an additional cluster to Argo CD

To add an additional AKS cluster to Argo CD, you must first install the Argo CD CLI on your machine. Retrieve the latest version and download the corresponding executable with these commands:

$version = (Invoke-RestMethod https://api.github.com/repos/argoproj/argo-cd/releases/latest).tag_name
Invoke-WebRequest -Uri "https://github.com/argoproj/argo-cd/releases/download/$version/argocd-windows-amd64.exe" -OutFile "argocd.exe"
Enter fullscreen mode Exit fullscreen mode

Then, add the path of the downloaded executable to your system’s environment variables to facilitate access:

$ [Environment]::SetEnvironmentVariable(“Path”, “$env:Path;C:\Users\Ivan\ArgoCD-CLI”, “User”)
Enter fullscreen mode Exit fullscreen mode

To add a cluster, update your kubectl context with the new cluster's details using the commands you previously used, this time pointing to the applications' AKS:

$ az account set — subscription 36e90881–8db4–4f13-aa7a-8e9d83febdf4
$ az aks get-credentials — resource-group rg-training-dev — name aks-training-dev-01
Enter fullscreen mode Exit fullscreen mode

You can verify the new cluster’s context with the following command:

$ kubectl config get-contexts
CURRENT   NAME                                                         CLUSTER                                                      AUTHINFO                                                     NAMESPACE
*         aks-training-dev                                             aks-training-dev                                             clusterUser_rg-training-dev_aks-training-dev
          aks-training-dev-02                                          aks-training-dev-02                                          clusterUser_rg-training-dev_aks-training-dev-
Enter fullscreen mode Exit fullscreen mode

Once the context is set, log in to the Argo CD CLI using the same credentials you use for the web application:

$ argocd login 20.253.8.230
WARNING: server certificate had error: tls: failed to verify certificate: x509: certificate signed by unknown authority. Proceed insecurely (y/n)? y
Username: admin
Password:
'admin:login' logged in successfully
Context '20.253.8.230' updated
Enter fullscreen mode Exit fullscreen mode

Acknowledge any certificate warnings and proceed to log in. After successful login, you can add the cluster to Argo CD with:

$ argocd cluster add aks-training-dev-02
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `aks-training-dev-02` with full cluster level privileges. Do you want to continue [y/N]? y
time="2023-11-10T10:04:55+01:00" level=info msg="ServiceAccount \"argocd-manager\" created in namespace \"kube-system\""time="2023-11-10T10:04:55+01:00" level=info msg="ClusterRole \"argocd-manager-role\" created"
time="2023-11-10T10:04:56+01:00" level=info msg="ClusterRoleBinding \"argocd-manager-role-binding\" created"
time="2023-11-10T10:05:01+01:00" level=info msg="Created bearer token secret for ServiceAccount \"argocd-manager\""
Cluster 'https://aks-training-dev-02-dns-uxajaze6.hcp.eastus.azmk8s.io:443' added
Enter fullscreen mode Exit fullscreen mode

Image description

This will create a service account argocd-manager on the cluster referenced by the context aks-training-dev-02 with full cluster-level privileges. Confirm when prompted to continue. After the service account argocd-manager is created, along with the associated ClusterRole and ClusterRoleBinding, verify that the new cluster is managed by Argo CD via both the CLI and the web application with:

$ argocd cluster list
SERVER                                                             NAME                 VERSION  STATUS      MESSAGE  PROJECT
https://aks-training-dev-02-dns-uxajaze6.hcp.eastus.azmk8s.io:443  aks-training-dev-02  1.26     Successful
https://kubernetes.default.svc                                     in-cluster           1.26     Successful
Enter fullscreen mode Exit fullscreen mode

Add a new user

When you first install Argo CD, it automatically creates a built-in admin user with full access. A best practice is to create local users and disable the built-in admin account. To create a new user, define a new ConfigMap resource as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  accounts.gtrekter: apiKey, login
  accounts.gtrekter.enabled: "true"
Enter fullscreen mode Exit fullscreen mode

The username follows the ‘accounts’ string and accepts two values:

  • apiKey to allow generating API keys
  • login to allow login using the UI. As soon as the user is created, you can proceed with disabling the admin account. Edit the ConfigMap and set the admin.enabled key to false as shown below: 
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  accounts.gtrekter: apiKey, login
  accounts.gtrekter.enabled: "true"
  admin.enabled: "false"
Enter fullscreen mode Exit fullscreen mode

The password of the newly created user it’s going to be the same as the admin. However, you can easily change it by using the following command:

$ argocd account update-password --account gtrekter --current-password xxxxxxxxxxxxxxxx --new-password xxxxxxxxxxxxxxxx
Enter fullscreen mode Exit fullscreen mode

Even if we allow our new user to log in to the UI and generate a new token, there is little they can do without RBAC permissions. RBAC roles are managed via the argocd-rbac-cm, so the first step is to pull it using:

$ kubectl get cm argocd-rbac-cm -n argocd -o yaml > argocd-rbac.yml
Enter fullscreen mode Exit fullscreen mode

The content of the file will look as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-rbac-cm","app.kubernetes.io/part-of":"argocd"},"name":"argocd-rbac-cm","namespace":"argocd"}}
  creationTimestamp: "2023-11-09T20:06:17Z"
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-rbac-cm
  namespace: argocd
  resourceVersion: "2778"
  uid: b8369700-a83e-47d9-8ae6-5f2eb2b0110d
Enter fullscreen mode Exit fullscreen mode

The permissions are defined with the following formats:

p, <role/user/group>, <resource>, <action>, <object>
Enter fullscreen mode Exit fullscreen mode

You can specify a variety of resources, such as clusters, projects, applications, applicationsets, repositories, certificates, accounts, gpgkeys, logs, exec, and extensions. The actions available for these resources include get, create, update, delete, sync, override, and action/<group/kind/action-name>.

In this demo, we will grant the user ‘gtrekter’ the rights to create new applications and view clusters. We will add the following policy to the ConfigMap that we have obtained:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-rbac-cm","app.kubernetes.io/part-of":"argocd"},"name":"argocd-rbac-cm","namespace":"argocd"}}
  creationTimestamp: "2023-11-09T20:06:17Z"
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-rbac-cm
  namespace: argocd
  resourceVersion: "2778"
  uid: b8369700-a83e-47d9-8ae6-5f2eb2b0110d
data:
  policy.csv: |
    p, role:developer, applications, *, */*, allow
    p, role:developer, clusters, get, *, allow
    p, role:developer, repositories, get, *, allow
    g, gtrekter, role:developer
Enter fullscreen mode Exit fullscreen mode

To apply the updated RBAC configuration, run the following command:

$ kubectl apply -f argcd-rbac.yml
Enter fullscreen mode Exit fullscreen mode

Deploying your first ArgoCD Application

You can deploy a new application using either the Argo CD CLI or the Argo CD UI. In this demo, we will focus on using the Argo CD UI. After logging into the UI, click on the New App button.

Image description

In the panel that appears, configure several settings in the General section: name the application, select the project for grouping and access control, choose the sync policy (automatic or manual), and decide whether to enable cascading deletion with the “Set deletion finalizer” option. Sync options also allow you to manage schema validation, resource pruning order, differential ignores, namespace creation, and server-side application, as well as applying changes only when they are out of sync.

  • Skip schema validation: Bypasses schema validation steps during the sync process in Argo CD, useful for custom resources not recognized by Argo CD.
  • Prune last: Deletes resources no longer present in the desired state after other changes are applied, maintaining service availability during updates.
  • Respect Ignore Differences: Prevents differences between desired and live states that are intentional from triggering a sync.
  • **Auto-create namespace: Automatically creates the specified Kubernetes namespace for the application if it does not already exist.
  • Apply out of sync only: Applies changes only to resources that are out of sync with the repository’s desired state, allowing for selective syncing.
  • Server-side apply: Utilizes Kubernetes’s server-side apply feature during the sync process.

Regarding the prune propagation policy, there are three options:

  • Foreground: Deletes “dependent” objects first, ensuring all related objects are cleaned up before the main object is removed from the cluster.
  • Background: Deletes the main object immediately, with Kubernetes cleaning up dependent objects in the background.
  • Orphan: Leaves dependent objects in the cluster when the main object is deleted, effectively “orphaning” them.

Image description

The next section should contain the repository URL, the revision (branch or tag) to be monitored by Argo CD, and the path to the folder with relevant files. For this demo, the public training repository with manifests stored in the ‘argocd’ subdirectory will be used.

Image description

In the Destination section, specify where to deploy the application and its related namespace.

Image description

The Directory section offers configuration options for managing the deployment:

  • Top-Level arguments: Parameterize the application definition to override configurations within Argo CD directly.
  • Directory recurse: Specify whether to recursively sync all subdirectories within the path.
  • External variables: Define external variables used by templates or configurations.
  • Include/Exclude: Specify which files or directories to include or exclude during the sync process.

After filling out the configuration, click the Create button.

Image description

Once you completed filling up the configuration, click the create button. The modal will close, and your new application will be listed with the status OutOfSync and the health status Missing.

Image description

This indicates that the Argo CD application has been created but not yet deployed. In fact, if we list all the deployments in the cluster, we will see just the default and the Argo CD related resources

$ kubectl get applications -n argocd
NAME        SYNC STATUS   HEALTH STATUS
guestbook   OutOfSync     Missing
$ kubectl get deployments --all-namespaces
NAMESPACE         NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
argocd            argocd-applicationset-controller   1/1     1            1           2d22h
argocd            argocd-dex-server                  1/1     1            1           2d22h
argocd            argocd-notifications-controller    1/1     1            1           2d22h
argocd            argocd-redis                       1/1     1            1           2d22h
argocd            argocd-repo-server                 1/1     1            1           2d22h
argocd            argocd-server                      1/1     1            1           2d22h
calico-system     calico-kube-controllers            1/1     1            1           2d22h
calico-system     calico-typha                       1/1     1            1           2d22h
kube-system       coredns                            2/2     2            2           2d22h
kube-system       coredns-autoscaler                 1/1     1            1           2d22h
kube-system       konnectivity-agent                 2/2     2            2           2d22h
kube-system       metrics-server                     2/2     2            2           2d22h
tigera-operator   tigera-operator                    1/1     1            1           2d22h
Enter fullscreen mode Exit fullscreen mode

To deploy the application, click the Sync button on the application card.

Image description

Then, in the panel that appears, select the resources to synchronize, define the synchronization and prune propagation options, and click the Synchronize button.

Image description

Upon completion, the application’s status will change to Synced, and its health status will update to Healthy.

Image description

Clicking on the card will redirect you to the application detail view, where you can see the various resources and their relationships.

Image description

Update the application

Let’s assume we are anticipating an increase in traffic and need to update the number of replicas for our application. We will create a new branch and update the number of replicas from 2 to 5 as shown below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: guestbook-ui
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: guestbook-ui
  template:
    metadata:
      labels:
        app: guestbook-ui
    spec:
      containers:
      - image: gcr.io/heptio-images/ks-guestbook-demo:0.2
        name: guestbook-ui
        ports:
        - containerPort: 80
Enter fullscreen mode Exit fullscreen mode

The next step is to raise a new Pull Request so other team members can review and either approve or reject the changes before they are merged into the production branch.

Image description

Assuming the Pull Request is approved and our code is merged, the desired state (with 5 replicas) will no longer match the live state. Since we set the sync policy of our application to Manual, this discrepancy will highlight that the application is out of sync.

Image description

To update the live state, we simply click the Sync button on the application card, and Argo CD will handle the rest.

Image description

References

Top comments (0)

Read next

samuelarogbonlo_64 profile image

How To Setup Password Hash Synchronization In Microsoft Azure

Samuel Arogbonlo -

marcuskohlberg profile image

Building and Deploying TypeScript Microservices to Kubernetes

Marcus Kohlberg -

imsushant12 profile image

Launching EC2 Instances with AWS CLI and Advanced Features

Sushant Gaurav -

nolunchbreaks_22 profile image

How Hackers Exploit Ansible for Configuration Attacks: A Technical Deep Dive

Osagie Anolu -