Updating Azure DevOps Pipelines for Terraform Post SSH-RSA Deprecation

Microsoft's recent announcement regarding the deprecation of SSH-RSA for connecting to Azure Repos in Azure DevOps represents a significant shift towards more secure encryption methods. This change, anticipated with the version post-2022.3 of Azure DevOps Server around the end of 2024, aligns with a broader trend of enhancing security protocols across the tech landscape. However, this transition poses challenges, particularly for organizations leveraging a centralized approach for Terraform module management. These modules, typically stored in dedicated repositories and referenced via SSH in other repositories, are essential for provisioning infrastructure for services or applications. The deprecation of SSH-RSA necessitates an adaptation of these practices.

Centralized module strategy

Adopting a centralized module strategy for Terraform comes with significant advantages, including:

• Single Source of Truth: Centralizing Terraform modules in one repository ensures consistency, minimizes duplication, and simplifies infrastructure code management. Updates or bug fixes to modules can be easily propagated across all dependent services, eliminating the need to update each service's repository individually.
• Ease of Updates: Enhancements, bug fixes, or security patches to a module can be seamlessly rolled out across all implementations by simply updating the module reference in the dependent Terraform configurations.

Terraform Modules

A common practice is to store Terraform module configurations in version control repositories, such as GitHub or Bitbucket. Terraform accesses these modules using Git, cloning the content over HTTPS or SSH. For private repositories, this process necessitates proper authentication:

• HTTPS: Requires embedding a Personal Access Token (PAT) within the URL, which poses security risks if exposed in publicly accessible or shared codebases.

module "key_vault_key" {
   source = "git::https://<PAT>@dev.azure.com/GTRekter/Training/_git/Terraform-Modules//keyvault-key?ref=main"
}

• SSH: Preferred in automated systems due to its non-interactive nature, allowing seamless access to private repositories.

module "key_vault_key" {
    source = "git::[e-mail]:v3/GTRekter/Training/_git/Terraform-Modules//keyvault-key?ref=main"
}

After executing terraform init, Terraform clones the referenced external repository into the .terraform/modules directory. This local copy is then utilized in the plan and apply stages, ensuring that the Terraform configurations are self-contained and reproducible.

.
├───.terraform
│   ├───modules
│   │   └───key_vault_key
│   │       ├───.pipelines
│   │       ├───aks-cluster
│   │       ├───aks-node-pool
│   │       └───keyvault-key
│   └───providers
│       └───registry.terraform.io
├───pipelines
└───main.tf

SSH and issues with the current Microsoft update

With the deprecation of SSH-RSA, developers have started encountering unexpected access issues with their repositories overnight.

The problem arises because the agents execute the clone operation over HTTPS, not relying on SSH-RSA keys. The authentication mechanism at play here is OAuth, which is embedded in the pipeline's configuration as an http.extraheader. However, when SSH is specified as the mechanism for downloading the source code of the Terraform modules, the process fails.

Migrate from SSH-RSA to RSA-SHA2–512

To move to the new ciphers you will first need to generate public/private key pairs for your client. To do so execute the ssh-keygen command with the option -t set to rsa-sha2–512 to specify the type of key to generate. Once completed you will have a *.pub file (public key), and a file without an extension (private key).

$ ssh-keygen -t rsa-sha2-512
Generating public/private rsa-sha2-512 key pair.
Enter file in which to save the key (~/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~\.ssh\id_rsa
Your public key has been saved in ~\.ssh\id_rsa.pub
The key fingerprint is:
SHA256:zWr3tUHtDLjICoxb6+GjacGmHpYgdVcV9SGVEu2htos gtrekter@Sample
The key's randomart image is:
+---[RSA 3072]----+
|        ..oo++o. |
|       .    .o+. |
|  . . .      +.. |
| . . .   o  o... |
|o  .    S o...o .|
|.. .+o   o ..o + |
|  +o..= o +...o o|
| ....=.= oE... o |
| ...+o+..   . .  |
+----[SHA256]-----+

Next, integrate this key with Azure DevOps by setting up a service account associated with the new key, which the agent will use for downloading Terraform modules. Here's how to set it up:

• Navigate to Organization Settings in Azure DevOps.

• Under Users, select Add User.

• Enter the user's email, assign them to the project, and add them to the Project Contributor group for necessary permissions.

Note: The new account requires a mailbox since SSH public keys can only be generated via the Azure DevOps UI due to security protocols.

• Once the account is set up log in with the new user credentials and click the icon next to the avatar and select SSH public keys.

• Click + New Key, paste the public key into the Public Key Data field, and click Add.

With your RSA-SHA2–512 SSH Key ready for Azure authentication, it's essential to set up another crucial component for SSH configuration: the known_hosts file. This file records the public keys for the servers you connect to via SSH, pairing each server's public key with its URL and ensure the integrity of the SSH connection by verifying the identity of the servers you're connecting to, and protecting against Man-in-the-Middle (MitM) attacks. You can generate this file via the following command:

$ ssh-keyscan ssh.dev.azure.com >> ~/.ssh/known_hosts
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10

(Optional) Add the Keys to Azure Key Vault

For those utilizing Azure, it's a smart move to secure sensitive data like SSH keys in the Azure Key Vault. This not only centralizes your secrets management but also bolsters your project's security posture. Once you created or selected the KeyVault that will store the private SSH key and the known_hosts file, add a new secret for each one of them. For this guide, we'll name them ado-ssh-key for the SSH private key and ado-ssh-host-key for the known_hosts file.

Note: Since the SSH key is multiline, ensure you replace new lines with \n to prevent formatting issues that could corrupt the key.

To enable your Azure DevOps agent to access these secrets at runtime, a service connection to Azure is necessary. To do so:

• In your Azure DevOps project, navigate to Project settings, select Service connections and then click New Service connection.

Choose Azure Resource Manager and follow the prompts to select your subscription.

Behind the scenes Azure DevOps will automatically create and configure a service principal in Azure.

With the Key Vault and service connection ready, the final step involves updating your pipeline to fetch the SSH keys before initiating tasks like terraform init. Insert the following task at the appropriate point in your pipeline:

- task: AzureKeyVault@1
  displayName: 'Get Azure DevOps SSH key and Host Key from KeyVault'
  inputs:
    azureSubscription: VSES Training
    KeyVaultName: kv-training-ssh-dev-krc
    SecretsFilter: 'ado-ssh-key, ado-ssh-host-key'

Setting up the agent SSH key

With the new SSH keys generated using RSA-SHA2–512 SSH, the public key linked to an Azure DevOps user, it's time to integrate these components into your pipelines. If you've opted not to use Azure Key Vault, Azure DevOps variable groups offer an alternative for storing secret values, including SSH keys and known_hosts files. Remember to reference these in your pipeline to ensure they're accessible when needed.
Before executing tasks like terraform init, the agent's SSH environment must be configured with the new keys:

• Create the SSH Directory and set up a .ssh directory with the correct permissions:

echo "Create SSH directory and set permissions"
mkdir -p ~/.ssh/
chmod 700 ~/.ssh/

• Generate the private key file, naming it uniquely (e.g., ado_id_rsa) to avoid conflicts with keys from other services:

echo "Set SSH keys"
echo "$(ado-ssh-key)" > ~/.ssh/ado_id_rsa
chmod 600 ~/.ssh/ado_id_rsa

• Append the known_hosts content coming from Azure KeyVault or Azure DevOps variable group to ensure the agent recognizes trusted hosts:

echo "Set known_hosts entries"
echo "$(ado-ssh-host-key)" >> ~/.ssh/known_hosts

• Create an SSH Config File to instruct SSH to use the specific key (in our case called ado_id_rsa) when the target is a specific host.

echo "Create the SSH config file"
echo -e "Host ssh.dev.azure.com\n  HostName ssh.dev.azure.com\n  User git\n  IdentityFile ~/.ssh/ado_id_rsa\n  IdentitiesOnly yes\n\nHost bitbucket.org\n  HostName bitbucket.org\n  User git\n  IdentityFile ~/.ssh/bitbucket_id_rsa\n  IdentitiesOnly yes" > ~/.ssh/config
chmod 600 ~/.ssh/config

• Verify that the SSH setup is functioning correctly:

ssh -Tv [e-mail] || true
echo "SSH connection test completed"

The final task will be the following:

- task: CmdLine@2
  displayName: Setup SSH key
  inputs:
    script: |
      echo "Create SSH directory and set permissions"
      mkdir -p ~/.ssh/
      chmod 700 ~/.ssh/

      echo "Generate and set SSH keys"
      echo "$(ado-ssh-key)" > ~/.ssh/ado_id_rsa
      chmod 600 ~/.ssh/ado_id_rsa

      echo "Update known_hosts entries"
      echo "$(ado-ssh-host-key)" >> ~/.ssh/known_hosts

      echo "Create the SSH config file"
      echo -e "Host ssh.dev.azure.com\n  HostName ssh.dev.azure.com\n  User git\n  IdentityFile ~/.ssh/ado_id_rsa\n  IdentitiesOnly yes" > ~/.ssh/config
      chmod 600 ~/.ssh/config

      ssh -Tv [e-mail] || true
      echo "SSH connection test completed"

With these configurations in place, your Azure DevOps agent is now equipped to securely connect and deploy your Terraform infrastructure using the updated SSH keys.

Reference

• Microsoft SSH-RSA deprecation announcement: https://devblogs.microsoft.com/devops/ssh-rsa-deprecation/