Resolving dpkg Lock Contention in Ubuntu Deployments

Are you seeing the following error causing a failure in your deployment process?

module.nginx.digitalocean_droplet.nginx (remote-exec): E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 7264 (apt-get)
module.nginx.digitalocean_droplet.nginx (remote-exec): N: Be aware that removing the lock file is not a solution and may break your system.
module.nginx.digitalocean_droplet.nginx (remote-exec): E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
╷
│ Error: remote-exec provisioner error
│ 
│   with module.nginx.digitalocean_droplet.nginx,
│   on nginx/main.tf line 55, in resource "digitalocean_droplet" "nginx":
│   55:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_1926178835.sh": Process exited with status
│ 100
╵

and different variations of it occasionally popping up

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 7264 (apt-get)

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 9983 (unattended-upgr)

This is a race condition, the issue occurred because two system processes were running simultaneously: an automated package update service (unattended-upgrades) and an initialization process for cloud-based environments (cloud-init script) which includes a apt-get command. unattended-upgrades runs as soon as the container starts up and it is responsible for automatically updating software packages within aptitude, the underlying package management system. When unattended-upgrades runs, it acquires a lock on the dpkg frontend to ensure that no other processes modify packages until all updates are complete. This lock prevented other processes from installing or removing software until the update process was complete.

Investigation

Check /var/log/unattended-upgrades for the frequency of unattended-upgrades, it should be running periodically once enabled but have seen it ran at irregular frequencies unsure why. It is also a good idea to check syslogs, dpkg logs and check the individual config for apt-daily timers, and cron config. You can run the following to see the scheduled upcoming timers and last ran, look for apt-daily-upgrade.timer and apt-daily.timer Unit in the output table:

systemctl list-timers

Resolving dpkg Lock Contention in Ubuntu Deployments

We should look to disable unattended-upgrades during the cloud-init process and re-enabling it afterwards.

1. Increase DPKG lock timeout in script
   

   sudo apt-get -o DPkg::lock::Timeout=600 [insert-your-command]
   

   unattended-upgrades can take up to 10 minutes (or more) to complete,
   ensure you set an appropriate timeout to allow sufficient time for the
   upgrades. See Waiting for apt locks without the hacky bash scripts for more.

2. Disable unattended-upgrades during the initialisation process and re-enabling it afterwards.

Glossary

What is dpkg?

dpkg (Debian Package Manager) is the low-level package management tool used in Debian-based Linux distributions (like Ubuntu). It handles the installation, removal, and management of .deb packages. However, dpkg does not handle dependencies automatically—if a package requires other software to function, dpkg won’t fetch them automatically.

What is aptitude?

aptitude is a higher-level package manager that works as a user-friendly interface for dpkg. It provides advanced dependency resolution and additional features. aptitude is similar to apt (apt-get), but it offers a more interactive interface, allowing users to browse and manage packages easily.

Key features of aptitude:

• Handles dependencies automatically (unlike dpkg)
• Provides a text-based interactive UI
• Can be used as a command-line tool like apt-get