DEV Community

Cover image for How to become more productive using Makefile

How to become more productive using Makefile

Ousseynou Diop on June 27, 2020

Originally posted on my blog Introduction You want to become more productive, automate your workflow, there are many tools to help you ...
Collapse
 
tartley profile image
Jonathan Hartley • Edited

I love Makefiles and use them in almost all my projects, but I think the main benefits of them is walking DAGs of dependencies, which the article doesn't really mention.

For example, if your toolchain generates files:

source-file
   |
   tool-a
   |
intermediate-file
   |
   tool-b
   |
final-file

Then you can define some Makefile targets:

# final-file depends upon the state of intermediate-file
final-file: intermediate-file
     tool-b intermediate-file >final-file

# intermediate file depends on source-file
intermediate-file: source-file
    tool-a source-file >intermediate-file

Then the command 'make final-file' will run tool-b to generate the intermediate file, and knows that it first needs to run tool-a to generate the intermediate-file, and (the clever bit) it only runs each step if the output of the step is older than any of its inputs.

This means you eliminate needless typing (from running a tree of pre-requisite steps), and you eliminate needless (and possibly slow) rebuilding of files which don't need it. This is the core value of make, and if you can't take advantage of this, then there isn't much value over a set of script files.

I'd classify my current employer (Canonical) as high on the technically savvy scale, and my team uses a Makefile or two on every project. Even in cases where we can't make use of the above dependency analysis, it's nice for us to standardize on one approach to controlling how to build/test/etc projects.

Collapse
 
xarala221 profile image
Ousseynou Diop

Very interesting, thanks.

Collapse
 
tartley profile image
Jonathan Hartley

I added some missing "dependencies" (the parts after the colon) in my Makefile example.

Collapse
 
simo97 profile image
ADONIS SIMO

Great article, i use to handle some of those task with custom django commands... but it really depends on what the task has to achieve, for example initializing app with default data... but it's a good idea to use makefile also.

But i am not sure if it can do everything, for example can it help you to deploy on remote server via SSH like fabfile ?

Collapse
 
xarala221 profile image
Ousseynou Diop

Thank you, friend.
Yes, you can do that.
I use this file to deploy my app to a docker container or Heroku server.

Collapse
 
simo97 profile image
ADONIS SIMO

yeah i use it also but for VPS deployment and sometime on EC2....

Collapse
 
hammertoe profile image
Matt Hamilton

TABS! Make sure you mention you need to use tabs not spaces to indent the start of a command. That is something that still gets me two decades of using Makefile's later.

Collapse
 
xarala221 profile image
Ousseynou Diop

Good to know.
Thank you.

Collapse
 
gwutama profile image
Galuh Utama • Edited

But you don’t really save time though since the commands are short already. I question the productivity improvements here and would use bash aliases instead.

Make was invented for compiling (C/C++) codebases which involves a lot of long commands for compiler, linker, finding libraries, dependency tracking, etc. And it’s still a pain in the ass to maintain makefiles for those purposes. Nowadays I see less and less makefiles at work and in the opensource community.

Collapse
 
stefanjarina profile image
Štefan Jarina • Edited

Hi, yeah, the example is a little bit too simplistic, however with a bit more involved example, one can start seeing the benefits:

include Makefile.settings

.PHONY: init build clean publish log jenkins slave

# Jenkins settings
export DOCKER_GID ?= 100
export JENKINS_USERNAME ?= admin
export JENKINS_PASSWORD ?= password
export JENKINS_SLAVE_VERSION ?= 2.2
export JENKINS_SLAVE_LABELS ?= DOCKER

# AWS settings
# The role to assume to inject temporary credentials into your Jenkins container
AWS_ROLE ?= `aws configure get role_arn`
# KMS encrypted password - the temporary credentials must possess kms:decrypt permissions for the key used to encrypt the credentials
export KMS_JENKINS_PASSWORD ?=

init:
    ${INFO} "Creating volumes..."
    @ docker volume create --name=jenkins_home

build:
    ${INFO} "Building image..."
    @ docker-compose build --pull
    ${INFO} "Build complete"

jenkins: init
    @ $(if $(and $(AWS_PROFILE),$(KMS_JENKINS_PASSWORD)),$(call assume_role,$(AWS_ROLE)),)
    ${INFO} "Starting Jenkins..."
    ${INFO} "This may take some time..."
    @ docker-compose up -d jenkins
    @ $(call check_service_health,$(RELEASE_ARGS),jenkins)
    ${INFO} "Jenkins is running at http://$(DOCKER_HOST_IP):$(call get_port_mapping,jenkins,8080)..."

publish:
    ${INFO} "Publishing images..."
    @ docker-compose push
    ${INFO} "Publish complete"

slave:
    ${INFO} "Checking Jenkins is healthy..."
    @ $(if $(and $(AWS_PROFILE),$(KMS_JENKINS_PASSWORD)),$(call assume_role,$(AWS_ROLE)),)
    @ $(call check_service_health,$(RELEASE_ARGS),jenkins)
    ${INFO} "Starting $(SLAVE_COUNT) slave(s)..."
    @ docker-compose up -d --scale jenkins-slave=$(SLAVE_COUNT)
    ${INFO} "$(SLAVE_COUNT) slave(s) running"

clean:
    ${INFO} "Stopping services..."
    @ docker-compose down -v || true
    ${INFO} "Services stopped"

destroy: clean
    ${INFO} "Deleting jenkins home volume..."
    @ docker volume rm -f jenkins_home
    ${INFO} "Deletion complete"

log:
    ${INFO} "Streaming Jenkins logs - press CTRL+C to exit..."
    @ docker-compose logs -f jenkins
Enter fullscreen mode Exit fullscreen mode

what I see is that makefiles are less used with some languages, but are still quite heavily used with others.
Though I agree that I also haven't seen complex makefile in a while, but that is probably because more and more languages have their own tooling.

  • JS: npm scripts, gulp, grunt, etc.
  • C#: Powershell (Invoke!), maybe Cake
  • F#: FAKE
  • Rust: cargo custom tasks
  • JVM languages had their own for a long time also: sbt (Scala), gradle (a lot of JVM langs)
  • Python: Fabric, but here I've seen a lot of makefiles :-)
  • Ruby: Rake
  • Elixir: mix with custom tasks
  • Go: maybe Task, but here I've also seen quite a lot of makefiles

Can't tell for the rest, these are the ones I am familiar with.

Collapse
 
stefanjarina profile image
Štefan Jarina • Edited

Small updates:

PYTHON:

Fabric was used for this in it's 1.x version.

Since version 2.0 Fabric was split essentially into 2 packages with separate functionalities:

  • Fabric 2.0 - now used only for remote shell/commands using ssh
  • Invoke! - task execution tool & library extracted from Fabric 1.x into it's own library

GENERIC:
There is now another tool, that is generic and can be used for many languages, it is even cross-platform

Haven't used it though, in most cases I go with language native solution and if there is none, I usually either:

  • still write Makefile if only linux/mac/docker/kubernetes is needed
  • use Powershell 7 if I need it to run on MAC/Linux/Windows
Collapse
 
mintypt profile image
mintyPT

What do you have inside your Makefile.settings?

Thread Thread
 
stefanjarina profile image
Štefan Jarina

Hi,

The code in this file is probably a bit more involved and harder to decipher, but it is just parsing functions really.
To get useful data from docker and/or env variables

Makefile.settings:

YELLOW := "\e[1;33m"
NC := "\e[0m"
INFO := @bash -c 'printf $(YELLOW); echo "=> $$1"; printf $(NC)' MESSAGE
ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))

SHELL = bash

# Slave arguments
ifeq ($(firstword $(MAKECMDGOALS)),$(filter $(firstword $(MAKECMDGOALS)),slave))
  SLAVE_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
  SLAVE_COUNT = $(if $(SLAVE_ARGS),$(firstword $(SLAVE_ARGS)),1)
endif

# Docker host settings
DOCKER_HOST_IP := $(shell echo $$DOCKER_HOST | awk -F/ '{printf $$3}' | awk -F: '{printf $$1}')
DOCKER_HOST_IP := $(if $(DOCKER_HOST_IP),$(DOCKER_HOST_IP),localhost)

# Image and Repository Tag introspection functions
# Syntax: $(call get_image_id,<docker-compose-environment>,<service-name>)
# Syntax: $(call get_repo_tags,<docker-compose-environment>,<service-name>,<fully-qualified-image-name>)
get_container_id = $$(docker-compose $(1) ps -q $(2))
get_image_id = $$(echo $(call get_container_id,$(1),$(2)) | xargs -I ARGS docker inspect -f '{{ .Image }}' ARGS)
get_container_state = $$(echo $(call get_container_id,$(1),$(2)) | xargs -I ID docker inspect -f '$(3)' ID)
filter_repo_tags = $(if $(findstring docker.io,$(1)),$(subst docker.io/,,$(1))[^[:space:]|\$$]*,$(1)[^[:space:]|\$$]*)
get_repo_tags = $$(echo $(call get_image_id,$(1),$(2)) | xargs -I ID docker inspect -f '{{range .RepoTags}}{{.}} {{end}}' ID | grep -oh "$(call filter_repo_tags,$(3))" | xargs)

# Port introspection functions
# Syntax: $(call get_port_mapping,<service-name>,<internal-port>)
get_raw_port_mapping = $$(docker-compose ps -q $(1) | xargs -I ID docker port ID $(2))
get_port_mapping = $$(echo $$(IFS=':' read -r -a array <<< "$(call get_raw_port_mapping,$(1),$(2))" && echo "$${array[1]}"))

# Service health functions
# Syntax: $(call check_service_health,<docker-compose-environment>,<service-name>)
get_service_health = $$(echo $(call get_container_state,$(1),$(2),{{if .State.Running}}{{ .State.Health.Status }}{{end}}))
check_service_health = { \
  until [[ $(call get_service_health,$(1),$(2)) != starting ]]; \
    do sleep 1; \
  done; \
  if [[ $(call get_service_health,$(1),$(2)) != healthy ]]; \
    then echo $(2) failed health check; exit 1; \
  fi; \
}

# AWS assume role settings
# Attempts to assume IAM role using STS
# Syntax: $(call assume_role,<role-arn>)
get_assume_session = aws sts assume-role --role-arn=$(1) --role-session-name=admin
get_assume_credential = jq --null-input '$(1)' | jq .Credentials.$(2) -r
define assume_role
    $(eval AWS_SESSION = $(shell $(call get_assume_session,$(1))))
    $(eval export AWS_ACCESS_KEY_ID = $(shell $(call get_assume_credential,$(AWS_SESSION),AccessKeyId)))
    $(eval export AWS_SECRET_ACCESS_KEY = $(shell $(call get_assume_credential,$(AWS_SESSION),SecretAccessKey)))
    $(eval export AWS_SESSION_TOKEN = $(shell $(call get_assume_credential,$(AWS_SESSION),SessionToken)))
endef
Enter fullscreen mode Exit fullscreen mode
Thread Thread
 
mintypt profile image
mintyPT

Just the first 3 lines are worth it

Collapse
 
xarala221 profile image
Ousseynou Diop

Great content, Thank you.

Collapse
 
xarala221 profile image
Ousseynou Diop

Thank you,
I find this very interesting.

Collapse
 
patarapolw profile image
Pacharapol Withayasakpunt

I have thought about this a while ago, and Make seems to be both quite standard, and preinstalled (if not using Windows).

Collapse
 
ryands17 profile image
Ryan Dsouza

Great article! I'm trying it for a workflow for Node apps :)

Collapse
 
xarala221 profile image
Ousseynou Diop

Great.
Thank you.

Collapse
 
jingxue profile image
Jing Xue

It's an interesting approach, but why use make for a bash script's job?