Why You Need To Stop Using Git-Hooks

Originally published at blog.mphomphego.co.za on Oct 03, 2019

---

The Story

TL; DR

Native Git-hooks are very useful but have some shortcomings around installation, maintainability, and re-usability. But there's a great alternative...

TS; RE

Git-Hooks is a great way to execute custom actions triggered by various git events on the local machine, to identify bugs/issues before committing and pushing our work for review. Ideally, these hooks run on every commit to automatically point out issues/bugs in
the code, for example missing semicolons, trailing whitespace, docstrings
violations, Python module sorting, etc.

For the longest time, I have been using git-hooks mainly for pre-commit and prepare-commit-msg. You can find my deprecated git-hooks here

• pre-commit: A script is written in bash/shell that gets called by a git commit, and then execute linting depending on the file being staged. Exiting this hook with a non-zero status will abort the commit, which makes it extremely useful for last-minute quality checks.

• prepare-commit-msg: A script called by git commit that automatically prepends the commit message with the Jira ticket link based on the branch name.

Githooks are great for pointing out any issues before committing any form of work and submitting for review, this allows the reviewer to direct their focus on the architecture of a change than wasting time on trivial style nitpicks.

IMHO, they have some shortcomings around installation, maintainability, and re-usability, let me try to explain.

• Installation:

  Git hooks normally live inside the .git/hooks folders of local repositories. They can contain one executable per trigger event, such as pre-commit or prepare-commit-msg, which is run automatically by various git commands. It sounds magical, so you'd think what's the problem with this?

  The .git folder is excluded from version control (i.e., You won't be able to track its life-cycle), it only lives on your machine. To get a hook in place, you either have to copy an executable file with the trigger’s name into the .git/hooks folder or symlink one into it. It becomes cumbersome sharing hooks across various libraries or repositories and manually making changes to suit each repository.

• Maintainability:

  The next problem is that you can have one of these files per event. You can then write a long Bash script (see my pre-commit, for instance) to execute multiple actions for one trigger or split the actions to several individual files, and then create the main entry point to run the additional functionalities in the divided files. It is fully feasible, but it is tedious to maintain

• Re-usability:

  The last point concerns the re-usability, both for the setup and the hooks. Doing so manually, you need to get the hook script and copy it or symlink it in each repository you want to use. You would also like to see your setup move with you when switching workstations/systems, without them going through the above-mentioned setup for each project.

  You may also want to have these hooks executed for all same type of projects, for example, of updating Python dependencies or executing flake8 checks before each commit, etc. You would have to copy and create your hook for each project automatically, with an out-of-the-box setup, which over time became painful and tedious for me to use which led me to use my hooks less.

With all that said, do not get me wrong, I love the concept of Git hooks, and they can be very useful in several ways. However, portability and maintenance is an issue.

This post will detail why you should probably consider ditching your native git-hooks (Assuming you currently use them), reasons stated above. But nature abhors a vacuum, so we would need to replace native git-hooks with an alternative. Enters pre-commit.

The How

According to the website:

Pre-commit is a framework for managing and maintaining multi-language pre-commit hooks, which is far more flexible and feature-rich than native git-hooks.

pre-commit makes it easier for the developer to specify a list of hooks they want and pre-commit manages the installation and execution of any hook written in any language before every commit, in a completely isolated environment and does not require root access.

Why Do You Need It

• Gently enforced best consistent practices between all team members or contributors.
• No need to fight with the reviewer about code formatting (PEP-8 violations), pre-commit can enforce it using the same coding standards across your team.
• Improved code quality == More green pipelines on your CI.
• Human time is expensive i.e. less time spent reviewing and quoting (PEP-8) coding standards.

The Walkthrough

Installation

Setting up pre-commit is straight forward, Run (alternatively read the docs.):

pip install pre-commit

Configuration

Once the package is installed you would need to configure pre-commit, so that it knows what you want - right!

Follow these simple steps and you should be good to go:

• cd to any repository you want to install pre-commit hooks.
• create a file named .pre-commit-config.yaml (alternatively you can generate a template by running: pre-commit sample-config > .pre-commit-config.yaml
• If you opted for the option to create .pre-commit-config.yaml manually then copy and paste the following code to it.

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.0.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files
-   repo: https://github.com/psf/black
    rev: 19.3b0
    hooks:
    -   id: black

Note: pre-commit (hooks) are not restricted to one programming language.

You can add all kinds of hooks - linting, automatic formatting, safety checks, tests, custom scripts the list is endless alternatively you can use the official hooks.

Hooks Installation

To have pre-commit run every time you commit, run;

pre-commit install

Now every time you execute git commit, pre-commit will run the hooks defined above.

Note: git commit -n bypasses/disables pre-commit hooks, This is not advised consider benefits gained vs time.

Alternatively, if one would like to run pre-commit manually from time to time.
Running pre-commit run --all-files will run all hooks against current changes in your repository or,

If you wish to execute an individual hook use pre-commit run <hook_id>.
Example:

• pre-commit run black, this will execute black on the current working directory. Black is the uncompromising Python code formatter.

If like me you rather prefer to have pre-commit hooks installed every time your clone or create a new repository. Run;

• mkdir -p ~/.git-template
• git config --global init.templateDir ~/.git-template : This tells git to copy everything in ~/.git-templates to your project .git/ directory when you run git init
• pre-commit init-templatedir ~/.git-template: This will install the current hooks into the template directory and ensure that every time you clone or create a repository pre-commit hooks will be installed.

Custom Hooks

It's easy to create your custom hooks and share them with the community.
Since I deprecated my native git-hooks, I have been constantly updating (WIP) my custom hooks which you are more than welcome to check out and contributions are welcome.

Alternatively see my .pre-commit-config.yaml

##########################################################################################
#                                                                                        #
#                       Pre-commit configuration file                                    #
#                                                                                        #
##########################################################################################
---
default_language_version:
    python: python3

repos:
    ####################################### Various Checks ###############################
    # Various checks
    -   repo: https://github.com/pre-commit/pre-commit-hooks
        rev: v2.3.0
        hooks:
        -   id: check-ast
            name: check-ast
            description: Simply check whether files parse as valid python.
        -   id: check-builtin-literals
            name: check-builtin-literals
            description: Require literal syntax when initializing empty,
                         or zero Python builtin types.
        -   id: check-docstring-first
            name: check-docstring-first
            description: Checks for a common error of placing code before the docstring.
        -   id: check-added-large-files
            name: check-added-large-files
            description: Prevent giant files from being committed.
        -   id: check-merge-conflict
            name: check-merge-conflict
            description: Check for files that contain merge conflict strings.
        -   id: check-symlinks
            name: check-symlinks
            description: Checks for symlinks which do not point to anything.
        -   id: check-yaml
            name: check-yaml
            description: Attempts to load all yaml files to verify syntax.
        -   id: check-toml
            name: check-toml
            description: Attempts to load all TOML files to verify syntax.
        -   id: debug-statements
            name: debug-statements
            description: Check for debugger imports and py37+ breakpoint() calls in python source.
        -   id: detect-private-key
            name: detect-private-key
            description: Checks for the existence of private keys.
        -   id: end-of-file-fixer
            name: end-of-file-fixer
            description: Makes sure files end in a newline and only a newline.
        -   id: trailing-whitespace
            name: trailing-whitespace
            description: Trims trailing whitespace
        -   id: requirements-txt-fixer
            name: requirements-txt-fixer
            description: Sorts entries in requirements.txt

    -   repo: local
        hooks:
        # Python minor syntax related checks
        # https://github.com/Pike/pygrep
        -   id: python-check-mock-methods
            name: python-check-mock-methods
            description: Prevent common mistakes of `assert mck.not_called()`, `assert mck.called_once_with(...)`
                        and `mck.assert_called`.
            language: pygrep
            entry: >
                (?x)(
                    assert .*\.(
                        not_called|
                        called_
                    )|
                    \.assert_(
                        any_call|
                        called|
                        called_once|
                        called_once_with|
                        called_with|
                        has_calls|
                        not_called
                    )($|[^(\w])
                )
            types: [python]

        -   id: python-no-eval
            name: python-no-eval
            description: 'A quick check for the `eval()` built-in function'
            entry: '\beval\('
            language: pygrep
            types: [python]

        -   id: python-no-log-warn
            name: python-no-log-warn
            description: 'A quick check for the deprecated `.warn()` method of python loggers'
            entry: '(?<!warnings)\.warn\('
            language: pygrep
            types: [python]

        -   id: python-use-type-annotations
            name: python-use-type-annotations
            description: 'Enforce that python3.6+ type annotations are used instead of type comments'
            entry: '# type(?!: *ignore *($|#))'
            language: pygrep
            types: [python]

        # Python security check
        # https://bandit.readthedocs.io/en/latest/
        -   id: bandit
            name: bandit
            description: Find common security issues in your Python code using bandit
            entry: bandit
            args: [
                '-ll',
                '--ini', 'setup.cfg',
                '--recursive',
            ]
            language: python
            types: [python]

        # Vulture
        # https://github.com/jendrikseipp/vulture
        -   id: vulture
            name: vulture
            description: Find dead Python code
            entry: vulture
            args: [
                "--min-confidence", "90",
                "--exclude", "*env*", "docs/",
                ".",
            ]
            language: system
            types: [python]

        # Jira Ticket Link Prepender
        -   repo: https://github.com/mmphego/prepend-jira-ticket-link
            rev: master
            hooks:
            -   id: prepend-jira-link
                description: Appends ticket number and link below commit message based on the branch name

    ####################################### Linters ######################################
    -   repo: local
        hooks:
        # Flake8 Linter
        # https://flake8.pycqa.org/en/latest/
        -   id: flake8
            name: flake8
            description: Python style guide enforcement
            entry: flake8
            args: ["--config=setup.cfg"]
            additional_dependencies: [
                flake8-2020, # flake8 plugin which checks for misuse of `sys.version` or `sys.version_info`
                flake8-blind-except, # A flake8 extension that checks for blind except: statements
                flake8-bugbear, # A plugin for flake8 finding likely bugs and design problems in your program.
                                # Contains warnings that don't belong in pyflakes and pycodestyle.
                flake8-builtins, # Check for python builtins being used as variables or parameters.
                flake8-comprehensions, # It helps you write a better list/set/dict comprehensions.
                flake8-copyright, # Adds copyright checks to flake8
                flake8-deprecated, # Warns about deprecated method calls.
                dlint, # Dlint is a tool for encouraging best coding practices and helping ensure we're writing secure Python code.
                flake8-docstrings, # Extension for flake8 which uses pydocstyle to check docstrings
                # flake8-eradicate, # Flake8 plugin to find commented out code
                flake8-license,
                pandas-vet, # A Flake8 plugin that provides opinionated linting for pandas code
                flake8-pytest, # pytest assert checker plugin for flake8
                flake8-variables-names, # flake8 extension that helps to make more readable variables names
                flake8-tabs, # Tab (or Spaces) indentation style checker for flake8
                pep8-naming, # Check PEP-8 naming conventions, plugin for flake8
            ]
            language: python
            types: [python]

        # MyPy Linter
        # https://mypy.readthedocs.io/en/latest/
        -   id: mypy
            name: mypy
            description: Optional static typing for Python 3 and 2 (PEP 484)
            entry: mypy
            args: ["--config-file", "setup.cfg"]
            language: python
            types: [python]

        # PyDocstyle
        # https://github.com/PyCQA/pydocstyle
        -   id: pydocstyle
            name: pydocstyle
            description: pydocstyle is a static analysis tool for checking compliance with Python docstring conventions.
            entry: pydocstyle
            args: ["--config=setup.cfg", "--count"]
            language: python
            types: [python]

        # YAML Linter
        -   id: yamllint
            name: yamllint
            description: A linter for YAML files.
            # https://yamllint.readthedocs.io/en/stable/configuration.html#custom-configuration-without-a-config-file
            entry: yamllint
            args: [
                '--format', 'parsable',
                '--strict',
                '-d', "{
                    extends: relaxed,
                    rules: {
                        hyphens: {max-spaces-after: 4},
                        indentation: {spaces: consistent, indent-sequences: whatever,},
                        key-duplicates: {},
                        line-length: {max: 90}},
                    }"
            ]
            language: system
            types: [python]
            additional_dependencies: [yamllint]

        # Shell Linter
        # NOTE: Hook requires shellcheck [installed].

        -   id: shellcheck
            name: shellcheck (local)
            language: script
            entry: scripts/shellcheck.sh
            types: [shell]
            args: [-e, SC1091]
            additional_dependencies: [shellcheck]

        # Prose (speech or writing) Linter
        -   id: proselint
            name: proselint
            description: An English prose (speech or writing) linter
            entry: proselint
            language: system
            types: [ rst, markdown ]
            additional_dependencies: [proselint]


    ################################### Code Format ######################################
    -   repo: local
        hooks:
        # pyupgrade
        # Upgrade Python syntax
        -   id: pyupgrade
            name: pyupgrade
            description: Automatically upgrade syntax for newer versions of the language.
            entry: pyupgrade
            args: ['--py3-plus']
            language: python
            types: [python]
            additional_dependencies: [pyupgrade]

        # Sort imports
        # https://github.com/timothycrosley/isort
        -   id: isort
            name: isort
            description: Library to sort imports.
            entry: isort
            args: [
                "--recursive",
                "--settings-path", "setup.cfg"
            ]
            language: python
            types: [python]

        # Manifest.in checker
        # https://github.com/mgedmin/check-manifest
        -   id: check-manifest
            name: check-manifest
            description: Check the completeness of MANIFEST.in for Python packages.
            entry: check-manifest
            language: python
            types: [python]

        # pycodestyle code format
        # https://pypi.python.org/pypi/autopep8/
        -   id: autopep8
            name: autopep8
            description: A tool that automatically formats Python code to conform to the PEP 8 style guide.
            entry: autopep8
            args: [
                '--in-place',
                '--aggressive', '--aggressive',
                '--global-config', 'setup.cfg',
            ]
            language: python
            types: [python]

        # Python code format
        # https://github.com/psf/black/
        -   id: black
            name: black
            description: The uncompromising Python code formatter
            entry: black
            args: [
                '--line-length', '90',
                '--target-version', 'py36'
            ]
            language: python
            types: [python]

################################### Test Runner ##########################################
    -   repo: local
        hooks:
        -   id: tests
            name: run tests
            description: Run pytest
            entry: pytest -sv
            language: system
            types: [python]
            stages: [push]

Assuming both pre-commit and pre-push are installed (pre-commit install && pre-commit install -t pre-push), it will:

During Commit Stage

• Check whether files parse as valid Python.
• Require literal syntax when initializing empty, or zero Python builtin types.
• Checks for a common error of placing code before the docstring.
• Prevent giant files from being committed.
• Check for files that contain merge conflict strings.
• Checks for dead symlinks.
• Check for yaml syntax.
• Check for TOML syntax.
• Check for debugger imports and py37+ breakpoint() calls in python source.
• Checks for the existence of private keys.
• Makes sure files end in a newline and only a newline.
• Trims trailing whitespace.
• Sorts entries in requirements.txt.
• Prevent common mistakes of assert mck.not_called(), assert mck.called_once_with(...) and mck.assert_called.mck.called_once_with(...)`
• Check if eval() built-in function is used.
• Check if deprecated .warn() method of python loggers.
• Enforce that python3.6+ type annotations are used instead of type comments.
• Find common security issues in your Python code using bandit.
• Append JIRA ticket number and link below commit message, if branch name contains ticket number.
• Python style guide enforcement using flake8.
• Optional static typing for Python 3 and 2 (PEP 484)
• Static analysis tool for checking compliance with Python docstring conventions.
• A linter for YAML files.
• A static anaylsis tool that automatically finds bugs in your shell scripts.
• An English prose (speech or writing) linter.
• Automatically upgrade syntax for newer versions of the language.
• Sort imports.
• Check the completeness of MANIFEST.in for Python packages.
• Automatically format Python code to conform to the PEP 8 style guide. (Optional)
• The uncompromising Python code formatter.

During Push Stage

• Runs also pytest with verbose and no-capture flag.

You should see something like this in the command line while committing/pushing:

bash
check-ast............................................(no files to check)Skipped
check-builtin-literals...............................(no files to check)Skipped
check-docstring-first................................(no files to check)Skipped
check-added-large-files..................................................Passed
check-merge-conflict.....................................................Passed
check-symlinks.......................................(no files to check)Skipped
check-yaml...............................................................Passed
check-toml...............................................................Passed
debug-statements.....................................(no files to check)Skipped
detect-private-key.......................................................Passed
end-of-file-fixer........................................................Passed
trailing-whitespace......................................................Passed
requirements-txt-fixer...............................(no files to check)Skipped
python-check-mock-methods............................(no files to check)Skipped
python-no-eval.......................................(no files to check)Skipped
python-no-log-warn...................................(no files to check)Skipped
python-use-type-annotations..........................(no files to check)Skipped
bandit...............................................(no files to check)Skipped
flake8...............................................(no files to check)Skipped
mypy.................................................(no files to check)Skipped
pydocstyle...........................................(no files to check)Skipped
yamllint.............................................(no files to check)Skipped
shellcheck (local).......................................................Passed
proselint............................................(no files to check)Skipped
pyupgrade............................................(no files to check)Skipped
isort................................................(no files to check)Skipped
check-manifest.......................................(no files to check)Skipped
autopep8.............................................(no files to check)Skipped
black................................................(no files to check)Skipped


Conclusion

If you’ve followed along this far, you should be able to see the different ways and benefits that pre-commit hooks can help automate some of your tasks. They can help you deploy your code, or help you maintain quality standards by rejecting non-conformant changes or commit messages. Go here, further reading on how to use pre-commit in your CI environment.

Reference

• [Podcast] Keep Your Code Clean Using pre-commit with Anthony Sottile - Episode 178
• [Manual] pre-commit
• [Blog] Githooks: auto-install hooks
• [Blog] Pre-commit is awesome
• [Blog] Automate Python workflow using pre-commits: black and flake8
• [Tutorial] How To Use Git Hooks To Automate Development and Deployment Tasks
• [Blog] Automatically check your Python code for errors before committing.