How to use testament in Nim?

Introduction

Testing is a tedious work but a worth-while work. You should make sure your project is robust before it is released.

In Nim language, there are many ways to test your program.

For simplicity, you can combine when isMainModuleand doAssert to test your program.

Programs below produce no error:

proc hello*(num: int): int =
  result = num + 4

when isMainModule:
  doAssert hello(1) == 5
  doAssert hello(2) == 6
  doAssert hello(3) != 6

Or you can use unittest module.

Testament

Testament which is different from unittest is integrated test. It is shipped with Nim distribution as test tools.

How to use

If you want to use testament, make sure your project have a tests directory. For example:

- example
  - src
    - main.nim
  - tests
    - test1.nim
    - test2.nim
    - dir
      - test1.nim
      - test2.nim

Programs in src directory:

# main.nim
proc hello*(num: int): int =
  result = num + 4

Notes that your tests file should start with character 't'.

Single File

It is simple to test single Nim File. Change your current working directory to example. You can omit the name of tests. Just run testament r test1.nim or testament r dir/test1.nim. Of course, testament r tests/test1.nim and testament r tests/dir/test1.nim are right too.

In test1.nim, you can write tests like this:

import ../src/main


doAssert hello(1) == 5

block:
  doAssert hello(3) == 7


doAssertRaises(ValueError):
  raise newException(ValueError, "specific errors")

May produce results like this:

D:\cookbook\example>testament r test1.nim
PASS: tests/test1.nim C                                            ( 2.65 sec)

Multiple Files

testament all will only test something in the directory. Namely, tests/*/ta.nim will be tested. However tests/ta.nim won't be tested.

If you want to test all files in tests directory, you can use command testament cat / or testament cat ..

May produce results like this:

D:\cookbook\example>testament cat /
PASS: tests/dir/test1.nim C                                        ( 0.83 sec)
PASS: tests/test1.nim C                                            ( 0.84 sec)
PASS: tests/dir/test2.nim C                                        ( 0.85 sec)
PASS: tests/test2.nim C                                            ( 0.88 sec)

D:\cookbook\example>testament cat .
PASS: tests/dir/test1.nim C                                        ( 0.23 sec)
PASS: tests/dir/test2.nim C                                        ( 0.44 sec)
PASS: tests/test1.nim C                                            ( 0.44 sec)
PASS: tests/test2.nim C                                            ( 0.57 sec)

All parameters

Run command testament to watch all parameters.

Usage:
  testament [options] command [arguments]

Command:
  p|pat|pattern <glob>        run all the tests matching the given pattern
  all                         run all tests
  c|cat|category <category>   run all the tests of a certain category
  r|run <test>                run single test file
  html                        generate testresults.html from the database
  stats                       generate statistics about test cases
Arguments:
  arguments are passed to the compiler
Options:
  --print                   also print results to the console
  --simulate                see what tests would be run but don't run them (for debugging)
  --failing                 only show failing/ignored tests
  --targets:"c c++ js objc" run tests for specified targets (default: all)
  --nim:path                use a particular nim executable (default: $PATH/nim)
  --directory:dir           Change to directory dir before reading the tests or doing anything else.
  --colors:on|off           Turn messages coloring on|off.
  --backendLogging:on|off   Disable or enable backend logging. By default turned on.
  --megatest:on|off         Enable or disable megatest. Default is on.
  --skipFrom:file           Read tests to skip from `file` - one test per line, # comments ignored

Specify configuration

You can specify configuration in the front of your program.

For example, you can specify configuration in main.nim:

discard """
  cmd:      "nim c -r --styleCheck:hint --panics:on $options $file"
  matrix:   "--gc:arc; --gc:arc --d:release"
  targets:  "c"
  nimout:   ""
  action:   "run"
  exitcode: 0
  timeout:  60.0
"""

import ../src/main


doAssert hello(1) == 5

block:
  doAssert hello(3) == 7

Often I just need change matrix parameters and specify multiple configurations separated by ;.

You can also specify strings printed by program:

discard """
output: '''
true
false
1
'''
"""

echo true
echo false
echo 1

Specify error messages:

discard """
  errormsg: "undeclared identifier: '++'"
"""

echo 12 ++ 1

For more details, surf the Nim github repo to get more intuitions.

https://github.com/nim-lang/Nim/tree/devel/tests

Real world use

I use testament in my project:

Prologue

https://github.com/planety/prologue/tree/master/tests

https://github.com/planety/prologue/blob/master/prologue.nimble

Official docs

http://nim-lang.github.io/Nim/testament.html

Conclusion

Nim is an elegant and efficient programming language. And testament is also a powerful test tool. Find a fun project and enjoy your life!

Comments

[Constantine Molchanov]

Great read, thanks!

Do you know a way to run different test categories in parallel? I couldn't find any info in the docs.

[Timothee Cour]

there's currently:

• testament all (all tests)
• testament pcat <cat>, eg: testament pcat js or testament pcat vm

there's currently no testament pcat js,vm but it's not hard to implement, PR welcome

finally, github.com/timotheecour/Nim/issues... would be the most flexible, doing away with hierarchy and filtering by labels instead, PR welcome

[Constantine Molchanov]

Thanks!

But what is pcat exactly? There's no help for it. Does it mean "parallel cat"? How do I run two categories in parallel?