Tsonnet #2 - Cram tests to the rescue

In the previous post, I added the parsing of JSON literals to Tsonnet:

Tsonnet #1 - JSON literals

Hercules Lemke Merscher ・ Jan 29

#tsonnet #compiler

There's something important still missing though and it is tests. Better not to proceed without guaranteeing the language specification is being validated by automated tests.

Let's do it!

Adding the testing libraries

We'll add alcotest for writing automated tests -- it seems like the favorite testing library for OCaml programmers nowadays instead of ounit. Let's also add bisect_ppx for coverage report:

diff --git a/dune-project b/dune-project
index cf8c03f..383bbef 100644
--- a/dune-project
+++ b/dune-project
@@ -25,7 +25,15 @@
   (dune
    (>= 3.16.0))
   (menhir
-   (= 20240715)))
+   (= 20240715))
+  (alcotest
+   (and
+    :with-test
+    (>= 1.8.0)))
+  (bisect_ppx
+   (and
+    :with-test
+    (>= 2.8.3))))
  (tags
   (jsonnet interpreter compiler)))

In dune, we can specify that a dependency belongs to a specific environment -- in this case, the test environment. This way we don't carry testing dependencies to the release binaries.

Running dune runtest is convenient, but the coverage command isn't. Let's add a Makefile to free our minds of the details:

diff --git a/.gitignore b/.gitignore
index 69fa449..39136cd 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1,2 @@
 _build/
+_coverage/
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..688c42d
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,15 @@
+default: test
+
+.PHONY: test
+test:
+       dune runtest
+
+.PHONY: coverage
+coverage:
+       dune runtest --instrument-with bisect_ppx --force
+       bisect-ppx-report html
+
+.PHONY: clean
+clean:
+       dune clean
+       rm -rf _coverage/

I appreciate that simply running make targets the make test. I always like to have the test command as the default for my projects. The make coverage command is now convenient to run locally and we can leverage it when setting up a CI job later on.

Now, let's configure dune.

A little bit of configuration

diff --git a/bin/dune b/bin/dune
index 300e220..b274fc4 100644
--- a/bin/dune
+++ b/bin/dune
@@ -2,3 +2,8 @@
  (public_name tsonnet)
  (name main)
  (libraries tsonnet))
+
+(cram
+ (deps
+  %{bin:tsonnet}
+  (source_tree ../samples)))
diff --git a/test/cram/dune b/test/cram/dune
new file mode 100644
index 0000000..4c224ef
--- /dev/null
+++ b/test/cram/dune
@@ -0,0 +1,4 @@
+(cram
+ (deps
+  %{bin:tsonnet}
+  (source_tree ../../samples)))
diff --git a/test/dune b/test/dune
index f8dbe1e..bd42c05 100644
--- a/test/dune
+++ b/test/dune
@@ -1,2 +1,5 @@
 (test
- (name test_tsonnet))
+ (name test_tsonnet)
+ (libraries tsonnet alcotest)
+ (deps
+  (source_tree ../samples)))

We are going to specify that we can write cram tests in the bin directory and the tests will have two dependencies:

1. The Tsonnet binary that dune will compile for us
2. The samples directory, where we store the Jsonnet file samples, to be used as input in the tests

We are also adding a similar configuration to the test directory, where we specify that cram tests will live in the test/cram folder and the binary and sample files dependencies.

The main entry-level test folder will have the sample files as a dependency and the alcotest library.

I'll go into more detail about cram tests in the next section.

For the coverage:

diff --git a/lib/dune b/lib/dune
index 9660313..3452131 100644
--- a/lib/dune
+++ b/lib/dune
@@ -1,5 +1,7 @@
 (library
- (name tsonnet))
+ (name tsonnet)
+ (instrumentation
+  (backend bisect_ppx)))

 (menhir
  (modules parser))
diff --git a/lib/lexer.mll b/lib/lexer.mll
index a581fcd..585673b 100644
--- a/lib/lexer.mll
+++ b/lib/lexer.mll
@@ -1,4 +1,5 @@
 {
+  [@@@coverage exclude_file]
   open Lexing
   open Parser
   exception SyntaxError of string
diff --git a/lib/parser.mly b/lib/parser.mly
index 6872459..2396dfe 100644
--- a/lib/parser.mly
+++ b/lib/parser.mly
@@ -1,3 +1,7 @@
+%{
+  [@@@coverage exclude_file]
+%}
+
 %token <int> INT
 %token <float> FLOAT
 %token NULL

We need to instrument the bisect_ppx coverage tool to set the lib directory as its target, otherwise, it is going to completely ignore this directory, which is where all our compiler logic lives.

Since the lexer and parser are generated, there's no point in checking coverage for them, so we basically annotate them to be excluded.

Now we are ready to finally write some tests.

 Testing with cram tests

Do you know what are cram tests?

TLDR, cram tests are integration tests that describe a shell session. They are amazing to test CLI programs!

Do you know what else is cool? Dune has built-in support for writing cram tests. The reason why no new library has been added to handle that.

We can write runnable documentation with that. Remember that we configured dune to have cram tests in the bin folder? Here are cram tests to cover some usage cases of Tsonnet:

diff --git a/bin/usage.t b/bin/usage.t
new file mode 100644
index 0000000..7d13b46
--- /dev/null
+++ b/bin/usage.t
@@ -0,0 +1,10 @@
+Using the Tsonnet program:
+
+  $ tsonnet ../samples/literals/int.jsonnet
+  42
+
+  $ tsonnet ../samples/literals/string.jsonnet
+  "Hello, world!"
+
+  $ tsonnet ../samples/literals/object.jsonnet
+  {"int_attr": 1, "float_attr": 4.200000, "string_attr": "Hello, world!", "null_attr": null, "array_attr": [1, false, {}], "obj_attr": {"a": true, "b": false, "c": {"d": [42]}}}

Your cram test files need to have the .t extension.

You can mix free text with properly indented shell commands. The line right below the command is the expected output.

How cool is that?!

In the test/cram directory we can test each sample source file, along with its expected output:

diff --git a/test/cram/literals.t b/test/cram/literals.t
new file mode 100644
index 0000000..c7b1c2a
--- /dev/null
+++ b/test/cram/literals.t
@@ -0,0 +1,36 @@
+  $ tsonnet ../../samples/literals/int.jsonnet
+  42
+
+  $ tsonnet ../../samples/literals/float.jsonnet
+  4.222222222222222
+
+  $ tsonnet ../../samples/literals/negative_int.jsonnet
+  -42
+
+  $ tsonnet ../../samples/literals/negative_float.jsonnet
+  -4.222222222222222
+
+  $ tsonnet ../../samples/literals/true.jsonnet
+  true
+
+  $ tsonnet ../../samples/literals/false.jsonnet
+  false
+
+  $ tsonnet ../../samples/literals/null.jsonnet
+  null
+
+  $ tsonnet ../../samples/literals/string.jsonnet
+  "Hello, world!"
+
+  $ tsonnet ../../samples/literals/array.jsonnet
+  [ 1, 2.0, "hi", null ]
+
+  $ tsonnet ../../samples/literals/object.jsonnet
+  {
+    "int_attr": 1,
+    "float_attr": 4.2,
+    "string_attr": "Hello, world!",
+    "null_attr": null,
+    "array_attr": [ 1, false, {} ],
+    "obj_attr": { "a": true, "b": false, "c": { "d": [ 42 ] } }
+  }

To finalize, let's run the tests:

$ dune runtest

There are no errors when all the tests have passed.

Let's introduce a small error, changing the output value of one or two of the tests, and see how it is presented:

Here's our intentionally introduced error changes:

diff --git a/test/cram/literals.t b/test/cram/literals.t
index c7b1c2a..54d455f 100644
--- a/test/cram/literals.t
+++ b/test/cram/literals.t
@@ -1,8 +1,8 @@
   $ tsonnet ../../samples/literals/int.jsonnet
-  42
+  666

   $ tsonnet ../../samples/literals/float.jsonnet
-  4.222222222222222
+  4.23

   $ tsonnet ../../samples/literals/negative_int.jsonnet
   -42

The output of the tests:

$ dune runtest
File "test/cram/literals.t", line 1, characters 0-0:
diff --git a/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t b/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t.corrected
index 54d455f..c7b1c2a 100644
--- a/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t
+++ b/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t.corrected
@@ -1,8 +1,8 @@
   $ tsonnet ../../samples/literals/int.jsonnet
-  666
+  42

   $ tsonnet ../../samples/literals/float.jsonnet
-  4.23
+  4.222222222222222

   $ tsonnet ../../samples/literals/negative_int.jsonnet
   -42

It is super easy to spot the expected value and the actual result we got. Isn't it nice?!

Concluding

Cram tests have become one of my favorite tools. I can't live without them if I'm writing a CLI program seriously.

This is not a new idea. The first time I saw something similar was in Elixir via doctests. I read somewhere that this cram Python package is where it originated, but I can't say for sure. The last time I tried Rust was many years ago, but as far as I know, Rust has documentation tests.

I don't know who copied who, but I hope this trend continues. This is a really cool feature that I hope every single programming language implements, either fully as cram tests or partially as documentation tests.

Disclaimer: alcotest is basically sitting idle for now, as there are no unit tests or integration tests besides the cram tests. Eventually, there will be. I just anticipated adding the dependency right away.

What do you think about cram tests?

---

Thanks for reading Bit Maybe Wise! Subscribe to receive new posts about Tsonnet and cram tests.

Photo by Tai Bui on Unsplash