Tsonnet #10 - There and back again… concatenating strings

Welcome to the Tsonnet series!

If you're just joining, you can check out how it all started in the first post of the series.

In the previous post, we added support for identifiers:

Tsonnet #9 - ID please

Hercules Lemke Merscher ・ Feb 12

#tsonnet #jsonnet #compiler

Now let's go back to one topic we covered already, concatenating strings.

Why revisit string concatenation?

In Jsonnet, the plus (+) operator is overloaded by design, meaning it can adhere to the object type and do a certain operation depending on the context. For example:

{
  // concatenating strings
  a_string: "abc" + "def",
  // arithmetic operation
  a_sum: 1 + 2,
  // object merging
  an_object: { name: "John" } + { surname: "Doe" }
  // this also works
  mixed: "asdf" + 123 + [4]
}

Those are all valid. Let's run this through Jsonnet and see what we get:

$ jsonnet concat.jsonnet
{
   "a_string": "abcdef",
   "a_sum": 3,
   "an_object": {
      "name": "John",
      "surname": "Doe"
   },
   "mixed": "asdf123[4]"
}

If you're curious about the nitty-gritty details of this behavior, I've dug deep into it in another post:

The Sneaky String Concatenation Of Jsonnet

Hercules Lemke Merscher ・ Jan 20

#jsonnet

This is not very sound when it comes to type-checking. A safer and expected behavior would be std.toString to convert whatever type we want to concatenate, to match the string type, explicitly. Not implicit conversions.

However, it is what it is. If I want to keep compatibility with Jsonnet, I need to support this behavior.

BREAKING CHANGE: the odd case where a string and an object side-by-side, without using the plus operator, result in a string concatenation won't be supported in Tsonnet -- really, this is possible in Jsonnet. How mindblowing is that?! I covered this in the Jsonnet post above, in case you haven't read it.

Eventually, I'd like to add a flag to the compiler, to strictly check and fail on such cases. But until the stdlib is implemented, and for compatibility's sake, the default behavior will be the implicit conversion.

Let's concatenate strings, the universe, and everything

First things first. Let's add a new Jsonnet sample to cover the string concatenation to varied values:

diff --git a/samples/strings/concat_to_multiple_types.jsonnet b/samples/strings/concat_to_multiple_types.jsonnet
new file mode 100644
index 0000000..dd52512
--- /dev/null
+++ b/samples/strings/concat_to_multiple_types.jsonnet
@@ -0,0 +1,5 @@
+"2 apples"
++ ", " + 42.0
++ ", " + true
++ ", " + [42]
++ ", " + { answer: 42 }

And a cram test targeting it -- plus a tiny refactoring renaming the cram test file:

diff --git a/test/cram/concat_strings.t b/test/cram/concat_strings.t
deleted file mode 100644
index 5efda56..0000000
--- a/test/cram/concat_strings.t
+++ /dev/null
@@ -1,2 +0,0 @@
-  $ tsonnet ../../samples/concat_strings.jsonnet
-  "asdfhjklasdfhjkl!!!"
diff --git a/test/cram/strings.t b/test/cram/strings.t
new file mode 100644
index 0000000..e506cb1
--- /dev/null
+++ b/test/cram/strings.t
@@ -0,0 +1,5 @@
+  $ tsonnet ../../samples/strings/concat.jsonnet
+  "asdfhjklasdfhjkl!!!"
+
+  $ tsonnet ../../samples/strings/concat_to_multiple_types.jsonnet
+  "2 apples, 42.0, true, [ 42 ], { \"answer\": 42 }"

Now comes the fun part -- we need to make Tsonnet handle all these cases. Here's how we're going to restructure our code:

diff --git a/lib/tsonnet.ml b/lib/tsonnet.ml
index ae6eb91..94e185c 100644
--- a/lib/tsonnet.ml
+++ b/lib/tsonnet.ml
@@ -10,7 +10,7 @@ let parse (s: string)  =
   try ok (Parser.prog Lexer.read lexbuf)
   with | Lexer.SyntaxError err_msg -> error err_msg

-let interpret_bin_op (op: bin_op) (n1: number) (n2: number) : expr =
+let interpret_arith_op (op: bin_op) (n1: number) (n2: number) : expr =
   match op, n1, n2 with
   | Add, (Int a), (Int b) -> Number (Int (a + b))
   | Add, (Float a), (Int b) -> Number (Float (a +. (float_of_int b)))
@@ -29,17 +29,28 @@ let interpret_bin_op (op: bin_op) (n1: number) (n2: number) : expr =
   | Divide, (Int a), (Float b) -> Number (Float ((float_of_int a) /. b))
   | Divide, (Float a), (Float b) -> Number (Float (a /. b))

+let interpret_concat_op (e1 : expr) (e2 : expr) : (expr, string) result =
+  match e1, e2 with
+  | String s1, String s2 -> ok (String (s1^s2))
+  | String s1, expr2 ->
+    let* s2 = Json.expr_to_string expr2 in
+    ok (String (s1^s2))
+  | expr1, String s2 ->
+    let* s1 = Json.expr_to_string expr1 in
+    ok (String (s1^s2))
+  | _ -> error "invalid string concatenation operation"
+
 (** [interpret expr] interprets and reduce the intermediate AST [expr] into a result AST. *)
 let rec interpret (e: expr) : (expr, string) result =
   match e with
   | Null | Bool _ | String _ | Number _ | Array _ | Object _ | Ident _ -> ok e
-  | BinOp (Add, String a, String b) -> ok (String (a^b))
   | BinOp (op, e1, e2) ->
     let* e1' = interpret e1 in
     let* e2' = interpret e2 in
-    match e1', e2' with
-    | String a, String b -> ok (String (a^b))
-    | Number v1, Number v2 -> ok (interpret_bin_op op v1 v2)
+    match op, e1', e2' with
+    | Add, (String _ as expr1), (_ as expr2) | Add, (_ as expr1), (String _ as expr2) ->
+      interpret_concat_op expr1 expr2
+    | _, Number v1, Number v2 -> ok (interpret_arith_op op v1 v2)
     | _ -> error "invalid binary operation"

 let run (s: string) : (string, string) result =

1. The interpret_bin_op function was doing arithmetic operations only, so it makes sense to rename it to interpret_arith_op
2. The pattern match on the binary operation will check if one of the expressions is a string to know if it is a string concatenation operation
3. In the interpret_concat_op, the non-string expression is transformed to a string before concatenation

And voilà:

$ dune exec -- tsonnet samples/strings/concat_to_multiple_types.jsonnet
"2 apples, 42.0, true, [ 42 ], { \"answer\": 42 }"

Conclusion

In this post, we've implemented Jsonnet's string concatenation behavior in Tsonnet, including its distinctive approach to type coercion. While this implicit type conversion might seem counterintuitive from a type safety perspective, it maintains compatibility with Jsonnet's specification.

This addition to Tsonnet highlights an interesting trade-off: the balance between type safety and convenience. While automatic type coercion can make code more concise and easier to write, it may also introduce subtle bugs that are hard to catch. That's why I'm planning to add that strict mode flag -- you'll get to pick your poison: Jsonnet compatibility or stricter type checking.

---

Enjoying this wild ride through Tsonnet's development? Hit subscribe to follow along as we build a programming language, one questionable operator at a time!

Photo by oliver spicer on Unsplash