Joker: Flex Your Scripting Muscles In (Almost) Clojure

If you read enough dev blogs, and have a touch of imposter syndrome,
it starts to look like our field is full of Uber-coders who can constantly
switch gears between, say, writing machine learning code in Python,
complex user interfaces in JavaScript and React, and then jump down and throw together a quick device driver in C. For fun, I imagine these Uber-coders then go home to write retro-games in 6502 assembly.

These people probably exist. I am not one of them. I've coded professionally
in many languages, everything from C, to PL/1, to Java, to Lua. But for day-to-day work, I specialize. I like Clojure. And I'm terrible at switching gears and languages, unlike my mythical Uber-coder.

Clojure is intensely good at what it does: REPL oriented development,
functional thinking, data transformation, and leveraging existing Java libraries.

But you don't write shell scripts in Clojure, even if you are as hopeless at Bash scripting as me.

Not that you don't want to, it's just that startup time is an obstacle: all the JVM startup time, plus all the Clojure startup time, plus all the time to load whatever libraries you need.

It adds up.

If you are working on an application, you may only fire up your REPL once in an entire day, so this startup time is inconsequential. But if you are writing a script ... either a throwaway, or something to help you automate your environment, you don't want to be twiddling your thumbs waiting for that script to load (even a one second wait can feel like forever).

That's why Joker is such a revelation; Joker is an interpreted dialect of Clojure, written in Google's Go, intended just for scripting: we use it internally on my team for all of our automation.

Nearly everything that makes Clojure amazing exists in Joker, including immutable data types, macros, and dynamic typing, and the vast majority of Clojure's core library.

Joker starts up very quickly:

> time joker -e '(println "There and back again")'
There and back again

real    0m0.050s
user    0m0.038s
sys 0m0.017s

That's virtually instantaneous. By comparison, Clojure has a noticable start up time:

> time clj -e '(println "A price must be paid")'
A price must be paid

real    0m1.011s
user    0m1.695s
sys 0m0.124s

Better yet, Joker is batteries included: it has a selection of built-in libraries, that cover all of your typical tasks, including sending HTTP requests, JSON and YAML encoding, basic cryptography ... all the things you need to, say, work with Amazon AWS.

One feature near and dear to me is joker.tools.cli, a port of Clojure's
clojure.tools.cli library. This is first class support for GNU-style command line options.

Using Joker

Let's say you got yourself a sweet job working for the Department of Defense, and wanted to create a script to control a few things at NORAD:

You can name this file norad, and make it executable (chmod u+x norad).

Because we made this a first class script, with properly parsed options, we can ask norad what it does:

> norad --help
norad OPTIONS

  -d, --defcon VALUE  Set DEFCON level
  -v, --version       Show version information
  -h, --help          Show this summary

That's a relief; a sloppy script that expected the user to remember its non-standard options might have changed the DEFCON to null, or launched some missiles or something. Real options can be a safety net.

And even though this script is non-trivial, it's still hella fast:

> time norad --defcon 3
Setting DEFCON to 3

real    0m0.086s
user    0m0.088s
sys 0m0.020s

And as someone who crusades for good feedback from my tools and libraries, it's great to see this:

> norad -d 0
norad OPTIONS

  -d, --defcon VALUE  Set DEFCON level
  -v, --version       Show version information
  -h, --help          Show this summary

Errors:
Failed to validate "-d 0": level must be 1 to 5

... even in a throwaway script. More safety net.

How it Works

That first line:

#!/usr/bin/env joker

That's standard Posix for "find the joker command and run the rest of the file through it". Any additional arguments to the command will be provided in *command-line-args*, a sequence of strings.

Because this is a script, and not a program, there's no main; there's just Clojure forms, evaluated one after another.

The ns form is convenient for bringing in the other built-in libraries that are needed. We gave this namespace the name script, but that's pretty arbitrary.

The next few forms define constants and utility functions needed to parse command line options and perform actions based on them. A real implementation of set-defcon might, for example, send an HTTP request to update the big display in the war room.

The last form, the let block, is where the real work starts; this is where those command line arguments are parsed into options, and then actions are dispatched. This is all very familiar to any Clojure programmer.

But that's it. No main. No classes. No compilation. No CLASSPATH. No overhead. Just your code, in (very nearly) The One True Language, but running instantly.

Comments

[Howard M. Lewis Ship]

I wasn't sure quite how to weave this into the narrative, but at one point I wrote a 20 line Python script that parsed some arguments and figured out the exact docker command to run for me. It was mostly for my own personal use, but I shared it with the rest of the team.

I did it in Python just because Python had a good built-in library for parsing command line options.

This triggered a multi-person-hour discussion about the merits of one language vs. another, whether it was appropriate to introduce yet another language into the system (beyond Clojure, Ruby, and some Bash scripts) and so forth. Ultimately, someone rewrote it as a Bash script, adding in nearly 100 lines of boilerplate, mostly dealing with parsing the command line options.

I kept my personal copy of the Python version.

Anyway, it's great to have a familiar, functional, fast scripting option ... with great command line arguments parsing.

[Daniel Pritchett ⚡]

Really appreciated this post! I'd been trying to find a good clojurish replacement to my sprawling ~/bin folder full of bash and ruby stuff. I tried lumo a bit but didn't really enjoy it. Joker is working out well!

Is there any way to find other folks' joker code in the wild? The API docs are nice but I could really use a lot more examples.

[Howard M. Lewis Ship]

Most of my Joker code is internal, but here's something I've put together for my own purposes.

github.com/hlship/dialog-tool

It's pretty much the kind of brutal, pragmatic kind of code you'd put in a Bash script, but in Joker, and it runs fast and nice.

[Daniel Pritchett ⚡]

Appreciate it, thank you.

[Daniel Pritchett ⚡]

Having a good time now - this is probably the first time I've actually gotten value out of Clojure at work in ten years of wishing :P

Silly convenience wrappers:

> cat ~/bin/jokes/lib/shell.joke
#!/usr/bin/env /usr/local/bin/joker

(ns lib.shell (:require [joker.os :refer [exec sh]]))

(defn sh-v
  "Verbosely execute a script via joker.os/sh."
  [bin & args]
    (apply println "Executing:\t" bin args)
    (apply sh bin args)
    (println "Complete."))

(defn exec-v
  "Verbosely execute a script via joker.os/exec."
  [bin args]
  (let [opts { :args args :stdin *in* :stdout *out* :err *err*}]
    (apply println "Executing:\t" bin args)
    (let [result (exec bin opts)]
      (println result)
      (println "Complete."))))

Usage:

> cat wtf-docker
#!/usr/bin/env /usr/local/bin/joker

(ns script
  (:require [lib.cli :refer [argv-str]]
            [lib.shell :as shell]))

(defn docker-shell
  "Open a bash shell inside the supplied docker tag or hash."
  [tag-or-hash]
  (let
    [sh-args ["run" "-ti" tag-or-hash "/bin/bash" "--login"]]
    (shell/exec-v "docker" sh-args)))

(def results (docker-shell argv-str))

(println results)

Demo:

> wtf-docker ubuntu:latest
Executing:   docker run -ti ubuntu:latest /bin/bash --login
root@b84798f0415c:/# logout
{:success true, :exit 0, :err }
Complete.
nil

[Bob Nadler]

Another tool in the toolbox. Thanks for sharing!

Note for Emacs users. If you don't have a file extension you can set the mode at the top of the script like this:

#!/usr/bin/env joker
;; #-*- mode: clojure -*-
...

[Alan Thompson]

Thank you for the review, but there is something even better available now that GraalVM is out and able to convert normal Clojure code into a statically linked executable.

Details are here: github.com/BrunoBonacci/graalvm-cl...

I just tried out the "Hello World" yesterday and it works great. Looks like it could be a game changer for resource constricted environments (startup time

[Howard M. Lewis Ship]

I'm sure GraalVM has its uses, but for actually getting things done in a tiny script, Joker is incredibly effective. Joker is a 15M self-contained executable and Joker scripts are tiny. For scripting, where you often want to iterate quickly, it's nice not to have a build process to generate a large executable.

Having fast startup via GraalVM would be terrific for, say, AWS Lambda.

For what I used Joker for, I want to be able to make changes locally and quickly and not have a build process.

A current example is our internal DevOps script, fleet; it's about 1300 lines of code, with 25 sub-commands. fleet help executes in .22 seconds - that's time for Joker to initialize, load several libraries, and the 1300 lines of code, organize a bunch of stuff, and print out the command list. Fast enough for me.

[Blake Miller]

Thank you for writing this!

[phil jones]

Have you tried github.com/dundalek/closh ? Would be interesting to compare.