Building and testing Raku in AppVeyor

Originally published on peateasea.de.

Trying to get an old Raku project up and running again led me down a deep rabbit hole. I ended up working out how to set up, build, and test Raku projects on Windows with the AppVeyor CI platform. These notes are a guide to help anyone wanting to create an AppVeyor configuration in the future.

This is quite a detailed discussion, so strap yourself in!

A bit of online CI platform history

AppVeyor is an online continuous integration (CI) platform made available to Open Source software projects for free. It links with GitHub, GitLab, BitBucket, kiln, and Visual Studio Team Services and runs automated CI checks on pushes and pull requests.

About 5 or 6 years ago, AppVeyor was the best system to test Open Source software (such as Perl or Raku distributions) on Windows systems. At the time Travis-CI, the other major online CI platform available for Open Source projects, only supported Linux systems. It was good to have an alternative where, even as a Linux dev, one could test code on Windows. Travis-CI has since fallen out of favour, as it is “no longer free for Open Source accounts”, but that’s a different story.

With the addition of GitHub Actions, AppVeyor has gone out of fashion somewhat. Even so, it still exists, is still available for Open Source use, and can still test code on a variety of Windows systems.

I’m stubborn: I want software to “just work”

The motivation for this post came from me trying to fix some issues in an old Raku project and to bring its code and configuration up to date. The project still used AppVeyor, yet the build was failing. Since I’d been able to run the tests on my local Linux box, I knew that it worked on Linux, but I wanted to make sure that builds still worked on Windows. For that, I needed a working AppVeyor config. Thus I needed to update it to use the current tools used in the Raku community for installing Raku and for building and installing any project-level dependencies (things have come a long way in the last 6 or so years!). In particular, the old tool for “brewing” a Raku installation (rakudobrew) has long since been replaced by rakubrew. This was the main failure with the current AppVeyor config and something I wanted to fix before moving on to other issues.

“Why bother?” one might ask. Well, I’m stubborn and I want software to “just work”, especially for users of that software, who in this case are likely to be other developers. I mean, if it’s hard to install and use someone’s software (in this case a module distribution), then people are less likely to use it. I want to reduce barriers and minimise friction for others if at all possible, so that using software is easy and a pleasure. Or put another way: I don’t think people should have to fight their tools only to get stuff done.

Two working AppVeyor configurations

Ok, enough about my motivations for wanting to get AppVeyor working again, what does a working configuration look like for a Raku project? Mauke already discussed this topic in his post about building Perl projects with AppVeyor. That post helped me iron out some of the wrinkles in the configuration that I present here. What follows are two working Appveyor configurations. The main difference between them is the script engine used to install prerequisites and run the tests.

For AppVeyor to trigger builds, you will need to give AppVeyor access to your repositories. To do this, sign in to AppVeyor (e.g. with your GitHub credentials) and add your project to the list of projects AppVeyor monitors for push events. All this is described in the AppVeyor documentation. You configure a build via aYAML file called appveyor.yml in your project’s base directory.

The main block I will focus on here is the install section of the configuration. This is where we do most of the hard work in setting up the Raku environment. Doing this work up front makes the test_script section a simple prove call. There are two ways to install the base dependencies: use either Windows Command Prompt, or Windows PowerShell as the script engine¹. The remaining AppVeyor configuration options are the same for both environments.

Let’s first discuss the basic options common to each config before focussing on the details of the install section for each script engine flavour.

Basic config options

The options common to each configuration are, image, platform, build, test_script, and shallow_clone.

image

This option specifies the build worker image (VM template) to initialise and hence defines the software that is pre-installed on the virtual machine for running the CI tasks.

image: Visual Studio 2022

AppVeyor currently lists Visual Studio 2013 through Visual Studio 2022 as the available Windows-based images. Thus, there is some flexibility for users when choosing exactly which image to use. I chose to use Visual Studio 2022 because it’s the most up-to-date version.

platform

This parameter is optional and can be set to x86, x64 or Any CPU. We want to focus on x64 systems, hence we specify the platform here explicitly.

platform: x64

build

As mentioned in the AppVeyor Build phase docs:

After cloning the repository, AppVeyor runs MSBuild to build project sources and package artifacts.

Because we don’t want to run MSBuild (we only need to install Raku and zef), we disable the automatic build step. Hence we set:

build: off

in the config file.

test_script

This section specifies the list of commands to run when running the tests. Basically, all we do is run the relevant prove command, using raku to execute the tests. The only special thing to note is the addition of the Strawberry Perl path to the main PATH environment variable. This extra line of code is necessary so that prove can be found by the respective shell. The Visual Studio 2019 and Visual Studio 2022 images provide Strawberry Perl by default. Thus, if you wish to use an older Visual Studio version, you will need to install and set up Strawberry Perl, optionally using caching to speed up builds.

The test_script section looks like this

test_script:
  - SET PATH=C:\Strawberry\perl\bin;%PATH%
  - prove -v -e "raku -I." t/

for CMD, and like this

test_script:
  - ps: $Env:Path = "C:\Strawberry\perl\bin;$Env:Path"
  - ps: prove -v -e "raku -I." t/

for PowerShell.

If you’re used to seeing raku -Ilib as the program passed to prove and are wondering why I’ve used raku -I. here, there’s a reason. This is because the -Ilib variant²

doesn’t use META6.json, thus isn’t equivalent to what the installed version might be. For example, if you fail to list a module in META6.json it would fail very obviously with -I., whereas -Ilib guesses what files should be included based on the files it sees.

In other words, doing things this way ensures that we catch any configuration-related errors before they end up affecting any users.

shallow_clone

This parameter defines how Git should clone the upstream repository. We only want to build and test everything for the commit that triggered the build, hence we avoid cloning the entire upstream repository. Not only would a full clone waste a lot of space and network resources, but it also makes the build take an unnecessarily long amount of time. Thus we ensure that shallow cloning is switched on:

shallow_clone: true

Notes about the install section

Now that we’ve discussed the basic options, let’s look at a complete configuration and discuss the core of the build configuration: the install section.

Setting up rakubrew in a CI environment can be tricky because the installation procedure is different to what one would use on, say, one’s laptop. One reason is that the shell used on the CI VM image only runs once; any startup routines will have already run before we get a chance to change them. Thus we can only extend the environment within the current shell session. Also, build images are ephemeral, meaning that any changes to the environment will be lost after the CI run has finished. That was a long-winded way of saying that we have to set up paths ourselves, and we have to use the rakubrew’s shim mode rather than the default env mode.

Since some of the steps won’t be obvious, I’m going to spend some time discussing each of the commands within the respective install section. This way it’s clearer what their purpose is and why they’re needed.

A deep dive into the install section (Windows Command Prompt)

AppVeyor uses the Windows Command Prompt (a.k.a. CMD) by default, in what the docs sometimes refer to as “batch”. If you read more of the docs, you’ll find that CMD and “batch” are sometimes referred to as slightly separate concepts. At other times they seem completely interchangeable, which can be a bit confusing. From my experience, running commands without specifying a script engine in the install section runs each line via CMD.

To be explicit about using CMD in the install or test_script sections, prepend each line with cmd: . This will ensure that the command runs via Windows Command Prompt. For instance:

install:
  - cmd: echo Hello World

But why use CMD? That’s such hard work!

Well, if I tried to use only CMD, then someone else would also try to do it and most likely run into the same issues I found. Therefore the hope is that this information will help them (assuming, of course, that they find it!).

Here’s the configuration I ended up with for the Windows Command Prompt use case:

# appveyor.yml
image: Visual Studio 2022

platform: x64

install:
  - curl https://rakubrew.org/install-on-cmd.bat -o install-on-cmd.bat && install-on-cmd.bat
  - SET PATH=C:\rakubrew\bin;%PATH%
  - SET PATH=C:\rakubrew\shims;%PATH%
  - rakubrew mode shim
  - rakubrew download
  - rakubrew build zef
  - zef --verbose install .

build: off

test_script:
  - SET PATH=C:\Strawberry\perl\bin;%PATH%
  - prove -v -e "raku -I." t/

shallow_clone: true

Let’s pick apart the install section line-by-line.

Install rakubrew (CMD)

The first thing we do is install rakubrew. To do this, we download a CMD batch script to install rakubrew and run it directly afterwards.

curl https://rakubrew.org/install-on-cmd.bat -o install-on-cmd.bat && install-on-cmd.bat

This command follows a similar pattern to other installation scripts one might see online used in combination with curl, i.e.

curl <https://some-url> | sh

Since there’s no such thing as a pipe in CMD, it’s not possible to pass the script code directly from curl into the shell to execute it.

To make the command have this common form we save the script to an intermediate file (via the -o install-on-cmd.bat option). Then we run the downloaded file (install-on-cmd.bat). Note that entering a batch script’s name in CMD runs the code in the file. The trick here is to join the two commands together with && so that they appear on the same line.

It’s nice that this command has the same shape as an already familiar pattern for the same concept used on other platforms. This way one can understand it quickly and intuitively without needing to dig into the command’s details.

Set up the main PATH for rakubrew

Now that we’ve installed rakubrew, we need to extend the PATH environment variable so that we can run the rakubrew command.

SET PATH=C:\rakubrew\bin;%PATH%

Set up the shim path for rakubrew

As mentioned earlier, we need to use rakubrew’s shim mode, thus we need to add the shims path to the PATH. In contrast to the command mentioned in the bare bones installation section of the rakubrew docs, we hard-code the value. After all, we know that we installed rakubrew into its default location C:\rakubrew.

SET PATH=C:\rakubrew\shims;%PATH%

Diversion: complexities of the general shim path setup in CI

The documented way to set up the shims path is:

FOR /F "delims=" %i IN ('"rakubrew" home') DO SET PATH=%i/shims;%PATH%

This command runs rakubrew home and uses its output to construct the shim path and then adds that path to the PATH environment variable. It seems like we’re doing an awful lot of work here to effectively substitute the output of rakubrew home into a variable. If you’re used to command substitution from Unix-y shells, you’ll find it’s not possible to do command substitution directly in CMD.

As mentioned in the StackOverflow answer explaining how to do this:

Yeah, it’s kinda non-obvious (to say the least), but it’s what’s there.

So it looks like that’s what one has to do in the general case where rakubrew’s home isn’t known in advance.

What does the FOR loop used here do exactly? Well, it loops over the output generated by the command specified after IN one line at a time. The delims= quantifier ensures we ignore any delimiters, thus avoiding splitting the output on spaces. The loop then puts each element of the command’s output into the loop variable %i for each loop iteration. We then update the PATH environment variable in the loop body with the value of %i/shims, adding the path <rakubrew-home>\shims to the PATH.

Note that the FOR loop solution from the rakubrew docs mentioned above doesn’t work as-is within a scripted CI environment. The situation is subtle and one has to be careful to get the invocation correct.

One issue is that the double quotes around rakubrew aren’t necessary. In other words, using only rakubrew home works as a single command as one might expect. Hence, one can simplify the FOR loop to this:

FOR /F "delims=" %%i IN ('rakubrew home') DO SET PATH=%%i/shims;%PATH%

Unfortunately, due to the many quotes in this command, this isn’t valid YAML and we have to enclose it in single quotes. Making this change blindly also isn’t valid YAML due to the embedded single quotes surrounding rakubrew home. For YAML to handle these embedded single quotes and for the script engine to receive the correct code, one needs to double up the single quotes. In other words, putting two single quotes together in the YAML produces a single, erm, single quote in the shell. In the end, this is what the command looks like in the YAML config:

  - 'FOR /F "delims=" %%i IN (''rakubrew home'') DO SET PATH=%%i/shims;%PATH%'

There is another subtlety floating around here as well. Were we entering commands straight into the command prompt, we would use a single percent sign for the loop variable %i. This is the form mentioned in the rakubrew docs. But in a script, one needs to use two percent signs, hence why the above command uses two percent signs for the loop variable.

All this information is very well and good (and it works!) but it’s unnecessary. The reason is that we know, in this case, that we installed rakubrew into its default location (i.e. C:\rakubrew). Hence we hardcode the PATH value we need:

SET PATH=C:\rakubrew\shims;%PATH%

Use shim mode

As mentioned in Notes about the install section, we need to use rakubrew’s shim mode, so we simply turn that on here.

rakubrew mode shim

Download and install Raku

Now that rakubrew is set up, we download and install Raku itself. This lets us use raku and any pre-installed libraries the core distribution delivers. To do this we use the download command to rakubrew

rakubrew download

Note that this not only downloads the core Raku distribution but also installs it as well.

Download, build and install the zef package manager

We need to install our dist’s upstream dependencies as well, hence we install the Raku package manager, zef

rakubrew build zef

Install the dist’s upstream dependencies

With zef installed, we’re ready to install the dist’s upstream dependencies

zef --verbose install .

The --verbose option ensures that we see all output so that we can debug any issues should they arise. With install . we install not only all upstream dependencies but the dist itself. This has the advantage that everything is precompiled and thus we catch any compile time errors that might not be exercised by the test suite.³

Note that this is in contrast to when one develops code locally. In such a situation one usually only wants to install the dist’s dependencies and not the dist itself and thus we test the dist in isolation. In such a case, one would use

zef --verbose --deps-only install .

to install only the upstream dependencies.

Ready to go!

Now with the setup complete, we can run the Raku dist’s test suite as part of the test_script section.

All you need to do now is copy the YAML from A deep dive into the install section (Windows Command Prompt)), and paste it into a file called appveyor.yml. Then place this file in your project’s base directory (be sure to check it in to the repository) and you should be good to go!

A deep dive into the install section (Windows PowerShell)

The PowerShell install config section is very similar to that for the Windows Command Prompt. Still, there isn’t a 1-to-1 mapping between the two script engines, so a simple translation isn’t possible. Also, there are a few edge cases that definitely weren’t obvious when I started using PowerShell for the preliminary project setup.

Without further ado, here’s the configuration I landed upon in the Windows PowerShell use case:

# appveyor.yml
image: Visual Studio 2022

platform: x64

install:
  - ps: . {iwr -useb https://rakubrew.org/install-on-powershell.ps1 } | iex
  - ps: $Env:Path = "C:\rakubrew\bin;$Env:Path"
  - ps: $Env:Path = "$(rakubrew home)/shims;$Env:Path"
  - ps: rakubrew mode shim
  - ps: rakubrew download
  # Git reports "chatty" output to stderr thus causing errors to be raised on
  # PowerShell, hence we redirect stderr to stdout here.
  # See https://stackoverflow.com/a/47232450/10874800,
  # https://stackoverflow.com/a/54624579/10874800 and
  # https://stackoverflow.com/a/37561629/10874800 for more details.
  - ps: $env:GIT_REDIRECT_STDERR = '2>&1'
  - ps: rakubrew build zef
  - ps: zef --verbose install .

build: off

test_script:
  - ps: $Env:Path = "C:\Strawberry\perl\bin;$Env:Path"
  - ps: prove -v -e "raku -I." t/

shallow_clone: true

The main obvious difference here to the CMD use case is that we have to prefix each line with ps: for PowerShell to execute it.

As with the discussion of the Windows Command Prompt, let’s pick apart the install section line-by-line.

Install rakubrew (PowerShell)

The first thing to do is download and install rakubrew.

. {iwr -useb https://rakubrew.org/install-on-powershell.ps1 } | iex

This rather cryptic-looking command downloads a script to install rakubrew and runs it straight away. The environment variables set within the script are made immediately available to the running shell. Although the details are different, it uses the same pattern as

curl <https://some-url> | sh

as I discussed in Install rakubrew (CMD).

The leading dot “.” is the Dot sourcing operator and it

Runs a script in the current scope so that any functions, aliases, and variables that the script creates are added to the current scope, overriding existing ones.

In other words, any environment variables set up within the script are now available within the currently running shell. This is equivalent to sourcing scripts in Unix-y shells.

The command used instead of curl, in this case, is iwr, which is an alias for the PowerShell command Invoke-WebRequest and

Gets content from a web page on the internet.

After reading through the Invoke-WebRequest docs, I think that the -useb option is shorthand for -UseBasicParsing. The iwr documentation states that UseBasicParsing has been deprecated and as of PowerShell version 6.0.0 all web requests use basic parsing only. Thus we could remove this option from the call to iwr because the Visual Studio 2022 image provided by AppVeyor (and used here) comes with PowerShell 7.4.0. Still, I’ve decided to include the option here because it matches the rakubrew documentation.

iwr downloads a script from the rakubrew website and pipes its contents into iex which is an alias for the PowerShell Invoke-Expression command which

Runs commands or expressions on the local computer.

This is like piping the downloaded file straight into a shell, like the | sh invocation common in Unix-y settings.

Set up the main PATH for rakubrew

Setting up the PATH for the shell to be able to find the rakubrew binary is like the Windows Command Prompt case

$Env:Path = "C:\rakubrew\bin;$Env:Path"

The syntax is only slightly different: instead of PATH, the environment variable is $Env:Path. Also, it’s possible to put whitespace around the equals sign used for assignment, which makes reading the code much easier.

Set up the shim path for rakubrew

Unlike the case with CMD, PowerShell does support command substitution. This makes adding the shim path for rakubrew much easier within this environment.

$Env:Path = "$(rakubrew home)/shims;$Env:Path"

Here, the syntax for command substitution is the same as that used in shells like bash or zsh:

$(command-name)

Thus the result of the command can be directly substituted into the string constructing the shim path.

Using shim mode and installing Raku

To set up rakubrew’s shim mode, as well as download and install Raku, we use the same commands as in the CMD case:

rakubrew mode shim
rakubrew download

Redirect Git’s stderr stream to stdout

You read that correctly. Git’s stderr stream needs to be redirected to stdout.

# Git reports "chatty" output to stderr thus causing errors to be raised on
# PowerShell, hence we redirect stderr to stdout here.
# See https://stackoverflow.com/a/47232450/10874800,
# https://stackoverflow.com/a/54624579/10874800 and
# https://stackoverflow.com/a/37561629/10874800 for more details.
- ps: $env:GIT_REDIRECT_STDERR = '2>&1'

Hang on. What? Why do we have to do that? What’s that got to do with setting up a Raku build and test environment?

This is one reason why I’ve added lots of explanatory comments to the config here is because it’s really not obvious why this is necessary. It turns out that some Git commands (such as git clone) produce “chatty” output and this output gets sent to stderr. For instance (from the Git coding guidelines documentation):

An example of a chatty action command is git clone with its “Cloning into '<path>'...” and “Checking connectivity…” status messages which it sends to the stderr stream.

Other commands (such as git log or git show) produce “primary output”, sending it to stdout.

The StackOverflow answers mentioned in the code comments provide much more information, including references to individual commits.

There are more GIT_REDIRECT_* environment variables, yet they are only relevant on Windows. If you spend your time on Unix-based systems, you’re not likely to have run across them until now.

What’s nice about the GIT_REDIRECT_STDERR value is that it uses the familiar redirection syntax from Unix shells to redirect and append (>&) stderr (filehandle 2) to stdout (filehandle 1), i.e. 2>&1.

But wait, that still doesn’t explain why we need to do this at all, does it? After all, we’re not running any Git commands. True, we’re not running any Git commands directly, however when fetching the zef package manager, rakubrew clones the upstream Git repository. As mentioned above, git clone has “chatty” output which appears on stderr, and PowerShell reacts to this allergically. This means builds will fail with an error message like this:

git clone https://github.com/ugexe/zef.git
The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Cloning into 'zef'...

Download, build and install the zef package manager

Now that we’ve redirected Git’s stderr stream to stdout, we can install zef as we did in the CMD use case above

rakubrew build zef

Install the dist’s upstream dependencies

With zef installed, we’re ready to install the dist and its upstream dependencies

zef --verbose install .

which is the same process as we used for Windows Command Prompt.

Ready to go!

Now that we’ve completed the setup, we can run the tests in the test_script section.

All you need to do now is to copy the YAML code from A deep dive into the install section (Windows PowerShell)), and paste it into a file called appveyor.yml. Then place this file in your project’s base directory (be sure to check it in to the repository) and you should be good to go!

Extra AppVeyor configuration possibilities

While working out what an up-to-date working configuration looked like, I stumbled across some extra configuration possibilities worth mentioning. These are especially relevant when building and testing on older VM images.

Strawberry Perl installation

Strawberry Perl is pre-installed on Visual Studio 2019 and Visual Studio 2022 images. So, if you want to test your dist on an earlier image (Visual Studio 2017 and below), you’ll need to install it explicitly.

On Windows Command Prompt add the following code to the start of your config’s install section:

  - if not exist "C:\Strawberry" choco install strawberryperl -y
  - SET PATH=C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;%PATH%

We install Strawberry Perl via Chocolatey

choco install strawberryperl -y

only if its base directory (C:\Strawberry) does not already exist. The if check is useful when caching is enabled in the AppVeyor configuration.

On Windows PowerShell, the Strawberry Perl installation process looks like this:

  - ps: 'if ( -not (Test-Path -Path "C:\Strawberry") ) {
           choco install strawberryperl -y; refreshenv
         }
         else {
           $Env:Path = "C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;$Env:Path"
         }'

As with the CMD variant, we only install Strawberry Perl if its base directory doesn’t already exist. Note also that Test-Path is the PowerShell equivalent to exist in CMD).

The refreshenv command refreshes the $Env:Path after having installed something via Chocolatey. This path needs to be set explicitly if a cached directory exists because Chocolately and refreshenv won’t have run and thus perl wouldn’t be available within your path.

Caching installation artefacts

Often, the dependencies for a given project can be rather large. Downloading and installing these dependencies each time a CI build runs wastes resources. Thus one wants to avoid such expensive processes wherever possible. This is what the cache keyword is for: it tells AppVeyor a list of directories to keep after a successful build. AppVeyor reuses these cached directories in later builds, thus speeding things up.

In the particular case here, we want to cache the Strawberry Perl installation, as it downloads a lot of data (~170MB). Strawberry Perl gets installed into the C:\Strawberry directory, hence we list this name under the cache section:

cache:
  - 'C:\Strawberry'

To then use the cache, we check within the install section whether the directory already exists and if not, only then do we install Strawberry Perl.

Changing into the APPVEYOR_BUILD_FOLDER

Several AppVeyor configuration files I found online contain a command to change into the APPVEYOR_BUILD_FOLDER directory. For instance, in the case of Windows Command Prompt:

  - cd %APPVEYOR_BUILD_FOLDER%

or

  - ps: cd $Env:APPVEYOR_BUILD_FOLDER

for the PowerShell.

This is unnecessary as this is the current directory when the build starts. If you’ve copied this code from someone else’s configuration, you can most likely simply delete it: in most cases, it’s not doing anything.

Closing up

Crikey! That got much longer than intended!

So, that’s it basically: all the gory details of getting everything set up on AppVeyor to build and test your Raku distributions.

If you got this far, you’ve done really well! Thanks for sticking around until the end ☺️

1. It might even be possible to mix them, but I haven’t tried that. ↩

2. Thanks again to ugexe for letting me know about this finer point of raku behaviour! ↩

3. Many thanks to ugexe for pointing this out to me! ↩