A Guide On Debugging Tests in Stencil

Prerequisites: This article assumes you have some basic familiarity with writing and running tests in Javascript, Stencil.js, and VSCode's debugger. There are links at the bottom if you need some brushing up on any of the above.

I'm going to be focusing on end to end tests here because these are mostly the types of test you'll be writing more of to give you insight your UI is working. Unit (Spec) tests fall short of that when working on front-end components.

My tests don't work and I don't know why

So the story starts something like this. You started working with Stencil.js, whether for hobby or profit, and are creating web components. Then you start writing some end to end tests for some of your components. The tests aren't passing, but they should be. Or you're getting an error when running the tests. If you're like me, you may have noticed that debugging the tests isn't as simple as just adding a breakpoint - there are some gotchas. I'll share what worked for me.

A little Background Helps

Stencil is using Jest and Puppeteer under the hood. You'll do well to take a bit of time to try and understand what part of your testing code is using which API. In general, Jest is handling the boilerplate of defining tests (e.g. describe() and it()) and it's also handling the assertions (e.g. expect() and all its chain functions). Puppeteer sets up the page in a browser-like environment (a.k.a. headless browser) called Chromium (e.g. const page = await newE2EPage()) and adds your component to the page (e.g. page.setContent()). You're also using Puppeteer to find elements in the DOM and inspect properties, attributes or other state like isVisible.

It's also worth noting that Stencil has extended these APIs with some custom functions and helpers. newE2EPage shown above is actually provided by Stencil, not Puppeteer, but the page element it returns is a Puppeteer page Class. This is a blessing and a curse. Stencil handles most of the Puppeteer boilerplate for you, and offers some helpful utilities for verifying component custom events. But Stencil has also renamed/wrapped some of these APIs with function names that are different than what's found in the respective libraries docs. And as of this writing, the Stencil team hasn't explicitly documented a list of alias' or notable differences yet. I've taken one of the example snippets from their testing page and added comments to give you an idea. For example, Puppeteer's original API uses page.$() and page.$$() for find one element or find many elements, but Stencil has extended these and is calling them page.find() and page.findAll(). The piercing selector >>>, which is quite useful, is only available from these extended methods.

// Stencil
import { newE2EPage } from '@stencil/core/testing';

// Jest
describe('example', () => {
  // Jest
  it('should render a foo-component', async () => {
    // Stencil wrapping Puppeteer. 
    const page = await newE2EPage();
    // Puppeteer
    await page.setContent(`<foo-component></foo-component>`);
    // Stencil wrapping Puppeteer    
    const el = await page.find('foo-component');
    // Jest
    expect(el).not.toBeNull();
  });
});

The most comprehensive way to see these is to look at the Stencil type definition file, which has solid code comments.

Node or Chromium? Which environment is my code running in?

The reason the above is relevant is two-fold. First is it helps to know whose docs you should be reading. And second, there's two different environments, each with their own scope, your test code runs in - and knowing where is necessary for you to setup breakpoints and logging statements properly. Node is where Jest code and Puppeteer is running. Puppeteer start's an instance of Chromium, and that's where the actual component code is running. It's also possible to run code in Chromium using Puppeteer's page.evaluate() which we'll look at in a moment.

Show me the code Already

Debugging in Node

1. Add these configs to your VSCode Debug configurations. And Run Debug Stencil Tests.

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Stencil tests",
      "cwd": "${workspaceFolder}",
      "program": "${workspaceFolder}/node_modules/.bin/stencil",
      "args": ["test", "--spec", "--e2e", "--devtools"],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "disableOptimisticBPs": true,
      "windows": {
        "program": "${workspaceFolder}/node_modules/@stencil/core/bin/stencil"
      }
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Spec Test Current File",
      "cwd": "${workspaceFolder}",
      "program": "${workspaceFolder}/node_modules/.bin/stencil",
      "args": ["test", "--spec", "${fileBasename}"],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "disableOptimisticBPs": true,
      "windows": {
        "program": "${workspaceFolder}/node_modules/@stencil/core/bin/stencil"
      }
    },
    {
      "type": "node",
      "request": "launch",
      "name": "E2E Test Current File",
      "cwd": "${workspaceFolder}",
      "program": "${workspaceFolder}/node_modules/.bin/stencil",
      "args": ["test", "--e2e", "${fileBasename}", "--devtools"],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "disableOptimisticBPs": true,
      "skipFiles": [
          "${workspaceFolder}/node_modules/**/*.js",
          "<node_internals>/**/*.js"
      ],
      "windows": {
        "program": "${workspaceFolder}/node_modules/@stencil/core/bin/stencil"
      }
    }
  ]
}

The other's are helpful convenience formulas that allow you to run only one file at a time. You have to have that file open and that tab has to be in focus in order for it to work. You can also add "--devtools" to the args here if it's helpful.

2. Add a debugger; statement to your test file.

UPDATE: As of 1.3.0 Sourcemap support has been added for test files.

Why not set a breakpoint by tagging a line number? Well As of v1.2.5 of @stencil/core, the typescript compiler isn't using sourcemaps. So the line the code actually breaks at is different than the one you tagged in your file. Sometimes you can game this by moving it higher up and stepping over until you see your watch value populate, but that's annoying.

describe('example', () => {
  it('should render a foo-component', async () => {
    const page = await newE2EPage();
    await page.setContent(`<foo-component></foo-component>`);  
    const el = await page.find('foo-component');
    const isVisible = await el.isVisible();
    // Test execution will stop at this line.
    debugger;
    expect(isVisible).toBe(true);
  });
});

This will allow you to see what the value of el is, for example. Perhaps isVisible is undefined and you want to see if the reference to foo-component was even retrieved properly. This will help you figure out where you're using the test libraries API improperly, or if the methods you're using aren't working the way you expect.

Debugging in Chromium

What if we need to step-debug or see a console.log() from our component code because our test code looks fine, but it's not passing? Puppeteer's page.evaluate() command allows us to execute code from inside the browser context where the component is running.

You can add console.log() and debugger; statements in your component's code, but these won't have any impact if the tests are run in headless mode. More on that below.

Let's use a more complex example. Let's say your component has a button that, when clicked, should hide another element. Your test might look something like this.

it("should hide the content when the close button is clicked", async () => {
    const page = await newE2EPage();

    await page.setContent(`<foo-component></foo-component>`);
    const foo = await page.find("foo-component");

    const closeButton = await page.find(`foo-component >>> .close`);
    const content = await page.find(`foo-component >>> .content`);
    await closeButton.click();

    // Debugging Start
    // page.evaluate( () => { debugger; } );
    // Debugging End

    const isVisible = await content.isVisible();
    expect(isVisible).toBe(false);
  });

But for some reason this isn't working. Let's debug.
1. First let's add the following line (commented out above). page.evaluate( () => { debugger; } ); You can also do page.debugger() - which is a Stencil convenience method that does the same thing.

This adds a debugger; command which will execute in the browser context (instead of node's context).

2. Secondly, we need to modify our test configuration. Normally Puppeteer run's Chromium headless (in the background with no graphics), but we need to change that so we can see the browser and use the devtools.

Simple way

In Stencil v1.0.7, they introduced a flag you can pass to the Stencil test command in the cli called devtools which tells Chromium to run in headed mode, with dev tools turned on and slowed down so a human can see what's happening. It also adjust's Jest's timeout so you have more than 30 seconds to do your debugging before the tests cleanup and close Chromium on you.

$ node_modules/.bin/stencil test --spec --e2e --devtools

Manually

Note: Do this only if Stencil's defaults aren't working for you.
You have to tell Puppeteer to disable headless mode and enable devtools. This allows you to see the browser instance running and inspect the elements and source tabs like you would in Chrome. You'll want to also slow the tests down so you can see them in action. Lastly you'll need to set a timeout in Jest You'll need to adjust some timeouts in Jest to prevent the browser from closing on you mid-debug. Stencil exposes in its testing config browserHeadless, browserDevtools and browserSlowMo. And in your test file you can override Jest's default timeout by doing jest.setTimeout(n);

// stencil.config.ts
import { Config } from "@stencil/core";

export const config: Config = {
   // other stuff
   testing: {
     browserHeadless: false
     browserDevtools: true
     browserSlowMo: 1000 //milliseconds
   }
}

// test file
jest.setTimeout(100000); //milliseconds
/* Put the number at something absurd to give you plenty of time to work. 
 * Just don't leave this in the test once you get it working.
 */

Now you should see the Chromium browser open (should resemble Chrome quite a bit) which has Chrome's dev tools. The debugger you added inside page.evaluate() should pause the test, and you can then step over it and see how your business logic in your component is working (or not working ;D).

Conclusion

1. Using VSCode's debugger, we can step-debug through our test code and find bugs in the test code itself.
2. Use the --devtools flag when running the Stencil tests to disable headless mode. This let's us set debugger; statements in our component code tell puppeteer to pause. I find the latter is useful if you want to be able to see your component rendered, which sometimes helps you spot visually what's not rendering properly. The former is better if you need to inspect your component code itself, like verifying some state has the correct value.

From there it's a matter of learning the Puppeteer and Jest APIs (as well as Stencil's add-ons) and finding the right one for what you need. One thing that took me a bit of time to figure out was how to run some custom code on a page to do DOM manipulations. page.evaluate() is good for that too - and there are a couple other methods on the page class that can help with that.

Don't forget that almost all of Puppeteer's functions return a promise, so pretty much every method needs await in front of it if you need things to run synchronously.

The Stencil team is updating thing's pretty frequently. So keep an eye on their Changelog to see if anything has changed with their testing setup (that's how I found the --devtools flag :) ).

"But test everything. Keep what is good." 1 Thes. 5:21

References

Prerequisites

Stencil.js

Javascript Test Fundamentals

VSCode Debugger

Testing APIs

Puppeteer API

Page Class
Element Handle Class
Debugging Puppeteer

Jest API

Core
Expect
Debugging Tips

Stencil Testing

Overview

Typescript Definitions Files
The comments in these files serve as documentation in lieu of any official docs.

Stencil Testing Config

Stencil Puppeteer Declarations

Comments

[Brian]

Excellent article, worked a charm👌

I've was originally following the debugging steps from the docs which I could not get to work

[JV Lobo]

This doesn't seem to be working anymore?
I run my test and even if I have the debugger; statement and the timeout, the code never stops. The browser closes right away :(