Use Playwright and AskUI Together

Playwright is the de facto standard when it comes to reliably testing browser applications across browsers and operating systems.

But what if your workflow under test leaves your browser and requires you to automate on a desktop application?

AskUI is not confined to a specific application because it uses visual selectors and real human user-like input with keypresses and natural mouse movement. This makes it an ideal tool to accompany Playwright.

In this post you will learn to setup Playwright together with AskUI to define workflows that can leave the browser.

Prerequisites

• AskUI installed and configured on your system (Windows, Linux, macOS)

Install Playwright

Playwright for JavaScript/TypeScript consists of two components.

1. The playwright node package
2. A browser installed through Playwright library

First you have to install playwright as a dev-dependency. Execute the following command in your terminal:

npm install --save-dev playwright

For this blog you will use Chromium as a browser. The installation through playwright is done like this:

npx playwright install --with-deps chromium

Configure Playwright

You will configure Playwright and the Chromium browser in the file helpers/askui-helper.ts, so it will work correctly together with Jest.

import { UiControlClient, UiController } from 'askui';
import { AskUIAllureStepReporter } from '@askui/askui-reporters';

// Server for controlling the operating system
let uiController: UiController;

// Client is necessary to use the askui API
let aui: UiControlClient;

// 1. Playwright setup
import { chromium, devices, Browser, BrowserContext } from 'playwright';
let pwBrowser: Browser;
let pwContext: BrowserContext;

jest.setTimeout(60 * 1000 * 60);

beforeAll(async () => {

  // 2. Playwright setup
  pwBrowser = await chromium.launch({ headless: false, slowMo: 100 });
  pwContext = await pwBrowser.newContext(devices['Desktop Chrome']);

  uiController = new UiController({
    /**
      * Select the display you want to run your tests on, display 0 is your main display;
      * ignore if you have only one display
      */
    display: 0,
  });

  await uiController.start();

  aui = await UiControlClient.build({
    credentials: {
      workspaceId: '<your-workspace-id>',
      token: '<your-access-token>',
    },
    reporter: new AskUIAllureStepReporter(),
  });

  await aui.connect();
});

...

afterAll(async () => {

  // 3. Playwright setup
  await pwContext.close();
  await pwBrowser.close();

  aui.disconnect();

  await uiController.stop();
});

export { aui };

// 4. Playwright setup
export { pwContext };

1. Import the necessary classes from playwright and define the variables
2. Configure chromium to be not headless and also slow down its execution. Also we want a Desktop Chrome as we are on desktop.
3. Close the BrowserContext and the Browser after the execution finished.
4. Export the Context so you can use it in your workflow

Write a Workflow

With the BrowserContext set up, hop over to the file my-first-askui-test-seuite.test.ts and import it there. Add this to the start of the file:

import { pwContext } from './helpers/askui-helper';

Now you can implement a simple workflow:

1. Open a new tab
2. Navigate to https://google.com/
3. Click away the cookie banner
4. Fill the search field with Playwright framework
5. Press Enter on keyboard
6. Switch to image search by clicking on the Button architecture
7. Move the mouse cursor to the element that has an image with a red and a green mask in it (Playwright logo description)

describe('jest with askui', () => {
  it('should open askui.com in Chrome with playwright', async () => {
    let page = await pwContext.newPage();
    await page.goto('https://google.com/');
    await page.getByRole('button', { name: 'Reject all' }).click();
    await page.locator('//textarea[@name="q"]').fill('Playwright framework');
    await page.keyboard.press('Enter');
    await page.waitForTimeout(1000);

    // AskUI workflow
    // Make sure you have the focus on the browser
    await aui.mouseLeftClick().exec();
    await aui.click().button().withText('Images').exec();
    await aui.click().text('architecture').exec();
    await aui.moveMouseTo().element().matching('image with a red and a green mask in it').exec()

    await page.waitForTimeout(5000);
  });
});

Testrun

To run this you need to run this command in your terminal:

// Linux + macOS
npm run askui

// Windows
AskUI-RunProject

The following gif shows the execution of the whole workflow:

Conclusion

Combining AskUI with Playwright enhances automation capabilities to extend beyond web browsers. While Playwright is a solid choice for browser automation, AskUI takes the reins when dealing with desktop apps and operating systems.

This integration offers a practical toolkit for workflows across various applications for efficient automation.