Workbench/backend/target/backend-1/handsontable/visual-tests/README.md

Handsontable visual testing

To avoid unintended changes to Handsontable's UI, we use visual regression testing.

Overview

We run visual tests automatically by using the following tools:

Tool	Description
Playwright	An open-source testing framework backed by Microsoft. We use it to write and run visual tests.
Argos	An external visual testing service. We use it to compare screenshots.
GitHub Actions	GitHub's CI platform. We use it to automate our test workflows.

When you push changes to a GitHub pull request:

1. The Visual tests linter workflow checks the code of each visual test.
2. The Tests workflow runs all of Handsontable's tests.
3. After all tests pass successfully, the Visual tests job runs the visual tests and uploads the resulting screenshots to Argos.
4. Argos compares your feature branch screenshots against the reference branch (develop) screenshots (so-called "reference", "baseline" or "golden" screenshots).

If Argos spots differences between two corresponding screenshots, the Visual tests check on on your pull request fails, and you can't merge your changes to develop. In that case:

1. Open the log of the Visual tests job:
   At the bottom of your pull request, find the Visual tests check. Select Details.
2. Open the Argos URL and review the differences. You can:
   • Reject the modified screenshots, update your code, and re-run the visual tests.
   • Accept the modified screenshots. You can then merge your changes to develop. As a result, the modified screenshots become the new baseline.

Visual tests structure

Visual tests are divided into:

• multi-frameworks: tests run on Chromium using classic, horizon, horizon-dark, main and main-dark themes against Handsontable instance created in:
  • Vanilla JS
  • Angular
  • React
  • React (functional)
  • Vue 2
  • Vue 3
• cross-browser: tests run against vanilla JS Handsontable instance using:
  • Chromium
  • Firefox
  • Webkit

There is a separate Playwright config for cross-browser tests: playwright-cross-browser.config.ts

Visual tests demos

All the test examples are available at examples/next/visual-tests and configured to be served from localhost:8082

There main demo available for all frameworks is served on /. There are additional demos available only for vanilla JS (to be used with cross-browser tests):

• /cell-types-demo,
• /arabic-rtl-demo,
• /custom-style-demo,
• /merged-cells-demo,
• /nested-headers-demo,
• /nested-rows-demo,

Run visual tests through GitHub Actions

Our GitHub Actions configuration runs the visual tests automatically, but you can run them manually as well:

1. On GitHub, at the bottom of your pull request, find the Visual tests check. Select Details.
2. On the left, next to the Visual tests job, select 🔄.
3. Select Re-run jobs.

Run visual tests locally

You can manually run visual tests on your machine and then upload the resulting screenshots to Argos.

First, prepare your local visual testing environment:

1. Make sure you're using the Node and npm versions mentioned here.
2. From the ./visual-tests/ directory, run npm install.
3. In the ./visual-tests/ directory, create a file called .env. In the file, add the Argos token:

   ARGOS_TOKEN=xxx
   

   Ask your supervisor about the token's value.

To run the visual tests locally:

1. From the ./visual-tests/ directory, run one of the following commands:

   Command	Action
   npm run test	Run multi-framework visual tests, for all the configured frameworks, using Chromium only.
   npm run test:cross-browser	Run cross-browser visual tests, using vanilla JS framework, for all the supported browsers. You can pass the test name to run a single cross-browser test: npm run test:cross-browser borders
   npx playwright test {{ file name }}	Run a specific test. For example: npx playwright test mouse-wheel

   The resulting screenshots are saved in ./visual-tests/screenshots/.

2. From the ./visual-tests/ directory, run npm run upload.

3. Open the Argos URL displayed in the terminal.

Write a new visual test

To add a new visual test:

1. On your machine, in the ./visual-tests/tests/ directory, create a new .spec.ts file.
   Give your file a descriptive name. This name is later used in test logs and screenshot names.
   • ✅ Good: open-dropdown-menu.spec.ts.
   • ❌ Bad: my-test-1.spec.ts.
2. Copy the template code from ./visual-tests/tests/.empty-test-template.ts into your file.
3. Write your test. For more information, see:
   • Playwright's docs
   • Helpers
   • Take screenshots
4. Push your changes to a pull request.
   The Visual tests linter workflow checks the code of your test.

Take screenshots

To capture a screenshot and save it to a file, add this line anywhere in your test:

await page.screenshot({ path: helpers.screenshotPath() });

In each test, you can take as many screenshots as you want. For example:

await cell.click();
await page.screenshot({ path: helpers.screenshotPath() });
await anotherCell.click();
await page.screenshot({ path: helpers.screenshotPath() });

To take a screenshot of a specific element of Handsontable, use Playwright's locator() method. For example:

const dropdownMenu = page.locator(helpers.selectors.dropdownMenu);

await dropdownMenu.screenshot({ path: helpers.screenshotPath() });

For cross-browser tests we are using

  await page.screenshot({ path: helpers.screenshotMultiUrlPath(testFileName, url, suffix) });

for easier screenshot identification.

Helpers

To write tests faster, use the custom helper functions and variables stored in the ./visual-tests/src/helpers.ts file.

modifier

Returns the current modifier key: Ctrl for Windows or Meta for Mac.

// copy the contents of the selected cell
await page.keyboard.press(`${helpers.modifier}+c`);

isMac

Returns true if the test runs on Mac.

if (helpers.isMac) {
  // do something
}

findCell()

Returns the specified cell.

Syntax: findCell({ row: number, cell: number, cellType: 'td / th' }).

const cell = helpers.tbody.locator(helpers.findCell({ row: 2, cell: 2, cellType: 'td' }));

await cell.click();

findDropdownMenuExpander()

Returns the button that expands the dropdown menu (also known as column menu) of the specified column.

Syntax: findDropdownMenuExpander({ col: number }).

// select the column menu button of the second column
const changeTypeButton = table.locator(helpers.findDropdownMenuExpander({ col: 2 }));

await changeTypeButton.click();