Functional tests

Introduction

OpenRefine's web interface is tested with the Cypress framework. With Cypress, tests are performing assertions using a real browser, the same way a real user would use the software. Those are "end to end" (e2e) tests because they also rely on OpenRefine's backend (server).

Cypress tests can be ran

• using the Cypress test runner (development mode)
• using a command line (CI/CD mode)

If you are writing tests, the Cypress test runner is good enough, and the command-line is mainly used by the CI/CD platform (Github actions)

Cypress brief overview

Cypress operates insides a browser, it's internally using NodeJS. That's a key difference with tools such as Selenium.

From the Cypress documentation:

But what this also means is that your test code is being evaluated inside the browser. Test code is not evaluated in Node, or any other server side language. The only language we will ever support is the language of the web: JavaScript.

Good starting points with Cypress are the Getting started guide, and the Trade-offs

The general workflow of a Cypress test is to

• Start a browser (yarn run cypress open)
• Visit a URL
• Trigger user actions
• Assert that the DOM contains expected texts and elements using selectors

Getting started

If this is the first time you use Cypress, it is recommended for you to get familiar with the tool.

• Cypress overview
• Cypress examples of tests and syntax

1. Install Cypress

You will need:

• Node.js 10 or 12 and above
• Yarn or NPM
• A Unix/Linux shell environment or the Windows command line

To install Cypress and dependencies, run :

cd ./main/tests/cypress
yarn install

2. Start the test runner

The test runner assumes that OpenRefine is up and running on the local machine, the tests themselves do not launch OpenRefine, nor restarts it.

Start OpenRefine with

./refine

Then start Cypress

yarn --cwd ./main/tests/cypress run cypress open

3. Run the existing tests

Once the test runner is up, you can choose to run one or several tests by selecting them from the interface.
Click on one of them and the test will start.

4. Add your first test

• Add a test.spec.js into the main/tests/cypress/cypress/e2e folder.
• The test is instantly available in the list
• Click on the test
• Start to add some code

Tests technical documentation

A typical test

A typical OpenRefine test starts with the following code

it('Ensure cells are blanked down', function () {
  cy.loadAndVisitProject('food.mini');
  cy.get('.viewpanel-sorting a').contains('Sort').click();
  cy.get('.viewpanel').should('to.contain', 'Something');
});

The first noticeable thing about a test is the description (Ensure cells are blanked down), which describes what the test is doing.
Lines usually starts with cy.something..., which is the main way to interact with the Cypress framework.

A few examples:

• cy.get('a.my-class') will retrieve the <a class="my-class" /> element
• cy.click() will click on the element
• eventually, cy.should() will perform an assertion, for example that the element contains an expected text with cy.should('to.contains', 'my text')

On top of that, OpenRefine contributors have added some functions for common OpenRefine interactions. For example

• cy.loadAndVisitProject will create a fresh project in OpenRefine
• cy.assertCellEquals will ensure that a cell contains a given value

See below on the dedicated section 'Testing utilities'

Testing guidelines

• cy.wait should be used in the last resort scenario. It's considered a bad practice, though sometimes there is no other choice
• Tests should remain isolated from each other. It's best to try one feature at the time
• A test should always start with a fresh project
• The name of the files should mirror the OpenRefine UI organization

Testing utilities

OpenRefine contributors have added some utility methods on the top of the Cypress framework. Those methods perform some common actions or assertions on OpenRefine, to avoid code duplication.

Utilities can be found in cypress/support/commands.js.

The most important utility method is loadAndVisitProject.
This method will create a fresh OpenRefine project based on a dataset given as a parameter.
The fixture parameter can be

• An arbitrary array, the first row is for the column names, other rows are for the values
  Use an arbitrary array only if the test requires some specific grid values
  Example:

  const fixture = [
    ['Column A', 'Column B', 'Column C'],
    ['0A', '0B', '0C'],
    ['1A', '1B', '1C'],
    ['2A', '2B', '2C'],
  ];
  cy.loadAndVisitProject(fixture);

• A referenced dataset: food.small or food.mini
  Most of the time, tests does not require any specific grid values
  Use food.mini as much as possible, it loads 2 rows and very few columns in the grid
  Use food.small if the test requires a few hundred rows in the grid

  Those datasets live in cypress/fixtures

Browsers

In terms of browsers, Cypress is using what is installed on your operating system. See the Cypress documentation for a list of supported browsers

Folder organization

Tests are located in main/tests/cypress/cypress folder. The test should not use any file outside the cypress folder.

• /fixtures contains CSVs and OpenRefine project files used by the tests
• /integration contains the tests
• /plugins contains custom plugins for the OR project
• /screenshots and /videos contains the recording of the tests, Git ignored
• /support is a custom library of assertion and common user actions, to avoid code duplication in the tests themselves

Configuration

Cypress execution can be configured with environment variables, they can be declared at the OS level, or when running the test

Available variables are

• OPENREFINE_URL, determine on which scheme://url:port to access OpenRefine, default to http://localhost:333
• DISABLE_PROJECT_CLEANUP, If set to 1, projects will not be deleted after each run. Default to 0 to keep the OpenRefine instance clean

Cypress contains exaustive documentation about configuration, but here are two simple ways to configure the execution of the tests:

Overriding with a cypress.env.json file

This file is ignored by Git, and you can use it to configure Cypress locally

Command-line

You can pass variables at the command-line level

yarn --cwd ./main/tests/cypress run cypress open --env OPENREFINE_URL="http://localhost:1234"

Visual testing

Tests generally ensure application behavior by making assertions against the DOM, to ensure specific texts or css attributes are present in the document body.
Visual testing, on the contrary, is a way to test applications by comparing images. A reference screenshot is taken the first time the test runs, and subsequent runs will compare a new screenshot against the reference, at the pixel level.

Here is an introduction to visual testing by Cypress.

In some cases, we are using visual testing.
We are using Cypress Image Snapshot

Identified cases are so far:

• testing images created by OpenRefine backend (scatterplots for example)

Reference screenshots (Called snapshots), are stored in /cypress/snapshots. And a snapshot can be taken for the whole page, or just a single part of the page.

When a visual test fails

First, Cypress will display the following error message:

Then, a diff image will be created in /cypress/snapshots, this directory is ignored by Git.
The diff images shows the reference image on the left, the image that was taken during the test run on the right, and the diff in the middle.

CI/CD

In CI/CD, tests are run headless, with the following command-line

./refine e2e_tests

Results are displayed in the standard output

Resources

Cypress command line options Lots of good Cypress examples