Integrate TestRail with Cypress - TestRail

Integrate TestRail with Cypress

Cypress is a very popular next generation front end testing tool built for the modern web. It can be used for unit tests, integration and even end-to-end, packed with productivity features such automatic waiting, time travel, real time reloads and debugging tools.

In this tutorial, we will explore how to integrate the test results generated by a Cypress automated test run within TestRail using the TestRail CLI. This will allow you to centralize your automated test results and take advantage of all the analytical and reporting capabilities TestRail provides.

On this page:

    Overview

    Throughout this tutorial, we will be using a sample project to guide you through the process of setting up a Cypress automated tests project that is compatible with the TestRail CLI tool and uploading the generated test results.

    After reading through this tutorial, you will be able to:

    ✔ Execute tests from a simple Cypress project 

    ✔ Install the TestRail CLI Tool

    ✔ Configure your TestRail instance

    ✔ Run the CLI Tool 

    ✔ See your test cases and test results on TestRail

    Prerequisites

    To be able to run a Cypress tests project you can install it through npm or as a standalone, as per the installation docs, but for the purpose of this article we will be installing through npm. It also has its advantages such as Cypress being versioned like any other dependency and it also simplifies running Cypress in Continuous Integration. As a pre-requisite to install Cypress using npm all you need to do is install Node.js.

    You will also need Python to install and run the TestRail CLI Tool that will be used to import the test results to TestRail.

     

     

    Prerequisite Description
    Node.js Download the version for your operating system and follow the install wizard instructions.
    To make sure the install was successful, try executing the commands node –version and npm –version from your command line and they should output the installed version.
    Python 3.10.4 Download the version for your operating system and follow the install wizard instructions.

    To make sure the install was successful, try executing the commands python –version and pip –version from your command line and they should output their versions.

    Installing the sample project

    Let’s start by fetching the sample project code and installing the required dependencies.

    1. Clone or download the sample project
    2. Open your command prompt on the project root folder and execute the command below

     

    $ npm install

    The packages.json file contains the libraries we want to install for this project. In this case the only dependency is Cypress itself, which naturally depends on other libraries.

    Requirement Description
    cypress Test automation framework with web automation capabilities

    Exploring the sample project

    Use your favorite IDE to open the sample project and start digging into the test files. We’ve kept the automated tests code simple so that the focus of this tutorial remains on how to import the execution results. These tests consist of simple interactions with a TO DO list web app, hosted by the cypress team itself.

    Cypress test code

     

    Executing the sample project

    On the same command prompt, execute the following command to run the Cypress tests in the project and save the results in JUnit XML format.

     

    $ run --reporter junit --reporter-options "mochaFile=reports/TEST-[hash].xml"

    If the cypress execution was successful, you should be able to see your test results on the reports folder. It should contain multiple XML files which are JUnit style reports for each test suite. Notice that the report files all start with TEST-, which is according to the command we executed, so that we can later merge all the reports into one. This one file will be parsed by the TestRail CLI Tool to create the test run and upload your test results to TestRail on the next step. Below you can see an example of a partial report.

    <?xml version="1.0" encoding="UTF-8"?>
    <testsuites name="Mocha Tests" time="1.4800" tests="5" failures="1">
      <testsuite name="Root Suite" timestamp="2022-06-22T13:40:26" tests="0" file="cypress/integration/1-getting-started/todo_interactions.spec.js" time="0.0000" failures="0">
      </testsuite>
      <testsuite name="[interactions] example to-do app" timestamp="2022-06-22T13:40:26" tests="2" time="0.8810" failures="0">
        <testcase name="[interactions] example to-do app can add new todo items" time="0.6530" classname="can add new todo items">
        </testcase>
        <testcase name="[interactions] example to-do app can check off an item as completed" time="0.2280" classname="can check off an item as completed">
        </testcase>
        </testsuite>
        <testsuite name="with a checked task" timestamp="2022-06-22T13:40:28" tests="3" time="0.5990" failures="1">
        <testcase name="[interactions] example to-do app with a checked task can filter for uncompleted tasks" time="0.3060" classname="can filter for uncompleted tasks">
        </testcase>
        <testcase name="[interactions] example to-do app with a checked task can filter for completed tasks" time="0.2930" classname="can filter for completed tasks">
        </testcase>
        <testcase name="[interactions] example to-do app with a checked task can delete all completed tasks" time="0.0000" classname="can delete all completed tasks">
          <failure message="Timed out retrying after 4000ms: Not enough elements found. Found &apos;1&apos;, expected &apos;2&apos;." type="AssertionError">
            <![CDATA[AssertionError: Timed out retrying after 4000ms: Not enough elements found. Found '1', expected '2'.
            at Context.eval (https://example.cypress.io/__cypress/tests?p=cypress\integration\1-getting-started\todo_interactions.spec.js:186:31)]]>
          </failure>
        </testcase>
      </testsuite>
    </testsuites>

    Please Note: Do not create test files with describe blocks with the same name since that will cause name collisions and the import may not work as expected. Notice we added a prefix (i.e.: [interactions]) to the describe blocks on our project so that this doesn’t happen.

    Importing results to TestRail

    After the tests have been executed and the JUnit report files are generated, you can easily import your test results (and test cases!) to TestRail. This will bring visibility to your automated test runs and will enable you to look at the big picture of how you’re actually testing your app all within TestRail.

    Installing the TestRail CLI

    Given you already have Python installed on your machine, installing the TestRail CLI is as simple as executing the following command on your command line.

    $ pip install trcli

    Configuring TestRail

    Secondly, you need to configure your TestRail instance according to the instructions below.

    1. Enable the TestRail API by accessing the Administration panel, going to Site Settings, click on the API tab, and checking the Enable API option.
    2. Create a custom field in order to map your automated test cases code to the actual TestRail cases. You can do so by accessing the Administration panel, going to Customizations on the right side and clicking Add Field. After you’ve reached the field creation screen, there are two requirements for this custom field:
      1. The System Name must be automation_id 
      2. The Type must be String. 

    Sending results to TestRail

    After you’re done installing the TestRail CLI Tool and finished configuring your TestRail instance, there’s only one extra step before uploading your results. Since we have multiple JUnit XML files, we first need to merge them into one file which can then be parsed by the TestRail CLI Tool. To merge the files, use the command below.

     

    $ junitparser merge \
    >    --glob "reports/TEST-*" \
    >    "reports/junit-report.xml"

    Finally, you can upload your test results by simply using a one-liner such as the one below.

     

    $ trcli -y \
    >    -h https://INSERT-INSTANCE-NAME.testrail.io \
    >    --project "My Project" \
    >    --title "Cypress Automated Test Run" \
    >    --username INSERT-EMAIL \
    >    --password INSERT-PASSWORD \
    >    parse_junit \
    >    "./reports/junit-report.xml"

    Note that the file name after the -f option should match the path to your report file in case you change its default location. All others options should be according to your TestRail instance and project. You can check other command line options by checking the README.md file on the project repository, the TRCLI documentation article, or the embedded CLI help through the commands below.

    $ trcli --help
    $ trcli parse_junit --help

    Visualizing the results on TestRail

    Now, if you go to the Test Cases page in your TestRail project, you’ll notice that the TestRail CLI tool automatically created the test cases that were on your test results report. Notice that it added a unique Automation ID by combining the classname and name attributes on each test on the JUnit report. This Automation ID is used to map the tests on your automation code base to the test cases on TestRail. This means that each time you run the TestRail CLI tool, it first attempts to match an existing test case on TestRail and only creates a new one in case there is no test case with that Automation ID.

    Please Note: If you change the name of your tests, the name of the file or its location, the Automation ID for those tests will change and they won’t be mapped to the existing test cases on TestRail.

    TestRail test cases

    On the Test Runs & Results page, we can see that a test run with the name Cypress Automated Test Run was created. By opening it we can dig further into the details of each automated test result and perform a high level analysis of why any test is failing since the error message provided by the test automation framework is also registered on the test result, as you can see on the image below.

    TestRail test results

    What next?

    Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results! 

    You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.