Introducing the TestRail CLI
Send Automated Test Results to TestRail from Anywhere
Check out this recording of the live webinar “TestRail CLI”, where Solution Architect Diogo Rede shows how the new CLI feature lets you focus on writing test code instead of needing to write, test and maintain scripts to upload results.
By making it easy for you to parse and upload JUnit style test results into TestRail directly from the command line, the TestRail CLI makes centralized tracking of manual and automated testing faster and easier than ever.
Watch the full replay above, or read the highlights below.
What’s the TestRail CLI?
Before TestRail CLI, if you wanted to report the results from your test automation in TestRail, you would have to script a method to parse the results and upload them to TestRail with API calls written into your automation frameworks.
Now, as long as you’re using a test automation framework that outputs a JUnit-style XML report or another report style that you can easily convert to a JUnit-style format, you can automatically upload those to TestRail with the TestRail CLI.
Strictly speaking, a command-line interface (CLI) is a text-based user interface (UI) used to run programs, manage computer files and interact with the computer. The TestRail CLI provides a programming language agnostic mechanism that converts one of the most common test result formats—JUnit-style XML files—into a set of TestRail test results, automatically create a new test run and upload the results to TestRail, and even generate new test cases in TestRail if you hadn’t copied them from your automation framework to TestRail yet.
The TestRail CLI is an open source project, which allows users to collaborate on making improvements and sharing them with the rest of the TestRail community.
Github repository: https://github.com/gurock/trcli
What can you do with TestRail CLI?
Use a single command in the terminal to:
- Upload Jumit-style test automation results to TestRail
- Automatically generate new test cases if aren’t already documented in TestRail
- Report on results from automation frameworks such as: JUnit / TestNG / Cypress.io / Playwright / Mocha / Robot Framework / Pytest / NUnit / etc
Time saving and all your testing efforts in one place:
- Abstract the TestRail API
- Reduce maintenance effort
- Increase focus on test automation
- Track test progress
- Measure quality metrics
TestRail CLI (“trcli”) Functionality
How does the trcli map test results onto existing test cases? Simply matches the title?
The TestRail CLI maps your JUnit test cases to TestRail cases through the automation_id custom field by using a classname.name pattern, i.e., for <testcase classname=”tests.LoginTests” name=”test_case_1″ time=”159″></testcase> the automation_id is set to test.LoginTests.test_case_1.
If you’d like to use the trcli with test cases you already have in your TestRail test case repository, manually update the `automation_id` as described in the trcli docs with the classname.name pattern for the test as it’s written in your automation before running the trcli.
I have quite a long list of test cases already in TestRail. Does CLI automatically link my automation tests in the automatically created testrun to the test cases I have if I include the C12345 testcase ID for example in the test name ? i.e. it(‘C12345 should do this and that…’) in Cypress
As of now, the TestRail CLI can only map your JUnit test cases to TestRail cases through the `automation_id` custom field. This means you should add the JUnit report test case paths to your automation_id custom field for each test case using the classname.name pattern, i.e., for <testcase classname=”tests.LoginTests” name=”test_case_1″ time=”159″></testcase> you would set the automation_id to test.LoginTests.test_case_1 for the test case it would map to. The TestRail CLI won’t update any test cases based on the TestRail Case ID.
Can we manually override the test results after the results are exported to TestRail?
Yes! When you upload results via the trcli, it will leave the test run open by default. That way you can review the results in TestRail first, and manually update them, add comments or attachments, and create / link defects from inside TestRail as needed.
If you close the test run, however, you won’t be able to edit the test results afterwards. You can specify the –close-run option to do this, if desired.
Can you upload results to existing test cases in multiple suites?
Yes, even if you have a project with multiple suites, the automation_id on each test case should be unique, so the TestRail CLI will report the results for the correct test cases. If you want to auto-generate your test cases in a specific suite, you should use the –suite-id option.
Instead of creating a new test run for each report, is it possible to update an existing test run via CLI results? If I have to rerun failed tests in automation, how can I ensure it updates the same test run? Due to performance issues we may have to run our full suite multiple times to iron out all the issues.
By simply using the –run-id option you can specify you want to update an existing test run with a given id, instead of creating a new one.
Sometimes we create test runs with both manual and automated tests in them. Is it possible to run some tests manually and update the other half via automation?
By simply using the –run-id option you can specify you want to update a test run with a given id, instead of creating a new one. This will only update the tests with matching automation_id values in that test run, and won’t affect the tests you’ve queued to run manually in that run!
How to upload multiple test results in the same script? If I want to upload results with priority ‘high’ and ‘critical’, can I do that with the same script?
You can simply execute the TestRail CLI twice on the same script. Please note that this assumes you have two JUnit report files, so you can specify different values for each of them.
Are individual test step results supported by the CLI?
That’s a great question! At this time, the CLI doesn’t support individual test steps, but that’s definitely something we could look into in the future. Would you mind checking if there are any similar requests on the TestRail CLI GitHub issues page, and if not, submit one describing your use case? That way the rest of the community can weigh in as well and even possibly contribute to the project to add the functionality!
Is there a way to send attachments with test cases (like an execution log file) to a test run and then view them in TestRail?
That’s a great question! At this time, the CLI doesn’t support attachments, but that’s definitely something we could look into in the future. Would you mind checking if there are any similar requests on the TestRail CLI GitHub issues page, and if not, submit one describing your use case? That way the rest of the community can weigh in as well and even possibly contribute to the project to add the functionality!
What are the best practices to handle multiple environment (DEV, QA, PROD) executions?
As of now, we only support specifying the Test Run name and description, so we recommend stating the name of the environment on the Test Run name. If you’d like to see the functionality of updating custom test results fields in the future, please check if there are any similar requests on the TestRail CLI GitHub issues page, and if not, submit one describing your use case? That way the rest of the community can weigh in as well and even possibly contribute to the project to add the functionality!
Is test case versioning possible? (Can we pull up an older version of a test case)
At this time, the CLI does not support anything related to test case versioning. That said, if you want to change a test case, you can update that in TestRail—simply make sure the automation_id still matches your test as it’s written in your automation suite. And if you have TestRail Enterprise, you can review and revert to previous versions with explicit test case versioning—just need to make sure the `automation_id` field matches the way the test is written in your automation project.
Installation & Supported Platforms
How do I install the TestRail CLI (trcli)? Can this be installed via HomeBrew on MacOS?
You can install the TestRail CLI using the python-based package installer, `pip`.
What test automation platforms can I use with the trcli?
As long as your test framework can generate a JUnit-style report, you will be able to use the TestRail CLI to upload your test results.
Can we use the trcli in CircleCi?
Yes! Simply add it as a step in your CircleCI builds.
Can we use the trcli for Cypress?
Yes! Just upload the JUnit XML generated by Cypress as described in this tutorial: https://www.gurock.com/testrail/docs/integrate/testrail-test-automation/integrate-testrail-with-cypress/.
Using the tool currently and it’s great so far! – We’re using pytest to generate the xml report. When generating this xml I can choose to have the stdout included ( which includes test steps etc..) however there’s currently no mapping for sending the stdout to TestRail. Is this something that could be supported in the future perhaps? Thanks!
That’s a great question! At this time, the CLI doesn’t support adding the stdout to the test results, but that’s definitely something we could look into in the future. Would you mind checking if there are any similar requests on the TestRail CLI GitHub issues page, and if not, submit one describing your use case? That way the rest of the community can weigh in as well and even possibly contribute to the project to add the functionality!
Currently the way WebdriverIO generates junit XML creates a pretty noisy Automation ID, here’s how it looks like: chrome.<version>.macosx.<classname>.<name>. Obviously the “chrome.<CHROME_VERSION>.<OS_VERSION>” becomes a problem since it’s generated on the client side and is going to vary depending on platform and configuration. Please advise.
That’s a good point. The CLI isn’t particularly well-suited to handle uploads from WebdriverIO JUnit XML reports yet for that reason—we totally agree with you. This is definitely something we could see being valuable to fix in the CLI project: could you submit an issue on the TestRail CLI GitHub issues page describing your use case and ultimately how you’d like the CLI to handle WebdriverIO reports? That way the rest of the community can weigh in as well and even possibly contribute to the project to add the functionality!
Does the TestRail CLI work if used Java with BDD Framework in Selenium?
If your Java project is using a framework that supports JUnit reports, then the answer is yes! So, i.e., if you’re executing your Cucumber tests through the JUnit5 runner, you can make use of the Surefire reporter, which generates JUnit-style reports.
Puppeteer is a high-level API to control Chrome or Chromium over the DevTools Protocol. It all comes down to the actual test framework.
If you use mocha, it provides JUnit-style reports out of the box. If you use jasmine or jest, you can probably use plugins such as the ones below to generate the JUnit report.
Can we integrate TestRail with tools like ADO (Azure DevOps)?
Yes! You can run the TestRail CLI as part of a build pipeline in ADO, and integrate TestRail with ADO Work Items if you use those to track bugs/defects.
Is it also easy to integrate in GitHub Actions?
The TestRail CLI is a Python application, so you will need to have Python installed on the GitHub executor so you can install and execute the CLI, which you can do in GitHub Actions!
Does TestRail CLI work with TestNG? Will you be having documentation for TestNG?
TestNG creates JUnit based reports with some extras which can definitely be used to import test results using the TestRail CLI. You can check this article for more information on how to setup your TestNG project to work with the TestRail CLI:
Does Test CLI support Cypress with Cucumber?
Yes! As long as you’re exporting your BDD test run in Cypress to a JUnit XML, you can upload the results to TestRail with the CLI.
Does Test CLI support Extent reports?
(assuming this refers to extentreports.com) The TestRail CLI was designed to import JUnit test results to TestRail, it does not integrate with other vendor tools. However, if Extent can export results to JUnit style, then you can use the CLI.
You could use the API directly, but if you can generate JUnit reports on your cypress project, we would recommend giving the TestRail CLI a go first!
Check out these other useful resources to learn more:
⚡TestRail CLI Documentation
⚡How to Integrate TestRail with JUnit5
⚡How to Integrate TestRail with NUnit
⚡How to Integrate TestRail with Robot Framework