Test Reporting AMA
with TestRail Solution Architect Diogo Rede
Test reporting is an activity that, when done right, can bring great value to your organization. Through reporting, you can visualize your test data using charts and tables that highlight certain data. This can provide insights on your test coverage, test progress, and application health.
In this webinar, Diogo Rede, TestRail Solution Architect, answers questions about test reporting and how to solve some problems using TestRail along with his guests Shanu Mandot and Matt Caponigro.
Watch the full replay above, or read the highlights below.
Is requirements traceability still relevant in modern testing?
Diogo: Imagine that you start working with a new team in your organization. How do you know what requirements are being tested? How do you know what needs more test coverage? So my answer to this is: Yes, definitely. It is still relevant, whether you’re working with an Agile or Scrum team, or whether you’re still doing waterfall.
There are a lot of different scenarios. For instance, imagine that you are outsourcing your testing to a different company. You would like to know if they’re testing all the requirements. So, they will also have to do some sort of traceability there. There are also things like regulated industries where this is actually mandatory. And we can solve this problem on TestRail by referencing requirements in our test cases.
Matt: When you’re developing with Agile methodology, you don’t have time to be as detailed as folks used to be in waterfall style development. But it’s not just about the time pressure itself. Teams have found that, a lot of times, the level of detail they were reporting isn’t providing as much value as people thought it did, right?
A traceability report itself doesn’t tell you whether a test is actually sufficiently covering the requirements. It doesn’t tell you anything about the quality of the test. But there are different ways I see traceability still being important. It’s not about proving the value of the release, or helping to judge quality, it’s more about providing auditability for compliance.
“The trick here is finding a way to maintain traceability without the overhead that traceability used to require”
A current TestRail user sent me what they used to do in Excel a couple of months ago before they switched to TestRail. It took three hours every Monday afternoon for the test lead to update this Excel doc with all the current requirement numbers, the current tests, any defects. And they had to do it all by hand previously.
I think the trick for Requirements Traceability in modern testing is figuring out how to do enough that it provides the value along with the accountability or the auditability.
Shanu: I think it’s still relevant, but it highly depends on the type of industry you are in. I worked in healthcare, banking and finance and retail hospitality. Traceability was not that important in all of them. I mean, the type of requirement traceability we do now has changed a bit. There are tools and automated ways to still cover this requirement. But again, it depends heavily on the domain of the organization.
“With the Agile methodology, it’s not wise to have requirement traceability in Excel format, which is very time-consuming and prone to errors.”
How can I create meaningful reports to track sprint progress?
Diogo: Usually in a sprint, you have two sorts of activities for testers: you have test case specification -or design- and you have test execution. For test design, if you want to make sure you’re testing every user story, you can add the references on your tests. And then you can generate a Case Coverage report. That will give you the visibility between the tests you are writing and the user stories so you can track the progress of your test design work during a sprint.
Then you can also create the results reports for further requirements as well, where you will probably want to have a Milestone to track the sprint. In a Milestone, you can have your test plans, test runs, and you can track those back to the Jira sprint user stories. You can also add things like regression test runs, regression test plans. And then you can track them against the references from your sprint. A reference here could be an issue, a user story, a bug. Reference is a generic word we use for that.
In TestRail, a Reference can be an issue, a user story, a bug, a URL, an ID. You can use the References field on test cases, runs, plans, and milestones to link requirements that your team needs to test.
Matt: Ultimately, the references field in TestRail is going to give you a good way to link the test you’re writing to the requirement you’re going to test. What you could do is use the Jira TestRail add on. Right from the story in Jira, you can choose to add a test case, and it’s going to pop open to TestRail. Because you start with the actual story and click the “Add test case” button, TestRail already copies the reference for you. Just make that part of your workflow. You start with the story in Jira, use the Jira add on, and add the test case.
Now, as you test, the overall question is: How do you actually track this during the sprint? When you’re running a sprint, you may test against some specific issues, or maybe you have an epic that contains a lot of the issues you’re going to test. So, you have a Milestone in TestRail and its name is the same as the sprint you’re running in Jira. The sprint in Jira has a certain timeline. So does the Milestone in TestRail. That way you can align what you’re doing in TestRail with Jira.
Then, you’re going to run a couple of different configurations, maybe Windows 11, Mac OS 12, Firefox, Chrome, Edge, and Safari. And you want to run some regression tests with Selenium integrated with TestRail. Just select those test cases. Now, under this Milestone, you’ve got the functional testing that you’re going to track in real time, and regression tests automated with Selenium. All you have to do is open that Milestone, and you’ll see the status of all the testing planned to do as part of this sprint, right there under the Milestone in TestRail.
But how do you make sure that you’ve actually written enough tests for the references in this milestone? Let’s look at the Coverage for References report in TestRail. Once you run the report, you’re going to get a quick insight into whether the references, the issues right from your Jira story, actually have test cases and test runs. Not only does this help to build your traceability but also to see if there is coverage in your test case repository. This report is something you would run maybe once a week, or you’d run ad hoc to just double-check.
Then, you want to see what’s the status of all the tests running against these requirements. There’s another report that’s good for that: the Comparison for References report. You would do a similar thing: just download the reference IDs for the ones that are involved in your sprint, copy and paste them into TestRail. The Comparison for References report will show all the test results that you’ve added so far against any of the test cases you had those references linked to. You’ll be able to see: Are there any critical tests that are still failing? Should we delay this release or not? Because you set this up as a Milestone, and you sort of adopted this methodology, where a Milestone in TestRail is going to match a sprint in Jira, you’re just able to get that information at your fingertips.
Can I report on my automated test results using TestRail?
Diogo: You definitely can. We just released a couple of months ago a CLI tool to import your automated test results, as long as you have a framework that allows you to generate a JUnit style report. It’s open source, so you can also customize it, contribute and try to make it better for everyone. This is great for centralizing all your testing efforts in one place. If you have your automated test results isolated somewhere, and then you have your manual efforts on TestRail, how do you know if you’re testing everything you want to test?
“With the CLI tool you can combine all the automated results with the manual results in TestRail, so you can have an insight into the full testing.”
For automation, a lot of times there are things like flaky tests. You always need to analyze the results and debug a bit. One thing we can suggest in TestRail is using custom status for flaky tests. You can set a unique status. Then you can use the Status Tops report to show which tests have the highest failure rate. You will probably have to give them more attention because either the feature is problematic, has a lot of bugs in it, or the automated tests you’re running against it are very flaky. Either way, you have to take action.
Shanu: The Property Distribution report is another way to know about the test cases based on the automation status. Because it gives you the ability to group the test cases by attribute.
Diogo: That’s a very good example. The Property Distribution report works with any drop-down field you have on your cases. You can, for instance, create a custom field which would be automation status, for tests to be automated, or not to be automated, or currently being assessed, and so on. You will have a nice chart that shows your test automation progress.
The same works for test priority. Maybe you’re writing too many low priority tests and perhaps you need to do more critical tests. These tests will probably consume less and you will test more in less time if you don’t have a lot of tests that aren’t that meaningful.
Matt: In terms of actually integrating and test automation, there’s a couple of community supported plugins out there for different frameworks. There are two primary ways that I’ve seen teams actually integrate their automation tool with TestRail.
First one is the custom API development route, where you’re using TestRail’s API, which has tons and tons of endpoints for just about any test result reporting need that you’d have. Teams will write a custom script or custom logic for how they parse the results object, from whatever the framework is, to what information they actually send to TestRail. Actually, Shanu wrote a really good article about how to integrate TestRail with Selenium.
Then, the other way that I’ve seen people work with integration, and this is more recent, is the CLI tool. You already mentioned that, but maybe we can give a quick overview of how that works.
Diogo: Yes. CLI is a very simple tool, developed in Python and the code is open source. You can customize it even further to your particular needs. Like the name says, it’s a command-line tool. All you need to do is write one line on your console, where you will specify things like your TestRail instance, your credentials, and you actually don’t need much more than. Then you just need to add the project, give the report path, and it will parse through your JUnit report. Any JUnit compatible report will work. Almost all the test automation frameworks that I know of generate that sort of report, or you can just create some sort of converter. There are many ways to do that.
The CLI is very useful. It not only reports your results to TestRail, but it also creates test cases if they don’t already exist. If you want to do an approach where you do automation first, you don’t need to write test cases in TestRail first, you just write your automated test cases. Then, when you import your test results through the CLI, it will create the test cases on TestRail and add the results for your test run. It has an auto-mapping mechanism. There’s a custom field with the name space and the test name, which allows you to map your tests. If you imported the tests again, it will know the set already exists and report the result to the right test.
Can I share my reports outside TestRail?
Diogo: This is quite a straightforward one. Yes, you can. People are always worried because of licenses or pricing, but you can do this in two ways. If you have already generated the report, you can just go to the Share Report button on the top of the page, and send it via email as HTML or PDF. Or, when you schedule a report, you can also automatically send it to these emails as PDF or HTML. This also works for most pages on TestRail where you have charts, test executions, test runs, etc. You can also share those outside as a PDF.
Matt: Some reports can get lengthy, so sometimes choosing the HTML option, when you send that report as an email, might be the best way to do it. The person will just click the attachment, and it’ll open in the browser window.
TestRail Professional Vs TestRail Enterprise
Matt: All of our reporting capabilities are available on TestRail Professional. You don’t need to go up to the Enterprise plan. But in TestRail Enterprise, you do get other features that might be important. For example, TestRail Enterprise comes with test case approval and versioning. While that’s not a report, it may be something that is required for compliance. You might need to prove that someone else has reviewed any test case.
Another relevant thing is instance auditing. So everybody in TestRail can see the history of each individual test case, test runs, and things like that. But if you have a more specific security requirement, or compliance requirements, to audit activities like somebody’s created a test case, somebody’s invited someone else to the team, somebody’s deleted or updated a test case, then you might want to look at TestRail Enterprise. Those are features we hear people talking about, and ultimately made them decide it was worth going up to Enterprise. SSO compatibility is also really popular there.