Introduction to TestRail
TestRail is a web-based test case management tool. It is used by testers, developers and team leads to manage, track, and organize software testing efforts. TestRail allows team members to enter test cases, organize test suites, execute test runs, and track their results, all from a modern and easy to use web interface.
Teams benefit from four main advantages when adopting TestRail:
Manage test cases, suites and test runs
The traditional way of managing and documenting test cases on paper, in wikis, bug tracking tools, or general document management systems quickly reaches its limits. TestRail enables you to create, manage, and organize test cases and suites within an optimized user interface and application structure.
Boost testing productivity
TestRail’s todo lists, filters, and email notifications help coordinate test runs and increase the productivity and responsibility of testers. The goal is to make sure that every team member knows their tasks at all times and that team leads can assign new tasks to testers depending on their workload.
Real-time insights into your testing progress
To make important project decisions, it is critical to have access to detailed information about the testing progress and test results. Questions such as “What percent of a test run have we successfully completed?”, “How are we progressing on the upcoming milestone?” or “Which tests have been executed in the past 24 hours?” can only be answered efficiently by using a comprehensive test case management software such as TestRail, and TestRail makes these details readily available. The organization and coordination of multiple parallel tests, be it for different projects or releases, is often complex and time-consuming. To not lose track of your testing efforts, TestRail helps you manage important software-specific data and structures such as project milestones and makes it easy to integrate with bug tracking tools.
The organization and coordination of multiple parallel tests, be it for different projects or releases, is often complex and time-consuming. To not lose track of your testing efforts, TestRail helps you manage important software-specific data and structures such as project milestones and makes it easy to integrate with bug tracking tools.
After logging in to TestRail, the first page you usually see is the dashboard. The dashboard provides you with an overview of available projects, recent project activities, and your todos. When you navigate to a project (by clicking on a project title), you switch to the project view and land on the project’s overview page, showing project details such as test cases, active test runs, project activity, and so on. Whenever you need to switch to another project, just return to the dashboard by clicking the link in the upper left corner.
Test cases and sections
A test case in TestRail consists of a description of the test’s prerequisites, a list of test steps and the expected result. A test case can ideally be verified by a single tester in a short period of time and confirms a specific functionality, documents a task or verifies a project artifact.
In TestRail, test cases are organized into sections and sub-sections. Sections and sub-sections are a collection of test cases and are often created for specific project modules or areas of your product. How you organize your sections depends on the size of your project. If you have a lot of test cases for a project module, it is recommended to further break down your sections and create sub-sections for specific features or functionality.
For example, if you use multiple test cases to verify a specific application dialog or web page, these test cases should be grouped in a section.
Test runs and tests
To run a test and enter test results for the cases you added, you start a test run for a particular test suite. While a test suite is just like a plan that specifies how an application is tested, a test run is an actual test you are conducting.
For most projects you will most likely start multiple test runs for a particular test suite over time. For example, if you are releasing multiple versions of a software program, you might want to conduct a test run for each new release.
Likewise, you can have multiple test runs for a particular test suite active at the same time. This can make sense if you want to execute a particular test suite for multiple configurations (such as different operating systems). You can then start a test run for each different configuration you want to test against.
When you start a new test run (e.g. by clicking the Run a Test button in the toolbar of a test suite), you can choose to include all cases of a test suite, select specific test cases, or set a dynamic filter based on specific test case values. You can learn more about using test case selection for test runs in our documentation here.
A run consists of individual tests for each case that you add. By default, each test has one of five different statuses that are signaled in TestRail by specific colors. You can add a test result and change the status of a test either by clicking the Add Result button on the test page, or by changing it directly on the run page.
The following test statuses are available by default:
- Untested: By default, new tests have the status Untested. Once a test result has been added to a test, it can never receive the Untested status again.
- Passed: A test is marked as Passed when a tester verified the test steps and the expected results.
- Failed: A tester marks a test as Failed if one of the specified test steps resulted in an error or if the expected result differs from the actual test result.
- Retest: If a developer or tester determines that a test should be tested again, she marks the test as Retest. E.g., if a test originally failed and the developer fixed the problem, the developer can mark it to be retested.
- Blocked: The Blocked status is used to signal that a test cannot be executed currently because of some external dependency (such as a bug that needs to be fixed before being able to complete to test). It is often used in combination with the Retest status.
Once a test run has been completed, you can close it from the run’s edit page. Tests of a closed run cannot be edited or changed, making it ideal for archiving test runs. Additionally, if you change any of the associated cases’ attributes (such as the expected result), the change won’t be applied to the closed and archived run.
Projects and milestones
Projects are the main organizational unit in TestRail. It is generally recommended to create a TestRail project for every real software project you want to manage within TestRail. All other data such as test runs, test results, milestones etc. are directly associated with a specific project.
See the following illustration to see how the different entities relate to each other:
You can also add your project milestones to TestRail. A milestone can be any project milestone you deem to be important, such as a planned public software release, an internal test version or a new beta release for an important customer. Once you have added your milestones to TestRail, you can assign test runs to specific milestones.
Assigning test runs to milestones has the advantage that you can easily track the milestones’ test results and progress separately. Especially if you are working on multiple milestones in parallel or if you have many test runs active at the same time, managing milestones within TestRail is a great help.
Test plans and configurations
When you need to manage multiple test runs and configurations for a single project, TestRail’s test plans come in handy. A test plan allows you to start multiple test runs at once if you want to test against multiple configurations. A configuration can be anything you need to test your project against, such as different operating systems or web browsers.
To create a test plan, simply click the Add Test Plan button from the Test Runs & Results page. Just like you do with a single test run, you can configure all properties of the test runs such as the name, the cases you want to include, and so on.
You often have to test a specific test run against multiple configurations such as operating systems or web browser versions. To make it easier to create a separate test run for each configuration combination, you can specify and select all configurations for your test runs within a plan. To do this, simply click the Select Configurations link of a run and select or add your configurations.
Todos and notifications
Tests can be assigned to team members. You can either assign tests when a run is created or do so afterward from the run or test pages. Once a test has been assigned to a user, the test appears on the user’s todo list. Every user has its own todo list for each project she’s working on, while the Dashboard provides a quick way to see the user’s todos for all projects.
The todo page can also be used to view the todos of other team members or the entire team at once, providing an easy way for team leads to assign tests to team members with the least workload.
Another relevant feature is email notifications. Email notifications help test owners to learn about test changes made by other users. For example, if a test has been assigned to a user (i.e. the test owner) and another user adds a comment or result to the test, the test owner is notified by email. It is also possible to subscribe to tests of other users or even entire runs by clicking the small email icons on the relevant pages.
Tracking progress and activity
Besides making it easier to manage test cases and enter test results, providing an easy way for all team members to track the test progress is one of TestRail’s most useful features. You can view the test results and testing activity of runs, milestones and entire projects on the individual resource pages.
You can choose between the status, activity, and progress view on the test run and milestone pages from the sidebar, while you can switch between the project history and test results on the project’s overview page. Additional reporting features can be found on the Reports tab.
Learn more and try out TestRail
If you haven’t already, the easiest way to try out TestRail is to create a trial on our servers. This just takes a few minutes to set up.