Integrating TestRail and JUnit - TestRail

Example: Integrating TestRail and JUnit

Here is an end-to-end working solution for integrating TestRail and JUnit. 

The example starts with a real sample Java program in Eclipse (we used Eclipse 4.6.1 Java SE 1.8), with two JUnit (JUnit-413) test files. The program has a compile.sh (for Mac or Linux) and compile.bat (for Windows) shell script that compiles the code, runs tests and pushes the code into TestRail using the TestRail API with Java bindings, called ReportJUnitResults.

Through this example, you will learn how to integrate TestRail with JUnit to help manage your JUnit testing and keep track of all of your tests and test results in TestRail for visibility and traceability.

Getting Started

You’ll need a TestRail account and administrator access. Click Administration (in the upper right), Site Settings, and API, and then click the checkbox to enable API (the cloud icon with a gear) and save.

Once that’s done, you will want some software under test. To make things easy, we’ve pushed a “triangle” program into Git, with all the project fixings to run as an Eclipse project.

Triangle.java is a simple class that takes three integers and tells you what kind of a triangle you have. That could be equilateral, where all sides are equal (1,1,1). It might be isosceles, where two sides are the same (2,2,3). Or it could be scalene, where the sides are all different lengths (3,4,5). 

The tests triangleTest.java and triangleSpecialCasesTest.java are test cases for the software, which is so new it does not even have a user interface yet. This is what the Eclipse directory structure looks like:

 

The main function to test is a simple Java function found in triangle.java:

public String gettriangletype(int x, int y, int z) {
..
 }

Here are two of the tests, found in triangleTest.java:

@Test
    public void test_equilateral() {
                triangle t = new triangle();
                String result = t.gettriangletype(1, 1, 1);
                assertEquals(“111 is equilateral”, “equilateral”, result);
    }

   @Test
   public void test_scalene() {
                 triangle t = new triangle();
                 String result = t.gettriangletype(3, 4, 5);
                 assertEquals(“345 is scalene”, “scalene”, result);
   }
A conventional way to run the tests is through Eclipse; a developer would run them while writing code, using something like test-driven development (TDD). What we want to do is get the tests to run on every build kicked off by a continuous integration system like TravisCI or Jenkins. To do that, we need to create a batch script that generates our .jar files and then runs unit tests.

Examining the Example

The entire codebase is available in GitHub as an Eclipse workspace here. Clone the repository or download the entire directory structure. We’ve uploaded everything you need, including external references, but if you want to check, they are JUnit-4.13.jar, json-simple-1.1.jar, hamcrest2.2 and the TestRail API bundle for Java. Again, these are pre-installed in the correct directories, but you might want to read the documentation or download them into your own projects. 

There’s one more piece to do on the TestRail side; we need a project number and test cases to push our results to. To do that, log into TestRail and create or click on a project. Note the URL — it will be something like:

https://xndev.testrail.com/index.php?/projects/overview/1

That “1” at the end is the project number. Create two test cases within that project; these correspond to the two JUnit cases. When you click on “Test Cases” within the project, they will get an identifier, such as C1 and C2. Remove the letter and you have the number of the test case.

Now it’s time to download the program from GitHub, load it into Eclipse, and look at the code!

The /build directory contains the build script, build_and_test.sh. The last three lines run the tests and then call the program to push the test results back into TestRail.

#Excerpt from build_and_test.sh
javac -d . -cp junit-4.13.2.jar:. ../src/triangleTest.java
javac -d . -cp junit-4.13.2.jar:. ../src/triangleSpecialCasesTest.java
javac -d . -cp junit-4.13.2.jar:. ../src/triangletestsuite.java
javac -d . -cp junit-4.13.2.jar:. ../src/TriangleTestSuiteRunner.java
java -cp junit-4.13.2.jar:hamcrest-2.2.jar:. org.junit.runner.JUnitCore triangleTest > testResults1.txt
java -cp junit-4.13.2.jar:hamcrest-2.2.jar:. org.junit.runner.JUnitCore triangleSpecialCasesTest > testResults2.txt

At this point we have a mechanism to run our JUnit tests and get output that connects the test case number to the test. There are a lot of ways to do this; an Ant or Maven build script could do it in XML.

Now to write the code that takes those results and put them in TestRail. You can find that code in RestJUnitResults.java.

Let’s look at that main routine:

public static void main(String[] args) {
//create the test run
APIClient client = new APIClient(baseurl);
client.setUser(userid);
client.setPassword(pwd);
try {
String test_run_id;
test_run_id = api_call_create_test_run(client);
System.out.println(“Test Run Id ” + test_run_id + ” ” + System.getProperty(“user.dir”));
//Loop through the directory, finding test case runs and pushing them over the API
create_test_case_execution_records(client, test_run_id);
//Make run as complete – commented out for now
//close_test_run(client, test_run_id);
} catch (MalformedURLException e1) {
                 // TODO Auto-generated catch block
                e1.printStackTrace();
} catch (IOException e1) {
                // TODO Auto-generated catch block
               e1.printStackTrace();
} catch (APIException e1) {
                // TODO Auto-generated catch block
               e1.printStackTrace();
}
}

Note that this executes after build_and_test.sh has actually performed the new build and created the test file named testResult(n).txt, where N is the name of the test case ID. This code creates a test run, then calls create_test_execution_records(), which will loop through the build directory, picking up all the files and examining them to see if they are marked correct or not.

If you are a programmer writing these yourself, you’ll likely want to dig into the functions that call the TestRail APIs. Here’s a brief summary of the core three:

Api_call_create_test_run makes an API call to create the test run — a single database record for the entire run of the suite.

Api_call_add_test_case_run adds a run for a single test case using the API. 

 Close_test_run indicates the test run is complete and can be archived. 

Let’s dig into the details.

Taking a Low-Level Code Walkthrough

One thing to note is that the code updates every test case as “pass” or “fail.” To know if the test case passes, the program looks at the file output. A typical file output might look like this:

JUnit version 4.13.2
….
Time: 0.006
OK (4 tests)

A function process_test_case_run() uses this line to determine if a test cases passes:

if ( stringResults.contains("\nOK ("))

If you are using a different form of test output, that check may need to change.

When this completes, we get test run results imported into TestRail.

Notice that TestRail thinks the test is still running — that it is not complete. That is what the close_test_run function does for us: It makes the API call to close the test run. There is no need to indicate if the test run passed or fails; TestRail figures that out by based on whether all the test cases passed (or not).

Here’s the code for close_test_run, included with the sample app, which will run automatically.

private static String close_test_run(APIClient client, String test_run_id) throws MalformedURLException, IOException, APIException {
JSONObject obj = new JSONObject();
JSONObject r;
r = (JSONObject) client.sendPost(“close_run/”+test_run_id, obj);
String result = r.toString();
System.out.println(“Result of closing run is ” + result);
return result;
}

This function is called toward the end of main[]. After adding it on the test runs screen, if you scroll down, you will see that the test automation run is now marked as complete.

Run it Yourself

First, run build_and_test.sh to create the results file; then, edit reportJUnitResults.java to use your real user ID, password, project number, and so on. Finally, run the main routine of reportJUnitResults.java in eclipse.

Architecture and choices

This tutorial is designed to provide an integration that has the greatest level of flexibility. Many JUnit runners provide more verbose test output but are based on a specific runner — and this output tends to change in different versions. Likewise, we chose to connect test_case_id to the test using the name of the file. It is possible to do this integration with annotations in the test code itself, but that might change by version of JUnit.

Also note that the builder program does not build reportJUnitResults every time; if you would like to, we have provided an Ant build script to do that, called ant_script_to_build_ReportJUnitResults.xml, in the main workspace.

Here, we provided an example of an end-to-end JUnit test suite against real working software that updates TestRail with results based on tests run in JUnit.

This walkthrough was contributed by Matthew Heusser. Matthew is the Managing Director of Excelon Development, with expertise in project management, development, writing, and systems improvement. And yes, he does software testing too.