Skip to content

TestNG

Description

Pangolin provides a set of Java annotations which allows you to send your TestNG test XML reports right after test execution in your favorite IDE.

Technically it consists of three parts:

  • @Pangolin annotation
  • TestNG test run reporter: com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter
  • XML configuration file which contains TestRail connection and other details needed for functioning of pangolin annotations package

Prerequisites

  • Pangolin Server must be installed and be accessible via HTTP from user machine
  • Maven project with TestNG tests

- Installation and configuration

  • Modify your pom.xml file and add pangolin-annotations dependencies and Agiletestware repository information:
<dependencies>
    <dependency>
        <groupId>com.agiletestware</groupId>
        <artifactId>pangolin-annotations</artifactId>
        <version>2.5</version>
        <scope>test</scope>
    </dependency>
</dependencies>
<repositories>
    <repository>
        <id>public.maven.agiletestware.com</id>
        <url>http://public.maven.agiletestware.com.s3-website-us-west-2.amazonaws.com/release</url>     
    </repository>       
</repositories>
  • Add com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter to maven-surefire-plugin:
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.19</version>             
    <configuration>
        <properties>
            <property>
                <name>listener</name>
                <value>com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter</value>
            </property>
        </properties>
    </configuration>
</plugin>
  • Create pangolin_config.xml XML file in the project's root:
<?xml version="1.0" encoding="UTF-8"?>
<pangolin>
    <!-- Pangolin Server URL -->
    <pangolin_url>http://pangolinurl:9090</pangolin_url>
    <!-- TestRail URL -->
    <testrail_url>https://testrailurl:8080</testrail_url>
    <!-- TestRail user name -->
    <testrail_user>username@company.domain</testrail_user>
    <!-- TestRail encrypted password: please use http://server_name:port/pangolin/password to encrypt your plain text password -->
    <testrail_encrypted_password>encryptedPassword</testrail_encrypted_password>
    <!-- TestRail project -->
    <testrail_project>project name</testrail_project>
    <!-- Whether the Pangolin is disabled or not. If this is omitted, Pangolin is enabled -->
    <disabled>false</disabled>
    <!-- Whether a run in TestRail should be closed or not. If this is omitted, Pangolin will not close runs -->
    <close_run>true</close_run>
</pangolin>

Setting configuration file with Java System Properties

Sometimes it's not desired to store pangolin_config.xml file in project root directory, so it can be set with pangolin.config.xml Java System Property: "mvn -Dpangolin.config.xml=D:\temp\pangolin_config.xml clean test"

- Add Pangolin annotations to TestNG classes/methods

@Pangolin Java annotation can be added on class and method levels When class or method is marked with Pangolin annotation, its result will be exported to TestRail.

- Pangolin annotation attributes

Name Description Applicable Required
sectionPath Path to a section in TestRail project into which results should be exported. Must start with TestRail Suite name, e.g. Master\Section\SubSection Class, method Yes
runName Name of a test run in TestRail to which results will be added. If run does not exist, it will be created automatically. If it is not set, a new run with time stamp will be created Class, method No
planName Name of a test plan in TestRail to which results will be added. If plan does not exist, it will be created automatically Class, method No
milesonePath A path to a milestone to which test results will be added. If milestone does not exist, it will be created. E.g.: Milestone1\Milestone2 Class, method No
testName Name of a test in TestRail. If it's not set, then method name is used Method No
customFields An array of case fields name/value pairs to pass into TestRail Class, method No
disabled Set it to com.agiletestware.pangolin.annotations.BooleanValue.TRUE to disable results export for a particular method or class Class, method No

Overriding annotations' values on method level

Pangolin annotation on method level overrides values set by Pangolin annotation on class level.

- Execute Maven test phase

To run TestNG tests and send results to TestRail, just run Maven test phase on your project: e.g.: mvn test

Support of WebDriver and Appium

Pangolin can be used easily together with WebDriver and Appium tests and do not require any additional configuration.

- Examples

- Pangolin annotation on class level

package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection")
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        Assert.fail("test1 failed");
    }

    @Test
    public void test2() {
        System.out.println("test2 passed");
    }

    @Test
    public void test3() {
        System.out.println("test3 passed");
    }

    @Test
    public void test4() {
        Assert.fail("test4 failed");
    }

    @Test
    public void test5() {
        System.out.println("test5 passed");
    }
}

Execution log:

Execution

Results in TestRail:

Results in TestRail

A new run with time stamp is created:

Results in TestRail

Results in TestRail

Test case result:

Results in TestRail

- Adding results to a particular Test Run in TestRail

To add results to a particular run in TestRail, just specify runName attribute:

package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection", runName = "Maven run")
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        Assert.fail("test1 failed");
    }

    @Test
    public void test2() {
        System.out.println("test2 passed");
    }

    @Test
    public void test3() {
        System.out.println("test3 passed");
    }

    @Test
    public void test4() {
        Assert.fail("test4 failed");
    }

    @Test
    public void test5() {
        System.out.println("test5 passed");
    }
}

Results in TestRail:

Results in TestRail

- Adding results to a particular Test Plan in TestRail

To add results to a particular run in TestRail, just specify planName attribute:

package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection", planName = "Maven plan")
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        Assert.fail("test1 failed");
    }

    @Test
    public void test2() {
        System.out.println("test2 passed");
    }

    @Test
    public void test3() {
        System.out.println("test3 passed");
    }

    @Test
    public void test4() {
        Assert.fail("test4 failed");
    }

    @Test
    public void test5() {
        System.out.println("test5 passed");
    }
}

Results in TestRail:

Results in TestRail

- Add test results to TestRail Milestone

To add tests results to a Milestone in TestRail, define a path to a milestone with milesonePath attribute of Pangolin annotation:

package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection", milesonePath = "Milestone1\\Milestone2")
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        Assert.fail("test1 failed");
    }

    @Test
    public void test2() {
        System.out.println("test2 passed");
    }

    @Test
    public void test3() {
        System.out.println("test3 passed");
    }

    @Test
    public void test4() {
        Assert.fail("test4 failed");
    }

    @Test
    public void test5() {
        System.out.println("test5 passed");
    }
}

Results in TestRail:

Results in TestRail

Results in TestRail

- Setting test name for a method

To set test name to a particular test method, add Pangolin annotation to a test method and set testName attribute:

package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection")
public class SomePassSomeFailTest {

    @Pangolin(testName = "Some test name")
    @Test
    public void test1() {
        Assert.fail("test1 failed");
    }

    @Test
    public void test2() {
        System.out.println("test2 passed");
    }

    @Test
    public void test3() {
        System.out.println("test3 passed");
    }

    @Test
    public void test4() {
        Assert.fail("test4 failed");
    }

    @Test
    public void test5() {
        System.out.println("test5 passed");
    }
}

Results in TestRail:

Results in TestRail

- Passing values for custom case fields

If you need to pass value to some case fields in TestRail (e.g. for fields which are required in TestRail project), just add an array of com.agiletestware.pangolin.annotations.CustomField objects into customFields attribute.
CustomField object contains the following attributes:

  • name - name of a field in TestRail, e.g. custom_test_field
  • value - value to be set for a field
package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.CustomField;
import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection", customFields = { @CustomField(name = "custom_required_field", value = "Maven") })
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        Assert.fail("test1 failed");
    }

    @Test
    public void test2() {
        System.out.println("test2 passed");
    }

    @Test
    public void test3() {
        System.out.println("test3 passed");
    }

    @Test
    public void test4() {
        Assert.fail("test4 failed");
    }

    @Test
    public void test5() {
        System.out.println("test5 passed");
    }
}

Results in TestRail:

Results in TestRail

- Passing values for result fields

If you need to pass value to some fields of result in TestRail (e.g. for fields which are required in TestRail project), just add an array of com.agiletestware.pangolin.annotations.CustomField objects into customResultFields attribute.
CustomField object contains the following attributes:

  • name - name of a field in TestRail, e.g. custom_test_field
  • value - value to be set for a field
package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.CustomField;
import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection", customResultFields = { @CustomField(name = "custom_result_fields", value = "Maven"),
        @CustomField(name = "version", value = "12") })
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        Assert.fail("test1 failed");
    }

    @Test
    public void test2() {
        System.out.println("test2 passed");
    }

    @Test
    public void test3() {
        System.out.println("test3 passed");
    }

    @Test
    public void test4() {
        Assert.fail("test4 failed");
    }

    @Test
    public void test5() {
        System.out.println("test5 passed");
    }
}

Results in TestRail:

Results in TestRail

- Exporting all test methods results into single test in TestRail

In BDD frameworks, test class may represent one single test case and its methods are test steps of this test.
To support this scenario, Pangolin annotation's attribute hasSteps should be set to BooleanValue.TRUE:

package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.BooleanValue;
import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection", hasSteps = BooleanValue.TRUE)
public class MultipleStepTest {

    @Test(priority = 0)
    public void step1() {
        Assert.fail("test1 failed");
    }

    @Test(priority = 1)
    public void step2() {
        System.out.println("test2 passed");
    }

    @Test(priority = 2)
    public void step3() {
        System.out.println("test3 passed");
    }

    @Test(priority = 3)
    public void step4() {
        Assert.fail("test4 failed");
    }

    @Test(priority = 4)
    public void step5() {
        System.out.println("test5 passed");
    }
}

Results in TestRail

Results in TestRail

TestNG test method execution order

Please note that Pangolin relies on test method execution order provided by TestNG, so you have to provide a fixed test methods execution order somehow (e.g. via priorities).

By default, test step names are the same as names of corresponding test methods in your Java code.
To change test step names and some other export settings, a PangolinTestStep annotation was introduced.
This annotation can be applied to test methods only and has the following properties:

Name Description Required
name Name of a test step in TestRail Yes
expected Value for Expected Result field of a test step in TestRail No

e.g.: 

package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.BooleanValue;
import com.agiletestware.pangolin.annotations.Pangolin;
import com.agiletestware.pangolin.annotations.PangolinTestStep;

@Pangolin(sectionPath = "Master\\Section\\SubSection", hasSteps = BooleanValue.TRUE)
public class MultipleStepTest {

    @PangolinTestStep(name = "The first step", expected = "Something expected")
    @Test(priority = 0)
    public void step1() {
        System.out.println("step1 passed");
    }

    @Test(priority = 1)
    public void step2() {
        System.out.println("step2 passed");
    }

    @Test(priority = 2)
    public void step3() {
        System.out.println("step3 passed");
    }

    @Test(priority = 3)
    public void step4() {
        Assert.fail("step4 failed");
    }

    @Test(priority = 4)
    public void step5() {
        System.out.println("test5 passed");
    }
}

And it produces the following test steps in TestRail:

Results in TestRail

Tip

To map results into an existing test in TestRail, just add id attribute containing an ID of a test in TestRail, e.g.: @Pangolin(id=1234, hasSteps = BooleanValue.TRUE)

- Adding arbitrary attachments to Test Step in TestRail (since 2.2)

In some cases it could be beneficial to have some test data like screenshots or logs accessible in TestRail, so Pangolin supports adding arbitrary attachments to test results.

Adding PangolinTestNGListener into pom.xml

To be able to upload attachments, the com.agiletestware.pangolin.annotations.testng.PangolinTestNGListener class should be added into Maven Surefire plugin config as listener in addition to com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.19</version>         
    <configuration>
        <properties>
            <property>
                <name>listener</name>
                <value>com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter,com.agiletestware.pangolin.annotations.testng.PangolinTestNGListener</value>
            </property>
        </properties>
    </configuration>
</plugin>

Adding attachments in the code

To add an arbitrary attachment to your test results in TestRail you can use com.agiletestware.pangolin.annotations.CurrentTest class which provides several static overloaded addAttachment methods:

  • addAttachment(File) - adds a given file into step results in TestRail
  • addAttachments(List<File) - add a list of files into step results in TestRail
  • addAttachment(String, InputStream) - add attachment from generic input stream

Example: 

import static org.testng.Assert.assertEquals;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;

import org.junit.BeforeClass;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.annotations.AfterTest;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.CurrentTest;
import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\demo", runName = "Webdriver TestNG")
public class SimpleWebDriverTest {
    private final RemoteWebDriver webDriver = new ChromeDriver();

    @Test
    public void testPageOpens() throws IOException {
        webDriver.navigate().to("http://agiletestware.com");
        final File screenshot = ((TakesScreenshot) webDriver).getScreenshotAs(OutputType.FILE);
        CurrentTest.addAttachment(screenshot);
    }

    @Test
    public void testTitleCorrect() throws IOException {
        webDriver.navigate().to("http://agiletestware.com");
        try (InputStream stream = new ByteArrayInputStream(webDriver.getPageSource().getBytes())) {
            CurrentTest.addAttachment("page.html", stream);
        }
        assertEquals(webDriver.getTitle(), "something clearly wrong");
    }

    @AfterTest
    public void afterTest() {
        webDriver.quit();
    }

    @BeforeClass
    public static void setUp() {
        System.setProperty("webdriver.chrome.driver", new File("chromedriver.exe").getAbsolutePath());
    }

}

Results in TestRail:

Results in TestRail

Results in TestRail

- Mapping test method result into existing test in TestRail (since 2.5)

Pangolin Annotation package allows you to map results of your test into existing test in TestRail by simply marking your test method or class with Pangolin annotation with id attribute:

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.Pangolin;

public class MapToExistingTest {

    @Pangolin(id = 14223)
    @Test
    public void mappedTest() {
        Assert.fail("test1 failed");
    }

}

where id is an id of an existing test case in TestRail:

Existing test case in TestRail

Note, that only 'C' character in the ID value should be omitted.

Mapping all test method results into one test

If you export all methods of a class into one test in TestRail (by specifying hasSteps = BooleanValue.TRUE), just add id attribute on the class level

Results in TestRail:

Results in TestRail

- Disabling test results export

To disable exporting of test results into TestRail user can either:

  • Set disabled element of pangolin_config.xml file to true
<?xml version="1.0" encoding="UTF-8"?>
<pangolin>
    <!-- Pangolin Server URL -->
    <pangolin_url>http://pangolinurl:9090</pangolin_url>
    <!-- TestRail URL -->
    <testrail_url>https://testrailurl:8080</testrail_url>
    <!-- TestRail user name -->
    <testrail_user>username@company.domain</testrail_user>
    <!-- TestRail encrypted password: please use http://server_name:port/pangolin/password to encrypt your plain text password -->
    <testrail_encrypted_password>encryptedPassword</testrail_encrypted_password>
    <!-- TestRail project -->
    <testrail_project>project name</testrail_project>
    <!-- Whether the Pangolin is disabled or not. If this is omitted, Pangolin is enabled -->
    <disabled>true</disabled>
</pangolin>
  • Or set disabled attribute of Pangolin annotation to BooleanValue.TRUE for a test class or method:
package com.agiletestware.pangolin.dummytest;

import org.testng.Assert;
import org.testng.annotations.Test;

import com.agiletestware.pangolin.annotations.BooleanValue;
import com.agiletestware.pangolin.annotations.Pangolin;

@Pangolin(sectionPath = "Master\\Section\\SubSection", disabled = BooleanValue.TRUE)
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        Assert.fail("test1 failed");
    }

    @Test
    public void test2() {
        System.out.println("test2 passed");
    }

    @Test
    public void test3() {
        System.out.println("test3 passed");
    }

    @Test
    public void test4() {
        Assert.fail("test4 failed");
    }

    @Test
    public void test5() {
        System.out.println("test5 passed");
    }
}

Pangolin will print the following message for tests which test results will not be exported:

Pangolin: Skipping com.agiletestware.pangolin.dummytest.SomePassSomeFailTest.test3 method because it's disabled