TestNG¶
Description¶
Pangolin provides a set of Java annotations which allows you to send your TestNG test XML reports right after test execution in your favorite IDE.
Technically it consists of three parts:
- @Pangolin annotation
- TestNG test run reporter: com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter
- XML configuration file which contains TestRail connection and other details needed for functioning of pangolin annotations package
Prerequisites¶
- Pangolin Server must be installed and be accessible via HTTP from user machine
- Maven project with TestNG tests
- Installation and configuration¶
- Modify your pom.xml file and add pangolin-annotations dependencies and Agiletestware repository information:
<dependencies>
<dependency>
<groupId>com.agiletestware</groupId>
<artifactId>pangolin-annotations</artifactId>
<version>2.5</version>
<scope>test</scope>
</dependency>
</dependencies>
<repositories>
<repository>
<id>public.maven.agiletestware.com</id>
<url>http://public.maven.agiletestware.com.s3-website-us-west-2.amazonaws.com/release</url>
</repository>
</repositories>
- Add com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter to maven-surefire-plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19</version>
<configuration>
<properties>
<property>
<name>listener</name>
<value>com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter</value>
</property>
</properties>
</configuration>
</plugin>
- Create pangolin_config.xml XML file in the project's root:
<?xml version="1.0" encoding="UTF-8"?>
<pangolin>
<!-- Pangolin Server URL -->
<pangolin_url>http://pangolinurl:9090</pangolin_url>
<!-- TestRail URL -->
<testrail_url>https://testrailurl:8080</testrail_url>
<!-- TestRail user name -->
<testrail_user>username@company.domain</testrail_user>
<!-- TestRail encrypted password: please use http://server_name:port/pangolin/password to encrypt your plain text password -->
<testrail_encrypted_password>encryptedPassword</testrail_encrypted_password>
<!-- TestRail project -->
<testrail_project>project name</testrail_project>
<!-- Whether the Pangolin is disabled or not. If this is omitted, Pangolin is enabled -->
<disabled>false</disabled>
<!-- Whether a run in TestRail should be closed or not. If this is omitted, Pangolin will not close runs -->
<close_run>true</close_run>
</pangolin>
Setting configuration file with Java System Properties
Sometimes it's not desired to store pangolin_config.xml file in project root directory, so it can be set with pangolin.config.xml Java System Property: "mvn -Dpangolin.config.xml=D:\temp\pangolin_config.xml clean test"
- Add Pangolin annotations to TestNG classes/methods¶
@Pangolin Java annotation can be added on class and method levels When class or method is marked with Pangolin annotation, its result will be exported to TestRail.
- Pangolin annotation attributes¶
Name | Description | Applicable | Required |
---|---|---|---|
sectionPath | Path to a section in TestRail project into which results should be exported. Must start with TestRail Suite name, e.g. Master\Section\SubSection | Class, method | Yes |
runName | Name of a test run in TestRail to which results will be added. If run does not exist, it will be created automatically. If it is not set, a new run with time stamp will be created | Class, method | No |
planName | Name of a test plan in TestRail to which results will be added. If plan does not exist, it will be created automatically | Class, method | No |
milesonePath | A path to a milestone to which test results will be added. If milestone does not exist, it will be created. E.g.: Milestone1\Milestone2 | Class, method | No |
testName | Name of a test in TestRail. If it's not set, then method name is used | Method | No |
customFields | An array of case fields name/value pairs to pass into TestRail | Class, method | No |
disabled | Set it to com.agiletestware.pangolin.annotations.BooleanValue.TRUE to disable results export for a particular method or class | Class, method | No |
Overriding annotations' values on method level
Pangolin annotation on method level overrides values set by Pangolin annotation on class level.
- Execute Maven test phase¶
To run TestNG tests and send results to TestRail, just run Maven test phase on your project:
e.g.: mvn test
Support of WebDriver and Appium
Pangolin can be used easily together with WebDriver and Appium tests and do not require any additional configuration.
- Examples¶
- Pangolin annotation on class level¶
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection")
public class SomePassSomeFailTest {
@Test
public void test1() {
Assert.fail("test1 failed");
}
@Test
public void test2() {
System.out.println("test2 passed");
}
@Test
public void test3() {
System.out.println("test3 passed");
}
@Test
public void test4() {
Assert.fail("test4 failed");
}
@Test
public void test5() {
System.out.println("test5 passed");
}
}
Execution log:
Results in TestRail:
A new run with time stamp is created:
Test case result:
- Adding results to a particular Test Run in TestRail¶
To add results to a particular run in TestRail, just specify runName attribute:
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection", runName = "Maven run")
public class SomePassSomeFailTest {
@Test
public void test1() {
Assert.fail("test1 failed");
}
@Test
public void test2() {
System.out.println("test2 passed");
}
@Test
public void test3() {
System.out.println("test3 passed");
}
@Test
public void test4() {
Assert.fail("test4 failed");
}
@Test
public void test5() {
System.out.println("test5 passed");
}
}
Results in TestRail:
- Adding results to a particular Test Plan in TestRail¶
To add results to a particular run in TestRail, just specify planName attribute:
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection", planName = "Maven plan")
public class SomePassSomeFailTest {
@Test
public void test1() {
Assert.fail("test1 failed");
}
@Test
public void test2() {
System.out.println("test2 passed");
}
@Test
public void test3() {
System.out.println("test3 passed");
}
@Test
public void test4() {
Assert.fail("test4 failed");
}
@Test
public void test5() {
System.out.println("test5 passed");
}
}
Results in TestRail:
- Add test results to TestRail Milestone¶
To add tests results to a Milestone in TestRail, define a path to a milestone with milesonePath attribute of Pangolin annotation:
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection", milesonePath = "Milestone1\\Milestone2")
public class SomePassSomeFailTest {
@Test
public void test1() {
Assert.fail("test1 failed");
}
@Test
public void test2() {
System.out.println("test2 passed");
}
@Test
public void test3() {
System.out.println("test3 passed");
}
@Test
public void test4() {
Assert.fail("test4 failed");
}
@Test
public void test5() {
System.out.println("test5 passed");
}
}
Results in TestRail:
- Setting test name for a method¶
To set test name to a particular test method, add Pangolin annotation to a test method and set testName attribute:
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection")
public class SomePassSomeFailTest {
@Pangolin(testName = "Some test name")
@Test
public void test1() {
Assert.fail("test1 failed");
}
@Test
public void test2() {
System.out.println("test2 passed");
}
@Test
public void test3() {
System.out.println("test3 passed");
}
@Test
public void test4() {
Assert.fail("test4 failed");
}
@Test
public void test5() {
System.out.println("test5 passed");
}
}
Results in TestRail:
- Passing values for custom case fields¶
If you need to pass value to some case fields in TestRail (e.g. for fields which are required in TestRail project), just add an array of com.agiletestware.pangolin.annotations.CustomField objects into customFields attribute.
CustomField object contains the following attributes:
- name - name of a field in TestRail, e.g. custom_test_field
- value - value to be set for a field
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.CustomField;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection", customFields = { @CustomField(name = "custom_required_field", value = "Maven") })
public class SomePassSomeFailTest {
@Test
public void test1() {
Assert.fail("test1 failed");
}
@Test
public void test2() {
System.out.println("test2 passed");
}
@Test
public void test3() {
System.out.println("test3 passed");
}
@Test
public void test4() {
Assert.fail("test4 failed");
}
@Test
public void test5() {
System.out.println("test5 passed");
}
}
Results in TestRail:
- Passing values for result fields¶
If you need to pass value to some fields of result in TestRail (e.g. for fields which are required in TestRail project), just add an array of com.agiletestware.pangolin.annotations.CustomField objects into customResultFields attribute.
CustomField object contains the following attributes:
- name - name of a field in TestRail, e.g. custom_test_field
- value - value to be set for a field
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.CustomField;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection", customResultFields = { @CustomField(name = "custom_result_fields", value = "Maven"),
@CustomField(name = "version", value = "12") })
public class SomePassSomeFailTest {
@Test
public void test1() {
Assert.fail("test1 failed");
}
@Test
public void test2() {
System.out.println("test2 passed");
}
@Test
public void test3() {
System.out.println("test3 passed");
}
@Test
public void test4() {
Assert.fail("test4 failed");
}
@Test
public void test5() {
System.out.println("test5 passed");
}
}
Results in TestRail:
- Exporting all test methods results into single test in TestRail¶
In BDD frameworks, test class may represent one single test case and its methods are test steps of this test.
To support this scenario, Pangolin annotation's attribute hasSteps should be set to BooleanValue.TRUE:
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.BooleanValue;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection", hasSteps = BooleanValue.TRUE)
public class MultipleStepTest {
@Test(priority = 0)
public void step1() {
Assert.fail("test1 failed");
}
@Test(priority = 1)
public void step2() {
System.out.println("test2 passed");
}
@Test(priority = 2)
public void step3() {
System.out.println("test3 passed");
}
@Test(priority = 3)
public void step4() {
Assert.fail("test4 failed");
}
@Test(priority = 4)
public void step5() {
System.out.println("test5 passed");
}
}
TestNG test method execution order
Please note that Pangolin relies on test method execution order provided by TestNG, so you have to provide a fixed test methods execution order somehow (e.g. via priorities).
By default, test step names are the same as names of corresponding test methods in your Java code.
To change test step names and some other export settings, a PangolinTestStep
annotation was introduced.
This annotation can be applied to test methods only and has the following properties:
Name | Description | Required |
---|---|---|
name | Name of a test step in TestRail | Yes |
expected | Value for Expected Result field of a test step in TestRail | No |
e.g.:
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.BooleanValue;
import com.agiletestware.pangolin.annotations.Pangolin;
import com.agiletestware.pangolin.annotations.PangolinTestStep;
@Pangolin(sectionPath = "Master\\Section\\SubSection", hasSteps = BooleanValue.TRUE)
public class MultipleStepTest {
@PangolinTestStep(name = "The first step", expected = "Something expected")
@Test(priority = 0)
public void step1() {
System.out.println("step1 passed");
}
@Test(priority = 1)
public void step2() {
System.out.println("step2 passed");
}
@Test(priority = 2)
public void step3() {
System.out.println("step3 passed");
}
@Test(priority = 3)
public void step4() {
Assert.fail("step4 failed");
}
@Test(priority = 4)
public void step5() {
System.out.println("test5 passed");
}
}
And it produces the following test steps in TestRail:
Tip
To map results into an existing test in TestRail, just add id attribute containing an ID of a test in TestRail, e.g.: @Pangolin(id=1234, hasSteps = BooleanValue.TRUE)
- Adding arbitrary attachments to Test Step in TestRail (since 2.2)¶
In some cases it could be beneficial to have some test data like screenshots or logs accessible in TestRail, so Pangolin supports adding arbitrary attachments to test results.
Adding PangolinTestNGListener into pom.xml¶
To be able to upload attachments, the com.agiletestware.pangolin.annotations.testng.PangolinTestNGListener class should be added into Maven Surefire plugin config as listener in addition to com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19</version>
<configuration>
<properties>
<property>
<name>listener</name>
<value>com.agiletestware.pangolin.annotations.testng.PangolinTestNGReporter,com.agiletestware.pangolin.annotations.testng.PangolinTestNGListener</value>
</property>
</properties>
</configuration>
</plugin>
Adding attachments in the code¶
To add an arbitrary attachment to your test results in TestRail you can use com.agiletestware.pangolin.annotations.CurrentTest class which provides several static overloaded addAttachment methods:
- addAttachment(File) - adds a given file into step results in TestRail
- addAttachments(List<File) - add a list of files into step results in TestRail
- addAttachment(String, InputStream) - add attachment from generic input stream
Example:
import static org.testng.Assert.assertEquals;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import org.junit.BeforeClass;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.annotations.AfterTest;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.CurrentTest;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\demo", runName = "Webdriver TestNG")
public class SimpleWebDriverTest {
private final RemoteWebDriver webDriver = new ChromeDriver();
@Test
public void testPageOpens() throws IOException {
webDriver.navigate().to("http://agiletestware.com");
final File screenshot = ((TakesScreenshot) webDriver).getScreenshotAs(OutputType.FILE);
CurrentTest.addAttachment(screenshot);
}
@Test
public void testTitleCorrect() throws IOException {
webDriver.navigate().to("http://agiletestware.com");
try (InputStream stream = new ByteArrayInputStream(webDriver.getPageSource().getBytes())) {
CurrentTest.addAttachment("page.html", stream);
}
assertEquals(webDriver.getTitle(), "something clearly wrong");
}
@AfterTest
public void afterTest() {
webDriver.quit();
}
@BeforeClass
public static void setUp() {
System.setProperty("webdriver.chrome.driver", new File("chromedriver.exe").getAbsolutePath());
}
}
Results in TestRail:
- Mapping test method result into existing test in TestRail (since 2.5)¶
Pangolin Annotation package allows you to map results of your test into existing test in TestRail by simply marking your test method or class with Pangolin annotation with id attribute:
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.Pangolin;
public class MapToExistingTest {
@Pangolin(id = 14223)
@Test
public void mappedTest() {
Assert.fail("test1 failed");
}
}
where id is an id of an existing test case in TestRail:
Note, that only 'C' character in the ID value should be omitted.
Mapping all test method results into one test
If you export all methods of a class into one test in TestRail (by specifying hasSteps = BooleanValue.TRUE
), just add id attribute on the class level
Results in TestRail:
- Disabling test results export¶
To disable exporting of test results into TestRail user can either:
- Set disabled element of pangolin_config.xml file to true
<?xml version="1.0" encoding="UTF-8"?>
<pangolin>
<!-- Pangolin Server URL -->
<pangolin_url>http://pangolinurl:9090</pangolin_url>
<!-- TestRail URL -->
<testrail_url>https://testrailurl:8080</testrail_url>
<!-- TestRail user name -->
<testrail_user>username@company.domain</testrail_user>
<!-- TestRail encrypted password: please use http://server_name:port/pangolin/password to encrypt your plain text password -->
<testrail_encrypted_password>encryptedPassword</testrail_encrypted_password>
<!-- TestRail project -->
<testrail_project>project name</testrail_project>
<!-- Whether the Pangolin is disabled or not. If this is omitted, Pangolin is enabled -->
<disabled>true</disabled>
</pangolin>
- Or set disabled attribute of Pangolin annotation to BooleanValue.TRUE for a test class or method:
package com.agiletestware.pangolin.dummytest;
import org.testng.Assert;
import org.testng.annotations.Test;
import com.agiletestware.pangolin.annotations.BooleanValue;
import com.agiletestware.pangolin.annotations.Pangolin;
@Pangolin(sectionPath = "Master\\Section\\SubSection", disabled = BooleanValue.TRUE)
public class SomePassSomeFailTest {
@Test
public void test1() {
Assert.fail("test1 failed");
}
@Test
public void test2() {
System.out.println("test2 passed");
}
@Test
public void test3() {
System.out.println("test3 passed");
}
@Test
public void test4() {
Assert.fail("test4 failed");
}
@Test
public void test5() {
System.out.println("test5 passed");
}
}
Pangolin will print the following message for tests which test results will not be exported:
Pangolin: Skipping com.agiletestware.pangolin.dummytest.SomePassSomeFailTest.test3 method because it's disabled