Jenkins and TestRail Integration Guide¶
Why you need Pangolin
In many organizations, Jenkins is used to build software, run unit tests, and run various kinds of testing frameworks. Example: Selenium, JUnit, Pytest, TestNG, Visual Studio Test, etc. These test metrics are very important and should be reflected in TestRail to show the true health of any project. Agiletestware Pangolin TestRail Connector plugin for Jenkins allows users to integrate any testing framework with TestRail without making any code changes or writing custom tools.
Integrating of Jenkins and TestRail is very easy and requires user to install Pangolin TestRail Connector plugin from Jenkins repository and do a simple configuration
- Prerequisites¶
- Pangolin Server must be installed and accessible from Jenkins host
- Installation of plugin¶
Agiletestware Pangolin Connector For TestRail Jenkins plugin can be installed via the Jenkins update center:
- Navigate to "Jenkins" -> "Plugin Manager" -> "Available"
- Select "Agiletestware Pangolin Connector For TestRail" plugin and click on "Download now and install after restart" button
- Restart Jenkins server
- Configuration of plugin¶
Plugin configuration consists of two parts:
- Global settings configuration¶
Pangolin global configurations are accessible on "Mange Jenkins" - "Configure System" screen:
Field Name | Description |
---|---|
Pangolin URL | URL for Pangolin server. Example: http://some-server:8888 |
TestRail URL | URL for your TestRail instance. Example: https://testrail:8080 |
TestRail User | Name of user in TestRail |
TestRail Password | Password for TestRail user |
Upload Timeout | The number of minutes to wait for the Pangolin server to process the request. 0 means wait indefinitely |
When all fields are set, please click on "Save Configuration" button - Pangolin will check availability of entered URLs and encrypt your password.
- Uploading test results into TestRail¶
- "Pangolin: Upload test results into TestRail" post-build action configuration¶
To upload test results into TestRail, a new "Pangolin: Upload test results into TestRail" post-build action has to be added:
A few additional configuration fields can be accessed by clicking on "Advanced" button:
Field Name (pipeline property) | Required | Description |
---|---|---|
Project (testRailProject) | Yes | The name of project in TestRail to which results should be exported |
TestRail User (testRailUserName) | No | If specified, overrides TestRail user name defined in Global Configuration |
TestRail Password (testRailPassword) | No | If specified, overrides TestRail user password defined in Global Configuration |
TestPath (testPath) | Yes | Path to where Pangolin should test definitions, must contain suite name in the beginning (for single-suite project, suite name is always 'Master'), e.g. Master\Section1\Section2 |
Format (format) | Yes | Format of test results generated during the build. Available values: junit, nunit, testng, trx |
Results File Pattern (resultPattern) | Yes | Provide path to the report file(s) generated during the build. This is a comma separated list of test result directories. You can also use Ant style patterns such as **/surefire-reports/*.xml |
Test Run (testRun) | No | Name of test run in TestRail to which test results will be added |
Test Plan (testPlan) | No | Name of test plan in TestRail to which test results will be added |
Milestone Path (milestonePath) | No | Path to a milestone in TestRail to which test run/plan will be added. E.g. Milestone1\Milestone2 |
Case Fields (customProperties) | No | Values for case fields in TestRail can be specified in this field. The format is [TestRail field system name]=[value 1] and each field name\value pair should start with the new line. E.g.: custom_user_field1=foo |
Result Fields | No | Values for result fields in TestRail can be specified in this field. The format is [TestRail field system name]=[value 1] and each field name\value pair should start with the new line. E.g.: custom_user_field1=foo |
Configuration names | No | Configuration names in TestRail in format: [Config group name]\[Config name], e.g. Operating Systems\Linux. Each configuration name must start with the new line |
Fail build if upload unsuccessful (failIfUploadFailed) | N/A | If checked, the build will be marked as failed if for any reason the plugin was not able to upload the results. This could be due to Pangolin server issues, TestRail server issues, network issues, etc. |
Close Run | N/A | If checked, Pangolin will close the test run in TestRail and archive its tests and results |
Test Name to ID Mappings | No | This field contains mappings between test result and an existing test in TestRail. See details below |
Disable Grouping | N/A | If checked, Pangolin will ignore structure in report files and upload all test cases into one Section, defined by the Test Path parameter. |
Several configuration can be specified
To add more than one upload configuration, just click on "Add More" button.
- "Pangolin: Upload test results into TestRail" pipeline configuration¶
Here is an example of pipeline configuration for exporting test results to TestRail:
pangolinTestRail([
testRailProject: 'project',
testRailUserName: 'user',
testRailPassword: 'encrypted password',
configs:[[
testPath: 'Master\\Section1\\Section2',
format: 'JUNIT',
resultPattern:'**/surefire-reports/*.xml',
testRun:'test run',
testPlan:'test plan',
milestonePath:'milestonePath',
customProperties:'''custom_required_field=Pipelines 2
refs=cc,bbb,ddd''',
customResultFields: '''version=1
elapsed=2h''',
configurationNames: '''Operating Systems\\Linux
Browsers\\Chrome''',
failIfUploadFailed: true,
closeRun: true,
customResultFields: 'version=1, elapsed=5s, custom_result_field=Some value',
disableGrouping: true
]],
])
TestRail password must be encrypted
When pipeline script is used, value for testRailPassword must be encrypted. Password can be encrypted by navigating to Pangolin Server GUI page (http://pangolinUrl) and clicking to Password Encryption link.
- Execution¶
Once the job is configured, simply run the job. Here is Jenkins job output example:
And results in TestRail:
Test Cases:
Test Plan and Test Run:
Created Milestones hierarchy:
- Mapping test results into existing Test Cases in TestRail¶
Sometimes there is a need to map e.g. JUnit test results into Test Cases which already exists in TestRail.
To address this scenario Pangolin plugin offers "Test Name to ID Mappings" configuration fields which is accessible by clicking "Advanced" button:
The only way to identify test result in test report is to use fully qualified name of a corresponding test method, e.g. if you have the following class:
public class LoginFormTest {
@Test
public void loginCorrectCredentials() {
System.out.println("login with correct credentials");
}
@Test
public void loginErrorIncorrectPassword() {
throw new RuntimeException("Unexpected exception");
}
}
and want to map result of loginCorrectCredentials
method to Test Case with ID 42, then use the following mapping string:
com.agiletestware.pangolin.dummytest.LoginFormTest.loginCorrectCredentials=42
- Running TestRail report (since Pangolin Server 3.0)¶
- "Pangolin: Run TestRail report" post-build action configuration¶
To run report in TestRail, a new "Pangolin: Run TestRail report" post-build action has to be added:
Field Name (pipeline property) | Required | Description |
---|---|---|
Project (testRailProject) | Yes | The name of project in TestRail to which results should be exported |
TestRail User (testRailUserName) | No | If specified, overrides TestRail user name defined in Global Configuration |
TestRail Password (testRailPassword) | No | If specified, overrides TestRail user password defined in Global Configuration |
reportTemplateIds | Yes | A list of report template names or IDs to run. Each template name or ID should start with new line. |
- "Pangolin: Run TestRail report" pipeline configuration¶
Here is an example of pipeline configuration for running a report in TestRail:
TestRail password must be encrypted
When pipeline script is used, value for testRailPassword must be encrypted. Password can be encrypted by navigating to Pangolin Server GUI page (http://pangolinUrl) and clicking to Password Encryption link.
pangolinRunReport(
reportTemplateIds: 'Activity Summary (Cases) %date%',
testRailUserName: 'user',
testRailPassword: 'encrypted_password',
testRailProject: 'Pangolin Demo'
)
- Execution¶
Once the job is configured, simply run the job. Here is Jenkins job output example:
When report is triggered, a link to a report is attached to the build page in Jenkins: