This article was written by Tonislava Docheva from Xray.
The role of software testing is becoming increasingly important in the software development lifecycle to ensure that software is released with the highest possible quality. With an increased amount of testing, more advanced test plans and strategies, it is evident that a proper test management tool is crucial. Xray for Jira is a Test Management tool that provides the structure to organize, plan, and report with accuracy on the progress of testing as well as the readiness to deploy.
In this guide, you’ll learn about the most important benefits and features of Xray, including full requirements traceability, integrations with the leading automation frameworks like Cucumber and Robot Framework, and advanced reporting to level up your software test management.
So let’s get started.
Here’s what we’ll cover in this article:
- What is Xray for Jira?
- Why you Need a Test Management Solution like Xray for Jira
- What are some Core Features of Xray for Jira?
- How to Use Xray for Jira (a Step-by-Step Guide)
What is Xray for Jira?
Xray is the fastest growing cloud app for Jira and a Success Story in the Atlassian Marketplace.
It’s a full-featured tool that lives inside and seamlessly integrates with Jira. Its aim is to help companies improve the quality of their products through effective and efficient testing.
More than 4.5 million testers, developers, and QA managers trust Xray to manage 100+ million test cases each month. Xray is a mission-critical tool at over 5,000 companies in 65 countries, including 137 of the Global 500 like BMW, Samsung, and Airbus.
With Xray, managers can enhance agile boards by tracking the requirement status and test execution progress in real-time. It can also generate advanced reporting that can be exported to Docx, xlsx, or pdf using Xporter.
Integration with your favorite Continuous Integration tools, including Cucumber and Jenkins, is straightforward through our built-in REST API.
Why You Need a Test Management Solution like Xray for Jira
For companies still using traditional and outdated methods of test management like Word and Excel, it is becoming increasingly challenging to perform all of the necessary functions of proper testing and management.
Why is it important to have a test management tool? Without a test management tool, or simply using Excel and Word to organize your tests, you can not ensure proper quality management and the quality of your products will inevitably suffer.
Ensure you’re meeting all requirements
A proper test management tool will help you meet your testing requirements in different environments on any device, in any browser, and at any testing stage. Xray for Jira also provides traceability between requirements, tests, and defects while assisting with the root cause analysis that identifies the real cause of a problem or a bug.
Improve productivity with automated testing
Tests can be manual or automated. With automated tests, you need to choose your test automation platform like Cucumber, Robot Framework, or Selenium amongst others, and connect it with Xray using the REST API. This enables you to lessen manual work by automating test cases and covering your story with the right test cases. You can also review the code of an automated test to check what it’s doing, right from your issue tracker. This way, you can easily track your work and all your activities including testing ones in your issue tracker or agile board.
Prioritize your time with Risk-Based Testing
A test management tool like Xray will help you prioritize test cases by giving priority through risk-based testing. It also ensures compliance for auditing, which becomes especially important for companies that are in regulated fields like medical device manufacturers. Before releasing a software product, the test management tool can systematically ease the go/ no-go decision process and provides everyone involved in the software development lifecycle (the testers, developers, analysts, or project managers) with access to the testing progress.
Ensure proper organization and planning
A proper test management tool should define specifications/ authoring of the test cases, including the composition of the test cases and the reusable preconditions, and you can easily do that with Xray. There also needs to be a clear organization of the test cases since it is key to find the test you’ll need afterward. These tests can be organized either in multiple ordered lists or in a hierarchical structure like folders called the Test Repository which is an intuitive and easy way to organize your tests for maximum visibility. Next comes the right test planning which requires to be dynamic, which you can do using Xray’s Test Plan Boards.
Get advanced reporting to make smart decisions
Once testing gets going, there are many things to track, like the execution status, different versions, and requirements coverage. All of this data is available to you using Xray’s advanced reports and gadgets. Share this with your team and other stakeholders to give an overall status of testing progress. Once tests are running, you can evaluate coverage of the requirements/ stories through Xray’s reports and gadgets. Xray’s real-time dashboard will give you all of the information you need to make decisions on releases and your readiness to deploy.
What are some Core Features of Xray for Jira?
A Jira Native app that Integrates Testing into the SDLC
Xray is a Jira native application. It is the only test management app in the Atlassian Marketplace to have a native integration that uses Jira issue types for your testing activities. When you install Xray for Jira, you’ll see new issues for Tests, Preconditions, test set, test repository, test plan, test execution, and test runs.
When you integrate your testing into Jira, the leading agile software project management system, you merge development and testing together under one integrated toolset. This breaks down silos between testing and development teams which are otherwise working in separate tools. When testing and development are using the same tool, there’ll be more collaboration, transparency, and communication.
Providing Full Requirements Traceability
One of the most powerful features of Xray is its requirements traceability. Meeting your requirements is one of the most important aspects of software testing.
With Xray, you get full traceability between requirements, tests, defects, executions. This feature gives you immediate visibility into your testing progress and readiness to deploy.
For companies like medical device manufacturers who have to meet strict FDA and ISO audits, it is crucial to ensure they’ve met all the requirements and to provide proof of the requirements traceability. With Xray, this is as simple as generating the Requirements Traceability Report.
Integrating with the Leading Automation Frameworks
Xray has native integrations with test automation frameworks like Cucumber, Selenium, JUnit, NUnit, and Robot framework to improve your team’s efficiency.
With the popularity of Continuous Integration and Continuous Delivery practices, automated testing is gaining an increased slice of the overall testing specification. This is no surprise since automated tests provide time-saving benefits over manual testing. It makes the testing process faster, systematic, and reduces the effort associated with the execution and validation.
Automated tests are implemented as code. Usually, they are executed during the Continuous Integration process, triggered by code changes, or on a timely basis.
Providing Advanced Coverage Analysis
Understanding coverage for different requirements is a powerful metric that will guide your decision making on readiness to deploy and the status of a release.
Coverage is a key feature of Xray since it gives you the ability to analyze what the status of your issues are. You can analyze coverage by stories, requirements, and even bugs, making it a useful metric that you can customize to your needs.
Whenever talking about statuses, we may be talking about the statuses of requirements, Tests, Test Runs, and Test Steps. Coverage analysis provides real-time information on the status of coverable issues like requirements, so tests and coverable issues can be analyzed from different perspectives or scopes.
Integrating into your CI/CD Pipeline with REST API
With Xray for Jira, you can easily incorporate testing into your CI/CD pipeline.
DevOps and its components, Continuous Integration and Continuous Development are a way to speed up the rate of software delivery by integrating all of the software development functions together, including testing.
Automated testing is one of the most important parts of any CI/CD pipeline because it ensures that testing is reliable, fast, and integrated. Without the proper automated tests that run fast and have good coverage, there can be no successful DevOps workflow.
Advanced Reporting and Gadgets
Xray provides built-in reports that can be used to analyze your project in different ways, including testing progress, current requirement statuses, and others. With these reports, you can look at the bigger picture, like the Test Execution completeness, and hone in on more detailed reports like the Test Sets Overview and Test Plans overview.
Jira itself has gadgets that can be easily added and configured to use Xray custom fields and generate reports such as lists of Test Executions and Test Sets with their current status, among others.
Supporting Agile Methodology with Full Work Visibility
If your testing and development teams are currently working separately, then you’re already familiar with the silos that this causes. Without the proper tools to collaborate, there is often a lack of understanding between testers and developers, confusion on the status of everyone’s work, as well as poor requirements coverage.
With Xray, your testing and development teams have full visibility into each other’s work directly on Jira’s agile boards. This leads to better collaboration, visibility, and transparency. This is an agile mindset that all modern companies should be adopting in order to stay competitive in the ever-evolving market.
How to Use Xray for Jira (a Step-by-Step Guide)
Xray is an app that seamlessly integrates with Jira. When you install that app, you’ll have 7 new Jira issue types that you can use to maximize your testing activities.
Because Xray uses issue types for mapping test concepts, you can organize your projects to best suit your organizational needs. You can keep your current projects separate from Xray, or combine them together.
The typical testing life cycle follows a set of phases that do not necessarily need to be executed sequentially.
For each testing phase, you would use the following issues:
- Plan phase: Test Plan issues
- Design phase: The specification can be done with Pre-Condition and Test issue types. The organization of the tests can be done using Test Sets.
- Execute phase: Test Execution issues
- Report phase: Test Execution issues, along with the built-in requirement coverage reports and some custom ones using Jira tools
Here’s how to kick-start your testing project in Xray:
Specification
To start, you will specify the requirements for your project. Normally, these requirements will need to be reviewed by stakeholders. Once you have your requirements, you want to create tests to meet these requirements.
Creating tests in Xray is easy. Hit the “create” button in the menu bar and choose the “Test” issue type. In the test details, you’ll want to specify if it’s a manual, generic, or automated test. If the test is manual, you’ll write the description, assign a version of your software, and set the assignee and the reporter.
If you want to create an automated test, you can select the Test Type as Cucumber. On Cucumber Type, select either Scenario or Scenario Outline as test type. On Cucumber Scenario, enter the Cucumber Scenario clauses in Gherkin.
Keep in mind that automated tests are executed through an external tool which controls the execution of tests and the comparison of actual outcomes to predicted outcomes.
Organization
Organizing your tests is a crucial part of your daily testing activities. Since your project may have hundreds or thousands of tests, it is imperative that you organize them efficiently so you can find relevant tests quickly. We recommend you organize your tests using Xray’s Test Repository, which lets your team organize all their tests into a hierarchy of folders and sub-folders to improve test execution and visibility of results.
Another way to organize your tests is by using Test Sets, which is a collection of tests with a detailed description or goal. Test Sets are simple, flat lists and are grouped in a logical way. For example, you can group together all of your security tests or high-level business features in a Test Set. There are many combinations and ways to organize your tests, you can figure out what works best for your organization.
Planning
Planning is the activity where you decide your testing strategy, which requirements you want to validate, and how would you validate them. Depending on several factors, you may want to prioritize some of the tests or give more focus on some requirements.
To create a Test Plan, go to the menu and click on “Test Plan.” From here, you can create a new folder, and select all of the Tests you want to track for this Test Plan, for example, in a given version. Once you begin executing your tests, you’ll be able to see the results of your execution in a consolidated view in the Test Plan Board, giving you a quick overview of your testing progress.
Execution
When you are executing a test, you will fill the following fields: Begin Date, End Date, Revision, and Test Environments. A Test Execution is validated if all the Tests that verify the specific environment target passed. In order to execute, monitor the progress, and report the results, each Test has its own Execution page details.
To execute a test, click the “Execute Test” button in your test case. This will take you to the Xray execution screen. If you’re doing a manual test, you will execute your test then come back and set the Execution Status. This will describe the testing progress. You can set the status for a Test Run as TODO, EXECUTING, FAIL, ABORTED, and PASS.
You can also attach evidence as screenshots and notes on your test run. You’ll be able to see the status of your executions on your Jira agile board.
Common Use Case
With Xray, testers manage all their tests and executions as Jira entities with all the power of the customized screens, fields, and workflows. They also do automated testing using the REST API. You can also do load tests, Functional Testing, UI Testing, security testing, and others, as well as direct integration with CI tools like Jenkins and Bamboo. Managers can as well check test coverage status directly from the issue screen in Jira. Xray has enabled more collaboration, eased human-to-human communication, and helped decrease misunderstandings.
The Swiss Federal Railways is the backbone of the Swiss public transport system. Testing is a crucial element for SBB, and they have more than 600 applications. As a railway company, they test both software and hardware and they have over 100.000 test cases and 1.000+ trained employees to help ensure that operations stay on track. In 2018 they underwent an agile transformation, and they chose Xray to help them manage over 100,000 test cases and test management activities.
Conclusion
The benefits and features that we’ve listed above have a compound effect on the quality of your software testing. A test management tool like Xray for Jira alleviates many of the common challenges that software testing teams face.
Through proper organization, you are able to reduce the manual effort and focus on test strategy and risk mitigation. With integrations into many of the existing test automation frameworks, you can reduce manual work while properly organizing your test results.
Finally, with the right reporting and documentation, you are able to extract valuable data, meet requirements, and comply with regulations that require traceability.
If you’re interested in test management tools, you can find Xray in the Atlassian Marketplace and try it out.