Core Conversion Misstep #1: 1,500 Test Cases Are Missing

Letter dice spread out to spell "Risk"

This blog is the first in a series about things that are frequently overlooked or forgotten when a Credit Union is considering a Core Conversion. Subsequent blogs will be posted weekly and highlight each item in-depth. Make sure to follow along with the series on


Your core conversion planning is anxiously underway, and many months will be spent planning and executing testing. Often, a decision about the tool that will be used to house test scripts and track defects is made just as test planning is kicking off, which could end up being too late. Your test and defect repository will be as critically important as your overall project plan, so you need to have the right tool selected as early as possible.



There are two sets of testing data that need to be tracked during any test effort: the test cases and the results of executing the cases. While some organizations use home-grown tools, there are also many tools available in the market.  The important thing is documenting the work that needs to be done, tracking the outcome, and reporting the status to stakeholders.


For simple software projects, Excel is often used. However, Excel will have significant limitations if used for a core conversion project. Charts and graphs, which are used for analysis and communication, will need to be created and maintained manually, taking time away from more valuable project activities. Excel search functions are limited, particularly if multiple tabs or workbooks are used to separate different types of testing or test cycles. It will also be difficult to track current test and defect results against historical results to see how test progress and defects are trending.


The complexity of core conversion projects inherently means increased risk when compared to many of the other projects your credit union has executed. Thorough planning and tracking will be used to see and understand your biggest testing and defect risks.


Consider that your testing team will create hundreds of test scripts, all carefully designed to satisfy the myriad requirements of your core conversion project. If you are using an Agile project management methodology, user stories will also be written to ensure you are testing every critical and high priority function in your new core, as well as a portion of your medium and low priority functions.


You will need a tool that captures important metadata about each test script such as the test script’s author, the date it was written and the type of test being executed. Some of your core processes will be changing while you are writing and executing test scripts, so dates of authorship and versioning will ensure that the most current processes are being tested.


While some scripts will have relatively simple workflows, more complex tests may require dozens of steps. The tester will need to document the outcome of each step. Robust tools are designed to easily capture and attach the objective evidence of a successful or failed test.


You will also need (and want!) to convert your conversion test script suite into a regression suite for use in testing core upgrades and releases. The right tool will help to do this efficiently and with minimal effort.



There are multiple repository options available, from basic Excel functionality to the robustness of Micro Focus Application Lifecycle Management (ALM), Microsoft Azure DevOps (ADO), formerly known as TFS and VSTS, and Jira.



Several other robust tools compete with ALM, ADO, and Jira, but these are all well-known. They offer a full suite of testing functionality – requirements traceability, test planning and script tracking, execution details, and reporting and dashboards. They also have integration capabilities with other software applications.


A key piece of functionality in a robust tool is defect management. Some of your test scripts will encounter failures, as mentioned earlier, which are documented as defects that get assigned for research and solution, retested, and closed. Recording and tracking application and environment defects are critical to ensuring that they do not escape into your production environment. Production defects are far more costly to address than those that are identified and resolved in a test environment, particularly if they are visible to your members. If your repository does not include defect management functionality, you will need a second application for this.



There are clear pros to using a single application for test and defect management. When using multiple systems, you risk miskeying test information in the defect system or forgetting to log a defect altogether. A consolidated tool can identify test cases with a status of “failed” but no defect information, ensuring every defect gets logged.


With a consolidated system, you will have visible links between requirements, test scripts, and defects, making traceability easier to share and trends easier to identify.


Integrated reporting for stakeholders is often easier with charts and dashboards that summarize defect status with a comparison to test script status. Even when reporting test script and defect status separately, you will have the same reporting and dashboard format and easy filtering.




A chart showing the results of a team’s test plan in Microsoft Azure DevOps.


Lastly, there is less effort to manage, govern, and integrate these important applications when you are using one piece of software to accomplish both goals.


While there is a strong case for using one tool for test and defect management, there may be functionality in a stand-alone defect tracking tool that you do not find in a consolidated application, such as better functionality for tester and developer communication. You may also find the defect reporting and dashboards to be more refined in a stand-alone tool.


If your credit union needs a more robust repository solution to support your conversion project, you will need resources to identify the right tool for your environment, get it installed, and train your users.



The right tool(s) for your credit union depends on a number of factors. Robust tools can offer a wide range of valuable functionality, including mobile support, dynamic reporting, and monitoring to ensure no test case is left behind. You will want to weigh the value of those benefits against the cost of a new tool.


You should get a demonstration of each tool, so you can evaluate if it meets your project needs. You will want a tool that allows you to easily search for, or filter, test cases. If you want to import or export test case data and customize field values and field locations on screens or report layouts, ask to have that included in a demonstration.


Some tools are licensed and others are sold at a flat fee. If your IT department has limited resources, you may want to select a licensed tool so you have access to technical support.


Your organization may already have a defect tracking tool that is well-known to your testers and developers. You will need to weigh your current financial obligation to that software provider, if any, and user willingness to adopt a new software.


Most importantly, you will want to consider your credit union’s Quality Assurance (QA) strategy. The goal of a test and defect management repository is to supplement or improve an organization’s existing QA processes and functions. Will a robust, highly refined testing tool be valuable for other project initiatives? Is your organization advancing through the evolution of QA functions as you grow?


There are organizations that can assist with evaluating your test maturity model, aid in making educated tool-selection decisions and help evaluate current QA processes.



Sadly, test and defect repository tools do not manage or govern themselves. (We’re still waiting for that artificial intelligence!) You will need both short and long term resources for the work that is specific to your conversion project and the ongoing management of your repository tools. Well-developed processes will be important for the many people that will be dependent on your test process and tool.


A strong test lead or manager will own the QA and test strategy, requirement coverage/ traceability, testing risk assessment and communication, along with how the repository tool will be used to develop reports and identify test and defect trends. The test team and developers will be dependent on processes that ensure efficient use of their time and tools.


Your test lead will also establish defect severity and priority definitions and processes for defect triage, assignment, tracking, and retesting. For a core conversion project, a brief, daily defect review meeting is often used to ensure the testers and developers are aware of priority, trends and the work to be accomplished that day. Your repository tool can be used to facilitate the discussion.



If your organization commits to a new repository tool for use beyond your core conversion project, resources will be needed for periodic maintenance, user support and training, and software upgrades that will change the tool’s functionality over time.


No matter which tool you select, seamless integration within your credit union’s Software Quality Engineering ecosystem will make the difference between a clunky, overwhelming and frustrating experience and a highly efficient test planning, test execution and defect management experience that ensures successful project delivery.


One of the most critical components of a core conversion project is testing, and a sound repository decision – made at the right time – can go a long way in effectively supporting your test strategy and execution.


In addition to a test repository, an often overlooked component to a successful core conversion is performance testing. Next in this blog series, we will look at how performance testing is used to ensure your new core conversion can handle the weight of your business.


Olenick is a global software testing firm with headquarters in Chicago, IL. Special thanks to Steve Woods, Joe Culotta and Flip Nehrt for contributing to the content of this blog post. Learn more about them on LinkedIn.



Related Content: Change Management, Credit Unions, Functional Testing, Microsoft Azure, Quality & Testing, Requirements Management, Test Management, Test Tool Management Strategy

Don't Miss An Olenick Article!

Subscribe to receive our latest blog articles, right to your inbox.