Core Conversion Misstep #5: The Project Is Finished, Right? Nope, One More Step
This blog is the fifth in a series about things that are frequently overlooked or forgotten when a Credit Union is considering a Core Conversion. Subsequent blogs will be posted weekly and highlight each item in-depth. Make sure to follow along with the series on Olenick.com.
Core conversion projects have a seemingly endless number of things to consider. Your project team may have already discussed a test and defect management repository, performance testing your new core, the integrity of your new core’s data, and the downstream testing that is often forgotten.
Next in this blog series of neglected components is converting your test cases into a regression suite. Your new core will have regular upgrades, new integrations, and new development – all work that is designed for you to realize even more benefits. The ongoing testing you will be doing can be made much more efficient with a regression suite, and the best time to create a regression suite is immediately following your conversion.
Regression Testing Benefits
Your new core system vendor will periodically upgrade your core’s functionality. These upgrades will correct non-critical defects from prior releases and introduce new features.
You are also likely to apply hotfixes (typically used to correct critical defects) and implement new integrations that optimize the core’s functionality. Regardless of the reason your core is changing, it is important that you have a regression test suite developed so you can efficiently test as often as needed.
Simply stated, regression testing is used to verify that previously tested software still performs well after a change is made.
Throughout your core conversion project, you will write hundreds, if not thousands, of test cases. A regression test suite is created by taking a subset of those test cases and packaging them together. Often, two regression suites are created – a “baseline” suite and an “extended smoke test” suite. The suite that you use depends on the changes that are being made to your core.
Regression suites have a number of benefits and the most significant is efficiency. If a change is made to your core and you do not have a regression test suite prepared, your testers will need to write new test cases from scratch. That effort is likely to take a fair amount of time and duplicates the work that was already done during your conversion project.
An additional benefit to a regression suite is risk reduction. When rushing to write test cases, you may not spend time creating a thoughtful test plan. Your testers will jump into writing new test cases and are likely to miss important test scenarios. This increases the risk that defects will sneak into your production environment.
Spending a small amount of time creating regression suites now will save you a great deal of time and effort later. You are best positioned to do this immediately following your production conversion so add this task to your project plan and ensure you have resources to execute.
How To Select Test Cases for a Regression Suite
The first place to start is with your test case criticality, which we discussed in the first blog post in this series. Each test case should have been classified as critical, high, medium, or low.
Your “baseline” suite will include all of the critical and high priority test cases and a selection of medium and low priority. You will use this regression suite for upgrades and when you have done any coding or development against the core. This includes third party development using your core’s API.
The “extended smoke test” suite is used for smaller changes made to your core, typically associated with defect fixes or enhancements. This regression suite includes your critical priority test cases. With this suite, it is optional to run all of your high priority test cases.
It is not necessary to include test cases in either suite that are designed to test field formatting, drop down values or screen configurations, and layout. These are not likely to be impacted by upgrades, hotfixes, or new development, and if they were, the risk would be minimal.
Finally, make sure your regression test suites includes end to end test cases, as well as functional test cases. A healthy mix ensures adequate test coverage.
Your Core Has Upgrade Release Notes – What Do You Do With Them?
Software upgrades are typically accompanied by release notes – a multi-page document that outlines the changes included in the release. This will be a mix of defect fixes and new functionality.
Some software vendors plan their development so there is one “major” release per year, packed with new functionality and a myriad of code changes to support it, and smaller releases throughout the year that primarily serve to provide defect fixes.
If the release you are taking is a “major” release, you will run your entire baseline regression test suite. It’s important that all critical and high test cases are run against the new release to see if any defects snuck into the new code.
You will also need to review the release notes. Find the module or functionality you use and scrutinize the changes that were made. Grab a virtual or actual highlighter and start marking up the changes that impact your use of the core.
Cross-check those changes against the test cases in your regression suite. Traceability is key – you want to be able to tie together the module, the change that was made, and the associated test cases you will run.
If your regression suite does not have test cases for the release changes, you can review the full suite of test cases from your core conversion to see if any apply. When doing this, please remember to review those test cases for accuracy. Processes may have changed since your core conversion was completed, requiring you to update the test cases.
If your full suite of test cases does not apply to the release changes, you will write new test cases and add them to the regression suite.
There are two additional things to do at this point. First, consider if you are enabling any new functionality during your upgrade. If so, you will need to write new test cases and add them to your regression suite.
Lastly, based on all of the above information you have gathered, consider the potential need to test integrated applications. As discussed in our blog post about downstream testing, changes to your core can potentially impact many other integrated applications. Make sure you have accounted for those applications in your end to end test cases.
Hotfixes and Enhancements – What Do You Need To Test?
For hotfixes and software enhancements, your extended smoke test suite is your most efficient testing solution. Critical test cases should always be run against any change to your core. The number of critical test cases should be small enough to run manually in a short period of time.
You will also need to consider functionality that exists “around” the fix you are receiving. What else could have been impacted by the change or enhancement that may not be included in your critical test cases? Once you identify those, you’ll need to add test cases for them.
Last, if the change to your core is an enhancement, new test cases will need to be developed and executed that are specific to the new code you are applying. Prioritize those test cases as you write them and if any are critical or high, be sure to add them to the appropriate regression test suite for future test cycles.
Maintaining Test Cases
Your regression test suites are valuable tools for efficiently reducing risk when changes are made to your core, but they’re only valuable if the test cases are accurate.
As processes in your organization change over time, your test cases will need to evolve with them. Implement a process for regularly updating test cases when business processes change and defects are resolved.
If you choose to write procedures or other documentation to compliment your test suite, such as instructions for creating a test case in your repository or guidelines for deciding the test case priority, periodically review those as well.
The Power of Automation
What we will not cover in detail in this blog series, but is something additional to consider, is test automation. The next step in a mature, highly-efficient QA function is reducing your dependence on manual testing by automating your regression test suites.
With a strategy, resources, and tools that fit your organization, test automation can reduce effort, improve accuracy, expand test coverage, and reduce expense.
If test automation interests you, there are organizations that can assist with creating your credit union’s automation strategy, aid in making educated tool-selection decisions, and help evaluate current automation processes.
Our final recommendation for regression test suite conversion is to do it immediately following the completion of your core conversion project. Your team’s memory is fresh for decision-making, and you reduce the risk of other priorities constantly overshadowing the need to develop regression suites.
The creation of regression test suites is a final, highly valuable QA step in your core conversion project. Hotfixes, upgrades, and enhancements will quickly become an exciting, but sometimes painful reality of your new core system, which makes testing your new best friend.
Taking the time to create regression suites immediately after your conversion project concludes ensures that you can test efficiently and frequently, while preserving the integrity of your critically important core system.
We have covered five often overlooked components in this blog series on core conversions: test repositories, performance testing, data integrity, downstream testing, and now, creating regression suites. The final post will be published next week, summarizing the risks and benefits.
Olenick is a global software testing firm with headquarters in Chicago, IL. Special thanks to Mike Mokrzycki and Tony Dusek for contributing to the content of this blog post. Learn more about them on LinkedIn.