Defect Management: What Is It, and Why Do We Need It?
What is Defect Management?
You are starting a project and know some defects will probably be created. You also know that the person who found the defect will tell a developer. The developer will fix it right away, test it to make sure it works right this time, put it back into the application code, and all will be good with the world. Right?
Well, not so much. Pretty much every piece of that puzzle has problems. Yes, the testers will find defects– that much we have as a given. However, how do we know that defect exists? How do we know a developer was told about it? How do we know they are working on fixing it rather than something they feel is more important? How do we remember to check to see if it was fixed? When the developer does get around to fixing it, how do we know it gets back into the application code? How do we know that it now works and did not break something else in the meantime?
Simple answer: without a Defect Management process, we don’t.
Just what is Defect Management? We see it as a small (but very important) piece of the full Quality Assurance umbrella. This tail end of the QA process controls defects from when they are found through when they are closed, sets up the Defect Management environment before testing begins, manages and tracks the defects (and resources working on them), and reports on defect metrics throughout the process. Who is in charge of doing all of this?– The Defect Manager. This individual, whether part time or full time, needs to be the one main resource for making sure the process works smoothly from start to finish.
Without the Defect Management processes in place, we risk a defect not being fixed and that defect escaping into the wild of production. All the pieces of Defect Management are put in place as protection against that happening.
So, rather than assuming that defects will be handled well, we need to assure it.
We do this with planning the execution of Defect Management and executing the plan. Steps that you will want to ensure are executed include:
- Setting Up Your Defect Repository
- Publicizing and Training the Teams on the Defect Process
- Ensuring the Defects Are Entered Correctly Into the Repository
- Prioritizing and Assigning Defects During Triage
- Tracking and Accurately Reporting the Defect Population Lifecycle
- Verifying That Each Defect is Retested in a Timely Manner and Added to the Next Build
- Identifying Key Trends Hidden in the Defects
Each step needs some explanation (and several could generate a blog by themselves), so let’s take a look at them at a high level for this blog’s purposes:
1. Setting Up Your Defect Repository
The first step, before any defects need to be logged, is to set up (or verify/modify) the tool you are using for a defect repository – where defects will be logged, tracked, assigned, reported from, and closed. This is one of the steps that could use a blog all for itself, but let’s hit the highlights here:
- What does the defect form look like?
- Do you have all the necessary fields on the form for assigning, tracking, and reporting the defect?
- Are they laid out cleanly, in an intuitive flow?
- Are the correct fields made required or read only, or alternately not required or editable?
- Are drop-down fields fully and correctly filled with the list of possible values?
- Do you have (or should you have) different access rights to fields depending on the role of the user accessing the defect form? For instance, you may not want your developer to be able to close out a defect but only a tester or administrator.
2. Publicizing and Training the Teams on the Defect Process
Now you have everything set up correctly in your repository tool. But, does anyone know how to use it? You will need to set up one or more Archival Documents (PowerPoint, Word, etc.) for all future audiences and one or more Training Documents (most likely PowerPoint or something like that) for live classes to users. I say one or more, since there may be one for how to enter Test Cases for execution, how to execute those test cases, and then (as pertains to this blog) how to write defects found while either executing test cases or during exploratory testing (fooling around with the application).
The first set of documents – the descriptive ones – will be the most complete, having verbiage and images about every aspect of using the tool and creating defects. Those are for reference by any new or existing team member. The second set – the Training Documents – are more for classroom training of the team member doing the testing and step–by–step on how to log and update defects.
3. Ensuring the Defects are Entered Correctly Into the Repository
So, your testers are ready to start testing and logging defects, but it is up to the Defect Manager as part of the Defect Management process to ensure, correct, and “re-train” the users on how best to enter defects into the system if necessary. These could be corrections to selections (modifying the testing cycle or environment, entered incorrectly by the tester, for instance). Ultimately, though, it is part of the process to ensure that the fields are correctly filled out.
4. Prioritizing and Assigning Defects During Triage
Triage? Where did this come from? Simply put, the triage meetings should be held as many times as twice a day to keep the workflow of defect refactoring flowing smoothly, depending on the number and criticality of the defects. Triage meetings need to be in place for two reasons, to determine that:
- New defects are filled out and prioritized correctly, assigned to the correct resource as a next step, and that key people are informed of critical and high severity defects.
- Existing defects are reviewed, and it is verified that there is movement on them, that they are on their way to being fixed, and not forgotten in a corner somewhere. A review of critical and high severity defects needs to be part of the triage meetings to ensure action is being taken on them, and if nothing has been happening to contact the appropriate resource to find out why.
The appropriate people need to be in each of these triage meetings. These would include the Defect Manager, owners (department heads) of the application area(s) being tested, and key development resources in those same areas.
5. Tracking and accurately reporting the defect population lifecycle
Metrics describing the overall status of the defect refactoring effort may not directly address the fixing of any given defect but is invaluable to making sure that attention and effort is being directed properly and that resources are being focused appropriately. As the saying goes, you cannot fix what you do not measure, and measurement is key to determining if and where things might be going wrong.
In order to create metrics useful to Senior Management to make decisions, the proper fields need to be tracked in the defect form. These include, but should not be limited to:
- Defect Summary or Title
- Status or State
- Defect Type
- Defect Location
- Date Created
- Date Closed
- Closed Reason
- Root = Cause
- Detected By
- Assigned To
- Cycle/Release Detected In
- Environment Encountered In
- Business Process, if tracked
- Owner, either a resource or a department
Once your process includes the tracking and proper updating of all these fields, a variety of metrics can be generated and distributed to allow Senior Management to act if needed to address issues.
These metrics may include the following and many more, depending on what the management desires:
- Current number of New, Assigned, Fix in Progress, Ready to Test, Closed defects
- Simple number of open and closed defects by severity
- Velocity of Opened defects (how many opened per day, how many resolved per day, number currently open)
- Duration of defect lifetime (how many days, on average, defects were open before resolved, broken by severity to compare against SLAs for refactoring, if that is being tracked)
- Defect Status by Team/Application/Functional Area by Status/Severity
- Defects by Defect Type/Root Cause
- Count of defects over time by Team/Application/Functional Area
The list could go on and on. Note that each of those listed above are suggestions but be creative. TALK to Senior Management to determine what they need to see, and how they need the data presented. Tables, charts? What colors in the charts makes sense to them? Bar, column, pie, line or doughnut?
6. Verifying That Each Defect is Retested in a Timely Manner and Added to the Next Build
Once the defect state shows that it is ready to retest, ideally the original tester who found and logged the defect should retest it to verify the application now works correctly. It now should pass the test cases that caught the defect and test around the defect to verify that the fix did not accidentally break something else. Note that the title of this section includes the words “in a timely manner.”
A good suggestion is that every morning when the testers arrive, have grabbed their coffee or tea, and are ready to get started testing again, they first open up the defect repository and look for any defects that are assigned to them and are ready to test. They test those first, before they are consumed by the rest of the testing effort. In that manner, defects that need retesting are not neglected or sit for too long without being addressed.
7. Identifying Key Trends Hidden In the Defects
Metrics sent to Senior Management are one thing. The management may be focused on specific numbers on which they want to act – such as number of Critical Severity defects, but they only know what they know – that is, they are only aware of the numbers they have asked for.
It is part of the process of Defect Management and the job of the Defect Manager to scour the data in the defect repository and find trends that point to potential issues and communicate those to the Senior Management. For instance: if the Defect Manager notices that there happens to be a long delay once the defect gets to the developers’ desks or once they are ready for retest, that they sit in that state for a long while, and this is not something that the Senior Management is currently tracking, it is worth mentioning that this is a bottleneck and may need to be addressed.
The Defect Manager may find that many defects are being logged for poor data or perhaps coding errors, due to one developer in particular – this sort of defect reason or root cause needs to be reported up. These are not trends that are normally going to come out in standard metrics requested by Sr. Management, but part of the Defect Manager’s responsibilities within the Defect Management process is to look at the data in every way that they can imagine, pull this type of information out, and present it to the appropriate people if it looks to be an issue affecting the success of the project.
So, there it is – Defect Management in a nutshell.
Get things ready, make sure everyone is trained, ensure proper logging, prioritization and assignment, track and report appropriately, ensure defects are resolved quickly, and identify trends that could impact the project timeline and success.
As I had mentioned earlier, most of these key pieces in the above diagram could have a blog all on their own, as they each hold a very important place in the overall success of Defect Management and therefore of the project as a whole.
Stay tuned for more upcoming articles in this series on Defect Management.