Quality Assurance
We describe how NSL engineers resolve bugs reported by users, including CBC personnel. For each bug report, four NSL engineers involve in resolving the issue: Project Manager/Architect, Support Engineer, Test Engineer, and Software Engineer. Their responsibilities are described as follows.
- Project Manager/Architect: He oversees the direction of product development. He monitors the bug database, as well as the workload of individual engineers. He reassigns bugs to avoid unnecessary delays. E.g., a bug assigned to an alumni should be immediately reassigned. He also discusses any design and even implementation issues with software engineers.
- Support Engineer: He helps customers to fill out proper bug reports. A proper bug report should contain enough, but not too many, details. This requires a significant amount of discussions, considering most of the bug reports will be extremely vague. A vague bug report is not useful for resolving the problem, but rather is wasting our resource. Occasionally, a support engineer needs to login to customers' networks (such as CBC's Intranet), for diagnostics. In addition, he may need to set up a small testbed in NSL. However, for a larger-scale testbed, a support engineer should always work with a test engineer to accelerate the process.
- Test Engineer: He reproduces bug reports based on the info given by a support engineer. He clearly documents how to reproduce bugs, which includes the proper testbed set up. The description of the testbed should be crystal clear so that a software engineer may augment the setup for a more controlled environment, such as in a debugger or on a virtual network. A test engineer performs the same test upon the bug has been fixed and checked-in by a software engineer. A test engineer should never accept a patch that is not in the version control system. A test engineer documents the test results for the patches. He closes the bug report if the patched software passes the test. Otherwise, he assigns the bug back to the software engineer.
- Software Engineer: He fixes the bug based on the description given by customers, support engineers, and test engineers. He sometimes needs to set up a large testbed for non-reproducible bugs. Most of time, his test setup is smaller than what was developed by a test engineer. A software engineer discusses any structural issues/changes with his architect. A software engineer checks in his code changes, and returns the bug to the test engineer to validate the patch.
We describe the typical life-cycle of bug reports as follows. A customer creates an account in our bug database using web browsers if he/she hasn't done that before. He/she creates a bug report, and fills in as much info as possible. Our bug database automatically assigns this bug to a support engineer. This support engineer helps the customer to complete the bug report in case more info is required. The support engineer then assigns the bug to a test engineer. The test engineer develops one or more test cases for the bug, and clearly documents the test cases. If the given info is not sufficient to develop the test cases, the test engineer returns the bug back to the support engineer. Otherwise, the test engineer assigns the bug to a software engineer. The software engineer fixes the bug with the help of test cases developed by the test engineer. The software engineer may consider some test cases are invalid. He discusses and resolves this with the test engineer. Once a solution is found by the software engineer, he makes and checks-in the code changes to our version control system. The software engineer documents what has been changed in the bug database. He then reassigns the bug to the test engineer. The test engineer checks-out the latest software from the version control system. He re-runs all the test cases, and documents the results. He closes the bug if all tests are passed, otherwise, he documents his comments and returns the bug back to the software engineer.
Most of the time, the manager/architect does not directly involve in the bug resolving process. The only exception is for those bug reports that are tagged as new feature or functions as designed. A bug with new feature tag should be immediately directed to the manager/architect. A bug with functions as designed tag should be sent to the manager/architect (by the test engineer) before being closed. The manager/architect validates whether this bug is indeed an invalid one, and closes or reassigns it accordingly.
