Test Process Measurements and Improvement

Course available to schedule onsite

Concepts

Testing effectiveness is determined by one's testing process - the way we do things that produces our results. To improve test results, we must improve our testing process; and that requires meaningful, valid, and reliable objective measurement. Defect data are the most obvious testing artifact to measure, but they must be put in context and coupled with other key factors that are critical to enabling improvement. Not only do we tend to miss important measures, but we also often overwhelm ourselves with too many measurements. This interactive presentation describes a minimum set of metrics that testing needs to know, ways to make those measures within appropriate contexts, and methods to analyze and report to guide improvement. Techniques are suggested for overcoming resistance when getting started. Exercises enhance learning by allowing participants to practice applying practical techniques to realistic examples. This course has been designed for test, project, and system managers, as well as for te sters, analysts, designers, programmers, auditors, and users who are concerned about the efficiency and effectiveness of testing within the overall development process.

Participants will learn:

Relation of measurement to processes and distinguishing real from presumed processes.
Key variable dimensions that a measurement set needs to address.
Relating testing project measures to assessing, managing, and improving the testing process.
Ways to analyze and evaluate effectiveness of testing practices.
Presenting and reporting measurements so people pay attention.

Outline

Measurement and Processes
- What a process is, and is not, why it matters
- Distinguishing "real" from "presumed"
- Defined and documented processes
- Relation between process and measurement
- Silos, measuring a process to full end result
- Measuring results vs. guiding improvement
- Non-procedural process components
- Key measurement principles, core measures
- Validity, reliability, materiality, Hawthorne
- Project and product vs. process measures
- Measures vs. metrics, meters and predictors
- Goal-question-metric (GQM), process goals
- Why most measurement programs fail
- Avoiding resistance to measurement
Defining the Testing Process
- Test execution time, effort, quality measures
- Test automation, support, administration
- Test cycles, defect fix turnaround time
- Impact of level of quality required
Outcomes, Analysis and Reporting
- What is a defect; fault vs. failure, duplicates
- Operations and performance measures
- Reliability, durability, usability
- Incident/defect categorization, tracking
- Defects found vs. number projected
- Defect age, source including testing errors
- Charting over time, projecting delivery date
- Defect patterns, causal analysis
- Test sensitivity, detection rates and yields
- Find and fix time, effort, cost by phase
- Defect detection efficiency
- Historical profiles, subjective view of value
- Evaluating effectiveness of test techniques
- Reporting, using measures to improve
- Key testing process procedural elements
- Proactive Testing^TM Life Cycle
- IEEE Standards for testing, documentation
- What should be measured, how to do it
- Testing/development good metrics practices
- Statistical process control, capability
- Externally-defined software process models
- Test maturity, CMM-based vs. TPI models
- Test effectiveness indicators
- Benchmarks, surveys, assessments adequacy
Testing Activity Measures
- What is a test case, basic unit of size
- Development sizing, derived test sizing
- Test case coverage measures, limitations
- Functional test coverage, traceability matrix
- Degrees of structural, white box coverage
- Dynamic vs. active, passive static testing