Microsoft PowerPoint - The Metrics Minefield.ppt - metricsminefieldkwsqa.pdf

Tags

Top Highlights

• Most organizations have some projects that succeed, and some that fail. • When a project fails • the people on it are dispersed to other projects • the metrics that have been collected for the failed project often vanish • a retrospective (a.k.a. post-mortem) doesn’t happen for the failed project • in summary, it doesn’t get counted and therefore it doesn’t count • yet this might be the most instructive project

• Counting Ideas

• The Metrics Minefield

• The intention is not to suggest that metrics are useless, but to identify some examples of gross dysfunction, and to try to avoid it

• Mine #1

• measurement validity

• Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of objects or events with the intent of describing them.

• How Do We Measure?

• Metric models are based on some comparison, which may be explicit or implicit

• We can • count things

• compare individual things with each other • compare individual things with a reference

• count things over time (rates)

• Why Do We Measure? Implicit in our motive for measurement is some model of assessment, based on some comparison, made in accordance with some observation

• facilitating private self-assessment and improvement • evaluating project status (to facilitate management of the project or related projects) • evaluating staff performance • informing others (e.g. potential customers) about the characteristics (such as development status or behavior) of the product • informing external authorities (e.g. regulators or litigators) about the characteristics of the product

• Quality measurement depends upon our skill at observation, what we’re comparing, and the validity of the models that we’re using for assessment

• Mine #2 Comparisons and assessments aren’t necessarily numerical

• Lines of Code Metrics

• Mine #3 Numbers aren’t as descriptive as words and stories. Words can be vague or ambiguous, but numbers without clarifying words are just as bad or worse

• Mine #4 Many people in our field make sweeping, pseudo-statistical generalizations without proposing a measurement model.

• Mine #5 Most people in the testing and quality field haven’t studied measurement theory or statistics, but some of us are asked to implement measurement programs anyway.