Report R4: Midterm Report

LOC and time estimation accuracy (C++ only)

LOC data is taken from programs 2A through 6A; first graph is a depiction of rough-guess estimated LOC vs actual program LOC, showing the best-fit linear regression line through the data points. As the graph shows, the data is not particularly accurate, and the "best fit" approximation is not particularly good.

A more "calibrated" comparison is the comparison of PROBE- generated estimates vs actual LOC; the "actual" numbers are the same as the last graph, but the estimates are those generated from the full PROBE algorithm. A "perfect fit" line is plotted to show how the PROBE estimates varied from a perfect 1-to-1 match

In fact, it doesn't look like we're doing well on accuracy at all, until we take a closer look at accuracy by using the "percent error" equation on the numbers.

A look at the time figures reveals some similar trends:

...and a similar analysis of the time estimation error reveals an extremely similar curve. This makes sense, since the time estimates are made directly from size estimates; they should form similar curves. Still, it's good to see that they are similar-- and that our estimation abilities seem to be getting better over time. Note the very large spike in percent error near the beginning of the graphs; this is program 2a, which was significantly larger than predicted and certainly took a great deal of time.

Defect types injected/removed

Defect types found by the compiler

Defect fix times

It's interesting to note that defects found in test seem to be substantially larger, or at least require longer fix times. The too-simple conclusion is that errors found later in the development cycle "cost more" than errors found earlier. This is probably true, but it's also true that the compiler is catching very simple errors automatically-- errors which couldn't be caught during test, because the program simply won't compile enough to test without them.

Design Review Checkliste

The largest categories of design-injected failures are "missing design" and "interface capability" (which is essentially a lesser manifestation of "missing design"). Less frequent are "wrong name", "wrong algorithm", "wrong expression" (a lesser manifestation of wrong algorithm), and "wrong expression" (a boolean manifestation of wrong expression).

It's tough to know where to start with this. These are all fairly fuzzy things; I'll start with Humphrey's design review checklist on [Humphrey95] page 705, and attempt to expand it as necessary. The problem is, there isn't much necessary-- the "missing design" errors are covered by the "complete" category, and and "wrong algorithm" errors are covered mostly by the "logic" section. Still, I've added a few minor items, shown by emphasis.

Code Review Checklist

Big items injected in the coding phase are wrong name and syntax, with interface use, wrong algorithm, interface capability, and wrong type finishing up. With C++, wrong type is very sneaky, because it's good at automatically converting without any input from the programmer.

Humphrey gives us a C++ code review checklist, so I'll supplement it with emphasized lines (and modifications). I'll try and merge the Eiffel code review checklist with this, noting the obvious places.

Postmortem

Time preparing this report was spent as follows:

I'm not doing an extensive postmortem on this because I don't feel the time involved would justify the reward. If I continue using the PSP, this section will have to be heavily automated, and will differ so much from the current hand-drawn method that it will no longer be recognizable.