Improving Task Estimation Accuracy

In my opinion estimating how long it will take to write a piece of software is difficult if you haven’t done it before and with software we never have.  The more experience you have, the more you’ll have done similar things and thus the more accurate your estimates will be.  To help build this experience faster, it is important to track how long things actually take. 

Many software developers just start working and don’t really plan.  They have a vague idea what they want to accomplish, but haven’t really thought it through.  This inevitably leads to the task taking longer than expected.  There is a great discussion of this on the Stack Overflow Podcast.  Check out episodes 5 and 16 for the before/after.  There are transcripts available if you don’t have time to listen.  The reason why things take longer is because without careful planning, there are a lot of pieces of work that aren’t taken into consideration.  Basically the programmer makes an unfounded assumption that he doesn’t have to do some work that indeed he does.  In my experience, this is one of the largest contributing factors to things taking too long.

The good news is that most of these unconsidered items can be teased out if sufficient thought is given to the problem.  They are known unknowns.  The questions are usually known, just not the answers.  There are techniques for doing this. 

The simplest is breaking down the problem into small pieces.  On our projects we never let items get bigger than 5 days of work.  This is probably too large.  If you are estimating in chunks larger than a day or two, you are probably glossing over important details. 

Writing a spec in some level of detail is also important.  Be honest with yourself and explain how things will get done, not just that you need to do them.  “Break class Foo into two classes” may not be enough detail.  What are the two other classes going to do?  How will they work together?  My rule of thumb is that a spec isn’t done until someone could pick it up cold and implement a reasonable facsimile of what you intended.  If they can’t, it is too vague.

For estimation of larger projects, there is a process called Wideband Delphi estimation that can help to flush out the underlying assumptions.  It is very lightweight and can be done by a few people in a couple hours for many projects.

Once the task is broken down into small segments and these segments are estimated, it is important to track how long each item really took.  Be honest.  You are only cheating yourself if you report a 3-day work item as having taken 1 day.  This helps to build experience and make better estimation possible in the future.

One more thing to consider is whether you are staying on task.  When it isn’t something we failed to consider, the next highest cause of under-estimation is distraction.  How much time are you actually spending on a project?  If you spend an hour a day working on it and the other 7 hours doing e-mail, meetings, surfing the web, talking to people, etc. you won’t finish on time (unless you account for an 87.5% buffer).

Speaking of buffering, it is my experience that it is easier to estimate uninterrupted time than to try to estimate how much time all of the distractions will take.  On our team we estimate each work item in a vacuum.  This will take 1 day of coding and debugging time.  Meetings, vacations, e-mail and other distractions all fall into another category we call “buffer.”  We usually estimate 30% of an average person’s time will be spent in this buffer state.  This varies with the project of course.  Separating the two allows us to better see where the time is being spent.  It is a much different issue if someone takes 5 days actually working on what was estimated as a 2-day problem than if they take 2 days coding but are distracted with other items for 3 days.