Just like Merging, estimation is one of those things that benefit from being regularly trained, but resists training because of the strong negative error consequences. Fortunately, there are some common advices that transfer the burden from the intuition to rationality, and therefore it can be learned easier.

Read the fucking spec

This sounds obvious, but in order to increase estimation quality, you need to fully understand what is going to be built. In software, everything can be connected with everything else. This means, estimating only a small fraction of the whole spec with no knowledge about the other parts bears high risks. Because there might be one single requirement somewhere in the seemingly unrelated parts, that might require usage of some specific technology or framework that you wouldn’t necessarily assume, so that your estimation (based on another technology) might be rendered useless.

Therefore, read the whole spec. Read each page, each requirement, each sentence. Look at every available design comp, and understand what each of the UI elements is doing there and why. Ask questions. Ask a lot of them. Understand, what areas are not exactly defined. Understand, why they are still undefined, and how could they be defined in the future. Know exactly what you’re going to release at the end of the project.

Grade the parts

From what you understand about the spec, grade the product parts in the following four areas:

• Did that before 
• Know how to do 
• Optimistic challenge 
• The Horror

The Did that before might have a misleading name, because software developers never develop an exact copy of some older software. But still, there are a lot of factors that don’t affect the efforts. Writing a css style for orange button will take the same time as writing a css style for green button, if the only difference is the color. Making a 1:2 column layout is not that much different from the 2:1 column layout. Creating a SQL database storing furniture web shop products and writing ORM code for them is roughly the same as creating a database for a car dealer (when using proper time units, in this case, the weeks, and rounding up).

Let’s define two software parts to be effort-equivalent, if they differ only in factors that don’t influence the efforts needed to create the parts.

To improve your estimation skills, you need to build a catalog of the typical effort-equivalent parts, along with the information how long it took to implement them before. In my case, I haven’t written this information down – I just remember, that “Interfacing to a new web API (PayPal, Credit Card Processing, Facebook, etc)” would normally take me 3 days for the first working version. “Creating a simple UI screen” needs one day and “Creating a complex UI screen” needs 3 days, both meant to be code complete stages before bug-fixing. And “Writing a requirements document” is 2 to 3 pages per day.

And the estimation of Did that before area looks like this: you split it to the effort-equivalent parts you know, and sum up the efforts. The result you get is in units of Project Time.

Time measures

There are four different time measures used during estimation:

• Calendar Time. This is really the calendar time. If we start developing on March 3rd, and the estimated calendar time efforts are 15 calendar days, we plan to be ready on March 18th. This means, the Calendar Time includes weekends, the probability of official holidays, the probability of vacations and sick days, average number of projects developed at the same time, all kinds of interruptions and unproductive time, etc.
• Working time. This is calendar time without weekends, holidays, vacations and sick days, but including several projects at once, all kinds of interruptions and unproductive time. Basically, every day you appear to work counts to the Working time.
• Project time. This is Working time spent on one project. If you’re only doing one project at all, it is the same as Working time. If you have a contingent of 1 day per week for a second project, you might spend 5 Working days in some week, but only 4 Project days in the same week.
• Ideal time. This is the Project time, without any interruptions, without meetings, without smalltalk in the tea kitchen, without helping other team mates, without work-related communication with other colleagues, without writing documentation or managing the bug tracker. This is the most limited resource. As a software developer, I used to have only 2 to 6 hours per day of ideal time on average. There were days I didn’t have a single hour. Most of the days, I’ve managed to have around 4 hours. The rest was filled with the so-called “unproductive” things. Which were most of the time of course absolutely necessary and reasonable things to do, but still didn’t contribute to writing code.

Note that the time measures constitute a onion structure. The ideal time is in its core. Eight hours of ideal time corresponds to two project days, and about 3 working days, and about one calendar week. As for the exact conversion factors, YMMV, but the overall idea is the onion. Also because of the onion, usually you would need to use different time units for different time measures: the Ideal time is usually measured in hours, the Project and Working time in days, and the Calendar time in weeks.

Different stackeholders are interested in different time measures. Product and Marketing are interested in Calendar time. Project management is interested in Working and Project time. Developers value the Ideal time.

The safe zone

The Did that before and Know how to do constitute the safe zone. As previously discussed, with the Did that before you would typically estimate in Project days (or weeks or hours), because this is the time you can easily remember when building your own catalog of effort-equivalent parts.

The Know how to do part is different. This is the part you’ve never done before, and you don’t have any effort-equivalent part that is similar enough. But you still can immediately create a possible solution or at least an approach how to solve this task, using the programming language constructs and frameworks you already have and know. A typical example of this might be parsing of some unknown but documented file format (eg. MP3 to obtain the ID3 tags). You know that the ID3 tags are written somewhere in the file, and that there should be some file markers to detect their exact location. So you plan to learn how to open the file in binary mode, print the ID3 documentation, put it on your table, and then code the walking through the bytes of the MP3 until you read all the tags, and oh by the way you will define some data structure to hold the results. Nothing too fancy, especially if there are no requirements to make it as quick as possible, or support broken files, etc.

So, to estimate the Know how to do part, you create a possible solution, then mentally go through each coding (and unit testing) step and intuitively decide how many ideal hours you’d need for them. Then you add some buffer just to be sure, and here you go, you have the estimation. This time, it is measured in Ideal time though.

The fear zone

In the fear zone, there are the areas of the Optimistic challenge and The Horror.

Optimistic challenge is something you’ve never did before and you can’t create a solution, but still optimistic to be able to estimate it with some reasonable (but not too high) precision. For example, I never developed Flash apps, but I know it is a client-side programming (and I did WinForms, HTML/JavaScript and Silverlight) and I know it is from Adobe (and I did Adobe InDesign scripting). So I just add some time for getting into technology details, installing needed tools and finding authoritative information sources in the Internet. And estimate the rest, as if it was a Silverlight app. I also check the Internet to search for the things that are similar to the one we want to build, and if I find any, I have a confirmation that the task is doable. And if it is doable, I can do it, because I have more experience than many Flash developers.

The time measure of Optimistic challenge is a little complicated. In the example above, I will estimate it as if it was a Silvelight app. If I did effort-equivalent things before, my estimation will be in Project time. I will then add some Project time for getting into technology, and get the result in Project time. If I merely know how to do a similar Silverlight app, my solution will be in Ideal time. I will then have to convert it to Project time, then add up the Project time for getting into technology.

This means, I must know how to convert from Ideal time to Project time. This is a factor (similar to Velocity in agile methods) that you can easily calculate from the past activity – estimate its ideal efforts, and compare them with the actual spent project time. The difference to Velocity is that a) you do it for yourself, not for the whole team and b) you’ll get the Project time, not number of Sprints as the output.

The Horror area is completely different. You’ve never done something like this before, and you don’t even know what it takes to implement it. This is like, implementing a new video codec if you only know how to develop web pages. Or implementing a HTML5 mobile web app, if you only know how to program video codecs. When developers meet with The Horror area, they normally become very aggressive and cynical, and start bashing everyone around them. Perhaps, this behavior is due to pressure to perform? We want to be able to estimate anything, and not being able to do it with The Horror area is a hit for our self-image and self-expectations.

The interesting thing is though that very often, stakeholders don’t need a time estimation “at zero price”. Therefore, the correct thing to do is to describe exactly, where your Horror area lies and why. This will result in a conversation with the PM, and some of the possible outcomes might be:

• Getting another team member who is knowledgeable in this area.
• Removing this feature from the project scope.
• Giving you some days or weeks to get the know-how and to try out some things.
• Sending you to some courses or training to buy the know-how.

All of these outcomes are indefinitely better for your organization than just making a random guess and estimating The Horror area just to come to any number. Do not estimate The Horror. Talk about it.

Spec estimation

So, you’ve read the full spec, you’ve graded it, you’ve determined The Horror part and discussed it with the PM. The next step is to estimate all other parts, and to convert them to some common time measure. As developer, I prefer to convert everything to Project time: estimating Working or Calendar time might be infeasible, because the estimation might happen at the time when some important data about the number of parallel projects and the project begin day is unknown. When everything is summed up, I add some buffer to it just to be sure. This buffer cannot save me in 100% of cases, but still seems to be good idea, because I might forget to estimate some software parts or activities (such as deployment or data migration).

Team estimation

So far we’ve discussed estimation are a personal activity. When the estimation has to be delivered by a whole team, there are two factors to consider.

The first is the variation of estimation. It is very easy to explain. Not everybody is as good at estimation as everyone else. This is because not everybody has read this blog post (to the end) ;)

And even if everybody was good at estimation, everyone has different professional experience: things that are The Horror for one team member, might be in the Done before area for another team member. What for one is Know how to do, for other is the Optimistic challenge.

This is also the reason why it is not the best idea to get estimation from every team member and then average them out. The better way is to let the team members speak, and to check if the lower estimation is due to underestimating some tasks, or the higher estimation is just because someone is way too far in their fear zone. Also, combining estimation and task assignment is a good idea. It wouldn’t help if the team member A estimates (and actually has experience to do) a task for 3 Project days, if this team member won’t get this task. And the team member B who’ll later actually get the task, estimates (and actually needs) 8 Project days for it.

So, for the best team estimation:

• have each team member to grade each software part
• discuss the grades and assign parts to members to minimize their fear zone
• get estimation from each task owner
• sum them up and add the second buffer

The first buffer is added by every developer for his own estimation. The second buffer is different – it is added to compensate eventual shifts in team (sick days or other changes) meaning that not the original task owner will have to implement it.

Another thing to consider during the team estimation is the team size. The larger the team is, the higher are communication costs. This means, every team member will get less Ideal hours per Project day. My personal experience with geographically distributed teams makes me believe it costs 10% of the Ideal time for every other person in the team you have to communicate with actively. So, for the team of six persons, each of them will spend 50% of their Ideal time for communication. Which means, you need twice as many Project days for the same Ideal hours. Or, the total 6-person team productivity is only 3 times of a single developer productivity. And yes, teams of more than 10 persons (who have to constantly speak with each other and develop in the same set of files) are counter-productive.

Phase estimation

So far, in the examples I’ve used only the software implementation phase. But you can estimate everything, as long as it is not in your Horror zone. And the trick is, you don’t really need to do or implement something yourself to be able to estimate it. For example, after the code complete milestone, the testing phase starts. It is actually similar, no matter if you use Waterfall, Agile or something else. The differences lie only in when testing starts, and how much it is to test at once. But generally, you can watch how much time testers need to test the part you’ve implemented (and you to bug-fix it until it passes all tests). And you can add it into your catalog. In my catalog, there is just one simple record: “testing and bug-fixing phase” – takes the same time as the implementation phase. YMMV.

Another typical phase after or in parallel with testing is the customer feedback phase. It might consist of some more testing (in realistic scenarios), as well as of some change requests or improvements that didn’t make to the spec or were recognized only after using the actual software for the first time. In my catalog, there is the record “Customer feedback per UI screen” – on average two to tree smallish change requests, costing around 3 Project days extra.

Estimation Training

Estimation training is not different from any other mental training. Write down your estimations. Implement the software. Track the time really spent. Compare. Repeat.

Eventually you’ll learn, what factors are stable and can be reliably predicted, and what are volatile. You’ll become a better estimator.

And in my experience, being a better estimator is highly correlated with being promoted.