How do you estimate that which you’ve never done?

Have you heard about #noestimates? No? Well I’m sure you can guess what it is anyway. But reading the debates reminded me of a story.

While at Game Developer’s Conference a few years ago, I was arguing about estimation with a certain project manager, who, despite having no actual development experience, was in charge of development (Icelandic society is notoriously nepotistic).

“So, maybe no estimation works for your small projects, but when you have to do big projects, and you need to ask for budget, and coordinate many departments and offices, and you need to plan all this in advance, what do you do? How would you plan Incarna?”

Incarna was CCP’s expansion that introduced avatar/character-based “gameplay” into EVE Online. What shipped was the ability for your avatar to walk (not run!) around a room. It was massively over budget, behind schedule, and under delivered. A few months later, 20% of the company was laid off. There’s been no active development on Incarna since 2011, and World of Darkness- which continued to use Incarna’s core technology- was cancelled and the team laid off earlier this year. It was, quite simply, the biggest disaster I’ve seen or heard of my career.*

A character-based game is also something CCP had never done before. They are massively- MASSIVELY– more technologically complex than the “marbles in viscous fluid” EVE flight simulator. CCP did not have the in house experience, especially in Iceland, where most of the (very smart) engineering team had never worked on character based games.

So it was pretty hi-larious that a project manager was using Incarna as an example of why estimation is necessary. But cognitive dissonance is nothing new. Anyway, my response was:

“You don’t plan Incarna. You greenlight a month of development. At the end of a month, you see where things are. Do you keep going for another month? If you are happy with the spend and progress, keep going. If not, pull the plug. Once you can make a prediction at the start of a month, and it holds true for that month, and you do this two times in a row, maybe make a prediction for two months and see how it plays out.
You may pass a year this way. Well, a year isn’t a long time for developing a character-based MMO and game engine from scratch. But at the end of the year, you at least have some experience. But you keep going. If your velocity is consistently predictable, you estimate further out. Eventually, if you can get your velocity stable at the same time you’re growing and developing, you have a fighting chance.**
When your velocity isn’t stable, you reign things in and figure out. If you go through a year of missed month-long predictions, you need to change things drastically (or reboot entirely) if you hope to get something predictable.”

Nothing really insightful there of course- I’m just parroting what has worked me me and many, many others, from Lean-inspired methodologies (and this one in particular says traditional yearly budget cycles are responsible for many terrible business decisions).

A couple months ago I was asked if a significant new feature could get done by June. It would build on several months of foundation and other features. I responded that I was pretty confident that if we aim for June we would have it by September. My rationale, simply, was that previous similar projects shipped 3 or more months late, and I didn’t have enough experience with the team to give a more accurate estimate.

The best predictor of future behavior is past behavior. You need to create historical data before you can extrapolate and plan.

The historical data also needs to be “meaningful.” That is a much more nuanced topic, though.

* It should go without saying that disasters the scale of Incarna are 100% at the hands of management.

** On Star Wars: The Old Republic, management took an interesting strategy of driving velocity into the ground so that while it was terrible individually, it was at least stable. They could then increase the number of people resources and predict, pretty reliably, when it could ship. The game ended up costing about $200 million (I suspect much more, actually), but it wouldn’t have shipped otherwise.