Comparing assemble-your-own vs. kitchen-sink frameworks

In my previous post, I wrote about some of the “performance” tradeoffs in build-yourself vs. off-the-shelf approaches as an analogue for remote vs. collocated teams. I’d like to wade deeper into the build-yourself vs. off-the-shelf approaches, in this case going with an “assemble-your-own” framework (Sinatra with Sequel, Flask with SQLAlchemy) vs. “kitchen-sink” framework (Rails, Django). My goal is not to compare the pros and cons of each, but rather to explain why such arguments are missing the point.

I have spent most of my career in video games. Large video game studios often build (built?) nearly everything from scratch. Everything from audio middleware to the game engine itself is often modified at the source code level. Most systems are written in C++, which is difficult to make granular libraries for. All code has strict performance ramifications, so large, critical-path systems that could be generic are often written from scratch. Oftentimes there are proprietary or visual scripting languages for much of the gameplay. There is very little that is actually usable truly off-the-shelf.

The opposite is true in web development. Interfaces like Ruby’s Rack or Python’s WSGI provide an extensible way to add application behaviors. Granular libraries are often interoperable enough because the languages have relatively rich datatypes. Because performance is dominated by IO, we rarely care much about the performance of the libraries we depend on. And of course, we have the order of magnitude productivity speedup: we’re using a garbage-collected language.

Having this perspective, I have always seen debates about building a product with a kitchen-sink framework like Ruby on Rails compared to an assemble-your-own framework like Sinatra with Sequel as sort of missing the point. We’re incredibly productive in either one! Let’s not argue over scraps.

When I look at the entire value stream for a feature/product in terms of effort and time, it seems like programming works out to 30% or so. Other activities include research, explorations, product management, design, build, support, rework, and maintenance. I’m thinking of “programming” here as actual programming, even if a programmer is doing all of the above tasks.

Let’s also be generous and say that with Rails we can slice 33% off of programming time compared to if we “assemble-your-own.” Or you can take the opposite position if you think you go faster when you assemble-your-own, in which case you can invert my examples but the argument is the same. Or go up to twice as fast, it doesn’t really change anything.

That 33% savings in 30% of the total time a feature has devoted to it works out to a 10% reduction. In real terms, that means starting working on January 1 and shipping it Jan 9 vs Jan 10 (1 day in 10), or March 5 vs. March 12 (1 week in 10), or September 1 vs. October 1 (1 month in 10). Has there been a startup that has lived or died based on developing 10% faster? Of course not. This is not how startups succeed. They succeed because they have some fundamentally new idea, or can execute orders of magnitude faster, not because they can execute 10% faster.

There are a few other ways to think about this. Can we save 33% of our engineering headcount? I don’t think so. Engineers do more than program. They are involved upstream during research and design validation, and downstream in support and iteration. In other words, if 3 engineers can work on 3 features using Sinatra/Sequel, 2 engineers can’t work on 3 features using Rails. Each engineer has a high cost embedded in each feature they work on. Of course, we would save some money because the engineers would ‘move faster’. Has there been a startup that would have succeeded if its burn was 10% less?

Will our programmers have to maintain less code? It may seem this way, but no. We maintain the code we write, and then some. We’ll have to maintain the same amount of code in either scenario. Off-the-shelf means more of the code will be in the product, but we’ll have larger and more complex dependencies. There are really quite a number of tradeoffs here, from managing upgrades, to dealing with bugs in dependencies, but none of them are truly significant.

So no matter which approach you think is faster, we’re still arguing over like a hypothetical, best case 10% change to overall productivity either way.

It’s not that there is no difference between using a kitchen-sink framework vs. assemble-your-own. It’s rather than I think the differences are more subtle than “we can write less code,” but with much larger impacts.

• You’re going to attract different programmers. Some programmers will avoid Rails or Django, some will only really want to work in Rails or Django.
• Some changes are going to be easier in each system, some are going to be more difficult.
• You’ll often find plugins already available for Rails/Django, and not have to build it yourself. Sometimes this can lead to a curse of over-complexity (“sure I can add that, it’ll only take a minute” and now you suddenly have weird bugs or behaviors). Sometimes it saves you a month of development (“this plugin is confusing, I’ll write it myself” and then you keep adding features).
• Technical onboarding is going to be different. Home-rolled systems will have a steeper learning curve. You may be able to throw bodies at a Rails project. If you want junior developers in a home-rolled system, you will need to support them more, which can be a good or bad effect.
• The more you build and assemble yourself, the more programmers will need to learn how to do more library-level work, as opposed to only application work. You can see this as unnecessary, or you can see it as an opportunity for growth and development. There is a velocity cost, then, that you’d hope to make up by having programmers have a longer tenure or better skills.
• The codebase will be architected and tested differently, which will lead to another set of costs and benefits.

All of these effects are subtle, but will profoundly shape your long-term culture. Their impacts are far greater, I think, than any direct “productivity” or “performance” impact due to framework approach. There’s no right and wrong in the abstract, but there may be a right and wrong for you.

I largely consider choice in framework a religious argument akin to tabs-vs-spaces. Interesting for programmers to argue about but of minimal high-level import. Have an opinion, argue that opinion, but understand what you’re arguing. When the entire team is on the same page, when they understand the whole value stream and how they and their technology choices impact it (or don’t), you can avoid the religious argument and focus on the fundamental practices and culture that are truly responsible for success.