Your project probably isn’t epic, nor is it trivial. That’s precisely why it’s hard.
John D. Cook summarizes the mathematical “law of medium numbers”. In short: the hard stuff is in the middle.
Atoms are simple, and so are stars, but medium-sized things like birds are complicated. Medium-sized systems are where you see chaos.
With minimal effort, I humbly restate this as The Law Of Medium Software Projects:
For medium-sized software projects, we can expect that large fluctuations, irregularities, and discrepancies in time and effort will occur more or less regularly.
When you start from scratch with a small problem space, like “I want a portfolio site”, the solution is pretty clear-cut: Pick a basic publishing platform, get your images, prose, then organize and launch.
Super-large efforts can also be simple. If you’re building out a titanic data center, the question of “how much power we gonna’ need?” boils down to square-footage and math.
It’s the middle-sized stuff that is tricky: the four-month project, the 300-user web app, the blog that gets enough traffic to create non-trivial hosting costs, the database that’s big enough to need DBA’s, but not big enough for bleeding-edge storage stuff.
The normal curve tells us that most projects fall in the middle. The realities of our profession dictate the same.
If a job is so small as to be trivially simple, people rig it themselves or unload it to a third party provider.
If a project is titanic in scope and size, then we staff a lot of people and plan for a lot of time. While the details can be gnarly and intricate, the 10,000 foot-view of the project boils down to predicting person-hours and hardware costs.
What the Law Means to You
We need to be careful which methodologies and anecdotes we latch onto. Just because Google leans on Bigtable doesn’t mean it’s time to drop Oracle like a bad habit. Just because 37Signals says “simple is best” doesn’t mean you can ignore 95% of the feature requests from your department.
The trick is to map these successes onto our own situation. Think of it like the breakfast buffet at Denny’s. Sure, they’ve got cantaloupe, but you want eggs and pancakes, so admire the melon. Appreciate what it can lend to breakfast. Maybe even take a teeny piece, but don’t dump what you’ve accumulated just to make room for honeydew.