So here’s the thing. In three weeks this little project grew in a way that deserves a post about the process alone, because it became a living example of something I’ve been preaching for years: real software is grown, like a plant. Nobody builds good software piece by piece the way you snap a Lego together. Let me show you the numbers, and mostly the story they tell.

The numbers, today

ai-memory has, in 24 days of life (from the first commit on May 21 until now):

• 482 commits on main, averaging 20 a day.
• 26 people committed to the repo.
• 10 Rust crates in the workspace.
• ~62.7k lines of production Rust, plus ~11k lines of tests in tests/, more than 1,000 test functions, and 90 #[cfg(test)] modules scattered through the code.
• ~19.7k lines of markdown documentation (docs/).
• 70 pull requests, of which 59 merged (84% acceptance).
• 27 issues, 100% closed.
• We cut v1.0.0 on June 12 and we’re already on v1.0.6.

Twenty-four days. Hold onto that number, because the interesting part isn’t the final size, it’s the way it got here.

How it started: one of everything

ai-memory was born small and dumb on purpose. In the first two days I nailed down the whole skeleton: workspace bootstrap, MCP server, hook capture, cross-agent handoff, git wiki versioning, an LLM provider trait, embeddings. All of it running for one user, one machine, one workspace, one architecture (Linux, basically). No multi-user, no permissions, no scopes, no Windows or macOS support. It was my agent brain, on my machine, for me.

On day 3, when the first external contribution landed (a fix from Lucas Oliveira for OpenAI-compatible provider URLs, PR #6), the project had 102 commits and around 22.9k lines of source. Already the same 10 crates as today, but with a fraction of the functionality. It was a one-of-everything MVP that worked for me and nobody else.

And this is where most projects die, or worse, where most inexperienced programmers get it wrong.

The Lego fallacy

There’s a widespread belief, especially among people starting out, that you can design everything a piece of software will be before writing the first line. You sit down, plan the entire architecture, break it into neat little pieces with all the folders and layers and interfaces mapped out, and then build it piece by piece. When the last Lego brick clicks in, the software is done. Beautiful on paper.

In practice, it’s the fastest road to rotting code. The sooner you crystallize a structure, the sooner it becomes a straitjacket. You predicted an elaborate permission system before having any users? Congrats, now you carry the weight of maintaining an abstraction nobody uses, and that’s almost always wrong, because you designed it guessing the future instead of answering a real need. The inexperienced programmer over-engineers at the start, and no matter how much they plan, the result starts to rot fast, because they’re solving problems that don’t exist yet and getting wrong the ones that do.

I don’t believe in software that finishes. I already wrote about this in “Software Is Never Done”, and it connects straight to the original Agile methodologies, the real ones, before they turned into a corporate post-it ritual. Good software is a clay sculpture: malleable, always wet, that you keep adjusting and reshaping, and that never gets finished. It’s clay you mold while it’s being used. The Lego tower that one day clicks into place and is done is a fantasy.

What the contributors built

And this is where the best part of open-sourcing it comes in. That one-of-everything MVP became a cross-platform, multi-user system because a lot of people put their hands on it. I want to thank them by name, because without these people the project would be a fraction of what it is:

Djalma Júnior drove a big chunk of the heavy architecture (multi-user, admin, cross-project recall), with 25 merged PRs. wlech with dozens of commits. Pedro de Freitas Jr on embeddings and web fixes. Mauro Couto de Paiva and Gustavo Cateim drove a big part of the Windows support. Pablo Winter, Lucas Oliveira (author of the first external PR), and Said Rodrigues, among many others. Thank you so much, folks.

What those contributions added, the highlights:

• Real Windows, macOS, and Linux support. What was Linux-only got macOS bash 3.2 compatibility, an NTFS file index for tracking inodes on Windows, and native per-platform hook commands.
• Frontend API (/api/v1) and a customizable web UI. It stopped being only MCP + CLI and got a real HTTP surface, with clickable wiki links and search by file path.
• Multi-user with attribution. A users table, writer/reader commands, and write attribution in the same SQL transaction as the data (pages.author_id + frontmatter).
• Multi-workspace and cross-project recall. Global search, links between wikis of different projects, and active-project isolation per session and per actor.
• More providers and more agents. Gemini, Copilot via OAuth, OpenAI-compat strict mode; plus support for Antigravity, Grok, and VS Code Copilot on top of the originals.
• Operational robustness. Checkpoint recovery, reindex-from-wiki to rebuild the index from the markdown, self-describing _meta.md manifests.

Notice something: none of this was in the original plan. There was no original plan. Each of these features landed because someone needed it, opened an issue or a PR, and the structure gave way to make room. Multi-user wasn’t predicted: it emerged.

And only now did we stop to consolidate

Here’s the central point of the post. Over these three weeks, the code grew a little messy on purpose. When multi-user support came in, the logic for “which workspace and project is this request aiming at” ended up embedded right in the middle of several handlers in the MCP server, the web API, the admin, scattered and duplicated. Each spot resolved scope its own way. There was duplication. There was code that wasn’t pretty. And that’s fine, because it was working, and working for a lot of people.

Only after the software was already running on dozens of people’s machines, across three operating systems, with real multi-user in production, did I stop and say: “okay, now is a good time to refactor and rethink the internal wiring.” The second-to-last big commit, refactor: centralize scope auth and wiki consistency, did exactly that, with 1,239 lines added and 249 removed, creating a central 601-line ScopeResolver and a consolidated ActorContext.

Look at the “before.” This scope-resolution snippet lived duplicated, with variations, across several methods:

// BEFORE: scope logic embedded, repeated in every handler
let active = self.active_project.get_for(actor);
if let Some(name) = explicit_project.map(str::trim).filter(|s| !s.is_empty()) {
    if let Some((active_ws, _)) = active
        && let Ok(Some(pid)) = self.reader.find_project(active_ws, name.to_string()).await
    {
        return Ok((active_ws, pid));
    }
    if active.map(|(ws, _)| ws) != Some(self.workspace_id)
        && let Ok(Some(pid)) = self
            .reader
            .find_project(self.workspace_id, name.to_string())
            .await
    {
        return Ok((self.workspace_id, pid));
    }
    return Err(McpError::internal_error(
        format!("project '{name}' not found in the active or default workspace"),
        None,
    ));
}
Ok(active.unwrap_or((self.workspace_id, self.project_id)))

And the “after.” The same resolution, now delegated to a single authority in the store crate:

// AFTER: one central resolver, one call
fn scope_resolver(&self) -> ScopeResolver<'_> {
    ScopeResolver::new(&self.reader, self.workspace_id, self.project_id)
        .with_writer(&self.writer)
        .with_active_project(&self.active_project)
}

// ... and each handler becomes this:
self.scope_resolver()
    .resolve_current_or_project(explicit_project, actor)
    .await
    .map(ai_memory_store::ResolvedScope::as_tuple)
    .map_err(Self::scope_error)

Three different methods that each used to carry 30 lines of embedded logic, with lookup_ids, workspace fallback, and their own error handling, collapsed into one call each. Alongside came the sibling commits: harden multi-user admin auth boundaries and prevent scoped fallback bleed. Centralizing permissions, workspace and user scope, and wiki consistency, all in one place, with a clear rule.

The detail that matters: ScopeResolver wasn’t invented at the start of the project by guessing there’d be multi-user one day. It was extracted from code that already existed, already worked, and had already shown, in practice, which patterns kept repeating. The abstraction came after the usage, not before. I let the structure emerge from the code, and only once it was obvious did I consolidate it.

The right order: make it work, make it right, make it fast

This is the summary of everything. There’s an old saying, attributed to Kent Beck, that I carry like a mantra: “make it work, make it right, make it fast.” Make it work, then make it right, then make it fast. In that order. Most inexperienced programmers want to make it “right” and “fast” before making it “work,” and so they invent rules with no code to justify the rule.

ai-memory followed the right order without me even thinking much about it. First I made it work for one person, simple scope. Then I made it work one feature at a time, refactoring and hardening the code along the way, but never falling into over-engineering and never rewriting from scratch to fit a preconceived structure. I let the current structures emerge from the code that was being added. That meant living with some duplication and some so-so code for a while, until that moment hit of “okay, now it’s time to refactor and rethink the wiring.” And only now, with the software mature and in use, did I establish better patterns and rules for new code. Rules backed by real code, not by guesswork.

KISS, “keep it simple,” is about not solving a problem you don’t have yet. It has nothing to do with writing little code. It’s about letting architecture emerge from the code, not the other way around. You don’t design the sculpture and pour the clay into the mold. You take the clay, start shaping, and the form appears as your hands work. The mold, if you ever need one, you build later, around the form that already exists.

Twenty-four days, 482 commits, 26 people, and none of them followed a diagram drawn on day 1. Because there was no diagram on day 1. And that’s exactly why it worked.

ai-memory is open source and keeps changing every week: github.com/akitaonrails/ai-memory. Want to help shape the next piece? There’s an open issue waiting.

–

On February 10, 2026, I posted the tweet above, just throwing out there something that had been on my to-do list for years, ever since I was publishing videos on YouTube. I never did it because I hate HTML/CSS, and having to build an email template was enough to keep me from starting. On top of that, doing yet another newsletter, which is just a boring list of links, didn’t motivate me.

But that day I was already wrapping up my second vibe coding project, FrankMD, and I think I figured out the main tricks and the methodology to build a complete product the right way. So a bunch of ideas started popping up. I dumped them all into this little document.

I thought: what if I vibe coded not only the newsletter system’s code, but also had part of the actual content generated by AI in a quirky way, in the form of commentators: me and a robot? And, of course, with Hitchhiker’s Guide in mind, it had to be MARVIN, or rather, “M.Arvin” (so nobody hassles me about copyright).

Obviously, there would be a weekly email version and an archive blog. But since we were going all in, why not rewrite the whole newsletter as a podcast script? And since Qwen3 TTS recently came out, open source, why not train a LoRa with my voice (obviously, I have hundreds of hours of my voice on YouTube) and put together a fully automated podcast?

Finally, one of the things that would demotivate me is building a traditional Web Admin and having to log in manually to register news links and my comments. I knew I’d get lazy and, at some point, burn out. There was only one solution: build a smarter bot I could talk to via DISCORD. That way I can quickly jump in, whether from my PC or my smartphone, without having to log into a separate system or anything. I already chat with people on Discord, so the friction would be zero.

So:

• Marvin Bot to manage all the content (the stuff I send in and the automated stuff, with comments and everything else)
• An X.com monitor to check what I posted and grab it for the newsletter. Including the replies I give, which will become Q&A!
• Newsletter website to subscribe, unsubscribe, etc
• Blog for newsletter archives and podcast transcripts (also with terms of service and privacy — yes, I’m LGPD compliant!)
• Automatic voice generator and podcast assembler that uploads automatically to make it available on Spotify
• And also blog and email templates, with light and dark theme versions.

You, an experienced programmer, without AI, how long would it take you to build all of that?

Well, I started on February 10. I finished everything on Sunday, February 15. And today, the 16th, the production jobs ran for the first time, which generated the newsletter, blog posts, podcast, Spotify upload, email delivery.

And it wasn’t done the crappy way every amateur vibe coder does it:

Rails folks must be surprised: a project with a 1 to 1 test ratio! 1 line of tests for every 1 line of code: 100% coverage. And that’s without counting a separate full end-to-end integration project (which I used to make sure production would be flawless).

All the strategies I’ve learned over the years putting projects into production: security, scalability, durability, delivery guarantees without spamming duplicate emails, etc. I’ll see later if I write more specific posts about some of the technical tricks.

I’m not going to open source this project. I built it 100% custom for me, my workflow, and my style. It’s not hard for another competent programmer to do the same thing with vibe code: if I pulled it off, anyone can. So just don’t be lazy about thinking and doing. Honestly, I don’t get how tech YouTubers still aren’t launching 1 complete project in production, per video, for everyone to test and watch.

The screenshot also doesn’t show the Qwen3-TTS server, which produces audio. That one is much simpler, a PyTorch server. Here I actually used Python. The most important part is the fine tuning of the model to generate voices exactly the way I wanted:

The blog is all Hugo/Hextra, same as this one. And my bot publishes there automatically.

M.Arvin Bot

M.Arvin has a personality I spelled out exactly the way I wanted. And it’s a Discord bot, so my workflow looks like this:

I just need to send a link with my comment (the same way I’d do on X.com), or I can post directly on X and it’ll pick it up the same way. Then it fetches the content, writes a little summary, does a small fact-check, and M.Arvin generates its own comment on top of mine.

I programmed it to respond to a bunch of commands to make my day-to-day easier. This is just a small part of it:

Here’s an example: I have control over how much AI it’s using (it’s pretty cheap):

Deployment

In this post I’m not going to go into too much detail, but one thing I spent a lot of time on, going through several discussions and versions with Claude Code, was the infrastructure architecture. And in the end it managed to give me a Deployment Guide that I was able to follow all the way through, knowing at a minimum it’s set up the way I would have done it myself if I’d built everything alone:

No software is guaranteed to be 100% perfect; this one shouldn’t be either. Even with me reviewing for days, going through every step multiple times, there’s always some bug I didn’t anticipate. I “think” that, at least, everything I know usually causes problems in production, I “think” I managed to cover satisfactorily. If it were a human team, I would have pushed it to production in a worse state than what I reached with Claude.

Everything I could protect with Cloudflare, I configured. Every good practice for email/mailing, I followed. Every precaution against accidents I took (every deploy automated with Kamal) and every subscriber’s email was encrypted with AES-256 in the database (even if they leaked by accident, the user data doesn’t leak). I didn’t implement profiling, cookies, or tracking pixels, nothing. Privacy was the focus.

Everything I could reduce costs on, I reduced; after all, I’m paying out of pocket. So things like the RunPod GPU for generating the audio are serverless. I only spin it up at the moment of generation; it runs the whole process and then the machine shuts down. The server is all organized into a lightweight Docker image. All the AI models live on a separate network volume.

Process Spoiler

Later I plan to do a bigger post about my methodology, but it’s very simple: treat Claude Code/Codex like a hard-working human programmer — one who’s going to make lots of mistakes, but knows how to obey super explicit, direct, clear, unambiguous orders. My role is product manager, Tech Lead, or QA. It has to be all THREE roles.

In total, Claude Code made 201 commits. But these aren’t normal commits; I figure that in a normal project with humans, each one would have been roughly 150 Pull Requests.

Question for you folks doing sprints on real projects: how long does it take to get 150 user stories/bug fixes approved and in production on your actual project? How many user stories can you get done and approved (with QA and everything) in ONE week?

Here’s an example of a commit/“user story”:

When I was around commit 50, I was already thinking: “It works already!”* That’s the stage where every content creator (who has never actually put real projects into production) would already be posting, saying: “I vibe coded it in 2 days”. In fact, in 2 days I already had the bulk of the functionality ready.

The other 4 days were:

• 1 day for production deploy (when everything is planned ahead of time, all the contingencies already thought through, it’s easy).
• 3 days just testing, writing automated/integration tests, generating and regenerating, exercising every step of every part, multiple times, and making adjustments, refactors, or even throwing away entire pieces to change them, because the tests showed it was worth it.

That took from commit 50 to 200! And that’s the part every vibe coder skips. To be fair, it’s the part anyone, with or without AI, would skip. And that’s why projects always go wrong and end up in production full of errors and obvious security holes.

Even after asking Claude Code to review security best practices and test coverage, multiple times. Then I opened Codex and asked it to do the same thing, and it still found more security holes and more missing tests, multiple times. Even after both of them did these checks several times, I still found more holes on my own that I had them fix. At no point was either of them able to find “EVERYTHING”. Nobody can, not even AI.

Conclusion

Vibe Coding the right way has only one path: Extreme Programming!

Yes, all the “boring” and “bureaucratic” things every developer hates doing are what guarantees the AI will give you back a result acceptable for production:

• Pair Programming: you need to tell the AI to plan first and spell out exactly everything it’s going to do, BEFORE starting. I review, I tweak, and only when I’m satisfied do I let it go. I watch every single thing it does and, at the end, I tell it to fix whatever I think it missed.
• Test-Driven Development: every feature must come with unit tests. Every bug fix needs regression tests.
• Continuous Integration: every time it finishes implementing, you need to run the CI script to see if anything broke somewhere else. More than that: on the master branch, there are only commits with passing tests. I can revert at any point and the project runs.
• Small Releases: I really liked this about Claude, even if I end up forgetting to tell it to commit a task and send it off to do another one, in the end it doesn’t do that ugly git add . thing — no: it separates into 2 commits and describes each one the right way. On top of that, I pushed to production on Friday, and kept going Saturday and Sunday testing integration locally and in the production environment, to make sure there’s none of that “it only works on my machine” stuff.
• Coding Standard: I started with the bare minimum of technical requirements, but during development I had it refactor my way over and over again.

Those are the main tasks that need to be done. Let me get ahead of it: “Spec Driven Development” isn’t totally useless, but the structure doesn’t matter: it’s your text that counts.

It’s the same thing as User Stories. Big deal, you wrote “As the end user, I want that when I type X…, the system responds with Y”. The format is NEVER important. What’s important is HOW you wrote what goes in the middle. If you can’t write, the AI won’t know how to respond to you the way it should. I’ll do another article just to show how terrible all of you are at writing.

The message is simple: Vibe Code works, in part.

What actually works is “Senior Agile Vibe Coder”

I’ll ask again: how long, without AI, would it take you to produce 150 user stories/bug fixes, in the scope I outlined at the beginning? All the way to production with everything tested? I doubt it would be in 6 days like I did it.

One of the most popular ones answers the question “What is the hardest thing you do as a software engineer?”. I wrote something similar in Brazilian Portuguese in the post “Estimativas são Promessas. Promessas devem ser cumpridas.”.

In a nutshell, you can never give an estimate that is “correct”. If you could, it would not be an “estimate”, it would be a “prediction”.

Let us assume we have neither precognition powers nor magic crystal balls to tell us the future.

To estimate something is to “guess” the value of something. It’s always a guess, the same thing as a valuation. And like any guess, it can never be deemed “correct”: it’s just a likely candidate among an infinite range of possible values.

There is zero connection between a guess and the outcome. Understand this simple truth: saying something can happen does not MAKE it happen.

Estimating tomorrow’s weather as rainy does not MAKE it rain. Estimating the result of the Super Bowl does not MAKE it happen. There is zero correlation between an estimate and the actual outcome.

I stated in those articles that “Estimates are Promises”. The intention was to provoke a reaction, since most people assume that estimates can never be promises, exactly because of what I just explained.

What makes a promise special is that once you promise something, you’re expected to ACT on delivering it.

No one that has no “skin in the game” should give estimates.

If you’re not an active player in the game, you shouldn’t be handing out guesses. The same way no one makes promises on someone else’s behalf.

Can you actually make credible promises and keep them? You can, but first you have to understand a few more truths about reality.

You’ve probably already seen plenty of articles explaining project management methodologies and techniques. Most of them manage to explain the Whats and the Hows, but as usual, they fail on the Whys.

Why do we need those methodologies? Why are they necessary? Why do they work? What hidden mechanisms do they set in motion?

What makes Agile techniques any different from your usual homeopathy or a budget self-help cliché?

It’s always all about Pareto

There is no such thing as a precise project scope. There is a limit to adding detail, and past it you just get diminishing returns.

The most precise level of detail a feature scope can have is the actual code of the feature. And this matters: the lines of code are NOT the outcome. What end users actually experience is the execution of that code at runtime.

The programming itself is the actual blueprint. The diagrams, the use cases, the user stories, and anything else that comes before the programming are just a draft, a mere sketch.

A naive architect or designer might think that detailed diagrams, use case documents, and fancy PowerPoint slides are worth as much as an engineering blueprint. They’re worth less: they’re the equivalent of a sketch on a napkin. Volatile and, in practice, mostly worthless.

“But isn’t the programming the same as the construction phase itself, laying bricks on top of bricks?” NO, that’s not what a programmer does. The bricklaying is the job of the language interpreter or the compiler that spits out the executable binary.

This is the metaphor that drives non-programmers crazy. In engineering, the construction itself is the most expensive and time-consuming part. In programming, the expensive part is the blueprint (writing the code), and the “construction” is trivial: just compiling, which is automatic.

In the same way, a project scope is just a set of sketches. We have to drop the notion that a “complete” scope will ever exist. There’s no way to say “100% of the scope” or “closed scope”, because a software project scope is, by definition, always incomplete.

On top of that, I’ll argue that roughly 80% of this so-called “scope”, what I prefer to call a sketch, is mostly worthless to most end-user activities: the admin section, the institutional pages nobody reads, convoluted sign-up flows, and so on.

This is why every feature list MUST be prioritized. You can usually get by with 20% of the features, which is roughly what people mean when they say “MVP” or “minimum viable product”. Release as early as you can, get user feedback, and refine the rest of the “sketch” you call a backlog.

So instead of aiming for an all-or-nothing proposition, hunting for stupidly complex equations to figure out a “precise” estimate of an incomplete sketch, assume you can actually deliver EARLY the first 20% that really matters and figure out the rest in iterations.

Oh yeah, this is what we call “Agile”, by the way.

Agile is about Risk Management

People assume Agility is about managing the project management instruments themselves: the backlog, the rituals, the metrics.

Having Agile-like instruments doesn’t make you Agile.

Being Agile is keeping Risk under control.

Instead of treating projects as an all-or-nothing endeavor, start thinking about them the way an investor thinks about a portfolio of stocks. You don’t expect the whole portfolio to turn a profit; you assume some stocks will underperform. You just don’t know which ones, so you dilute your risk.

Trying to predict the stock market is an exercise in futility.

Trying to predict the precise implementation of a project, especially the long ones, is also an exercise in futility.

So you have to deal with uncertainty the right way: by becoming Antifragile.

“Some things benefit from shocks; they thrive and grow when exposed to volatility, randomness, disorder, and stressors and love adventure, risk, and uncertainty. Yet, in spite of the ubiquity of the phenomenon, there is no word for the exact opposite of fragile. Let us call it antifragile. Antifragility is beyond resilience or robustness. The resilient resists shocks and stays the same; the antifragile gets better” - Nassim Taleb

Instead of the preposterous exercise in futility of trying to predict uncertainty and random events, you do the reasonable thing: you assume Black Swans exist and you can’t predict them. So you prepare for uncertainty the only reasonable way there is, by not trying to predict them.

Expose the small mistakes early and correct them often. Implementing everything in a black box and doing a Big Bang deployment is the easiest path to failure. Delivering often, exposing the bugs, and fixing them constantly is accepting that mistakes will happen and gaining strength along the way.

A Better Metaphor

Imagine that you, whether the non-programmer client or the programmer who has no clue how to explain the process to that client, have an iron ore furnace to manage.

The thing about those furnaces is that if you heat them too much, they can explode in your face. If you cool them down too much, the ore hardens.

Your job is to add coal to the furnace, and you decide the rate. Too fast and it gets exploding hot. Too slow and you might put out the fire and lose your furnace.

Now try to estimate a constant intake of coal to keep the furnace in good shape.

You can’t.

The easiest way out is to install a thermometer that tracks the furnace’s current temperature.

You stay safe within a certain margin of temperatures, speeding up or slowing down the coal intake by checking the thermometer all the time.

ALL THE FREAKING TIME.

This is what Agile-based “Velocity” (or any of the fancier Monte Carlo simulations) actually is: a thermometer.

If the Velocity is too high, your team is probably working extra hours or delivering lower-quality code. This will backfire: either your team burns out too fast, or the code piles up technical debt too fast and you won’t be able to pay it back. Keep that pace and Velocity drops to a halt (the furnace explodes).

If the Velocity is too low, your team is slacking off, your backlog is a useless piece of shit no one can understand even after 10-hour meetings, or you already let Velocity run too high and now you’re paying back technical debt, or your team is dead from burnout (fire extinguished).

You want to keep Velocity steady, constant. This is what being Agile is all about: keeping an eye on the thermometer and responding.

The Triforce of Projects

Welcome to the Iron Triangle of Project Management.

Repeat after me: if I want to lock down the time, the cost, and the scope, I am a moron.

Repeat again.

You should lock down time and cost. If you’ve read this far, you know you can never lock down “scope”; you can only make it fat, and not necessarily more valuable. This is why I always say the very definition of a Product Manager or Product Owner is to be the bastion of ROI (Return on Investment).

Now, why?

Because the Iron Triangle has the following corollary:

• If you want it fast and good, it can’t be cheap
• If you want it fast and cheap, it can’t be good
• If you want it good and cheap, it can’t be fast

This is a Law, you can’t fiddle with it. Pick your choice.

How do I keep Promises then?

Now, with those 3 truths in hand:

• You can’t have your lunch and eat it too (cheap, fast, and good)
• You’re managing the temperature of the furnace
• You only need 20% of the “sketch” you call “scope”

Yes, any experienced developer can give you a “ballpark” estimate. A ballpark goes like this:

• 1 month (maybe 1 month and a half, but definitely less than 2)
• 3 months (it’s more than 2 months, less than 6 months)
• 6 months (it’s more than 4 months, less than 9 months)
• it’s more than 6 months, probably less than 1 year

Don’t even try to granulate more than that. It’s useless.

Lock the time. And lock the cost (that being the number of developers times the hourly rate times the total amount of estimated hours). That’s it.

Now jot down what the client calls “scope” as user stories in a backlog and have him prioritize it.

Start the iterations. After each one, release to a staging environment. Non-programmers, beware: ALWAYS make sure whichever programmer you hire delivers testable versions of the finished stories to a publicly available URL you can actually visit and test.

If your programmer refuses to do that or gives excuses: FIRE HIM.

If your programmer or company or whatever promises you a “closed scope” price, promises to do everything, and you believe them, you’re too gullible.

Do you think it’s funny to play this stupid game of “I’ll pretend to tell you that I know the truth and you will pretend that you believe what I tell you.”

No serious professional has time for stupid games, and the only honest thing anyone can say about any software project is: “given my experience I believe the ballpark for this kind of project is within X months, given the Y and Z assumptions”.

You don’t have to believe him. You just need to start checking the thermometer. Any non-programmer can assess the quality of the delivery based on the frequent deliveries of the prioritized features.

“But what if after 2 weeks I don’t like the results?” Easy: FIRE HIM.

And sometimes “firing” isn’t even the right word. Sometimes the relationship is difficult and the best thing to do is to part ways.

You have to accept losing 2 weeks, or any short period of time, as part of Risk Management. It’s better to accept losing 2 weeks of your project budget than to blindly trust someone for 6 months and lose the entire project budget and then some.

Pareto again: Agility is about Risk Management. You accept that losing 20% of your budget is okay, and you play with that. And that’s fine, because you only really need a bit more than 20% of the sketchy scope you have.

See what I just did with the Math here?

We stop playing the pretend game and actually manage the project’s risks. You collaborate on the equivalent thermometer, which combines the prioritized backlog (the scale) and the velocity (the temperature), and you keep an eye on the partial deliveries in the staging environment.

Conclusion

So yes, Estimates should be taken as seriously as Promises. You can give reasonable Estimates as long as you can manage the Risks and the Client accepts the rules of the game: there is no “closed, complete scope”, priorities first, and testing and accepting delivered features every week.

The idea behind Promises is that you have to MANAGE in order to meet them. The best way to do that is to frequently stop, re-assess, and keep going. It feels like you’re wasting time, but you’re actually saving yourself from wasting it.

If you don’t have skin in the game, back off.

Velocity is meant to be kept steady, at a predictable rate. Use the variance as a sign of being too fast or too slow, adjust the other variable, measure again, and go. Just like a furnace thermometer.

There are several versions of “thermometers”, from Joel Spolsky’s Evidence Based Scheduling all the way to fancy Monte Carlo simulations and other stochastic processes (they are all thermometers, and NOT estimation tools).

What stops you from doing that is using the wrong metaphors and the wrong references.

Instead of hunting for equivalent metaphors in construction, factories, and other “hard”-ware assemblies, look elsewhere, where you’ll find other “soft”-ware processes.

Musicians have deadlines. Painters have deadlines. Choreographers have deadlines. Sports have deadlines. Research laboratories have deadlines. How do they meet them? By constantly checking the current state, comparing it to the goals, assessing whether what they’re doing is actually working, and changing what doesn’t.

Hollywood has deadlines. They have far worse variables to control than any software project you’ll ever run into, and they still manage to deliver. And profit.

And accept that you can’t control every variable, so stop trying. Think of it as the financial markets. One day you have Ethereum skyrocketing 4,000% and the very next day you have it falling down to ashes in a flash-crash.

Don’t aim to become resistant or resilient. Prepare to become Antifragile.

Now, you’re a small tech startup or even a small team in a big corporation. How to even start?

And the recommendation is: do the smallest thing that works first. Always. Of course, avoid “quick and dirty” as much as you can. But don’t get paranoid and over-engineer too much.

Over-engineering is as expensive as doing things “dirty”.

Add “Time” to your equation

What most people do very wrong is disregarding the variable of “Time”.

Everybody goes to tech conferences or read fancy blog posts, or watch flashy screencasts. Their wrong conclusion is: “Netflix uses it, therefore it should be good for me, because I want to become Netflix”.

The wrong assumption is that Netflix, or Google, or Facebook, or Spotify, is a static system that always functioned the way they advertise it today.

People forget that every unicorn company had a day-1. And on day-1 they did not use microservices. They did not have React.js. They did not have Go or Elixir. Some of them didn’t even have good programmers to begin with. Netflix started as a VHS renting service, remember that? Google started off in a dorm-room with off-the-shelf hardware components assembled in a Lego-based rack.

You are a fan of some celebrity. That celebrity has a Lamborghini car. So you want to be like that celebrity one day. What do you do? Do you buy the same Lamborghini he has?

If you are that stupid, the only thing you will end up with is a HUGE debt to pay.

Stop envying the rich guy’s Lamborghini.

(Technical) Debt

This is neither a new term nor a new theme. Every decision you make in programming is a compromise between what the “future-ideal” should be and what you can actually do right now. Loan to buy the equivalent Lamborghini too soon, and you will have to deal with an unpayable (Technical) Debt. You will stall, you will stop, and you will do nothing but pay that debt from now on.

You want to “code faster” therefore you skip writing automated tests, because you assume it will slow you down. And you’re right. Test-driven development doesn’t make you fast at first, and it isn’t supposed to. It protects your future self from your present self, and keeps you from piling up Technical Debt.

Again, you forget the “Time” variable.

You write and deliver fast for the 1st month. After the first version is deployed, now you will be obliged to start paying the debt. Your productivity will slow down. Regression bugs will show up. Every new feature you try to add breaks something in unexpected ways. Because you don’t have automated tests, you will keep fixing the same things many times over. Debt will catch up, and you will pay. One way or the other.

Minimal test suites are like Insurance. You don’t need it right now, on day-1. But on day-100 you will be so glad you have it.

Doing too much microservices on day-1 is debt. It feels great that you’re doing that fancy thing you read in a blog post. On day-100, with your 3 people team and a dozen microservices, from now on you will do nothing but pay the accumulated debt, with Interest. Every new microservices deployment breaks your system. And of course you didn’t add monitoring, you didn’t add integrated tests. So every time you code something, something else unexpectedly breaks.

Enjoying that early Lamborghini now?

The Mythical Man-Month

Philip Calçado writes very good posts and presentations about microservices in the right way. If you’re serious about that subject I strongly recommend that you read his posts, such as his “Prerequisites” post or his “Economics” presentation.

He correctly remembers Fred Brooks’ The Mythical Man-Month. I urge developers to read this small book. It’s uncanny how the entire industry is still repeating the very same mistakes Brooks reports in his book from projects in the 60’s!

To me, microservices is a by-product of tech companies with too many developers. Once you break through the 5-developers team a “monolithic” system with poor organization and few automated tests can become cumbersome to deal with. So the consequence is the desire to break it down. You make 2 teams and 2 microservices and coordinate. And the teams start to isolate, and play the finger-pointing game when new bugs are reported (“it’s the other team’s microservice’s fault”).

That’s Conway’s Law if you do it wrong:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Also in play is another chapter in Brooks’ book: The Second System Effect. Every developer, tech startup, has to assume that the first version of anything is not the best design. And the second system (the first big rewrite) is the most dangerous system you will write. As Brooks states, it goes like this:

An architect’s first work is apt to be spare and clean. He knows he doesn’t know what he’s doing, so he does it carefully and with great restraint.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used “next time.” Sooner or later the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one. The result, as Ovid says, is a “big pile.”

Then, what happens? The CEO, the board, the investors, or whatever, start doing the wrong thing: hiring more and more people. Loan more and more. Instead of paying the debts they end up making more debt. They fell for what Brooks reported in the first chapter and what gives the name to the book: The Mythical Man-Month.

Adding manpower to a late software project makes it later.

Complex programming projects cannot be perfectly partitioned into discrete tasks that can be worked on without communication between the workers and without establishing a set of complex interrelationships between tasks and the workers performing them.

Therefore, assigning more programmers to a project running behind schedule will make it even later. This is because the time required for the new programmers to learn about the project and the increased communication overhead will consume an ever increasing quantity of the calendar time available.

Again, it’s foregoing the sense of the “Time” variable. And confusing “debt” for “investment”.

First of all, do yourself a favor and read the goddamn book, twice.

Do Agile right! And no, it’s not Kanban!

If there is one good thing the whole Lean noise generated is the notion of “Minimum Viable Product” or MVP. People call it “prototype”. Some call it “launching Beta” or simply “version 1.0”. Doesn’t matter. It’s the realization that you don’t know much at the beginning so overengineering on the first version is a waste of time. Lean is about controlling “waste”, so we do the minimal that works, gauge results and work from there.

The cynical will say that any prototype that reaches production will never die. And they’re not wrong.

The balance is to not do the “quick and dirty” version. That’s what we have Agile techniques for. Do the minimal, organize the minimal. That’s why we have object-oriented design patterns, from GoF to DDD. You don’t need to do EVERY pattern, and that’s exactly what “overengineering” means. But you can do the minimal monolith that will allow you to evolve later.

People criticize Ruby on Rails for not being organized “enough”. Nick Sutterer, the creator of Trailblazer, has a good point.

Rails done wrong is bad. The conclusion should NOT be: “let’s do microservices” though. That jump in faith is idiotic and makes no sense for all the reasons I mentioned above.

“Let’s do proper Agile stuff and proper object-oriented stuff” should be the initial answer.

And by “Agile” forget about idiotic post-its, idiotic numerology-based estimation equations (this is a whole post in itself, because story points and velocity are useful, but adding Montecarlo and other Gaussian-based stuff are not). The only “Agile” things you should be concerned about are the Extreme Programming (XP) techniques, including Iteration-based timeboxes.

You MUST do timeboxes. Stop, re-assess, change directions, and then keep going. The model of “Pull” is only reasonable when your direction is very, very clear, written in stone and unchangeable, like in a factory production line! (Which is exactly where the concept of Pull, and Lean, was actually born!)

Iterations, like automated tests, are like Insurance. You can never avoid all waste, but you can minimize it. You can afford to throw away an Iteration-worth of work. After the iteration you measure the results, and throw it away if necessary, changing direction in the process. Throwing useless stuff away is as important as adding new stuff. If you just add, you have a hoarding disorder!

If in doubt, do XP. Yes, it feels more “difficult”, and Kanban is “simpler” to explain. Now, is that a good reason?

Forget about Raw Performance when you DON’T need it!

Another problem is choosing languages or framework because of performance.

If you’re in the web development business, this is a huge WASTE.

Understand this truth: your servers will IDLE most of the time. And if you feel like using your web app is slow, the slowness comes from the POOR programming you did, not from the language. And no good language will rescue a bad programmer. I always say that if performance was that important, we should all be doing C.

Most of what you serve in HTTP-based apps, be it user-readable content, be it API GET results, can be CACHED! If you’re not using a CDN, you’re doing it wrong.

Yeah, yeah, yeah, you think you’re building the next Spotify. You’re not, at least not 99% of you. And the 1% doing custom engineering, with custom techniques and custom stack, successfully: you’re the 1%. Actually I’d say you’re a fraction of the 1%. Do not assume that what you do is good for the rest of the population.

90% of what most small companies and solo web developers need is just a Shopify account and a vanilla Wordpress installation. And that’s it.

Are you doing infrastructure development like building custom components to Docker, Kubernetes, Terraform? Command-line tools? Daemons? Then choose GO.

Are you doing embedded libraries? Maybe the next generation OpenSSL? Drivers? Then choose RUST.

Are you doing mobile development? Then don’t have a lot of choice: do Swift for iOS, do Kotlin/Java for Android. Or do React Native for simpler apps.

Are you really doing the next Whatsapp? The next Waze? The next Snap? You have hundreds or thousands of users in need of long-lived connections over unreliable network doing broadcasts of messages? Or you’re building the next evolution of distributed NoSQL databases? Or anything that actually has the proper meaning of “Distributed” in its definition? Then choose ELIXIR.

Are you doing a CRUD based web application? Go RAILS and never look back.

Are you doing all of the above? Use all the alternatives then. And I hope you have the budget, because you need a big team.

Conclusion

The Economics only makes sense when you consider the day-1. Do not forget the “Time” variable. It makes all the difference.

You will always get Debt to pay. So be smart in which kind of debt you choose. Because you will have to pay it back eventually.

Again, read Brooks. He said it decades ago: THERE IS NO SILVER BULLET. Don’t bullshit yourself. A new language, a new framework, a new architecture. None of those will save you.

In any tech endeavor, code is not the only thing to worry about. I would go as far as to say that in a tech startup, code is only 20% of the problem. A tech startup is just a company, like any other. If you’re the founder or CEO, you have to deal with all the remaining 80%: marketing, accounting, human resources, legal, etc. It’s already difficult enough without your tech team getting you unwanted Technical Debt, the unneeded Lamborghini, that you won’t be able to pay.

You can start humble on day-1. Keep evolving, continuously. This is the core of any Agile or Lean process: KAIZEN. Choose smart debts, pay them a small bit at a time, continuously. It’s the same reasoning as getting a loan in the bank.

Netflix day-7,200 is NOT your day-1.

Facebook day-4,800 is NOT your day-1.

Google day-6,809 is NOT your day-1.

Be humble. Deliver fast. Pay your Debts. Keep evolving continuously. Stop believing in fairy tales and silver bullets.

I’ve been using Pivotal Tracker since 2010 up to 2015 and this is because I believe it is far superior to Trello and other glorified to-do lists out there.

The Cost Conundrum

As with all the other reports, let me start with the cost argument. Pivotal Tracker is priced based on the number of users.

If you have a small team and not many outside clients, a basic 10-user plan would be enough, and you will pay just USD 420 a year. Just pay it already.

If you have up to 25 users, it’s still not so expensive, and you will pay USD 1,800 a year. Well worth it.

Even if you have 50, the maximum plan, you will pay USD 3,600 a year. More than that and you have to contact them directly.

The cost is not prohibitive and if you can pay, I advise you to do so.

In my case, the strategy is to get full control over our data and knowledge, which is why I’d rather have ownership of the service.

And because Fulcrum was “almost” good enough, I decided to try it out and see if we could evolve it over time. And I think that even though we are still very far from the full-featured Tracker, what we have is enough for most of our projects.

Let me say it again: you most probably won’t benefit if you’re a small team. Pivotal Tracker is, to this day, the best project management tool for software development teams, and you should definitely try it out first, get used to its workflow (as I will explain below), and then decide if you want to take control and install the open source alternative instead.

There is the factor of cost, but it’s not the driving incentive for this move.

The Important Features

Doing some research I found the Fulcrum project. It’s a very simple, very bare-bones Rails project that mimics Tracker’s most important features.

Any competent project management tool must have at the very least 4 pillars:

• Stories (with discussion threads, assets uploading, tags, Epics organization, and searchability)
• Proper numeric estimation (story points, proportional size for stories)
• Time-boxed Sprints (short, fixed periods of time)
• Velocity (the actual productivity of a particular team in a particular project)

Any tool that has all of the above is useful.

Fulcrum itself has almost everything. We made an internal fork of the project and we started adding some of the missing features. We’ve been using it for around 1 year already and it’s been good enough for our routine. There are still a few bugs and missing features that we intend to add.

If you’re interested, I am making this fork publicly available to conform to the Affero GPL license.

Any contribution is welcome.

On Proper Agile Management

Now sidetracking a bit, I’d like to explain why this kind of tool (Tracker-like) is the best compared to what I called “glorified to-do lists”.

The principles are very simple:

• Every project must start with the most complete backlog possible (uncertainty is at its highest in the beginning, and details will be missing, but it’s important to have a vision of what “minimally complete” means and the order of importance of each story, a proper “priority”).
  • The level of detail doesn’t matter. A story must have enough information that any developer in the team can implement the required tasks. Otherwise it must be tagged “blocked” so the Product Owner knows that (s)he has to add more details and no developer will touch it until the tag is removed.
• All stories must be estimated (more on that on the next section). The unit really doesn’t matter. You can think in hours, in fractions of a day or for what it really is: a proportion. A story sized “2” should be roughly half the complexity of a Story sized “4”. That’s it. Even “blocked” stories should receive sizes.
• Projects must be divided into time-boxed, fixed-sized Sprints, so the team has the opportunity to reassess frequently. This is where estimation can be refined, where more levels of detail in the stories should be required.
• Bugs can’t have any numeric estimation. They will serve to slow down the team’s velocity and they’re an indication of poorly made deliveries.
• Velocity is the actual amount of story points delivered by the end of a Sprint.
  • The total amount of story points of all stories in the project backlog divided by the team’s velocity will give a roughly estimated “End Date”. This is important, because this is how you know when certain groups of features will be delivered over time, and this is what the Product Owner needs to know to prioritize stories, simplify stories or even cut down unnecessary features.
  • The Product Owner’s sole responsibility is to be accountable for the Return on Investment of the implemented features. And the only way to measure cost-benefit is to know “how much” a feature will “cost” in terms of story points divided by velocity.

What the tool will expose:

• If a backlog has a lot of “blocked” stories or if it has many stories delivered without acceptance, it’s a symptom of a Product Owner not doing the work.
• If the velocity varies too much between sprints, it’s a symptom of either poorly written stories (and therefore developers having a hard time understanding what to do), or developers not concentrating on their work and delivering inconsistent results.
• Inconsistent velocities with many bugs in each sprint is the symptom of developers not doing code with proper quality (automated tests to avoid regressions for example).
• A “bug” is not the same thing as the Product Owner changing their mind on a story and rejecting. The team must not implement undocumented changes. Acceptance is tied to what the story says. The Product Owner must change his mind before the story is implemented or create a new story to add the change (and therefore add more cost to the project, there is no free lunch).
• Stories in the “Chilly Bin” (in Fulcrum, or “Ice Box” in Tracker) are “wish-lists” and they will never be implemented until the Product Owner moves the story to the backlog.
• Velocity should be roughly the same between sprints. Peaks or valleys in the velocity are symptoms of a problem that needs to be addressed by the next sprint. This is how a project evolves for the better over time.

On Estimation

Many people worry too much about “proper estimation” to the point of creating so much tension that developers start screaming about “no estimates”. This is nuts. This is why “estimation” is a different word than “prediction”, one is uncertain, a ballpark, the other is exact.

There is no such thing as a “correct” estimation.

I would argue that in many projects if the stories are minimally described (meaning that a developer is able to implement what is written), we could estimate all of them to be size “1” and forget about it.

We run 2 or 3 sprints (usually one sprint should be one work-week - 5 days of at most 8 hours, no more, no less) and measure how many stories were delivered. And this is the velocity.

Now we can see how many stories we have until the end of the project and divide by this velocity. And this is how many weeks it will take to finish the project.

There is no need for consensus based estimation.

Consensus on estimation is usually a waste of time. Estimation is merely the realization that it’s difficult to write down every story to be the exact same size, so we add proportions to them (A story size 2 is half the size of a story size 4).

But if a project is 3 months or larger, the less the proportions matter (unless you do the anti-practice of having heavily disproportionate stories, some size 1 and many sized 5, 8 or even 10, for example). But if you stay closer to less than size 5 for every story, then the larger the project, the less the individual estimates matter. Velocity will average out the differences over time.

Then the task of the Project Manager, Product Owner, the team in general, is to keep an eye on the total amount of points divided by this velocity, which derives the estimated end date of the project. And everybody can negotiate, beforehand, which stories can be done, which cannot, and change priorities, remove stories or simplify stories.

Don’t waste time trying to create formulas to derive estimation consensus.

If you manage the Velocity then you can manage the entire project. This is the core of all proper Agile methodologies. Everything else is an accessory to that end.

Trying to artificially increase velocity is one of the many sins. You can’t. It’s better to invest time in adding more details to a story, or to simplify a story or to decide that some stories are just not needed to deliver a proper product in the end.

Monte Carlo simulations, Six Sigma techniques, Kanban concepts, none of them matter. All traditional industry techniques are meant for production lines in factories, where the time of each task is well defined and you manage the variance, the exception. It’s a Gaussian, Normal world.

In software projects, or any craft and research, you manage expectations on results based on milestones and cutting excesses and uncertainties. It’s a Pareto world, one where there is no such thing as an “average”, there is no medium, no variance.

In a Pareto world everything is interconnected, not statistically independent (a requirement for Gaussians). It’s therefore 80/20: where 20% of the project will yield 80% of the results, where 20% of the scope can be cut down.

Anything that is not “statistically independent” (like tossing a coin) can’t be averaged out, can’t fit in a Normal distribution. It’s most possibly a Pareto, Power Law, Zipf, Exponential, or even a Poisson, not a Gaussian. Software parts are all intertwined in a network, they are not independent events, they can’t be randomly rearranged, they are not easily interchangeable.

Conclusion

This article is very short and I can expand each bullet point above into its own, very long, article. Let me know in the comments section if I should, and which points raise more questions.

I’ve been refining those techniques for the past 10 years, and at least 5 within the constraints of a tool such as Tracker. And I believe I was able to come up with the minimal set of principles that lead to proper management, not superstition, fantasy or delusion you see everywhere nowadays.

Projects suffer when the recommendations above are not followed:

• Product Owners don’t perform their responsibilities: they fail to add details to stories, they neglect delivered features and don’t test them properly, they don’t participate.
• Backlogs are incomplete, lacking detail, lacking estimation, lacking prioritization.
• Developers don’t properly start and finish their tasks with proper code quality.
• Time is wasted in bikeshedding instead of delivery.

Our fork of Fulcrum, called “Central”, is not perfect by any means, but it works well enough and gives us enough control to keep adding features that make our routine more comfortable, and we are just getting started.

I hope this technique and tool is valuable to more teams and I’d love to hear feedback from anyone.

Start simple with the article that lends its title to this compilation, Processes and Methodologies Won’t Help You, which I published on Abril’s blogs in 2011. It’s the starting point for understanding what processes, methodologies, and procedures are.

Next, go straight to the more recent Agile: the Truth Behind the Method and understand once and for all: no methodology or magic recipe will save incompetent or bad-character members of any team. After trying to improve attitude and training without success, you only rid the team of these elements one way: by replacing the members. Nothing saves a Net Negative Producing Programmer.

Many are left with that feeling of “waste”, like “man, but we’ve invested so much in him, what if we invest a little more?”. That comes from failing to understand that Sunk Cost is better than increasing your Opportunity Cost.

On the same “Agile is Dead” theme, or, more explicitly, “Agile-by-bikeshedding-consultants is Dead”, it’s worth recalling the old theme of The Problem of Credentials and Argumentation through Intimidation. The credential guy has the easiest job in the world: he shows up at your company to implement “practices” he doesn’t really practice. And trying to implement it at one or half a dozen places is ridiculously the same as nothing.

When it “seems” to work, he puts the case on his portfolio list as a “success.” When it goes wrong, he says “you didn’t do it right.” Terrible.

Forget pompous names and different schools of methodologies. Everything starts from exactly the same principle: Scientific Method vs Cargo Cult, where it’s clear no one understands the main thing about the method: it doesn’t pick sides or root for any team.

The phrase “it’s proven to work because I’ve seen it working in 100 places” doesn’t exist.

Anyone who uses a phrase similar to this is clearly an amateur. By Karl Popper’s principle of “Falsifiability”, instead of trying to prove something works, it’s better to try to disprove that it works. Because it doesn’t matter how many tons of evidence we manage to accumulate that something is true, a single piece of evidence that makes it false is enough to destroy any hypothesis. That’s why the scientific body is robust, not because we have mountains of evidence in its favor, but because what we’ve accumulated so far hasn’t been proven false.

But human beings are fallible. Our non-rational brain, a leftover of our long evolution, is still exceptional at seeing patterns where none exist, as I describe in We Are Mathematically Ignorant and Processes, Methodologies and the Human Brain.

The most accepted hypothesis is that we tend to confuse correlation with causation. Because “A” came before “B”, we tend to think that “A” caused “B”, and most of the time this isn’t true. The world is more random than we’d like to accept. Shit just happens.

This leads to the pseudo-intellectuals who want to push their “Southbeach Diet” of the day and become mere trolls or haters, as I describe in Math, Trolls, Haters and Internet Discussions. They create more and more diets, as I talk about in #noEstimates Debunked, Demystifying the Kanban method, and Watch Out for Kanban-butt.

And all of that I linked above is just the introduction to the most important thing of all: be skeptical all the time, nothing is automatically true, no matter who’s saying it. And as Carl Sagan would say: “extraordinary claims DEMAND extraordinary evidence.” In the end, we always come back to the theme of Enemies of Reason.

The problem is: if none of this works, what do you do? It’s much easier to say what’s wrong than to say what to do. The problem was never the recipe. Everything comes down to the decisions you make.

Each decision brings consequences, which lead to new dilemmas, which require more decisions. As Neo would say in The Matrix Reloaded: “The problem is choices.” There simply is no magic recipe, diet, or methodology that will make you make better decisions all the time.

There’s a desire to seek an external tool, an external source of “validation,” one of the worst words invented in recent times. The catch is that decisions carry consequences, and most people simply wish they didn’t have to bear them. That’s the root of the problem. This is the comfort zone: everyone complains and wants things, but no one wants to make decisions, much less face their consequences.

The only way to make better decisions is to study and train, as much as possible, all the time, continuously. Experience isn’t doing the same thing for 10 years straight, but becoming very good at 10 different things over those 10 years. Following the trend won’t help you, just like following magic diets. They leave you a bit thinner at the beginning, and soon the rebound comes and leaves you heavier than you were before. That’s why Processes and Methodologies Won’t Help You.

The trouble is using the phrase as an excuse. It turns into “we don’t need processes, just let people choose,” or “people choose, so we’ll have consensus on everything,” the fallacy of the democratic organization. At bottom, it’s a way to justify not understanding the value of having processes.

Agile, derived from Lean, is entirely based on processes. And the pillar of original Lean (not the derivations like Lean Startup or Kanban) is “Kaizen”, that is, “continuous improvement”.

The idea is that today should be at least a little better than yesterday. What we do is measurable, and because of that we can say that today came out better than yesterday. The principle is simple; the execution, not so much. If you haven’t read them yet, follow the links in this paragraph to understand these concepts.

Now comes the bad part: how do we talk about people? Or, more specifically, about people’s behavior?

Since we’re all “people,” we’re sure we know how people think. And nothing could be further from the truth. We carry preconceived ideas, fallacious, superstitious, and prejudiced, many of them long outdated. From being repeated so often as clichés, we still believe them, the same way plenty of people still believe we only use 10% of our brain or that you can’t have mango with milk.

And we don’t have to guess. Just like with processes, there is a long literature on people: psychology, sociology (which evolved a great deal since the 90s thanks to the study of networks), behavioral economics, among others. It’s impossible for someone from tech to become a specialist in the Humanities, but if the responsibility is to prioritize people, on top of processes, we’d better equip ourselves well enough.

This is a Small Bite because I don’t intend, this time, to go into detail on the subject. I’d rather leave a few books I really like to give some grounding.

Within the tech field itself there are some classics worth reading:

• Peopleware by Tom DeMarco
• The Mythical Man Month by Fred Brooks
• The Psychology of Computer Programming by Gerald Weinberg

From there we can move on to Behavioral Economics and Psychology:

• Thinking, Fast and Slow by Nobel laureate Daniel Kahneman
• Predictably Irrational by Dan Ariely
• The Upside of Irrationality: The Unexpected Benefits of Defying Logic by Dan Ariely
• The (Honest) Truth about Dishonesty by Dan Ariely
• The Lucifer Effect by Philip Zimbardo
• Talent Is Overrated by Geoffrey Colvin
• Descartes’ Error: Emotion, Reason, and the Human Brain by Antonio Damasio
• The Black Swan by Nassim Taleb
• Fooled by Randomness by Nassim Taleb

And, more specifically, about the behavior of various agents in groups (network theory):

• Linked, the New Science of Networks by Albert-László Barabási
• Six Degrees: The Science of a Connected Age by Duncan Watts
• Everything Is Obvious: How Common Sense Fails Us by Duncan Watts
• Sync: How Order Emerges From Chaos In the Universe, Nature, and Daily Life by Steven H. Strogatz
• Chaos: Making a New Science by James Gleick
• Nonlinear Dynamics and Chaos by Steven H. Strogatz
• The (Mis)behavior of Markets by Benoit Mandelbrot

Yes, a lot of these books are pretty superficial (none is technical or specialist), but the intent is just to give an introduction to the various areas. Over time, I dropped some that were too superficial, already in the entertainment and self-help category, like Malcolm Gladwell, Dan Pink, Clayton Christensen, and Clay Shirky. Even so, some of them are worth reading.

The main thing is the conclusion: contrary to what we usually think, we, people, aren’t as rational as we’d like, and our behavior is heavily influenced by external agents and by the environment.

A few articles that can set up the rest of the discussion:

• by Dave Thomas: Agile Is Dead (Long Live Agility)
• by Richard Bishop: Agile Is Dead: The Angry Developer Version
• by Giles Bowkett: Why Scrum Should Basically Just Die In A Fire

At my company, since day one, we don’t use the word “Agile,” “Scrum,” or any other buzzword. We do have “sprints,” but things like grooming and planning poker (just kidding!) are by no means “imposed.” The practices get used, with or without the names, whenever they’re needed. I don’t like saying “Scrum,” “Kanban,” or any other label; mentioning these names gives me a mild sense of embarrassment.

For anyone who actually applies agile practices day to day, none of this is news. Nobody argues about things like Collective Code Ownership (Github!), Continuous Integration (Jenkins, Travis, Semaphore, etc.), Test Driven Development (Rspec, Jasmine, Capybara, etc.), Refactoring, YAGNI, User Stories/Requirements/Use Cases (or whatever you want to call it: basically, scope). Look at how open source projects run: in terms of programming practices, it’s not much more than that.

The problem is the process: project management itself. People talk endlessly about Waterfall vs Agile, but that’s a load of bullshit. What’s worth lamenting is how the world of consultancies “specialized” in “Agile processes” has prostituted Agility.

And it’s true. That’s why I don’t “sell” things like “methodology implementation,” and I honestly don’t recommend anyone hire a third party for it. If you really must, size up how much hands-on time in the actual code and the day-to-day the candidate really has. Most of them rack up more “verbal” hours than hours of action.

In this article I won’t give the solution, that’s for a future one. Here I just want to state the one big truth nobody warns you about, which is why any implementation attempt will fail: implementing any Agile practices, processes, and methods requires a good team, programmers and managers and everyone else.

Does “good team” mean a team of nothing but “seniors”? No. It requires people committed to their craft, whether juniors or seniors. And by the way, the folks many people today call “seniors” or “ninjas” (argh, whether it’s because they give talks or because of their many years of “experience”) are almost all barely committed and merely arrogant. Again, more talk than action.

This point is crucial: plenty of people will try to implement “Agile” hoping to turn a bad team, or a bad company, into something better. That’s not going to happen. At first it’ll look like things “improved” or at least that something “changed.” But of course: you’re implementing a few merely “different” practices, so naturally something will change for a while.

Programming is a profession of practice. You can’t turn a bad programmer into a good one just by bolting on methods, the same way making a rookie soccer player sit through a two-day training will never turn him into a goal-scorer. Only hard, uninterrupted, consistent practice, made of small continuous improvements, will maybe get him there. In my cynicism, I told Bob Martin directly my theory about what motivated Ken Schwaber to create something as silly as Scrum Certification, at Railsconf 2010.

Anyone who was programming in the 80s and 90s didn’t see anything that disruptive when the descriptions of the principles and practices called “Agile” showed up in the late 90s, up to the publication of the manifesto in 2001. What was different is that someone finally condensed everything into a marketable commercial package, mostly Scrum (which on its own includes none of the important programming practices from Extreme Programming). Want to see how the practices have been evolving for far longer? Read The Cathedral and the Bazaar and The Mythical Man Month.

So no: implementing Scrum, or even starting to implement XP, will not save a bad team. One here or there, who already had the disposition, the talent, and the drive, will manage to become agile after practicing, screwing up, and practicing more. The rest, who don’t have the talent or the disposition, or worse, are flat-out bad-character, won’t change.

No methodology on Earth will ever change a person’s personality, at least not in so little time. Think of it this way: his mother couldn’t get the guy to quit being lazy and a stall-tactic artist, you won’t either, and that’s not your job anyway. Your job is to deliver value.

The unspoken truth of the Agile world is that applying agile practices comes naturally among good programmers who already have experience. It won’t be useful on a rookie team, which still needs a lot of practice, with or without agile principles.

I said you just have to look at how the open source world works. Contrary to what a lot of newbie Brazilians imagine, open source has, in principle, nothing to do with inclusion or leftist movements. I’d sum up open source as “the most efficient capitalist way to maintain commodities.” It’s highly exclusive to developers.

With no employment ties, on a purely voluntary basis, the interests revolve around marketing: company exposure, developer exposure, cost reduction, opportunities. You really do need to be “good” to stay at the top of the list. Does everyone’s opinion carry equal weight? Ask Linus Torvalds what he does when a bad idea shows up.

The open source world exposes problems immediately. Just because you think your project is “cool” doesn’t automatically translate into volunteers and exposure. The rate of projects that die is orders of magnitude higher than that of projects gaining traction. Compared to the commercial world, open source kills projects far faster.

Agile was built to expose problems as early as possible. Understand: Agile practices are problem-exposure mechanisms! Agile is a Risk reduction and management mechanism! Catch the obstacles the moment they happen and deal with them immediately to head off unnecessary future risk, the kind that only leads to waste.

Except nobody wants to do this. If the problem is the manager, what consultancy is going to remove him from his post, unless the owner above him is committed to cutting into his own flesh? If the problem is the bad-character programmer everyone likes (the most dangerous type: the one who knows how to work a room and build zones of influence among peers and superiors), how do you fire that guy?

Yes: fixing a team in a short window necessarily means replacing its members. Worst case, replacing all of them. If the method is applied well, it immediately demonstrates and exposes the problems. And since in the vast majority of cases the problem is behavior, not practice, you have to resolve it on the spot. If you, the stakeholder responsible for your company, aren’t firing anyone after you start implementing Agile, you’re doing it wrong.

And here’s the big problem: nobody gets this commitment. There’s no free lunch. There’s no making an omelet without breaking some (or all) of the eggs. Without that understanding and that commitment, there’s no agility. And this isn’t the first time I say it, if you read my 2009 article: Net Negative Producing Programmer. But since most people still seem not to have gotten it, I decided to spell it out more clearly.

This was an introduction. Depending on the comments, maybe I’ll walk down a few more steps to explain exactly the mechanisms of what I laid out very quickly here.

The year was 2001. A group of senior programmers gathered at a ski resort in Snowbird, Utah. The main names of programming in the United States were there: Kent Beck, Alistair Cockburn, Ward Cunningham, Martin Fowler, Jim Highsmith, Ron Jeffries, Brian Marick, Bob Martin, Ken Schwaber, Jeff Sutherland, Andy Hunt, Dave Thomas, Mike Beedle, Arie van Bennekum, James Grenning, Jon Kern, Steve Mellor. A “who’s who” list of the field.

From that historic meeting a manifesto of 4 values and 12 principles emerged, which became known as the “Agile Manifesto.”

Thirteen years later, Dave Thomas, author of “The Pragmatic Programmer” and one of the people most directly involved in writing the manifesto text, published a frustrated article called “Agile is Dead, Long Live Agility”.

The original agilists meant to break the outdated way of thinking about and executing software engineering, the same obsolete model still taught today in most of the country’s universities. It caused a huge stir in the industry.

Along came the derivations, the trainings, the tools, the certifications, the conferences, the consultancies. A huge portfolio of options, for any size and any price. In the middle of this whole circus, what it really means to be an agilist got forgotten.

And, as Dave Thomas summed it up well in his vent, you don’t “do” Agile or “buy” Agile. Agility is a quality, an adjective. To be agile, you have to go back to the principles. Quoting:

• Find out where you are
• Take a small step towards your goal
• Adjust your understanding based on what you learned
• Repeat

How to do this:

“When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier.”

And that’s it: four lines and one practice.

Thirteen years after something so simple was first published, the world seems to have gotten way too complicated. Scrum, Kanban, Scrumban, Lean Programming, Lean Six Sigma, Agile PMI. In a moment where chaos reigns, whoever “seems” able to explain the chaos becomes a guru.

And so associations, certifications, trainings, and conferences emerge. That’s how it works for diets, how it works for therapies, how it worked for software engineering. That’s what Dave Thomas laments.

But it’s an inevitable result. People are easy to fool: they always look for someone or something to follow and rarely question. They’ve been dogmatized never to challenge authority and to root for soccer teams without any criteria.

In 2010 I was at a conference in Baltimore and had the chance to talk to Bob Martin, who organized the Snowbird meeting. It was in his Object Mentor office that Ken Schwaber’s first Scrum certification training took place.

I was already immune to the “Agile Hype,” but I wanted to tell Bob Martin what I thought of it. For those who don’t know, the “official” Scrum Master training is a two-day course, and at the end you walk out certified. You don’t even need to show up: just pay for the course.

I told him, cynically, that Ken’s idea probably backfired. “Ken probably thought ‘man, how do I convince these large corporations to adopt agile practices?’ and then he thought ‘of course, large companies love certifications,’ and then he created a Trojan Horse, a joke 2-day course where serious people would know you can’t learn anything serious in 2 days, but that way he could insert the idea into the companies. But everyone took it seriously, and that’s when the shot backfired.” Skip straight to minute 6:24 of the video below if you want to hear that story.

Learning to Learn

Now let’s step out of software a bit and enter the world of “startups.” That world has its own “Agile,” dubbed “Lean Startup.” Eric Ries coined the term in 2008 and made it famous with the book of the same name, from 2011, a direct derivation of Toyota’s Lean principles. It’s a repackaging of old ideas.

This Lean comes from the same ideas of W. Edwards Deming about PDCA, “Plan, Do, Check, Act”. You’ll notice the resemblance to Dave Thomas’s four principles I cited above.

Pompous names, celebrities, gurus, expensive tools, and prominent titles don’t matter. The basic principles of Agile and Lean are the same: deliver value as fast as possible with the least waste possible, learning and using that learning in the process, planning and replanning to reach goals that become clearer along the way. That’s it.

The biggest problem shows up when an amateur programmer asks “what’s the next step?” and the answer is “the goal is to set up a Kanban.” Or when an amateur entrepreneur asks “what’s the next step?” and the answer is “the goal is to build a Business Canvas.”

Kanban and Business Canvas aren’t goals, they’re means. If those means don’t help reach the ends, delivering value to the customer and turning a profit in your business, then they’re disposable. The goal is to have profit. MVP, prototype, Lean UX, and Business Canvas are just possible paths to get there. Which of them serve?

If you want to learn anything, the steps to start are simple:

• Don’t create idols, never “idolize” anyone, at least not seriously.
• Ignore almost everything you see or hear in groups. Take notes, but don’t take everything seriously.
• Ignore advice and mentoring. It’s like a bank manager giving investment tips: if he were any good, he’d be rich instead of selling advice. He’s a salesperson by another name.
• Everything, absolutely everything, starts with a few principles. Nobody becomes a heart surgeon wanting to cut the aorta before learning to wash their hands. Look for the principles and ignore the pompous titles.
• Assume you know little, always. Approach every subject with the eyes of an ignorant person. Whoever thinks they already know everything learns nothing more. But be a smart ignoramus, not a dumb one.

Learn to question. The whole world of the Sciences, from the absolutely small quantum world to the infinite and immense cosmological world, only exists because it accepts one principle: the Scientific Method.

It’s basically PDCA: define the problem, make hypotheses, run controlled experiments, discard the hypotheses proven wrong, repeat. To be more accurate, Deming’s PDCA is the one that draws directly on the Scientific Method.

There is the principle of everything. From Agile to Lean to PDCA, we come back to Karl Popper and his philosophy of “Critical Rationalism”, which teaches that everything should stay open to rational criticism.

That’s why Lean, as an umbrella for selling products and services you don’t need, is dead. Long live the principle of achieving value while avoiding waste through continuous learning, in other words, Efficiency. There are no magic formulas. If magic diets worked, there would be no obesity in the world.

PS: I know, there’s #tongueincheek in the title and in the article, but the intention, as usual, is actually to provoke ;-)

The distinction matters. A junior professional, who still has technical deficiencies and wants to grow, learns techniques, practices, and tools, and gains technical quality, productivity, and efficiency. A bad professional, who may even have some technical capacity, is more interested in little work, little effort, and little responsibility, and turns bad-character in conduct and attitude. That one doesn’t change.

Agile is another formula whose image (result) is the promise of hyper-productivity that can reach 500% of the current. Again, what is the Domain (the premises)?

The most basic premise for Agile is that the team has technical capacity and commitment. No methodology teaches or changes that. Either each individual on the team is a capable person or it simply won’t work. Well-made, well-organized code, following the known good practices, with tests and discipline, is the minimum to start, the rice and beans.

If that doesn’t exist, give up. No methodology, no voodoo, no exorcism, no feng shui will help. Dropping Scrum, XP practices, Kanban, or anything else into a bad team with weak leadership produces some of the following prostitutions of the good practices:

Planning for Show

Good planning is only possible if the Product Owner commits to keeping a prioritized, well-defined backlog. And understand this: the definition of the User Stories should be ready before the Sprint Planning. A vague, poorly defined User Story helps nothing at all.

While the team works on the Sprint, the PO defines the upcoming User Stories, and can pull in one or more members of the team itself or of parallel teams to help. By the Sprint Planning everything should already be well defined, ideally pre-pointed and prioritized. The team just points the Sprint and, according to the current Velocity, sees what fits in the next one. A well-run meeting lasts 1 or 2 hours at most.

Symptoms: if it takes more than 2 hours, has too many people just to make quorum, poorly defined User Stories that spawn endless discussions and no action, you’re doing it wrong.

Pair Programming for Show

Pair Programming is a very good practice, and the team decides how to run it. Except you can end up with a member who almost never programs well alone and, when doing the “pair,” turns into more of a listener than a participant. That’s the famous “dead weight.” Sits there tossing out opinions, or just sits in silence playing Candy Crush and browsing Facebook. No value at all.

Symptoms: this happens more easily on very large teams (above 6), since it becomes easier to “hide” doing pair-fake with different people. In fact, I never recommend more than 4 people on an agile team. Even for a single product, if there are 15 people, for example, better to split into 3 teams of 5. Better still if people rotate across those teams so that “cliques” don’t form where one bad guy covers for another.

Departmental Protectionism

Professional programmers today need to be multi-functional, versatile. Obviously each one is stronger at some type of capability than another. For example, what we’d call a “front-end” should be the master of HTML 5, CSS 3, JavaScript, but should also know the minimum of Rails, Node.js, Python, or whatever the back-end is, enough to organize the templates and integrate with controllers, models, helpers, etc. And the “back-ends” should also minimally know HTML, and even infrastructure.

Pointing User Stories (whether in Story Points, Working Hours, etc.) should always be the same no matter who pulls the User Story. There’s none of: “these 3 stories are mine, these 2 are so-and-so’s, these 4 are what’s-his-name’s.” Stories are prioritized and each person pulls one story at a time as they finish the previous one.

Symptoms: stories pointed differently depending on who’s going to pick them up, or “reserved” stories, or types of stories only one “type” of professional can pick up. There’s none of “this story is infra, only an infra guy should get it, and the back guy shouldn’t even be looking at it.” Sprints are the responsibility of the entire team.

One of the advantages of pair programming and small teams is that each person learns new capabilities from the others. A back-end dev can pull a Story with something of front-end in it, without being the best at front-end, and nothing stops them from asking to pair with a good front-end on the team when they need it. A User Story may take 2 hours if done by a good back-end, but it may take 5 hours if done by a front-end.

The right thing is for the front-end guy to get close to those 2 hours, not to reserve the story. The right thing is for the team’s velocity to climb gradually, and for that the individual velocity of each member also has to climb. And for that they have to keep learning new capabilities. That’s how everyone evolves. Reserving stories is the best example of a punch-the-clock worker wanting to ‘protect’ their job and hide their mediocrity.

Quality for Show

Every member of the team should care about both the technical quality of the code and the functional quality, that is, whether the feature actually behaves correctly with end users. Many times teams allocate people with the sole goal of being a “Q&A” (Quality Assurance), responsible for building test cases, automating integration tests, aligning with the PO and with business analysts, marketing, etc.

I’ll say that personally I don’t like this distinction, but I confess that in some cases it makes a lot of sense. Except it has to be done in a way that doesn’t strip the team of the responsibility to deliver features that actually work. There can’t be a feeling of “who cares if there are bugs, the Q&A will catch them later.” But that’s what often happens.

Symptom: look for User Stories that get delivered, then rejected, then redone, then rejected again, and stay that way for a long time. A User Story that goes 1 month without being accepted is a monstrous absurdity, an abomination. Even worse if there are team members who, when you look at the past backlog, are always involved in stories that always get rejected and always take days to be accepted. It’s the typical case of the professional who keeps their job by solving problems they created themselves. Remove those rotten apples from the team as soon as possible, because they contaminate the whole barrel very fast.

Excuses, Excuses, Excuses

“I couldn’t do it because we were missing a test, but the guy who does tests was overloaded so I got stuck.”

“I couldn’t do it because there was an obstacle in the other team that they didn’t resolve, so I got stuck.”

“I couldn’t do it because it was poorly specified, I sent an email to clear it up, no one answered, so I got stuck.”

A good professional gives solutions, doesn’t point fingers or keep looking for excuses to justify their own incompetence. Does that mean nobody can complain? Of course not. There are always hard problems to solve, and people can and should seek help when needed.

Symptoms: it’s easy to spot the professional procrastinator. The guy is never at his desk, spends more time chatting at the coffee machine than working. Shows up late every day, leaves early every day. When he’s at his desk, you glance over and the poor soul is on YouTube or on Hacker News.

And let it be very clear: none of these activities, on its own, is a problem. Even I take a break and hang out on Facebook. It’s just that whoever behaves this way constantly, routinely, is clowning around. Worse: if by the end of the day his task were finished, with well-made code and no bugs, I wouldn’t complain. Usually it isn’t. The task comes in incomplete, the code poorly made, always needing to be redone, and the guy shows up with excuses like the ones above.

That’s being bad-character, that’s a criminal’s attitude. He’s deliberately stealing from the company: his salary is being paid and the value isn’t being delivered. He’s a thief.

A good professional, who works seriously, doesn’t wait to be pressed to say “ah, I asked, no one answered, so I didn’t do it.” When they hit an obstacle, they chase the problem. Technical question, they ask the colleagues next to them (in person, email, chat, gtalk). Functional problem, they go straight to the PO. Departmental problem, they get up from the chair and walk over to the other team to clear it. If the task depends on a third party who wouldn’t collaborate, they go straight to the boss to solve it.

The basic symptom of the bad professional is the opposite: he doesn’t go after it. He’s happy when he hits the obstacle, because a week from now, only at the next Review meeting, he’ll say “daaaamn, I even tried, but so-and-so didn’t collaborate, so I couldn’t.” Give me a break.

Conclusion

If your team has just 1 of the problems described above, it’s already a catastrophe. If it has more than one, it’s a military-intervention problem, Iraq level. Anything past that, only a nuclear bomb solves it. And worse still, someone who exhibits all the bad behaviors described above is a criminal sociopath. Scolding is useless, trying to fix them is useless: there’s no fixing it… but there’s a solution: swap out the rotten apples as soon as possible.

And again, to be clear, obviously there are all kinds of bad professionals, and I’m not talking only about programmers. There are bad managers, bad analysts, bad coordinators, bad supervisors, bad directors, etc. This applies to everyone.

If you’re a programmer and you see all this, chase after it. If the problem is in upper management, I’m sorry to say there’s nothing to do. Now, if you’re a manager, a supervisor, someone in a position of authority, and you’re taking no action knowing all this: resign, you’re doing it very wrong and being complicit with an environment whose culture is bad character.

Applying ‘some’ Agile practices, interpreted the wrong way, isn’t being Agile, it’s fooling yourself. It’s like a nutritional diet. During the day you follow the diet to the letter, but when no one is looking you eat chocolate and sweets, and the next day you say “damn, I don’t know why this diet isn’t working, I’m following it right, I guess the diet’s no good, let’s try another one.” #lamentable

Let’s make this even: you do #noEstimates if, and only if, you’re willing to do #noSalary. Your employer holds your pay until you deliver, and in that scenario the amount gets depreciated the longer you take. Labor law doesn’t allow that, unfortunately. But it would be an interesting scenario.

There are a lot of people talking about #noEstimates lately. I’ve read most of the arguments in its favor, and anyone can run a quick Google search, so I won’t keep referencing each of them. The gist is this: estimates will never be good enough, and the more specification and planning you do, the less the quality of the estimates seems to improve. In a dynamic market the specs change all the time, and the bigger the estimation effort, the bigger the waste. If estimation is that much of a waste, why not throw it all out at once?

It feels like a noble idea, especially for software developers. Software is malleable, it’s abstract, and it just seems not to fit the traditional notions of project management. And while we’re at it, why not throw out the entire notion of projects too? That’s how another trend came along, #noProjects.

My intention here isn’t to answer every argument, and that isn’t even the point. What I’ll do is explain why the whole idea is absurd right at the root. So let’s start with the basics.

One thing I’ve argued since at least 2008 is looking at project management and markets in general through the models of Complex Adaptive Systems, Chaos Theory, and Evolutionary Biology. I’ve been heavily influenced by the ideas of Nassim Nicholas Taleb and his magnum opus “The Black Swan”. The idea is incredible: markets are driven by chaotic agents that influence a complex system, far from any linear path.

Every company is managed to deal with averages, with limited sigmas as the margin of operational error. But once something big happens, a “Black Swan” like the 2008 economic crisis, most aren’t prepared. No model can predict it, and the whole system locks up and collapses.

If you don’t know the idea, Google it for a second and you’ll see that companies, markets, and human relationships in general are dynamic systems that obey the rules of evolutionary biology. Decentralized systems seem to be the way to go, and those concepts influence many of the Lean movements we see today. So, yes, I’m well aware of these effects.

To sum it up, the ones with the most chance of surviving in a complex system like that are the most adaptable, far more than the ones who stick rigidly to plans. Following long-term plans with rigidity is the easiest way to fall down when the Black Swans show up.

That led me to another concept: Einstein’s General Theory of Relativity. In modern cosmology it superseded Newton’s Theory. When I first learned that, my first thought was: if Newton is “wrong”, why aren’t we using Relativity to calculate everything in our daily lives?

The answer is that Newton is only “wrong” if you demand that it calculate anything. It doesn’t apply to the very very large, to gravitational calculation, to galaxy-level calculations. But at Earth-like sizes, computing the path of an airplane or the trajectory of a bullet, it still holds, because the margins of error are negligible. In day-to-day life you can reduce the problem to Newton and leave General Relativity aside. It’s a crude simplification, of course, but bear with me.

The same applies to companies. We’re all subject to Power Law distributions, to Evolutionary Biology, to the relentless forces of Chaos that make everything behave like Complex Adaptive Systems. But in a constrained environment I’ll argue that you can reduce the calculation back to Bell Curves. That’s the hardest part to “prove”, so I won’t try right now, but the explanation that follows may get you there.

Let’s define what a company is: a set of operations. Operations are repetitive activities, like “pay a supplier”, “send a purchase order”, “process the payroll”, or “transport products”. The set of all those activities defines what a company is.

The core idea of a company is to run those operations as efficiently as possible. You do that by continuously refining the process in small steps, or through a breakthrough that completely changes the way you run a particular operation.

An example: back in the day there were entire groups of people dedicated to filling out paper forms and organizing them so information could flow inside the company. With the rise of digital systems, of ERPs, all that paperwork stopped being necessary. We got rid of an entire profession of typists and added efficiency and precision to the system. Breakthroughs are usually the digital automation of manual labor, or getting rid of a process entirely.

To reach those breakthroughs there are Projects. Projects are temporary endeavors where a group of people concentrate to hit a pre-established goal. They usually have a fixed start and due date, a fixed budget, and a fixed number of people involved.

And here we get to the Estimation part. Every project wants to hit some goal, and in the case of a software project we write code to get there. To do that, we come up with features and break them into Use Cases, User Stories, Requirements, or any artifact that describes what has to be built. Then we estimate how much resource (time, money, people) it takes to implement each piece and integrate them into a “solution” that solves the problem.

What developers complain about is that you can’t estimate those pieces with precision, and that projects will always come out late and over budget. Hence the idea of estimating nothing, just starting to code, delivering value as fast as possible, and calling it done only when it’s done.

Some developers get so fed up with this whole notion that they want to leave their companies and start their own startups, where they’ll get to do whatever they want with no controls at all. Then off they go looking for investors, because they need a lot of money and a lot of time. Of course they do. What they don’t realize is that EVERYBODY needs a lot of money and a lot of time.

And investors know this: the idea is irrelevant, execution is everything. The ones who deserve more money and more time are exactly the ones who can pressure themselves into delivering under budget and ahead of everyone else. Just doing something without worrying about constraints is the realm of the mediocre. And the mediocre deserve nothing.

Back to basics: there’s a thing called Economy precisely because resources aren’t infinite. Everything that has value has a price, whether it’s a physical product or working hours.

Understand this: in the services business, where we developers all live, whether as an employee or a co-founder, value has only two variables, quality and efficiency. We tend to see quality as the only thing that matters. Worse: we tend to see what we think is quality as the only thing that matters.

This brings us back to CONTEXT. Most programmers are bad at estimates, and the root of it is that they’re usually utterly incompetent at understanding context. As someone with a Math background, I read all those articles about processes, methodologies, and things like “#noEstimates” as if they were “formulas”.

Formulas alone don’t mean a thing. Any mathematician knows you have to define a Domain and an Image, the source and the destination of every input and output. If I show you a formula like “f(x) = 1 / x”, you can say it’s invalid, because you can’t divide when x is zero. But if I tell you the Domain is every Natural number except zero, now it’s a perfectly valid formula for the Image of the Rationals.

So when someone says “#noEstimates”, the natural question is: in which Domain and for what Image? That’s the origin of the confusion in most Internet arguments. People argue for or against an idea because each one is in a different Domain. Same goes for Agile in general, Lean Startups, Lean Manufacturing, and so on: they tend to define only practices, procedures, formulas, but they rarely define Domain and Image. That creates confusion and misses the point.

What I’ll define is this: Projects are necessary every time there’s a defined goal to hit. That’s the domain. And I’ll also state that whoever says “#noEstimates” is thinking about the Domain of “Ongoing Operations”, while Projects are a separate Domain. That’s exactly where Lean Manufacturing in general fits too.

I already explained Operations above, and it’s where the small improvement steps (Kaizen) emerge. Sometimes the feedback from an operation gives enough input to justify a Project and take a bigger step, a breakthrough.

Projects, on the other hand, remain temporary endeavors. The whole idea is to set boundaries, like time and cost constraints. And we come back to the original question: is it impossible to predict the effort needed for something as malleable as software development?

First of all, yes, it’s impossible to predict with exact precision. Let’s define it again: it’s impossible to predict a number with zero margin of error. Estimation is prediction with a margin of error. And why do some projects cost more than twice as much and take more than twice as long as the estimate?

Most of the time, in 90% of cases, because the team is incompetent. The problem isn’t the estimate, it’s the execution. Estimation is the establishment of an expectation, and expectations have to be managed. An estimate is only good if the context is taken into account.

Most people don’t like to estimate because of the “what if” scenarios. What if the client changes their mind? What if we hit a hard obstacle? What if a meteor strikes the Earth and all living creatures perish? “What if” can’t be managed; what can be managed is what you know, by creating constraints. A project’s constraints start with its goal. To hit that goal, we also set the rules of engagement, the premises. Without goals and without premises, there’s no game.

None of that guarantees a Prediction. An estimate is only as good as the execution. Now we have to manage it, and everybody has to manage it.

It’s no use setting clear rules and then suddenly seeing a programmer sitting idle. You ask why he’s idle, and the answer is “oh, because I emailed the client about some requirements and he never replied, so I was waiting”. You ask “and did you try calling him?”, and the answer is usually “No, I didn’t”. No amount of process or methodology “fixes” an incompetent employee. Lack of technical skill can be fixed. Bad faith cannot.

The #noEstimates advocates can argue they’re not like that, and I believe them. But 90% of the projects that failed had employees like that. Programmers tend to blame the client, the bosses, the market, but never themselves. And as a programmer I’ll argue that most projects fail because of lazy employees, far more than because of changing requirements or limited time. Want to make projects go right? Start with Human Resources, then go find methodologies and practices.

The trouble with methodologies that don’t state the Domain their formulas work in is that most people don’t see where that Domain begins: at “having competent, committed, and skilled employees”. Most adopt methodologies hoping to turn incompetent employees into competent ones, and that just doesn’t happen.

With that settled, why do we need estimates? Or, more generally, why do we need constraints? Because that’s the essence of the value of any system. Nature puts pressure on every living species: changing climate, limited food, predators. The most adaptable species evolve, and the ones that can’t adapt perish.

When someone says “it’s impossible to do X”, that’s probably the most valuable goal to chase. Because the full sentence is “it’s impossible to do X with what we know today”. It was impossible to go around the globe in 24 hours, now it isn’t. It was impossible to talk to the other side of the world in real time, now it isn’t.

Constraints are the foundation of innovation. If I had to define innovation, I’d say it’s “the process by which you accomplish something that was deemed impossible before”.

If you have infinite resources, or you simply don’t need to worry about constraints, which is what happens in bubbles, you don’t innovate. This is so important that I’ll repeat it:

Innovation is the byproduct of Constraints.

We estimate based on past knowledge. If we can’t outdo our own past selves, how incompetent are we? You can argue that software isn’t predictable, and I agree. But being off by orders of magnitude when recreating similar software just smells of incompetence to me.

Most of the software we produce is the same old thing: content websites, ecommerce, elearning, social networks, social commerce, forums, polls. It’s rarely a brand-new idea or a breakthrough algorithm. Unless you work in research programs, what kind of truly different software have you built lately?

Estimation is rarely the problem. Every estimate starts from a set of premises, and the problem is not managing those premises. If the requirements change, fine, that we can always manage. What we can’t manage is the pile of problems on the last day of the project, after the programmers kept pushing everything under the rug. Again, this is a Human Resources problem.

Meritocracy only exists in a system of scarcity, where one stands out against another in the face of a constraint. In a system of pure abundance, there’s no need to innovate or to have merit. Companies that just got a humongous pile of cash invariably show symptoms of laziness, far more than of innovation.

The confusion comes from some of the “practices” being pushed coming out of this temporary, unreal bubble situation, and they can’t take the pressure of time. Give it enough time and you’ll ask yourself: “how did we, with so much money and so much time, get so little done, and that little startup, with such limited resources, managed to outdo us?”. That’s the answer to why a Yahoo! buys a Tumblr, why a Facebook buys an Instagram, why a Google buys a Waze.

If you advocate #noEstimation, why not go one step further and advocate #noWork? It’s just one more step.

I got back last weekend from a trip to Baltimore, Maryland, where I spoke at RailsConf. It was an event full of great talks, some aimed not just at Ruby on Rails but at the programming career in general. One of them was by no one less than Robert Martin, of Object Mentor. I translated an article of his in my previous post.

I recommend watching the talk in full.

Since I started this blog, I’ve tried to explain the conditions and difficulties a Project Manager faces. But I’ve been careful not to repeat the term “Agile” at every turn. We’re in the middle of an “agility” fever. Every CIO, CTO, manager, or consultant, to seem modern, says they know “Scrum” and that they’re “agile”.

That has a good side: some companies might finally adopt these ideas the right way. It also has a bad side, and a big one. We’re burning the name “Agile” because of bad consultancies that implement everything wrong at the client. The result is the perception, among the uninformed, that “Agile doesn’t work”.

Well, during RailsConf, I had the chance to talk to and interview Robert Martin, whose recording you can watch below:

From this interview, the part I want to present is the story of how “Agile” came about. Robert Martin, also known as “Uncle Bob”, has been a programmer for decades, certainly more than any of us. In the 90s he was one of those who noticed the rise of “light” or “lean” methodologies. Much of that came from the manufacturing concepts that revolutionized Japanese industry from the 60s onward.

Several big names started proposing smarter ways to develop software. Kent Beck with Extreme Programming (XP); Ken Schwaber and Jeff Sutherland with Scrum; Alistair Cockburn with Crystal.

They all followed a similar line: short iterations, constant feedback, and a lot of communication. Seeing that pattern, Uncle Bob reached out to Martin Fowler, another strong name in the world of software architecture, and together they began organizing a meeting, summoning a group of these professionals.

The legendary meeting took place at the Snowbird ski resort in Utah, between February 11 and 13, 2001. Seventeen professionals spent three days there and walked out with the “Agile Manifesto”. It brings together 4 values and 12 principles that all the participants accepted as the minimum common denominator in the practice of software development. The manifesto says:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

As I said before, there’s a growing fever in the adoption of agile methodologies. Worse than that, there’s a tendency to expropriate the term “Agile” and its derivatives, like “XP” and “Scrum”, creating real frankensteins that blend the old way with the new. It’s the case of trying to join RUP with Scrum, or PMI with Scrum, and so on.

These are opportunistic attempts riding the fever to make easy money. Some companies, for example, insist on a “Certified Scrum Master” in a manager’s resume, among other equally irrelevant things.

And, to make it worse, many try to implement agile methodologies “by the book”, that is, dogmatically. Every dogma, by definition, is dumb.

Back to the 4 values: the priority is working software and value to the client. Implementing a methodology comes after that, if it comes at all. The first value is clear: it values Individuals and Interactions much more than Processes and Tools. What we see today is the opposite, an effort to push processes, methodologies, and tools.

Now is a good time to start demystifying the Agile world. This here is just an introduction, but the advice Uncle Bob gives in the interview holds: follow the originals. If you’re really committed to improving things and delivering projects with quality and value, drop this new generation of consultancies. Start with the first books by Ken Schwaber, Kent Beck, Alistair Cockburn, and Mike Cohn.

Ask yourself all the time: am I in agreement with the 4 values?

If you’re putting more emphasis on the process than on people, you’ve already started wrong. Very wrong.

The human brain is a fantastic machine, and saying that is basically a redundancy of the term. But there are many anecdotes and mysticism when people talk about it.

Everyone has at some point discussed the separation of “Emotion” from “Reasoning.” Since the time of Plato, emotions have been seen as something that gets in the way of logical thinking. Worse: most people believe that, at least in the most important decisions, we’re capable of setting emotions aside. The idea is to think according to classical economics, based on facts, evidence, and cost-benefit.

But in the vast majority of cases, that’s not how we behave. When we go to the supermarket, we rarely stand there doing arithmetic, weighing each product’s ingredients or nutritional value. Usually we look at a product and decide whether to buy it based on how we feel.

Among dozens of brands for the same type of product, we choose based on past experiences, on some memorable event, on memories, and all in fractions of a second. We prefer to buy an “80% light” product over a “20% fat” one, even though both phrases say the same thing.

Antonio Damasio is a well-known neurologist who studies the brain and our behavior. He followed patients who literally lost the ability to have emotions because of brain damage, whether from a tumor or a genetic problem. We might imagine, as the philosopher Plato did, that a person without emotions would make the best decisions all the time, since they wouldn’t be influenced by them.

But without emotions, these people can’t learn. Any trivial decision becomes a cost-benefit analysis. Everyday choices like what clothes to wear, what TV channel to watch, tea or coffee, all the mundane things we decide almost instantly, can take hours before this type of patient reaches a conclusion.

A fact we already know: all emotions, sadness, compassion, joy, depression, pleasure, are chemical reactions in the brain. And we already know how to influence or control many of them.

Drug users know the sensation of pleasure. People who take antidepressants do too. They’re all chemicals that act on the brain and modify our emotions. Schizophrenia and Parkinson’s, on the other hand, are damage to this chemical system.

Another fact is that our brain evolved to learn. We learn through trial and error. We learn by observing and inferring patterns. Emotions strongly influence our decisions, and that’s good.

When we learn some pattern, dopamine is released in the brain and creates a sensation of pleasure. We like to discover the pattern of things and to try to predict future events.

Take an experiment with monkeys: we ring a bell and then give a banana. The monkey learns the pattern “after the bell comes the banana.” You can measure dopamine being released the moment the bell rings, because the monkey starts to anticipate that event.

If we turn on a light, ring the bell, and then give the banana, the monkey learns that the “light” event precedes the bell and then the banana. The pattern can be repeated with several stages, and the monkey learns it. But if we follow the pattern and the banana doesn’t come at the end, the monkey gets sad, because the prediction failed.

We also work this way.

For many simple events, with few variables, this works well. Perhaps that’s why many like routine, where the results are well defined.

Our brain, however, has a flaw: in its eagerness to find patterns, it tries to fit them where they don’t exist. That’s how superstitions are born. Before we understood meteorology, primitive tribes thought “it’s raining” was tied to a dance they had just done, and imagined that repeating the dance would bring the rain back.

I translated on my other blog a Scientific American article showing how we are “mathematically ignorant.” We try to find a pattern for everything. A passage that illustrates our superstitions and explanations about coincidence is this:

It’s always possible to combine random data and find some regularity. A very well-known example is the comparison of coincidences in the lives of Abraham Lincoln and John Kennedy, two presidents with 7 letters in their last names, and elected 100 years apart, 1860 and 1960. Both were assassinated on a Friday in the presence of their wives, Lincoln at the Ford Theater and Kennedy in an automobile made by Ford. Both assassins have 3 names: John Wilkes Booth and Lee Harvey Oswald, with 15 letters in each full name. Oswald shot Kennedy from a warehouse and ran to a theater, and Booth shot Lincoln in a theater and ran to a kind of warehouse. Both successor vice presidents were Southern Democrats and former senators named Johnson (Andrew and Lyndon), with 13 letters in their names and born 100 years apart, 1808 and 1908.

But if we compare other relevant attributes, we fail to find coincidences. Lincoln and Kennedy were born and died on different dates (day and month) and in different states, and none of the dates are 100 years apart. Their ages were different, as were their wives’ names. Of course, if any of these had matched, they would be on the list of “mysterious” coincidences. For any person with a life reasonably full of events, it’s possible to find coincidences between them. Two people meeting at a party often find shocking coincidences between them, but what they are – birthdays, hometowns, etc. – aren’t predicted in advance.

The real world is full of randomness, and it isn’t hard to find patterns in it. Actually it’s very easy. Complex situations have many random aspects, more than we imagine. And our brain hasn’t evolved enough to feel comfortable in a totally random world.

An example is successful people. Imagine a mega-investor like Warren Buffett. Books upon books get printed, with analyses, biographies, and theories, trying to explain the process that takes an ordinary person to earning billions on the stock exchange. Just as speculation: what if Warren Buffett is nothing more, nothing less, than a statistical accident?

Nassim Taleb explains this in a Malcolm Gladwell article:

Suppose that there were ten thousand investment managers out there — which is not an outlandish number — and that every year, entirely by chance, half of them made money and half of them lost money. And suppose that every year the losers were tossed out, and the game replayed with those who remained. At the end of five years, there would be three hundred and thirteen people who had made money in every one of those years, and after ten years there would be nine people who had made money every single year in a row — all out of pure luck. And who’s to say Buffett isn’t one of those nine?

Our brain takes pleasure in looking for patterns in everything, especially in success stories. We want to find the recipe, the magic formula. Sometimes we get that feeling of “what does he have that I don’t?”

Sometimes two people are technically equally capable, but one gets rich and the other doesn’t. The difference can simply be randomness. “But that isn’t fair!” you complain. The world wasn’t made to be fair. Justice is a human concept, and reality has no sense of morality: it simply “is.”

Science, fortunately, has a procedure to avoid this trap of the brain: the scientific method. Everything can start with one of these misfirings, thinking it found a pattern and generating a theory. Now that theory has to be put to the test.

For it to become a good theory, we need not only to try to prove it but, mainly, to try to disprove it. And even if we don’t find a way to disprove the hypothesis, that doesn’t make it a “Law.” Science is very careful with the claims it makes.

Even if a theory works well a thousand times, a single failure is enough for it to be rejected. It’s thanks to this way of thinking that we have solid and reliable foundations in science, free of superstition and mysticism.

Processes, Methodologies, and Superstitions

I gave that whole explanation to get to the main topic of this blog: project and people management. Anyone following my posts noticed that at no point do I preach a methodology, a process, a cookbook recipe. Things like “how to make correct estimates” or “how to predict the success of a project.”

If you’re well informed, you know there’s a whole market of individuals and companies that try exactly this: to sell you processes and methodologies. Whether traditional Software Engineering, PMI, or the so-called “Agile methodologies.”

First I need to leave a warning: all these methodologies were thought up with the best of intentions. The idea was to share the factors that led a certain project to success.

My explanation is more or less this: the leaders of these projects, or rather, their brains, tirelessly start looking for patterns. Maybe it’s the post-it on the wall. Maybe the kind of document used. Maybe the way people sit at the table.

All of this forms a set of “pseudo-truths” that the brain understands as the “lights” and “bells” that lead to the “banana.” These leaders apply the recipe to other projects and, as Nassim Taleb explained, succeed consecutively.

Eureka! It seems we found the formula for turning iron into gold! Quickly these pseudo-truths get formatted as methodologies, canned, packaged, and sold.

But think for a moment: what if all these methodologies are exactly like the rain dance? As I said, it’s very easy to find patterns in randomness.

When we start discussing minutiae, what level of detail to make an estimate at, what metrics to use, what mechanisms for team communication, I can’t help thinking of primitive tribes discussing “What feathers should we wear? What colors on our face? Should we take long or short steps in the dance?”

But the main question never comes up: is the rain dance the thing that actually makes it rain? Or, in our case, is the methodology the thing that actually takes the project to success?

And the scientific method? The problem is that a “project” is a very complex situation. Even the smallest projects are complex, in the sense that they have more variables than it’s possible to measure.

One way to test a theory is to run a double-blind experiment, like drug labs do: one group receives the real medication and another receives a placebo, without knowing. If both have approximately the same result, the drug doesn’t work.

But a methodology is more complicated than a drug. Even if we create two isolated and identical work environments, put two groups of “approximately” similar people in each, start at the same time and with the same requirements, one side with methodology “A” and the other with “B,” we still have no way to compare the results.

That’s because methodologies involve the micro-decisions and capabilities of each person, and “approximately similar” is very different from “equal.” It’s impossible to duplicate the same group of people in each environment.

We would first have to successfully clone the people. Then, with exactly identical groups, we could run a double-blind experiment. But that’s impossible.

The conclusion is that proving a methodology of this type as “correct” is impossible, just as it’s hard to disprove it. So it’s useless to try to claim they work or not. Of course they influence, and separately, each technique can be tested more carefully. But I don’t think it’s possible to scientifically claim that they work.

“But I tested methodology X on my project and it worked!”

Excellent, good for you. But for every project that succeeded with a certain methodology, you can find a case where it went wrong.

And the individuals and consultancies that sell these methodologies are clever: if the project worked, it was thanks to the methodology; if it went wrong, it’s because the people didn’t follow the methodology as they should have. The methodology is never at fault. It’s excellent rhetoric for building a sales argument.

And since people aren’t used to evaluating things rationally, and as I explained, most of the time they don’t reason at all, we’re left with lots of consultants making a lot of money without having to commit to results.

That matters because many people believe that, with the help of the fashionable methodology, simply applying it to a team of nothing but inexperienced people will make a good result appear, as if what mattered were the process and not the people. The opposite happens.

Most of the time these methodologies don’t describe the exact qualifications each member needs to have, because it’s impossible to list them all. At most they offer vague descriptions of “roles,” which generate more discussion than answers. A project with a chance of succeeding starts with at least some very experienced people, capable of guiding the rest of the team.

Anyway, my recommendation: study the so-called “methodologies,” because, as I said, some techniques there have value. But understand that none of them represents the truth.

And there’s no point in adding two half-methodologies together thinking it makes them “more complete.” Projects are executed by people, human behavior is hard to predict, and the real world is full of unpredictable variables, forming a complex system with a lot of randomness involved.

Start by accepting that the world is random and beyond your control. In complex situations it’s better to err sooner than later, because errors bring information that helps refine the next decisions.

Accept that predicting the future in a complex system is impossible, and adapt: some things will go as planned and others will need to change. Maybe some unforeseen feature will need to be built, and the deadline will have to stretch. Maybe some planned feature wasn’t so necessary after all and can be dropped. That’s how you deal with a complex and unpredictable environment: adapting.

Biology explains it: all living beings on the planet are the result of a constant sequence of trial and error. Nature isn’t “perfect,” in the sense that there is no creator, no intelligent design, no grand plan, no destiny. Every living being came from a very bad biological draft that, little by little, via natural selection, discarded what didn’t work and favored what did.

We humans carry biological leftovers, like the appendix. We are a product in constant evolution, and for plenty of things the current stage is “good enough.” In terms of a project, that’s exactly the point where it can be launched: when it’s “good enough,” still with rough edges to fix, but working overall.

Trial and error. Adaptation. Evolution. That’s how you manage anything.

If you want a more detailed view on this topic, I recommend starting with the following books:

• How We Decide by Jonah Lehrer
• Descartes’ Error by Antonio Damasio
• Predictably Irrational by Dan Ariely
• Fooled by Randomness by Nassim Taleb
• The Lucifer Effect by Philip Zimbardo

The text below is a translation of the excellent article Standards: excellence vs mediocrity, written by Jason Yip, a Thoughtworks consultant I had the pleasure of meeting in person last year.

Before the translated text, a short introduction. Plenty of concepts we treat as foundational in the Western world carry a radically different meaning in the Eastern world. One of the hardest to carry across is precisely “standard.”

In the Western world, a “standard” is a common denominator: static, rigid, hard to change, the status quo. In the Eastern world, “standard” means “the best.” If tomorrow another “best” shows up, that becomes the new standard. It isn’t something unreachable that we admire from below, knowing we’ll hardly get there, like a “record.”

Imagine a world where the “record” is the “standard.” I wrote about this in another article, Standards, Commodities, and Innovations, and it’s worth the read. Here’s the translation of Jason’s article:

A few years ago, I went on a Lean study tour in Japan. As expected, we did a plant visit at Toyota. Unexpectedly, this visit was hosted by the plant manager who also accompanied us on the standard guided tour. At one point, we were looking at a demonstration area of fundamental skills training – developing the basic skills necessary to be a productive member on the assembly line. Most of this was about hand-eye coordination and motor skills. For example, there’s an exercise where you take a thin rope and wrap it around a set of pegs following a known clockwise or counter-clockwise sequence. I’ve sketched an example of what this might look like:

I was able to do the exercise in 8 seconds. We then watched a video showing the plant champion doing it in 4 seconds. We asked the plant manager what the standard was, expecting the answer to be something like 5 - 6 seconds. His answer? 4 seconds, of course. If someone is able to do 5 - 6 seconds, they’d train them to achieve 4 but slower than that, and they’d probably find the candidate something else to do.

Our basic assumption was that “standard” is supposed to be something easy enough that everyone can do it. Sometimes that “standard” comes with an expectation of training but many times it ends up being about lowest common denominator in order to achieve consistency. Obviously, we can’t achieve consistency if our standards can only be achieved by a subset of people. The consequence of this mindset is that “standards” inevitably pull the organisation toward mediocrity.

The Toyota assumption was that the best should be the standard. There is still an expectation that people can do it (with training) and an expectation of consistency but it’s a standard of excellence rather than lowest common denominator. This creates a much higher expectation of performance and challenge. The consequence of this mindset is that “standards” inevitably pull the organisation toward excellence, especially as people’s capabilities continue to develop over time.

In one case, standards are an anchor; in the other case, standards are an engine.

Is this any different in our world of information technology and software development? I don’t think so. Think about the last standard you encountered (e.g., coding standard, design standard, etc.). Was it designed as an anchor or as an engine? And what would it be like if every standard was designed to be an engine?

Ron Jeffries is one of the 17 original signatories of the Agile Manifesto and one of the 3 founders of Extreme Programming. He publishes articles in the Pragmatic Programmers e-magazine, and the April 2013 issue ran “Estimation: The Best We Can Do.” I asked editor Michael Swaine for permission to translate this article into Portuguese, since it was a subject I meant to write about myself and Ron did a better job of it. I still have plenty to say about estimates, but for now I’ll go straight to the translation:

Two months ago in these pages, Ron Jeffries told us that estimates are evil. Now he’s back to tell us they’re a necessary evil and, done the right way, they aren’t even evil.

Yes, there are many abuses perpetrated by organizations around the notion of estimation. Many developers have suffered grievances from these abuses more than once in their careers.

Unfortunately, this leads to the common naïve view among would-be agile practitioners who are still early in learning: they want to abolish estimates completely. That’s probably not possible, and certainly not ideal. We have the ability to estimate how quickly we can do things, and companies can make better decisions if we share what we know.

Software development isn’t a perfect machine that spits out features quickly. I know it’s comfortable to think we have no responsibility for business concerns like money, time, or dates. But that isn’t true. What we build isn’t solely influenced by some Lord of the Products who decides what we’re going to do and takes on all the risks. Executives are the ones with the final decision, but many of the decisions are better made by the team — and not only in how to do things, but what to do. We have the responsibility to help guide the projects, using our creativity and our special knowledge.

The Agile Manifesto says, “the best architectures, requirements, and designs emerge from self-organizing teams,” and that’s what we meant (my emphasis). Agile developers usually have a good sense of how long things will take, and that’s a valuable component of selecting and refining requirements. Developers need to have the attitude of talking about possible costs of what they’re asked to do.

Concerns About Estimates

There really are several serious problems related to estimating a software job. In a previous article “Estimation is Evil”, I described some of them. Let’s review them:

• Pre-defined work backlogs reduce creativity and inhibit adjusting the project for success.
• Demanding delivery of “everything” on some fixed date is a trick that never works.
• Treating estimates as promises leads to conservative teams and frustrating results.
• Trying to improve the “quality” of the estimates is attention that could give better results if turned toward improving the quality of the product.
• And so on.

In that article, I argued that the result of this focus is “weak Agile.” In essence, teams focused on estimates are usually on the weak side of the balance between cost and value. Better teams focus more on value and less on cost. They still consider cost, because cost is an important component in delivering value. But value is their primary interest.

It’s true that many first-rate teams don’t estimate in the real sense of the term. They work on small stories, small enough that to get a sense of progress you can just count them. They work on the highest-value things first — using whatever notion of value they have. They adjust their vision frequently, and adjust what they’ll work on next according to what is happening. They work to produce a continuous flow of valuable software, and they deliver as frequently as possible, usually daily.

Since estimation is often associated with weak and dysfunctional teams, and since great teams seem to work without estimates, many Agile practitioners strongly resist any estimates of any kind. And it’s a good thing to build a team — and an organization — that is strong enough to work without estimates. But it’s an advanced way of working, never the way to start. It isn’t good to refuse to estimate, or even to feel resentful about it, in an organization where it’s necessary. On the contrary, we need to learn to do it right, in a way that influences the organization positively.

Budgeting New Projects

Now, let’s look at a legitimate need — helping the organization decide whether to proceed or not with a project.

When something goes wrong with our house or car, and we don’t have all the money in the world, we probably get estimates to fix those things. We probably consider alternatives: if the trunk lid is scratched, should we repaint the whole car to match the color perfectly, just spray the lid, try to polish the scratches, or just live with the scratches? We choose the option that seems to be the best use of our money.

Business people budgeting projects have the same problem of allocating funds. They need to consider how to deliver the greatest value possible within the budget. In a small organization trying to create wonderful new products, the concern is even greater. We need to start generating positive cash flow before the money runs out!

In situations like that, we really need to have estimates. We need to know whether it’s a wise decision to go ahead with the new project. Business people already have some estimates in mind: how many people will be interested? What percentage of prospects will become buyers? How much will buyers pay for the product?

But they need to decide: can we start bringing in enough money to stay alive before our money runs out?

The decision-maker knows that all these estimates are estimates, and uses the best judgment to value them. But there’s a key area they don’t know much about: how much of this product can be done, by what date, for how much money? That’s a technical question, and it needs to be answered by technical people.

Yes, of course, we don’t know either. We really don’t know how long it will take. But let’s face the facts — we have a better idea of how long things will take compared to the poor entrepreneur, and they need our help.

I believe we can give a better guess whether it will take a week, a month, a quarter, or a year before we start having a good idea of how quickly we can progress. And this guess, however broad, is a better guess than a business person can give.

We can’t just say “Should we try?” It seems we can say something like:

“Our team costs $10K per week. Let’s start working on the most important, riskiest, most informative parts of your product, and create a sense of how hard it will be and how long things will take. You’ll see us building what you asked for. You’ll see us working. You’ll be able to decide whether you want to keep investing or stop.”

It seems to make sense to say that — unless it’s your $10K coming out of your pocket every week. Then you have more questions. “Will I know after a week? Will I have to wait a whole month to know? Will I know by June? Jeez, guys, that’s about $100K from now until June. That’s a ton of money! Give me a break here! How much is this going to cost?”

We need to take hold of this kind of challenge. We should give the most solid answer we can, even when we don’t have an absolutely concrete answer.

We’re walking on sand. How do we get concrete?

The question is, “How long will it take, and how much money will it cost, to get to a product people can buy?”

We’d like to say, “Well, start spending some money with us, and if you get bored, stop,” but that doesn’t help at all. Can we change this idea a little? Let’s start with a pretty broad estimate just to get a sense of where we are.

We probably already have some notion of whether it will take a week, a month, or six months to do something usable. If so, let’s talk about it. Our investor already has a value in mind that they can spend. Let’s find out how much that is, or get a sense of it in the conversation.

Suppose we imagine that we might get a rudimentary but usable version of the product in six months, and the investor is willing to invest up to $1 million until we achieve positive cash flow. $1 million is roughly a year of our time, and if we take 9 months instead of 6, we’ll need to bring in $1 million in three months to reach positive. Let’s talk about that. How quickly does he expect revenue to grow with the first version? Suppose he thinks revenue will grow quickly, so if we take 9 months, we’re fine. That sounds risky to me, but it seems the project at least looks good enough to explore.

Remember how Agile works. We want business people to be fully engaged in decision-making about spending money based on what they see. So let’s start thinking about selecting features, not just dates.

Estimates That Won’t Make You Regret

“OK, intuitively, we think we can deliver an initial version in six months, and we think that if we take nine months, the project would be viable. It sounds good, but it still sounds risky. We’d hate to see you spend all this and get nowhere, and what we have is still rough guesses.”

“So let’s do this: let’s spend two weeks to produce a more concrete answer. Two weeks will cost you only $20K, which is much less than $1 million. In these two weeks, you’ll actively work with us on the most important aspects of the product. Some will be interface things, some will be driven by business or technical risks we can predict now. At the end of these two weeks, we decide whether we want to continue. Our job will be to show you, concretely, what we can build that convinces us of whether to proceed or not. Your job will be to guide us by describing what you want, and to work with us to identify risks and concerns.”

“If after these two weeks, things look bad, we’ll know it, and we’ll recommend you stop. If things look good by then, we’ll decide together what the next decision point will be. It could be a month from now, or three months from now. Frankly, we’re perfectly happy to do this every two weeks.”

“Yes, that’s right. Every two weeks we’ll talk with you about what to do next. Let’s do it and show you. If it’s good enough, we continue. If not, we stop.”

“That way, your risk will never be greater than another $20K. You can decide whether to use that money to get more information, to redirect your efforts, or to see whether you should spend that money some other way.”

“Can we work this way?”

If he says yes, let’s start. If he says no, we need to find out why, and even whether this project is for us.

This is definitely estimating, and it’s a kind of estimating that we can do in conjunction with our business people rather than in conflict with them.

We estimate, with some certainty, that we can produce useful information in two weeks. If we don’t think we can do that, we’d better suggest four or six or eight weeks.

More than that, we estimate that a viable product is probably possible in six or nine months. If we had no idea, or worse, if we honestly doubted it, then we can’t legitimately invite him to find out — at least not on the terms above. We owe it to him to say “Frankly, this looks like a two- or three-year project to us. We could be wrong: here’s what we need to learn to find out. Here’s what it would cost to find out, and our best guesses now are that the answer will be bad news.”

Sometimes we have a good idea of what we can do. Sometimes we know less. Either way, by estimating what we know, we can design experiments — investments — that improve our ability to decide what to do.

When we’re talking about spending other people’s money, I think we owe them to think and act this way.

This Is Estimation, Not Negotiation!

An estimate doesn’t have to be “This project will be finished on Tuesday, May 14, at 2:35 PM.” An estimate is supposed to be expressed with a range of possibilities. It would be perfectly OK to say something like, “We see no possibility of this being done before April. If everything goes perfectly, we can be ready by mid-May. Based on our experience, we’re thinking June, maybe July.”

Whether we say this or not, chances are we know it, or at least suspect it. And if that’s what we think, the project needs that information. However, if we put it that way, we’ll be pushed back. Someone will challenge us to prove, without a shadow of a doubt, that we can’t deliver by April 30, and if we prove that, they’ll say “How about May 5?” Bah, we hate it when they do that. So how can we deliver information the project needs without playing this game?

Move from Estimated Date to Estimated Velocity

Let’s use the Five-Card method from previous articles, not to estimate a date, but to break the whole project into sub-stories small enough that our team does, say, five of them in two weeks. Of course we can end up having only three done. We may have six, but we doubt it. Then we arrive at an estimate in terms of velocity:

“We broke the project into smaller stories. We believe that with stories of this size, we’ll be able to do between three and five every two weeks. And at the end of a two-week period, everything we do will be visible, tested, integrated, and working. So whether it’s five or six or two per week, we’ll know, and you’ll know.”

“If you can remove or defer some of these stories and still have a viable product, you’ll have the product sooner. If you can simplify, you can have the product sooner. If you add stories, or make things harder, you’ll delay the product to later.”

“It’s up to you. We can break things into this size, we can guess our velocity now as three items every two weeks, and you’ll know every two weeks what’s really happening. You can use that information to decide what to do, what to defer, in order to get the best product by whatever dates you choose.”

Help Management Learn About Velocity

Now management may still want to negotiate about this with us. If they try to add up the numbers to get dates, and then start adjusting the dates, don’t negotiate: turn the conversation back to the rate of progress. “The date is up to you. If you choose to do more, it’ll be later. If you choose to do less, it’ll be sooner. We think we’ll do between three to five items every two weeks. We may be wrong: if we are, you’ll know as soon as we do.”

Someone may try to push velocity. “Can’t you do seven every two weeks? Then you could finish on time.”

Remember how Agile works: developers work at a sustainable optimal pace; developers aren’t responsible for the dates; the business side owns the dates, and owns the responsibility of reaching those dates.

The developers’ job is to deliver at the best possible pace: “We think we can do three to five. If we happen to go faster, you’ll know. If we’re slower, you’ll know too. When we finish depends on that, but depends even more on how little you give us to do. What goes into those dates depends on you. You should plan on three to five items every two weeks, and check what’s actually happening.”

This can press more than once to go faster. But we have the facts at our disposal, as they are. “Based on our experience, we’ll have three to five done. It wouldn’t be wise to assume anything greater than that, because it probably won’t happen. If something happens that makes us go faster, we’ll all know it and you can select more items into the version.”

All of this, of course, has much more to do with communication, not estimation, and emphatically not negotiation. The estimation was trivial: we did it right at the beginning when we separated the cards. Now we’re communicating what we know, what we guess, what we believe.

We give management our best estimates and production rates, and we commit to them, improving as we evolve. We continue to make it clear that the business side controls the dates by choosing how much work they ask for.

Optimizing the Product

Whether you’re in favor of estimates or not, you should know that Agile is about building products incrementally, in good condition, from day one to the last day, and extracting value from it as early as possible. That requires great technical skills, but technical skills aren’t enough. The people paying for our work need to allocate their resources. They need to manage their money, and they need to coordinate our work with activities and stakeholders outside our team.

To do this efficiently, they need information about how long things will take, which translates into how much things will cost, and how long it will take for us to start getting our money back. They need the information we have to do this work well. The fact that our information is vague, flawed, and uncertain isn’t a reason not to expose it. We need to put our information in their hands, and do it in a way that gives them the best opportunity to use it as well as possible.

Refusing to estimate isn’t the right thing to do. Estimating is necessary, and effective communication of the estimate is necessary too.

The best thing to do is to develop small features at the best sustainable pace possible, and use the information generated to help business people decide what to ask for, and what to defer. To do this, we need to estimate. We break stories into small estimable sizes, and we estimate our rate of delivering them. This type of estimation isn’t evil at all: it’s incredibly useful and is what the best agile teams do.

Ron Jeffries has been developing software longer than any living person. He holds advanced degrees in mathematics and computer science, both earned before negative integers had been invented. His teams built operating systems, compilers, relational database systems, and several large applications. Ron’s software products generated more than half a billion dollars in revenue, and he wonders why he didn’t earn any of it.

Send feedback to the author or discuss the article on the magazine forum.

My problem is the image he uses in the article:

It’s a low-resolution image and looks really bad when I put it in a slide. The example is clear and nice to show, but this time I decided to prepare something more polished.

So I decided to use exactly the same idea as the previous image (read the original article to know what it’s about) and edited this video. See what you think :-)

At least I think it’s more fun :-)

I’m against anything turned into dogma. But if you insist on dogmatizing the Manifesto, never pull a single sentence out of it in isolation. If you’re going to take one principle literally, take all of them literally, otherwise it’s the same as dogmatizing the ten commandments and saying “I obey the ten commandments: I’ve committed adultery, but it doesn’t matter because I never killed anyone, never stole, and it wasn’t with the neighbor’s wife that I committed adultery.”

A few of the principles:

“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”

This one principle alone has fueled years of debate. Worth reading Our Highest Priority and Why Satisfy the Customer?.

I like to think like Eliyahu Goldratt, who says, in simplified terms, that the priority of any company is to make money. Now, calm down, anti-capitalists and on-duty populists :-) Nobody is talking about “making money at the expense of people, customers, and society” or anything like that.

The priority is to make money. Creating innovation, creating products, and generating jobs are means to get there, and we keep confusing the means with the goals.

“Delivering fast and continuously” is important to keep in mind. It isn’t always possible, for example, when you depend on suppliers or other external factors. But let me pull in other principles tied to this first one:

“Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”

Time is never fixed, because every professional knows you can’t predict the future. Delivering value quickly is a principle that guides the work, and nobody should treat it as a rigid rule. It means doing everything possible to deliver something that works, with quality, in the shortest viable timeframe.

For that, you try to remove impediments, improve processes, optimize procedures, and improve communication. It’s a reminder for every agilist whenever deliveries start taking longer than normal.

“Working software is the primary measure of progress.”

Same idea in different words: software that hasn’t been delivered is useless. An agilist should feel bad working for months on software nobody uses.

Every piece of software is different. Sometimes it takes a whole day to find a one-line solution, and sometimes you can write a hundred lines in an hour. It’s the old productivity dilemma the industry tried to solve with failed techniques like LOC (lines of code), Function Points, and other nonsense.

I imagine Martin Fowler was one of the people who suggested this principle, mostly because of his article Cannot Measure Productivity. Software is measured by value. Raw productivity says very little, and whoever defines the value to be reached in development is the client or the company.

If the delivered software brings no value, the problem is in the definition of that value. The software did what it was told. A tool, on its own, has no value.

That’s why I say LOC and function points are useless. I can have a team shipping ten thousand function points per sprint, and it means nothing if what was requested brings no value to the client.

This is one of a Product Owner’s jobs: making sure whatever enters the development queue will actually bring value, short or long term, depending on the future vision they have. A PO with no vision, no strategy, and no clear direction reaps exactly that: lots of software and little value. The blame there lands on the PO’s and the company’s lack of vision, never on the technical team.

And speaking of POs, we can’t leave out stakeholders in general, the people interested in the value the project will generate:

“Business people and developers must work together daily throughout the project.”

Personally, I’d hate bureaucrats around a development environment every day :-) Cynicism aside, this is another reminder that whoever cares about the value of a project is the one who should seek out the team and check if anything is missing. A stakeholder who never approaches the technical team shows they don’t care about the result.

And I’m not talking about a daily meeting. Five or ten minutes of conversation a day are enough, no ritual like a Daily Scrum required. An interested stakeholder gets up from the chair and walks over to the teams. An uninterested stakeholder stays seated, waiting for things to happen on their own.

Guess which of the two is more efficient? If the stakeholder, who should be the most interested, shows no interest in what’s being produced, why would anyone on the team bother?

Teams are a reflection of the company’s organization. If the upper layers are disorganized, indecisive, uncommunicative, and irresponsible, the teams will be the same. What are teams but sub-systems, copies of the larger complex adaptive system called “company”?

A company can hire a hundred Linus Torvalds, a hundred Guido van Rossums, a hundred John Resigs, and even then don’t expect them to come up with iPods, iPhones, and other products that sell millions all on their own. Apple works because, I speculate, a Steve Jobs gets out of his chair and gets personally involved in the day-to-day of the research and development teams, even as CEO.

Put this way, it sounds like one side is off the hook for the other’s responsibility. Of course not. The best way to work is collaboratively, but arriving at the best technological solutions, with technical quality and efficiency, is the technical team’s job. Stakeholders, in turn, answer for market studies, marketing research, and product vision.

One side can and should collaborate with the other, but not everyone can do everything equally. Stakeholders bring the vision and the technical team does what it can to fulfill it. Except the vision isn’t authoritarian, just as some technological decisions shouldn’t be either.

The criticism shows up when orders come from a single direction, with no discussion. The technical team can be doing everything right and still head the wrong way, and in the end the criticism comes back on them for the failure of a vision nobody ever discussed.

“The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”

To me, this is a corollary of the previous principle. Forget automated communication systems, information-flow systems, and all that nonsense. Everyone has seen 80s and 90s movies mocking “memos,” that technique for spreading information. To this day people still try something similar, only now they call it a spreadsheet, an email, a wiki, and so on.

What matters is sitting next to the person who’s going to deliver value to you, clearly stating the expectations, what changed, what stayed, and then getting out of the way. A stakeholder who can’t talk live with the teams shows, again, no interest in the result. And the message is clear: what’s being developed has no value, because if it did, they’d show interest.

As long as a stakeholder keeps the old-fashioned mentality of “these people who work for me are obligated to guess what I’m thinking and deliver what I want, when I want it,” they’ll never get any value, and they’ll keep asking themselves “are my teams so incompetent they can’t deliver anything?” The obvious conclusion would be the other one: “maybe if I just ’tell’ them what I’m thinking, they can deliver what I want.”

And there’s nothing worse than an absent stakeholder who only shows up when something goes wrong, to yell, threaten, and terrorize. Where was he during the whole process that led to this accident in the first place?

“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”

Every agile methodology has a “Retrospective” phase, one of the rituals created for teams that weren’t used to talking. No process is a silver bullet, and every process needs constant refinement. Ideally, each retrospective, at the end of each sprint, points at something to change. There’s no perfect scenario; what there is is continuous, uninterrupted improvement. If nothing changes after a retrospective, if nobody suggests adjustments at the end of the sprint, something is wrong.

The other principles speak for themselves:

“Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.”

Changes are welcome, as long as they bring competitive advantage to the customer. Random changes based on the mood of the people involved stay out.

“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”

Obvious again: either you trust your teams, or you don’t. If you don’t, fire them. If you do, stop micro-managing and demanding reports all the time. Simple as that. Anyone who doesn’t trust them and still thinks they can control everything through micro-management is a terrible manager, and should fire themselves first.

About the environment, remember Maslow’s Hierarchy of Needs. If the environment is cramped, hot, dirty, and disorganized, don’t expect motivated, proactive, organized teams. Nobody motivates anybody, but demotivating is very easy. Don’t demand the top of the pyramid while not even the base is satisfied.

“Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.”

It means all the principles above are being respected: there’s a good environment, stakeholders are involved, live and frequent communication happens, and priorities are clear. With that, you can have frequent deliveries at a constant pace and, better still, accelerating.

“Continuous attention to technical excellence and good design enhances agility.”

The first principle, in isolation, sometimes gives the false impression that value has to be delivered at any cost, sacrificing quality, maintainability, and good design. That’s false, and that’s exactly why I keep insisting on considering all the principles together.

Valuable software should be delivered as fast as possible, but never at the cost of wrecking future maintainability, right when the Return on Investment starts happening. Each case is its own case, and that’s why stakeholders and teams need to talk often and decide based on cost-benefit.

“Simplicity–the art of maximizing the amount of work not done–is essential.”

Together with the previous principle, this means not getting stuck on “maybe they’ll need this in the future” and cranking out “bloat” software, fat, heavy, and complex to try to make it “future-proof”. That’s impossible and only leads to software that takes too long to deliver and shows up with dubious value.

Again, it’s a matter of communication, negotiation, and cost-benefit, starting from the value the stakeholder said they need.

“The best architectures, requirements, and designs emerge from self-organizing teams.”

This topic I’ve explained dozens of times, in all sorts of ways, but I recommend reading my article Agility, Chaos, Self-Organization.

Anyone who doesn’t understand Emergence and Self-Organization doesn’t understand Agility. That brings me back to my article You Don’t Understand Anything About Scrum, which I also recommend reading now.

As you can see, the world of Agility rests on important principles. None of them holds up in isolation: either you consider them all together, or you’ll have none of them.

But be careful: a lot of people confuse Agility, Self-Organization, and Participatory Management with the Search for Consensus. Nothing could be further from that. I’m taking longer than I’d like to finish the book Systems Thinking, but here’s an excerpt I like:

“Finally, fear of rejection and a strong tendency toward conformity among members of a social system and others are obstacles to social changes. An example is the experiment in a city with a dry law (which doesn’t allow the sale of alcohol) whose constituents were to vote on the ban against alcohol. A pre-voting poll indicated that 75% of voters were in favor of abolishing the ban. However, each of the voters thought that the majority preferred the dry law. When the results were tabulated, 60% of voters voted to keep the dry law. Not surprisingly, after the poll was published, the next election on the subject produced a 65% majority in favor of abolishing the ban.”

Democracy based on the Tyranny of the Majority is useless. This subject is much more complicated than just having people vote on the options, and it can’t be treated lightly. Political scientists, philosophers, and various researchers have been studying it for a long time, and I guarantee there’s vast literature on it.

And this whole subject is more complex than just reading the twelve principles. An engineer, a doctor, or a lawyer study for years and are still far from being masters of their fields. A manager, on the other hand, studies almost nothing, and so decides based more on folklore than anything else. It’s terrible: in these cases, when they’re right it’s by luck, and when they’re wrong it’s just what you’d expect.

On top of that, I’m anxious about my own research, which always seems to have pieces missing. Since there’s almost no one to talk to about it, the frustration grows. So, at the start of the article, I was already out of patience and began abusing the first and third person, in a near 1-to-1 dialogue. That wasn’t helping the main point, so I decided to trim the most aggressive parts. Let’s see if it’s more “digestible” now :-)

Last year I put together a list of books I read in my search for the essence of true management. Instead of going through the more trivial authors, like Peter Drucker, I went after the principles of Agility.

There are a few categories of managers: the “agile” ones, in the broadest sense of the word, and the “traditional” ones. The second group, prone to not wanting to change or learn anything, is sad, period. The first is on the right path, but there’s still a long way to go. The discussions I see today in agile communities also fall into the inertia of “good sense” and “common sense,” and end up shallow.

An agile manager should want to dedicate a lot more. Scrum itself is an example. As I said in the article You Don’t Understand Anything About Scrum: Jeff Sutherland and Ken Schwaber researched the relevant subjects long ago and gave the tips, though few follow them.

If you really want to understand Scrum, this literature helps a lot. To start, there are the Scrum Papers, which give you the introduction.

If you read them, you’ve already bumped into terms like “Emergence,” “Edge of Chaos,” “Complex Adaptive Systems,” and “Self-Organization.” And if you passed through those terms and left it at that, think again: what’s the point of reading without understanding?

I don’t know everything yet either, but I’m committed to understanding this properly. Without the principles, all that’s left is to “follow a recipe,” and that’s very little.

Whoever looks for rules or recipes is trying to shed their own responsibility. If the method, the procedure, or the process “fails” or doesn’t deliver, you can always hide behind “but I followed it to the letter, it’s not my fault.” That misses the point. What result are you after: following rules or reaching results?

I say this with “aggressiveness,” but some reluctance, because I used to think this way myself. Anyway, after some time reading academic articles, right now I’m working through these books:

• Systems Thinking, Second Edition: Managing Chaos and Complexity: A Platform for Designing Business Architecture. Once you fall into Complexity, you leave classical, reductionist, deterministic thinking behind and start needing the holistic thinking of Systems, which has itself gone through several paradigms. Highly recommended: it opens by describing the different management models and their evolution with precision.

• The Essence of Chaos. Complex Adaptive Systems fall under the broader subject of “Complexity,” and that leads straight to “Chaos.” I’d rather read something by Ilya Prigogine, but since I’m on a Kindle, the only things I found available were titles like this one, by none other than Edward Lorenz, one of the “fathers” of Chaos.

• Surfing the Edge of Chaos: The Laws of Nature and the New Laws of Business. Systems and Complexity also require understanding the Edge of Chaos.

• The Greatest Show on Earth: The Evidence for Evolution. One of the clearest examples of Complex Adaptive Systems is the nature around us. You need to understand Darwin’s Theory of Evolution. Most people “think” they understand it, but watch out: much of it is folklore, and plenty of people still doubt it. If you have doubts about Evolution, don’t even start.

More than that? From experience, every time someone uses words like “chaos” or “balance,” I notice they’re working from the casual definition, not the technical one. Many think “chaos” is just mess, something completely unwanted.

Worse still: they believe the “secret” is to reach balance and stay there, and they confuse the goal with seeking balance. That’s not quite it.

Here’s what the introduction of Surfing the Edge of Chaos says:

Equilibrium is a precursor to death. When a living system is in a state of equilibrium, it is less responsive to changes occurring around it. This places it at maximum risk. In the face of threat, or when galvanized by a compelling opportunity, living things move toward the edge of chaos. This condition evokes higher levels of mutation and experimentation, and fresh new solutions are more likely to be found.

When this excitation takes place, the components of living systems self-organize and new forms and repertoires emerge from the turmoil.

Living systems cannot be directed along a linear path. Unforeseen consequences are inevitable. The challenge is to disturb them in a way that approximates the desired outcome.

This is a definition of the biological evolution of living beings. But if you’ve read at least the basics of Scrum and Agile, it’s also a way to describe the Agile process. Reread the Principles of the Agile Manifesto and you’ll find similar texts.

As for Complex Adaptive Systems:

A complex adaptive system is formally defined as a system of independent agents that can act in parallel, develop “models” as to how things work in their environment, and, most importantly, refine those models through learning and adaptation. The human immune system is a complex adaptive system. So is a rain forest, an ant colony, and a business.

There’s vast literature. Search Amazon for terms like “Complexity,” “Edge of Chaos,” and “Systems Thinking,” and you’ll see that for decades researchers from the most varied fields and industries have been refining this body of knowledge. Techniques like the Toyota Production System/Lean, Six Sigma, Scrum, and Theory of Constraints all derive from this school of thought.

From my previous list, I reinforce the need for readings like:

• The Machine that Changed the World: The Story of Lean Production. Understand how the Lean process came about at Toyota.

• The Goal. Process engineering and cost control. Without understanding TOC, it’s futile.

• The End of Management, and the Rise of Organizational Democracy. A summary of all these subjects and, maybe, a provocative introduction to the rest.

More concerning: techniques like Lean, TOC, Six Sigma, and derivatives like Kanban are terms plenty of people use the wrong way nowadays. They were all born and refined in Production Engineering, always talking about the “production line” and the “manufacturing” of material goods, “physical” things.

In Agile, we’re talking about “Software,” something abstract and without parallel in the physical world, and therefore without the same references.

It’s impossible to migrate any Production Engineering technique verbatim. It doesn’t work that simply. A Scrum Sprint is a conscious adaptation by Sutherland and Schwaber, who knew this limitation and attempted an approximation in the abstract world.

But this isn’t obvious or trivial to grasp. I haven’t fully grasped it myself yet, and I believe most people in the agile world never stopped to study this difference.

Be very careful: “Management” is much more than “experience” or “good sense.” Just as a good programmer needs to read and research a lot to sharpen their technique, a manager needs to as well. And I’m saying this to myself.

As I said in a previous article, agile methodologies share the same foundation. To understand TPS it’s worth going back to the original literature, and one of those sources is the book A Study of the Toyota Production System from an Industrial Engineering Viewpoint, by Shigeo Shingo, published in 1996. In the Preface he says:

Many believe that when implementing a new system, only “know-how” is needed. However, if you want to succeed, you must understand, as well, “know-why”

With the know-how, you can operate the system, but you won’t know what to do if you encounter problems under conditions different from the usual. With the know-why, or “knowing why,” you understand why you have to do what you’re doing, and thus face changing situations.

Specifically about Kanban he says:

The biggest problem I encountered while studying TPS from an Industrial Engineering viewpoint is the fact that it’s often considered synonymous with the Kanban system. Mr. Ohno writes:

• TPS is a production system
• The Kanban method is a technique for its implementation

Many publications are confused on this issue and offer an explanation of the system, claiming that Kanban is the essence of TPS. Once again: TPS is a production system, and the kanban method is merely a means of controlling the system.

Superficial analyses of TPS give special attention to the kanban method because of its unique characteristics. Consequently, many people conclude that TPS is equivalent to the Kanban method.

A Kanban method should only be adopted after the production system itself has been rationalized. That’s the reason this book repeatedly insists on the fact that TPS and the Toyota method are separate entities.

I should add that 90% of Toyota’s excellent management performance was attributed to TPS itself, and only 10% to the Kanban method — a clear demonstration of the greater importance of TPS.

Worth repeating: Shigeo wrote this in 1996. It’s striking that, more than a decade later, we’re still making the same interpretation mistakes.

Just like the Agile Manifesto, the Toyota System has a set of 14 principles, known in the West as The Toyota Way. And the same way it works in Agile, sticking post-its on the wall and running Sprints doesn’t make anything Agile. Following the Toyota method takes a lot more than using Kanban.

To understand Toyota it’s mandatory to understand the Toyota Way, and one of the best books to start with is The Toyota Way by Jeffrey Liker. If you intend to take this seriously, you also need to understand Toyota’s management revolution, narrated in the classic The Machine That Changed the World by James Womack.

If you’re still not convinced, at the conclusion of the chapter on Kanban, Shigeo Shingo himself says:

Kanban systems can only be applied in factories with repetitive production. (…)

Kanban systems are not applicable in companies with non-repetitive project-based production, where orders are infrequent and unpredictable.

The type of production most likely to benefit from Kanban is that which uses common material transformation processes.

A tip: software development is a non-repetitive task. Even so, TPS principles stay very applicable when the know-why is clear.

The Toyota method is more generically known as Lean Manufacturing. The best work adapting Lean to the software world is the book Lean Software Development, by Tom and Mary Poppendieck. Before talking loosely about Kanban, it’s mandatory to read those works; otherwise it’ll just be one more tool doomed to fail, and we’ll end up with a wave of Kanban-butts.

Everything that comes easy goes easy. It “seems” easy to implement Agile. It “seems” easy to implement Kanban. There’s no free lunch. Take things superficially and don’t expect anything beyond mediocre results. That’s how it works.

My research style is quite Web-like, I’d say :-) For example, by some cosmic accident, only this week did I decide to read about the origin of Scrum, the Agile methodology that ended up becoming the most well-known today. I was very surprised by what I found.

I always defend that people need to constantly, obsessively research and never accept anything without argument. We should all already know that there are no silver bullets. Simply quoting someone to justify something is a fallacy I don’t accept, as I already said in my article Obedience to Authority.

That is, even to defend agile methodologies I need a better argument than saying “Kent Beck is awesome, Kent Beck invented XP, so I should use XP.” Or something like “Toyota is awesome, Toyota uses Lean, so I should use Lean.” That’s not an argument.

Fact: many companies today say they use Scrum, many people say they understand Scrum, but I’d bet a lot of money that the overwhelming majority don’t know more than Scrum-butt, or what we’d call, in a very Brazilian way, “half-baked Scrum.” Better than nothing, yes, but quite poor compared to the level of hyperproductivity that Scrum promises: 400% or more.

David Hume said:

A wise man, therefore, proportions his belief to the evidence.

Laplace’s Principle (Pierre Simon Laplace) says:

The weight of evidence for an extraordinary claim must be proportional to its strangeness.

Carl Sagan popularized:

Extraordinary claims require extraordinary evidence.

Given that I think this way, I need much more. I can use something based on my hunch that it works, but it can’t turn into a comfort zone. This led me into reverse engineering and a research path that has already lasted more than a year: dozens of books, dozens of technical papers, multidisciplinary research spanning the history of management, production engineering, psychology, sociology, and physics.

It’s a search for the roots of what should make a real “agile organization.” For me, agile methodologies are the current stage in the evolution of project management, and still far from the last one. Implementing an agile methodology is a means to get somewhere bigger, never the end goal.

In reality, the last few months have only been the most constant stretch. Worth remembering that I certified as a PMP in 2005, and back then I was already studying PMBOK and OPM3. Two years before, my research line was RUP, SW-CMM (before CMMi), and Indian software factories. My first contact with agile methodologies was in 2003, with an introduction to Extreme Programming, and in the 90s I followed the evolution of UML and RUP: Coad & Yourdon, Rumbaugh, Booch, Jacobson.

This whole introduction is a mea culpa about recent talks I’ve given, especially “Beyond Chaos: Random Thoughts on Agility.” It was the most recent and most daring version of a series of talks I gave throughout 2009 in several cities. It became clear that many people didn’t have the background to follow all these concepts.

In that talk I dared to mention concepts like:

• Complexity Theory and Nonlinearity
• Chaos Theory (specifically the “Edge of Chaos”)
• Entropy Systems Theory
• Emergence and Self-Organization
• Complex Adaptive Systems
• Evolution and Punctuated Equilibrium
• Game Theory

I really did try to explain each of these topics, but I found out that talks of less than 1 hour are terrible for explaining such dense concepts at the same time. I’ll still find a better way.

Each of these topics represents an entire branch of scientific research. And I swear I’m not talking about pseudo-science, this is serious research. In fact, these branches are precisely the “answer” to the question: “how do agile methodologies, if properly implemented, lead teams to a stage of sustainable hyperproductivity?”

The first time I mentioned “Self-Organization” at the beginning of 2009, I heard many people literally “making fun” of what I said. Some thought I had “invented” this “strange” concept. Unfortunately I’m not that smart :-) By chance, I reached the same conclusions as the creators of Scrum. Watch Jeff Sutherland, co-creator of Scrum, himself listing some of these concepts:

This excerpt is from the talk Self-Organization: The Secret Sauce for Improving Your Scrum Team, which he recently presented at Google Tech Talk.

I confess it was very strange to hear that reaction, since I’m talking about people who, in theory, should be agilists, given that they claim to know how to implement Scrum. That’s why I published the article The Agile Manifesto, or How to Become Google, where I started to interpret what everyone should have already memorized: the Agile Manifesto:

“The best architectures, requirements, and designs emerge from self-organizing teams.”

Yes, this is one of the 12 principles of the Manifesto. But if you don’t understand the scientific concept of “emergence” and “self-organization,” then you don’t understand the Manifesto and will have many problems trying to be Agile.

You know what else? Ken Schwaber was co-creator of Scrum in the mid-90s, together with Jeff Sutherland. He wrote a formal paper presented at OOPSLA-95. I can’t understand how anyone can say they “know Scrum” and have never read this paper. In it, Schwaber says:

“The closer the development team operates to the edge of chaos, while still maintaining order, the more competitive and useful the resulting system will be.”

Sutherland and Schwaber were some of the first Westerners to understand and publish about the enormous difference between Defined (predictable) and Empirical (unpredictable) processes. The Japanese had already gotten there with Deming years earlier.

Waterfall, RUP, and CMM are defined processes. Scrum, XP, and other Agile methodologies are mixed, somewhere between defined and empirical. This is the first big difference that makes the two groups incomparable, and Schwaber explains it very well in Controlling Chaos: Living on the Edge, presented at OOPSLA-96.

Even today I hear someone wanting to start a discussion like “Scrum vs. XP,” “Scrum vs. Lean,” “Adding Scrum to Kanban,” or “Adding Scrum to CMMi.” It’s hard not to get frustrated with the futility of these conversations, where people only argue about the top of the iceberg and never come close to touching the base of the principles that precede each of these topics.

In the words of Scrum co-creator Jeff Sutherland, from November 2007:

After thinking about the early days of Scrum I think the root of both Scrum and Lean is complex adaptive systems theory. When we created Scrum we did not talk about Lean, we talked about complex adaptive systems. I think Scrum and Lean are complementary implementations of ways to deal with physical reality where things are often not linear, not simple, and not predictable.

The first Scrum implemented all the practices we know as Scrum today plus all of what later became known as XP practices plus some secret sauce that creates hyperproductive teams.

(…) But most important, we captured the “punctuated equilibrium” effect, something I have yet to see another Scrum team do. It is one of the secrets of hyperproductive teams. Before a developer chooses a task in the Scrum meeting, the entire team discusses with the developer whether that task will be the fastest path to an observable end-use feature. This was a sophisticated implementation of “self-organization” and “test first” development.

What’s funny is that, as I said before, I only read this account of Scrum’s history this week. I arrived at the same topics of Complex Adaptive Systems, Punctuated Equilibrium, and the others I listed above through different paths.

I already knew the novelties of scale-free network theory, which gave rise to my 2008 talk “Killing the Average.” And, reading and rereading the Agile Manifesto, I came across the word self-organization. The sum of these two things led to dozens of readings that culminated in Complex Adaptive Systems. It’s a research path that seems somewhat inevitable, and it seems Sutherland and Schwaber reached the same conclusion more than 15 years ago.

And more than that, Sutherland also says the following about the origin of Scrum:

The first Scrum team was created directly from a paper which is required reading for any Scrum practitioner. It explains how to set up self-organizing teams and clearly outlines management’s role in the process.

Takeuchi and Nonaka. The New New Product Development Game. Harvard Business Review, 1986

A longer paper from a book which is out of print goes into more depth. Some may find this easier reading.

Ken-ichi Imai, Ikujiro Nonaka, and Hirotaka Takeuchi. Managing the New Product Development Process: How Japanese Companies Learn and Unlearn.

Hirotaka Takeuchi and Ikujiro Nonaka are Japanese professors also known for their research in Knowledge Management. The Harvard Business paper Sutherland mentions costs a mere USD 6.50. I just bought it, and it’s telling that the “Scrum” analogy from “Rugby” is already explained and named right there. That is, the name “Scrum” comes from Takeuchi and Nonaka.

More than that, they already explain in that paper what the concept of Self-Organization is and how it works. They also explain sequential style (Type A) and overlapping development phases (Types B and C). Sutherland usually talks about hyperproductive teams as Type C Scrum.

What Takeuchi and Nonaka explain in detail in these papers is the following:

From interviews with organization members from the CEO to young engineers, we learned that leading companies show six characteristics in managing their new product development processes:

1. Built-in instability
2. Self-organizing project teams
3. Overlapping development phases
4. “Multilearning”
5. Subtle control
6. Organizational transfer of learning

Once again, each of these items is the subject of an entire branch of research, so don’t try to extract any meaning from just these few words. The goal here is only to give a “taste.” Buy the paper and read more about the two’s work, especially about Knowledge Management, which is a bit different from what most companies understand by knowledge (for a change).

Another thing some may not know:

“In the same year, Sutherland provided support for development of eXtreme Programming by giving Kent Beck all background information on the creation of Scrum and the results of two years of product development with the Scrum process from 1993-95. XP engineering practices evolved along with Scrum and the two leading Agile development processes work well together. Scrum and XP are the most widely used Agile processes worldwide and their creators are signatories of the Agile Manifesto.”

This is in the Scrum Papers, which obviously every agilist should read. The point is: Scrum is founded on the Japanese principles that gave rise to Lean, while also serving as inspiration for the creation of XP. There is no conflict between Lean, Scrum, and XP. They have similar roots! That’s why it’s important to study principles more than just practices.

Be careful: I’m not citing all of this as a way of “proving” my point. I’m citing studies that happened to coincide with what I started doing on my own. It’s all research material that shouldn’t be used as dogma, but as a reference for research.

To close, I like to mention Mary Poppendieck. Among the agilists, she and Tom decided to take the most purist line and wrote Lean Software Development. With this explanation, it should be clear that what Mary/Tom and Sutherland/Schwaber say is complementary and totally compatible, each with their own experience background. And from Mary, I think this article on ‘Blame’ is important to read:

The purpose of what became today’s organization chart was clear: The assignment of responsibility would enable “prompt detection of derelictions of duty… and point out the delinquent.”

(…) There is something called standard work, but standards should be changed constantly. Instead, if you think of the standard as the best you can do, it’s all over. The standard work is only a baseline for doing further kaizen. It is kai-aku [change for the worse] if things get worse than now, and it is kaizen [change for the better] if things get better than now. Standards are set arbitrarily by humans, so how can they not change?

That last paragraph is, again, for those who still think silver bullets exist, that there’s some body of unshakable practices (CMMI, ITIL, RUP, etc.) where simply “following the recipe to the letter” will magically make everything work. After all, they’re “industry standards.”

Bullshit! Do your homework: keep researching!

PS: Read all my articles from the Off-Topic section for more related topics.