Last week I posted about my distrobox-gaming setup, with more than ten retro racing games running on Linux, from Gran Turismo 1 on DuckStation to Forza Motorsport 4 on Xenia. At the end of that piece, I described Driveclub on shadPS4 as the “final boss”: the game boots, the menu loads, the race starts, the controller responds.

Except I mentioned two problems in passing and treated them as “accepted limitations”:

1. During daytime, the image is a bit darker than on a reference PS4. Not alarming, but noticeable.
2. At night, the race start in Canada or Munnar went pitch black for 10 to 30 seconds before the game “recovered on its own” around the 1:30 mark. During those seconds, only the HUD was visible. Track, car, headlights — all absolute black.

In the previous article I pushed the explanation to “it’s just not fixed in the emulator yet, I’ll live with it”. But inside I couldn’t make myself play the game in a visually unplayable state. I wanted to play DriveClub at its best, and I didn’t want to sit around waiting for the project to fix it for me. I decided to take matters into my own hands. Thirty seconds of absolute black isn’t “a bit darker”. It’s a serious bug, the kind that makes a game unplayable in practice.

This article is what happened between Sunday night and Thursday morning.

The blackout, live

This video is the behavior I wanted to fix. Race start in India (Munnar), night race. You see the track, headlights turn on, the first few seconds run. Then it darkens progressively. Within about 15 seconds everything is black. Stays that way for another 60-90 seconds. Then the game “recalibrates” on its own and the track reappears.

If you’re a player, you don’t care why. You just want to drive.

Why shadPS4 is hard to debug

shadPS4 still has no stable release. The project is under active development. The main branch changes several times a week, configs migrate from TOML to JSON without notice, settings hide behind “Advanced” so users with incompatible hardware don’t file issues, open PRs compete over different approaches to the same problem.

Anyone trying to configure DriveClub today will find:

• September/2025 Reddit guides using readbacks = true (it was a boolean back then) saying “for DriveClub always enable readbacks”.
• November/2025 guides saying “disable readbacks, it kills performance” (because in 2025-07 readbacks became an enum Disabled | Relaxed | Precise with different performance profiles, and Bloodborne started hanging with it on).
• January/2026 YouTube videos using some custom fork nobody documented.
• A thread on shadps4.net with three different, mutually exclusive explanations for the blackout.
• My own notes from last week in distrobox-gaming/docs/driveclub-shadps4.md saying readbacks_mode: 0. The exact opposite of what I found this week.

So the information exists, but it’s scattered, dated, contradictory, and most of it depends on version details that changed between the original post and today. Reproducing a setup from a YouTube video is like trying to solve a Rubik’s cube blindfolded.

Spoiler: the solution is one integer

After 31 phases of investigation, 44 commits on my shadPS4 fork, 15,668 lines of instrumental code added, and three days of nearly uninterrupted debugging, the answer is one line in the per-game config:

{
  "GPU": {
    "readbacks_mode": 2
  }
}

readbacks_mode: 2 is the Precise mode. Explaining what it does requires understanding how DriveClub implements auto-exposure, and that’s where the journey starts. To know WHY this toggle fixes things, you have to understand the whole chain.

The short version: DriveClub implements auto-exposure as a GPU→CPU feedback loop.

1. The scene renders to an HDR target.
2. A compute shader calculates a luminance histogram into an SSBO.
3. The CPU reads that SSBO on the next frame and derives a target exposure.
4. The CPU writes that exposure back into a 1936-byte lighting UBO (at slots [38] [48] [50]).
5. Fragment shaders read that UBO and multiply scene luminance by it.

Without readbacks_mode: Precise, step 3 reads stale zeros. The memory page where the GPU wrote the histogram is never synchronized to the CPU side. The CPU exposure integrator concludes “scene is dark, open the aperture all the way” and ramps its output value monotonically: 2.59 → 7.84 → 24 → 90 → 179 → 255. Within 60-90 seconds the scale saturates so high that everything clips to zero. That’s the pitch black.

With readbacks_mode: Precise on, the emulator marks the pages the GPU just wrote as kernel-protected (via mprotect on Linux). When the CPU tries to read one of those pages for the first time, a page fault fires. The emulator intercepts it, issues a vkCmdCopyBuffer download from the GPU buffer to the host staging area, waits on the scheduler, and the CPU finally sees fresh data. The real histogram enters the integrator, exposure converges normally, and the race start opens bright as it should.

Why isn’t it the default? Three documented reasons. First, per-page mprotect + per-fault GPU stall + compute dispatch + scheduler wait has a real cost: on NVIDIA’s warm path it’s tolerable, on AMD it tanks below 12 FPS, with an open issue about it. Second, Bloodborne hangs on loading screens with Precise enabled. Third, the maintainer tucked the option behind “Advanced” in the UI precisely to keep users with sensitive hardware from enabling it by mistake. The mode has existed since v0.15.0 (September/2025), but nobody had publicly connected Precise + DriveClub + auto-exposure feedback loop. The compat tracker already admits that DriveClub “requires readbacks enabled to function properly”, but doesn’t specify the mode.

In my setup the performance cost is irrelevant. I have an RTX 5090 on my desktop, plenty of hardware to swallow the mprotect, the page fault, the copy and the stall per frame. The call is trivial: I flip Precise on, pay the price, the game runs. On more modest hardware or on AMD the conversation is different, which is exactly why this setting isn’t default and hides behind “Advanced”.

One thing I want to make clear: this whole explanation above (Precise vs Relaxed vs Disabled, GPU→CPU feedback loop, per-page mprotect syscall) is not knowledge I had on Sunday night. I didn’t know what a buffer readback was, didn’t know DriveClub implemented auto-exposure as a GPU→CPU feedback loop, didn’t know readbacks_mode had become an enum in 2025 with three values. To arrive at the right answer, I had to learn every piece from scratch. I couldn’t guess. Had I tried to “enable readbacks” last week without understanding the chain, I’d have picked the wrong value (Relaxed isn’t enough for this feedback loop, as I’ll explain below) and given up thinking “it doesn’t work”.

Two more things need to be in place for this to work, both covered in the previous article:

• v1.28 patch applied on top of the v1.00 base install (without it, content stays locked with “not yet released”).
• 60fps XML patch disabled (it raises the render rate but the logic tickrate is fixed; with it on, the game runs in slow motion).

The result

Same race in Canada, night, with readbacks_mode: 2 on. Race start opens bright, auto-exposure converges immediately, the night transition unfolds naturally as the game’s TOD (time of day) advances. No blackout, no “recalibration” at 1:30, no 30-second window of absolute black. This is DriveClub running like it does on PS4.

Now the interesting question is not “which setting”. It’s: how did we get here?

How does a junior learn today?

There’s a question I hear often: “with AI doing everything, how does a junior learn?”

The premise is wrong. AI doesn’t do everything. AI does what you ask it to. It doesn’t discover on its own. It has no domain initiative. It’s not capable of telling you “hey, this looks like a broken GPU→CPU readback feedback loop” unless you already know there’s a thing called a readback and that broken feedback loops are a class of bug.

Worse: if you go to an AI agent today and ask “fix this black screen in DriveClub on shadPS4”, it will give you a list of possible answers based on old forum posts, with a high probability of steering you wrong. It will suggest the 60fps XML patch (wrong — causes slow motion). It will suggest a tonemap override (wrong — the game already writes correct SDR). It will suggest a vblank_frequency tweak (useless for the real problem). It will suggest manual gamma (treats the symptom, not the cause).

The way to discover is to dive into the chain, phase by phase, ruling out wrong hypotheses until the terrain is familiar enough that you recognize the right answer when it appears.

That’s where a lot of people misread me. “Akita is already a senior, he has 25+ years of experience, of course he knows.” False. A senior is someone who has been a junior in every topic they master. And they keep becoming a junior every time they pick up a new domain. I have 25 years of web programming, distributed systems, Ruby, Erlang, Go. But I’ve never programmed for PS4. I’d never looked at the shadPS4 codebase. I didn’t know PS4’s graphics architecture. On Sunday I was an absolute junior in this domain.

But junior doesn’t mean totally green. A while back I had explored on my YouTube channel the low-level architecture of older consoles (from the NES’s 6502 onward) and how emulators work internally in general. The two videos below helped build the basic vocabulary for how a game console works under the hood, and made sure I wouldn’t trip on concepts like fetch-decode-execute, opcode, interrupt, or HLE vs LLE.

I had also made an older video on the evolution of CPUs, GPUs, DirectX, and Vulkan, so at least the vocabulary of “modern graphics pipeline” and “shading language” wasn’t alien.

Knowing that VkCommandBuffer exists is very different from knowing where in shadPS4 a readback vkCmdCopyBuffer fires. But it gives you a base. That kind of old curiosity paid interest now.

The rotation: Claude Code and Codex

AI (in my case Claude Code) didn’t “know” more than I did. It has the same dated forum text you’d find on Reddit. What it did well was aggregate: read shadPS4 source code in parallel with me, index comments, cross-reference between phase docs, compile, run probes, parse 61 MB logs, compare decompiled binary against UBO diffs, shine light on corners of the system faster than I could alone.

But the decision to “keep investigating” was mine. The perseverance of “don’t accept a mitigation, want the root cause” was mine. And when the AI suggested “honestly, we should just accept the current result and document it” (and it suggested this several times over the four days), I was the one who had to say “no, we keep going”.

The central point about the rotation: no single LLM managed to close this problem alone.

Over the four days, Claude Code repeatedly got stuck looping on the same hypotheses with no new direction. When that happened, I’d switch to Codex for a few hours to bring in outside ideas, different probes, a re-read of the codebase from a fresh angle. Then back to Claude Code to integrate what Codex had surfaced. And so on, alternating.

The final resolution happened on Claude Code, but the journey switched models several times. Each LLM had its biases and blind spots. Left alone, each one would have given up sooner.

What broke the impasse was me continuing to bring in new ideas to try, forcing the model to reassess, switching models whenever one started to repeat itself.

At two points things went past “just repeating hypotheses”. Codex literally had an agentic-loop glitch: it started repeating the exact same “solution” it had just tried, over and over, in sequence, without noticing what it was doing. It was the first time I’d ever seen that happen in any AI agent. I had to kill the session and start a fresh one from zero to break the cycle.

That says something about context size and session duration. The complexity and pace of this investigation were enough to bring both LLMs to their knees, each in their own way.

The whole journey below is documented in 33 phase docs on the gamma-debug branch of my shadPS4 fork. Each section here links to the original doc for anyone who wants the complete detail. Here I summarize the essentials: the hypothesis we had, what we tried, what new concept we had to learn, and why the phase didn’t close the case.

Session zero: baseline and reproducibility

Every serious investigation starts before Phase 1. The first thing, for any project, is to make sure the baseline works as expected and that the situation you want to research is reproducible every time. If you can’t reproduce the bug on demand, you’re chasing ghosts. Every probe after that will be inconclusive because the bug “appeared and vanished” without control. And if the baseline is broken in some way you haven’t noticed, every experiment measures the wrong problem. Without those two guarantees, you can’t make progress.

In my case, on Sunday night I confirmed:

• The game boots all the way to the race screen without crashing, using the distrobox-gaming config.
• DriveClubFS extracts v1.28 cleanly, 8018 files, ~47 GB.
• MSAA depth resolve applied on the fork; night tracks render shapes instead of uniform absolute black (covered in the previous article).
• 8BitDo controller detected.
• Night race in India 19:30 reproduces the blackout on 100% of attempts, with the same temporal pattern: progressive fade in the first ~15s, pitch black between ~20s and ~90s, “recalibration” at 1:30.

That baseline is the only solid ground that lets me compare each experiment reliably. Without it, a probe that “worked” might just be a different cache state, not the real fix. With it, every change has a before and an after that are measurable.

Beyond that, zero specific knowledge of:

• PS4 PKG format (PFS, param.sfo, disc_info, keystone).
• shadPS4 codebase (src/core, src/video_core, shader_recompiler, buffer_cache).
• PS4 graphics pipeline (GCN ISA, forward+ lighting, MSAA depth resolve).
• Vulkan beyond the surface (descriptor sets, UBO binding, push constants, pipeline cache).
• SPIR-V disassembly and patching.
• PS4 OELF loader, Itanium RTTI, SCE dynamic relocations.
• mprotect-based page-fault tracking for CPU-GPU synchronization.

The journey, phase by phase

I’ll group the 30+ phases into buckets that make sense as a narrative. Each phase links to the full doc for anyone who wants technical detail.

Note for anyone who doesn’t want to read all 30+ phases: after the phase sequence there are reflection and methodology sections you can read on their own. The shotgun technique explains the debugging methodology I used throughout the investigation. What I knew on Sunday vs Thursday sums up what I actually learned. The AI suggested accepting. The perseverance was mine discusses AI’s real role in this process. The real cost of this journey catalogs what got produced. If you want to jump straight to the argument, those are the shortcuts.

Phase 01: pipeline cache, the warm-up

phase-01-shader-compile-stalls.md

• Hypothesis: menus run at 2-3 fps on first launch. Must be Vulkan pipeline compilation happening in real time.
• Action: enable pipeline_cache_enabled: true in the global config.
• New concept: Vulkan pipeline caching. When the emulator translates a PS4 GCN shader into SPIR-V and then into driver-specific Vulkan binary, it can cache that binary. Without the cache, it recompiles every cold launch (~864 shaders + ~590 pipelines).
• Outcome: solved. First session pays the cost; subsequent sessions read from disk and launch immediately. Easy. Also my first lesson about the shadPS4 codebase.

Phase 02: gamma dim, starting in the wrong direction

phase-02-gamma-dim-image.md

• Hypothesis: DriveClub’s image is dim compared to reference. Must be wrong sRGB encoding or an HDR tonemap applied badly by the emulator.
• Action: instrumented the swapchain to log format, added SHADPS4_PP_* env vars to experiment, built a post-processing pipeline with three knobs (exposure, ACES tonemap, gamma curve).
• New concept: sRGB OETF, HDR tonemap (ACES vs Reinhard), Bayer dither as anti-banding.
• Outcome: via bypass path (SHADPS4_PP_BYPASS=1, writes raw game output with no post-processing) I discovered the game writes correct SDR already into the framebuffer. The emulator’s post-processing pipeline was mangling the signal. Final shader became simply sRGB encode + Bayer dither. Dim persists, but it’s not a gamma problem.

Three hours on this phase. The most important lesson: when “the image looks wrong”, the first diagnostic should be a bypass path that shows raw game output. If bypass looks right, the emulator is the one mangling the signal. Stop touching the tonemap.

Phase 03: v1.28 content access

phase-03-v128-content-access.md

• Hypothesis: v1.28 extracts, but the game says “content not released yet, download required”.
• Action: merge v1.28 on top of v1.00, restore param.sfo, disc_info.dat, keystone.
• New concept: PS4 cumulative patches, package metadata.
• Outcome: solved. Detailed in the previous article.

Phase 04: MSAA depth + slow motion

phase-04-slowness.md

• Hypothesis: night tracks are pitch black and the game “feels slow”.
• Action: two separate things. Disable the 60fps XML patch (which was decoupling render rate from logic tickrate). Implement ReinterpretMsDepthAsColor on my fork (to let the emulator resolve 4x MSAA depth to 1-sample color, which DriveClub needs for forward+ lighting on night tracks).
• New concept: forward+ lighting (a renderer that writes the scene to an MSAA depth target and then reads it as color for SSAO and volumetrics), MSAA depth resolve.
• Outcome: both fixed. Detailed in the previous article.

Phase 05: second pass on brightness

phase-05-brightness-followup.md

• Hypothesis: since the game writes correct SDR, maybe a static boost in post-processing helps without touching the UBO feedback loop.
• Action: tried linear boost (1.5x, 2.0x), gamma pre-encode (pow curve), scene-aware auto-exposure with peak+mean sampling.
• Outcome: all rolled back. Linear boost clips highlights. Gamma introduces banding. My auto-exposure layer cancels the user’s manual brightness slider. There’s no free lunch in post-processing. Dim stays dim without real HDR or shader patches.

Narrowing the blackout: the shared gate

race-start-blackout-040420.md

• Hypothesis: the blackout has a curious signature. Main view goes dim/black, rearview mirror goes absolute black, HUD stays bright. Three layers with different behaviors suggest a shared “gate”, not a common fade.
• Action: instrumented texture-cache aliasing, probed with live fragment shader substitution on suspected inputs.
• New concept: texture-cache aliasing (multiple images sharing a memory address, which the emulator cache reuses), render-target lifetime.
• Outcome: points to a shared “presentation permission” between the main scene and the mirror, not a scripted fade.

Phases 07-09: texture torture, UI surgery, binary patch plan

phase-07-aggressive-torture-probe.md · phase-08-ui-asset-surgery.md · phase-09-eboot-binary-patch-plan.md

• Hypothesis: the race-start dim overlay is an asset. Small texture, UI panel, or a method of the FreeplayGetInCar controller in the eboot.
• Action: null-substituted BC-compressed materials, small texture candidates, four-vector weight masks during critical draws. Disabled UI panels (loading_freeplay.txt, get_in_car_animation.txt, vehicle_select_background). Extracted the OELF, located FreeplayGetInCar ASCII strings, identified the constructor at VA 0xf89b00 and the vtable at 0x15dc3e0.
• New concept: PS4 OELF loader (Sony’s variant of ELF), Itanium C++ RTTI (how C++ encodes type info in vtables), SCE dynamic relocations (placeholders that only the dynamic loader resolves at runtime).
• Outcome: nulls work but the fade persists. The overlay renders directly into HDR scene targets, not through the panel system. Binary patching has relocation placeholders that aren’t statically resolvable without a Ghidra + PS4 plugin. Asset-side surface exhausted. Six hours of debugging, final conclusion: the bug lives in state, not in an asset.

Phase 10: rethink + animlib

phase-10a-rethink-040422.md · phase-10b-video-diagnostic-animlib.md

Tuesday morning. Sat down, rebooted mentally. New model: the blackout is a state-machine gate. It’s not “scene not ready”, it’s “permission transition”. The gate closes after the starting grid appears. Next probe: runtime dispatch trace, enough of the blind asset surgery.

In parallel: frame extraction from the recording at 10 Hz, cross-referenced against keyframes of the animation library (animlib) in india_posteffects.lvl. Found a smoking gun. The MasterBrightness track has keyframes 0.003 and 0.0007 (0.63% brightness) where the default would be 1.0. Animlib beats inline defaults every frame.

• New concept: animation library, keyframe encoding, binary float-scanning.

Video above: the process of probing the fade in real time.

Phase 11: multi-scalar patch

phase-11-multi-scalar-patch.md

• Hypothesis: patching MasterBrightness + AutoTargetLuminance + ManualAutoMix simultaneously will lift the blackout.
• Action: extended PatchAnimlibFloatValues to rewrite three scalars. Instrumented with SHADPS4_DC_DRAWLOG + SHADPS4_DC_UBOLOG.
• Outcome: the patch produces a real numeric lift (1.2 → 6.5 in 255, a 5×), but still sub-perceptual black. And the crucial finding: the mirror works normally after the patch. It’s not gated, just reflecting a dark scene. At least 285× of additional attenuation lives downstream of the scalars we can patch.

Phase 12: bisect closes the asset side

phase-12-bisect-closes-asset-side.md

• Hypothesis: one of the 60 accumulated patches broke the mirror independently from the blackout. Bisect.
• Action: binary bisect across 60 overlay files over 5 rounds.
• Outcome: x_live_lobby_pre_race.txt (rerouting loading-spinner transition from ZoomInAndFade to NoFade) was the mirror killer. Rolled back. Asset side completely closed. The driver lives in compiled code or post-process state, not in a game asset.

Asset sweeps 01-09 (consolidated)

asset-sweep-09-conclusions.md

In parallel with the phases above, I swept 12 RPK substitutions: FreeplayGetInCar page swaps, postfx edits in india_landscape_gui.rpk, globaldata RTT-only edits, prerace state-string rewrites. All clean misses. No asset family touches the blackout. Lesson: stop chasing by names; chase by state transitions.

Phase 13: UBO dump breakthrough

phase-13-ubo-dump-breakthrough.md

• Hypothesis: eye-adaptation feedback loop with stale input is the culprit. A luminance scalar growing without converging.
• Action: dump 128 bytes of UBO in hex during the race window across six hand-picked pipelines. Look for monotonically changing values.
• New concept: UBO (Uniform Buffer Object), a shared memory region the CPU writes and shaders read. Auto-exposure integrator, a classic eye-adaptation component that takes time to converge toward a target.
• Outcome: found it. UBO offset 3 climbs monotonically from 2.59 → 7.84 → 24 → 90 → 179 → 255 over 2 seconds. Textbook feedback loop with broken input. First real hit in 12 phases.

Phase 14: scene-pipeline UBOs are the read side

phase-14-scene-pipeline-ubos-wrong.md

• Hypothesis: clamping the climbing scalar will stop the blackout.
• Action: SHADPS4_DC_LUM_CLAMP=2.0 forces offset 3 to a fixed value during the race window.
• Outcome: the clamp fires 1093 times, blackout unchanged. Scene-pipeline UBOs consume the dim, they don’t produce it. The dim is upstream, in a fullscreen tonemap or post-fx compute.

Phases 15-16: runtime texture dump + mutation

phase-15-runtime-texture-dump.md · phase-16-runtime-texture-mutation.md

• Hypothesis: the blackout is a fullscreen texture composited over the scene.
• Action: SHADPS4_DC_TEX_DUMP=1 dumps every bound texture during the race window (1079 textures captured). Convert tiled format to PNG for visual triage. Then, deterministically mutate each texture via hash-based tint + InvalidateMemory to force re-upload.
• New concept: GCN tile format (Sony interleaves bytes a specific way for cache locality), texture de-swizzle, buffer cache invalidation.
• Outcome: 14 rounds of mutation (BC textures, float data, render targets, UI atlases, post-fx buffers, LUTs). Blackout unchanged across all of them. Texture content 100% ruled out as the source.

Phases 17-19: UBO smashing, push-constant smashing, compute dispatch probe

phase-17-ubo-batch-smashing.md · phase-18-push-constant-smashing.md · phase-19-compute-dispatch-probe.md

• Hypothesis: random-fill on UBOs, push constants, and compute input buffers can bracket and isolate the dim field without crashing.
• Action: SHADPS4_DC_UBO_SMASH=1, SHADPS4_DC_PC_SMASH=1, SHADPS4_DC_DISPATCH_SMASH=1 with cb-index and size filters.
• New concept: push constants (128 bytes of fast params the driver pushes straight into the pipeline, no descriptor), compute dispatches (fires compute shaders with a grid of threads), ud_regs (GCN user data registers — how shadPS4 maps user data to push constants).
• Outcome: safe brackets produce overlay bokeh + blinks, but the dim stays put. Wide brackets that reach the dim also hit matrices/transforms and crash. Random-fill is the wrong instrument. Needs diff methodology.

Phase 19b-c: dim-vs-lift dispatch diff

phase-19b-dim-vs-lift-diff.md · phase-19c-dispatch-draw-skip-null.md

• Hypothesis: compute dispatches that only fire during dim (and stop once it lifts) are part of the chain.
• Action: split the clean log into pre / dim / bright windows (via user wall-clock), count dispatch frequency per pipeline.
• Outcome: identified 8 compute pipelines (histogram 256→downsamples, tile-grid reductions, auto-exposure 1x1 scalars) that stop exactly when dim lifts. Shapes strongly suggest the eye-adaptation chain. First genuine diagnostic hit. Skip test on those 8 + 101 correlated draws: blackout unchanged. They’re downstream effects, not drivers.

Phase 20-21: tonemap inline, fragment → flip buffer

phase-20-shader-dump-tonemap-inline.md · phase-21-fragment-to-flip-buffer.md

• Hypothesis: the auto-exposure compute shader is broken, or the tonemap compute has the wrong math.
• Action: recompile all shaders, dump SPIR-V, disassemble the histogram auto-exposure. Patch its output to constant 1.0 and test.
• New concept: SPIR-V disassembly (Khronos intermediate format with readable assembly). Compute shader verification. Later, pipeline-to-shader mapping.
• Outcome: patch has no effect. Discovery: DriveClub has no separate tonemap pass. Scene geometry writes directly to the flip buffer, with exposure inline in material shaders. Each material shader bakes the dim scalar from a shared UBO. Pipeline cache wipe + SHADPS4_DC_PIPEMAP=1 dumps every pipeline→shader mapping. Filtering drawlog for pipelines writing to 0x5000900000 / 0x5000108000 (the actual swapchain buffers) yields 14 unique pipelines. None with Exp2/Log2 tonemap ops. Exposure is applied upstream.

Phase 22: tonemap compute identified

phase-22-tonemap-compute-identified.md

• Hypothesis: among the game’s 510 compute shaders, one has a tonemap signature (many Exp2/Log2).
• Action: automated scan, cross-reference with dispatchlog.
• Outcome: cs_0x2c918c06 is the strong candidate, with 8 Exp2 + 8 Log2, 5 SSBOs (scene params, TAA, bloom, camera, exposure), 11 images. Skip test on that specific pipeline: stops the blackout cycle, removes color grading, introduces temporal ghosting. That’s the final scene→swapchain tonemap path.

Phase 22+: tonemap patches → playable

phase-22plus-tonemap-compute-patches.md

• Hypothesis: two surgical SPIR-V edits (histogram no-op + tonemap exposure clamp/boost) make the game playable.
• Action: histogram patch to never update; tonemap patch to clamp exposure floor at 0.5 and boost blend by ×10. Both via SPIR-V patching + shader/patch/ drop-in.
• Outcome: playable! Dim shrinks from pitch black to mild. A real mitigation, not a root-cause fix. The fade curve still drags exposure, and each track needs per-threshold tuning.

Here the AI suggested stopping for the first time. “The game is playable, this is a decent mitigation, we should document and accept.” I pushed back: “it’s playable but it’s wrong. Don’t accept. We keep going.”

Phase 23: CPU-side exposure intercept

phase-23-cpu-exposure-intercept.md

• Hypothesis: clamping the exposure scalar at the tonemap compute’s BindBuffers point prevents over-dimming.
• Action: SHADPS4_DC_EXPOSURE_PIN=1 intercepts BindBuffers, overwrites exposure offsets with a clamp-min threshold, invalidates buffer_cache.
• New concept: CPU-GPU buffer binding (the moment when the CPU hands a descriptor to the GPU). Buffer cache invalidation (how the emulator keeps consistency between a buffer’s CPU-side and GPU-side state).
• Outcome: the scalar oscillates at race start, then climbs monotonically. Per-track tuning is needed. India wants 3.0, Canada blows out at the same threshold. A single-scalar solution doesn’t converge cross-track.

Phase 24: dim upstream of tonemap

phase-24-dim-upstream-of-tonemap.md

• Conclusion: after the whole probe chain (identity tonemap, exposure pin, compute skip), the dim is in the HDR target before the tonemap reads it. Three upstream suspects: (1) atmospheric scattering / volumetric fog, (2) reflection-probe / envmap update, (3) pre-tonemap exposure apply.
• New concept: atmospheric scattering (simulating how light travels through atmosphere, Rayleigh/Mie-style), volumetric fog (fog rendered as a froxel volume).
• Outcome: plan set. Frame-order dump, walk backwards from tonemap through the whole chain.

Phase 25: frame-order trace

phase-25-frameorder-trace.md

• Action: SHADPS4_DC_FRAMEORDER=<N> logs strict per-submit pipeline/shader chain over N submits during race window.
• New concept: frame-order reconstruction (listing every GPU submission in exact order), froxel volumetrics (frustum volume voxels — a 3D grid aligned to the camera’s frustum that stores light info).
• Outcome: three suspects: (A) half-res compute (60,34,1), probably SSAO; (B) progressive dispatches (256,1,1)→(2560,1,1)→(10240,1,1), froxel volumetric; (C) fullscreen apply-fog-to-HDR draw. Suspect B is the strongest fit (sunset best-fit, per-pixel non-uniform dim).

Phase 26: The UBO fade pin, the “aha moment”

phase-26-ubo-fade-pin.md

This is where everything changed.

• Hypothesis: a direct memory diff (bright vs dim vs recovered) identifies the exact bytes the game writes to control the blackout.
• Action: capture periodic UBO snapshots across visual state transitions. Diff three snapshots byte by byte. Look for fade signatures.
• New concept: state diff methodology. Instead of “which pass looks suspicious?”, ask “which bytes actually change with the visual state?”
• Outcome: breakthrough. Two UBOs control the fade: a 1936-byte one with slots [38/48/50] (light intensity) being multiplied by 0.094 (fade factor), and a 224-byte sun UBO with slot [50] dropping 97%. Pinning both: race start opens bright. TOD animation intact. Content-aware three-state lifecycle (idle → engaged → expired).

Wednesday night. Claude suggests stopping again. “We have the pin working, three-state lifecycle is robust, Canada + Munnar are running. This is upstream quality, we should PR and close it.” I insist: “there’s still residual dim on other tracks, it’s not closed. We keep going.”

Phase 28: pivot to emulator code

phase-28-pivot-to-emulator-code.md

• Realization: the residual dim in Canada dusk/night isn’t a wrong UBO value. It’s an emulator translation gap. The game runs correctly on a real PS4, so shadPS4 has some path that translates or synchronizes incorrectly.
• Action: ranked 5 candidate emulator bugs: HLE shader misidentification, image storage classification, layout transitions, IMAGE_STORE_MIP fallback, OpImageFetch LOD restriction.
• New concept: HLE (High-Level Emulation). Replacing complex PS4 OS functions with host-side implementations. Different from LLE (low-level) which emulates bit by bit. HLE is faster but can diverge from real behavior whenever the host implementation doesn’t cover every case.

Phase 29: calibrated state + recording harness

phase-29-calibrated-state.md

• Action: SHADPS4_DC_RECORD=1. Recording harness that dumps lighting UBOs periodically + cross-correlates with wall-clock screenshots.
• Outcome: important discovery. Slots [144..295] of the 1936-byte UBO (30+ fields) stay denormal/uninitialized for the first ~90 seconds of a race. At the 1:30 “auto-recalibration”, the game writes the real values. It’s not magic recovery, it’s the CPU finally writing values that should have been there since frame 1. On PS4 they’re ready before the first frame; on shadPS4 they’re delayed by 90s. Root cause lives in an emulator init gap.

Phase 30: UBO writer audit

phase-30-ubo-writer-audit.md

• Action: exhaustive audit of every path that writes into slot [38] of the 1936-byte UBO. 7 suspects ranked: (1) ObtainBuffer stream-copy race, (2) lazy RegionManager in memory_tracker, (3) histogram compute skip, (4) page-fault delayed invalidation, (5) readback gating, (6) buffer-coherency race, (7) push-constant.
• Status: audit incomplete. Before I could verify each suspect, Phase 31 short-circuited the whole thing.

Phase 31: readbacks_mode = Precise, the turning point

phase-31-readbacks-mode-fix.md

Thursday morning. Tired, thinking about suspect #5, readback gating. That one itched at me because it was the only one of the 7 writers that touched the timing of the read rather than the correctness of the write.

• Hypothesis: flipping readbacks_mode: 2 (Precise) in per-game config makes the feedback loop work by synchronizing the CPU’s read to fresh GPU output.
• Action: one edit in custom_configs/CUSA00003.json. Boot. Canada night.
• Outcome: race start opens bright. Auto-exposure converges normally. Zero blackout. Zero 1:30 recalibration. Night TOD natural.

Tested on Munnar. Same result. One integer. 31 phases. 44 commits. 15,668 lines. Answer: one integer in the config.

Why did this slip past 30 phases? The Phase 31 doc has the honest analysis:

1. Symptom disguise. Progressive darkening looks exactly like a broken tonemap, bad shader, or miscompiled exposure scalar. Every dump-driven probe found wrong values in the UBO, which is true but describes the downstream read, not the upstream gap.
2. readback_linear_images red herring. I tested that flag in Phase 29, saw no effect, and concluded “the readback surface is audited”. I forgot that flag is for image readbacks (linear images via TextureCache::DownloadImageMemory). The histogram SSBO is a buffer, a separate path.
3. Pinning making it worse, not better. Overwriting slots [38/48/50] looked promising but never converged. The game rewrote on top of the pin every frame. I interpreted that as “the pin mechanism lands too late in the pipeline”. I spent phases 27-29 moving the pin earlier. The real fix was to stop clobbering the GPU-side input the integrator reads.
4. Never audited the buffer readback path. The Phase 30 audit listed 7 writer candidates, but all of them were about the correctness of the write. None asked “what is the CPU reading, expecting the GPU to have filled?”

The shotgun technique

One thing I want to put on record: torture probes aren’t waste, they’re learning infrastructure. And the methodology behind them has a name: shotgun.

If you have a suspicion (say, “the dim comes from some texture”), the worst thing you can do is start investigating one texture at a time, in depth. DriveClub has 1079 bound textures during the race window. If a detailed triage on one texture takes 20 minutes, you spend three days on a single branch of that tree and still don’t know if the dim even comes from a texture. It might not be in the whole tree.

Shotgun fixes that. Instead of investigating one at a time, you fire at all of them simultaneously. Mutate every bound texture with a hash-based tint, force re-upload, run the game. In 30 seconds you know whether any of them affects the dim. If it does, the dim is texture-side and only then is it worth starting to narrow down. If nothing changes, congratulations: you just ruled out the entire “texture” category in 30 seconds. Switch categories, shotgun UBOs. No hit? Shotgun push constants. No hit? Shotgun compute dispatches. Keep going until something reacts.

Don’t start by narrowing down a tree that might be huge. You’ll spend weeks in irrelevant leaves. Shotgun the trunk first. See which branch reacts. Then go deep inside that branch.

That’s what I did, without knowing I was doing it, for the first 30 phases. 32 probe env vars (SHADPS4_DC_UBO_SMASH, SHADPS4_DC_PC_SMASH, SHADPS4_DC_DISPATCH_SMASH, SHADPS4_DC_TEX_NUKE_*, SHADPS4_DC_UBO_NUKE, and more). Random batches, hash-based tints, null substitutions. Most of them never pointed at the real fix. But every shotgun round ruled out an entire category of mental model. “Not texture.” “Not push-constant.” “Not those 8 compute pipelines.” “Not the UBO write side anywhere in that region.” Every ruling-out narrows the problem.

And while I was doing it, I was learning the system. Every smash that crashed told me “there’s a critical descriptor pointer here”. Every color tint that appeared told me “this UBO feeds that composition pass”. Every dispatch that fired differently between dim/bright showed me where the interesting boundary lives.

When you don’t know the system, shotgun probes are a map. You don’t use them to find the answer. You use them to learn the terrain.

What I knew on Sunday vs Thursday

Sunday: zero everything. All the areas below were black boxes to me.

Thursday:

• PS4 package format end-to-end (PKG → PFS → param.sfo → disc_info → keystone → npbind). How v1.28 cumulative patches layer on top of a v1.00 base install.
• shadPS4 architecture in outline: src/common, src/core (OS emulation, HLE libraries), src/shader_recompiler (GCN→SPIR-V), src/video_core (Vulkan renderer, buffer cache, texture cache, page manager).
• Full graphics pipeline: G-buffer → forward+ lighting (writes MSAA depth) → read depth as color for SSAO / volumetric froxel → post-fx compute → tonemap → composite → swapchain.
• UBO vs push constant vs descriptor set. When to use each.
• SPIR-V disassembly and re-assembly for surgical patching. How to identify a tonemap compute by its Exp2/Log2 signature.
• Forward+ lighting + MSAA depth resolve + how to implement ReinterpretMsDepthAsColor.
• shadPS4 buffer cache: MemoryTracker, RegionManager, FaultManager. How page faults turn into GPU→CPU sync.
• readbacks_mode: Disabled / Relaxed / Precise. What each tradeoff looks like. Why Relaxed isn’t enough for feedback loops (it only write-protects, not read-protects).
• PS4 OELF → Itanium RTTI → SCE relocation → dynamic linker reasoning, even without actually doing the binary patch (the plan is documented).

I didn’t become an expert in any of it. But I went from total outsider to reasonably oriented operator. And that jump, as I’ll argue below, was the single most valuable thing of the four days.

The AI suggested accepting. The perseverance was mine

I’ve said a few times in this article that “Claude suggested stopping”. I want to be fair and specific: it wasn’t an AI failure. It was exactly the correct behavior of an intelligent tool: when you have something that works reasonably well, stopping and documenting is frequently the right call. The Phase 22+ tonemap patch made the game playable. The Phase 26 UBO pin made the image pretty. The Phase 29 recording harness covered the critical window. All were legitimate stopping points.

If you read codex-conclusion.md, the document the AI produced after Phase 26 (still not knowing about Phase 31 yet), it describes the then-current solution as “a mitigation, not a root-cause fix”. It honestly admits what we knew: the UBO pin was a countermeasure, not an explanation. That’s good behavior.

The difference is that I had a stubborn hunch that something was still wrong. It worked in the playable sense. But the cost (DriveClub-specific pin + tonemap SPIR-V patch + per-track threshold tuning + doesn’t generalize) was too high for a supposedly engineering-quality fix. A senior professional recognizes when the asymmetry between cost and understanding points to something deeper. You don’t always get it right. But that hunch deserves to be heard.

Four days without interruption, 8 a.m. to midnight, Sunday, Monday, Tuesday, Wednesday, resolved Thursday morning. How many times did the AI suggest “honestly, we should just accept the current result and document”? Four or five. Each technically defensible. Human perseverance is the difference between “the game works well enough” and “the game works, and I know why”.

The real cost of this journey

investigation-effort-accounting.md catalogs everything before cleanup. Some highlights:

• 3 days of active work (Apr 21-23, 2026).
• 33 numbered phases + sweeps + side threads = 54 docs, 412 KB, 6,867 lines of prose.
• 44 commits on the gamma-debug fork. 15,668 lines added, 1,204 removed.
• src/video_core/renderer_vulkan/vk_rasterizer.cpp grew from 1,364 to 4,680 lines (3.4×) with instrumentation, pin lifecycles, recording harness.
• 32 probe env vars defined. 1 effectively used to find the bug (SHADPS4_DC_LOG_STREAMCOPY, in Phase 30).
• 61 MB of shadPS4 log accumulated. 57 MB of shader dumps (5,803 files). ~470 MB of total reclaimable artifacts once closed.

Resolution: 1 integer in the config.

The ratio of 6,867 lines of prose : 1 integer amuses me. But it isn’t a joke about inefficiency. It’s the real cost of learning a system you didn’t know. Every line of prose ruled out a hypothesis. Every commit taught a layer of the stack. The integer was the end; the path was the product.

An honest note: despite the 15,668 lines added on the gamma-debug fork, almost none of it is useful to contribute upstream to shadPS4. What’s there is instrumentation: torture probes, recording harness, pin lifecycles, calibrate-at-arm, UBO snapshots, dozens of debug env vars. Diagnostic code, not production code. Diagnostics that only make sense for DriveClub under those specific conditions.

The fork is going to stay public, but not as a contribution branch to the shadPS4 project. Unfortunately. It stays as a study document: 54 md files covering the full process, 44 commits showing the sequence of hypotheses, rule-outs and failures. It’s material for anyone who wants to understand how the debugging process unfolded, or to reuse some of the techniques in similar investigations. No upstream-ready PR is coming out of that tree.

Closing

Yes, the final solution was a config toggle that already existed. It’s been in shadPS4’s code for seven months. But on Sunday night, with zero PS4 knowledge, zero emulator knowledge, zero notion that auto-exposure was a GPU→CPU feedback loop, I had no technical authority to say “flip readbacks Precise, done”.

I had to see the whole chain to understand why it works. Why Relaxed isn’t enough (it only write-protects, and DriveClub reads, it doesn’t write into the histogram SSBO). Why Precise is expensive (per-page mprotect + per-fault GPU stall). Why the maintainer hid it behind “Advanced” (Bloodborne hangs, AMD tanks to 12 FPS). Why nobody in the community had connected Precise to DriveClub before: the compat tracker admits “readbacks enabled” but doesn’t cite the mode, and nobody had seen the monotonic-drift signature through the GPU→CPU lens.

That’s the difference between knowing the answer and knowing the terrain well enough to recognize the answer. To the junior asking me “how do I learn today, in the AI era”: that’s my answer. Don’t ask. Dive in. Use AI to speed up the search, to read code in parallel, to index old forums, to compile, to parse logs. But insist on understanding the why. When the AI suggests “let’s accept and document” (and it will, because that’s often the right behavior), ask yourself whether you actually understood the chain. If the itch “this is wrong but I don’t know why” is still there, keep going.

You’re probably going to spend four days on something that has a one-integer solution. You’re probably going to discover five wrong paths before the right one. Probably half your probes will be torture shotguns that never pointed at the fix.

And you’ll probably, when you finish, know a domain you didn’t know before.

• Fork gamma-debug: github.com/akitaonrails/shadPS4/tree/gamma-debug
• 54 investigation docs: docs/driveclub-investigation/
• Operational runbook (recipe): distrobox-gaming/docs/driveclub-shadps4.md
• Phase 31 resolution: readbacks_mode: 2

Now I’m going to go play DriveClub. Night race in Canada, natural brightness, no blackout. Good Thursday.

The Blender community is so passionate and committed that they frequently produce high quality, almost Hollywood grade short movies inside Blender. The point is to stress test the tool and flush out bottlenecks and usability issues in a real world workflow.

This is primarily a post for “future me” to jump back into a single resource list. Being a 3D modeler is not my full time job, so I’ll have large gaps between Blender sessions. And I know I’ll regret it if I don’t dump my brain into a post while it’s still fresh :-)

Beginner First Steps

First things first. If you’re in this blog, you’re probably a programmer. And let me tell you that the Graphics folks see “usability” very differently from what we, programmers, usually think of. The sheer amount of customization, options, shortcuts and combinations will rival even Vim and Emacs users out there.

Oh, by the way, you’re going to need a real 3-button mouse. Touchpads are useless with Blender, and the Mac mouse is terrible. Any cheap PC mouse will do.

Out of the box, the main button is the right one, not the left one we’re used to. Change that in the user preferences to select with Left, and things start to make sense. While you’re there, enable Numpad emulation too.

By the way, it’s well worth having a keyboard with a numpad, or even a separate numpad. You can use the number row on top of your normal keyboard, but Blender was built assuming both an inverted 3-button mouse and a numpad.

If you have never studied Blender before, you will not figure things out just by randomly exploring the UI. The UI is useless until someone teaches you the ins and outs. There are hundreds of terms you just have no idea what they mean, such as Meshes, Edges, Seams, Nodes, Viewport, Subsurface, Modifiers, etc.

The main thing you MUST do before proceeding is watching this entire 9 part series from Blender Guru, called Blender Beginner Tutorial Series. Optionally you can slowly do the Intermediate Blender Tutorials later.

The Beginner Series will teach you enough that you’ll finally start feeling confident with the UI and tools. And don’t forget to print and hang this handy “cheat sheet”. You won’t believe how much easier your life becomes once you get used to the most important keyboard shortcuts.

Better Defaults

For historical reasons, some things are not how they should be. Knowledge in the 3D rendering field is evolving very fast.

The very first thing to do: change the default render engine from Blender Render to the Cycles Raytracing Engine.

Then, color grading. The default sRGB EOTF is basically wrong. You must download Sobotka’s filmic-blender configuration. If you’re on Arch Linux you can basically do:

pacaur -S filmic-blender-git

I created a script at ~/bin/filmic-blender with this:

env OCIO=/usr/share/blender/2.78/datafiles/filmic-blender/config.ocio blender

And I always start Blender from the terminal like this:

optirun ~/bin/filmic-blender

This does 2 things: first, it pre-configures OpenColorIO to use the Filmic replacement. Second, it enables the external GPU of my notebook to be available to Blender. Read my post on “Enabling Optimus NVIDIA GPU on the Dell XPS 15 with Linux, even on Battery” for more details. On Windows or Mac this is not necessary, but you’ll still need to load the filmic configuration.

Then, on the Scene tab you must reconfigure “Color Management” to be like this:

Then you need to configure Cycles. If you’re on Linux and Optimus is correctly installed as I explained before, you should have the CUDA option enabled in the User Preferences:

In the Render tab, you should have something like this:

In the Dimensions section, you’ll notice it has the Full HD (1920x1080 px) size but only 50% (so it will only render half the size). Increase it to 100%. If you want a 4K shot, increase the dimensions to 3840x2160 px. 4K makes it 4 times slower to render than 1080p, obviously.

In the Performance section you’ll see 2 input fields for “Tile size”. Blender will slice your full image into tiles, and each tile renders in parallel on an available CPU or GPU core. My notebook has 8 CPU cores, so 64 is a good size, because it will render 8 tiles of 64 pixels each in parallel.

The fewer cores you have, the larger you should make the tile sizes. For my NVIDIA GPU, I only have 2 available cores (and not a lot of video memory either!), so it’s worth bumping both fields to 512. On the GPU it only renders 2 tiles at once, and larger tiles optimize the render.

As you probably guessed, Blender Guru has a very useful “18 ways to speed up Cycles Rendering” tutorial.

These should cover the basic defaults.

You must think as a Photographer!

You will want to watch A LOT of online tutorials, because it’s really not obvious how to best use the tools. Another channel I really like is BornCG and CG Masters. Pick a few of their videos to get a more in-depth view on specific tools, modeling techniques and so on.

And really, you must practice as much as possible while studying a lot along the way. One important area is Photography. This is what you see when you select the Camera tab:

There’s a lot of customization you can do here. For example: Focal length: 35.00, which is what photographers know as a 35mm lens, a good default choice. You can also use a really wide angle such as 200mm or 300mm, and indoor shots, in small rooms, could use a 24mm or even 18mm lens. This article “Understanding Color Lenses” should give you the basics.

Then, you must understand “Depth of Field”. This can be done at this configuration before rendering, or you can simulate it after rendering (if you chose to separate the render in layers), in the Compositor.

Speaking of which, another way to control Depth of Field is reconfiguring “f-stop”, which is the measurement of exposure, or aperture. The default is “128.0”, which is “f/128”. As a reference, your iPhone 7s camera has an f/2.2 aperture. Again, let’s study more about this starting with the article “What’s the Best F-Stop?”

Taking a photo (or rendering a still, in our case) is a whole lot more than just point-and-click. You also have to worry about proper composition techniques such as the Rule of Thirds, the golden ratio, and so on. You can change that in the “Composition Guides” combo box as shown above.

You really should dive deep into the field of Photography to improve the final result of the renders. I’m an amateur and it’s really exciting to be able to apply real world techniques to 3D rendering.

Material Design and Physically-Based Render (PBR)

The golden standard in 3D modeling and rendering is certainly Disney/Pixar RenderMan. Every award winning Pixar movie was made with it.

But Blender learns fast. Every Pixar paper eventually becomes part of Blender itself. Material design, for example, has always been quite cumbersome in the tool. I did some of the tutorials myself, and creating materials with the proper real world characteristics, like proper Fresnel and the right dielectric versus metallic distinction, was a challenge.

If you subscribed to Blender Guru’s channel you should really watch the tutorials “How to Make Photorealistic PBR Materials - Part 1” and “Part 2”. And you will end up with this complicated Nodes configuration for PBR materials:

And Blender Guru just posted a new video introducing a feature for the upcoming Blender 2.79 (currently at 2.78): the implementation of the “Physically-Based Shader at Disney” paper as a proper and optimized new Blender Shader named “Principled Shader”. It’s quite literally the Ultimate Shader, because it makes creating and customizing realistic materials very easy, and compatible with Renderman and Substance.

Conclusion

I am still a beginner at Blender, there is a very long road to walk here. But it’s a very exciting environment and I am learning tons of new and useful stuff all the time. Anyone should try it!

Over time I hope I find the time to integrate the Blender workflow with other tools such as Unreal Engine or Unity3D to create interactive stuff as well.

This is by no means a complete tutorial or reference. The idea was to highlight a few things that are not immediately obvious when you start with Blender and that can give you a broader sense of what Blender can do.

If you want to go really in-depth in the customization, watch this CG Master setup video:

I’m a longtime proponent of killing gods; it’s practically a hobby. They make excellent fiction characters in bedtime stories, but obviously they don’t exist. Like the boogeyman, Santa Claus, or the Easter bunny. And no, this isn’t another anti-religion article, relax.

For the same reasons, I want to dismantle the myth of “Talent” and bring the demigods back to humanity.

Talent Is Highly Overrated

I really hate the use of the word “Talent.” It should be used as a compliment, a quality, but it’s very often used as an excuse. The Latin root of the word simply means a large amount of money. Today’s mistaken and common use devalues the real efforts of truly “talented” people.

Every time it’s used to denote something like “I’ll never be an Einstein, so I won’t even bother trying.” It’s the favorite excuse of lazy people.

Although environmental and circumstantial factors play a crucial role, there is ABSOLUTELY NOTHING in our biological material, like a gene, that would stop someone from reaching similar levels of success given the same circumstances.

Yes, some people end up, randomly, in the right place at the right time. It’s like winning the lottery. And unfortunately, it’s impossible to recreate the same network of circumstances for everyone and replicate levels of success (more on this at the end of the article).

At the same time, it doesn’t help at all to believe in the mysticism that “talent” can be defined physically, “genetically.” That’s the thinking that creates Josef Mengeles and much of racism and other forms of segregation. There is no predestination.

These days you’ve probably heard of the concept of 10,000 hours of deliberate training to become a master at something, first presented in the book “Outliers” by the prolix Malcolm Gladwell.

From Bill Gates to Oppenheimer to the Beatles, the book tries to describe how these people’s continuous, focused effort, the famous 10,000 hours (roughly 5 years full-time), helps explain their levels of success.

Geoff Colvin, in his book “Talent Is Overrated: What Really Separates World-Class Performers from Everybody Else” tries to go further than Gladwell and explains that, obviously, 10,000 hours of training alone isn’t enough if it isn’t “deliberate practice”: obsessive training focused on continuous improvement, above the mere repetition of the same routine.

In summary: do you want the chance to become a master at something? Have the obsession to train focused on improving by millimeters and milliseconds over 10,000 hours. Repeating the same routine, without focusing on improving results, doesn’t take anyone anywhere. It’s just a pastime.

This kind of deliberate training that Gladwell and Colvin detail in their books is one possible factor of success.

10% of the Brain: the Myth

This one is easy to debunk, so I’ll start here.

One of the “mystical” aspects of talent is linked to the belief in people who can activate more than 10% of the brain. The myth that we don’t use all of our “potential.”

Recent fiction films like “Lucy” by Luc Besson don’t help at all because they perpetuate this popular belief. Let’s describe it this way:

“The human brain constitutes 1/40 of the total mass of the human body, on average, but consumes 1/5 of all the calories we absorb. From an evolutionary point of view, where every organ in our body was created and naturally selected over ages for efficiency, having a brain that sucks up 20% of all our daily energy reserve for a mere 10% of efficiency simply makes no sense.”

Think about yourself right now. Are you using your muscles to sit there now? Using your hands to scroll your computer mouse (or finger on your phone screen)? Maybe eating or drinking something? Listening to some music? Breathing? Rest assured, you’re using more than 10% of your brain right now - Nature

With that factor out of the way, let’s continue.

Humanizing Einstein

In general, I don’t respect biographies. I rarely waste my time reading any. I’m not against them, but to me a biography is a story told by a writer, who took into account every technique to keep me hooked on the story he wants to tell. Every biography is limited to the writer’s interests, not the character whose name is on the book’s cover.

In fact, I consider biographies as any other kind of entertainment story. In exactly the same category as watching Captain America or Superman. Because that’s exactly how writers and directors produce “based on true facts” films.

By the way, hats off to the Coen brothers for the film “Fargo” which opens with the line “this is a true story.” And many people, even today, believe it’s real. A trolling of every film that begins with the infamous “based on a true story” while everyone thinks they’re watching facts, not entertainment.

Put in a name that existed, a situation that existed, and everyone associates the entire fiction as if it were real. I hope nobody thinks Lincoln was a vampire hunter because of the film “Abraham Lincoln: Vampire Hunter”.

But if I were to ask anyone in the world who the greatest super-genius scientist of all time was, with the theories that most revolutionized humanity, no one would hesitate to say: Albert Einstein!

But why?

Physicist Roger Schlafly, in his book “How Einstein Ruined Physics: Motion, Symmetry, and Revolution in Science” brings a ton of evidence to the contrary.

In summary, yes, Einstein really did publish many very interesting things, but the super-inflated myth created around his figure is much more harmful than beneficial for the future of physics and his successors. And it keeps fueling the myth-making market.

Einstein was a person full of flaws, like all of us. A long way from a demigod.

Mark Green, in his article “New Book Says Einstein Highly-Overrated and Ruined Physics” describes:

“What made Einstein so great? The official story goes this way: Albert Einstein, a young clerk in a Swiss patent office, single-handedly transformed physics from a static, three-dimensional science to a four-dimensional, mind-blowing, time-space universe via brilliant and solitary ’thought experiments’ involving gravity, motion, space and time. Einstein also made unprecedented inroads into understanding the nature of light and energy and was the first to comprehend the equivalence between energy and mass. Einstein’s discoveries not only transformed modern physics but the way we view the universe.”

It’s what we all know. And I strongly recommend you read Mark Green’s full article, but let’s get to the points that matter.

E=MC²: was not invented by Einstein!

The equation was published 2 years earlier by Olinto de Pretto. As Schlafly says in his book, “Einstein’s understanding of special relativity … was inferior to (Henri) Poincaré’s.” By the way, if you’ve never heard of the French physicist Poincaré, I recommend reading the book “Chaos: Making a New Science” by James Gleick.

The notion of motion and time as a 4th dimension separate from three-dimensional space.

H.G. Wells speculated about the concept in the famous fiction piece “The Time Machine” in 1894. That same year Lorentz wrote the scientific paper where he “proposes the concept of local time in a moving object.” Poincaré wrote one in 1898 and another in 1900 exploring the relationship between motion and time.

And the relationship of E=MC² to the atomic bomb?

As Schlafly describes, E=MC² isn’t even necessary for the atomic bomb. The equation gives no clue about how to split the atom, or how to create a nuclear chain reaction, or any of the other steps necessary in making the atomic bomb. Relativity isn’t even necessary to understand the energy release of a uranium or plutonium bomb, since the release can be explained by electromagnetism considerations … predictions about relativistic mass were being tested (by German physicist Walter Kaufmann) in 1901, before Einstein wrote anything about it.

And, to top it off, it was also H.G. Wells who first published about the concept of “atomic bombs”.

There were many exceptional scientists, who worked hard, before and during Einstein’s era. The French mathematician and physicist Henri Poincaré, whom British philosopher Bertrand Russell (remember Russell’s teapot?) called the greatest man France ever produced. The Dutch physicist and Nobel laureate Hendrik Lorentz. And the Scottish physicist and mathematician James Clerk Maxwell.

According to Schlafly, Maxwell was the first to coin the term “relativity” and created the first true theory of relativity of mass and energy. Electromagnetism generated the most important equations in the history of science.

Poincaré’s subsequent work on relativity provided the theorems that were “mathematically identical to Einstein’s,” and most of his work came before Einstein. Lorentz and Poincaré had all the main aspects of the theory of relativity, and published before Einstein.

Remember another detail of the “official story.” Einstein worked at a patent office. He understood the importance of “intellectual property” and had access to a lot of work before it had been patented!

And so a talent is born, irreplaceable and of inestimable value … for the mass entertainment market.

Hold onto another detail: what if Einstein hadn’t existed? Would we have been deprived of the Theory of Relativity? No, that work was practically complete before him. Others would have served the role and the media and folklore would just have a different name to call God.

Talent, is relative …

Every Genius Is Actually a Group!

An important fact is that the Theory of Relativity wasn’t the invention of one person. A lone wolf who had a “Eureka” moment. Eureka moments are very different from what you imagine.

Understand that today, the Theory of Relativity is basic material for any physics student. Nothing remarkable.

Everything we have today is the result of hundreds of individuals and groups, who throughout history have used what was known before, thrown away what no longer works, refined what works, and added new pieces for the following generations.

Nothing we do today is truly done “from scratch.” Everything we have at our disposal today makes us orders of magnitude superior, more efficient, faster, than the generation that lived 200 years ago, millennia ago.

We don’t create anything based on absolute truths written in stone millennia ago. Evolution exists because everything is volatile. Everything that has been proving itself correct given years, centuries of scrutiny, becomes the strongest tools to create new things. Engineering, Medicine, etc.

Thanks to that we went from the era of subsistence to agriculture. To the Industrial Revolution. To the Information Age. And to where we’re heading toward the Age of Artificial Intelligence, and so on.

“Talent” is a way of pointing out moments in this continuous and uninterrupted history where society as a whole has been climbing on the shoulders of giants to take longer strides into the future. But no “one” individual was necessarily pivotal. The achievements would be made by someone, but from time to time, we elect a few individuals.

It’s easier to tell the story that way. That’s all. Innovation is always a group effort.

Serendipity, Randomness, and Complex Systems

Unfortunately we, human beings, are terrible at understanding randomness. The world is highly random.

More than Einstein, the scientist who published the most significant work for humanity was Charles Darwin and his magnum opus “On the Origin of Species” where he explains how complexity emerges in a world of random events through natural selection.

I wrote in part about this in the post “Processes, Methodologies, and the Human Brain” from 2013 and in the post “We Are Mathematically Ignorant” from 2008. I’ll try not to repeat myself too much.

The very story of how we survived and evolved in this world, until reaching the humanoid form we have today, is the result of what we had to face in the natural environment, or we wouldn’t be here.

And a crucial factor in our primitive brain is finding patterns, things that repeat, and doing it fast. If, in the dead of night, in the middle of a forest, we see a small pair of shining circles moving around us and don’t immediately connect them to the eyes of a lurking wolf so we can run as soon as possible, we wouldn’t be here.

That same mechanism is what makes us see “faces in the clouds,” “images in coffee grounds,” “the face of Christ on a dirty cloth,” and so on. Identifying patterns amid randomness guaranteed our survival, but in the modern world this capability of our brain leads us to totally wrong conclusions of cause and effect.

That’s why I’ve written so much about “correlation isn’t the same as causation.”

“cum hoc ergo propter hoc”

The real world is “complex,” in the mathematical sense, which describes the exponentiality of results given a network of factors. There is no linear, step-by-step path in anyone’s biography. Not even in twins who were born and lived together, under the same set of factors, at least in theory.

“Complexity” isn’t what we’re used to imagining it is, and it’s what prevents us from predicting the weather more than the short term. It’s what leads to Lorenz’s famous butterfly, Mandelbrot’s fractals, and Chaos Theory. And more recently to Modern Network Theory (including Social Networks) published by authors like Barabási, Strogatz, or Duncan.

Anyone who knows me knows I’ve been fascinated by this subject for a decade and have already recommended all these authors’ books. It was even the topic of my first public talk in 2008, “Killing the Average”.

Beyond those 10,000 hours of obsessive deliberate training toward continuous improvement, you also need to be exposed to as many situations and people as possible to increase your chances of “serendipity” which is the combination of “being in the right place, at the right time, and with the capability to seize the opportunity.”

This is what can lead to such “success.” There’s no simple recipe, only hard work and a good dose of randomness. When all these factors align, the results can really seem “miraculous” or “superhuman.” And given enough time and the human habit of inflating stories each time they’re retold, in a few years we create new Demigods.

Killing Gods

In every mythology there are many gods. Both popular manifestations of beliefs around the “inexplicable” facts of nature (at the time) like storms, volcanoes, the seasons, the weather.

And there are also the demigods, human beings who demonstrated some kind of superiority or so-called “miraculous” feats (at the time) that no other human could ever achieve, with a good dose of creativity in telling the story.

Both cases are explainable. Unfortunately the logical explanation is much less dramatic and doesn’t make as good a story, most of the time. Curing diseases and advancing humanity isn’t as “sexy” as a so-called “miraculous” feat.

But our society’s style of mass communication (newspapers, magazines, television, films, social media) loves “sexy stories.” Negative news sells. Sexy sells. Logical facts don’t generate audiences. And, worse, they elevate false myths to the status of “truth.”

“Tell a lie enough times and it becomes truth.” - Joseph Goebbels

Just because you yourself don’t know the explanation for something, the argument that “therefore there is the possibility of something mystical” doesn’t hold. If I can’t explain that your God doesn’t exist, neither can you prove that he does. It’s a fallacious argument and has no place in a serious discussion.

And that’s being nice, because whoever tells the story is the one who carries the burden of proof. Not me, the one listening, who’s required to disprove you.

The last 200 years have been fantastic, because we’ve been killing gods at rates never before recorded in the history of humanity. It’s the reflection of how much we’ve advanced in our knowledge about nature and how it works. There aren’t many left, and we don’t need to keep creating more.

Conclusion

The circumstances of why certain individuals are elevated to the status of demigods isn’t an easy answer. Much less a linear sequence of events. Much less an entertainment film, with high doses of fiction and drama, that claims to be “based on real facts.”

We all live in a complex world, where everything is intertwined in an infinite network of possibilities. Each individual has very different circumstances.

Many have no conditions at all and even so they exceed expectations, with superior determination, with superior dedication, with superior patience, and some really reach success.

“Natural” talent?

Many have the conditions, but don’t have dedication, don’t have patience, don’t have determination. And they fail. How many stories of heirs of great fortunes destroying the legacy of their ancestors?

Lack of Talent?

Everyone has chances. But there’s no point in waiting. The best thing to do is to build the capabilities as soon as possible so that, if randomness arrives and you’re in the right place at the right time, you have the conditions to seize the opportunity and surf on it.

But there are no simple answers, no recipes, no miracles, and no talents naturally born with genius ready and, therefore, unreachable ideals.

Now, if you have the conditions and don’t get success handed to you on a plate, it isn’t the “bad luck” of having been born without the talent gene. It’s pure laziness. Simple as that.

Don’t have idols. Have references.

I can probably pinpoint why a certain argument is wrong. But I am very incompetent the other way around: given a proper data collection, how to do proper exploratory data analysis? What are the correct methodologies for any given scenario? And of course, all the math involved.

I just posted 2 articles to shed some light on exactly that. Most developers’ “common sense” on dealing with data is primitive aggregations such as sums and averages. Start talking about “standard deviation” and you lost half of the developer population. Start talking about binomial or Poisson distributions and you lost the remaining half. Now get into linear regression and bayesian statistics, and almost everybody left the room already.

We live in the 21st century. Every 60 seconds, Facebook receives an extra 3.3 million new posts, YouTube receives 400 hours of videos, Instagram receives 55,555 uploaded photos, and WhatsApp exchanges 44.4 million messages. Even e-mail: more than 206 million of them are fired off. By the time you finish reading this post, you can multiply that amount by 5 to 10.

“We are drowning in information,

but we are starved for knowledge”

– Various authors, original probably John Naisbitt

I will spend a few weeks diving into R. Yes, many people will talk about Julia, but we can’t deny the amazing body of knowledge, the experience, and the robust and extensive set of packages available for R, including learning material. There’s a great tool called RStudio, which just saw its 1.0 release.

And within the many materials I gathered, one stood out just for its introduction, and I still have to review the book as a whole. It’s called “Learning Statistics with R”, by Daniel Navarro from the University of Adelaide, Australia. I find it interesting because it’s a psychology professor teaching proper statistics through the basics of R, which is exactly what I wanted. You can purchase a hardcopy or download the free PDF.

I like the introduction so much that I wanted to share a few paragraphs to motivate you to join me in learning better statistics. So let’s dive right into it:

The cautionary tale of Simpson’s paradox

The following is a true story. In 1973, the University of California, Berkeley got into some trouble over its admissions of students into postgraduate courses. Specifically, the thing that caused the problem was that the gender breakdown of their admissions looked like this …

         Number of applicants     Percent admitted
Males           8442                    44%
Females         4321                    35%

… and they got sued. Given that there were nearly 13,000 applicants, a difference of 9% in admission rates between males and females is just way too big to be a coincidence. Pretty compelling data, right? And if I were to say to you that these data actually reflect a weak bias in favor of females, you’d probably think that I was either crazy or sexist.

Oddly, it’s actually sort of true … after Berkeley got sued, people started looking very carefully at the admissions data (Bickel, Hammel, & O’Connell, 1975). And remarkably, when they looked at it on a department by department basis, it turned out that most of the departments actually had a slightly higher success rate for female applicants than for male applicants. The table below shows the admission figures for the six largest departments (with the names of the departments removed for privacy reasons):

                      Males                          Females
Department   Applicants   Percent admitted   Applicants   Percent admitted
A                825             62%            108              82%
B                560             63%             25              68%
C                325             37%            593              34%
D                417             33%            375              35%
E                191             28%            393              24%
F                272              6%            341               7%

Remarkably, most departments had a higher rate of admissions for females than for males! Yet the overall rate of admission across the university for females was lower than for males. How can this be? How can both of these statements be true at the same time?

Here’s what’s going on. Firstly, notice that the departments are not equal to one another in terms of their admission percentages: some departments (e.g., engineering, chemistry) tended to admit a high percentage of the qualified applicants, whereas others (e.g., English) tended to reject most of the candidates, even if they were high quality.

So, among the six departments shown above, notice that department A is the most generous, followed by B, C, D, E and F in that order. Next, notice that males and females tended to apply to different departments. If we rank the departments in terms of the total number of male applicants, we get A > B > D > C > F > E (the “easy” departments are in bold).

On the whole, males tended to apply to the departments that had high admission rates. Now compare this to how the female applicants distributed themselves. Ranking the departments in terms of the total number of female applicants produces a quite different ordering C > E > D > F > A > B.

In other words, what these data seem to be suggesting is that the female applicants tended to apply to “harder” departments.

And in fact, if we look at all Figure 1.1 we see that this trend is systematic, and quite striking. This effect is known as Simpson’s paradox. It’s not common, but it does happen in real life, and most people are very surprised by it when they first encounter it, and many people refuse to even believe that it’s real. It is very real. And while there are lots of very subtle statistical lessons buried in there, I want to use it to make a much more important point . . . doing research is hard, and there are lots of subtle, counterintuitive traps lying in wait for the unwary. That’s reason #2 why scientists love statistics, and why we teach research methods. Because science is hard, and the truth is sometimes cunningly hidden in the nooks and crannies of complicated data.

Before leaving this topic entirely, I want to point out something else really critical that is often overlooked in a research methods class. Statistics only solves part of the problem. Remember that we started all this with the concern that Berkeley’s admissions processes might be unfairly biased against female applicants. When we looked at the “aggregated” data, it did seem like the university was discriminating against women, but when we “disaggregate” and looked at the individual behavior of all the departments, it turned out that the actual departments were, if anything, slightly biased in favor of women.

The gender bias in total admissions was caused by the fact that women tended to self-select for harder departments. From a purely legal perspective, that puts the university in the clear. Postgraduate admissions are determined at the level of the individual department (and there are very good reasons to do that), and at the level of individual departments, the decisions are more or less unbiased (the weak bias in favor of females at that level is small, and not consistent across departments). Since the university can’t dictate which departments people choose to apply to, and the decision making takes place at the level of the department it can hardly be held accountable for any biases that those choices produce.

That was the basis for my somewhat glib remarks earlier, but that’s not exactly the whole story, is it? After all, if we’re interested in this from a more sociological and psychological perspective, we might want to ask why there are such strong gender differences in applications. Why do males tend to apply to engineering more often than females, and why is this reversed for the English department? And why is it the case that the departments that tend to have a female-application bias tend to have lower overall admission rates than those departments that have a male-application bias? Might this not still reflect a gender bias, even though every single department is itself unbiased? It might.

Suppose, hypothetically, that males preferred to apply to “hard sciences” and females prefer “humanities”. And suppose further that the reason why the humanities departments have low admission rates is because the government doesn’t want to fund the humanities (Ph.D. places, for instance, are often tied to government funded research projects). Does that constitute a gender bias? Or just an unenlightened view of the value of the humanities? What if someone at a high level in the government cut the humanities funds because they felt that the humanities are “useless chick stuff”. That seems pretty blatantly gender biased. None of this falls within the purview of statistics, but it matters to the research project.

If you’re interested in the overall structural effects of subtle gender biases, then you probably want to look at both the aggregated and disaggregated data. If you’re interested in the decision making process at Berkeley itself then you’re probably only interested in the disaggregated data.

In short there are a lot of critical questions that you can’t answer with statistics, but the answers to those questions will have a huge impact on how you analyze and interpret data. And this is the reason why you should always think of statistics as a tool to help you learn about your data, no more and no less. It’s a powerful tool to that end, but there’s no substitute for careful thought.

Download the Book

Did you get intrigued? So download the book and let’s study some statistics beyond the mere basics. I am not saying this is the best book, just one that seems interesting, and if you happen to know a good book teaching statistics for novices through the use of R without any prior knowledge, let me know in the comments section below.

When you’re learning Erlang through its official documentation, the first thing you build in the chapters on Processes is a very simple Ping Pong code that looks like this:

-module(tut15).

-export([start/0, ping/2, pong/0]).

ping(0, Pong_PID) ->
    Pong_PID ! finished,
    io:format("ping finished~n", []);

ping(N, Pong_PID) ->
    Pong_PID ! {ping, self()},
    receive
        pong ->
            io:format("Ping received pong~n", [])
    end,
    ping(N - 1, Pong_PID).

pong() ->
    receive
        finished ->
            io:format("Pong finished~n", []);
        {ping, Ping_PID} ->
            io:format("Pong received ping~n", []),
            Ping_PID ! pong,
            pong()
    end.

start() ->
    Pong_PID = spawn(tut15, pong, []),
    spawn(tut15, ping, [3, Pong_PID]).

It’s not pretty, it’s Prolog-ish. The Clojure article claims how close they got with lightweight threads (true green threads) and this is the same exercise done in Clojure:

(defsfn pong []
 (if (== n 0)
    (do
      (! :pong :finished)
      (println "ping finished"))
    (do
      (! :pong [:ping @self])
      (receive
       :pong (println "Ping received pong"))
      (recur (dec n)))))

(defsfn pong []
  (receive
   :finished (println "Pong finished")
   [:ping ping] (do
                  (println "Pong received ping")
                  (! ping :pong)
                  (recur))))

(defn -main []
  (register :pong (spawn pong))
  (spawn ping 3)
  :ok)

People that like the Lisp-y aesthetics of programming directly in the AST representation through the structure of parenthesis based blocks will find this very pretty.

I personally spent years looking into code like this (Common Lisp, Elisp, Scheme, etc) and I still can’t get used to it. Once you have a competent editor such as Emacs, that can deal with the proper parenthesis handling, it’s easier, yes, but I still can’t find the joy in doing this kind of hackish syntax.

Elixir is much more than a new syntax on top of Erlang, as the great book Metaprogramming Elixir will teach you. It opens up the entire Erlang BEAM AST through the usage of the quote/unquote mechanics, making programming directly into the AST through “Hygienic Macros” a breeze. It’s really the best of both worlds of having a modern, good looking, joyful syntax and the same power a Lisp-y language gives you in terms of well behaved macros.

Now, this is the same example as above, in Elixir:

defmodule ExPingPongTut15 do
  def ping(0, pong_pid) do
    send pong_pid, :finished
    IO.puts("ping finished")
  end

  def ping(n, pong_pid) do
    send pong_pid, {:ping, self}
    receive do
      :pong -> IO.puts("Ping received pong")
    end
    ping(n - 1, pong_pid)
  end

  def pong do
    receive do
      :finished -> IO.puts("Pong finished")
      {:ping, ping_pid} ->
        IO.puts("Pong received ping")
        send ping_pid, :pong
        pong()
    end
  end

  def start do
    pong_pid = spawn(fn -> pong end)
    spawn(fn -> ping(3, pong_pid) end)
  end
end

Because of the power of pattern matching in the function argument signatures, you can define 2 separated functions, avoiding the “if” as in the Clojure example. Of course, because it’s Clojure and it has a complete macro system, and because its core.match has the proper pattern matching mechanisms, you can emulate the same thing through external libraries such as defun though.

This was just a simple exercise, I hope it shed some light on the basic similarities between these 3 languages. And as “ugly” as Erlang may feel, I still feel more comfortable with its quirks than nested parenthesis.

This is not my first time touching Erlang though. I was fortunate enough to be able to participate in a small workshop at QCon San Francisco 2009 with no other than Francesco Cesarini. Thanks to him I was able to understand some of Erlang’s exquisite syntax, the correct concept of Erlang processes, how immutability and pattern matching govern their programming flow. It was very enlightening. Unfortunately I couldn’t see myself doing Erlang full time. I just hoped those mechanisms were available in a language such as Ruby …

Between 2007 and 2009 Erlang had a renaissance among language aficionados because of the “Programming Erlang” book released by The Pragmatic Programmers, written by no other than Joe Armstrong himself, Erlang’s creator. Dave Thomas tried to push Erlang a lot in 2007, but even he wasn’t able to sell Erlang’s powerful engine because of the strange presentation of the syntax.

After 2009, José Valim had a long run to release the controversial Rails 3.0 big rewrite (which was fortunately a success) and he decided to step aside and try something else. His own research led him to Erlang for the reasons I mentioned above, but he decided that he could solve the “quirky syntax” problem. You can see some of his very first talks about Elixir in the Rubyconf Brasil 2012 and 2013 recordings. The very early beta was released in 2012 and he finally released the stable 1.0 in 2015. Chris McCord was able to release Phoenix stable soon after.

When I first heard about this, Elixir found its place under my radar. I didn’t jump right in though, between 2009 and 2015 we had a surge in “functional programming” interest because of the JavaScript renaissance, the release of Scala, Go Lang, Clojure, the promise of Rust, and so on. So I waited, carefully following every one of them.

Then 2014 came and suddenly everybody else found out about Erlang, with its spartan infrastructure that enabled WhatsApp to serve half a billion users with absurdly low costs and made Facebook buy them for a hefty USD 19 billion! We all knew about the WhatsApp case since at least 2011, but it was not until 2014 that everybody noticed. But not even this was able to steer Erlang to the forefront just yet.

When Elixir stable was released this year, followed by Phoenix stable, I knew it was time for me to start investing some quality time with it. Erlang’s core sells itself: everybody else is doing concurrency by means of immutability and lightweight threads (or green threads, NxM strategy between green threads and real threads). It is actually quite trivial to max out the machine just by spawning millions of light processes nowadays. What makes it difficult is to create a system that has the potential to actually achieve 99.9999999% reliability. Spawning processes is easy, how do you coordinate them in the same machine? How do you coordinate them between different machines? How do you update a living system without bringing it down? How do you handle failures? How do you supervise everything?

These are the questions that Erlang solved decades ago (20 years ago) with the now famous OTP, Ericsson’s Open Telecom Platform. Something that was created to meet the performance and reliability needs of telecommunications at large scale. When we say it like this it feels like it will be a royal pain to learn, something akin to the JEE (Java Enterprise Edition), but worse.

And I can tell you that learning enough OTP to be productive is actually very easy (you won’t achieve the legendary 99.9999999% reliability out of the blue, but you’ll be able to build something reliable enough). Think of it as a collection of half a dozen modules, with a couple of function interfaces to implement, a few words of configuration and you’re basically done. It’s so easy and lightweight that in fact many small libraries are written with OTP in mind and it’s easy to just “plug and play”. It’s not a heavyweight server-side only thing.

To harness that power you will need to learn Elixir, unofficially, a language that has an uncanny resemblance to Ruby, built to spill out Erlang bytecode for its BEAM virtual machine. You can’t find a better combination.

One Week

Having said all that, let’s cut to the chase. You definitely want to get acquainted with the Functional Programming concepts such as immutability, higher order functions, pattern matching. I made a list of links for those concepts in my previous post, I recommend you read it.

Assuming you’re already a programmer in a dynamic language (Ruby, Python, JavaScript, etc.) and you want the fast crash course, start by buying “Programming Elixir” by Dave Thomas and actually do each piece of code and the exercises in order. It’s a book so easy to read that you will be able to finish it in less than a week. I did it in 3 days. The Elixir-Lang official website has very good documentation as well and they link many good books you want to read later.

Then subscribe to Josh Adams’ Elixir Sips. If you’re a Rubyist, it’s like watching Ryan Bates’ Railscasts from the beginning all over again. Although it’s more akin to Avdi Grimm’s RubyTapas show, with very short episodes just for you to have your weekly fix on Elixir.

You can watch some of the episodes for free in low resolution, but I highly recommend you subscribe and watch the HD versions. It’s well worth it. There are more than 200 episodes. I’ve watched more than 130 in 12 hours :-) So I figure I’d take another 2 days to watch everything.

You should definitely watch everything if you can, but if you can’t, let me list the ones I think are the essentials. First of all, keep in mind that Josh has been doing this for quite a while, when he started Elixir was version 0.13 or below and Erlang was version 17 or below.

For example, episode 171 - Erlang 18 and time highlights the new Time API. You must know about this. Episode 056 - Migrating Records to Maps shows a new feature in Erlang 17 and Elixir where it makes Maps more preferable than the previous Records. Maps are explained in episodes 054 and 055. If you learn the Phoenix web framework, it uses the ORM Ecto underneath and Ecto models are Maps, so you must know this.

It means that the first 180 episodes, at least, are using previous versions of Erlang, Elixir, Phoenix, etc and you must keep in mind that new versions will have different APIs. This was one of the reasons I waited for the stable releases, because it was only natural that projects evolve and take time to reach stable APIs, and chasing several moving targets is really difficult for the uninitiated.

Having said that, watch this list first:

• 001 - Introduction and Installing Elixir.mp4
• 002 - Basic Elixir.mp4
• 003 - Pattern Matching.mp4
• 004 - Functions.mp4
• 005 - Mix and Modules.mp4
• 006 - Unit Testing.mp4
• 010 - List Comprehensions.mp4
• 011 - Records.mp4
• 012 - Processes.mp4
• 013 - Processes, Part 2.mp4
• 014 - OTP Part 1_ Servers.mp4
• 015 - OTP Part 2_ Finite State Machines.mp4
• 016 - Pipe Operator.mp4
• 017 - Enum, Part 1.mp4
• 018 - Enum, Part 2.mp4
• 019 - Enum, Part 3.mp4
• 020 - OTP, Part 3 - GenEvent.mp4
• 022 - OTP, Part 4_ Supervisors.mp4
• 023 - OTP, Part 5_ Supervisors and Persistent State.mp4
• 024 - Ecto, Part 1.mp4
• 025 - Ecto, Part 2_ Dwitter.mp4
• 026 - Dict, Part 1.mp4
• 027 - Dict, Part 2.mp4
• 028 - Parsing XML.mp4
• 031 - TCP Servers.mp4
• 032 - Command Line Scripts.mp4
• 033 - Pry.mp4
• 041 - File, Part 1.mp4
• 042 - File, Part 2.mp4
• 044 - Distribution
• 045 - Distribution, Part 2
• 054 - Maps, Part 1.mp4
• 055 - Maps, Part 2_ Structs.mp4
• 056 - Migrating Records To Maps.mp4
• 059 - Custom Mix Tasks.mp4
• 060 - New Style Comprehensions.mp4
• 061 - Plug.mp4
• 063 - Tracing.mp4
• 065 - SSH.mp4
• 066 - Plug.Static.mp4
• 067 - Deploying to Heroku.mp4
• 068 - Port.mp4
• 069 - Observer.mp4
• 070 - Hex.mp4
• 073 - Process Dictionaries.mp4
• 074 - ETS.mp4
• 075 - DETS.mp4
• 076 - Streams.mp4
• 077 - Exceptions and Errors.mp4
• 078 - Agents.mp4
• 079 - Tasks.mp4
• 081 - EEx.mp4
• 082 - Protocols.mp4
• 083 - pg2.mp4
• 086 - put_in and get_in.mp4
• 090 - Websockets Terminal.mp4
• 091 - Test Coverage.mp4
• 106 - Text Parsing.mp4
• 109 - Socket.mp4
• 112 - Benchfella.mp4
• 113 - Monitoring Network Traffic.mp4
• 124 - Typespecs.mp4
• 125 - Dialyzer.mp4
• 126 - Piping Into Elixir.mp4
• 127 - SSH Client Commands.mp4
• 131 - ExProf.mp4
• 132 - Randomness in the Erlang VM.mp4
• 135 - Benchwarmer.mp4
• 138 - Monitors and Links.mp4
• 139 - hexdocs.mp4
• 141 - Set.mp4
• 142 - escript.mp4
• 144 - Erlang’s calendar module.mp4
• 145 - good_times.mp4
• 153 - Phoenix APIs and CORS.mp4
• 155 - OAuth2_ Code Spelunking.mp4
• 156 - Interacting with Amazon’s APIs with erlcloud.mp4
• 157 - Playing with the Code Module Part 1 - eval_string.mp4
• 159 - Simple One for One Supervisors.mp4
• 160 - MultiDef.mp4
• 171 - Erlang 18 and Time.mp4
• 172 - Arc File Uploads.mp4
• 174 - ElixirFriends_ Saving Tweets with Streams and Filters.mp4
• 175 - Pagination with Ecto and Phoenix using Scrivener.mp4
• 176 - Prettying Up ElixirFriends.mp4
• 178 - Memory Leaks.mp4
• 179 - Rules Engine.mp4
• 180 - Collectable.mp4
• 182 - Phoenix API.mp4
• 183 - React with Phoenix.mp4
• 184 - React with Phoenix Channels.mp4
• 185 - Mix Archives.mp4
• 186 - Automatically Connecting Nodes.mp4
• 187 - Compiling a Custom AST Into Elixir Functions.mp4
• 190 - Testing Phoenix Channels.mp4
• 193 - Linting with Dogma.mp4
• 194 - Interoperability_ Ports.mp4
• 196 - Crashing the BEAM.mp4
• 200 - Custom Types in Ecto.mp4
• 201 - Tracing and Debugging with erlyberly.mp4
• 202 - Exception Monitoring with Honeybadger.io.mp4
• 203 - plug_auth.mp4
• 204 - Behaviours.mp4

This is roughly half of what’s available in Elixir Sips. All the other episodes are also interesting, but if you’re just getting started, this list should be enough to get your feet wet in the language.

Railers will enjoy Phoenix and the ecosystem that is growing around it. You can already authenticate through OAuth2, do will_paginate-style pagination with Scrivener, do CarrierWave-style file uploads with Arc, deploy to Heroku.

For more exercises, you can easily connect to HTTP endpoints using HTTPoison, parse HTML with Floki, parse JSON with Poison. For more libraries, you can follow this Github page called Awesome Elixir which lists many new Elixir packages that you can use. But make sure you walk yourself through the basic concepts first. Elixir has a Rake-like task management system with built-in Mix, you can add dependencies in a Gemfile-like file called Mix.exs, which every project has. You can add dependencies through Github urls or from Hex.pm which is like Rubygems.org.

In this learning process, the concepts that I find more important to learn first are:

• Elixir basic syntax and concepts (pattern matching, loops through recursion, immutable state, primitive types including Maps, the pipe operator)
• The concept of processes, nodes, and intercommunication between processes and nodes, including Monitors and Links.
• OTP Basics, learn what GenServer, GenEvent, GenFSM are.

After you learn that, you can figure out how to build OTP applications and do something practical for the Web using Phoenix. In particular, you will want to learn everything about Phoenix’s Channels, the infrastructure for robust, fast and highly concurrent WebSockets.

That’s it. This is my first week learning Elixir, and my next step is to train myself by doing more exercises and also learning more about Phoenix. Even though Phoenix is inspired by Rails, it is not a clone, it has its own set of unique concepts to learn and this is definitely going to be a very interesting ride.

If you have more tips and tricks for beginners, feel free to comment below.

Below, I’ll transcribe the professor’s questions and my answers.

Professor: My name is Rosenclever, I’m a university professor currently teaching Object-Oriented Analysis and Design (basically UML, Design Patterns, and Agile methodologies) and Object-Oriented Programming 2 (Java for Web with JPA and JSF)

Well, since you’re a developer who today also plays the role of an entrepreneur, and I, on the other hand, am in academia trying to prepare professionals to work in the market, I always try to update my content with market demand, and it’s very common to hear from IT managers, through the media, that there’s a very big gap between what’s taught in academia and what the market needs…

So I’d like to ask for your contribution so that I can help my students leave better prepared for the market. I’d like to know your opinion on the syllabuses of the courses I mentioned at the beginning of this email…

Professor: 1) Does it make sense today to master UML diagrams?

AkitaOnRails: “Master” perhaps not (in the sense of knowing every little detail of every diagram). But knowing they exist, having a general notion of each, and knowing the most useful ones like Use Cases, Sequence Diagram, State Diagram, Class Diagram, I think is appropriate and useful. And never as a way to design ALL the software, but a few parts that maybe become clearer if you diagram first.

Adding here in the post: plenty of people who never programmed professionally think that “any and every” type of diagramming and planning is unnecessary. They still imagine (without knowing it) that the so-called “corporate” world runs on mountains of diagrams and plans. That’s almost never true.

Of course there are exceptions, but in the common case a programmer needs to communicate an idea through something beyond code. That matters most when explaining a complex architecture to the team before starting to code.

It doesn’t mean diagramming 100% of the classes, 100% of the states, 100% of the sequences. It means diagramming what’s most critical and hardest to understand, and letting the rest emerge naturally during programming.

Professor: 2) Which Design Patterns and Agile concepts do you consider fundamental to be taught in academia? Or is that also not considered important?

AkitaOnRails: Design Patterns are important, but as a reference. They’re possible solutions, and they need to be evaluated case by case. In Agile, only worry about Extreme Programming, because all XP techniques matter. Scrum and Kanban can be introduced, but they’re dispensable. About patterns:

• Brazilian Portuguese Can Confuse Us: Standard vs Pattern
• GoF Design Patterns - Did it survive the test of time?
• Design Patterns represent defects in Languages

Adding here in the post: the important thing is to teach that nobody needs to invent everything from scratch all the time. That would be a redundant effort, since someone has probably already solved the problem before.

At the same time, it’s worth explaining that what we know today is just what seems to work best, and none of it is final. If someone has a result that beats what we know, great, show it to everyone.

It became fashionable to talk about “Inversion of Control,” and few people know what it really means. They only know it seems to improve “modularization,” also without understanding why modularizing is a benefit in some cases and not so much in others.

And speaking of Agile, I recommend reading these posts of mine:

• Lean is Dead, Long Live Efficiency
• Agile: the Truth Behind the Method

Professor: 3) Does the OO Analysis and Design discipline still make sense today? What would be interesting to be addressed in it?

AkitaOnRails: Of course, but today you have to emphasize that OO (object orientation) is one approach among several, and not always the “best.”

By the way, OO only gets interesting in academia through the eyes of languages that really explore OO, like Smalltalk and Ruby. It’s worth showing the whole landscape: class-oriented programming in Java and C#, functional in Lisp and Scheme, prototype-oriented in JavaScript and Io, and programming with concurrency and actors in Go, Scala, and Elixir/Erlang.

Adding to the post: this is the kind of subject with no “right” answer. Any attempt to measure the strength of one side or the other always turns into a flamewar or a mediocre bikeshedding.

The reality is that software factories will keep using languages with good commercial tooling support, like IBM Websphere and Microsoft Visual Studio.NET. Tech startups and smaller, more “best of breed” technology-oriented companies will prefer open source solutions that sometimes look like a “patchwork quilt.”

Agencies and small producers will keep using whatever delivers in the shortest possible time, even if it’s dirty. Advertising campaigns last very little, so you’re left with the “quick and dirty,” the WordPress derivatives, and so on.

Professor: 4) Is it definitely still worth teaching Java in academia? Do you think Ruby and Rails or even Python and Django are more appropriate?

AkitaOnRails: Yes, it’s worth it, as long as Java comes in as what it really is: the commercially most viable language. It isn’t the only solution to everything nor the best example of an OO language.

Ruby and Python can be explored as dynamically typed languages, taking the chance to explain the difference between static and dynamic typing. But in academia, especially in Computer Science, I’ve always been in favor of teaching dead languages like Smalltalk, Lisp, and Eiffel. That keeps the student from falling into the temptation of staying only with the language they learned in college. A bachelor’s degree should prioritize foundation, and commercial use is the job of technical and technologist schools.

Adding to the post: Academia, especially in Computer Science chairs, must emphasize science. Going 100% toward the market creates a generation that becomes obsolete very quickly and, worse, that never learns to update itself on its own.

In a hypothetical scenario, if 100% of universities turned 100% toward what the “market” wants, in ten years we’d have our entire computing field junked. Universities need to elevate the “Science” of Computer Science.

Let me leave posts I wrote about university and career topics:

• Should I go to college?
• Career in Programming - Coding isn’t Programming
• Letter to a Young Programmer Considering a Startup

Professor: Sorry for the long email, but unfortunately there was no opportunity for us to talk at the event…

AkitaOnRails: Not at all, if the subject is relevant, it’s worth discussing it. And I encourage everyone who attends events I’m at to call me if they want to discuss ways we can help improve education. This subject will never be irrelevant.

You can take some initial knowledge, say web and HTML, throw in a few frameworks and libraries (Rails, Django, WordPress), and actually put something functional online. But that’s little. Very little.

The fact that it’s so simple, bordering on trivial, to put “something functional” online hides all the potential in a programming career. Worse, the immediacy of the result creates the illusion that this alone is enough and that any more advanced knowledge is unnecessary.

The result shows up in the growth of technical programming courses and the low demand for Computer Science bachelor’s degrees. The fewer people we have working on the fundamentals, at the origin of everything, the more the market decays from here on.

You have a coder job today: Photoshop cutter, HTML assembler, WordPress plugin-sticker. Stop and think about how your career is going to evolve from here. Are you always going to be just an assembler? Are you going to take the easy path and become a half-baked “manager” of assemblers?

In a single post it’s impossible to list and explain every important aspect of Computer Science. But I want to try naming at least some of the subjects most people think are unnecessary, just to provoke you into digging further.

Learn to at least “READ” English fluently

I love the Portuguese language, and it will always be my first language. But pseudo-patriotism and laziness don’t change one fact: if you don’t commit right now to reading English fluently, you’ll live behind, at a sorry level.

The first reason is that in the Western world, everything new comes out in English first. (I’m leaving out the East because I don’t read Chinese, and the market there is enormous too.) Waiting for someone to bother translating it costs work and time.

If you rely on the Portuguese version, you’ll be reading stale material. And by the time the subject finally grabs you, it may already be obsolete. At that point the gap stops being one step and turns into hundreds of kilometers.

The second reason is that the internet is globalized. It’s common to pick up code written in other countries and carry it forward, and it’s common for the code you write to be shared with programmers abroad. Do yourself a favor: don’t embarrass yourself.

Religious Wars

Programming is science. It has no horoscope, no guru, no soccer team, no numerology, no astrology. Remember the name: Computer SCIENCE. Nobody calls it Computer Astrology.

Everyone has subjective tastes. Turning taste into dogma is stupid. The laziness of having to learn something new produces the famous “I’m good at Clipper, this language will evolve in the future and will always exist, I just have to defend it tooth and nail no matter what comes along later.”

Swap “Clipper” for anything else: Cobol, Basic, Pascal, Algol, Eiffel, Smalltalk, all the way to the newest ones like Java, C#, JavaScript, Python. Something existing today guarantees nothing about tomorrow. Think about how ridiculous whoever said that line must feel now.

Don’t play that role. In Science we’re not loyal to “teams”: we root for whoever is winning and switch the moment they’re proven wrong. That’s why Science always evolves.

And surprises happen. In the 80s and 90s nobody gave Objective-C a second look; it seemed doomed to failure. Then, in 2007, the iPhone shows up and, surprise, you needed to know Objective-C. Suddenly it became one of the most successful languages of the end of the first decade of the 21st century. Every ten years the market transforms in some way. It’s Bell’s Law.

Accept It: Your Solutions Today Are Bad

Back to the theme of “Ignore the Religious Wars”: how do you figure out what to do? Stick to the principles. Throw out what the gurus say, don’t idolize anyone, don’t blindly follow anyone. Go down to the fundamental questions and the more obvious direction shows up.

When you know how things work, strip the varnish, pop the hood, take the engine apart, understand the chemistry of combustion, only then do you master the craft. If you barely remember what the hell changing the oil is for, you’ll always be a mediocre driver, at best. What’s your goal? If it’s to be the car’s engineer, listening to gurus talk about the color of the steering wheel won’t take you far.

Computer Science usually gets ignored because learning Math looks both boring and useless.

Let’s run through a few examples. If I ask a beginner how to search for words inside a text, the most obvious things that should come to mind are:

• use substring functions and a loop to scan the text (brute-force solution)
• use a regular expression, or in a database use a “LIKE” (generic solution)
• install a SOLR or Elasticsearch (correct solution in many cases, but “black magic” in terms of understanding)

Most people wouldn’t even think of the third solution. And whoever does doesn’t know why. What if I told you that, in an absolutely raw and boiled-down way, the solution is to turn the document and the search terms into vectors and compute the relevance between them by cosine similarity? That’s exactly what the Vector Space Model (VSM) means, which you’ll find in several search engines.

The knowledge behind that is called Linear Algebra. Remember it, from the FIRST year of Computer Science? On this topic I recommend a talk I gave, “How not to do searches using LIKE”.

And when you need to build a process to filter inappropriate content? Most people would think like this:

• create a blacklist of words and keep adding to it as offensive words come to mind. And use the first or second option I listed before to search for those words.

As you can already imagine, the most “obvious” or “simple” answer (given the lack of knowledge) is usually the wrong one, and here it also isn’t the most efficient.

Get to know machine learning and one of its simplest forms, the Naive Bayes Classifier. Someone might think “damn, but this machine learning stuff is way too advanced for the day-to-day.” Not at all. In fact, any cheap anti-spam filter you find on a free-downloads site uses a classifier.

The idea is that isolated words aren’t enough to tell whether content is inappropriate. The way sentences are built, the “tone” of the writing, all of that forms a pattern that can be classified and learned. The more inappropriate content gets classified, the more efficient the algorithm becomes.

If you want to try a simple version in Ruby, see this blog. If you want to learn about more advanced classifiers, see the Apache Mahout project.

And what is that? It’s a subject of Statistics and Probability. The fundamentals for understanding it are, again, in the FIRST year of Computer Science.

“Oh, but you’re talking about things nobody needs to know. To make websites this is unnecessary.”

A good web framework needs to map routes to the underlying programming (the controllers) efficiently. Ruby on Rails has a routing component called Journey, which we configure through the “config/routes.rb” file. Below is an excerpt:

ImageUploadDemo::Application.routes.draw do
  devise_for :admin_users, ActiveAdmin::Devise.config

  resources :photos

  authenticated :user do
    root :to => 'photos#index'
  end
  root :to => "photos#index"
  devise_for :users

  require 'sidekiq/web'
  mount Sidekiq::Web => '/sidekiq'
  ActiveAdmin.routes(self)
end

How hard can this be? What might most people imagine?

• Easy, just make a set of arrays or a hash (dictionary), and when the URL arrives, break the terms with a regular expression and find the controller, passing the parameters to run it.

Let me repeat myself: yes, this works, for very small applications. Anything much bigger than the example above already runs into performance problems.

How about a look at a snippet of the Journey code?

class Journey::Parser

token SLASH LITERAL SYMBOL LPAREN RPAREN DOT STAR OR

rule
  expressions
    : expressions expression  { result = Cat.new(val.first, val.last) }
    | expression              { result = val.first }
    | or
    ;
  expression
    : terminal
    | group
    | star
    ;
  group
    : LPAREN expressions RPAREN { result = Group.new(val[1]) }
    ;
  or
    : expressions OR expression { result = Or.new([val.first, val.last]) }
    ;
  star
    : STAR       { result = Star.new(Symbol.new(val.last)) }
    ;
  terminal
    : symbol
    | literal
    | slash
    | dot
    ;
  slash
    : SLASH              { result = Slash.new('/') }
    ;
  symbol
    : SYMBOL             { result = Symbol.new(val.first) }
    ;
  literal
    : LITERAL            { result = Literal.new(val.first) }
  dot
    : DOT                { result = Dot.new(val.first) }
    ;

end

In plain English, now you’re screwed. Part of Journey uses Racc, a parser generator.

If you made it at least through the SECOND year of Computer Science, then you went through Algorithms and Data Structures and Assemblers. And if you reached the THIRD year, you learned about Compilers (and saw the famous Dragon book). Racc should have reminded you of Yacc, Flex, Bison.

To wrap up, you probably think you know what Object Orientation is, right? I bet you consider your favorite language (whether Java, C#, or JavaScript) object-oriented. If I ask you to define what that means, it’s on the tip of your tongue:

• My language supports Encapsulation, Inheritance, and Polymorphism, therefore it’s object-oriented.

And what if I told you that procedural, imperative, and functional languages also support encapsulation, inheritance (via delegation or not), and polymorphism? And if that’s the case, then what defines object orientation?

Whoever got into the subject might remember Alan Kay, who coined the term “object orientation.” But how many stopped to research the Simula 67 language? What did Simula introduce in 1967? Objects, classes, inheritance, subclasses, virtual methods, coroutines, discrete event simulation, garbage collection.

And how many have heard of the creators of Simula 67, Kristen Nygaard and Ole-Johan Dahl?

You don’t know who they are? I’m sure you don’t. Well, let me introduce you to the fathers of object orientation.

On the Shoulders of Giants

What I mentioned in the previous section is a drop of water on the tip of the iceberg. What matters is for you to realize that everything you think you know is close to nothing. I want you to accept that everything you think you know is either wrong or totally incomplete.

That matters because whoever thinks they already know everything, or close to it, will never learn anything. You need to empty the cup to fill it. (by Bruce Lee)

Beyond that, I want you to understand you’re not alone. Before you came dozens of great minds. And I’m not talking about those superficial gurus who merely teach techniques and tools. Forget them, everyone will forget them soon too.

Everyone knows who Linus Torvalds, Bill Gates, Steve Jobs, Zuckerberg are. Forget them for now. Stick to the immortals, the names who really made a difference in the history of Computer Science. A few examples:

• Charles Babbage
• Ada Lovelace
• George Boole
• Alan Turing
• Alonzo Church
• John von Neumann
• John McCarthy
• Niklaus Wirth
• Bertrand Meyer
• Dan Ingalls
• Donald Knuth
• Edsger W. Dijkstra

And that’s just a sample. Science is a broad field, where one scientist’s work complements the previous one’s. Step by step, always moving forward. It’s a cumulative work of tens, hundreds of years. Our advantage? Hundreds of people already walked this path in the past, and we can use what they learned and left us, instead of making the same mistakes ourselves until we learn on our own.

Almost everything you see out there called “innovation” is the rediscovery of things already documented in the past, but that were ahead of their time. It was like that with the mouse: Douglas Engelbart, who created it in 1968, had to wait sixteen years, until Steve Jobs launched the Macintosh, to see his invention go mainstream. Stop and think: how many discoveries are sitting in the past, just waiting for someone to dig them up and finally make them real?

Want to know about the immortal books of Computer Science? Here are a few:

• Structure and Interpretation of Computer Programs (SICP)
• The C Programming Language (K&R)
• Algorithms + Data Structures = Programs
• Compilers: Principles, Techniques and Tools, the Dragon book
• Modern Operating Systems
• Computer Networks
• The Art of Computer Programming, which even I haven’t read, and I don’t know anyone who has read and understood it, but I’ll leave it here because of its importance.

Don’t want to go to college? No problem, try to follow the material MIT makes available online on Electrical Engineering and Computer Science. If you’re still a complete amateur even in the programming world, see this introductory Computer Science course on Coursera.

Note that I’m not citing in this article any of the names many might expect, like Martin Fowler, Bruce Eckel, Robert Martin, Michael Feathers, Kent Beck, Steve McConnell, Tom DeMarco, Dave Thomas. Forget them for now. Without the earlier ones, these make little difference.

Don’t put the cart before the horse. Go one step at a time.

Career in Programming

One thing I always repeat is this:

"Programming isn't writing any code, the same way cooking isn't throwing any ingredient into a pan."

Understand: writing code is very simple. Anyone with the minimum motor coordination not to try shoving a square peg through a round hole, or who has at least stacked one Lego brick on top of another, is capable of writing code. There’s no merit in it at all.

Downloading a Twitter Bootstrap, using a Yeoman generator, installing MySQL on Ubuntu, copying and pasting a jQuery snippet: anyone can do it.

In career terms, what “anyone” can do is a mere “commodity”. Being a commodity means the value the market is willing to pay will only drop. Every so often some novelty shows up trying to create a differentiation (“look, Angular JS,” “look, HTML 5”), but it quickly dissolves into the downward trend in value.

Value lives in the creativity of the solution: extracting the greatest result for the lowest cost. And creativity only exists when you master all the elements around you. When one hour of work swaps an idiotic word-search algorithm for a vector space model, cuts your fleet from ten machines to two, and answers your user in a fifth of the time, that’s applied knowledge. And knowledge like that has value, and it grows.

A full toolbox makes you a “jack-of-all-trades with many tools.” The engineer or architect capable of building the next World Trade Center or Freedom Tower is a different thing.

But, like everyone who wants to grow, we all start as jacks-of-all-trades. There’s nothing wrong with that. Just don’t fool yourself into thinking that a box with more tools turns you into anything different from that.

As incredible as it sounds, there is no good, simple, clear, short definition for what it means to be a “manager.” Manager isn’t a profession. You can’t “learn” to be a manager in a school or a course. The only way to learn is by trying, erring, and improving, continuously and indefinitely.

One of the best definitions I’ve found is from no one less than Professor Henry Mintzberg, in his recent book “Managing,” published in 2009. A mature and light read.

I recommend the read. If you’re a beginner manager, or want to be one, maybe the book gives you some sense of what’s coming. If you’re already a manager, you’ll be surprised. So many theory books paint scenarios you’ve never seen in practice, and they leave you second-guessing whether you’re on the right track. This one does a better job of naming the real problems you deal with day to day.

I want to bring a few excerpts from the first half of the book.

Management Isn’t a Profession

After years of seeking Holy Grails, it is time to recognize that managing is neither a science nor a profession; it is a practice, learned primarily through experience, and rooted in context.

It is certainly not a Science. Science has to do with developing systematic knowledge through research. That’s nowhere near the purpose of management, which has to do with helping get things done in organizations. Management isn’t even an applied science, because that’s still science. Management certainly applies science: managers need to use all the knowledge they can. And they certainly use analysis, rooted in the scientific method (which means here more scientific proof than scientific discovery).

But effective management is more dependent on art and is especially rooted in craft. Art produces “insight” and “vision,” based on intuition (Peter Drucker wrote in 1954 that “the days of the ‘intuitive’ manager are numbered.” Half a century later, we’re still counting.) And craft has to do with learning from experience — working things out as the manager moves forward.

So it’s worth repeating what Mintzberg often says: it’s impossible to form managers in school. That much is a certainty. In school you learn accounting techniques, marketing, repeatable technical knowledge. None of that makes a manager, much less a leader.

When a school or an MBA course slaps “We Form Leaders” on the brochure, it’s lying. Those schools form technicians. Those technicians go work in technical activities until they grow, take on responsibilities, and maybe, one day, actually manage something.

Management Is a Practice

Put together a good portion of craft with the right touch of art along with some use of science and you end up with a job that is, above all, a practice. There is no “one best way” to manage; everything depends on the situation.

It isn’t even a Profession. It has been pointed out that engineering also isn’t a science or an applied science so much as a practice in its own right. But engineering applies a good portion of science, codified and certified as to its efficiency. And therefore it can be called a profession, meaning it can be taught before practice, out of context. In a way, a bridge is a bridge, or at least iron is iron, even if its use has to be adapted to the situation at hand. The same can be said of medicine. But not of management:

Many medical diagnostic, inference, and treatment skills … assume that a disease can be decomposed into separate problems that don’t differ much between patients and can be treated by reasonably standard remedies … In contrast, much of management work involves dealing with problems that are highly interdependent with other parts of the organization, are specific to this particular firm, market and industry and not easily reducible to a standard general syndrome that can be treated by specific techniques.

Managing is a work of negotiation, of trades, of yielding, of conquering. There’s no procedure, no methodology, no rigid body of knowledge that defines this role. You can’t know “what level” a manager is at because there’s no “standard” for comparison. There are good managers and bad managers, just that. “Bad managers,” in reality, couldn’t even be called “managers” because they’re actually not managing.

Real Management

It has been said that an expert is someone who knows more and more about less and less until finally he or she knows everything about nothing. The manager’s problem is the opposite: to know less and less about more and more until finally he or she knows nothing about everything.

Many people think that taking a pile of courses, postgrad, MBAs, and the like will necessarily turn them into good managers. That correlation doesn’t exist. Yes, there are excellent managers with good training, but what really formed them was practice. Correlation isn’t causation. In the general case, this person is just postponing the thing that would actually give them a shot at becoming a good manager: practice.

Managers who don’t do and deal, and so don’t know what is going on, can become incapable of coming up with sensible decisions and robust strategies.

There’s also the idea that managers shouldn’t get their hands dirty, since it’s either they manage or they execute. That doesn’t mean the manager can be ignorant about the area where they work. Quite the opposite: a manager with excellent technical knowledge of their area has a better shot at understanding the context.

Thinking is heavy — thinking too much can exhaust a manager — while acting is light — acting too much can make the manager unable to stop.

Similarly, too much leadership can result in a job without context — without aim, without focus, without action — while too much relationship can produce a job disconnected from its internal roots — public relations instead of tangible connections. The manager who only communicates never finishes anything, while the manager who only “does” ends up doing everything alone. And the manager who only controls risks ends up controlling an empty shell of yes-men and yes-women. We don’t need people-oriented, information-oriented, or action-oriented managers; we need managers who operate on these three planes. Only together, across all three planes, do these roles provide the balance that is essential to the practice of management.

Mintzberg points out something obvious: if you want to be a renowned management author, writing books and giving talks, you have to focus on a single aspect and exalt it as the “missing piece.” But in reality, a manager who focuses on only one aspect of the practice is certainly doing the wrong thing. A good manager balances the different roles as the situation demands, and uses all of them whenever needed.

Anyway, as you can see, the role of the Manager is much broader than it seems. Just studying procedures doesn’t make you a manager, just as studying a color table doesn’t make you a painter, or knowing how to read a score doesn’t make you a composer.

Art, Craft, Technique, and Practice. This is the definition of Management. A manager who doesn’t understand that will always be a terrible manager.

As an addendum, one word of caution. Sometimes we see very active professionals, and not only managers, who seem to do a lot because they talk to a lot of people, act a lot, and are always running hot. That’s usually the symptom of the “hero-professional.” In my definition, it’s precisely the person who is doing the job the wrong way. They look productive, but in practice they’re just being inefficient and, worse, cleaning up the mistakes they generate along the way. It looks like they’re solving plenty when, in fact, they’re just chasing their own tail.

After so much time talking about methodologies, processes, and procedures, there’s one thing that should already be obvious but that most companies ignore: methodologies will never replace good professionals.

If I say this to anyone, everyone will say: “but that’s obvious.” And my urge is to retort: “then why the hell are you trying to implement this methodology?”

First of all, let’s go through some definitions:

• Methodology: a system of methods used in a particular area of study or activity.
• Processes: a series of actions or steps taken to reach a particular goal.
• Procedure: an established or official way of doing something.

Notice that in software development we always talk about processes or methodologies, never procedures. That’s because there’s no established or official way of making software. The problem is that most people confuse processes and methodologies with procedures and try to apply them as if they were the “official” way of making software. And they aren’t.

That’s exactly why disputes like Agile vs PMI make no sense. PMI calls its set of processes PMBOK, where BOK means Body of Knowledge precisely because it isn’t a procedure. It’s just a body of knowledge that may or may not be useful depending on your sector of activity. Agile is no different: a set of methodologies and techniques that may or may not be useful in your company. None of them, and none of the others, are procedures. Understand this first.

Once you understand that, comes the second mistake. People who don’t practice software development notice that the software being produced is of low quality. The indicators are obvious: customer complaints, internal complaints, the sense of excessive delay in completing any software work, and so on.

Seeing this, the first conclusion they reach is: “we’re missing procedures.” And the search for the missing link begins. They find various processes and methodologies and try to institutionalize them. For a short period it seems the result is what they expected, but it doesn’t take long to see that things haven’t changed all that much. Then they blame the methodologies for “not working.”

Software development is an activity of “practice.” And watch out: “practicing” and “being practical” aren’t the same thing. So one more definition:

Practice: 1. the actual application or use of an idea, belief, or method as opposed to theories about its application or use. 2. repeated exercise in or performance of an activity or skill so as to acquire or maintain proficiency in it. (Read a more refined definition on Wikipedia.)

Repeat with me: if you don’t practice software, you don’t understand software. It’s like saying you were good at soccer 10 years ago, but you don’t practice anymore and still want to believe you’re good. If you haven’t practiced for 10 years, you’ve been bad for 10 years, and there’s nothing to discuss.

In software, not practicing continuously, daily, deliberately, for 5 years is the same as a civil engineer saying he hasn’t practiced engineering in 50 years. 5 years in software age is an eternity.

And if you don’t practice software, you have nothing to say about it, by definition. So here’s the tip: processes and methodologies won’t help you, precisely because they aren’t procedures.

If software had a procedure, it wouldn’t be a practice. You’d just mechanically follow the series of steps and, at the end, you’d have quality software. More than that: if it were a procedure, it would already be possible to have software that generates software on its own, with no human help. But that doesn’t exist, and never will.

If a restaurant has bad food, processes won’t help it; changing the cook will. If your orchestra plays badly, processes won’t help it; changing the musicians will. If your soccer team plays badly, processes won’t help it; changing the players will.

Replacing them is perhaps drastic. Before that, I’d say “practicing” and “training” will certainly help. But here’s the thing: if your soccer player believes he only needs to play after he punches in his time card, and can stop when he punches out, he isn’t a good player. So forcing them to train won’t help and, consequently, processes and methodologies definitely won’t help you.

With a good programmer it’s the same. You don’t need to force him to put his code into a versioned repository: he himself will miss it and do something about it. You don’t need to force him to test his code: he himself will miss it and do something about it. You don’t need to force him to refactor: he himself will feel that it’s bad and do something about it.

A good practitioner doesn’t need others to tell them what to do. They know what constitutes good work and will perform accordingly, or they will look for what they’re missing and train until they become proficient at it.

A good practitioner lives by the idea that they have to be better today than they were yesterday, and keep that pace every day. A procedure professional knows that today’s work is the same as yesterday’s.

That’s the difference. If your company manufactures screws, it follows procedures. Today there are people assembling those screws, following procedures; tomorrow you’ll probably replace them with machines that will do the same job better and faster. That’s the destiny of every procedure professional.

If your company is in gastronomy, music, sports, literature, art, or software, you want practitioners. If you don’t have them, remember: processes and methodologies aren’t procedures. Processes and methodologies can’t be installed as procedures. Processes and methodologies will never transform people with a procedure mindset into practitioners. And without practitioners, your result will always be bad.

We’ve been developing software for over 70 years. If the solution were that simple, don’t you think everyone would have done it that way decades ago, with excellent results sustained for decades? Don’t be naïve thinking you’ve found the “missing link”: it doesn’t exist.

The only thing we can do, always, is to try to be better tomorrow than today. But that isn’t done with meetings, committees, procedures, magic recipes, gurus, or talks. It’s done with practice. Want to be better at software? Practice software.

Now, do you have a good team, good professionals who really care about practice? Then yes, refining the processes and applying good techniques and tactics will improve things even more. But the order is this: good professionals first, then good methodologies. Repeating: good methodologies don’t create good professionals, but good professionals will certainly know how to take advantage of good methodologies.

“Good artists copy, great artists steal”

It reaches us through Steve Jobs, in the context of the graphical interface being copied first by Apple from Xerox and then by Microsoft from Apple.

The phrase became something of a cult after the famous TV movie Pirates of Silicon Valley. And for a lot of people it ended up wrongly tied to “plagiarism” and “piracy.” For a long time I never stopped to think about it myself, I just tossed it into that same bucket.

The phrase is usually credited to the great artist Pablo Diego José Francisco de Paula Juan Nepomuceno María de los Remedios Cipriano de la Santísima Trinidad Ruiz y Picasso, or simply Pablo Picasso :-). One aside, though: that attribution is shaky. The earliest documented version of the idea comes from T.S. Eliot, in 1920 (“Immature poets imitate; mature poets steal”), and it was Steve Jobs himself who started pinning Picasso’s name on the saying back in the 1980s. Either way, it’s the Picasso version that stuck, and that’s the one I’m starting from here.

Those who’ve followed me for a while and attend my talks know I spend a lot of time trying to explain the term “artist” applied to “software developer,” to “software craftsmanship,” and more specifically to the “practice of programming.”

Programming isn’t something where you read a book, do an exercise once, and “boom,” you already know how to program. Like any art, whether painting, music, or dance, it takes discipline and a lot of practice. And anyone who devotes themselves to an art knows that repetition is a technique, but repeating the same thing gets you nowhere.

The “practice” process starts with mastering a technique, repeating it many times, figuring out where you’re going wrong, and refining it. Once it’s mastered, you move quickly to new and harder techniques. It’s continuous, with no “end.” There’s no “arrival point” for anyone who practices an art. Even martial artists know a black belt doesn’t mean “the end.”

If I had to sum all of that up, it would be with Picasso’s phrase:

“Good Artists Copy, Great Artists Steal Take For Themselves”

I swapped “steal” for the more literal “take for themselves.” That’s the gotcha that changes the meaning for me.

Everything we have today, knowledge, culture, technique, technology, comes from centuries of meticulous accumulation and careful refinement by hundreds of great men and women. Literally, we are small elves standing on the shoulders of giants.

A material property, something physical, when taken from someone, is theft: for you to have it, someone had to go without. Copying that property is plagiarism. With knowledge and ideas, though, “copying” or “stealing” takes on several more abstract meanings, and I want to lay out one of them here.

Any artist has to start somewhere. A contemporary painter already starts light-years ahead of colleagues from a few centuries back. Back then, access to paint wasn’t obvious.

From what material in nature do you extract yellow? Maybe from egg yolk. And ultramarine blue? Things get complicated. Years of accumulated knowledge today hand us every possible shade without our having to think about it.

The same goes for pointillism, watercolor, oil paint. Today we don’t need to “discover” these techniques, they’re already exhaustively documented. In other words:

“To become ‘good’ artists, we must ‘copy’ the hundreds of techniques the giants created before us, until we master them.”

And let’s be honest: a painter who copies the techniques well enough to be indistinguishable from the masters who created them is already an excellent artist.

Every practice starts by copying what has already been done. Repeating, learning the nuances, understanding why the techniques ended up the way they are today. Whoever follows this philosophy, copying to master and pushing forward until they cover the whole field, is walking in the footsteps of the giants of the past.

Then comes an extra step. We geeks know the cliché of when “the disciple becomes the master.” In the movies it looks much easier, but that step is the moment the artist stops merely copying what they mastered:

“To become ‘great’ artists, after mastering what we copied, we can adapt it, refine it, and who knows, improve it or innovate on it. That’s when we stop just copying and start literally ‘stealing,’ making that property effectively our own, taking it for ourselves.”

That’s when an Einstein succeeds a Newton, or a Plato succeeds a Socrates. Here “steal” carries the sense of “making the property effectively your own,” far from the common use of “undeservedly taking what belongs to someone else.”

Every good painter or visual artist has dozens of giants and centuries of material to “copy” and “practice”: Michelangelo, Da Vinci, Picasso.

Wall Painter vs Painting Painter

Another thing I’ve been repeating ad nauseam is that there are programmer-coders and programmer-developers. Wall painters and painting painters.

Coders don’t understand their role in this story. They learn to copy a few techniques, maybe master some of them, and quickly call themselves “masters.” It’s mediocre thinking, the kind that would embarrass the giants of earlier eras: Babbage, Von Neumann, Bob Noyce. They’re mere wall painters.

Developers, as I like to call them, understand their role in history better. They recognize the richness of the material that already exists, practice without limit and without believing in an end, and master technique after technique, always aiming a level above what exists now. But they’d never put themselves in the position of painting a Mona Lisa without first perfecting the sfumato. Those are painting painters.

It embarrasses me to see programmers who read little, practiced miserably little, and think themselves in a master’s position. The same way it would embarrass a painter to hear someone claim they can paint a fresco without even knowing how to thin a plaster base.

To sum up my personal version:

“Good Artists Copy, Great Artists Take For Themselves, Terrible Artists Are Full of Themselves.”

If you’re learning Japanese this will definitely prove to be an invaluable tool. I am Japanese but I didn’t pay attention to proper Kanji training when I was a child. It shows now, as I can read Kanji only to what would be considered below high-school level in Japan. Still very useful, but it means I can’t read most websites in Japanese fast enough.

I’ve known of a Firefox add-on I’m very fond of for a long time now called Rikaichan. When enabled, you hover your mouse over the kanji text and it will popup a box with the translation. You install the Firefox add-on in the previous link and a proper dictionary to your native language from Rikaichan’s webpage. And that’s it.

But if you like to read Manga (Japanese Comics), you’re probably familiar with Furigana. It’s kind of like “subtitling” over the Kanjis with Hiragana or Katakana, the Japanese syllabaries (the first for Japanese only words, the second for foreign words).

That makes reading the more difficult Kanjis way easier and faster than hovering over each one or looking manually into the dictionary ideogram by ideogram.

A normal snippet of a Japanese webpage looks like this:

But with Furigana Inserter the same snippet looks like this:

To install it, first you install the add-on. Then you install the HTML Ruby add-on as well. Finally, you install this dictionary. It’s a 7-zip file you must decompress and manually drag it over Firefox to install.

If you’re on a Mac, I’m assuming you know what Homebrew is and already have that installed. Because then you can install Mecab like this:

brew install mecab
brew install mecab-ipadic

Finally, it seems like you must manually make Furigana Inserter aware of Mecab by adding a symlink into its extension folder:

ln -s /usr/local/lib/libmecab.dylib ~/Library/Application\ Support/Firefox/Profiles/454dy2eg.default/extensions/furiganainserter@zorkzero.net/mecab/libmecab.dylib

Understand that the Firefox Profile folder will have a different name on your computer. The one in the example is in my Mac. Open the terminal and tab to autocomplete each folder as you type and it should work just fine.

Restart Firefox, right-click in Japanese pages and you will have an “Enable Furigana” option waiting for you. Rikaichan seems to misbehave when hovering over Furigana enabled Kanji, so you may need to disable Furigana to use Rikaichan. But it’s a good compromise and may open a whole lot of new content in Japanese for us to consume.

One more reason to use Firefox.

On top of that, I’m anxious about my own research, which always seems to have pieces missing. Since there’s almost no one to talk to about it, the frustration grows. So, at the start of the article, I was already out of patience and began abusing the first and third person, in a near 1-to-1 dialogue. That wasn’t helping the main point, so I decided to trim the most aggressive parts. Let’s see if it’s more “digestible” now :-)

Last year I put together a list of books I read in my search for the essence of true management. Instead of going through the more trivial authors, like Peter Drucker, I went after the principles of Agility.

There are a few categories of managers: the “agile” ones, in the broadest sense of the word, and the “traditional” ones. The second group, prone to not wanting to change or learn anything, is sad, period. The first is on the right path, but there’s still a long way to go. The discussions I see today in agile communities also fall into the inertia of “good sense” and “common sense,” and end up shallow.

An agile manager should want to dedicate a lot more. Scrum itself is an example. As I said in the article You Don’t Understand Anything About Scrum: Jeff Sutherland and Ken Schwaber researched the relevant subjects long ago and gave the tips, though few follow them.

If you really want to understand Scrum, this literature helps a lot. To start, there are the Scrum Papers, which give you the introduction.

If you read them, you’ve already bumped into terms like “Emergence,” “Edge of Chaos,” “Complex Adaptive Systems,” and “Self-Organization.” And if you passed through those terms and left it at that, think again: what’s the point of reading without understanding?

I don’t know everything yet either, but I’m committed to understanding this properly. Without the principles, all that’s left is to “follow a recipe,” and that’s very little.

Whoever looks for rules or recipes is trying to shed their own responsibility. If the method, the procedure, or the process “fails” or doesn’t deliver, you can always hide behind “but I followed it to the letter, it’s not my fault.” That misses the point. What result are you after: following rules or reaching results?

I say this with “aggressiveness,” but some reluctance, because I used to think this way myself. Anyway, after some time reading academic articles, right now I’m working through these books:

• Systems Thinking, Second Edition: Managing Chaos and Complexity: A Platform for Designing Business Architecture. Once you fall into Complexity, you leave classical, reductionist, deterministic thinking behind and start needing the holistic thinking of Systems, which has itself gone through several paradigms. Highly recommended: it opens by describing the different management models and their evolution with precision.

• The Essence of Chaos. Complex Adaptive Systems fall under the broader subject of “Complexity,” and that leads straight to “Chaos.” I’d rather read something by Ilya Prigogine, but since I’m on a Kindle, the only things I found available were titles like this one, by none other than Edward Lorenz, one of the “fathers” of Chaos.

• Surfing the Edge of Chaos: The Laws of Nature and the New Laws of Business. Systems and Complexity also require understanding the Edge of Chaos.

• The Greatest Show on Earth: The Evidence for Evolution. One of the clearest examples of Complex Adaptive Systems is the nature around us. You need to understand Darwin’s Theory of Evolution. Most people “think” they understand it, but watch out: much of it is folklore, and plenty of people still doubt it. If you have doubts about Evolution, don’t even start.

More than that? From experience, every time someone uses words like “chaos” or “balance,” I notice they’re working from the casual definition, not the technical one. Many think “chaos” is just mess, something completely unwanted.

Worse still: they believe the “secret” is to reach balance and stay there, and they confuse the goal with seeking balance. That’s not quite it.

Here’s what the introduction of Surfing the Edge of Chaos says:

Equilibrium is a precursor to death. When a living system is in a state of equilibrium, it is less responsive to changes occurring around it. This places it at maximum risk. In the face of threat, or when galvanized by a compelling opportunity, living things move toward the edge of chaos. This condition evokes higher levels of mutation and experimentation, and fresh new solutions are more likely to be found.

When this excitation takes place, the components of living systems self-organize and new forms and repertoires emerge from the turmoil.

Living systems cannot be directed along a linear path. Unforeseen consequences are inevitable. The challenge is to disturb them in a way that approximates the desired outcome.

This is a definition of the biological evolution of living beings. But if you’ve read at least the basics of Scrum and Agile, it’s also a way to describe the Agile process. Reread the Principles of the Agile Manifesto and you’ll find similar texts.

As for Complex Adaptive Systems:

A complex adaptive system is formally defined as a system of independent agents that can act in parallel, develop “models” as to how things work in their environment, and, most importantly, refine those models through learning and adaptation. The human immune system is a complex adaptive system. So is a rain forest, an ant colony, and a business.

There’s vast literature. Search Amazon for terms like “Complexity,” “Edge of Chaos,” and “Systems Thinking,” and you’ll see that for decades researchers from the most varied fields and industries have been refining this body of knowledge. Techniques like the Toyota Production System/Lean, Six Sigma, Scrum, and Theory of Constraints all derive from this school of thought.

From my previous list, I reinforce the need for readings like:

• The Machine that Changed the World: The Story of Lean Production. Understand how the Lean process came about at Toyota.

• The Goal. Process engineering and cost control. Without understanding TOC, it’s futile.

• The End of Management, and the Rise of Organizational Democracy. A summary of all these subjects and, maybe, a provocative introduction to the rest.

More concerning: techniques like Lean, TOC, Six Sigma, and derivatives like Kanban are terms plenty of people use the wrong way nowadays. They were all born and refined in Production Engineering, always talking about the “production line” and the “manufacturing” of material goods, “physical” things.

In Agile, we’re talking about “Software,” something abstract and without parallel in the physical world, and therefore without the same references.

It’s impossible to migrate any Production Engineering technique verbatim. It doesn’t work that simply. A Scrum Sprint is a conscious adaptation by Sutherland and Schwaber, who knew this limitation and attempted an approximation in the abstract world.

But this isn’t obvious or trivial to grasp. I haven’t fully grasped it myself yet, and I believe most people in the agile world never stopped to study this difference.

Be very careful: “Management” is much more than “experience” or “good sense.” Just as a good programmer needs to read and research a lot to sharpen their technique, a manager needs to as well. And I’m saying this to myself.

This method bears a certain resemblance to the fallacy ad hominem, and comes from the same psychological root, but is different in essential meaning. The ad hominem fallacy consists of attempting to refute an argument by impeaching the character of its proponent. Example: “Candidate X is immoral, therefore his argument is false.”

But the psychological pressure method consists of threatening to impeach an opponent’s character by means of his argument, thus impeaching the argument without debate. Example: “Only the immoral can fail to see that Candidate X’s argument is false.”

In the first case, Candidate X’s immorality (real or invented) is offered as proof of the falsehood of his argument. In the second case, the falsehood of his argument is asserted arbitrarily and offered as proof of his immorality.

In today’s epistemological jungle, that second method is used more frequently than any other type of irrational argument. It should be classified as a logical fallacy and may be designated as “the Argument from Intimidation.”

The essential characteristic of the Argument from Intimidation is its appeal to moral self-doubt and its reliance on the fear, guilt or ignorance of the victim. It is used in the form of an ultimatum demanding that the victim renounce a given idea without discussion, under threat of being considered morally unworthy. The pattern is always: “Only those who are evil (dishonest, heartless, insensitive, ignorant, etc.) can hold such an idea.”

The classic example of the Argument from Intimidation is the story The Emperor’s New Clothes.

In that story, some charlatans sell nonexistent garments to the Emperor by asserting that the garments’ unusual beauty makes them invisible to those who are morally depraved at heart. Observe the psychological factors required to make this work: the charlatans rely on the Emperor’s self-doubt; the Emperor does not question their assertion nor their moral authority; he surrenders at once, claiming that he does see the garments—thus denying the evidence of his own eyes and invalidating his own consciousness—rather than face a threat to his precarious self-esteem. His distance from reality may be gauged by the fact that he prefers to walk naked down the street, displaying his nonexistent garments to the people—rather than risk incurring the moral condemnation of two scoundrels. The people, prompted by the same psychological panic, try to surpass one another in loud exclamations on the splendor of his clothes—until a child cries out that the Emperor is naked.

This is the exact pattern of the working of the Argument from Intimidation, as it is being worked all around us today.

We have all heard it and are hearing it constantly:

• “Only those who lack finer instincts can fail to accept the morality of altruism.”
• “Only the ignorant can fail to know that reason has been invalidated.”
• “Only black-hearted reactionaries can advocate capitalism.”
• “Only war-mongers can oppose the United Nations.”
• “Only the lunatic fringe can still believe in freedom.”
• “Only cowards can fail to see that life is a sewer.”
• “Only the superficial can seek beauty, happiness, achievement, values or heroes.”

As an example of an entire field of activity based on nothing but the Argument from Intimidation, I give you modern art—where, in order to prove that they do possess the special insight possessed only by the mystic “elite,” the populace are trying to surpass one another in loud exclamations on the splendor of some bare (but smudged) piece of canvas.

The Argument from Intimidation dominates today’s discussions in two forms. In public speeches and print, it flourishes in the form of long, involved, elaborate structures of unintelligible verbiage, which convey nothing clearly except a moral threat. (“Only the primitive-minded can fail to realize that clarity is oversimplification.”) But in private, day-to-day experience, it comes up wordlessly, between the lines, in the form of inarticulate sounds conveying unstated implications. It relies, not on what is said, but on how it is said—not on content, but on tone of voice.

The tone is usually one of scornful or belligerent incredulity. “Surely you are not an advocate of capitalism, are you?” And if this does not intimidate the prospective victim—who answers, properly: “I am”—the ensuing dialogue goes something like this:

• “Oh, you couldn’t be! Not really!”
• “Really.”
• “But everybody knows that capitalism is outdated!”
• “I don’t.”
• “Oh, come now!”
• “Since I don’t know it, will you please tell me the reasons for thinking that capitalism is outdated?”
• “Oh, don’t be ridiculous!”
• “Will you tell me the reasons?”
• “Well, really, if you don’t know, I couldn’t possibly tell you!”

All this is accompanied by raised eyebrows, wide-eyed stares, shrugs, grunts, snickers and the entire arsenal of nonverbal signals communicating ominous innuendoes and emotional vibrations of a single kind: disapproval.

If those vibrations fail, if such debaters are challenged, one finds that they have no arguments, no evidence, no proof, no reason, no ground to stand on—that their noisy aggressiveness serves to hide a vacuum—that the Argument from Intimidation is a confession of intellectual impotence.

The primordial archetype of that Argument is obvious (and so are the reasons of its appeal to the neo-mysticism of our age): “To those who understand, no explanation is necessary; to those who don’t, none is possible.”

The psychological source of that Argument is social metaphysics.

A social metaphysician is one who regards the consciousness of other men as superior to his own and to the facts of reality. It is to a social metaphysician that the moral appraisal of himself by others is a primary concern which supersedes truth, facts, reason, logic. The disapproval of others is so shatteringly terrifying to him that nothing can withstand its impact within his consciousness; thus he would deny the evidence of his own eyes and invalidate his own consciousness for the sake of any stray charlatan’s moral sanction. It is only a social metaphysician who could conceive of such absurdity as hoping to win an intellectual argument by hinting: “But people won’t like you!”

Strictly speaking, a social metaphysician does not conceive of his Argument in conscious terms: he finds it “instinctively” by introspection—since it represents his psycho-epistemological way of life. We have all met the exasperating type of person who does not listen to what one says, but to the emotional vibrations of one’s voice, anxiously translating them into approval or disapproval, then answering accordingly. This is a kind of self-imposed Argument from Intimidation, to which a social metaphysician surrenders in most of his human encounters. And thus when he meets an adversary, when his premises are challenged, he resorts automatically to the weapon that terrifies him most: the withdrawal of a moral sanction.

Since that kind of terror is unknown to psychologically healthy men, they may be taken in by the Argument from Intimidation, precisely because of their innocence. Unable to understand that Argument’s motive or to believe that it is merely a senseless bluff, they assume that its user has some sort of knowledge or reasons to back up his seemingly self-confident, belligerent assertions; they give him the benefit of the doubt—and are left in helplessly bewildered confusion. It is thus that the social metaphysicians can victimize the young, the innocent, the conscientious.

This is particularly prevalent in college classrooms. Many professors use the Argument from Intimidation to stifle independent thinking among the students, to evade questions they cannot answer, to discourage any critical analysis of their arbitrary assumptions or any departure from the intellectual status quo.

“Aristotle? My dear fellow—” (a weary sigh) “if you had read Professor Spiffkin’s piece in—” (reverently) “the January 1912 issue of Intellect magazine, which—” (contemptuously) “you obviously haven’t, you would know—” (airily) “that Aristotle has been refuted.”

“Professor X?” (X standing for the name of a distinguished theorist of free-enterprise economics.) “Are you quoting Professor X? Oh no, not really!"—followed by a sarcastic chuckle intended to convey that Professor X had been thoroughly discredited. (By whom? Blank out.)

Such teachers are frequently assisted by the “liberal” goon squad of the classroom, who burst into laughter at appropriate moments.

In our political life, the Argument from Intimidation is the almost exclusive method of discussion. Predominantly, today’s political debates consist of smears and apologies, or intimidation and appeasement. The first is usually (though not exclusively) practiced by the “liberals,” the second by the “conservatives.” The champions, in this respect, are the “liberal” Republicans who practice both: the first, toward their “conservative” fellow Republicans—the second, toward the Democrats.

All smears are Arguments from Intimidation: they consist of derogatory assertions without any evidence or proof, offered as a substitute for evidence or proof, aimed at the moral cowardice or unthinking credulity of the hearers.

The Argument from Intimidation is not new; it has been used in all ages and cultures, but seldom on so wide a scale as today. It is used more crudely in politics than in other fields of activity, but it is not confined to politics. It permeates our entire culture. It is a symptom of cultural bankruptcy.

How does one resist that Argument? There is only one weapon against it: moral certainty.

When one enters any intellectual battle, big or small, public or private, one cannot seek, desire or expect the enemy’s sanction. Truth or falsehood must be one’s sole concern and sole criterion of judgment—not anyone’s approval or disapproval; and, above all, not the approval of those whose standards are the opposite of one’s own. (note: ever notice those arguments that open with “I respect you a lot, but…”, clearly fishing for approval?)

Let me emphasize that the Argument from Intimidation does not consist of introducing moral judgment into intellectual issues, but of substituting moral judgment for intellectual argument. Moral evaluations are implicit in most intellectual issues; it is not merely permissible, but mandatory to pass moral judgment when and where appropriate; to suppress such judgment is an act of moral cowardice. But a moral judgment must always follow, not precede (or supersede), the reasons on which it is based.

When one gives reasons for one’s verdict, one assumes responsibility for it and lays oneself open to objective judgment: if one’s reasons are wrong or false, one suffers the consequences. But to condemn without giving reasons is an act of irresponsibility, a kind of moral “hit-and-run” driving, which is the essence of the Argument from Intimidation.

Observe that the men who use that Argument are the ones who dread a reasoned moral attack more than any other kind of battle—and when they encounter a morally confident adversary, they are loudest in protesting that “moralizing” should be kept out of intellectual discussions. But to discuss evil in a manner implying neutrality, is to sanction it.

The Argument from Intimidation illustrates why it is important to be certain of one’s premises and of one’s moral ground. It illustrates the kind of intellectual pitfall that awaits those who venture forth without a full, clear, consistent set of convictions, wholly integrated all the way down to fundamentals—those who recklessly leap into battle, armed with nothing but a few random notions floating in a fog of the unknown, the unidentified, the undefined, the unproved, and supported by nothing but their feelings, hopes and fears. The Argument from Intimidation is their Nemesis. In moral and intellectual issues, it is not enough to be right: one has to know that one is right.

by Ayn Rand – last chapter of “The Virtue of Selfishness”, 1964

The first article is What makes a good programmer?

An Analytical Thinker

Programmers need to be problem solvers. The process of programming requires that we systematically break complicated problems down, plan and implement solutions and find / eliminate small inconsistencies in code (bugs).

Analytical thinking also manifests in the ability to follow complicated logic through disparate code segments and understand it. It allows us to grasp abstract concepts such as Object Oriented methodology and design patterns and implement it in practice.

Has Clear Priorities

If I would ask you to rate the following according to priority, how would you order them?

• Security
• Maintainability
• Usability
• Performance
• LOC (lines-of-code) count

Take a moment to think about that, and then consider:

1. If you picked LOC count first, you failed big time in my book. In fact, LOC optimization can often go directly against the other metrics (such as maintainability). A lower LOC count should never be a goal, only a result of careful application of well factored architecture.
2. If you picked performance first, you are probably the guy who keeps writing articles about how you should use a while loop instead of a for loop since it came out a few milliseconds faster in your benchmarks. You might be inflicted with a case of premature optimization.

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. – Donald Knuth

Performance should only be good enough to satisfy the requirements of the application. Aside from caveats to well-known pitfalls (such as executing queries in each iteration of a long loop), performance optimizations should be deferred to the very last and even then should be used appropriately (profile … profile … profile … optimize).

The only exception to this is if you are primarily developing performance dependent applications (such as low-level system drivers).

3. Security is on somewhat of a middle ground. Depending on the application and distribution model it can be completely useless or mission critical. It’s mostly somewhere in between, and thus can’t be ranked as number one.
4. Maintainability is definitely one of the most important attributes of a software application. High maintainability allows you to improve other attributes (such as performance), when it is needed.

Maintainability is the single most important factor for keeping productivity up and costs down. For a long time I strongly believed this to be the most important attribute of software design. However… 5. The most important attribute is usability. In the end, the worth of your application is only as much value as it delivers to the end-user.

We should always remember - software is not written to serve its developers or the systems they run on. They are written to solve problems. If those problems are not solved, then the project is a failure.

I wrote usability here as a more general term than just UI/UX effectiveness. Even a command line application or a background service has its usability factor in the sense of how well it answers a specific need.

Gets Things Done

In principle, it’s simple. You’re looking for people who are

1. Smart, and
2. Get things done.

Joel Spolsky

Quite possibly the single most important trait in a developer. You can excel at all the previous attributes and still be a mediocre programmer if you just can’t get things done. One average but productive developer could easily replace several highly talented but slowly moving developers, depending on his responsibilities.

At the end of the day you definitely want more highly-productive developers than those who are high on theory but not actual work.

Does More Than “Just Enough”

Getting things done is important. Getting things done “the right way” is even more important.

Constantly paying off your technical debt is crucial - if you keep accruing debt by “hacking” quick fixes that work now but are not maintainable, you only create the appearance of progress. In reality, the cost of getting rid of a large technical debt could become prohibitive before you know it.

Taking the time to constantly refactor code into a more maintainable state is the best way to prevent the spiral into project oblivion.

Accountable

A person could be a very capable programmer on technical ability alone, however if he does not own up to his mistakes and does not respect deadlines he could become a liability very quickly.

Responsibility also means to know where to let go of your ego for the good of the project. We developers often have large egos as we consider ourselves experts on many things. Putting the project first is a sign of a good developer.

Good Human Relations

Another all-around useful trait, this one applies to programmers as well. There is some stereotype that programmers are reclusive, unsociable creatures - programmers are still people.

In order to be a part of a team or handle clients, a programmer must have above basic social skills. Rudeness, arrogance, short-temper - do not have a place in a professional work environment. All it takes is one bad apple to ruin the mood for everybody.

That’s It

If you answer to all of the above, you are probably a pretty good programmer.

You might notice I didn’t mention passion or technological diversity as qualifying traits. Simply put, I don’t think they’re very relevant to the quality of a programmer.

Passion is nice to have, however I’ve known many very professional and high-quality developers who were just content to go about their work professionally from 9 to 5 and then go home and have a meaningful and fulfilling family life. A programmer can definitely be completely professional without being passionate about programming.

Technological diversity is another nice to have but not a prerequisite - as long as you are in command of the technologies you work with, a lack of diversity shouldn’t affect you too much. Decision makers need to be well aware of all the options before starting a project, however nowadays the choice of technology simply is not that important.

You can achieve good results regardless of the programming language and database engine among other consideration. The biggest consideration should be the type of skills available to your personnel.

[Translation] Why Good Programmers Are Lazy and Dumb

Source: blogoscoped

I realized that, paradoxically enough, good programmers need to be both lazy and dumb.

Lazy, because only lazy programmers will want to write the kind of tools that might replace them in the end. Lazy, because only a lazy programmer will avoid writing monotonous, repetitive code – thus avoiding redundancy, the enemy of software maintenance and flexible refactoring. Mostly, the tools and processes that come out of this endeavor fired by laziness will speed up the production.

Of course, this is only half the truth; for a lazy programmer to be a good programmer, he (or she) also must be incredibly unlazy when it comes to learning how to stay lazy – that is, which software tools make his work easier, which approaches avoid redundancy, and how he can make his work be maintained and refactored easily.

Second (and I will elaborate a bit more on this because I find the concept to be less known than the first) a good programmer must be dumb. Why? Because if he’s smart, and he knows he is smart, he will:

a) stop learning b) stop being critical towards his own work

Point a) will make it hard for him to try to find new techniques to allow him to work faster. Point b) will give him a hard time debugging his own work, and refactoring it. In the endless battle between a programmer and the compiler, it’s best for the programmer to give up early and admit that it’s always him and never the compiler who’s at fault (unless it’s about character encoding issues, which is the part even the compiler gets wrong).

But there’s a more crucial point why a good programmer must be dumb. That’s because for him to find the best solutions to problems, he must keep a fresh mindset and manage to think out of the box (or rather, know its actual shape). In a way, this leads to the mindset of a child; incredibly creative because he never heard “no” for an answer.

The direct opposite approach would not be very constructive; to be knowledgeable about the parameters at hand, and accept them. Because who knows how many limits you think are there for real? The less you know, the more radical will your approaches be; the better the tools you develop, and the better the product you develop with them.

I know from my work that a good programmer, when confronted with a problem from management, will adopt this mindset of being dumb; he will start asking the most simple, child-like questions. Because he doesn’t accept the parameters suggested to him that someone thinks make up the problem. Here’s a typical conversation from the lost land of web development:

• “Since yesterday, our client can’t see the logo on the web site.”
• “Did he restart his browser?”
• “Yes.”
• “Did he restart his computer?”
• “Yes.”
• “Did he empty his cache?”
• “Yes.”
• “Does he run Internet Explorer 6?”
• “Yes.”
• “Is he sure he can’t see it?”
• “Yes.”
• “Did he look at the web site on the screen?”
• “What?”
• “Well, he might have printed it out.”
• “No, he was looking on it on the screen.”
• “Did he also not see other images besides the logo?”
• “What? Well, I will ask him.”

For the sake of argument (and this was entirely hypothetical) let’s say the client indeed turned off images in his browser. Or his son did. Whatever the case, this answer could not have been found if you would work in “smart” mode¹. None of the questions asked by the programmer required any programming skills. No; simply because the problem is so stupid, only stupidity can tackle it.

¹ Some years ago, I had a long telephone discussion about the whole web site being messed up since my last update… it turned out the guy disabled stylesheets in his browser. Back then, I would have suspected everything but such a simple solution and was listening to half an hour of complaints about quality standards etc. In the end, the assumption that my update was at fault was just that… an assumption. You better listen to facts only if you start debugging, and never to what people think might be the reason.

In similar fashion, when one of my co-programmers asks me: “Why isn’t this working?” most of the time it’s because they’re working on the wrong file (e.g., they linked to library 1 but they’ve altered library 2, and their revision isn’t showing, or they simply didn’t link the library at all). When you ask a colleague for help, particularly in programming, you want him to know less about your project… so he will ask the stupid questions you sub-consciously avoided asking yourself because you thought you knew the answer, when in fact you didn’t.

There’s another side to it. The too-stupid person will just run off and, without a second thought, do something wrong. The too-smart person will sit down and plan something right, without taking any action. A pragmatic programmer is sort of in-between; he knows making the wrong decision 1 out of 10 times doesn’t hurt the goal as bad as making only right decisions 5 out of 10 times, and making no decision at all the other 5 times.

It’s like the story of the centipede. The centipede was very good at walking with its hundred legs. It never spent a thought on just how it could walk. Until one day, when a big black bug asked the centipede “How can you manage to walk with all those feet? Don’t you find it hard to coordinate their rhythm?” The black bug already left, when the centipede was still sitting down, pondering how it could walk, wondering, and (for the first time in his life) even worrying a little bit. From that day on, the centipede couldn’t walk anymore.

So you better not think too much if you want to achieve something. And of course this is only half the truth, too…

You’re a “switcher”: you’ve decided to take the big leap, step away from the Windows world a bit, and buy your first Mac. The timing is good and prices are accessible, especially if you can buy from someone abroad. Keep one thing in mind from the start: this is a paradigm shift.

The first week or two are the worst. You’ll always have that nagging thought, “On Windows I did this more easily…” or “On my Linux I did this more easily…”. The temptation to go back is strong the first time you leave your comfort zone, same as with any change. Push yourself to learn the new tricks instead of falling back on the old ones.

As soon as you log in to the Mac for the first time, it arrives in a state where you need to configure very little. A few important things to get you started:

• The first difference is that application menus are all centralized. The bar at the top is the only menu, and it changes according to the application that currently has focus. The first option is the icon, which gives access to global system options like “About,” where you can view details of your hardware; “Software Update,” the equivalent of Windows Update; and options like Shutdown. By the way, the very first thing to do when starting your Mac is to run Software Update and install the latest updates. Unlike Windows, you tend to get large packages rather than dozens of small updates.

• The second option is always the name of the focused application. Initially it will always be “Finder,” the equivalent of Windows Explorer. This menu always gives access to things like “Preferences,” which on Windows usually lives under the “Edit” or “Tools” menus.

• Some shortcuts are global and constant. To start, understand that there is the Ctrl (control, as on a PC) key and a special one called Command ⌘. They are different, and although it isn’t a strict rule, things done on a PC with Ctrl are done on a Mac with ⌘. For example, copying text to the clipboard is ⌘+C, not Ctrl+C. Pasting is ⌘+V. Every application closes with ⌘+Q. The “Preferences” option is always ⌘+, (comma). The “Help” menu is always ⌘+Shift+?. A tip on that last Help option: use it to discover menu items whose shortcut you don’t remember or whose submenu you can’t find. Just type a few letters and Mac OS will open the menu and highlight the right item for you.

• Unlike Windows, the Mac paradigm is based on one window per document rather than a single application window with embedded document windows. ⌘+W closes the application’s current window. ⌘+N or ⌘+Shift+N usually opens a new window. We don’t normally close applications very often; they tend to stay open. That’s indicated in the Dock, at the bottom of the screen, by a glowing circle below the icon of an open application.

• Also unlike Windows windows, the red circle at the top of each window closes only that window, not the entire application. It’s the equivalent of ⌘+W. Another thing: you can move a window from the title bar or the footer bar, when there is one. So even if a window is partly off the physical desktop, you can still move it. On the other hand, the only way to resize a window is from the lower-right corner, not from any edge.

• In the title bar of applications that manipulate files, like Finder, Pages, and most apps that show the filename with an icon next to it, the icon is “normally” draggable. You can drag it to a folder and the file being edited moves there. You can also click while holding ⌘ (Command) and it will show the parent folders, letting you navigate them.

• Another thing that doesn’t exist on Windows: “Hide Windows.” Get used to ⌘-H to hide all windows of an application. It isn’t the same as minimizing, since hidden windows don’t appear in the Dock. A hidden application keeps a transparent icon in the Dock. Even better: if you have dozens of applications open in a mess of windows behind windows, use ⌘-⌥-H (Command-Option-H) to hide every window except the one currently in focus. It’s a very satisfying way to clean up your workspace and focus on what you’re doing.

• To switch between applications, on Windows we’re used to Alt-Tab. On the Mac the near equivalent is ⌘-Tab. But it switches between Applications, not between Windows like on Windows. It’s a subtle difference. If you have Photoshop with 4 windows and Word with 3 windows open, ⌘-Tab shows only 2 icons, one per application. To switch between windows of the same application, use ⌘-` (Command-Backtick). Or, more commonly, use Exposé to pick between windows.

• Note the icons on the F1 through F12 keys; they clearly show the functions you get when you press them. F1 and F2 are screen brightness. F3 is Exposé. F4 is the Dashboard, where you install Widgets. F11 and F12 are volume. Those are the main ones. Don’t forget that on newer MacBooks with a Multitouch Trackpad you have options like swiping 4 fingers down to activate Exposé or up to show the Desktop.

• Spotlight is very useful. ⌘+Space activates the search field. If you use external hard drives or USB drives and don’t want to waste time indexing their files, open “System Preferences,” option “Spotlight,” and drag the volumes that should be excluded.

• If you’re used to Explorer, you’ll miss a lot of things in Finder. You can copy and paste folders and files, but there’s no cut. You move items to different volumes by dragging and dropping while holding ⌘. To force a copy, use the Option/Alt (⌥) key. You’ll end up using drag-and-drop far more than on Windows. You also get different ways to organize your files, especially since ⌘+I opens the Info window, where you can add metadata like keywords for searching and colors (yes, you can even search for files and folders by color in Spotlight).

• Another difference between Finder and Explorer is how you rename files and run programs. In Explorer you press “Enter” to run a file and a single mouse click to rename. On the Mac you press “Enter” to edit the name of a file or folder and ⌘+O to run a program. That avoids the situation where you select dozens of files, accidentally hit Enter, and set off a cascade of programs opening. On the Mac that accident basically never happens.

• The Dock gives you quick access to your favorite applications. You can drag an icon out to remove it, or drag anything onto it to add it. If you open an application that wasn’t there, its icon appears anyway, because the Dock also shows every running application, with a glowing indicator below it. To keep an application in the Dock, drag it there or, while it’s open, use the context menu (right-click) on its icon and choose “Keep in Dock.” If you want that application to launch every time you log in, use “Open at Login.”

• Mac OS Leopard (10.5) doesn’t ship with the US International keyboard layout that’s common on Windows for non-ABNT-2 keyboards (which lack the cedilla). The Mac default is to use shortcuts like ⌥+c for cedilla, ⌥+e for acute accent, ⌥+` for grave accent, ⌥+n for tilde. But if you want the traditional way, download the US International layout for Mac from Brazilian developer Rainer Brokerhoff’s site and follow his instructions. From Snow Leopard (10.6) onward, this is no longer necessary.

• As on Windows and other systems, the Mac’s default key repeat rate is somewhat slow. Open any text editor and use the arrow keys to move the cursor and you’ll notice the lag. I prefer maximum speed. Go to “System Preferences, Keyboard” and set “Key Repeat” to “Fast” and “Delay Until Repeat” to “Short.” You’ll thank me:

• Say you have hundreds of files, like photos from your digital camera, sitting in JPG, and you want to rename them all to another convention and convert them from JPG to PNG. You can do this and much more with a feature called “Automator.” In Finder, select the files, right-click (on the Trackpad, place two fingers in the middle and use your thumb to click once at the bottom), then choose “More,” “Automator,” “Create Workflow.” That opens the selected items in Automator, a linear flow of “filters” where you chain the output of one filter as input to the next, like “Change Type of Images,” then “Rename Finder Items,” and even “Add attachments to Front Messages,” and so on. It’s an extremely powerful way to chain actions, a lot like Unix/Linux command-line pipes, but far more intuitive.

• Few people figure out how to take screenshots at first, since there’s no “Print Screen” key on a Mac. But there’s something far better: ⌘+Shift+3 takes a full-screen shot and saves it as a PNG on your Desktop. If you don’t want the file, ⌘+Ctrl+Shift+3 sends the shot to the clipboard so you can paste it into another application. Even more useful, ⌘+Shift+4 turns your cursor into a crosshair for capturing just the small area you select. Position the crosshair over any window and press “Space” and it captures that whole window. For clipboard-only, use ⌘+Ctrl+Shift+4.

• Another important thing: every print dialog opens with ⌘+P, and it always has the option to generate a PDF instead of sending to a printer. That’s extremely useful if you don’t have a printer or want to save a receipt from the web or something like that. Beyond that, the “Preview” application that opens images and PDFs can also do simple editing: resizing images, cropping, copying pages between PDF files, reordering pages, deleting pages. In other words, PDFs are quite editable on a Mac without a commercial Adobe product.

• Speaking of “Preview,” learn to use “Quick Look.” To view images, videos, audio, PDFs, Office files, iWork files, and so on, just select the file (or a set of files) and press “Space.” A floating window opens where you can quickly view the file. “Space” again closes it. You can preview most major file formats without opening their respective programs.

• On PCs we tend to create custom folders for videos, photos, and the like. On the Mac, try to use the default folders it already gives you. More than that, get in the habit of using iTunes to organize your music and videos, iPhoto for your photos, and iMovie to edit your videos. That keeps your Mac tidy, makes backups easier, simplifies searches, and keeps everything more accessible. Many Apple applications integrate with the iLife suite (iTunes, iPhoto, GarageBand, iMovie, iWeb), so it’s simpler to let these apps manage your files. Watch the screencasts on Apple’s website if you don’t know how they work yet; they’re simple, intuitive, and powerful.

• Installing applications on a Mac usually trips people up at first. You normally download one of two formats: DMG or MPKG. The first stands for (D)isk i(M)a(G)e and is similar to an ISO. When you double-click it, the DMG “mounts” inside /Volumes. The application is usually already there, and you just drag it to the “Applications” folder on your Mac. Then you “eject” the DMG with the ⏏ icon or with ⌘+E. The second format, (M)ac (P)ac(K)a(G)e, is the equivalent of Windows’ MSI: you double-click it and it runs like a standard installer. Usually only programs that need to install a lot of system components use MPKG, like Microsoft Office and Adobe CS4. Sometimes you download a ZIP that contains a DMG or an MPKG. The important part: if it’s a DMG with no MPKG inside, always drag the application to Applications before running it. Safari does an extra step: it downloads the DMG, mounts it, extracts the application, and deletes the DMG, so you drag it from the Downloads folder. A technical detail: MPKGs come with a BOM (Bill of Materials) that works as a recipe of the original system file permissions for scripts that adjust permissions. The same BOM helps uninstall, since it records everything that was installed.

• All application preferences are stored in your directory (/Users/your_login), then Library/Preferences. They’re files in plist format, which is XML stored in binary. Regardless of where the application is copied, it saves preferences in that location. Don’t confuse the “Library” local to your home folder with the global “Library” at the root of the system, since you’ll rarely need to touch the latter. On the Mac there’s no Registry, and so no nightmare of maintaining one. When an application starts behaving strangely, the first thing to try is deleting its .plist file.

• On the Mac, Finder doesn’t show every directory. Most are hidden, like /usr, /bin, /tmp, /dev, /var, /etc, and others that Unix or Linux users will recognize. Some important ones like System and Library at the root of your Mac are rarely accessed. User accounts live in the Users folder and applications in the Applications folder. Inside your “Home” folder are folders like Desktop, Documents, Downloads, Music, Movies, Pictures, Sites, Public, and Library. In that Library are things like “Application Support,” where plugins and other application dependencies live, and “Preferences,” which I explained earlier. Poke around these folders; the naming is fairly obvious.

• Inside the “Applications” folder there’s another called “Utilities” (also accessible via ⌘+Shift+U) with things like “Activity Monitor,” similar to the Windows Task Manager. There’s also “Console,” an easy way to review system log files. Another important one is “Disk Utility,” which does advanced things like repartitioning your disk non-destructively, creating DMG files including encrypted volumes, burning DMGs to DVDs, and more.

• Another difference from Windows: Macs don’t have drive letters (C:, D:, E:, and so on). The Mac, being a BSD Unix, behaves more like Unix/Linux, where DMG and ISO files are “mounted” inside the /Volumes directory and show up normally as disks in Finder that can be ejected. The Mac is very flexible with disk images: you can use Disk Utility to create bootable USB drives, create multiple boot partitions, copy DVDs into DMGs, and such. Network shared folders are also mounted in /Volumes. The behavior is consistent and fast, and Finder, unlike Windows Explorer, rarely crashes and almost never returns to an inconsistent state.

• A technical detail you’ll notice: Finder (including the top menu and Desktop), the Dock, and applications are very independent. On Windows, especially XP, you’ll remember an application sometimes getting sluggish, hard even to drag, making it difficult to click the Start menu or taskbar. On the Mac, even when Finder crashes or gets slow, the Dock behaves independently. Other applications aren’t affected either, including their window refresh. Another tip: if an application truly freezes, use ⌘+⌥+⎋ (Command+Option+Esc) to open a window where you can force-quit it. And ⌘+Shift+Q forces the current user to log out.

• Speaking of deleting: select files and use ⌘+Shift+⌫. That ⌫ key is called “Delete” but it’s the Mac equivalent of the PC’s “Backspace.” The Mac equivalent of the PC’s “Delete” key is Shift+⌫. You can also do more, like ⌥+⌫ to delete the whole word to the left of your cursor, or ⌘+⌫ to delete to the beginning of the line.

• For navigating text, get used to combinations like ⌘+← and ⌘+→ to go to the beginning or end of the line, ⌘+↑ or ⌘+↓ for the equivalents of Page Up and Page Down, and ⌥+← and ⌥+→ to move by word left or right. These keys become second nature because you use them constantly.

• If you’re in a text editor (Mail.app, Safari, TextEdit, and so on) and the text is too small to read comfortably, use ⌘-Shift-’+’ to increase the font size and ⌘-Shift-’-’ to decrease it. To zoom in on an image or small detail, hold Ctrl and drag two fingers up on the trackpad to zoom in, or two fingers down to zoom out.

• Still on text: if you select a passage in any application, you can drag it to the desktop or another folder in Finder. It’s saved as a “snippet,” like a persistent clipboard item you can drag back into another text editor or leave on the desktop until you decide what to do with it.

• All MacBooks, including older ones, have always had a two-finger multi-touch action for scrolling. Just slide two fingers, without clicking, up or down on the trackpad to scroll, for example in Safari.

• Before Lion, there was a feature to encrypt your files in real time. It was called FileVault and was bad enough to ignore. But from Lion onward, FileVault 2 arrived, completely different and this time so good it would be foolish not to use it. Open “System Preferences, Security & Privacy,” the “FileVault” tab, and enable full disk encryption. VERY IMPORTANT: it generates secure keys that allow decryption of your data. Write them down somewhere safe and never lose them. One tip: grab your iPhone or any smartphone with a camera and take a photo of the screen when it shows you the keys.

• From Lion onward there’s Mission Control, which you activate by placing four fingers on the trackpad and swiping up. It’s a replacement for the old Exposé and genuinely handy once you get used to it. When “maximized,” applications become their own parallel “Space,” and you navigate between them by swiping 3 fingers left or right. The default, though, has these Spaces rearranging automatically based on usage. I don’t like that, because I prefer my applications and desktops always in the same position (in Mission Control you can drag Spaces to wherever you like). To disable the forced rearrangement, go to “System Preferences, Mission Control” and turn off “Automatically rearrange spaces based on most recent use.”

• Also from Lion onward, the way you save files changed a lot. There’s now an “Auto Save” feature. So if an unexpected crash happens, or you have to force-shutdown the machine, or any drastic situation where you’d lose what you were editing, don’t worry, because Lion is saving automatically. There is one annoyance, though: the file sometimes gets a “Lock” that prevents you from editing it. You can unlock it manually, but it’s a hassle. I prefer to turn it off by going to “System Preferences, Time Machine,” clicking the “options” button, and disabling “Lock documents.” While you’re there, check the photo below for how to do that, and also what I keep on my Time Machine exclusion list, things I’d rather not back up either because they don’t matter (Software Images, for example) or because I prefer to back them up manually (iPhoto).

• Still on the new Lion way of handling documents and versions: in new applications you now have a “Save a version” option in the menu, so you can save moments in time in case you need to go back and recover an earlier state. Think of it as a mini Time Machine or granular backup. One option that no longer exists is “Save As,” which let you save the same file under a different name. Instead the operation is a bit more annoying: first “Duplicate” (all in the same main “File” menu) and then “Save” to store it under a different name. Get used to it; it’s annoying at first but becomes trivial.

• The Mac has plenty of unexpected surprises. In the main menu, the item with the application’s name always has a sub-item called “Services.” Try selecting a block of text in any application, then go to the “Services” menu, then “Speech,” then “Start Speaking Text,” and the Mac reads the selected text aloud. Unfortunately there’s no option in Portuguese.

• In Finder you have the “Go” menu, then “Connect to Server,” or the shortcut ⌘+K, to connect to shared folders on other Macs or on your company’s Windows file server. For a Windows server you’re used to typing something like \server\folder. On the Mac you type something like smb://server.domain/folder. If your server is Windows 2000 or later, include the full server name, including the root domain of your directory. Usually that’s all you need.

• There’s no shortcut to force-lock your computer so nobody can access it while you’re away. One way is the screen saver. First, open “System Preferences” and “Security” and enable “Require password to wake this computer from sleep or screen saver.” Now, still in “System Preferences,” choose “Desktop & Screen Saver,” then “Screen Saver.” On that screen there’s a “Hot Corners” option where you can pick a corner that starts the screensaver when you move the mouse there. That gives you a quick way to lock your computer.

• Every time you download a program from the internet, the first time you run it the Mac shows you the URL it was downloaded from and asks for confirmation. To be prompted whenever you or something tries to modify a system setting, go to “System Preferences,” “Security,” and enable “Require password to unlock each System Preferences pane.” The Mac warns you whenever something tries to escalate authority, and you enter your administrator password to confirm. For Linux users, it’s like using “sudo” in the Terminal. The Mac is quite secure: its architecture keeps trivial malware from taking over the machine the way it does on Windows, and that has nothing to do with market share, since the user has to explicitly do something foolish to get compromised. Antivirus is only useful to make sure you don’t forward someone an infected attachment you received. Your machine stays fine, but you might pass the virus along by accident.

• You’ll miss programs like Norton Utilities because you simply don’t need them. For a defragmenter, for example, Mac OS defragments automatically for you. Also understand that your filesystem is HFS+, not FAT or NTFS. One quirk is that HFS+ has Resource Forks, so when you copy Mac files to FAT USB drives you end up with what look like “duplicate” files with an underscore at the start of the name, which is the resource fork representation on filesystems that don’t support it. Also understand that every Mac “application” is actually a folder with a “.app” extension and an icon. When transporting applications, don’t copy them straight to a FAT USB drive; first put them in a DMG or just zip them (the “Archive” option in the context menu), then copy. Remember too that Unix files have permissions and FAT doesn’t preserve them.

• The equivalent of Windows Startup Items is in “System Preferences,” “Accounts,” your user, “Login Items.”

• The Mac is a BSD Unix. As such, it has a Terminal application in the Utilities folder. Use Spotlight to reach it easily. It opens a BASH shell, and from there you can manipulate files and programs easily in a POSIX environment. If you come from Linux, you’ll feel right at home.

• Speaking of Linux, one weak spot is the lack of a good package manager. On Ubuntu/Debian we have apt-get; on Redhat/Fedora/CentOS we have Yum. On the Mac there’s no standard. My recommendation is HomeBrew, a system for installing and maintaining open source projects. It’s not as powerful or robust as Aptitude on Ubuntu, but it does its job well. You’ll also come across others like MacPorts and Fink, but they’re either outdated or more hassle, so I don’t recommend them. See the next tip.

• HomeBrew is the best installer for open-source software like MySQL, PostgreSQL, Redis, Git, Android SDK, and much more. In some cases it downloads the Mac binary directly, but usually it downloads the source code, applies necessary patches, and compiles. For that you need compilers and other tools. The simplest way is to use the App Store to buy and download XCode. This tip is more for software developers; if you’re not one, you can ignore it.

• Every Mac comes with Apache 2 pre-installed. It’s hidden under the name “Web Sharing.” You’ll find it in System Preferences, “Sharing,” then “Web Sharing.” When you turn this on or off, you’re dealing with Apache 2, which by default looks at the “Sites” folder in your Users/Home directory. Another option, “Remote Login,” enables or disables OpenSSH access. “Screen Sharing,” in turn, activates a VNC server, so a VNC client on Windows can also share the screen of your Mac.

• One weakness fixed from Snow Leopard onward is the Firewall. On Leopard and earlier it ships disabled. Open System Preferences again, option “Security,” then “Firewall.” It comes set to “Allow all incoming connections.” Change it at minimum to “Allow only essential services” or, better, “Set access for specific services and applications,” which works as a whitelist. Everything is blocked by default, and only the applications you select in the box below get open ports.

• Lion is the most practical version for disaster recovery. There’s no more boot DVD. Every Mac now has a hidden partition with the installer, for when you need to reinstall, check disks, or do other things. Just boot while holding (Command-R) ⌘-R, or hold the (Option) ⌥ key during boot to get a menu with the available partitions and choose “Recovery HD.”

• Finally, to impress your friends: when you minimize a window, use Exposé, or activate the Dashboard, all these actions have animated transitions. Hold the “Shift” key at the same time and the animation plays in slow motion, showing every step of the transition at slow speed :-)

Applications

Some of the applications below are on the Mac App Store, and I always recommend checking there first. Otherwise, follow the links and download manually. At the end of this section I have a tip on keeping your applications up to date.

• If you have an iPad, you can use it as a second monitor connected to your Mac over Wi-Fi (both need to be on the same network). For this, buy Air Display.

• If you have a new Apple TV (the small black model), you can mirror the screen of your iPhone 4S, iPad 2, or later over Wi-Fi using the Air Play feature on iOS. The next version of Mac, Mountain Lion, will let you do the same from your Mac. Until then, you can still mirror your Mac to an HDTV via Apple TV by buying Air Parrot. It’s the simplest way to watch movies and videos from your Mac straight on your TV.

• I personally love reading a lot of RSS and Atom feeds in my Google Reader account, and the best Mac and iOS app for that is easily Reeder.

• With so much internet content, you download a lot but don’t always know which files are eating the most disk space. To find them (tip: most of the time it’s your video podcasts in iTunes) there are excellent paid apps like DaisyDisk, but I still like the free GrandPerspective.

• OS X handles compressed formats like zip and gz on its own. But to decompress more interesting formats like 7-zip and rar, be sure to install The Unarchiver.

• For applications, use sites like Apple’s own to see what’s most popular. To start, you’ll want Perian for more video codecs, Flip4Mac for Windows codecs, and VLC for a player that handles all the strangest codecs.

• To rip DVDs and create mp4 files, use the freeware HandBrake. To convert videos for iPod and iPhone, use iSquint. To create, edit, and convert subtitle files, use Miyu and Submerge.

• For presentations, forget Powerpoint. It’s very worth buying iWork to use the famous Keynote. The iLife and iWork suites are all on the App Store, so buy them there.

• For communication, use Adium, a client for Jabber, Messenger, Yahoo, AIM, and more, one of the best around, though it doesn’t yet support video chat for Messenger. The classic Skype is another great option. If you like IRC, one of the best clients is Colloquy.

• You want to run Windows or some Linux distro. The option that comes with the Mac is Bootcamp (in Utilities), to repartition your HD and install a second operating system. Before that, though, I recommend virtualization products like the open-source VirtualBox, or the commercial Parallels Desktop or VMWare Fusion. They’re all very good. With them you can run Windows XP or Windows 7 (I don’t recommend Vista unless you have a desktop Mac with more than 4GB of RAM), Ubuntu, Fedora, and so on. In most cases, even heavy stuff like SQL Server with Visual Studio works great. For games, no, better to dual-boot, or better yet, buy a PlayStation 3 or Xbox 360.

• The Mac is quite secure. Things like saving passwords in Safari go into an infrastructure called Keychain, an encrypted, secure storage. You can even encrypt your entire Home folder if you want. For other things, like sharing authentication data across multiple browsers or just storing notes securely, I suggest 1Password, one of the best out there.

• Macs normally don’t need uninstallers. For the vast majority of programs you just drag them to the Applications folder, so you delete them from there. They do leave behind a few artifacts, like files in the Library/Preferences directory. Normally it’s fine to leave them; they’re tiny and don’t corrupt anything like a Registry. But if you’re the paranoid type who needs to remove everything, use AppZapper, which does it for you.

• Anyone who’s watched Mac screencasts online might wonder, “How do they do that?” One of the secrets is the commercial ScreenFlow, by far one of the best, most performant, and easiest to use. Worth every cent. To record the keyboard shortcuts you’re using as a layer on top of your video, use KeyCastr. For mouse effects like focus and others, use OmniDazzle.

• The Mac needs very little maintenance, and most tools are already built into the operating system. As a visual shell to reach them more easily, use Onyx, which runs all the maintenance scripts at once. Sometimes, when your Mac starts behaving strangely, running Onyx helps a lot, though that situation is very rare.

• As a text editor there’s no getting around it: after 3 years my favorite is still TextMate. If you’re more of a Linux purist, you might prefer MacVim, especially with my Ruby snippets package :-) Beyond those, there are dozens of other editors, but none really appealed to me. A plugin that makes TextMate even nicer is MissingDrawer, which I recommend trying. If you’re a developer, follow my vimfiles repository to get your Vim pre-configured with the best of what’s available.

• For downloaders, Mozilla Firefox has various extensions, or if you’re a solid Linux user you’ll know how to use good old curl. But if you want something more visual, an excellent commercial alternative is Speed Download, which has options for parallel downloads, resuming interrupted downloads, and so on. Still on downloads, if you’re into BitTorrent, one of the best clients is Transmission.

• For burning DVDs, the Mac handles it reasonably well on its own. But if you want more control and detail, one of the most traditional and best commercial options is Toast by Roxio, the equivalent of Nero on Windows. For something simpler there’s Disco, one of the most visually beautiful, with a feature that keeps a database of DVDs you’ve burned to make searches easier. And a decent freeware is Burn.

• Even though Time Machine already does an excellent job of backup, SuperDuper! remains an excellent alternative. It makes a bit-for-bit image of your system, supports delta updates, and lets you create bootable external hard drives. It’s a good thing to do before anything drastic, like installing a new version of Mac OS. It’s the equivalent of Norton Ghost on Windows.

• If you’re a developer using Git with Gitk, you can install Git via HomeBrew (with HomeBrew installed, just open a Terminal and use “brew install git”) and then use Gitx instead of the ugly Gitk. It uses Cocoa components, so it has the look and feel of a native Mac app.

• If you have an iPod, know that it can play Audiobooks. Unlike mp3 and aac, the standard audiobook format is m4b, or mpeg-4 with bookmark support. That’s very important because mp3 has no bookmarks, and when a book is 20 hours long it’s painful if your iPod loses your place, or worse, if you want to navigate between chapters of a 40-hour book. If you have audiobooks in mp3, use Audiobook Builder, which compiles all the mp3s into a single m4b that your iTunes syncs to your iPod.

• You were a Windows developer with dozens of e-books in Microsoft’s CHM (compiled HTML) format. Fortunately, on the Mac you have the open-source reader Chmox.

• Say you have two computers, or some external hard drives, and you need to keep some folders in sync. Again, if you’re a solid Linux user, you can comfortably use “rsync” from the command line. But if you want a visual shell for rsync, a great option is ChronoSync, which allows bi-directional synchronization and much more. It’s how I keep two 500GB external hard drives synced through my Mac Mini, a poor man’s RAID 1 :-)

• A well-known subsystem is Growl, a notification infrastructure. For example, when an application like Transmission finishes a download, it sends a message to Growl, which pops up on your desktop to notify you. Adium also uses it to show who came online, and so on. Growl ships with plugins for iTunes and Mail.app, plus a command-line tool to integrate with other applications.

• Mail.app is organized with a list of emails on top and a view of the selected email below. Some prefer vertical bars instead of horizontal, more like Outlook. In that case use the LetterBox plugin for Mail.app.

• A famous but more advanced application is the legendary QuickSilver, which some see only as a fast way to find applications, but which is really a sophisticated environment for chaining functionality. You can, for example, find and select several files, attach them to an email, and send it to an address in your Address Book. It has dozens of plugins and themes for customization. It’s interesting at first, and there are various clones with the same idea, but in the end you’ll notice you mostly use it just to find applications. And for that, the simplest thing is Command ⌘ + Space to activate Spotlight (the magnifying glass in the upper-right corner) and type the application’s name. Same for finding documents. I find it more practical and faster.

• Those coming from Windows might still need to reach their old machine, or at the office deal with Windows programs. Fortunately, Windows has RDP (Remote Desktop Protocol) for remote access. On the Mac the best option is CoRD, the equivalent of a Terminal Client.

• A very useful utility for monitoring your CPU, RAM, disk, and network usage is iStat Pro, which has a widget version for the Dashboard, or iStat Menu for the top menu bar. I personally prefer the second, but both are very good.

• A well-known benchmarker in the Mac world is XBench, which tests your Mac and lets you compare results on their website against other Macs.

• The last tip is to create an account on MacUpdate and then download the MacUpdate Desktop software. Open it occasionally and it tells you everything that’s outdated on your machine, letting you pick what to update:

History

To understand Mac OS X, you need to understand its history. For that, the best set of articles are the ones ArsTechnica has written since 2000:

• Mac OS X DP2: A Preview

• Mac OS X Update: Quartz & Aqua

• Mac OS X DP3: Trial by Water

• Mac OS X DP4

• Mac OS X Q & A

• Mac OS X Public Beta

• MWSF: Mac OS X Post-Beta

• Mac OS X 10.0 Cheetah

• Mac OS X 10.1 Puma

• Mac OS X 10.2 Jaguar

• Mac OS X 10.3 Panther

• Mac OS X 10.4 Tiger

• Mac OS X 10.5 Leopard: the Ars Technica review

• Mac OS X 10.6 Snow Leopard: the Ars Technica review

• Mac OS X 10.6 Lion: the Ars Technica review

Also understand that Mac OS X is completely different from its predecessor, the Mac OS, which ended its life at version 9. When Steve Jobs was forced out of Apple in 1985, he founded the company Next and built the NextStep operating system, a 32-bit, multitasking OS with a Mach-based kernel. Watch Steve Jobs himself demonstrating NextStep version 3. NextStep was later opened as a standard together with Sun as OpenStep. The WindowMaker window manager is inspired by OpenStep and supports its open-source implementation, GNUStep. Later, in 1997, Apple reacquired Next, and NextStep became the foundation for Mac OS X. Its development framework was renamed Cocoa, which is why the development APIs have names like NSString and NSWindow, where “NS” stands for “NextStep.”

Currently Reading

Free: The Future of a Radical Price

The Visible Ops Handbook: Starting ITIL in 4 Practical Steps

A Study of the Toyota Production System from an Industrial Engineering Viewpoint (Sistema Toyota de Produção: do Ponto de Vista da Engenharia de Produção) – before understanding a specialization like software, it’s better to understand production in general

The Elegant Universe: Superstrings, Hidden Dimensions, and the Quest for the Ultimate Theory

Already Read

The Fabric of the Cosmos: Space, Time, and the Texture of Reality – every manager should understand relativity, nonlinearity, chaos, etc.

Linked: How Everything Is Connected to Everything Else and What It Means – the explanation of the foundation of self-organization

Outliers: The Story of Success

The Tipping Point: How Little Things Can Make a Big Difference – the practical effect of what Linked explains

Blink: The Power of Thinking Without Thinking

Turning the Tables (Virando a Própria Mesa) – the seed of democratic organizations, far more than a curiosity: it’s the only way out

You’re Crazy! (Você Está Louco!): A Life Managed a Different Way

School Without Walls (Escola sem Sala de Aula) – today’s education no longer works: the reason we don’t have motivated, proactive, creative workers.

Blue Ocean Strategy: How to Create Uncontested Market Space and Make Competition Irrelevant – another definition of “think different”

The Innovator’s Solution: Creating and Sustaining Successful Growth

The Black Swan: The Impact of the Highly Improbable – the war of Pareto vs Gauss

Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets

Wikinomics: How Mass Collaboration Changes Everything

Long Tail, The: Why the Future of Business is Selling Less of More

Lean Thinking: Banish Waste and Create Wealth in Your Corporation, Revised and Updated

Lean Solutions: How Companies and Customers Can Create Value and Wealth Together

Going Lean: How the Best Companies Apply Lean Manufacturing Principles to Shatter Uncertainty, Drive Innovation, and Maximize Profits

Beyond the Goal: Eliyahu Goldratt Speaks on the Theory of Constraints

Freakonomics [Revised and Expanded]: A Rogue Economist Explores the Hidden Side of Everything – learning to ask the right questions

Lean Software Development: An Agile Toolkit

To Read

The Great Game of Business – one of the key elements for creating democratic organizations

Open-Book Management: Coming Business Revolution, The

The Goal: A Process of Ongoing Improvement

The End of Management and the Rise of Organizational Democracy

Managers Not MBAs: A Hard Look at the Soft Practice of Managing and Management Development – I need to read at least one Mintzberg

The Balanced Scorecard: Translating Strategy into Action – still not sure if it’s worth it

Marketing Para o Século XXI – I need to read at least one Kotler

The Six Sigma Way: How GE, Motorola, and Other Top Companies are Honing Their Performance – I’m sure it’s not worth it, but I want to know it with more arguments

Conclusion

I’ll admit I’ve still read very little. There’s more research sitting in the queue, and this counts books only, without the hundreds and hundreds of academic papers, articles, and Wikipedia pages that also went into it.

Even so, I’d say everyone should read at minimum something close to what’s on this list, counting only the general-knowledge titles, not the technical ones.

My specific interest is understanding how Agile and Lean culture leads to concepts like integrated quality, zero defects, and what you read in The Goal. Especially self-organization. From there you get to Open Book Management and The Great Game, which lead to The End of Management and democratic organizations. All of it rests on what’s in Linked and books like The Fabric of the Cosmos. In the end, the whole theory closes in a simple, elegant way.

The short answer? When in doubt, go. Okay, let’s get to the longer answer.

Today’s Colleges

Like Paulino, most people have the sense that college “doesn’t prepare students for the job market.” Others say exactly the opposite: that the problem is colleges focusing only on the job market and neglecting the academic side.

For me there are two types of college. The first are the traditional, serious ones, like the federal universities and a few state ones: USP, Unicamp, UFRJ, ITA. These are seen as the ones that “don’t prepare students for the real job market.”

The second type are the commercial ones, with their fast-food two-year degrees or less, instant diplomas, do-two-courses-in-one deals. I won’t name names because they’re obvious. These are seen as the ones that “focus, barely, on the job market.”

A college’s goal should be much bigger than preparing someone for “the job market.” It’s the final stage of the formal educational process that starts in grade school and prepares a person “for life.” Preparing them to grow on their own, to value self-knowledge, respect for others, civic duty. Building a taste for discovery, for rational and independent thought, for systematic trial and error in the name of continuous improvement. Over more than ten formal years, the system should teach students to think and act on their own, aware of their own actions, with the scientific method above all.

Instead, students leave the educational system with a different proposition: I’ll do whatever I’m told to do, everything I’m told is true and needs no verification, and everything I’ve learned so far is enough to consider myself a good professional. In short, the student leaves college with their values completely corrupted. Half the blame goes to the system, the other half I place on the student, because I went through the same system myself.

The colleges considered top-tier, like USP, still have a more academic scope, and therefore a more experimental, more challenging, and more interesting one. If the student is willing, that can genuinely open doors. Students at these schools have more time, usually more purchasing power, and, with the right mindset, access to professors and material with far more depth. I’m generalizing: not all professors are good, not all students are engaged. But that’s not the norm there.

Commercial colleges have a single objective: being diploma factories. They present, in an absolutely superficial and disposable way, all the technical terms and basic procedures that entry-level jobs tend to ask for. They’re pieces of a cruel puzzle whose only goal is to trivialize education and the craft. Again, I’m generalizing: good professors and well-meaning people exist. They’re just not the norm.

The traditional educational system as a whole is built on dogmatization far more than on actual learning. Through the end of high school, every student is a memorization machine. Commercial colleges keep the same rhythm: memorize, take the tests, get a diploma. Simple, easy, no effort (I repeat: no effort).

My Experience

I’ve told this story before. About six years ago I decided to try a certain renowned commercial college, just for kicks. I scheduled the entrance exam for a Saturday, spent the previous Friday night hopping through three or four clubs with no sleep at all, and took the test literally still in post-club mode. I finished in half an hour, because it was absolutely trivial, and obviously passed. Let’s just say this college’s name is recognized by every company in the market.

It was so frustrating that I decided I wasn’t going to waste my time somewhere that easy to get into. I didn’t even bother to enroll.

Fifteen years ago I took the entrance exams for Fuvest, Unicamp, and Fatec, only those three, and passed all three. It was much harder: a full year at the same frantic pace of study, at least ten hours every day, weekends and holidays included. Thousands of exercises and practice tests, and even then I didn’t reach the top of the list.

Spending some time at USP was a great experience, even though I dropped out three years in. My reasons are personal and apply only to me, which is why I say I don’t recommend anyone copy me. I left because I was certain I could learn the rest of the curriculum on my own, and faster. But don’t get me wrong: I never thought what I was learning was unimportant. I had already been a programmer for at least five years. The first years at USP massively accelerated what I already knew and helped me take much bigger steps.

Even though I was at IME (Institute of Mathematics and Statistics), my best friends back then were electrical engineering students at Politécnica. That’s the crowd that taught me Unix: guys who played with assembler over lunch, did computer graphics in their spare time, and still had time to design microprocessors before dinner. I watched Java, Delphi, HTML, and JavaScript being born alongside those people. And I had broadband internet access fifteen years ago, thanks to the university.

That crowd left a strong impression on me: students only a year or two older than me who seemed to have ten more years of experience.

Talent and Effort

For me, college was extremely valuable and a hugely important step toward who I am today. I didn’t finish by choice. Now comes the hard part: software development is not a repeatable manual craft. Like advertising, music, literature, medicine, architecture, and every great profession, it demands innate talent. Nobody learns to be a great musician: you’re born with the talent and learn to refine it. You don’t learn to be Da Vinci, or Mozart, or Niemeyer, or Villa Lobos. Either you’re born Da Vinci or you aren’t. Sure, you can always learn the techniques, but you won’t get very far.

College won’t make you a talented person; at best it helps shape your talent. Many people discover in college that they’re on the wrong path, and many only find out much later. Unfortunately that’s common. Some people are aware they still don’t know what they want. Worse: many don’t even know that they don’t know yet.

Something else that matters: a talented person is never satisfied with what the system hands them, with what college or some course teaches. They always want more, much more. People like a Linus Torvalds, who creates a kernel from scratch. Like a Miguel de Icaza, who creates platforms from scratch. Like a Joe Armstrong, who challenges the status quo and creates a language like Erlang; like a Damien Katz, who creates different ways of handling old problems like databases, creating CouchDBs. People like Don Chamberlin and Raymond Boyce, who created the original SQL. People like Alonzo Church, who went deeper into mathematical disciplines like lambda calculus and opened up new forms of programming. People like John von Neumann, who literally laid the first brick in the foundation of computing.

Education is much more than learning a micro-trade, a series of mechanized tasks that can be repeated. There’s a place for that kind of work, sure. But if you’re reading this article, we’re definitely not talking about painting walls or stacking bricks.

Many of the best programmers were already programmers before college. It’s not a requirement, but it’s common, especially nowadays, for the urge to program to show up well before entrance exams. That makes things easier, because there are fewer doubts. In that case, invest as much as you can in the talent you believe you have and get into a college capable of shaping what you know.

The biggest mistake I see is great potential entering bad colleges just to grab a diploma. They believe they’re already good (they aren’t yet, they only have potential) and that therefore they don’t need college. That’s two mistakes. The first: a bad college makes you waste time chasing a diploma with no value at all. I personally don’t hire anyone based on a diploma, and neither does anyone truly serious who values knowledge.

The second mistake is reducing software development (including web development) to a mere set of specific pieces of knowledge. Thinking that knowing JavaScript, HTML, Linux, Apache adds up to a sufficient collection. Think about the painter again: a great painter either knows how to paint the whole work or doesn’t know how to paint at all. A good chef either knows how to make every dish or none. A musician either knows how to compose a complete piece or nothing. There’s no knowing the pieces, however well, and thinking you know enough.

The Worker and the Artist

That mistake is fatal, because the individual will readily believe they know enough and give little or no attention to everything else. Software development is an art like any other. You’ll never be perfect in all of its aspects, and the peculiarity of all these creative fields is precisely that they prize the “pursuit of perfection.” What counts is the rung above the one you’ll be on tomorrow, and the wisdom that this ladder has no end. What matters is the journey, and the journey in pursuit of perfection is the engine of continuous improvement.

As they said of Socrates: “the more I learn, the less I know.” Stop for a moment and think about that again: most of us forget this principle. Diplomas and certificates do harm precisely because they discourage the long-term journey in exchange for false short-term trophies.

You think you really don’t need college? Do you know probability and statistics? Because without them you can’t build the most advanced anti-spam and anti-virus mechanisms. Think you understand linear algebra and calculus? Without them you’ll never create anything in computer graphics. Do you really think you understand algebra? Without it you’ll never grasp how cryptography mechanisms work. Did you actually learn boolean logic? Without it you don’t understand the foundation of modern computing. Think quantum physics is irrelevant? Without it you can’t even begin to imagine the possibilities of future quantum computing. Before the cynics take this too literally: I’m not asking for expertise in any of these areas, and I don’t have it myself. But I know where to look to solve each problem involving each of them.

Plenty of people find all of this very irrelevant. After all, why would a programmer need to know all this nonsense to be a mere “form coder”? The thing is, in the end, the bulk of “the job market” is building forms: forms in Visual Basic, forms in HTML, it’s all forms. For that I don’t need Da Vinci; a house painter will do.

If your purpose as a “programmer” stops there, this article ends here and I’m wasting my time. I build forms too, but I’m not content with that and I keep inventing different ways to build them. A Da Vinci doesn’t choose or plan when or how his Mona Lisa will come out. It’s the result of a long, laborious, difficult journey. The Mona Lisa began to be painted in 1503, he put the final touches on it until shortly before his death, around 1519, and that journey took him from Italy to France in 1516. It only started becoming truly famous in the mid-19th century. He made hundreds of other paintings along the way, some better known, some ignored.

On top of that, knowing every aspect of the profession makes even the task of “coding forms” more interesting. A mediocre coder will code forms the same way for the rest of his life. A good programmer always looks for different approaches, experiments with new techniques, refines the ones he already knows, exactly as they did in the Renaissance. That’s where the whole school of software architecture came from, the best practices, the best architectures, the entire process and discipline later taught as rote memorization to the mediocre.

Data structures, basic algorithms, relational databases, compilers, assemblers, operating systems, protocols: these are all disciplines that should be considered trivial. The challenge lies elsewhere. For an artist it’s the same as knowing how to hold a brush or handle a piano’s keys: what makes the difference is how you use them. That takes talent, that takes experience. Even the most talented person will need years of practical experimentation, thousands of hours of practice, trial and error, refinement, before reaching expert status. A good programmer should aspire to become a polymath.

By polymath I mean knowing programming, some administration, some marketing, general culture, communication, and various other areas that add to your main craft. The current “social networks” craze makes much more sense once you become familiar with the work of scholars like Albert-László Barabási on scale-free networks, a discipline that didn’t originate in computing but precisely explains the evolution of this kind of natural formation. For those leaning toward a management career, understanding Frederick Taylor gives you another perspective. More “modern” topics like agility or lean make much more sense after you’ve seen the work of others like W. Edwards Deming or Taiichi Ohno.

My point: you won’t pull anything useful out of your own head alone. Forget that idea. You only sharpen what you know, at genuinely high speed, by standing on the shoulders of giants. In my case, I think I just absorbed that insight and consciously changed my attitude by going through college, in an environment that let me experience my own smallness before decades of cultural and academic evolution. You need to feel like a dwarf in order to pursue surpassing giants.

Today’s traditional and commercial environments of teaching and work are unfortunately terrible places for that, I’d say the worst in the world. In them you live the illusion of achieving more without pushing for more, in a predatory environment that rewards the fast and the superficial. Fast-food culture is completely hostile to the concept of continuous improvement, which by definition depends on time and disciplined effort. Today everyone wants a short college that’s easy to get into, easy to pass, preferably cheap and low-effort. Courses and certificates follow the same path: in no time you get to call yourself a “sage” in some field. Instead of a meritocratic environment that values learning and wisdom, we have an opportunistic one that values smallness, minimal effort, and low culture.

A Recipe?

With all of that:

• If you’re still of school age, in high school, do everything you can to figure out what you think you like. You’ll rarely be certain, but you need to try.

• If you’re past school age and still don’t know what you want to do, I’m sorry: few people leave that condition for a position of distinction. Rare isn’t impossible, but in that case understand that you’ll need to work much harder than average, much harder than most naturally talented people. It will literally cut to the bone, it will hurt, and there’s no guarantee of any return.

• If you know what you want, go for it. But really go. Understand that the journey is long and has no end, that you’ll never reach the top, and that there will always be many people above you, with the aggravating detail that they keep climbing fast. The reward: if you engage with the journey, you’ll never have to separate work from fun. Let others decide whether what you’re doing is work.

Now the most complicated part of the equation: few people have the conditions for any of this. Most aren’t just opportunists; they’re in a situation of real financial distress, sometimes with even graver family problems. It’s that terrible situation of having to choose between what you want and what others want, where either choice will be a bad one.

I have no magic solution for that, nor do I have that life experience. But I know up close, in detail, cases of people who managed to escape the negative cycle and became far more successful than me. There’s a price: it will hurt. No great reward comes for free. No great value comes in the short term.

What I see most are people wasting a lot of time feeling sorry for themselves. I hate that type, acting like the victim of everything: the system is to blame, the company, the parents, everybody but themselves. There are always people in much worse situations than yours. Never assume you’ve hit rock bottom, because you have no idea how deep that pit goes. And if you’re here reading this blog, I can assure you that you’re very far from that bottom, so you have no excuse.

The old saying still holds: “no pain, no gain.” The important thing is not to be a conformist. Most people who criticize me when I say these things start from the wrong premise: they separate what is work from what isn’t. That person already is a conformist: they’ve already accepted that their job is something they don’t want to take home. And I’m not talking about taking paperwork home to do at midnight. I mean the journey, the mindset, the thinking, the reflection. What’s the right balance? Each person knows their own.

Outliers

As Malcolm Gladwell tells it in his recent Outliers, the Beatles went to Hamburg five times between 1960 and 1962. They played eight hours a night, seven days a week, trying to win over an audience that didn’t speak their language and was more interested in the strippers. That was 270 nights in Hamburg. When “I Want to Hold Your Hand” finally broke in America in January 1964, the Beatles had already performed some 1,200 times. That’s far more experience than most new artists accumulate in a lifetime.

You don’t need to become the Beatles; that’s nearly impossible. Nor a Bill Gates, a Steve Jobs, a Ray Kroc, a Henry Ford. It makes no sense to try to imitate them, follow their footsteps, take the same courses, do the same things, wear the same clothes. It’s futile and it gets you nowhere: a biography comes nowhere near being a recipe for success. You need to absorb what drove them to the journey, not what they did during it.

What they all have in common is talent, a lot of hard work, and a lot of experience. Mozart started composing at six, but it was only after his twenties that he finally composed a work of art: more than ten thousand hours of practice later.

So I return to the question: “Is college worth it?” Certainly, if it’s a good college, with the intent of sharpening what you know and learning more. Very much worth it, as long as you understand that college is only a fraction of your journey of at least ten years of learning before you can finally say you know something.

My journey in computing began timidly in 1986 and picked up steam from 1989. My “formal” résumé starts around 1997. I’m still very much a novice in many areas I’d like to know more about. My journey has barely begun. Don’t think yours is already over.

Do You Need a College?

After all of that, you don’t expect an easy answer, do you? :-)

To be mediocre, by definition doing what everyone else does, yes: you’ll definitely need a college, any college, grab your diploma, and be just another one in a crowd of millions. Good luck, you’ll need it.

To be above average, the only path worth taking, it depends. If you think you can learn everything on your own, that works too. I know several cases like that, of extraordinary people who never went to college (or went to a bad one, just for appearances, without letting it influence them) and who know every aspect of the field. Understand: knowledge is not a secret. There’s nothing you’ll only see after opening some secret door inside a college. All the information is out there. It’s up to you to decide whether you’ll go after all of it on your own. If you don’t think you will, go to college, but a decent one, not one of these disposable commercial operations on every corner. I think it’s worth it.

But if the question you really wanted to ask was “Is college decisive for a successful career?”, then the answer is no. Success depends solely and exclusively on the individual. If you succeed, the credit is all yours; if you fail, the blame belongs to no one else either. And once more, for the cynics (whom for some reason I suspect are intelligent and love pulling things out of context): no, nothing I said here is new. I’m just repeating what should already be common sense.

In recent posts I explained why “mediocre” thinking is a problem and why there is no such thing as “average” in the real world.

Some readers already asked me for book recommendations, and I dropped a few important ones in recent posts. The Black Swan is at the top of the list. It explains how we ignore “black swans” and why trying to predict the long-term future from past events is an exercise in futility. The core point: we instinctively “reduce” and “simplify” everything to make it digestible, and that is a problem.

Now I want to explain how the modern hero has trivialized the great myths.

Modern Heroes

Another book I recommend, not directly related to the subject, is The Power of Myth by the great Joseph Campbell. It’s a fantastic book and it describes, among other things, the Hero’s Journey. Every civilization had its myths, populated with heroic figures.

The short version: the Hero’s Journey starts with someone just like most of us, at the “average” or below it. The road is packed with danger, temptation, and trials. In the end the hero completes the journey and returns to preach what he learned. It’s one reason myths like Buddha’s or Jesus’ are still told today: they are heroic allegories baked into every culture.

Every Star Wars fan knows George Lucas is a Campbell disciple. Luke Skywalker’s story is the hero’s tale beat for beat: a poor kid from the backwaters of Tatooine faces trials and the temptations of the Dark Side of the Force until he transcends himself and returns to his people as a hero. It’s one reason Star Wars is a classic to this day.

The modern world still has its heroic stories. These days some of the heroes are successful businessmen: Steve Jobs, Jack Welch, Henry Ford, and many others.

We all like having these “heroes”. Their stories inspire us and, as a side effect, sell mountains of books and merchandise, like courses and certifications.

Today’s self-help is a poor man’s version of ancient mythology. And in this market bent on manufacturing the next heroes, pitiful copies fight over the spot of the next Perseus.

Charlatans

All this hero talk was the setup for the point that matters: how easily charlatans fool us, and how to break the cycle.

Ask Campbell, but the need for heroes to model ourselves on is probably in our nature. Nothing wrong with that: the names I mentioned above are reasonably worthy of the “modern hero” title.

The problem is the charlatans selling themselves as heroes, and plenty of them pull it off. Today’s “Leader” is yesterday’s “Hero”. I searched for “leader” on Submarino’s site and the results alone show the size of the gold rush: a flood of books on the same theme, “How to become a leader”. Probably the best-selling category in corporate self-help.

First things first: self-help is a giant exercise in futility. If you can avoid it, avoid it, because almost all of it is a waste of time. Publishers were never good filters, so there will always be a mountain of worthless books. The job is separating the wheat from the chaff.

Plenty of people say you should read a lot of books, and they’re right. They just forget to add that not every book counts. Reading bad books makes you dumber; if those were the only books around, I’d tell you to stop reading. Remember Pareto: maybe 20% of all books are worth anything.

A book titled “How to Be…”? Skip it, waste of time. To see why, ask the most obvious question: did any of history’s great leaders need a self-help book, course, or lecture to become memorable? Can you picture John Kennedy, Martin Luther King, or Abraham Lincoln reading “How to Become a Leader” backstage? Hardly.

Now the second question: how many people around you actually became good leaders because of that material? I’ll go further: there are dozens of project management “methodologies” promising miracles, and everyone buys in.

The Great Fallacy

The logic goes like this: you apply the technique or methodology and, if the project succeeds, the method gets all the credit. “Of course it worked, thanks to the revolutionary methodology.”

If the project fails, it’s your fault, because you obviously didn’t follow the methodology properly. Had you followed it, everything would have worked!

See the absurdity? All those management, leadership, and results books and courses hide this exact trick: they list dozens of people and companies that benefited from their “techniques”. I’d bet those projects succeeded because of the quality of the teams, the people who executed them. The methodology only showed up afterwards to take the credit.

These were teams full of people able to route around every obstacle, who subverted the “methodology” until they hit the goal. They would have gotten there with or without the recipe, and probably sooner without the magic-formula baggage.

I personally like the Agile philosophy. Many people prefer to say Agile “methodologies”, but I prefer “philosophy”. Methodology implies a systematic recipe you can repeat in any scenario, which isn’t true. If the team lacks awareness and discipline in the philosophy, you can implement the techniques all you want and the project will never be truly Agile. Writing on sticky notes, slapping them on the wall, and timing 15-minute meetings doesn’t make any company agile.

In the modern age we think we’re smarter, sharper. Are we? We chase the heroes who beat the odds and, as Nassim Taleb would say, we want to take their experience and their recipes and apply them to our own projects. In other words, we plan our future based on the past success of the people we call heroes.

Worse, we still “adapt” those recipes, convinced we know better. Take the reputation of the so-called “Waterfall process”, the staircase process, considered one of the worst ever created for software development.

Waterfall was described in the paper Managing the Development of Large Software Systems, from 1970, by Winston Royce. That’s the paper with the famous staircase diagram. What few people know is that Royce himself wrote in it: "…the implementation described above is risky and invites failure." He already proposed running the process at least twice, which means he knew it would evolve into something iterative.

We love leaning on past successful projects, on masters and scholars with magic solutions. The vast majority simply doesn’t do the homework and falls for the fairy tale like naive kids.

Here’s another thing almost nobody questions: has anyone empirically checked how many of these training-formed managers go on to have consistently successful careers? I never met one. I also never saw a single credible number on it, from a credible institution, in any relevant independent study.

Manufacturing Managers

I’ve sat through every kind of lecture and course on managing projects, dealing with people, being a leader. In a Gaussian world, where the whole educational system incentivizes the average, students hit the job market and suddenly there’s a shortage of people above average.

Of course there is. We raise people in a paternalistic, protectionist way, the worst way to raise anyone. In school we only study what we’re handed, only do the homework we’re assigned, only study to scrape a passing grade. We’re so used to taking orders that the outcome is plain for everyone to see: excellent factory workers, few thinking professionals, and very few great leaders.

I like to say that “leaders aren’t formed, leaders form themselves.” The way to encourage future leaders, starting in childhood, is to rip out the average mindset and raise them in a meritocratic environment: one where merit gets rewarded far more than mediocrity. And when I say “far more”, I mean absolutely unequal.

A democratic, evolving civilization was never about forcing equality, artificially lifting the weak and dragging down the strong until everyone lands on the average. “Equality” simply means nobody will stop you from rising above the average. Most people confuse that with making everyone the same.

The correct solution is the opposite: keep a high level of inequality based on merit. That’s how the real world works: the successful are orders of magnitude wealthier than those who barely try. Wealth in capital, but also in recognition and influence.

Companies keep doing the opposite: they rerun the grade-school playbook, stick employees in a classroom, and expect great leaders to walk out.

I know how this works: I’m a PMP-certified Project Manager, certified by the PMI, the “renowned” Project Management Institute. And I’ll say it out loud: it’s one more of the great corporate lies so common these days.

It works like this: you’re required to sit through 2 or 3 months of classes to learn the “trade” of being a manager. You learn to break a project into a WBS, build a Gantt chart, calculate slacks, draw a project chart, analyze risks (in Gaussian fashion!), and so on.

You get a workbook, you do homework every week, just like grade school. At the end comes the “entrance exam”: the certification test, 200 questions. Beat the cut score, barely more than half, and you get a handsome PMP diploma, a lapel pin, a signature template for your business card, and the right to sign your emails “Fabio Akita, PMP”.

Does any of that make you a better manager? If you were truly awful, at least you climb out of absolute awfulness and land near the average. Don’t be too proud of that.

But does it create good managers? Absolutely not. At best it proves you can read well enough to get slightly more than half the questions right in 4 hours, which proves exactly nothing. After the brainwashing and the rote memorization, anyone gets good at passing tests. Anyone who survived a cram school knows: a management course is the same deal.

I’ve met many project managers, more than I’d like, who are notoriously terrible. And they passed the PMP exam! When you get certified you’re happy, of course, I was. But if you’re smart, you start feeling embarrassed about the title once you see the other bad PMPs out there. That’s when I stopped using mine, even after a process that cost me around R$ 5,000 (a piece of paper doesn’t come cheap), plus dozens of hours of mock exams.

Oh, and we become Leaders with an expiration date. Mine expires in 2009…

Manufacturing Leaders

Anyway, if your resume is weak and you know you can’t deliver, go ahead and get certified in as many things as possible. The certification market exists because people never learned in school how to actually learn.

Everyone only learned to study what they’re handed and to pass the test, at the average. I’ve met few truly self-taught people with real street smarts. Those are exactly the ones who couldn’t care less about certifications. A very smart acquaintance of mine, one I unfortunately had little contact with, put it to me straight: “good people don’t get manufactured.” Indeed.

And by the way, acing a test and being intelligent are two very different things. You can get perfect scores across the board and still be far from smart. I’ve known plenty of straight-A nerds who turned into slightly-better-than-average factory workers.

Our concept of the hero has changed a lot. For me, the Hero figure was always a strong influence on how I act: outliers who single-handedly went from zero to empire. I always admired the great Renaissance artists, like Leonardo da Vinci, who broke barriers as a painter, sculptor, inventor, philosopher, anatomist, physicist, and mathematician. And other giants like Thomas Edison, Benjamin Franklin, Carl Sagan, Stephen Hawking, and Charles Babbage, all of whom I met reading encyclopedias in junior high. Yes, encyclopedias were not made to be decoration.

Luckily, my upbringing always pushed me past the average. If I came home with a 9, my mother would say: “the test goes up to 10; if you’re missing 1 point, you didn’t try hard enough.” If I got a 5, the average, I was grounded. My father, for his part, always prized self-knowledge: he bought books and science magazines, and once gave me a world globe for my birthday.

You’re a company and you want to grow great leaders? Put big challenges in their hands, push them to the limit, squeeze out everything they have. Be a tyrant without being a dictator. Demand perfection, and know how far each person can go. Only those who overcome extraordinary obstacles are extraordinary people. Leaders are Heroes, by definition!

Now, you don’t earn “leader” from a job title. Subordinates’ respect through sheer hierarchy is worth nothing, zip, zero. A structure where the only form of command is the old “chain of command”, where the subordinate obeys because his job depends on it, is a company extracting a tiny fraction of what it could. That’s why I’m against hierarchical organizations. As I mentioned in the other post, hierarchies are one more Gaussian form of control through the average, and it doesn’t work. Insisting on the mistake only makes everything more wrong.

The Death of Heroes

Here is the problem with Gaussian thinking: in a world that incentivizes the average, there’s little room left for Heroes. By that logic, the heroes who do exist are “accidents”: Gaussian probability says the odds of a hero appearing are so small they’re nearly zero, something no “normal” person could ever reach. Not even worth considering. The “safe” move is staying at the average.

Except heroes exist, and new ones show up every day. Open Forbes and look at the list of the world’s richest people. Take the top 500: the last name on that list is already infinitely richer than any of us. Stretch it to the 10,000 richest and the last one is still far richer than anyone you’ve ever met. Keep going down and you realize there are thousands of small heroes: people who left the “normal” path and won. And not only in money.

There’s another problem in the world of the average: plenty of people know the “trick”. People would even like to be heroes, but it’s work. Everyone knows the hero’s path takes a monumental, highly risky effort: the middle of the Journey is packed with trials, dangers, and the devil’s temptations. Nobody likes suffering for an uncertain future. People are naturally short-term thinkers; they love to “say” they plan for the future, but that’s just rhetoric.

And the people who know this trick best are precisely the ones who invent the magic recipes, the methodologies, the step-by-step procedures for becoming a hero in 24 hours, no sweat, no effort, no risk. For a small fee and a few hours of your day, anyone becomes a hero, certificate included! Companies hire these fast-food heroes, hand them command positions, and subordinates obey through the sheer force of the title. Could it get any easier?

As I said, there’s no empirical data proving these methods work. Companies simply “believe” they do. The trainers’ pedigree looks good, other companies swear by it, the references are great. Why doubt it?


Everyone falls for the fallacy of showcasing only the projects that succeeded, where the people did the heavy lifting, and “forgetting” to list the ones that applied the methodology and sank.

It sounds so smart when the trainer says: “as a Leader, you need to motivate your team.” He just forgets to explain, step by step, how you actually “motivate” someone. Of course nobody explains it. Worse: as Taleb describes, people get reduced to stereotypes and roles, inanimate puppets that, according to the magic recipe, are predictable and easy to control. Just “motivate”, just “incentivize”, just “be communicative”. Building attractive, empty rhetoric is easy. Actually leading a project to success is hard.

An acquaintance of mine used to say: “He who can, does. He who cannot, teaches.”

And according to Pareto, that’s normal. At least 80% of the population, companies, and professionals will think this way. Great news for the other 20%: less competition.

The world was not made to be fair or equal. The real world is, by nature, highly unfair and unequal. In it, true heroes are battle-scarred, thick-skinned from taking so many hits, and they rise solely and exclusively through their own effort. The real world runs on meritocracy. Many will fool themselves for a while, thinking they’ve found a place that will stay stable and equal forever, but that period is temporary and doomed to disappear, replaced by a merit-based system that will crush them without mercy. Most people are terrified of that, terrified of losing their comfort zone.

All these cake recipes dressed up as self-help, masquerading as serious courses, are cheap mythology in homeopathic doses. I won’t describe them as anything more than what they really are. They all come loaded with “advice”, and the best definition of advice I’ve ever seen is in the famous Sunscreen video:

“Advice is a form of nostalgia. Dispensing it is a way of fishing the past from the disposal, wiping it off, painting over the ugly parts and recycling it for more than it’s worth.”

Campbell’s Heroes don’t live in the comfort zone: leaving the common path is the hero’s routine. The journey begins exactly when he decides to challenge what everyone else considers impossible. The hero becomes a myth when he conquers the impossible, and the impossible never yields to “normal” paths.

Myths are rare, unfortunately (or fortunately).

Go is an ancient strategy game with simple rules and a profound degree of complexity.

Software development is the art of managing complexity using a limited number of rules, structures, and patterns.

Programmers should play Go.

Go in 2⁸ words or less

The beauty of Go is its combination of simplicity and complexity. On the one hand, go has only a handful of rules. Place stones, don’t get completely surrounded, control territory. Like chess, the mechanics can be picked up in a few minutes, though Go only has a single type of “move”, and only one edge case (the ko rule). And like chess, one can spend a lifetime discovering the strategic and tactical layers of the game.

While chess is quite complex and rich, such that it took a 30-node supercomputer to defeat the reining chess champion, no computer comes close to defeating even a skilled amateur Go player. There are 361 positions on a Go board, and with two players, there are 2.08168199382×10¹⁷⁰ valid positions. That’s quite a bit bigger than a googol (yes, that is the correct spelling). Realistically, there are something on the order of 10⁴⁰⁰ possible ways that a typical game could play out. And the number of possible moves roughly follows 361!, which means that only 20 moves in, there are many googols of possible ways that the game could shake down. (As a fun exercise, try plugging 361! into an online factorial calculator.)

Managing complexity

So how does one play Go, given this near-infinite complexity? On a tactical level, a player approaches Go like chess, thinking several moves ahead. But this only works in small spaces, like a tight battle in a small sector of the board. Beyond there, there are just too many possibilities. So on a strategic level, a player must think in shapes or patterns. These shapes provide shorthand ways of managing the complexity of Go. As a non-master, I may have no idea how things will proceed in one sector of the board, but I may be able to recognize strong and weak patterns of stones, vulnerable shapes and effective formations.

But there’s more: Go has several sorts of patterns. Beyond shapes, there are Go proverbs. These can be general: “Your opponent’s good move is your good move”; specific: “Don’t try to cut the one-point jump”; funny: “Even a moron connects against a peep”; and meta: “Don’t follow proverbs blindly.” These proverbs are principles which help a player make good decisions. They are less specific than shapes, and so they provide guidelines for whatever situations may arise on the Go board. Proverbs often conflict, and a player must determine when and how to apply them.

Finally, there are joseki. Joseki are patterns of play that are considered even for both sides. They typically happen in the corners of the board, and typically at the beginning of the game. Interestingly, there is a Go proverb that says “Learning joeski costs two stones,” meaning that memorizing these patterns isn’t helpful. Instead, a player should learn from joseki by understanding what is going on in each move.

Patterns in Go, patterns in software design

Each of these Go patterns has a rough programming analogue.

Shapes in Go aren’t unlike software design patterns. While there is nothing preventing you from placing logic in your views, this shape is recognized to be a weak one. Think of Gang-of-Four design patterns: the MVC, Adapter, and Factory patterns are recognized to be helpful in some circumstances (and not appropriate in others). On a lower level, iteration and recursion have commonly recognized shapes, as do database normalization vs. denormalization. Even if you can’t hold an entire program or algorithm in your head at once, recognizing common shapes helps you to understand what is going on.

Go proverbs are like another type of pattern in software: CapitalizedPrinciples (for lack of a better term) made popular by Extreme Programming. Think DontRepeatYourself, YouArentGonnaNeedIt, CollectiveCodeOwnership, DailyBuild, TestFirst. These aren’t specific code “shapes”, like a singleton class – they are general principles that guide the practice of programming.

Because joseki is about exchange between competing parties, its programming parallel is a little less clear. The closest comparison, in my mind, is programming exercises. This article, for instance, suggests 9 exercises to help you become a better OO programmer, like:

• Use only one dot per line
• Use only one level of indentation per method
• Don’t use setters, getters, or properties

In a real-world program, you’re unlikely to stick to these principles 100% of the time. But forcing yourself to write code in this way can be an eye-opening experience and can make you a better developer.

So what can Go really do for you?

Obviously, these parallels are structural. Specific Go proverbs (“Your opponent’s good move is your good move”) may not have direct relevance to software development. So can Go really make you a better developer?

I think it can, and I’ll go one further. I think Go can make you smarter. There is a lot of anecdotal evidence to this effect 1, 2, 3, for example, 4:

In fact, all of our minds can benefit from playing Go, which officially has the capacity to make you smarter. Research has shown that that children who play Go have the potential for greater intelligence, since it motivates both the right and left sides of the brain.

The research mentioned isn’t footnoted, unfortunately, so take statements like this with a grain of salt.

But it makes sense: like chess, Go requires pattern recognition, a mix of strategic and tactical thinking, and comprehension of complex structures, though in Go the patterns are larger and the complexity is greater. A mind trained to think in these ways is going to have an easier time attacking similar problems in other spheres.

Like software development.

Akita’s Notes

I’ve liked Go for a long time. As someone of Japanese descent, of course I’d been exposed to Go, Shogi, and other oriental games. But, like any good westernized Japanese, I paid very little attention to them. Because of this, today I know no more than the basic rules and philosophies behind Go.

My interest was particularly renewed when I read the Hikaru no Go series, which is an entire story built around Go, and one of my favorite series, by the way. I even bought a Go board and some joseki books, but didn’t get very far. These days I just peek at online games on the IGS servers.

I don’t remember where I read this, but someone once mentioned something interesting: chess is primarily a game of destruction. Go is essentially a game of conquest and expansion. Not literally true, but there are nuances in Go that bring this to mind.

And in the parallel to software development, design patterns aren’t just structures you copy and paste wherever you “think” they’re needed. A handful of design patterns doesn’t make good software.

What Jon was getting at makes a lot of sense. Just like a Go player, a software developer needs to be an artist. Playing is a creative activity. Strategy is a creative activity. Given a set of constraints, what’s the best path forward?

More than that: the only way to learn Go is by playing, hundreds of thousands of times over many, many years. Go professionals are formed around age 10 and climb from there into old age. The only way to learn is by failing, failing, and failing again. Which circles back to what Ryan said about Hurting Code.

Refactoring is something similar to this: conquest and expansion. Only a good developer understands the real reasons for refactoring. Nobody refactors because someone told them to, just as an artist doesn’t make a certain stroke because someone told them to.

A good developer values the form and philosophy of what is being built. Acronyms, brands, and fashionable names say nothing to them.

There’s a lack of eastern philosophy in today’s programmers. Being a brick-stacking coder is very easy: any grunt can do it. But reaching the 10th dan of programming is only for those who have put in the work in the art.