So here’s the thing. In three weeks this little project grew in a way that deserves a post about the process alone, because it became a living example of something I’ve been preaching for years: real software is grown, like a plant. Nobody builds good software piece by piece the way you snap a Lego together. Let me show you the numbers, and mostly the story they tell.

The numbers, today

ai-memory has, in 24 days of life (from the first commit on May 21 until now):

• 482 commits on main, averaging 20 a day.
• 26 people committed to the repo.
• 10 Rust crates in the workspace.
• ~62.7k lines of production Rust, plus ~11k lines of tests in tests/, more than 1,000 test functions, and 90 #[cfg(test)] modules scattered through the code.
• ~19.7k lines of markdown documentation (docs/).
• 70 pull requests, of which 59 merged (84% acceptance).
• 27 issues, 100% closed.
• We cut v1.0.0 on June 12 and we’re already on v1.0.6.

Twenty-four days. Hold onto that number, because the interesting part isn’t the final size, it’s the way it got here.

How it started: one of everything

ai-memory was born small and dumb on purpose. In the first two days I nailed down the whole skeleton: workspace bootstrap, MCP server, hook capture, cross-agent handoff, git wiki versioning, an LLM provider trait, embeddings. All of it running for one user, one machine, one workspace, one architecture (Linux, basically). No multi-user, no permissions, no scopes, no Windows or macOS support. It was my agent brain, on my machine, for me.

On day 3, when the first external contribution landed (a fix from Lucas Oliveira for OpenAI-compatible provider URLs, PR #6), the project had 102 commits and around 22.9k lines of source. Already the same 10 crates as today, but with a fraction of the functionality. It was a one-of-everything MVP that worked for me and nobody else.

And this is where most projects die, or worse, where most inexperienced programmers get it wrong.

The Lego fallacy

There’s a widespread belief, especially among people starting out, that you can design everything a piece of software will be before writing the first line. You sit down, plan the entire architecture, break it into neat little pieces with all the folders and layers and interfaces mapped out, and then build it piece by piece. When the last Lego brick clicks in, the software is done. Beautiful on paper.

In practice, it’s the fastest road to rotting code. The sooner you crystallize a structure, the sooner it becomes a straitjacket. You predicted an elaborate permission system before having any users? Congrats, now you carry the weight of maintaining an abstraction nobody uses, and that’s almost always wrong, because you designed it guessing the future instead of answering a real need. The inexperienced programmer over-engineers at the start, and no matter how much they plan, the result starts to rot fast, because they’re solving problems that don’t exist yet and getting wrong the ones that do.

I don’t believe in software that finishes. I already wrote about this in “Software Is Never Done”, and it connects straight to the original Agile methodologies, the real ones, before they turned into a corporate post-it ritual. Good software is a clay sculpture: malleable, always wet, that you keep adjusting and reshaping, and that never gets finished. It’s clay you mold while it’s being used. The Lego tower that one day clicks into place and is done is a fantasy.

What the contributors built

And this is where the best part of open-sourcing it comes in. That one-of-everything MVP became a cross-platform, multi-user system because a lot of people put their hands on it. I want to thank them by name, because without these people the project would be a fraction of what it is:

Djalma Júnior drove a big chunk of the heavy architecture (multi-user, admin, cross-project recall), with 25 merged PRs. wlech with dozens of commits. Pedro de Freitas Jr on embeddings and web fixes. Mauro Couto de Paiva and Gustavo Cateim drove a big part of the Windows support. Pablo Winter, Lucas Oliveira (author of the first external PR), and Said Rodrigues, among many others. Thank you so much, folks.

What those contributions added, the highlights:

• Real Windows, macOS, and Linux support. What was Linux-only got macOS bash 3.2 compatibility, an NTFS file index for tracking inodes on Windows, and native per-platform hook commands.
• Frontend API (/api/v1) and a customizable web UI. It stopped being only MCP + CLI and got a real HTTP surface, with clickable wiki links and search by file path.
• Multi-user with attribution. A users table, writer/reader commands, and write attribution in the same SQL transaction as the data (pages.author_id + frontmatter).
• Multi-workspace and cross-project recall. Global search, links between wikis of different projects, and active-project isolation per session and per actor.
• More providers and more agents. Gemini, Copilot via OAuth, OpenAI-compat strict mode; plus support for Antigravity, Grok, and VS Code Copilot on top of the originals.
• Operational robustness. Checkpoint recovery, reindex-from-wiki to rebuild the index from the markdown, self-describing _meta.md manifests.

Notice something: none of this was in the original plan. There was no original plan. Each of these features landed because someone needed it, opened an issue or a PR, and the structure gave way to make room. Multi-user wasn’t predicted: it emerged.

And only now did we stop to consolidate

Here’s the central point of the post. Over these three weeks, the code grew a little messy on purpose. When multi-user support came in, the logic for “which workspace and project is this request aiming at” ended up embedded right in the middle of several handlers in the MCP server, the web API, the admin, scattered and duplicated. Each spot resolved scope its own way. There was duplication. There was code that wasn’t pretty. And that’s fine, because it was working, and working for a lot of people.

Only after the software was already running on dozens of people’s machines, across three operating systems, with real multi-user in production, did I stop and say: “okay, now is a good time to refactor and rethink the internal wiring.” The second-to-last big commit, refactor: centralize scope auth and wiki consistency, did exactly that, with 1,239 lines added and 249 removed, creating a central 601-line ScopeResolver and a consolidated ActorContext.

Look at the “before.” This scope-resolution snippet lived duplicated, with variations, across several methods:

// BEFORE: scope logic embedded, repeated in every handler
let active = self.active_project.get_for(actor);
if let Some(name) = explicit_project.map(str::trim).filter(|s| !s.is_empty()) {
    if let Some((active_ws, _)) = active
        && let Ok(Some(pid)) = self.reader.find_project(active_ws, name.to_string()).await
    {
        return Ok((active_ws, pid));
    }
    if active.map(|(ws, _)| ws) != Some(self.workspace_id)
        && let Ok(Some(pid)) = self
            .reader
            .find_project(self.workspace_id, name.to_string())
            .await
    {
        return Ok((self.workspace_id, pid));
    }
    return Err(McpError::internal_error(
        format!("project '{name}' not found in the active or default workspace"),
        None,
    ));
}
Ok(active.unwrap_or((self.workspace_id, self.project_id)))

And the “after.” The same resolution, now delegated to a single authority in the store crate:

// AFTER: one central resolver, one call
fn scope_resolver(&self) -> ScopeResolver<'_> {
    ScopeResolver::new(&self.reader, self.workspace_id, self.project_id)
        .with_writer(&self.writer)
        .with_active_project(&self.active_project)
}

// ... and each handler becomes this:
self.scope_resolver()
    .resolve_current_or_project(explicit_project, actor)
    .await
    .map(ai_memory_store::ResolvedScope::as_tuple)
    .map_err(Self::scope_error)

Three different methods that each used to carry 30 lines of embedded logic, with lookup_ids, workspace fallback, and their own error handling, collapsed into one call each. Alongside came the sibling commits: harden multi-user admin auth boundaries and prevent scoped fallback bleed. Centralizing permissions, workspace and user scope, and wiki consistency, all in one place, with a clear rule.

The detail that matters: ScopeResolver wasn’t invented at the start of the project by guessing there’d be multi-user one day. It was extracted from code that already existed, already worked, and had already shown, in practice, which patterns kept repeating. The abstraction came after the usage, not before. I let the structure emerge from the code, and only once it was obvious did I consolidate it.

The right order: make it work, make it right, make it fast

This is the summary of everything. There’s an old saying, attributed to Kent Beck, that I carry like a mantra: “make it work, make it right, make it fast.” Make it work, then make it right, then make it fast. In that order. Most inexperienced programmers want to make it “right” and “fast” before making it “work,” and so they invent rules with no code to justify the rule.

ai-memory followed the right order without me even thinking much about it. First I made it work for one person, simple scope. Then I made it work one feature at a time, refactoring and hardening the code along the way, but never falling into over-engineering and never rewriting from scratch to fit a preconceived structure. I let the current structures emerge from the code that was being added. That meant living with some duplication and some so-so code for a while, until that moment hit of “okay, now it’s time to refactor and rethink the wiring.” And only now, with the software mature and in use, did I establish better patterns and rules for new code. Rules backed by real code, not by guesswork.

KISS, “keep it simple,” is about not solving a problem you don’t have yet. It has nothing to do with writing little code. It’s about letting architecture emerge from the code, not the other way around. You don’t design the sculpture and pour the clay into the mold. You take the clay, start shaping, and the form appears as your hands work. The mold, if you ever need one, you build later, around the form that already exists.

Twenty-four days, 482 commits, 26 people, and none of them followed a diagram drawn on day 1. Because there was no diagram on day 1. And that’s exactly why it worked.

ai-memory is open source and keeps changing every week: github.com/akitaonrails/ai-memory. Want to help shape the next piece? There’s an open issue waiting.

Then it became an industry. Pinecone, Weaviate, Qdrant, Chroma, Milvus, pgvector, LangChain, LlamaIndex, Haystack. Tutorials everywhere, “build your chatbot with your PDFs,” entire consultancies feeding off this. It became the “hello world” of applied LLMs: document → chunk → embed → vector DB → query.

Today, in April 2026, Claude Opus 4.6 has 1 million tokens of context. Sonnet 4.6 too. Gemini 3.1 Pro too. GPT 5.4 has a smaller window but still in the comfortable range, in the hundreds of thousands. And some models already have experimental 2M token modes. The question that keeps nagging at me: what on earth do I need a vector stack for, to solve a problem that fits inside the model’s window?

And there’s more: vector databases have real problems nobody wants to talk about. False neighbors. Arbitrary chunking that splits a definition from its usage. Embeddings that age badly. Not to mention that when the result is wrong, you have absolutely no idea why.

The thesis I’ve been chewing on is simple: in most cases, a well-aimed grep plus a generous context window beats a full RAG stack. It’s cheaper, it’s easier to maintain, and when it breaks you can actually debug it. Let’s break this down.

What the Claude Code leak showed

Before we get into the theory, let me bring up something that happened a few days ago that backs this whole argument up. On March 31, 2026, Anthropic, by accident, published version 2.1.88 of the @anthropic-ai/claude-code package on npm with a nearly 60 MB source map attached, and roughly 512,000 lines of TypeScript from their internal tool leaked into the wild. I already wrote about the incident last week, with more detail on what showed up in the code.

The part that matters for this discussion is Claude Code’s memory system. Instead of dumping everything into a vector DB, the architecture has three layers. There’s a MEMORY.md that stays permanently loaded in context, but it doesn’t hold any actual data: it’s just an index of pointers, around 150 characters per line, kept under 200 lines and about 25 KB. The real facts live in “topic files” that get pulled on demand when the agent needs them. And the raw transcripts from previous sessions are never reloaded whole, only searched with grep, hunting for specific identifiers. No embedding. No Pinecone. Just write discipline (topic file first, index after) and lexical search. That’s it.

Claude Code’s main loop also has a tiered system for handling a context that’s filling up. As I detailed in the previous post, there are five different context compaction strategies, with names like microcompact (clears old tool results based on age), context collapse (summarizes long stretches of conversation), and autocompact (which fires when the context gets close to the limit). The CLAUDE.md file, which a lot of people thought was just a convention, is first-class in the architecture: the system re-reads it at every iteration of the query.

What this tells me: the best coding agent on the market right now, built by the company selling the most expensive model out there, does not use a vector DB. It uses files on disk, a markdown index, lexical search, and smart compaction strategies for when the context overflows. They could’ve slapped embeddings on top, they have the money to run whatever they wanted, and they chose not to. The reason, in my reading, is exactly what this post is arguing: to retrieve text from files you control, with generous context available, a vector DB is dead weight. Better to invest in compacting the window you already have than indexing everything into an external store.

There’s a curious security detail that came along with the leak: people noticed the compaction pipeline has a vulnerability they’re calling “context poisoning.” Content that looks like an instruction, coming from a file the model reads (say, a CLAUDE.md from a cloned repo), can end up being preserved by the compaction model as if it were “user feedback,” and the next model takes that as a real user instruction. It’s a new attack vector. But that’s a topic for another post.

The “Dream” system and memory consolidation

But what really caught my eye for the RAG debate, which I unpacked in detail last week, is the system called autoDream. It’s a forked subagent, with read-only bash access to the project, that runs in the background while you’re not using the tool. Its job is literally to dream: to consolidate memory. The name isn’t accidental, and the obvious analogy (which I couldn’t resist) is the human brain consolidating memory during sleep, turning short-term experience into something more stable.

For a dream to actually run, three gates have to open at once: 24 hours since the last dream, at least 5 sessions since the last dream, and a consolidation lock that prevents concurrent dreams. When it fires, it goes through four phases. Orient (does an ls on the memory directory, reads the index). Gather (looks for new signals in logs, stale memories, transcripts). Consolidate (writes or updates the topic files, converts relative dates into absolute ones, deletes facts that have been contradicted). And Prune, the final cleanup that keeps the index under 200 lines.

The decision to make autoDream a forked subagent is the detail that matters here. It does not run in the same loop as the main agent. Why? Because memory consolidation is a noisy process. The model has to re-read old transcripts, compare them against what’s in MEMORY.md, decide what stays and what goes, form hypotheses about things it saw in earlier sessions. If that ran in the main context, it would pollute the “train of thought” of the agent that’s trying to help you with your current task. By forking, you keep the two separate. The main agent stays focused on what you asked for, and autoDream does the housekeeping in parallel, with no write permission on the project.

And the way it figures out what needs to be consolidated is plain old lexical search. The transcripts live as JSONL files on disk, and autoDream uses grep to look for new signals. Just grep, on text logs. Stop and think about that for a second. The memory consolidation of the most advanced agent in the world, built by one of the richest AI companies out there, is a forked subagent running grep on text logs. If a vector DB were the right answer for this kind of problem, Anthropic would’ve put a vector DB in there. They didn’t.

And there’s a detail that, to me, is the buried gold of the entire leak, and it fits this argument like a glove. In autoDream, memory is treated as a hint. The system assumes that what’s stored may be stale, wrong, contradicted by something that happened later, and the model has to verify before it trusts it. The vector DB pitch is the opposite of that: index everything, search by similarity, return the top-k, trust the result. Claude Code went the conservative route. Index little, search by word, return a hint, and stay skeptical until you’ve laid eyes on the actual fact.

The whole strategy works in two layers. Inside a single session: generous context plus grep plus smart compaction (microcompact, context collapse, autocompact). Between sessions: a subagent that consolidates memory asynchronously, using grep on the transcripts and treating the result as a tip, not as truth. Embeddings and vector DBs don’t show up in either layer. The deliberate choice was a smart reader chewing on raw text, not a dumb reader being spoonfed the top-k of an embedding.

The practical lesson for our debate is simple. The most advanced agents on the market are heading toward generous context, lexical search, and smart compaction, not toward classic RAG pipelines. If Anthropic, with all the infrastructure and talent they’ve got, picked this path for Claude Code, those of us building internal applications on a fraction of that budget should at least think about going the same way.

Where the story started turning

When the ceiling was 32k of context, retrieval was the bottleneck of the entire problem. You had to pre-filter aggressively, because anything that made it into the window was sacred space. A vector DB was the only halfway-decent way to do that semantic pre-filtering. The logic was: “the reader (LLM) is expensive and dumb, so the retriever has to be smart and selective.”

Today the equation has flipped. The reader is now the smartest one at the table, and the window grew big enough to hold an entire document. So the retriever can (and maybe should) go back to being dumb. The dumber, the better. You want high recall and low precision, and you let the model do the fine work. Grep does exactly that. So does BM25. And ripgrep flies through millions of lines without breaking a sweat.

And this isn’t just my hunch. The BEIR benchmarks have shown for a while now that BM25 matches or beats a lot of dense retrievers when the domain drifts away from where the embeddings were trained. Anthropic itself published a post on Contextual Retrieval that basically says the same thing: a lexical signal plus an LLM’s judgment beats pure embeddings on most knowledge tasks. And take a look at Claude Code, the tool I’ve been using every day for 500 hours: it navigates the repo with Glob and Grep. No vector DB, no embedding, no LangChain. It works ridiculously well.

The real problems with vector databases nobody advertises

The vector DB marketing sells the dream of perfect semantic search. Reality is messier.

False neighbors come first. Cosine similarity rewards topical similarity, not relevance. You ask “how do we handle authentication errors” and the DB returns every chunk that mentions authentication. The chunk that actually answers the question may be in tenth place, or may not have been retrieved at all because the doc author wrote “login” instead of “auth.”

Chunking is the second one, and it’s a disguised disaster. A 512-token window with a 64-token overlap sounds reasonable, until you realize your important table got cut in half, the function definition ended up separated from its usage, and the piece of documentation with the exact command got orphaned without the context of its section. The chunk boundary tends to land exactly where the answer was living.

When it fails, it fails without leaving a trace. When BM25 misses, you know why: the word isn’t there. When a vector DB returns garbage, you get a plausible-looking wrong chunk, with no diagnostic signal at all. Good luck debugging that in production at two in the morning.

The index gets stale. Every document update calls for re-embedding. If you have 10,000 docs and 200 of them change per day, that turns into a batch process, monitoring, a queue, retries, embedding API costs, and an unavoidable inconsistency window between what’s on disk and what’s in the index. Grep has none of that. File changed? The next query already sees it.

And there’s the operating cost nobody adds up. Pinecone charges per vector. Weaviate wants a cluster to maintain. pgvector saves you a new server but you still own a schema, an index, and a re-embedding pipeline. Each of those things wants engineer time, monitoring, tests, deploys. All of that to do a search that rg would often crack in 200ms.

Comparing the complexity

Look at the diagram:

On one side, eight steps, four or five services, an external index that needs to be maintained and kept up to date. On the other, four steps, zero new infrastructure. This isn’t a caricature: it is literally what you have to set up for each case.

The honest question: does the left column pay off? In 2023, yes, because the right column didn’t exist (no LLM had a 200k window). In 2026, in most cases, it doesn’t.

Pros and cons of each side

Classic RAG (vector DB)

For:

• Works for huge document bases, on the order of hundreds of GB, where even rg won’t cut it without prior indexing
• Handles heavy paraphrase and cross-lingual queries (“how do I cancel” vs. “subscription termination process”) where the user’s vocabulary doesn’t match the document’s
• Works for non-textual modalities (image, audio) where grep has nothing to look at
• Saves input tokens if you’re tight on budget or absolute latency

Against:

• Complex stack: embedding, vector DB, chunking, reranker, re-indexing pipeline
• Opaque failures, hard to debug
• Chunking destroys the context of tables, code, long definitions
• Operational overhead (index, queue, monitoring, re-embedding cost)
• The semantic search the marketing is selling rarely works the way the marketing promises

Grep + long context

For:

• Practically zero new infrastructure: ripgrep, sqlite, or a plain LIKE in Postgres
• Always fresh: file changes, the next query sees them
• Transparent failures: the word is either there or it isn’t
• Loads the document in generous chunks, the model does the fine filtering with actual semantics
• Cheaper in dev and ops, cheaper to pivot domains

Against:

• Doesn’t scale to terabytes of raw text without some kind of indexing
• Suffers when the user’s vocabulary is very different from the document’s
• Doesn’t work for non-textual modalities
• Per-query latency is higher in absolute terms (loading 100k tokens always costs more than loading 5k)
• Per-query input cost is higher if you don’t have prompt caching

But what about cost?

This is the argument I get hit with the most when I defend the “load everything into context” thesis. “It’ll get crazy expensive, 200k tokens of input per query is absurd.” Let’s actually run the numbers.

In yesterday’s LLM benchmark post I mapped out the per-token price of every model. Take Claude Sonnet 4.6: $3 per million input tokens, $15 per million output. Take GLM 5 (which I proved actually works): $0.60 input, $2.20 output. Take GPT 5.4 Pro at the top of the heap: $15 input, $180 output (yeah, that one stings, I know).

Before we turn “200k tokens” into dollars, let’s land that number on something tangible, because “100k tokens” doesn’t mean anything to anyone. A token, on average, is roughly 0.75 of a word in English (Portuguese is similar, maybe a touch heavier because of longer words). So, translating:

• 100k tokens ≈ 75,000 words ≈ a whole short novel like Hemingway’s The Old Man and the Sea with room to spare, or about three long Wikipedia articles glued together.
• 200k tokens ≈ 150,000 words ≈ a big novel, like Crime and Punishment in full, or half of the first Game of Thrones book (which clocks in around 298k words, so roughly 400k tokens).
• 400k tokens ≈ 300,000 words ≈ A Game of Thrones in full, the entire first book of the series in your window.
• 1M tokens ≈ 750,000 words ≈ the entire Lord of the Rings trilogy plus The Hobbit, or the whole Bible (King James is around 783k words, roughly 1M tokens), or about two and a half Game of Thrones books stacked on top of each other.

So when I say “throw 200k tokens of input at the model,” what that actually means in the real world is “throw the entire Crime and Punishment in as the context for your question.” That’s a lot. And that’s exactly what makes the argument of this post viable: today’s models can read an entire novel in one go and still answer a specific question about it. In 2023, this was science fiction. In 2026, it’s the base case.

So picture a query that throws 200k tokens of input at the model (there goes Crime and Punishment again) and produces 2k tokens of output (about three pages of response):

Now throw prompt caching into the mix. Claude has a cache that drops cached input to a fraction of the full price (in the ballpark of 10%, depending on the model). Gemini has a similar mechanism. When you fire a sequence of queries against the same 200k-token dump, the cost of subsequent queries plummets to pennies. With Sonnet cached, you can fairly call it about $0.10 per follow-up query without making things up.

Now compare that to the cost of running a Pinecone, or a Weaviate, or a pgvector. Setting aside the price of the subscription itself (which varies a lot), you need an engineer to wire up the pipeline, maintain it, monitor it, deal with embedding failures, redo the chunking when the domain shifts. Conservatively, you’re looking at somewhere between 40 and 80 hours of engineering to make the thing stable. At R$ 200/hour, that’s between R$ 8,000 and R$ 16,000. In USD, somewhere between $1,600 and $3,200 just to stand it up.

With $3,200, on Sonnet 4.6 with prompt caching, you can run something on the order of 30,000 queries of 200k tokens each. Thirty thousand queries, depending on the scale of the project, gives you several months or even an entire year of an average internal tool. And you didn’t pay an engineer to wire up a pipeline. There’s no vector DB server to maintain. And if the document changes, the system already sees it on the next query.

The “RAG is cheaper in tokens” argument ignores that tokens are the cheapest thing in the entire equation. Engineers cost a lot, servers cost a lot, bugs in production cost a whole lot more. Tokens have become a commodity, and they’re getting cheaper with every new model release.

The classic RAG argument was “the model is expensive, retrieval is cheap.” Today it’s the opposite: the model is the cheap part of the stack, smart retrieval is what costs a fortune to build and maintain.

Where the thesis doesn’t hold

I don’t want to come off as a fanboy. There are cases where classic RAG still wins:

1. Massive corpora. If you have 500 GB of raw text, even rg won’t solve it in acceptable time. You need some kind of indexing. It can be indexed BM25 (Tantivy, Elasticsearch), it can be a vector DB. But notice: the first option is still lexical, not vector.
2. Wildly scattered vocabulary. Customer support, where the user types “my wifi’s down” and the documentation says “loss of connectivity at the physical layer.” BM25 won’t catch that. Embedding will. Vector DB scores a point here.
3. Non-textual modalities. Image-by-image search, audio-by-audio. Embedding is mandatory.
4. Critical absolute latency. If you have to answer in 100ms with a 5k input budget, a generous dump won’t fit. Pre-filtering is necessary.
5. Compliance and audit. If you have to prove that a specific document was consulted to answer a specific query, having indexed and trackable chunks helps. A 200k-token context dump is more opaque from an audit standpoint.

For those cases, classic RAG still makes sense. But notice the size of the list. These are specific cases. The general case, things like “chat with our internal docs” or “ask the product manual,” almost all of it falls into the “grep + long context handles it better” bucket.

Lazy retrieval: the recipe I’d defend

If I were building a “chat with docs” tool today, from scratch, it would look more or less like this:

1. Keep the documents raw. Markdown, converted PDF, code, whatever. On disk, organized in folders that make sense for the domain.
2. Fast lexical filter. ripgrep with regex, or BM25 with Tantivy/SQLite FTS5, or a LIKE in Postgres if you already have one. Returns 100-300 hits.
3. Load generously. Grab not just the matching snippet, but the entire file, or a wide window around it. Throw all of it into the context.
4. Let the LLM do the fine work. Pass the original question, tell the model to find what matters, drop the rest, and answer with citations.
5. (Optional) Add embeddings only for the query classes where lexical fails, after you have real data showing that it fails.

This is the opposite of the old advice (“start with vectors, fall back to keyword”). It’s: start with keyword, and add vector only if you feel the gap. In most projects, you never will.

A toy implementation in Ruby

To make it concrete. Here’s a Ruby script using the ruby_llm gem (the same one from yesterday’s benchmark) that does exactly this flow: grep through the files, load the snippets with context, send to Claude, get the answer back. No vector DB, no chunking, no embedding, no LangChain.

#!/usr/bin/env ruby
require "ruby_llm"
require "open3"

DOCS_DIR = ARGV[0] || "./docs"
QUERY    = ARGV[1] or abort "uso: ./ask.rb <pasta> <pergunta>"

# 1. Fast lexical filter with ripgrep.
#    -i case insensitive, -l file names only, --type-add covers md/txt/extracted-pdf.
def lexical_search(dir, query)
  terms = query.downcase.scan(/\w{4,}/).uniq.first(8)  # words with 4+ letters
  pattern = terms.join("|")
  cmd = ["rg", "-l", "-i", "-e", pattern, dir]
  files, _ = Open3.capture2(*cmd)
  files.split("\n").reject(&:empty?)
end

# 2. Load entire files (up to a reasonable cap).
def load_context(files, max_chars: 600_000)
  total = 0
  files.map do |path|
    body = File.read(path)
    next if total + body.size > max_chars
    total += body.size
    "## #{path}\n\n#{body}\n"
  end.compact.join("\n---\n")
end

# 3. Send to Claude with the question and the documents.
def ask(query, context)
  chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4-6")
  prompt = <<~PROMPT
    Você tem acesso aos documentos abaixo. Responda a pergunta do usuário
    usando apenas o que está nos documentos. Cite o nome do arquivo nas
    referências. Se a resposta não estiver nos documentos, diga isso.

    --- DOCUMENTOS ---
    #{context}
    --- FIM DOS DOCUMENTOS ---

    Pergunta: #{query}
  PROMPT
  chat.ask(prompt).content
end

files = lexical_search(DOCS_DIR, QUERY)
abort "nenhum arquivo bateu" if files.empty?
puts "Encontrei #{files.size} arquivos. Carregando contexto..."
context = load_context(files)
puts ask(QUERY, context)

About 40 lines. No Pinecone dependency, no vector schema, no re-indexing pipeline. You run it as ./ask.rb ./docs "how do I configure the payment webhook" and that’s it.

That example is one-shot. You run it, it answers, done. For a real chat, with multiple questions in a row over the same documents, the design changes. Instead of running lexical_search upfront and shoving everything into the context at once, you expose the search as a tool to the model. Then it’s the agent that decides when it needs to pull more docs, what term to look for, which file is worth opening in full. That’s how Claude Code actually works: Glob, Grep and Read are tools, and the model picks the sequence. ruby_llm supports tool calling, so you can do the same thing in Ruby. It looks something like this:

require "ruby_llm"
require "open3"

DOCS_DIR = "./docs"

class SearchFiles < RubyLLM::Tool
  description "Procura arquivos cujo conteúdo casa com o padrão dado (regex). Retorna lista de paths."
  param :pattern, desc: "Padrão regex pra busca lexical (case-insensitive)"

  def execute(pattern:)
    out, _ = Open3.capture2("rg", "-l", "-i", "-e", pattern, DOCS_DIR)
    out.split("\n").reject(&:empty?)
  end
end

class ReadFile < RubyLLM::Tool
  description "Lê o conteúdo completo de um arquivo do projeto."
  param :path, desc: "Caminho relativo do arquivo"

  def execute(path:)
    File.read(path)
  rescue => e
    "erro: #{e.message}"
  end
end

chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4-6")
            .with_tools(SearchFiles, ReadFile)
            .with_instructions(<<~SYS)
              Você responde perguntas sobre os documentos em #{DOCS_DIR}.
              Use search_files pra encontrar arquivos relevantes e read_file
              pra ler o conteúdo. Sempre cite o arquivo na resposta.
            SYS

loop do
  print "> "
  msg = gets&.chomp
  break if msg.nil? || msg.empty?
  puts chat.ask(msg).content
end

The model gets the question, decides whether it needs to search, calls search_files, sees what came back, decides whether it needs to open any file, calls read_file, and only then answers. On the next question it already has the previous context in the session and can ask for more if it needs to. The context only receives what the model asked for, not the whole grep dump from the earlier example.

The same idea works for databases: swap rg for a SQL query with LIKE or tsvector (Postgres full-text), load the relevant rows, throw them in the context. If you have 10k records in an internal database, this handles it. If you have 10 million, you start needing smarter pagination or a more serious pre-filtering layer. But the mental model is the same: dumb filter + smart reader.

The point that matters

The most interesting thing in all of this isn’t even the Pinecone savings. It’s that the nature of the bottleneck has changed. In 2023, the bottleneck was retrieval: the reader was small, slow, expensive, and you needed a clever retriever to fill the window with the bare minimum. In 2026, the bottleneck is reasoning over messy context: the reader is big, relatively fast, and cheap. So it makes more sense to have a dumb retriever with high recall and let the model do the heavy lifting.

Anyone still designing systems with the 2023 mindset is paying a premium to solve a problem whose shape has changed. RAG didn’t die, the “R” got dumber and cheaper, and that’s an upgrade. The vector DB vendors aren’t going to tell you this, but it’s the path the more experienced folks have been quietly walking.

The next wave of LLM applications, in my bet, is going to be dominated by the people who got this inversion. Smaller stacks, simpler infrastructure, generous context, and a whole lot less LangChain.

What the recent literature says

Before I close out, I went and checked what the research crowd published on this. Blog hot takes age in three months in this field, so it’s better to look at the papers.

Retrieval Augmented Generation or Long-Context LLMs?, out of Google DeepMind, published at EMNLP 2024, is probably the most cited piece in the debate. Their conclusion: when the model has enough resources, long context beats RAG on average quality, but RAG is still much cheaper in tokens. They propose Self-Route, an approach where the model itself decides whether it needs retrieval or whether it can just go straight through context. The token savings are big and the quality loss is small.

Then LaRA, presented at ICML 2025, is more measured. The authors built 2326 test cases across four QA task types and three long-context types, ran them across 11 different LLMs, and the conclusion was: there is no silver bullet. The choice between RAG and long context depends on the model, the context size, the task type, and the retrieval characteristics. RAG wins on dialogue and generic queries, long context wins on Wikipedia-style QA.

Long Context vs. RAG for LLMs: An Evaluation and Revisits, from January 2025, is the one that most reinforces this post’s thesis. Long context tends to beat RAG on QA benchmarks, especially when the base document is stable. Summarization-based retrieval comes close, and chunk-based retrieval lags behind. In other words: the old way, chunk plus embed plus top-k, is the one that comes out worst.

Worth keeping on the radar too is the original Lost in the Middle (Liu et al., 2023, published in TACL in 2024). That’s the paper that showed even models with big windows have performance that depends on the position of the relevant information. Stuff at the beginning or end of the context is found easily; stuff in the middle degrades. For a long time this got used as the argument against long context, but the paper is from 2023, with 2023 models. Today’s models, the Claude 4.x and Gemini 3.x line, handle the middle a lot better. It’s not a solved problem, but it’s much smaller than it was.

On the lexical retrieval side, BEIR is still the canonical reference. The classic result is that BM25, all the way from the 90s, is still ridiculously competitive in out-of-domain scenarios. Dense models only win consistently when you have in-domain data to fine-tune the embeddings. In zero-shot scenarios, which is where most projects live, BM25 is hard to beat without serious work.

To wrap up, the Anthropic post on Contextual Retrieval, from September 2024, is the most practical piece on the list. They show that combining contextual embedding with contextual BM25 drops the top-20 failure rate from 5.7% to 2.9%. Add a reranker and it drops to 1.9%. Important detail: BM25 is the centerpiece of their result, not a sidekick. The right reading is “lexical plus vector plus reranker is the combination that works.” Anyone who can only pick one picks BM25 and still gets pretty far.

To sum up what we can actually nail down: the literature isn’t claiming “RAG is dead.” It’s saying that long context, when you can use it, tends to win on quality. It’s saying RAG’s cost is still its main argument. It’s saying lexical BM25 is much stronger than the vector DB marketing makes it sound. And it’s saying that when you really do need heavy retrieval, the robust combination is hybrid (lexical plus vector plus reranker), not pure vector. All of that lines up with what I’ve been defending in practice.

Sources

• Li, Z. et al. (2024). Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. EMNLP 2024 Industry Track.
• Yuan, K. et al. (2025). LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing. ICML 2025.
• Yu, T. et al. (2025). Long Context vs. RAG for LLMs: An Evaluation and Revisits. arXiv:2501.01880.
• Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. TACL 2024.
• Thakur, N. et al. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks 2021.
• Anthropic (2024). Introducing Contextual Retrieval. Blog post.
• Akita, F. (2026). Claude Code’s Source Code Leaked. Here’s What We Found Inside. — my coverage of the leak, with more detail on the memory architecture, KAIROS and autoDream.

Now, you’re a small tech startup or even a small team in a big corporation. How to even start?

And the recommendation is: do the smallest thing that works first. Always. Of course, avoid “quick and dirty” as much as you can. But don’t get paranoid and over-engineer too much.

Over-engineering is as expensive as doing things “dirty”.

Add “Time” to your equation

What most people do very wrong is disregarding the variable of “Time”.

Everybody goes to tech conferences or read fancy blog posts, or watch flashy screencasts. Their wrong conclusion is: “Netflix uses it, therefore it should be good for me, because I want to become Netflix”.

The wrong assumption is that Netflix, or Google, or Facebook, or Spotify, is a static system that always functioned the way they advertise it today.

People forget that every unicorn company had a day-1. And on day-1 they did not use microservices. They did not have React.js. They did not have Go or Elixir. Some of them didn’t even have good programmers to begin with. Netflix started as a VHS renting service, remember that? Google started off in a dorm-room with off-the-shelf hardware components assembled in a Lego-based rack.

You are a fan of some celebrity. That celebrity has a Lamborghini car. So you want to be like that celebrity one day. What do you do? Do you buy the same Lamborghini he has?

If you are that stupid, the only thing you will end up with is a HUGE debt to pay.

Stop envying the rich guy’s Lamborghini.

(Technical) Debt

This is neither a new term nor a new theme. Every decision you make in programming is a compromise between what the “future-ideal” should be and what you can actually do right now. Loan to buy the equivalent Lamborghini too soon, and you will have to deal with an unpayable (Technical) Debt. You will stall, you will stop, and you will do nothing but pay that debt from now on.

You want to “code faster” therefore you skip writing automated tests, because you assume it will slow you down. And you’re right. Test-driven development doesn’t make you fast at first, and it isn’t supposed to. It protects your future self from your present self, and keeps you from piling up Technical Debt.

Again, you forget the “Time” variable.

You write and deliver fast for the 1st month. After the first version is deployed, now you will be obliged to start paying the debt. Your productivity will slow down. Regression bugs will show up. Every new feature you try to add breaks something in unexpected ways. Because you don’t have automated tests, you will keep fixing the same things many times over. Debt will catch up, and you will pay. One way or the other.

Minimal test suites are like Insurance. You don’t need it right now, on day-1. But on day-100 you will be so glad you have it.

Doing too much microservices on day-1 is debt. It feels great that you’re doing that fancy thing you read in a blog post. On day-100, with your 3 people team and a dozen microservices, from now on you will do nothing but pay the accumulated debt, with Interest. Every new microservices deployment breaks your system. And of course you didn’t add monitoring, you didn’t add integrated tests. So every time you code something, something else unexpectedly breaks.

Enjoying that early Lamborghini now?

The Mythical Man-Month

Philip Calçado writes very good posts and presentations about microservices in the right way. If you’re serious about that subject I strongly recommend that you read his posts, such as his “Prerequisites” post or his “Economics” presentation.

He correctly remembers Fred Brooks’ The Mythical Man-Month. I urge developers to read this small book. It’s uncanny how the entire industry is still repeating the very same mistakes Brooks reports in his book from projects in the 60’s!

To me, microservices is a by-product of tech companies with too many developers. Once you break through the 5-developers team a “monolithic” system with poor organization and few automated tests can become cumbersome to deal with. So the consequence is the desire to break it down. You make 2 teams and 2 microservices and coordinate. And the teams start to isolate, and play the finger-pointing game when new bugs are reported (“it’s the other team’s microservice’s fault”).

That’s Conway’s Law if you do it wrong:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Also in play is another chapter in Brooks’ book: The Second System Effect. Every developer, tech startup, has to assume that the first version of anything is not the best design. And the second system (the first big rewrite) is the most dangerous system you will write. As Brooks states, it goes like this:

An architect’s first work is apt to be spare and clean. He knows he doesn’t know what he’s doing, so he does it carefully and with great restraint.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used “next time.” Sooner or later the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one. The result, as Ovid says, is a “big pile.”

Then, what happens? The CEO, the board, the investors, or whatever, start doing the wrong thing: hiring more and more people. Loan more and more. Instead of paying the debts they end up making more debt. They fell for what Brooks reported in the first chapter and what gives the name to the book: The Mythical Man-Month.

Adding manpower to a late software project makes it later.

Complex programming projects cannot be perfectly partitioned into discrete tasks that can be worked on without communication between the workers and without establishing a set of complex interrelationships between tasks and the workers performing them.

Therefore, assigning more programmers to a project running behind schedule will make it even later. This is because the time required for the new programmers to learn about the project and the increased communication overhead will consume an ever increasing quantity of the calendar time available.

Again, it’s foregoing the sense of the “Time” variable. And confusing “debt” for “investment”.

First of all, do yourself a favor and read the goddamn book, twice.

Do Agile right! And no, it’s not Kanban!

If there is one good thing the whole Lean noise generated is the notion of “Minimum Viable Product” or MVP. People call it “prototype”. Some call it “launching Beta” or simply “version 1.0”. Doesn’t matter. It’s the realization that you don’t know much at the beginning so overengineering on the first version is a waste of time. Lean is about controlling “waste”, so we do the minimal that works, gauge results and work from there.

The cynical will say that any prototype that reaches production will never die. And they’re not wrong.

The balance is to not do the “quick and dirty” version. That’s what we have Agile techniques for. Do the minimal, organize the minimal. That’s why we have object-oriented design patterns, from GoF to DDD. You don’t need to do EVERY pattern, and that’s exactly what “overengineering” means. But you can do the minimal monolith that will allow you to evolve later.

People criticize Ruby on Rails for not being organized “enough”. Nick Sutterer, the creator of Trailblazer, has a good point.

Rails done wrong is bad. The conclusion should NOT be: “let’s do microservices” though. That jump in faith is idiotic and makes no sense for all the reasons I mentioned above.

“Let’s do proper Agile stuff and proper object-oriented stuff” should be the initial answer.

And by “Agile” forget about idiotic post-its, idiotic numerology-based estimation equations (this is a whole post in itself, because story points and velocity are useful, but adding Montecarlo and other Gaussian-based stuff are not). The only “Agile” things you should be concerned about are the Extreme Programming (XP) techniques, including Iteration-based timeboxes.

You MUST do timeboxes. Stop, re-assess, change directions, and then keep going. The model of “Pull” is only reasonable when your direction is very, very clear, written in stone and unchangeable, like in a factory production line! (Which is exactly where the concept of Pull, and Lean, was actually born!)

Iterations, like automated tests, are like Insurance. You can never avoid all waste, but you can minimize it. You can afford to throw away an Iteration-worth of work. After the iteration you measure the results, and throw it away if necessary, changing direction in the process. Throwing useless stuff away is as important as adding new stuff. If you just add, you have a hoarding disorder!

If in doubt, do XP. Yes, it feels more “difficult”, and Kanban is “simpler” to explain. Now, is that a good reason?

Forget about Raw Performance when you DON’T need it!

Another problem is choosing languages or framework because of performance.

If you’re in the web development business, this is a huge WASTE.

Understand this truth: your servers will IDLE most of the time. And if you feel like using your web app is slow, the slowness comes from the POOR programming you did, not from the language. And no good language will rescue a bad programmer. I always say that if performance was that important, we should all be doing C.

Most of what you serve in HTTP-based apps, be it user-readable content, be it API GET results, can be CACHED! If you’re not using a CDN, you’re doing it wrong.

Yeah, yeah, yeah, you think you’re building the next Spotify. You’re not, at least not 99% of you. And the 1% doing custom engineering, with custom techniques and custom stack, successfully: you’re the 1%. Actually I’d say you’re a fraction of the 1%. Do not assume that what you do is good for the rest of the population.

90% of what most small companies and solo web developers need is just a Shopify account and a vanilla Wordpress installation. And that’s it.

Are you doing infrastructure development like building custom components to Docker, Kubernetes, Terraform? Command-line tools? Daemons? Then choose GO.

Are you doing embedded libraries? Maybe the next generation OpenSSL? Drivers? Then choose RUST.

Are you doing mobile development? Then don’t have a lot of choice: do Swift for iOS, do Kotlin/Java for Android. Or do React Native for simpler apps.

Are you really doing the next Whatsapp? The next Waze? The next Snap? You have hundreds or thousands of users in need of long-lived connections over unreliable network doing broadcasts of messages? Or you’re building the next evolution of distributed NoSQL databases? Or anything that actually has the proper meaning of “Distributed” in its definition? Then choose ELIXIR.

Are you doing a CRUD based web application? Go RAILS and never look back.

Are you doing all of the above? Use all the alternatives then. And I hope you have the budget, because you need a big team.

Conclusion

The Economics only makes sense when you consider the day-1. Do not forget the “Time” variable. It makes all the difference.

You will always get Debt to pay. So be smart in which kind of debt you choose. Because you will have to pay it back eventually.

Again, read Brooks. He said it decades ago: THERE IS NO SILVER BULLET. Don’t bullshit yourself. A new language, a new framework, a new architecture. None of those will save you.

In any tech endeavor, code is not the only thing to worry about. I would go as far as to say that in a tech startup, code is only 20% of the problem. A tech startup is just a company, like any other. If you’re the founder or CEO, you have to deal with all the remaining 80%: marketing, accounting, human resources, legal, etc. It’s already difficult enough without your tech team getting you unwanted Technical Debt, the unneeded Lamborghini, that you won’t be able to pay.

You can start humble on day-1. Keep evolving, continuously. This is the core of any Agile or Lean process: KAIZEN. Choose smart debts, pay them a small bit at a time, continuously. It’s the same reasoning as getting a loan in the bank.

Netflix day-7,200 is NOT your day-1.

Facebook day-4,800 is NOT your day-1.

Google day-6,809 is NOT your day-1.

Be humble. Deliver fast. Pay your Debts. Keep evolving continuously. Stop believing in fairy tales and silver bullets.

If I were building a secondary Rails app connecting directly to the same database as the first, I could just add the dependency to the extracted gem:

gem 'central-support', github: 'Codeminer42/cm42-central-support', branch: 'master', require: 'central/support'

Then recreate the models including the same Concerns, and make sure I remove the ability to bin/rails db:migrate from the secondary app (by creating empty db tasks with the same name, for example).

By the way, this is one big caveat that I didn’t address in Part 1: up to this point, the schema was frozen in the central-support gem.

From now on, you must control the evolution of the tables mapped in the gem from within the gem. The best approach is to use the spec/support/rails_app and normally create new migrations with bin/rails g migration from there. Then you must move the migration to the lib/generators/central/templates/migrations folder.

The lib/generators/central/install_generator.rb will take care of making a central:install task available that will dutifully put new migrations into your application’s db/migrate folder as usual. You just have to bundle update central-support to get the newest changes, run bin/rails g central:install to create the new migrations (it will automatically skip existing ones) and run the normal bin/rails db:migrate. A migration generator code is very simple, you can do it like this:

require 'securerandom'
require 'rails/generators'
require 'rails/generators/base'
require 'rails/generators/active_record'

module Central
  module Generators
    class InstallGenerator < Rails::Generators::Base
      include ActiveRecord::Generators::Migration

      source_root File.expand_path("../templates", __FILE__)

      def create_migrations
        Dir["#{self.class.source_root}/migrations/*.rb"].sort.each do |filepath|
          name = File.basename(filepath)
          migration_template "migrations/#{name}", "db/migrate/#{name}", skip: true
          sleep 1
        end
      end
    end
  end
end

The migration_template will take care of adding the proper timestamp to the migration file, so you don’t have to add it manually and the template file name can be something plain such as migrations/add_role_field_to_user.rb.

All that having been said, there is a second challenge I added to my internal secondary app: I wanted it to have its own main database and use Central’s database as a secondary read-only source.

So migrations in the secondary app (let’s just call it Central-2) will run against its own main database, not against the main Central’s database. This adds the following problem: the test suite must be able to create and migrate both test databases, in isolation from Central. Only in production should Central-2 connect to Central’s database.

Every Rails application has a config/database.yml.sample and a db/schema.rb, so I started by creating a config/database_central.yml.sample and a db_central/schema.rb.

The config/database_central.yml.sample is already interesting:

development:
  adapter: postgresql
  encoding: unicode
  timeout: 5000
  database: fulcrum_development
  pool: 5

# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
  adapter: postgresql
  encoding: unicode
  timeout: 5000
  <% if ENV['DATABASE_CENTRAL_URL'] %>
  url: <%= ENV['DATABASE_CENTRAL_URL'] %>
  <% else %>
  database: central_test
  <% end %>
  pool: 5

production:
  adapter: postgresql
  encoding: unicode
  timeout: 5000
  url: <%= ENV['DATABASE_CENTRAL_URL'] %>
  pool: <%= ENV['DB_POOL'] || 5 %>

In production, it will use the DATABASE_CENTRAL_URL environment variable to connect to Central’s main database.

When running tests locally, it will simply connect to a local database named central_test.

Now, while running tests at Gitlab-CI (or any other CI for that matter), I have to configure the DATABASE_CENTRAL_URL to point to a secondary Postgresql test database.

For Gitlab, this is how I configure the build script:

image: codeminer42/ci-ruby:2.3

services:
  - postgres:latest

cache:
  key: central-bot
  untracked: true
  paths:
    - .ci_cache/

variables:
  RAILS_ENV: test
  DATABASE_URL: postgresql://postgres:@postgres
  DATABASE_CENTRAL_URL: postgresql://postgres:@postgres/central_test

before_script:
  - bundle install --without development production -j $(nproc) --path .ci_cache
  - cp config/database.yml.sample config/database.yml
  - cp config/database_central.yml.sample config/database_central.yml
  - bin/rails --trace central:db:create central:db:schema:load
  - bin/rails --trace db:create db:schema:load

test:
  script:
    - bundle exec rspec

Notice how I copy the “.sample” config files to make sure they exist. And then how I run tasks you know such as db:create db:schema:load to create the normal test database, but also tasks you don’t know such as central:db:create central:db:schema:load.

I defined those tasks in lib/tasks/db_central.rake like this:

task spec: ["central:db:test:prepare"]

namespace :central do

  namespace :db do |ns|

    task "environment:set" do
      Rake::Task["db:environment:set"].invoke
    end

    task :drop do
      Rake::Task["db:drop"].invoke
    end

    task :create do
      Rake::Task["db:create"].invoke
    end

    task :setup do
      Rake::Task["db:setup"].invoke
    end

    task :reset do
      Rake::Task["db:reset"].invoke
    end

    task :migrate do
      Rake::Task["db:migrate"].invoke
    end

    task :rollback do
      Rake::Task["db:rollback"].invoke
    end

    task :seed do
      Rake::Task["db:seed"].invoke
    end

    task :version do
      Rake::Task["db:version"].invoke
    end

    namespace :schema do
      task :load do
        Rake::Task["db:schema:load"].invoke
      end

      task :dump do
        Rake::Task["db:schema:dump"].invoke
      end
    end

    namespace :test do
      task :prepare do
        Rake::Task["db:test:prepare"].invoke
      end
    end

    # append and prepend proper tasks to all the tasks defined here above
    ns.tasks.each do |task|
      task.enhance ["central:set_custom_config"] do
        Rake::Task["central:revert_to_original_config"].invoke
      end
    end
  end

  task :set_custom_config do
    # save current vars
    @original_config = {
      env_schema: ENV['SCHEMA'],
      config: Rails.application.config.dup
    }

    # set config variables for custom database
    ENV['SCHEMA'] = "db_central/schema.rb"
    Rails.application.config.paths['db'] = ["db_central"]
    Rails.application.config.paths['db/migrate'] = ["db_central/migrate"]
    Rails.application.config.paths['db/seeds'] = ["db_central/seeds.rb"]
    Rails.application.config.paths['config/database'] = ["config/database_central.yml"]

    ActiveRecord::Tasks::DatabaseTasks.database_configuration = CM(Rails.root)
      .join("config", "database_central.yml")
      .File.read
      .ERB.new
      .result
      .YAML.load
      .unwrap.freeze
  end

  task :revert_to_original_config do
    # reset config variables to original values
    ENV['SCHEMA'] = @original_config[:env_schema]
    Rails.application.config = @original_config[:config]
  end
end

This is how I define a namespace of Rake tasks that start with central:, and every one of those connects to the secondary database as described in database_central.yml. The weird syntax is from my chainable_methods, don’t mind it too much.

The db_central/schema.rb is basically a copy of the spec/support/rails_app/db/schema.rb from the central-support gem, with the same tables. The spec runner of both the gem and this secondary app will just load the schema into the test database.

Now that we have the basic underpinnings for specs in place, we can focus on how the application itself can consume those external models.

We start by adding an initializer like config/initializer/db_central.rb:

DB_CENTRAL = CM(Rails.root)
  .join("config", "database_central.yml.sample")
  .File.read
  .ERB.new
  .result
  .YAML.load
  .unwrap.freeze

In this case I am reading from the sample file because, unlike the CI build, when I deploy to Heroku I don’t have a script to copy the sample to the final yaml file. This will populate the constant DB_CENTRAL with the database URL stored in the DATABASE_CENTRAL_URL environment variable that I have to set.

Then I create a new file called app/models/remote_application_record.rb that looks like this:

class RemoteApplicationRecord < ApplicationRecord
  establish_connection DB_CENTRAL[Rails.env]
  self.abstract_class = true

  unless Rails.env.test?
    default_scope -> { readonly }
  end
end

This is how you create a new connection pool for the secondary database configuration. You must only have this establish_connection in one place and have the models inherit from here. The abstract_class = true will make ActiveRecord not try to load from a table of the same name as this class.

Then we have a default_scope locking down the model as readonly. We don’t want that in the test environment because I still want to have Factory Girl populate the test database with fake data for the specs. But it’s a good idea to have it in the production environment just to make sure.

Finally, I can create all the models I need, such as app/models/central/team.rb:

module Central
  class Team < RemoteApplicationRecord
    self.table_name = 'teams'

    include Central::Support::TeamConcern::Associations
    include Central::Support::TeamConcern::Validations
    include Central::Support::TeamConcern::Scopes
  end
end

From here I can just call normal Arel queries such as Central::Team.not_archived.limit(5).

Conclusions

If you didn’t already, refer to Part 1 for more details.

This is a simple recipe to share model logic between a main read-write Rails app and a secondary read-only Rails app. They share most (not all) of the same models, they share the same logic (through some of the Concerns), and they share the same database.

In this particular case, the recommended approach is to create a Follower database, which is how Heroku calls a secondary replicated database, and make my secondary application connect to that (since it only needs a read-only source).

For more complicated scenarios, you will need a more complicated solution such as an HTTP API layer to make sure only one App manages model migrations and such. But the Rubygem approach should be “good enough” for many cases.

If I really need to go that way, it won’t be too difficult to transform this small gem into a full-blown Rails API app. If you can’t even separate the logic as a Concern, you won’t be able to separate them as APIs either, so consider this a quick exercise, a first step towards creating an Anti Corruption Layer.

And as a bonus, consider contributing to the Central and central-support open source projects. I intend to build a competitive Pivotal Tracker/Trello alternative and we are getting there!

That would be the quick answer if you ask any developer how to share business logic between applications.

Although it makes sense in many situations, it’s not a good answer all the time.

My TL;DR for some situations is that you can organize your model logic as ActiveSupport::Concerns (or plain Ruby Modules if you will) and move them out to a Rubygem that your applications can consume.

Notice that I am only speaking of Models, not Controllers or Views. To share those you would need a full-blown Rails Engine instead. But in many cases I’ve seen, the goal was to just share the business logic between applications while having separate front-end logic.

A small example of this scenario is the open sourced project I’ve been working on in the last few weeks. Central, which is a Pivotal Tracker/Trello alternative - if you’re interested.

A few days ago I started a new project (for internal use only) that would query the same models as Central. I didn’t want to implement HTTP APIs at this point, and the new application would itself have models with relationships to the models in Central (while treating them as read-only).

After a few refactorings, most of Central’s models look like this one:

class Team < ActiveRecord::Base
  include Central::Support::TeamConcern::Associations
  include Central::Support::TeamConcern::Validations
  include Central::Support::TeamConcern::Scopes
  include Central::Support::TeamConcern::DomainValidator
  ...
end

And I have this dependency in the Gemfile:

gem 'central-support', github: 'Codeminer42/cm42-central-support', branch: 'master', require: 'central/support'

Whenever I change the concerns, I do a bundle update central-support in the projects (this is the one caveat to keep in mind to avoid dealing with outdated models).

This was possible because most of those models were mature and stable and I will not be changing them often. I don’t recommend exposing unstable dependencies (as gems or APIs, it doesn’t matter), because this is a recipe for huge headaches of cascading breaking changes due to outdated dependencies that are changing too often.

You should ONLY expose business logic that is reasonably stable (changes only every week or so).

The whole endeavor was to build a certain Rubygems structure, organize the original models into Concerns (which breaks no behavior), make sure specs are still passing, and then move the content (models and specs) over to the new Rubygems and make sure the specs pass there.

That’s how I built a secondary open source dependency for Central, called Central Support. Like many gems, its main file lib/central/support.rb is nothing but a bunch of ‘require’s to load all the dependencies.

So I methodically organized logic as concerns, such as lib/central/support/concerns/team_concern/association.rb, which is just the extraction of the Active Record associations from the ‘Team’ model.

Cut from Central, Paste into Support. When all relevant logic has been moved, I could move the entire Team model spec, mostly without any changes, and make it run. Every time I moved a piece, I bundle updated the gem and ran the main spec suite to make sure nothing broke.

And this is the difficult part: make a sandbox where those concerns could run and be tested.

To begin, I needed to build a minimal Rails app inside the spec folder, at spec/support/rails_app. And there I could put fake models that include the concerns I had just extracted from Central.

There is scarce documentation on how to do that, but I think you can just do rails new and start from there, or copy my rails_app folder for the bare minimum. My case is simpler because this gem is not meant to be general-purpose, so I don’t need to run it against different Rails versions, for example.

This internal test app must have a carefully crafted Gemfile:

...
gem 'central-support', path: File.expand_path("../../../..", __FILE__)

gem 'devise'
gem 'pg'
gem 'friendly_id'
gem 'foreigner'

group :test do
  gem 'test-unit'
  gem 'rspec-rails'
...

You don’t have to add the gems from the main gemspec. But you can remove the development dependencies that you would put in the gemspec and keep them in the test app Gemfile.

Now, from the main Gemfile you can do:

source 'https://rubygems.org'

eval_gemfile File.join(File.dirname(__FILE__), "spec/support/rails_app/Gemfile")

Most tutorials to build a Rubygem will add a line to load dependencies from the gemspec, but here we are replacing it with the test app’s Gemfile. This is the manifest that will be loaded when we run bundle exec rspec, for example.

Speaking of which, this is the spec/rails_helper.rb:

ENV['RAILS_ENV'] ||= 'test'

require 'rails/all'

require 'factory_girl'
require 'factory_girl_rails'
require 'rspec/rails'
require 'shoulda/matchers'

`cd spec/support/rails_app ; bin/rails db:drop db:create db:schema:load RAILS_ENV=test`

require 'support/rails_app/config/environment'

require 'support/database_cleaner'
require 'support/factory_girl'
require 'support/factories'
require 'spec_helper'

To wrap your head around it:

• bundle exec rspec will load the main Gemfile
• the main Gemfile will load from the internal test app’s Gemfile
• that internal test app’s Gemfile will require the gemspec from ../../../.. and the development and test groups of gems (including Rspec, Factory Girl, etc)
• the gemspec will require the runtime dependencies such as “activesupport”, “enumerize”, etc
• finally, the rails_helper.rb listed above will load.

There at line 11, the runner will execute a command to cd into the internal test app’s root folder and run the db:schema:load, therefore you need a db/schema.rb ready to load, as well as config/database.yml.

The spec/spec_helper.rb is more standard, with optional configurations for test coverage, etc.

The models inside the internal test app are the important parts, because they are the means to include the extracted concerns into a runnable format. The ‘spec/support/rails_app/app/models/team.rb’ is such an example:

class Team < ActiveRecord::Base
  include Central::Support::TeamConcern::Associations
  include Central::Support::TeamConcern::Validations
  include Central::Support::TeamConcern::Scopes
  include Central::Support::TeamConcern::DomainValidator
end

And with that, I could move the unmodified specs directly from the main project (Central), such as spec/central/support/team_spec.rb:

require 'rails_helper'

describe Team, type: :model do
  it { is_expected.to validate_presence_of :name }
  it { is_expected.to have_many :enrollments }
  it { is_expected.to have_many :users }
  it { is_expected.to have_many :ownerships }
  it { is_expected.to have_many :projects }
  ...
end

If you go back in the Central project, some commits back, you will find the very same file as spec/models/team_spec.rb. And the main advantage of this approach is exactly being able to move most of the code out of the main project, together with their specs, into a dependency gem, without having to “rewrite” anything.

If I had to rewrite all or a big chunk of the code, it would’ve been a more expensive choice and I would probably have deferred it to another time and focus on more valuable features first.

This approach is not perfect but it was super cheap. I could move all the relevant business logic out of the main project without having to rewrite anything but a few lines of wiring code. The new dependency gem received all the relevant bits and specs, and everything just runs.

So, if you have 2 or more Rails apps that could share the same models, this is how you can start it. Of course, there are always a lot of caveats to keep in mind.

In my case, the Central project is the one that can read-and-write to the database. My internal secondary app is just using the models as read-only. When 2 different apps write to the same database, you may have a number of conflicts to deal with.

This approach is useful if your secondary application is akin to an Administration dashboard, for example. You need to have some of the same associations, scopes, even validations for eventual editing, but it’s limited to a few, controlled users.

This is also useful if you’re doing data analysis, and again you can use the same associations, scopes, to build reports and dashboards. Essentially, if you need read-only access, this is a no-brainer.

In the next article I will explain how I wired a secondary application, using the central-support gem and dealing with 2 different databases at the same time.

First and foremost, I am a Web Applications Developer. I deal with Ruby on Rails applications and infrastructure. So, I am not a Desktop developer, a Mobile developer, Games developer or a Tools developer. This is very important to get out of the way right from the start.

And by the way, as I will focus my programming efforts more and more into Elixir, it does not mean I am “switching” from Ruby to Elixir. I don’t need to make exclusive choices, and in my mind, at least for a period of time, Rails joining forces with Phoenix will be a very difficult combo to beat for my web development strategies.

I know, Phoenix is built like Rails so why not just switch altogether: because most content-based sites don’t need the concurrency aspects of Phoenix, and I said “for a period of time” because Rails still has a humongous ecosystem with more mature gems that make development easier and faster. This can change in the future, but for now the combo makes sense as I can build a normal website in Rails as I would normally do (with Devise, ActiveAdmin, Spree or whatever) and add Phoenix for stuff like WebSockets (real-time notifications, real-time chatting, background jobs that can be run more efficiently than Sidekiq, etc).

This article will summarize my personal take on 2 fronts:

• Other Developer Roles than just the Web
• “Functional” Concepts that Really Matter
  • Immutability and Opaque Message Passing are VERY important
  • Coroutines and Schedulers
  • Static vs Dynamic Typing is still controversial
  • Fault Tolerance: Don’t fear your code
• Summary

As anything that I want to argue about, this is lengthy, but it wouldn’t be nice to just state something without further elaboration. In this quest for understanding, I may have confused a bit or two, so let me know in the comments section below if there are things to fix or to explain in more detail.

Let’s get started.

Other Developer Roles than just the Web

In a Desktop environment, you will definitely want to make a combination of Node-Webkit with native libraries. If you’re in specific corporate environments, you won’t have any choices other than plain WFC based .NET development or Java Swing. Your options have been set for quite some time, and even Visual Basic.NET still has its place. Specific toolchains will be dictated by Microsoft and Oracle/Java Community Process.

In Linux environments you will still use wrappers around GTK+, Qt or similar toolkits. There aren’t a lot of ways around this.

If you’re a Mobile-first developer, you do need to learn your way into Swift and Objective-C (to some extent) for iOS and the specific Java flavor for Android’s Dalvik/ART. But I will argue that you have a lot to gain in native development if you use RubyMotion. Or, you can simply keep following Facebook’s React Native endeavor. There’s a lot of fragmentation in this environment, you can do as little as possible with Web Mobile and Phonegap/Cordova, and build compelling apps with tools such as Ionic Framework. The only consensus is that if you really want to build the advanced next gen stuff, you want to dive deep into the native frameworks each platform has to offer.

If you’re a Game Developer you want to be as close to the metal as possible. It’s definitely feasible to write perfectly playable mobile games using any number of Javascript libraries together with HTML 5’s technologies such as the Canvas and WebGL. But for the next gen blockbuster you will either use mature engines such as Unity or Unreal or even build your own if you’re really invested in this field. This will require you to actually know your way into C/C++ programming. Really depends on how deep you want to go down the rabbit hole.

If you’re a Tools Developer you will have the benefits from both old and new generation of languages. If you’re closer to the Linux Kernel you will really need C/C++ in your baggage. But for the new generation of light containers (LXC), Docker, you can benefit from Go, an application development environment suited to make life easier than having to handle C/C++ idiosyncrasies. Rust is another great new choice to make it easier to - among other things - write memory-leak free small libraries and tools (I am mentioning that in particular because it’s important for languages like Ruby or Python to be able to add performance by binding to C-based native libraries and Rust makes this task easier).

Different languages have different teams and different long term goals, which is why it’s not apples and oranges to compare languages. Go, for instance, is heavier than Rust, but both are good for command-line tools, specialized daemons, and in the case of Go, networking-heavy and concurrency-heavy endeavors.

In my eyes, Go is a “better” Java or C++. And don’t interpret this wrong: Java is still a very fine language and platform. There is hardly anything that comes close to the maturity of the JVM and the extensive ecosystem behind it. I would not think for a second of trying to rewrite complex systems written in Java such as the Lucene library or Elasticsearch/SOLR solutions, for example.

But the power of Standard Java is difficult to unleash without some warming up for the HotSpot to pick up steam. It makes it not a great solution for command-line tools. But now you have a good middle ground with Go. You also don’t have a good time embedding Java into other platforms, and then you would need to go back to C, but now you have another good middle ground with Rust.

If you want to unleash different programming models, specially those more suited to concurrent abstractions such as the Hoare’s CSP like Actor model, you can try Scala with Akka (which is now the standard actor library) and Clojure’s Pulsar/Quasar. Akka and Quasar are the ones that come “close” (but can never match) Erlang’s built-in OTP platform.

For the Web at large, you can do just fine with the current (ever changing, unstable) Node.js ecosystem, Python (Django, Plone), Ruby (Ruby on Rails, Sinatra), PHP (Zend, Laravel), even Perl has its place. Combined with mature services in different platforms (Elasticsearch in Java, PostgreSQL in C, RabbitMQ in Erlang), any big and complex Web Application can be written with any number of best-of-breed tools and combinations that best suit your needs.

This is an unfair, short overview, of course. I didn’t cover every aspect of computer science or industry. There are several other active and useful languages such as Lua, Haskell, Fortran, Ada, Julia, R. The message being: you don’t need to choose a single language, it will really depend on what you’re going to deliver. And a true craftsman will master many tools to get the job done in the best way possible.

“Functional” Concepts that Really Matter

I wrote about my opinions on the current surge in the so called Functional style of programming. I recommend you read it before continuing.

There are some aspects that are adamant if you really want to go beyond the academic research and into the real world of productivity.

Immutability and Opaque Message Passing are VERY important

In order for computation to be fast, we are used to share data between routines. We move pointers around and change data in place.

It’s not particularly fast to make things immutable and not shared. The more you make data mutable and the more you share, the harder it is to make your code run concurrently.

It is an important trade-off: if you see mutable data and shared state, you’re optimizing for performance.

One can’t say that a language is essentially good or bad for having mutable or immutable data. But for what it’s worth, my personal opinion is that it’s harder to convince users to follow conventions like “share a little as possible, mutate as little as possible.” Most will not even know about it if it’s not built-in and enforced. I prefer having immutability enforced by default.

In Erlang, data is immutable. Similar to Java it passes values by reference in routine calls (it is not copying values between calls, as many misunderstand it).

And in the case of recursion it optimizes through Tail Call Optimization to make it faster. By the way, this is one optimization that the Java VM just can’t quite do yet. Clojure needs special ‘recur’ and ’trampoline’ calls, for example. Scala can rewrite the tail recursion to a loop in compile time, with the ‘@tailrec’ annotation. Erlang has its own traps as well, so not so black and white at the moment.

In Erlang, as I explained before, you run functions within completely isolated processes. If the function recurs or blocks, it stays isolated. Processes can only communicate by sending (immutable and opaque) messages to each other. Messages get queued in a “run queue” or “mailbox” and the function can choose to receive and respond to those messages. That’s it.

So, you can pass values by reference between routines, or you can share data in a third party process as the mediator of data. One such infrastructure built-in to Erlang is the ETS, the Erlang Term Storage, which is part of the so called OTP platform. Think of ETS as a very simple and very fast built-in key value storage like Memcached. You use it for the same use cases as a cache and it’s as simple as just doing this:

table = :ets.new(:my_fancy_cache, [:set, :protected])
:ets.insert(table, {"some_key", some_value})
:ets.lookup(table, "some_key")

Many might argue that Erlang’s rigid process isolation and communication strictly restricted to opaque message passing are overkill and that you can pass by using something akin to Clojure’s MVCC Software Transaction Memory, or STM. You do have STM in Erlang, with the other built-in OTP tool, built on top of the ETS, called Mnesia. It offers the equivalent of ACID database transactions in-memory. It’s not a new concept, but STM is not available as a language feature and it’s still uncertain if it really is a good choice to have it.

An inspired result, I believe, from Clojure’s choice of having transactional memory with history queue and snapshot isolation is shown in its crown jewel, Datomic. The idea is not revolutionary by any stretch of the imagination as you have many other prior art such as RethinkDB, CouchDB, and extensions for existing databases. Good for a service, still I don’t think it’s a good thing to share state, even if you have a transactor around that state. Erlang’s immutability with rigid process isolation still has no match.

Coroutines and Schedulers

You already know sub-routines, you do it all the time by partitioning large portions of code into smaller functions or methods that call each other. You may already know about a specialized kind of Coroutines in the form of Fibers (as first implemented in Windows circa 1997).

Fibers offer a way for your current function execution to “yield” back to its caller, preserving its current state, and then the caller can “resume” the suspended Fiber to continue its execution from where it last yielded. This allows for non-preempted, cooperative multitasking. We have Fibers in Python, Ruby, and other languages and it allows the creation of constructions like Generators. Even Javascript can have some form of Fibers if you add libraries like node-fibers:

var Fiber = require('fibers');

function sleep(ms) {
    var fiber = Fiber.current;
    setTimeout(function() {
        fiber.run();
    }, ms);
    Fiber.yield();
}

The call to ‘yield’ suspends the current execution until the function in the ‘setTimeout’ is called. Then it calls ‘run’, which resumes the previously yielded function. This is still “rudimentary” compared to coroutines: because the function itself has to yield control out to the reactor event loop, in the case of a Node.js application. If you don’t, you will block the event loop in a single threaded Node.js process, and therefore you block everything until the function finishes, defeating the whole purpose. And this is one of those “conventions” that “good” programmers should follow, but most will forget.

Fibers are useful to make it less ugly to program in a rudimentary Reactor environment, where you depend on callbacks calling callbacks and you end up with the anti-pattern of callback pyramid of doom. With Fibers you can program as you would in a synchronous imperative language transforming this ugly Javascript code:

function archiveOrders(date, cb) {
  db.connect(function(err, conn) {
    if (err) return cb(err);
    conn.query("selectom orders where date < ?",  
               [date], function(err, orders) {
      if (err) return cb(err);
      helper.each(orders, function(order, next) {
        conn.execute("insert into archivedOrders ...", 
                     [order.id, ...], function(err) {
          if (err) return cb(err);
          conn.execute("delete from orders where id=?", 
                       [order.id], function(err) {
            if (err) return cb(err);
            next();
          });
        });
      }, function() {
        console.log("orders been archived");
        cb();
      });
    });
  });
}

Into this more manageable thing:

var archiveOrders = (function(date) {
  var conn = db.connect().wait();
  conn.query("selectom orders where date < ?",  
             [date]).wait().forEach(function(order) {
    conn.execute("insert into archivedOrders ...", 
                 [order.id, ...]).wait();
    conn.execute("delete from orders where id=?", 
                 [order.id]).wait();
  });
  console.log("orders been archived");
}).future();

The entire Promises, Futures debacle depends in part on proper Fibers. Javascript, being a very poor design, does not come with anything built-in and hence the proliferation of Fibers, Deferreds, Promises, Futures implementations that can never reach neither consensus nor people actually using them at large.

So, Fibers are ok. Coroutines are better because you have multiple points of suspending a function and more. And even better, in Erlang one doesn’t need to even think about rudimentary Reactor loops (yes, reactors are a rudimentary construction for concurrency when you have no other good choice): it has transparent asynchronous calls. Everything in Erlang is asynchronous and non-blocking but you don’t deal with callback pyramids because there is something better underneath: the Scheduler.

By the way, I find it very frustrating that Go’s “goroutines” are not proper “coroutines”.

For each Erlang process with SMP (symmetric multiprocessing) support there will be one real thread per CPU core available in your system, and for each thread there will be one single Scheduler to manage the internal green threads (processes) and run-queue.

As a programmer, I don’t have to “remember” to yield control back to a passive event loop. The scheduler will take care of balancing computation time to each concurrent process. If a process is taking too long the Scheduler can choose to suspend it and give time to other routines. Erlang defines a “reduction” and that there are different priority levels of functions. If a function takes more than 2,000 reductions, the Scheduler can choose to suspend it. If you have 8 CPU cores, but the computation on the processes are not heavy, the VM can choose to just use 1 or 2 Schedulers and leave the other 6 idle so the hardware can turn the cores off to save energy (!!). Yep, Erlang is even Eco Friendly!

And we need to repeat this again: because each process is rigidly isolated, with immutable data and no shared state, it’s easier to suspend a running process. In the case of the JVM this is usually implemented by raising checked exceptions and have everybody implement some Suspendable interface. It can be done using this 3rd party Java continuation library where you yield by raising an Exception (!) Nasty stuff.

Rust is still implementing something for coroutines as well, but still nothing as mature. But again, coroutines is just part of the story, you would need a heavier system with userland schedulers for it to make sense. Go is a better candidate to incorporate such a system in its runtime, but it also comes up short on being able to implement all this. There is Suture, an attempt to have some of OTP in Go, but it can’t be done. Even Akka, the first mainstream OTP clone for Scala, can’t come close because of the JVM shortcomings. Clojure with Pulsar/Quasar, comes closer, but is not there yet.

Now, the Erlang Scheduler is not only capable of suspending and resuming processes but it also takes care of message passing between them. So each Scheduler has its own run-queue to queue and dispatch messages. Again, because data is immutable, you only need some form of locking when you want another Scheduler (in another real thread) to take over a few processes in order to balance processing between cores. Erlang has supported SMP since OTP R12B (we are at R18 right now, and still evolving).

Most languages still rely on the OS real threads preemptive model to do multitasking. And this is heavy and slow because of all the context switching involved and all the locking logic most programmers will do wrong (the best practice for concurrency is: do not use threads, chances are you will screw up). Again, we make the right assumptions first: programmers can’t do proper multithreading, so let a Scheduler do it, when necessary, and avoiding slow OS context switching as much as possible. Suspendable green threads combined with a userland Scheduler coordinating cooperative switching is a way faster and safer choice.

If you want to learn more about coroutines this Lua Paper on the subject explains in more details what I just elaborated:

Implementing a multitasking application with Lua coroutines is straightforward. Concurrent tasks can be modeled by Lua coroutines. When a new task is created, it is inserted in a list of live tasks. A simple task dispatcher can be implemented by a loop that continuously iterates on this list, resuming the live tasks and removing the ones that have finished their work (this condition can be signalled by a predefined value returned by the coroutine main function to the dispatcher). Occasional fairness problems, which are easy to identify, can be solved by adding suspension requests in time-consuming tasks.

Static vs Dynamic Typing is still controversial

We saw a heavier movement from bureaucratic static typed systems (mainly Java prior to 6, C# prior to 4, C++) to purely dynamic languages such as Perl in the late 80’s, Python in the late 90’s, Ruby in mid 2000’s. We tried to go from “making the compiler happy” to “making programmers happy”, which makes more sense if you ask me.

Scala, Groovy, Haskell, Swift brought a very practical middle ground with Hindley-Milner derived Type Inference systems, in a way that we can code kinda like in dynamic languages but with the compiler doing more work to infer types for us before generating the final executable bytecode.

But there is a big catch: it’s very difficult to hot swap code inside the runtime if you have static signatures. I’m not saying that it is impossible, but a lot more difficult. You can reload code in Java (one example being Spring Loaded) or Haskell (there is a hotswap plugin and other alternatives). You don’t do granular reloads in a statically typed language because if you want to change the signature of a method, you have to change the graph that depends on that signature. It’s doable, albeit, cumbersome.

In Erlang, because there is no such hard dependencies, and again because of inherent advantages of it only having immutable data with no shared state, and dependencies limited to opaque message passing, you can granularly reload one single module and most important of all: you can implement simple callbacks to transform the old state of a process into a new state structure, because just reloading the code is half the story if you will have hundreds of old processes restarting into the new code but having to deal with previous state. In an Erlang GenServer, you just implement this single callback:

code_change(OldVersion, CurrentState, _Extra) -> {ok, NewState}.

So, while Type Inference is a nice middle ground, the flexibility of Dynamic Typing goes beyond being easy for programmers to use. As with Python, Ruby, Javascript, Perl and other dynamic typed languages, you will want to cover your code with proper test suites - which should not be optional in strong typed languages anyway. There is no doubt a compiler’s static analysis helps a lot, but it’s my personal take that dynamic typing allows me more flexibility.

Fault Tolerance: Don’t fear your code

With Dynamic Typing we will end up again in the endless discussion of “programmers never do it right, we need a compiler to enforce static rules”. And you are almost right: programmers screw up, but a compiler will not save your day anyway, and worst: it might give you a false sense of security. There is no worse security hole than false sense of security. A test suite is way better to assert proper implementation, and it’s still no hard guarantee.

I said that this is controversial, as a more rigid static typing system is more productive than having to unit test every input and output types of every function. But if you thought that, you’re at least wrong in that this is not what unit tests are for: it’s for testing unit behaviors, in spite of what the types are. Testing types is what we call “chicken typing” and it’s another form of defensive programming. You must test for behavior, not for stuff that a compiler would check. And again, having the static typing hinges my flexibility because now I have to constantly fight the typing, add more boilerplate, and ultimately if the behavior is wrong, the code is wrong, despite the type checks.

For a system to be “tolerant to buggy code or uncaught failures” is the opposite of littering your code with guarding statements like “try/catch” or “if err == x” or having some form static level checking with Result monads (that most people will just unwrap anyway). Guards can only go so far. And yes, this is anecdotal as there are no statistics that “everybody will unwrap” and most good programmers won’t, but if experience tells me one thing, it’s that bad programmers will fall for “try/catch” the whole thing when they don’t know what to do anyway.

You need a system that allows any buggy code to fail and not crash the environment around it, which would bring it into an inconsistent, corrupt, state.

The problem with faulty code is that it leaves the state in a position where it has nowhere else to go. And if this state is shared, you leave every other bit of code in a position where they can’t decide where to go next. You will need to shut everything down and restart from the last known good state. Almost all “continuous delivery” workflows are implemented around restarting everything.

Instead of having to bring everything down in case you forget to try/catch something, you can rely on Erlang’s underpinnings of not sharing state, having immutable data, and - most importantly - having the rigidly isolated lightweight process system to make the process that is holding the faulty code to shut down and warn its Supervisor. This is the general idea behind the so called OTP in Erlang.

The idea of a Supervisor is to have a small process that monitors other processes. The Supervisor has as little code as possible in order to rarely (or never) fail (the best kind of code is “no code”). Once a faulty process crashes because of uncaught exception or other undeterministic reasons, it sends a notification message to the Supervisor run-queue mailbox, and then die cleanly. The Supervisor then chooses what to do based on its underlying restart strategy.

Let’s say you have a list of URLs you’re scraping. But you didn’t anticipate dirty structures in your parsing logic. The process doing the scraping crashes and dies. The Supervisor is notified and chooses to restart the process, giving the new process the previous state - the URL list - and now that the faulty URL is not there anymore, the new process can happily continue the job with the next URL in the list.

This is a simple example of the dynamic between the Erlang VM, the Supervisor and its children workers. You can go further and have several Applications, which in turn start up new Supervisors, which in turn start up children processes. And you have a Supervisor Tree that can trap exits and restart granular bits of your runtime code without bringing the rest of the system into an inconsistent state.

Such is the beauty of the rigid process isolation concept.

Every I/O in the system is wrapped in Ports, which obey Async/callback logic transparently without you having to create pyramids of callbacks. The process consuming such ports just gets suspended by the Scheduler until the Async call returns and it can resume work. No callback pyramid hell. No need for rudimentary Fiber implementations to allow for rudimentary Promises/Futures systems. Just coroutines inside processes that can be suspended and resumed by the Scheduler. Fewer opportunities for programmer errors to stack up.

So, a programmer will forget to code every possible branch of execution and because he knows that, the very worst thing that can happen is not if he forgets an uncaught exception, but if he decides to program defensively and add general conditions to trap any error and never raise it. You’ve seen it before, when you find code that is trapped inside generic try/catch blocks, trying to avoid every error possible. But what really happens is that the system may not crash but your logic and your processing is faulty at its core. And you won’t find out about it, because it doesn’t crash, therefore no one is ever notified! You will not end up with fewer bugs, you will end up with a mountain of logic bugs that are never noticed because they are all swallowed!!

This is the core of Fault Tolerance: do not fear Erlang, fear the programmers! Instead, we should do what Joe Armstrong presented in his seminal paper “Making reliable distributed systems in the presence of software errors”. This is both a detailed guide into Erlang and OTP but also his arguments on how to write fault tolerant systems by making the programmer write code as clearly as he originally intended, without the need to be defensive, with the confidence that if he misses something, OTP will be there to catch it and not let the system die, but instead to give a chance to fix it and reload it without disrupting other good parts of the system.

This is the ultimate goal of good programming: not being defensive, not putting try/catch everywhere because you’re afraid.

Summary

This short summary is the reason why Erlang sounds very compelling for my Web Development needs, or for any scalable complex system, at least for me.

Every other language had to make trade-offs. Every new language on top of the JVM has to deal with limitations inherent to how the JVM was originally architected. Rust still can build better abstractions, but its scope is for smaller tools and libraries, not distributed complex systems. Yes, eventually it can do whatever C/C++ can do, and Mozilla is actually basing the core of its next generation browser core on Rust. This will bootstrap better higher level libraries and frameworks, the same way Apple using Objective-C created the entire set of Core frameworks that make implementing complex applications much easier.

Go made a choice for keeping familiarity with its C++ inheritance. Of course, it has tons of useful features, in particular the built-in goroutines that make concurrent coding much easier than in previous languages.

Haskell is too strict for most programmers (yes, Monads, still difficult for the average programmer to fully grasp) and despite contrary opinions, to me it still feels like it appeals more to researchers than everyday developers. Other dynamic languages such as Ruby, Python, may still go the way of Erlang with Ruby adding some immutability (in 2.3 with immutable String by default), but it’s still a long way to go.

Erlang has everything, as Joe Armstrong envisioned a fault tolerant system to be. It started as an exercise implemented in Prolog in 1986. It migrated from the previous JAM compiler to the current BEAM VM in 1998. It added SMP around 2008. It has been gradually evolving, polishing its rough edges, being really battle tested in really mission critical systems for decades. It’s ready for us, right now.

It has Fault Tolerance guaranteed by the principles of immutable data, no shared state, pure opaque message passing, and suspendable processes, all managed by Schedulers. This guarantees that faulty routines can crash one single process but not the entire system and definitely not bringing state of other process to inconsistent, corrupted, states.

Then you can instrument your virtual machine and check that Supervisors are restarting children more than you want and decide to fix the buggy code. And once you do, you can choose not to shut down and restart the entire system to reload the code fixes, you can do it granularly, on the fly, with running processes that will pick up the fixes once the Supervisor restarts it. The granular hot swap of code is guaranteed because there is no hierarchy of types to care about.

And because you have proper coroutines with no hierarchy of shared state dependencies, you can have asynchronous exceptions that can forcefully shut down processes without creating side effects to other running processes. A Supervisor may choose to restart its entire children list when one of its children crashes and another of its children depended on that previous process. You can switch from “one for one” restart strategy to “one for all” (like the Musketeers).

There is only one problem with Erlang: it was not designed for “programmer happiness”, a concept we’ve come to take for granted because of Ruby and newer languages.

Erlang has its roots in Prolog and it shows. Once you step up and really dive deep exercising with the language you can possibly get used to it. But if you came from more modern dynamic languages such as Ruby, Python, Groovy, you will definitely miss the comfortable modern constructions.

Elixir is the missing piece, the Philosopher’s Stone if you will, that can unlock all the 30 years of refinements, maturity, industry battle tested technologies in large scales, to the average programmer.

It brings many modern constructs such as making macros easier in order to allow for Domain Specific Languages, having testable comments in your code, adding a more modern standard library that is easily recognizable from a Ruby or Clojure point of view, polymorphism through Protocols, and so on.

This is one example of Elixir straight from its source code tests:

Code.require_file "../test_helper.exs", __DIR__

defmodule Mix.ArchiveTest do
  use MixTest.Case

  doctest Mix.Archive

  test "archive" do
    in_fixture "archive", fn ->
      File.write ".elixir", "~> 1.0.0"
      Mix.Archive.create(".", "sample.ez")
      archive = 'sample.ez'
      assert File.exists?(archive)
      assert has_zip_file?(archive, 'sample/.elixir')
      assert has_zip_file?(archive, 'sample/priv/not_really_an.so')
      assert has_zip_file?(archive, 'sample/ebin/Elixir.Mix.Tasks.Local.Sample.beam')
      assert has_zip_file?(archive, 'sample/ebin/local_sample.app')
    end
  end

  defp has_zip_file?(archive, name) do
    {:ok, files} = :zip.list_dir(archive)
    Enum.find(files, &match?({:zip_file, ^name, _, _, _, _}, &1))
  end
end

Code that can be written like this automatically makes me grin.

This is the perfect combination. Again, it is not without shortcomings. Erlang is by no means a fast language. It is way faster than Ruby, Python, or other interpreted languages. It can be made a bit faster with native compilation through the HIPE compiler, but still nowhere near the speeds of Go, Rust, or any Java derivative such as Scala, Groovy or Clojure.

So, if you really need raw power of computation, you will want Go or Java. Again, I’m not saying there are only those 2, but it’s the usual choices if you don’t want to go down to C/C++. Haskell has terrific performance, but its learning curve is less than stellar.

Erlang is a whole system, it has its own scheduling system, it controls living, breathing processes that each have its own garbage collector, it controls system signal trapping and so on. It was designed to be a full server. It is much smaller than a full blown Java Enterprise Edition container, so small that you can actually package command line tools that do start up fast enough. But this is not the sweet spot. For that purpose you will be better off with Go or even Rust.

For the same reasons it was not made to be an embeddable language the way Lua is. It was not made to create libraries that can be easily linked to via FFI or C-like function exports, the way Rust can.

There are ways to create desktop-class applications, specially with cross-platform wxWidgets, the way the built-in Observer instrumentation application is done, but Erlang was not built to be a desktop toolkit.

Also because it prioritizes correctness, rigidly isolated processes communicating just by opaque messages, immutable and non-shared states, it means that Erlang is not suited to hard data science processing. So I doubt it’s the best choice for Big Data analytics, DNA sequencing and other hard stuff that tools like Julia, R, Fortran, are better choices. It’s not the same thing to say that it can’t be a good database core, Riak and CouchDB proved that already. But complex queries on top of high volumes of data is not the sweet spot as well.

So, Erlang is good for distributed systems, with high concurrency of opaque message exchange and proxying. The exact scenario where the Web is. Web Applications heavy load of throughput that need real time chats and notifications, heavy and time consuming payment transactions, gathering of data from many sources in order to reduce them to consumable HTML or JSON responses.

But for the average Web developer (and by “average” I mean minimally able to architect the kinds of complex systems we deal with everyday in web development, not simple static website construction), Erlang used to be a real challenge, and now we can have the comfort of a real modern language with hints of Ruby and Clojure, without the complexities of strong typing but with the security of its built-in Fault Tolerance constructions in order to deliver highly reliable, highly scalable, modern Web applications.

For my learning process:

1. I read the entire official Elixir Documentation;
2. I read the entire Dave Thomas’ Programming Elixir book;
3. I watched almost all Elixir Sips screencasts;
4. I read the entire Benjamin Tan Wei Hao’s The Little Elixir & OTP Guidebook book;
5. I did a number of tutorials and exercises.

And finally, I documented everything I learned almost every day in the following articles:

• How Fast is Elixir/Phoenix?
• Personal Thoughts on the Current Functional Programming Bandwagon
• Phoenix Experiment: Holding 2 Million Websocket clients!
• My first week learning Elixir
• Ex Manga Downloader, an exercise with Elixir
• Ex Manga Downloadr - Part 2: Poolboy to the rescue!
• Phoenix “15 Minute Blog” comparison to Ruby on Rails
• Observing Processes in Elixir - The Little Elixir & OTP Guidebook
• ExMessenger Exercise: Understanding Nodes in Elixir
• Elixir 101 - Introducing the Syntax

Yes, I am prolific, and highly focused. I did the learning and writing all within less than a month (it would’ve been less if I could have used night time and weekends). So I’d say that the average developer would take at least 3 to 4 months to cover the same material.

Initial Thinking: Yocto Services!

For good or for worse, we are in the dawn of the Micro Services architecture. In summary it’s fragmenting your monolith into smaller applications that respond to HTTP based endpoints - that people call “APIs” - and that are responsible for a very narrow set of responsibilities. Then you create a “front-end” web application that will consume those services, such as analytics, payment methods, user authentication directories, and so on.

This is no novelty, of course, having HTTP APIs that return JSON structures is just a fancier way of doing the same, good, old Remote Procedure Calls, or RPCs, a technology we have had for decades to interconnect clients and servers in a network. But I digress.

If this is what people call “Micro” services, I think of Elixir processes as “Yocto” Services! (Milli > Micro > Nano > Pico > Femto > Atto > Zepto > Yocto, by the way - I may have just invented a new term here!)

I described a bit of the Processes infrastructure, how to spawn them, how to exchange messages between them, and how to link them together. So, go read my previous post if you haven’t already.

Inside an Elixir app you will find many processes, some from the VM running the show and some from your own app. If you architected it correctly, you’ve implemented an OTP application, with proper groups of Supervisors and Children, all organized in a Supervision Tree. A small worker dies, its supervisor knows how to deal with it.

Now, here’s the mental model: think of an Elixir process as a tiny micro-service - a Yocto service, if you will! - inside your application. In a Phoenix application, for example, you don’t import a “database library” you actually spin up a “Database Service” (which is an Ecto Repo) that runs in parallel with the Endpoint application that responds to HTTP requests coming from the internet. The code in your controllers and models “consume” and send messages to the Ecto Repo “Service”. This is how you can visualize what’s going on.

I have shown the Observer in the previous article as well. You will find a large tree if you open it up inside the IEx shell of a Phoenix application:

In summary you will find the following section in the Pxblog Phoenix App Supervision Tree:

Pxblog.Supervisor
- Pxblog.Endpoint
    + Pxblog.Endpoint.Server
    + Pxblog.PubSub.Supervisor
        * Pxblog.PubSub
        * Pxblog.PubSub.Local
- Pxblog.Repo
    + Pxblog.Repo.Pool

This is the Pxblog I explained in the Phoenix and Rails comparison article I published a few days ago.

I still haven’t read the source code of Phoenix, but if I am interpreting the Observer correctly, the Endpoint.Server controls a pool of processes that are TCP listeners that the application is ready to accept requests from, concurrently, with overflowing to accept more connections (I believe it’s a pool implementation like Poolboy, which I explained in Part 2 of the Ex Manga Downloader article).

Then, you have the PubSub.Supervisor and PubSub.Local applications that I believe support the WebSocket channels.

The Repo alone controls 10 initial processes in its pool, possibly a database connection pool. Notice how Endpoint and Repo groups are in parallel branches in the supervision tree. If Repo fails for some external database problems, the Endpoint group does not have to fail. This is what’s declared in the Pxblog Application definition at lib/pxblog.ex:

defmodule Pxblog do
  use Application

  # See http://elixir-lang.org/docs/stable/elixir/Application.html
  # for more information on OTP Applications
  def start(_type, _args) do
    import Supervisor.Spec, warn: false

    children = [
      # Start the endpoint when the application starts
      supervisor(Pxblog.Endpoint, []),
      # Start the Ecto repository
      worker(Pxblog.Repo, []),
      # Here you could define other workers and supervisors as children
      # worker(Pxblog.Worker, [arg1, arg2, arg3]),
    ]

    # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: Pxblog.Supervisor]
    Supervisor.start_link(children, opts)
  end
  ...
end

See how it defines Endpoint and Repo under the Pxblog.Supervisor.

I can go on and forcefully kill the entire Pxblog.Repo node from the Supervision Tree using the Observer, like I did in the previous article, and the proper strategy kicks in, the Phoenix Supervisor successfully restarts the Repo, and no one will ever notice something crashed underneath.

From the IEx I can still make more calls to the Repo like this and it responds as if it had never crashed:

iex(4)> Pxblog.Repo.all(Pxblog.User)
[debug] SELECT u0."id", u0."username", u0."email", u0."password_digest", u0."inserted_at", u0."updated_at" FROM "users" AS u0 [] OK query=78.2ms queue=3.2ms
[%Pxblog.User{__meta__: #Ecto.Schema.Metadata<:loaded>,
  email: "akitaonrails@me.com", id: 1,
  inserted_at: #Ecto.DateTime<2015-11-20T14:01:09Z>, password: nil,
  password_confirmation: nil,
  password_digest: "...",
  posts: #Ecto.Association.NotLoaded<association :posts is not loaded>,
  updated_at: #Ecto.DateTime<2015-11-20T14:01:09Z>, username: "akitaonrails"}]

And the way I think about this is: my IEx shell is sending a message to the Yocto Service called Pxblog.Repo (in reality it’s forwarding messages to the database adapter that then checks out a process from the pool). Just like I would consume external Micro Services through HTTP APIs.

So the landscape of your application is comprised of a series of processes and groups of supervised processes all working to compose a larger structure. As I said in previous articles, should one group of process collapse, its Supervisor kicks in, traps the error and uses its strategy to, for example, restart all of its children processes, bringing the app to a consistent state once more, and without you having to restart the entire Elixir application.

So each process can be a full blown Yocto Service, running online and waiting for other services to consume it, such as the Repo’s workers.

Divide and Conquer

Again, as a disclaimer, I am still new to Elixir but the way I find easier to understand it, is like this:

• If you must deal with external resources, be it a File, or a Network Connection, or anything outside of the Erlang VM, you will want it to be a GenServer.

• Then, if you have a GenServer, you want to start it up under a Supervisor (usually, the simple boilerplate that defines the children and restart strategy).

• The number of GenServer processes you want to start up depends on how many parallel processes you want to have running. For example, if it is a database service, there is no point in starting up more than the available number of maximum connections your database allows. If you have several files to process, you want at most the number of available files or - in practice - just a few processes to deal with batches of files. You will usually want a pool of GenServer processes, and in this case you want to use Poolboy.

• One GenServer may call other GenServers. You don’t want to use a try/catch exception handling mechanism because you just need the particular GenServer process to crash if something goes wrong: if the file is corrupted or doesn’t exist or if the network goes unstable or disconnects. The Supervisor will replace that process with a new GenServer process in its place and refill the Pool if needed.

• You can make GenServers talk remotely using the Node feature I explained 2 posts ago, with the ExMessenger example. Then it would be like a normal Micro Services architecture, but where the inside Yocto Services are actually doing the talking.

• Any transformation with no side-effects (transforming a simple input into a simple output), like getting an HTML body string and parsing it into a List of tuples, can be organized in a normal Module. Refer to libraries like Floki, to see how they are organized.

Each Erlang VM (called BEAM) is a single OS process, that manages a single real thread per CPU core available in your machine. Each real thread is managed by its own BEAM Scheduler, which will slice processing time between the lightweight processes inside.

Each BEAM process has its own mailbox to receive messages (more correctly called a run-queue). I/O operations such as file management will run asynchronously and not block the scheduler, which will manage to run other processes while waiting for I/O.

Each BEAM process also has its own separated heap and garbage collector (a generational, 2 stage copy collector). Because each process has very little state (variables in a function) each garbage collector stop is super short and runs fast.

So, each BEAM VM can be thought of as an entire application infrastructure, with many Yocto Services available for your application to call.

And as I said in the previous article, each BEAM VM can remotely call other BEAM VMs and exchange messages between them as if they were in the same VM. The semantics are almost the same and you have distributed computing the easier way.

Erlang implemented a fantastic set of primitives that quickly scale as large as you want or need, with the proper wirings to not let you down. And Elixir fixes the only problem most people have with Erlang: its old Prolog-inspired weird syntax. Elixir is a thin and effective coat of modern language design, partly inspired by Ruby (although not a port by any means, it’s still Erlang underneath).

I hope this series helped shed some light on why Elixir is the best choice among the new generation of high concurrency enabled languages and I also hope I made myself clear on why just high concurrency is not enough: you want high reliability as well. And in this aspect, the OTP architecture, built-in with Erlang, has no competition.

This is Elixir pre-1.0, and because it’s an exercise I refactored the original code and merged the server (ex_messenger) and client (ex_messenger_client) projects into an Elixir Umbrella project and you can find my code on Github here.

If you have multiple applications that work together and share the same dependencies you can use the Umbrella convention to have them all in the same codebase. If you mix compile from the umbrella root, it compiles all the apps (which are independent Elixir mix projects as well). It’s just a way to have related apps in the same place instead of in multiple different repositories.

The code shown here is in my personal Github repository if you want to clone it.

Nodes 101

Before we check out the exercise, there is one more concept I need to clear up. In the previous article I explained how you can start processes and exchange messages, and how you can use the OTP GenServer and Supervisor to create more robust and fault-tolerant processes.

But this is just the beginning of the story. You’ve probably heard how Erlang is great for distributed computing as well. Each Erlang VM (or BEAM) is network-enabled.

Again, this is one more concept I am still just beginning to properly learn, and you will want to read Elixir’s website documentation on Distributed tasks and configuration, that does an excellent job explaining how all this works.

But just to get started you can simply start 2 IEx sessions. From one terminal you can do:

iex --sname fabio --cookie chat

Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false]

Interactive Elixir (1.1.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(fabio@Hal9000u)1> 

And from a different terminal you can do:

iex --sname akita --cookie chat

Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false]

Interactive Elixir (1.1.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(akita@Hal9000u)1> 

Notice how the IEx shell shows different Node names for each instance: “fabio@Hal9000u” and “akita@Hal9000u”. It’s the sname concatenated with your machine name. From one instance you can ping the other, for example:

iex(akita@Hal9000u)2> Node.ping(:"fabio@Hal9000u")
:pong

If the name is correct and the other instance is indeed up, it responds to the ping with a :pong. This works only for nodes on the same machine, but what if I need to connect to an instance on a remote machine?

iex(akita@Hal9000u)3> Node.ping(:"fabio@192.168.1.13")

11:02:46.152 [error] ** System NOT running to use fully qualified hostnames **
** Hostname 192.168.1.13 is illegal **

The –sname option sets a name only reachable within the same subnet; for a fully qualified domain name you need to use –name, for example like this:

iex  --name fabio@192.168.1.13 --cookie chat

And for the other node:

iex --name akita@192.168.1.13 --cookie chat

And from this second terminal you can ping the other node the same way as before:

iex(akita@192.168.1.13)1> Node.ping(:"fabio@192.168.1.13")
:pong

And you might be wondering, what is this “–cookie” thing? Just spin up a third terminal with another client name, but without the cookie, like this:

iex --name john@192.168.1.13

And if you try to ping one of the first two nodes you won’t get a :pong back:

iex(john@192.168.1.13)1> Node.ping(:"fabio@192.168.1.13")
:pang

The cookie is just an atom to identify relationships between nodes. In a pool of several servers you can make sure you’re not trying to connect different applications to each other. And as a result you get a :pang. Instead of an IP address you can use a fully qualified domain name.

And just by having the node “akita@” pinging “fabio@” we can see that they are aware of each other:

iex(fabio@192.168.1.13)2> Node.list
[:"akita@192.168.1.13"]

And:

iex(akita@192.168.1.13)2> Node.list
[:"fabio@192.168.1.13"]

If one of the nodes crashes or quits, the Node list is automatically refreshed to reflect only nodes that are actually alive and responding.

You can check the official API Reference for the Node for more information. But this should give you a hint for the next section.

Creating a Chat Client

Back to the exercise: the ExMessenger server has “ExMessenger.Server”, which is a GenServer, and the “ExMessenger.Supervisor” that starts it up. The ExMessenger.Server is globally registered as :message_server, started and supervised by the “ExMessenger.Supervisor”.

The “ExMessengerClient” starts up the unsupervised “ExMessengerClient.MessageHandler”, which is also a GenServer, and globally registered as :message_handler.

The Tree for both apps looks roughly like this:

ExMessenger
- ExMessenger.Supervisor
    + ExMessenger.Server
ExMessengerClient
- ExMessengerClient.MessageHandler

We start them separately, first the message server:

cd apps/ex_messenger
iex --sname server --cookie chocolate-chip -S mix run

Notice that for this example we are starting with a simple name “server”, for the local subnet, and a cookie. It will respond as “server@Hal9000u” (Hal9000u being my local machine’s name).

Then, we can start the client app:

cd apps/ex_messenger_client
server=server@Hal9000u nick=john elixir --sname client -S mix run

Here we are setting 2 environment variables (that we can retrieve inside the app using System.get_env/1) and also setting a local node name “client”. You can spin up more client nodes using a different “sname” and a different “nick” from another terminal, as many as you want, all linking to the same “server@Hal9000u” message server.

I’m starting up like this instead of using a command-line escript like I did in ExMangaDownloadr because I didn’t find any way to set the –sname or –name the same way I can set –cookie using Node.set_cookie. If anyone knows how to set it up differently, let me know in the comments section down below.

Notice that I said “linking” and not “connecting”. From the “ExMessengerClient” we start like this:

defmodule ExMessengerClient do
  use Application
  alias ExMessengerClient.CLI
  alias ExMessengerClient.ServerProcotol

  def start(_type, _args) do
    get_env
      |> connect
      |> start_message_handler
      |> join_chatroom
      |> CLI.input_loop
  end
  ...
end

The get_env private function is just a wrapper to handle the “server” and “nick” environment variables that we passed:

defp get_env do
  server = System.get_env("server")
    |> String.rstrip
    |> String.to_atom
  nick = System.get_env("nick")
    |> String.rstrip
  {server, nick}
end

Now, we try to connect to the remote server:

defp connect({server, nick}) do
  IO.puts "Connecting to #{server} from #{Node.self} ..."
  Node.set_cookie(Node.self, :"chocolate-chip")
  case Node.connect(server) do
    true -> :ok
    reason ->
      IO.puts "Could not connect to server, reason: #{reason}"
      System.halt(0)
  end
  {server, nick}
end

The important piece here is that we are setting the client instance’s cookie with Node.set_cookie/1 (notice that we didn’t pass it in the command-line options like we did with the server instance). Without setting the cookie, the next line with Node.connect(server) would fail to connect, as I explained in the previous section.

Then, we start the “ExMessengerClient.MessageHandler” GenServer, linking with the Message Server instance:

defp start_message_handler({server, nick}) do
  ExMessengerClient.MessageHandler.start_link(server)
  IO.puts "Connected"
  {server, nick}
end

The Message Handler GenServer itself is very simple. It just sets the server as the state, handles incoming messages from the server and prints them out in the client’s terminal:

defmodule ExMessengerClient.MessageHandler do
  use GenServer

  def start_link(server) do
    :gen_server.start_link({ :local, :message_handler }, __MODULE__, server, [])
  end

  def init(server) do
    { :ok, server }
  end

  def handle_cast({ :message, nick, message }, server) do
    message = message |> String.rstrip
    IO.puts "\n#{server}> #{nick}: #{message}"
    IO.write "#{Node.self}> "
    {:noreply, server}
  end
end

Going back to the main “ExMessengerClient” module, after starting the (unsupervised) GenServer that receives incoming messages, we proceed to join the pseudo-chatroom in the server:

defp join_chatroom({server, nick}) do
  case ServerProcotol.connect({server, nick}) do
    {:ok, users} ->
      IO.puts "* Joined the chatroom *"
      IO.puts "* Users in the room: #{users} *"
      IO.puts "* Type /help for options *"
    reason ->
      IO.puts "Could not join chatroom, reason: #{reason}"
      System.halt(0)
  end
  {server, nick}
end

I defined this “ServerProcotol” module, which is just a convenience wrapper for GenServer.call/3 and GenServer.cast/2 calls, to send messages to the remote GenServer called :message_server:

defmodule ExMessengerClient.ServerProcotol do
  def connect({server, nick}) do
    server |> call({:connect, nick})
  end

  def disconnect({server, nick}) do
    server |> call({:disconnect, nick})
  end

  def list_users({server, nick}) do
    server |> cast({:list_users, nick})
  end

  def private_message({server, nick}, to, message) do
    server |> cast({:private_message, nick, to, message})
  end

  def say({server, nick}, message) do
    server |> cast({:say, nick, message})
  end

  defp call(server, args) do
    GenServer.call({:message_server, server}, args)
  end

  defp cast(server, args) do
    GenServer.cast({:message_server, server}, args)
  end
end

Pretty straightforward. Then, the main ExMessengerClient calls the recursive input_loop/1 function from the CLI module, which just receives user input and handles the proper commands using pattern matching, like this:

defmodule ExMessengerClient.CLI do
  alias ExMessengerClient.ServerProcotol

  def input_loop({server, nick}) do
    IO.write "#{Node.self}> "
    line = IO.read(:line)
      |> String.rstrip
    handle_command line, {server, nick}
    input_loop {server, nick}
  end

  def handle_command("/help", _args) do
    IO.puts """
    Available commands:
      /leave
      /join
      /users
      /pm <to nick> <message>
      or just type a message to send
    """
  end

  def handle_command("/leave", args) do
    ServerProcotol.disconnect(args)
    IO.puts "You have exited the chatroom, you can rejoin with /join or quit with /quit"
  end

  def handle_command("/quit", args) do
    ServerProcotol.disconnect(args)
    System.halt(0)
  end

  def handle_command("/join", args) do
    ServerProcotol.connect(args)
    IO.puts "Joined the chatroom"
  end

  def handle_command("/users", args) do
    ServerProcotol.list_users(args)
  end

  def handle_command("", _args), do: :ok

  def handle_command(nil, _args), do: :ok

  def handle_command(message, args) do
    if String.contains?(message, "/pm") do
      {to, message} = parse_private_recipient(message)
      ServerProcotol.private_message(args, to, message)
    else
      ServerProcotol.say(args, message)
    end
  end

  defp parse_private_recipient(message) do
    [to|message] = message
      |> String.slice(4..-1)
      |> String.split
    message = message
      |> List.foldl("", fn(x, acc) -> "#{acc} #{x}" end)
      |> String.lstrip
    {to, message}
  end
end

And this wraps up the Client.

Creating a Chat Server

The Chat Client sends GenServer messages to a remote {:message_server, server}, and in the example, server is just the sname “server@Hal9000u” atom.

Now, we need this :message_server and this is the “ExMessenger.Server” GenServer:

defmodule ExMessenger.Server do
  use GenServer
  require Logger

  def start_link([]) do
    :gen_server.start_link({ :local, :message_server }, __MODULE__, [], [])
  end

  def init([]) do
    { :ok, HashDict.new }
  end
  ...
end

And that’s it. When the “ExMessenger.Supervisor” starts this GenServer it registers it globally in this instance as :message_server. And this is how we address messages from what we called “clients” (the ExMessengerClient application).

When the ExMessengerClient calls the ServerProtocol.connect/1, it sends the {:connect, nick} message to the server. In the Server we handle it like this:

def handle_call({ :connect, nick }, {from, _} , users) do
  cond do
    nick == :server or nick == "server" ->
      {:reply, :nick_not_allowed, users}
    HashDict.has_key?(users, nick) ->
      {:reply, :nick_in_use, users}
    true ->
      new_users = users |> HashDict.put(nick, node(from))
      user_list = log(new_users, nick, "has joined")
      {:reply, { :ok, user_list }, new_users}
  end
end

First, it checks if the nick is “server” and disallows it. Second, it checks if the nickname already exists in the internal HashDict (a key/value dictionary) and refuses if it already exists. Finally, third, it puts the pair of nickname and node name (like “client@Hal9000u”) in the HashDict and broadcasts through the log/3 private function to all other nodes in the HashDict dictionary.

The log/3 just creates a log message concatenating the nicknames of all clients and prints it out, then broadcasts it to the Message Handler of all the clients listed in the HashDict:

defp log(users, nick, message) do
  user_list = users |> HashDict.keys |> Enum.join(":")
  Logger.debug("#{nick} #{message}, user_list: #{user_list}")
  say(nick, message)
  user_list
end

def say(nick, message) do
  GenServer.cast(:message_server, { :say, nick, "* #{nick} #{message} *" })
end

def handle_cast({ :say, nick, message }, users) do
  ears = HashDict.delete(users, nick)
  Logger.debug("#{nick} said #{message}")
  broadcast(ears, nick, message)
  {:noreply, users}
end

Up to this point it just casts a message to itself, the {:say, nick, message} tuple, which is handled by the GenServer and calls the broadcast/3 function defined like this:

defp broadcast(users, nick, message) do
  Enum.map(users, fn {_, node} ->
    Task.async(fn ->
      send_message_to_client(node, nick, message)
    end)
  end)
  |> Enum.map(&Task.await/1)
end

defp send_message_to_client(client_node, nick, message) do
  GenServer.cast({ :message_handler, client_node }, { :message, nick, message })
end

It maps the list of users and fires up an asynchronous Elixir Task (which is itself just a GenServer, as I explained before in the Ex Manga Downloader series). Because it’s a broadcast, it makes sense to make all of them parallel.

The important bit is the send_message_to_client/3 which casts a message to the tuple { :message_handler, client_node } where “client_node” is just “client@Hal9000u” or whatever “–sname” you used to start up each client node.

Now, this is how the clients send GenServer message calls/casts to {:message_server, server} and it sends messages back to {:message_handler, client}.

This is not your traditional TCP Client/Server example!

Now, we are calling the “ExMessenger.Server” a Chat “Server” and the “ExMessengerClient” a Chat “Client”. Although we have been calling them “Server” and “Client”, they don’t refer to the usual “TCP Server” and “TCP Client” examples you may be familiar with!

The ExMessenger.Server is indeed a Server (an OTP GenServer) but the ExMessengerClient.MessageHandler is also a Server (another OTP GenServer)! Because they both have Node behavior, it’s more like they are 2 peer-to-peer nodes instead of your old-school, simple client-server relationship. They can have client behavior (the Server sends messages to the MessageHandler) and server behavior (the Server receives messages from ExMessengerClient).

Let this concept sink in for a moment: built into the language you get a full-blown, easy-to-use, peer-to-peer network distribution model. You don’t need to have one single node elected as the sole “master”; you could have all nodes in a ring coordinating between them, avoiding single points of failure.

I believe this is maybe how Erlang-based services such as ejabberd and RabbitMQ work.

In the case of ejabberd, I can see that it keeps the state of the cluster in Mnesia tables (Mnesia being another component of OTP, a built-in distributed NoSQL database!) and it indeed uses the Node facilities to coordinate distributed nodes:

...
join(Node) ->
    case {node(), net_adm:ping(Node)} of
        {Node, _} ->
            {error, {not_master, Node}};
        {_, pong} ->
            application:stop(ejabberd),
            application:stop(mnesia),
            mnesia:delete_schema([node()]),
            application:start(mnesia),
            mnesia:change_config(extra_db_nodes, [Node]),
            mnesia:change_table_copy_type(schema, node(), disc_copies),
            spawn(fun()  ->
                lists:foreach(fun(Table) ->
                            Type = call(Node, mnesia, table_info, [Table, storage_type]),
                            mnesia:add_table_copy(Table, node(), Type)
                    end, mnesia:system_info(tables)--[schema])
                end),
            application:start(ejabberd);
        _ ->
            {error, {no_ping, Node}}
    end.

This is what a snippet of pure Erlang source code looks like, by the way. You should have enough Elixir in your head right now to be able to abstract away the ugly Erlang syntax and see that it’s a case pattern matching on the {_, :pong} tuple, using Node’s ping facilities to assert the connectivity of the node and updating the Mnesia table and other setups.

Also in the source code of the RabbitMQ-Server you will find a similar thing:

become(BecomeNode) ->
    error_logger:tty(false),
    ok = net_kernel:stop(),
    case net_adm:ping(BecomeNode) of
        pong -> exit({node_running, BecomeNode});
        pang -> io:format("  * Impersonating node: ~s...", [BecomeNode]),
                {ok, _} = rabbit_cli:start_distribution(BecomeNode),
                io:format(" done~n", []),
                Dir = mnesia:system_info(directory),
                io:format("  * Mnesia directory  : ~s~n", [Dir])
    end.

Again, pinging nodes, using Mnesia for the server state. Erlang’s syntax is uncommon for most of us: variables start with a capitalized letter (we intuitively think they’re constants instead), statements end with a dot, instead of dot-notation to call a function from a module it uses a colon “:”, unlike Elixir the parentheses are not optional, and so on. Trying to read code like this shows the value of having Elixir to unleash Erlang’s hidden powers.

So, up to this point, you know how processes are spawned internally, how they are orchestrated within the OTP framework, and now how they can interact remotely through the Node peer-to-peer abstraction. And again, this is all built into the language. No other language comes even close.

This behavior is what I also call “Amateur Hysteria.” And I say this with total peace of mind because I myself have had moments like this.

First of all, let’s define what is commonly called “Legacy”: basically any code done up until yesterday, especially if you weren’t the one who made it, is considered “Legacy.”

But what most programmers fail to understand is very simple: with that definition, any code will always automatically become “legacy” the minute after it’s finished: including your own.

The neighbor’s garden always looks greener? For programmers, the neighbor’s code always looks worse. How many times have we heard?

• “How could it take all this time to make this code? I would have done it in less time.”
• “How could the code have been made this ugly way? I would have done it better.”

Context, context, context

Why did a certain project take “more time than I think it should have?” Dozens of reasons: departments that didn’t collaborate, poorly defined business rules that changed several times, and so on.

Why isn’t the code as “elegant” or “well done” as “I would do it?” Because it was born with a goal in mind, and that goal changed, the way it always changes. The code piled up, the deadlines tightened, the refactorings didn’t happen the way they should. Technical debt grew despite the warnings of the programmers involved, and the result ended up much worse than it should be.

Another thing that should be obvious: every programmer will always find something they don’t like in any code, whether it’s effectively bad and poorly made or even well-made and well-structured. Software is so complex you can find a flaw in anything. It’s like pointing out flaws in a human being: everyone is imperfect.

The fallacy of whoever is listening is to think that just because someone pointed out a few flaws, everything is automatically bad. It’s the reasoning that goes: “my new programmer said they found this flaw and this other poorly done thing, so the system is bad.” Except you didn’t look at the qualities and didn’t weigh whether the flaws actually outrank them in volume, criticality, and severity.

More than that: any new code will also have points another programmer won’t like. And this cycle is infinite.

Now the truth: the programmer who arrived later and questioned all these points, had they been at the exact same moment when everything started, would have delivered the same bad code as the final result, or even worse.

Fear

Now the challenge is to understand the following. In most cases, “Legacy” code is code that is in production, in use, generating results. Something your “new and elegant” code isn’t doing yet.

Another story that many programmers prefer to ignore is the “Big Rewrite” that was never concluded and never went live. Especially when you were the one who worked on it.

The worst type of software is that which doesn’t generate value. And a factor that differentiates a professional programmer from an amateur is exactly how they deal with others’ code. Talking yesterday with Rodrigo Yoshima, he dropped a pearl: dealing with legacy is for adults, and any child can do “greenfield” (a project done from scratch).

It takes much more technique, much more skill, much more capability, to take a legacy and make it better, even if in some cases it is actually necessary to redo some parts. But that shouldn’t be the first automatic alternative without justification. “Because I would do it better” isn’t justification.

The Environment

The process that facilitates teams generating bad code is always one of the hardest problems to solve. Bosses, clients who ask for “urgent last-minute changes” is one of the symptoms. Departments with different priorities. In environments like this, any new software will quickly be made worse to the point where it seems necessary to redo everything again.

I won’t repeat everything the Agile communities have already said to this day, but the “fix” most of the time doesn’t just involve programming techniques, but rather programmers who can see beyond the code and understand how to help fix the environment around them. This includes the programmer colleagues around, the bosses above, the demands coming from surrounding departments, and all the feedback coming from the market in the outside world.

In the real world, software is made by people for people. Understanding that people aren’t difficult means the challenge for a professional is much greater than following good practices or being specialized in this or that technique. A good professional knows that just complaining and sulking doesn’t lead anywhere.

Opportunities

As always, stories aren’t rules, but they serve to illustrate a point. Years ago I was allocated alone to a big client who had a small “legacy” system that was extremely poorly made. A one-off allocation. And I can say “poorly made” because it effectively didn’t execute as it should. It was technically flawed and returned many errors. It was exactly the kind of code that would make any programmer say, immediately, “throw it out and do it again!”

I had 2 weeks to do something. Others had already tried, the client was already convinced to make a new one. No one would think it bad if I said nothing could be done besides starting from scratch. It was an attempt. And I hate saying I can’t solve a problem.

They needed the current one to work minimally. I was a Java programmer at the time, and the system was one of the worst ASPs I had ever seen up to that point, mixed with several DCOM components that miraculously executed somehow.

In 2 weeks I understood what the business rule they wanted was. I understood why the code didn’t work according to these rules. I understood where the data feeding it came from, how it was handled and stored, what results were expected, and how I should intervene to implement that processing.

I used exactly the same code, did a minimal sanity cleanup, removed what was unnecessary, and inserted the minimum necessary to reach the goals expected by the client within the deadline that was stipulated for me.

During those 2 weeks I understood more about the process, understood more about the client, understood more about the environment. Besides fixing the software I managed to build a good relationship. This good relationship, in a few months, led me to return to work for this client but this time on a 1-and-a-half-year project with a team of almost 10 people, where I could coordinate the development my way.

In this case, it was clear that the code was done in bad faith by a programmer who didn’t really know what they were doing. Many pieces of software are really done by simply bad programmers. But I understand that many cases are also good programmers who had to solve a problem within a series of restrictions they had to deal with.

To me, “legacy” is software done precisely by those who call others’ software “legacy.” For me, “legacy” can often be an opportunity, precisely because few can handle it. And for me, people who judge others’ code without knowledge of the whole context are just another amateur wanting to show off.

Of course, this isn’t a defense that all legacy software is justified and good. Not at all. My point here is focused on the automatic behavior of programmers who use this as an excuse.

He answered everything in colorful detail and we speak a lot about his open source projects, performance, scalability and, of course, lots of Git and Github stuff. Hopefully it will make people even more excited about how the Ruby/Rails community is moving things forward all the time.

aos leitores brasileiros: assim que tiver tempo irei traduzir esta entrevista.

AkitaOnRails: Ok, so, where are you from?

Chris Wanstrath: Right now I live in San Francisco but I grew up in Cincinnati, Ohio. I moved to San Francisco in 2005 to work for C|NET where I did PHP for a while on sites like gamespot.com, mp3.com, and tv.com

In 2006 I moved over to the chowhound.com site within C|NET to work on Rails, which is something I had been doing on the side for a while. While I was there P.J. Hyett and myself worked on chowhound.com then built chow.com, which we launched later that year. Then in April of 2007 we both left C|NET to form our consulting company, Err Free.

In December of 2007 we launched FamSpam, which is a site for keeping in touch with your family. It’s basically a simplified version of Google Groups with emphasis on participating in a thread at least once a week and easily finding photos or other file uploads.

While P.J. and I were working on FamSpam, Tom Preston-Werner and I were working on GitHub on the side. We started in October working nights and weekends on it. We basically wanted an easy, pretty, and feature-rich way to throw our Git repos up and share them with people. Then as we got more into it, we realized how things like the dashboard could change the way we deal with open source contributions. After that, all the social features began to fall into place. Github private beta launched in january, and in april we saw the public launch

AkitaOnRails: Wow, Cool, you are a very busy guy indeed, I want to go on in details within each topic you mentioned but first I knew you worked for C|Net because of your memcached keynotes. I have 2 questions: you were a PHP programmer, what made you switch to Ruby on Rails? And then, working for a high profile network like C|Net exposed you to very demanding public websites. How did you figured out about leveraging memcached with Rails and then your cache_fu plugin?

Chris Wanstrath: I started doing Rails at the start of 2005 and got my first PHP job in February of 2005. (before that I was doing asp and Perl, which I don’t like to talk about). So really, I never considered myself a “PHP programmer” over a Rails programmer. I really liked websites and I knew I could get paid doing PHP, so that’s what I did. In the meantime, I was constantly doing Ruby on the side.

When C|Net started doing Rails, I knew it would be a perfect fit for me. I really liked PHP and loved Gamespot, but it was just silly for me to spend all night playing with Ruby while doing PHP professionally.

In fact, one of the things that got me hired at Gamespot was my pure PHP yaml parser called SPYC (simple PHP yaml class). It was inspired (obviously) by _why’s syck yaml parser. I wanted an easy, extension-free way to deal with YAML in PHP because I was so spoiled by Ruby.

I think today the CakePHP and Symfony frameworks use it, and do their config stuff in yaml. But even when writing PHP I wanted to be writing Ruby. The main reason I picked up ruby in the first place is because I knew PHP, and I knew Perl, but I wanted something I could use to write both websites and command line scripts. Perl was great for scripts, and PHP was great for sites, but I was just tired of using them both

So while I was learning Python, Rails popped up.

As far as cache_fu goes, I was very interested in some of the “scaling” aspects of running a website while working at Gamespot.

While good OOP design and stuff like that can be learned by anyone writing enough code, you can’t really learn about scaling unless you’re working on a big website. There are just too many tricks and moving parts, and it’s too hard to simulate the load in a controlled environment.

So while at Gamespot, like I said, I tried to learn as much as possible about things like MySQL replication, clustering databases, sharding databases, and memcached based on how the smart guys at C|Net had set it all up over the years. They had a lot of machines serving a lot of traffic so it was perfect

At Chowhound, we were re-writing an old static html site in Rails and were worried about the transition from flat files to dynamic database calls and page generation. It had a fair amount of traffic from a group of rabid fans, and we knew the new “web 2.0” look of the site would not sit well with the old school users, so we at least wanted the site to be performant in order to give them one less thing to complain about. So we basically took a lot of what I had learned at Gamespot, and what information was freely available on the internet in mailing lists and discussions, and tried to do a more Rails friendly way of object caching with memcached.

Ruby is just so good at cleaning up repetition and creating friendly apis. So cache_fu does fragment caching and object caching, because in bigger sites you end up using both for different purposes which is how it was at Gamespot too, basically.

AkitaOnRails: Nice. And it is not very common for me to find people that have dealt with really big websites. Can you give us a glimpse of what kind of load your Rails apps did have to handle? Some pundits I know complain about the vanilla Rails distribution and recommend lots of tweaks like dumping ActiveRecord altogether and several other hacks. Do you have to hack Rails a lot in order to achieve the scalability you need or do you think most of it is good enough so you can avoid too much tweaking?

Chris Wanstrath: I wouldn’t really consider Chowhound to be a big Rails site anymore. Now that you’ve got scribd and twitter and friends for sale and Yellowpages, Chowhound doesn’t really make the list.

We ended up doing a lot of premature optimization on that site, and not enough real world optimization. I learned a lot of lessons there that I’ve applied to other sites along the way. However, I recently had the chance to work with the friends for sale guys on their Facebook app which gets millions of page views a day and handles an insane amount of db records.

To my knowledge, the only AR hack we did was apply Pratik Naik’s partial updates patch. This was because, with so many rows and such large indexes, you want to avoid modifying indexes when you don’t need to, so saving only the changed columns/attributes can have a real big performance gain. Luckily, partial updates are now in Rails Core.

In my experience, you spend a lot more time trying to scale SQL before you even need to address AR. And once you get to the AR part, it’s usually your application code. Fetching and saving too many records in before/after_save callbacks, full row writes like I was just describing, and generally not being a good SQL citizen. I would only ditch AR completely if I felt like I had no control over the SQL it generated. Which, as anyone knows, is not the case – you have a lot of control over it.

There are times, however, when AR is overkill and you really do need speed. On Github, we need to authenticate you every time you try and pull or push a repository over ssh. Loading an entire Rails environment, or even activerecord, was noticeably slow, so we switched to using sequel in a simple ruby script. Because we don’t need AR’s massive featureset and nice OOP capabilities for such a simple single row find, this made a lot of sense for us.

AkitaOnRails: I agree with that, that’s what I experienced too with regards to AR. So, and you’re also the author of several other famous Rails plugins like will_paginate, Ambition, the original Sexy Migrations. Can you comment briefly about each. Am I forgetting any other plugins? Some of what you did became the ‘de facto’ standards for Rails development, some people here use these plugins without really knowing you made them all.

P.J Hyett at SXSW'06

Chris Wanstrath: Err the blog is actually two people, P.J. Hyett and myself. I did not write will_paginate – P.J. wrote the original version then Mislav Marohnić re-wrote it. Mislav is now the maintainer. But I still get all the credit, which is fine by me.

As far as the other plugins go, there’s mofo, ambition, sexy migrations, cache_fu, acts_as_textiled, fixture_scenarios_builder, and gibberish.

Mofo is a microformat parser based on Hpricot. It’s up on Github and unfortunately doesn’t get enough love from me these days, but as far as I know it’s still the gold standard for microformat parsing in Ruby.

It was the first gem that I released which doubled as a Rails plugin, which is how I think most plugins should be distributed in the future. Gems already have a solid distribution system and versioning scheme, and if Github becomes a gem source then there’s almost no reason not to release plugins as gems

Sexy migrations were inspired by Hobo. Unfortunately their version was hobo-specific, so I wrote a plugin-friendly version. DHH later wrote his own version and rolled it into core, making a grand total of 3 distinct sexy migration implementations.

Gibberish is a pretty fun localization plugin I wrote pro bono for a client, just because I felt like the APIs of all the existing localization plugins were so difficult to work with. Hopefully, with the help of anyone interested, we can begin localizing Github in the near future.

Ambition was inspired by dabbling in Erlang. I read a blog post explaining that mnesia’s query syntax, which is all list comprehension (basically Ruby’s version of the Enumerable module), was made possible by the ability to walk the parse tree and build an mnesia query that way.

So I started playing with the parse_tree gem to see if I could do the same for Ruby. It’s now, of course, a full fledged framework for generating arbitrary queries for any RubyGem based on Enumerable. For simple queries, I really like treating tables as arrays, it feels very natural.

Acts_as_textiled was the first Rails plugin I released. It’s a testament to Rails that the plugin still works on edge today. The only other notable thing that Err started is the “vendor everything” term, which is now a part of Rails through config.gems.

That, and the Cheat gem (and site).

Tom Preston-Werner

AkitaOnRails: (Of course, pardon me P.J.!) And this raises another curiosity of mine: where did the name “Err” come from? It is both for Err the Blog and Err Free which is your company. And you mentioned P.J. and Tom. And you’re all full time Railers, right? How did you team up with them and decide to start your own company? Many times people ask me about entrepreneurship and you’re another good example.

Chris Wanstrath: “err” is kind of a play on “Typo,” which was the popular Rails blogging engine when we started our blog, so it may be kind of confusing, but I actually am cofounder of two distinct companies.

There is Err Free and Logical Awesome. Err Free is Ruby & Rails consulting and training. P.J. and I own the company, and we use it to do client work, speaking gigs, corporate training, that sort of thing. It also owns and operates FamSpam, which we use with our own families.

Logical Awesome, however, was founded by me and Tom. This is because I was developing FamSpam with P.J. and GitHub with Tom at the same time, separately. Today, however, P.J. is a part of Logical Awesome and is one of the GitHub developers.

Tom works at Powerset and doesn’t actually do much Rails for GitHub, he mainly does flash, haxe, C, css, design, and ruby. And because he’s the author of God, I make him write all the config files, too. As far as deciding to start a company, I am someone that needs to be involved in whatever I’m involved with on every level. At C|Net, there is only so much input you can have on a website that employs 30 people especially when there are dedicated product and “vision” people, whereas with Github, it’s just me Tom and P.J.

If the site is slow, confusing, sucks, or doesn’t improve your workflow, I can take responsibility for that. But if it’s fast, simple, awesome, and life changing, I get to take responsibility for that, too. Which is very rewarding.

It also means, when you’re cofounder of a mostly technical company, the discussions are a lot more logical. I saw decisions made at C|Net that directly contradicted a/b statistics we had gathered. At Logical Awesome, that would never happen. It is neither logical nor awesome. Basically: I wanted to find the ideal company so I found good people and started it.

AkitaOnRails: Ok, so we come to the main dish of the day: Github! I can’t think of a better example of an innovative product that leverages so well the power of Git. I am very curious to understand how did you come up with this idea? At the same time, lots of people still can’t understand why Railers as a whole started using Git so massively all of a sudden. Mercurial, Bazaar guys are very uncomfortable. What do you think Git has that attracted Railers in flocks like that? Or at least what do you think Git has that no other have? I started evangelizing Git here in Brazil last year after listening Randall Schwartz, and recently someone asked me what would be my pick for a Rails killer-app and Github popped out of my mind, right away. So you do have good mind share as well.

Chris Wanstrath: Github is super exciting to me because it’s really a website just for me. Tom and I are pretty similar in that we both have a fair amount of open source projects, all of which get used and patched by people we’ve never met.

Handling this the old fashion way was fine until me and P.J. started working on FamSpam… as my free time decreased, so did my ability to maintain my open source projects. And really, that’s a definite workflow problem. Sometimes when you can’t find enough time to do something, you just need to make the something take less time.

Github was literally written for chronic, god, ambition, will_paginate, and the rest of our projects. We wanted to make it dead simple for people to contribute, and really, it’s just not possible without Git.

I had started using Git right after viewing the Linus tech talk in May 2007, which is really good at explaining Git from a very high conceptual level. So when this time problem came up, I had already been using Git for a few months and Tom had serendipitously been trying to solve the same problem.

We talked after a local San Francisco Ruby meetup, at a bar of course (where all great ideas and partnerships begin), and decided to work on the problem together. I’d been playing with an err-specific Gitweb, but Github is obviously much more ambitious in its scope.

As far as the adoption of Git, I have no idea what single thing tipped it. When we began working on the app in October, we were worried about ./script/plugin and tarballs and windows and svn mirroring. we weren’t even sure we could build a business around Git, because subversion was still so utterly dominant.

As we all know, however, Git tipped at the start of 2008 and everyone somehow found the time to try it out. It maybe takes, like, two seconds to realize how much faster it is, and then a few more seconds after that to realize how much more awesome it is. So it’s not much of a surprise that it is grabbing converts.

As for the Rails community, it’s definitely because of the projects. When Merb, Rubinius, and Ruby on Rails all switch to Git, you have no choice. This is where the community is headed. As for other languages, we have a large number of non-ruby projects on Github. PHP, Java, Javascript, Lisp, Python — there are a handful of Django forks, for instance, the io and nu programming languages are also hosted on Github.

So really what it comes down to is early adopters, I think. Nu is a lisp written on Objective-C, very new and very cutting edge. The people using it right now are people who realize how powerful and awesome it is to run such a dynamic language on such a mature and stable platform. And naturally, the forward thinkers behind nu know that Git (and distributed source control) is an equally powerful, equally cutting edge concept.

With Prototype and Scripatculous moved over, and rumors of other Javascript frameworks switching, it’s only a matter of time before Git becomes even more widespread.

I think the Railers just like to blog a lot.

As far as other solutions go, it’s Rails vs Django all over again. If you use Mercurial, that’s fine. The two are so similar (yes, Git does work great on Windows) that as long as people move to distributed source control, it’s still a win.

Oh, but wait, Mercurial doesn’t have Github :-)

AkitaOnRails: There you go :-) And I wonder what were the challenges of putting something as Github together. It is not a simple Rails web app, with very fast, pure Ruby code. You’re probably dealing with thousands of system calls to Git command liners, several background jobs, maintenance, security. What do you think were the biggest challenges while assembling Github before recently releasing it to the public?

Chris Wanstrath: Well you pretty much nailed it right there… scaling Rails is the easy part. scaling sshd, Git, Git-daemon, and our background jobs is the hard part.

We call Git directly many times per page, we need to process jobs that get entered from many different places (create a repo on the website, push a repo through ssh), we need to be secure (ssl and ssh for all private repos at all times), we need to make sure it all runs fast (memcached for both Git calls and the db), we need to make sure you’re “in the loop” with a news feed aggregating information from all different sources within our system (almost 1m rows in the feed table when we launched).

So it definitely has not been easy. Not to mention that ssl is essentially the “slow flag” for http, so we need to make sure we’re going fast because things like your dashboard already have a speed handicap. As an example, sshd stores its keys in an authorized_keys file. Well, we weren’t even done with our beta yet and the keys file was over 4 megs of just plain text. So we had one of the C gurus at Engine Yard patch sshd for us to do mySQL based lookups, giving us faster searches and no pricey appending / writing of a massive file

Matthew Palmer is the said guru, and we plan to make that code open source in the near future.

We’ve also got post-receive (when you “Git push”) hooks running that we host and deliver – so if you push, we’ll POST to an arbitrary url for you with json of the commits. But you can also give us some info and we’ll post to campfire, irc, twitter, or lighthouse for you with custom payloads.

Since we have flash, flash with haxe, Python, c, ruby, bash scripts, lots of Javascript, and some forthcoming erlang under the hood, it’s definitely an interesting mix.

We are fortunate to be hosted on Engine Yard , though. If we didn’t have them we’d be screwed. Their awesome cluster setup with GFS means we can host our potentially 100s of gigs of repositories on a shared, raid’d drive with redundancy and backups. If we were running Github on a barebones VPS, I dont even know how we’d share access to those repos. It wouldn’t be fun.

They also have a lot of experts available 24/7, which is great because I have a tendency to forget to sleep and work until 7am. A lot of the pain of running a heavily unix-dependent site is taken away by those guys.

AkitaOnRails: Interesting, Erlang? I’d like to know what you are up to with it. Going back, I think I saw a Rails trunk Git clone at Github way before DHH announced it a few days ago. I think it was Michael Koziarski who first did it, without much fuzz. Were you planning this together with the 37signals guys, or the switch of the Rails Core to Github was something that happened naturally? And do you have any kind of partnership with the Lighthouse guys as well, because you already have some hooks to manage Lighthouse tickets through Github commit messages, right?

Chris Wanstrath: The Erlang stuff will be very cool, I promise. We’ll make a big fuss about it when it’s ready. As far as the Rails thing goes, koz did indeed have a mirror of his Git-svn repo at Github during the beta. It was unofficial. Then in february he wrote a post explaining that the core was thinking about moving to Git.

When I saw the post I emailed him and we talked a bit about what it would take to move Rails to Github, which was obviously something I love on a huge number of levels. He said he would get back to me, and then a few weeks later the Core team contacted us and wanted to discuss the move. The rest is history.

I really respect the work the Lighthouse team does. The bug tracker is great, and the new redesign is even better, but we don’t have any official partnership with them. We wrote the post-receive lighthouse hook because we wanted it, Rails wanted it, and a huge number of our users wanted it.

AkitaOnRails: Many others started moving to Github during the Beta. I think Merb was more obvious because of Ezra’s being involved in Engine Yard. Dr. Nic started using it for the Textmate bundles. Can you point to any other high profile Rails related projects hosted at Github right now?

Chris Wanstrath: Merb was actually the reason Github went into beta. We launched the beta so they could start using the site for their 0.9 rewrite of merb-core. As far as popular projects, the best thing to do is check http://Github.com/popular/watched and http://Github.com/popular/forked. Datamapper, Rspec, and Mephisto are some of the popular ones.

AkitaOnRails: Are you using vanilla Git code, or did you customize it in any way to better fit your environment? Have you ever talked to Junio Hamano or any other Git core maintainer? With Github probably eating up a lot of your time, how is Chow and FamSpam going right now? How are you managing all those products all at once?

Chris Wanstrath: We’ve patched Git-daemon to record statistics, which we’ll soon be surfacing on the website soon in the form of better, more granular activity. You’ll be able to see how many times your project was cloned, popular projects in the last 24 hours, all time most cloned projects, that sort of thing.

We’ve emailed the core Git guys about the site, just to say thanks for writing Git and to let them know about Github, but haven’t really had an occasion to chat with them about anything in particular. If I ever run into any of them, I’ll definitely be buying them a few drinks, though.

I don’t actually work at Chow anymore. My last day at C|NET was almost exactly one year ago, so my Ruby time is spent on FamSpam, Github, open source, and doing occasional client work.

FamSpam is going well. Now that Github is launched, there are a handful of features we want to get in and all. Luckily, however, the site is very polished and has been running smoothly without the need for much intervention. What we’d really love to do, because it’s really just a great Rails-based mailing list, is try out some different concepts.

Maybe one for open source, or a version for little league teams – anyone who needs to keep in touch and have their email workflow and document sharing streamlined. For right now, however, I try to spend as much time as possible on Github. There are so many places we want to take it.

AkitaOnRails: I probably more than exceeded my interview time (sorry about that), but this conversation is super-interesting. One project I just remembered is Sake. To me Sake is very neat because many people complain about “how difficult and complex” the Git command line is, and Sake-like solutions make the workflow a lot easier. For instance, my entire Git workflow for Rails projects is only 4 small sake tasks like Git:update and Git:push. If I am not mistaken someone at Err did Sake, right? Are you still evolving it?

Chris Wanstrath: Yeah, I wrote Sake. It’s not a Rails plugin but it was written to help in Rails development. The “Git tasks” for Sake are becoming quite popular, as Git really just facilitates a better workflow. If you want to implement it, or different branches of that workflow, you can pretty easily just wrap up a few commands into a rake task or bash script.

And while bash scripts are great, sake tasks are portable. A lot of sake was inspired, very loosely, on Git – you can pull in sake tasks from any text file, whether it’s on the web or local, and sake comes with its own daemon for serving your tasks.

So you could share tasks over wifi, or something, the same way you could share Git repositories over an adhoc network. Sake is up on Github, so feel free to add features you feel should exist.

AkitaOnRails: Another thing I would like to hear your opinion about. I recently made a presentation on Rails Deployment strategies for beginners. The most common simple architecture revolves around Apache/Nginx/Litespeed + Mongrel/Evented Mongrel/Thin/Ebb. It is usually a matter of load balancing between a Web server using some kind of IPC to distribute the load back to Ruby VMs running your Rails app.

Now we have mod_Rails. What’s your opinion on having all those choices, do you have any particular recipe you like or it depends on each application’s needs? Did you have time to test mod_Rails? I think Hongli Lai is doing great stuff in regards to things like mongrel_light_cluster and his efforts on making Ruby’s GC more copy-on-write friendly, did you take a look at that?

Chris Wanstrath: The nice thing about Engine Yard is they get to take care of all that stuff for me. So while I used to spend a lot of time benchmarking and playing with different solutions, like Mongrel vs Emongrel, I just let the experts take care of it so I can focus on Github..

So no, I haven’t played with mod_Rails yet because I’m not in the situation where I could deploy it even if I wanted to use it. Which is in my opinion the best situation to be in, and pretty much the reason mod_Rails was written.

AkitaOnRails: And by the way, are you going to make any presentation at RailsConf this year?

Chris Wanstrath: I am. I’ll be talking about “Beyond cap deploy”, which will pretty much be an in-depth look at the Github architecture, and then I’ll be on a panel with Ben Curtis, Geoff Grosenbach, P.J., and Tom about being a “profitable programmer” w/ side projects.

AkitaOnRails: I am very pleased to have the opportunity to talk to you.

Chris Wanstrath: Yes, you too. Thanks!

Think of services like Basecamp, for managing projects without the company spending on maintenance, or your favorite webmail. These are online services where you pay so you never have to think about infrastructure. A form of service outsourcing.

This week Google caused a small stir by launching its answer to Cloud Computing: Google App Engine. TechCrunch published a review here. But what is Cloud Computing? First things first, let’s go over the terms the market uses.

We developers will, at some point, want to put our application into production. That’s when the hairiest part of the process arrives: deployment!

There are several options. First, software:

• Buy a commercial application. Commodity cases like Exchange Server: you pick, buy, install, and start using it. Bigger applications sell under licensing schemes, per user or per processor. Some software can be extended and customized, but usually it’s you who has to adapt to the software.

• Get an open source solution. Download it from the internet, configure it, and use it, or hire a services company to do that work for you. The obvious advantage is zero license cost. The downside is that running that kind of setup has a cost nobody usually accounts for: TCO is not trivial. How easy the application is to modify varies case by case.

• Build the system in-house. Code the system yourself, tailored to your needs, with your own team or a hired software house. The bespoke model. It tends to cost more, but it may be necessary when no commercial software does what you need. In theory it should be the software that fits the company best; in practice the TCO is usually high and the outcome uncertain. Everything depends on the maturity of the company and of whoever was hired to build it. Your mileage will vary.

• Use an ASP. You ‘rent’ a market application, like SalesForce.com, without worrying about deployment, installation, or updates. It’s less customizable, but the total cost of ownership is usually far lower. In many cases it makes sense: all you need is a stable internet connection. The services model is a trend because today’s internet infrastructure allows it. And there’s a clear win for the developer: one fix in the central system instantly benefits the whole customer base, with no per-client installs.

As for hardware options:

• Host it yourself. Buy machines, build networks, manage everything inside your own physical perimeter. As the company grows, you buy more machines. It works best for intranets and for companies with a mature IT department. IT is normally treated as a cost, like paying taxes; running everything in-house only makes sense if your core business is those very systems.

• Host it yourself, but understanding that growth isn’t linear. Sometimes you need more resources only for a short window, and buying machines all the time just inflates your TCO. That’s what ‘on-demand’ servers are for, like some IBM lines: when you need more power, you call IBM and they unlock more processors on the machine sitting physically at your site. A limited solution.

• Co-location. The machines are yours, but they live on a third party’s infrastructure, in a data center outside your company’s perimeter. It’s a cost-benefit trade: you rarely have the expertise to keep infrastructure as stable as a third party’s, but the machines remain your responsibility. It fits companies that have the money for machines but no physical space, or branches that all need to reach the same system. The data center keeps everything running around the clock, with a stable network and 24/7 support.

• Shared hosting. You rent a piece of a machine shared with other clients. The entry cost is minimal, but you suffer when another client on the same box demands more resources, plus issues like shared system libraries. It’s a can of worms that providers keep trying to fix, but the nature of this kind of package prevents anything bulletproof. Growing in such an environment is limited. It favors freelancers and small companies that need a website online but lack the money for beefier setups.

• VPS (Virtual Private Server). You still share the machine with other clients, but you get the feel of an entire machine of your own, virtual and isolated. A ‘shared co-location’: the machine is rented in slices, using solutions like Xen or VMware, and it comes out cheaper than co-location. The isolation shields your application from the neighbors’ updates. It’s the second step for anyone who wants low cost with more flexibility. The problem remains when you need extra slices only for a short period: spinning one up takes time, and you can miss your time-to-market.

Cloud Computing

Terms settled, on to the trendy one. I don’t know if it was first, but the one making the most noise is certainly the Amazon Web Services model: Elastic Compute Cloud (EC2), Simple Storage Service (S3), and SimpleDB.

The main draw is the elastic on-demand concept. You pay for what you use, without manually reconfiguring everything at each change. Use more, pay more; scale back down, pay less. It’s payment for resource consumption, instead of a fixed monthly fee for fixed resources.

Amazon’s products cover three services:

• Storage (Amazon S3). In a traditional model you’d buy or rent a file server. S3 created a different kind of file system, based on the concept of ‘buckets’. The advantages are price and stability; the downside is a model that doesn’t map directly onto a regular file system. By now there are dozens of libraries in different languages that ease the transition.

• Database (Amazon SimpleDB). Instead of a relational Oracle or MySQL, Amazon offers an unstructured, document-based database. With no fixed schemas, the costly administration over your data domain disappears, and maintenance drops. On the other hand, there are still hardly any frameworks or libraries that migrate a relational (SQL) application to SimpleDB’s model trivially. Once again, the advantage may be price.

• VPS (Amazon EC2). Dynamic, elastic VPS instances. If your system demands more resources during a spike in visits and transactions, you spin up more slices dynamically and kill them when you no longer need them. Five slices in the morning, twenty in the afternoon, five again at night. The catch: they won’t necessarily be the same five slices. The model guarantees slices, but not which ones. Your application must store its data outside EC2, in S3 or SimpleDB, so another slice can take over without relying on local data. It’s Shared-Nothing taken to its ultimate conclusion.

What’s the advantage? You get what you need, when you need it. Instead of committing to expensive co-location or VPS plans, you leave the application in Amazon’s care and gain dynamism without worrying whether your machines ‘can take it’. It’s like paying for the gym by the hours you actually attended, instead of a fixed monthly fee. Quit mid-month, and you don’t lose the whole fee. It’s a trend in the services market.

What’s the disadvantage? With Amazon, you have to prepare your application around their limitations and requirements. Especially because having your slice killed is normal: local data is gone if it wasn’t persisted elsewhere first. Taken as a premise from day one, that stops being a problem. It’s a different paradigm from co-location or classic VPS, where local data is permanent and backups are the host’s job.

Google App Engine

Which brings us to the week’s news. In practice, App Engine maps Amazon’s services one to one:

• Amazon SimpleDB vs Google BigTable: neither is a relational database; forget the classic RDBMS.

• Amazon S3 vs Google GFS: again, no classic file system like the one on your machine.

• Amazon EC2 vs App Engine: similar, but different, as we’ll see.

Google’s marketing sold the product as “your application running inside our infrastructure.” Everyone knows that Google Search, Reader, Gmail, and Orkut run on a proprietary, highly scalable infrastructure, the one that made Google famous and is now ‘open’ (gratis, not libre) for others to enjoy. Let’s look at advantages and disadvantages:

• The downside on both sides is the paradigm break: applications must be born exclusive to one of these environments. Choosing one means locking yourself in, because supporting both costs more development.

• Amazon’s advantage is service independence: want just storage? Use S3 alone. The disadvantage is that their services demand more integration knowledge than anyone else’s.

• Google’s disadvantage is the all-or-nothing proposition: your whole application lives in there, not just pieces. The advantage is the online administration environment, which puts Google a step ahead of Amazon in usability.

Right now Google only supports applications written in Python. It ships with Django pre-available and with APIs that make handling data and storage easier. Certainly great news for the Python community.

That caused some commotion among developers of other languages, which is normal. Coming from anyone else, it would have passed unnoticed as “just another hosting provider.” Coming from Google, they know massive praise and massive criticism always follow. Praise is democratic, and so is criticism. If they didn’t see this coming, someone in Public Relations needs to be fired.

Either way, don’t pay attention to yet another round of language wars:

• they’re a flash in the pan: only fun right now

• they don’t amount to much: if Google anticipated this and has good PR, they walk away unscathed

• Google likes Python: that’s public knowledge, and there’s nothing wrong with it. Companies decide what they want, when they want, and how they want. The day you have your own company, you’ll understand this.

• and Google never said App Engine would be Python-exclusive

In the official announcement, Google presents the service as a “preview release”, the famous Beta of every self-respecting Web 2.0 application. Big news :-) The promise is to support other languages in the future. It’s not a perfect solution, but like everything coming out of a corporation the size of Google, it has huge potential. And like every Web 2.0 application, there’s a free account so you can test-drive the environment before deciding on a plan upgrade. At this Beta stage, though, it isn’t open to everyone yet.

What Does This Represent?

Once App Engine matures and starts supporting more development environments beyond Python, it can become the main threat to Amazon’s Web Services: it’s a blatant copy of their services, and Amazon is the obvious commercial competitor.

It can also threaten smaller services, like Joyent Accelerator, a hosting provider built on the same elastic-resources concept, or the newcomer Heroku, which at its core is an App Engine for Rails: a bridge over Amazon’s services, configured in a simpler way for Ruby on Rails developers. Granted, it’s not ready yet and is nowhere near Google’s scale.

Neither Google’s nor Amazon’s services are perfect. Which is an obvious thing to say: no service is. Everything depends on your requirements and on how you want to commit. The rule stays the same: the more commitment, the cheaper in the short term and the pricier in the long run; the less commitment, the pricier now and, maybe, the better later.

In short: want something cheap and flexible? Amazon and Google can help. But if one day you grow and want independence, detaching can be hard, because your application got intimately tied to their services. With a more traditional service, like co-location or VPS, you pay more now, but you stay isolated from third parties and a future migration to other hosts comes cheaper.

If App Engine succeeds, the likely effect is pressure on the hosting market to offer more creative, cheaper, higher-quality services. That’s market law: competition benefits us, the consumers. Let them fight it out while we watch. In the short term, for anyone wanting to put an application in the air, it’s just one more variable to consider.

What’s Google’s goal here? I’m not sure it’s the same as the hosting companies’: making money by outsourcing infrastructure in a commodity market. Google has specialized in turning good products into commodities, which is both good and bad, depending on your point of view. If it’s a free service, one of the goals is to help more and more websites appear: potential customers for Google’s real core business, AdSense.

Of course I wouldn’t host my Twitter-equivalent on a free service. Actually, if your data matters to you, never use free services. They’re great while they work; when they fail and you take a loss, the problem is uniquely and exclusively yours. It’s like driving a car without insurance.

(off-topic within the off-topic: I use Gmail, but every email that lands in my inbox gets automatically forwarded to my paid Apple .Mac account, every .Mac email lands on my machine, and my machine has two redundant backups. Just to illustrate. I like Gmail, but I wouldn’t bet even my emails on it.)

Understanding Google

When analyzing a Google move, never think ‘altruism’ or any such nonsense. They’re a publicly traded company whose main objective is keeping shareholders happy. Again: their business is profit, and charity pays no dividends.

Their core business happens to involve reaching people who get happy hearing stock phrases like “Don’t be evil.” Great for us, sure, but that also generates a conflict of interests.

(off-topic within the off-topic, no naivety: not doing evil is too vague, because ‘good’ and ’evil’ are relative concepts. What’s good for me can be bad for someone else, and vice versa; good and evil shift with place and time. Nobody will ever say ‘do evil’, obviously; that wouldn’t be very smart. ‘Doing good’, or at least what the majority currently perceives as ‘good’, is a very intelligent way to generate positive publicity with consumer opinion. Besides being cheap, that kind of publicity can come with tax exemptions, which for big companies is considerable savings. Brazil’s Lei Rouanet is an example: both sides win, but without laws like this there’d be little motivation for a company to ‘invest’ outside its core business. Bah, what do I know. And of course, I like Google too; so far it’s in my interest.)

To understand it better, read this. An interesting part:

“Evil,” says Google CEO Eric Schmidt, “is what Sergey says is evil.”

Wired: “moral compromise is just the cost of doing big business.”

App Engine is one more channel for AdSense, just like Android and Gmail. One reason a company like Mozilla exists is the revenue line from Google for keeping Google search as the default in its toolbar. Apple also gets paid to have Google’s search field in Safari. It doesn’t quite count as a ‘bundle deal’, since both browsers are free, but the cost is indirect: AdSense advertisers want to appear in most browsers. Yahoo! also announced a test to show AdSense ads on its own search results, which could increase Google’s market share even further.

That means indirect visits, which turn into the billions of revenue that keep Google’s shareholders happy. And geek users, who contribute the free word-of-mouth advertising, obviously love the idea that supporting Google or Firefox means ‘Microsoft losing’. I may be being pessimistic, but it happened with Rockefeller, a great philanthropist, and his Standard Oil. It happened with Bill Gates, another great philanthropist, and his Microsoft. It can happen with Google. To get an idea, read this article and an interesting excerpt:

Blogger and Mashable reader Constantine von Hoffman at Collateral Damage probably put it best when he said: “At this point, it would take a mashup of Wittgenstein, Quantum mechanics and LSD to make sense of Google’s various explanations for what it will and won’t censor and why.”

Anyway, Google App Engine by itself represents no revolution yet: it’s just one more player in the hosting market. Cloud Computing is one more marketing term for a solution derived from others that already existed: finer price granularity for services. Just as it did with Android, Google is commoditizing one more market segment.

The same old rule still applies: cost-benefit analysis. And that kind of analysis doesn’t come as a recipe, nor with the click of a button.

As I always say – and Avi is good at pointing out —, Ruby has its drawbacks – most of them being improved on Ruby 1.9, JRuby and Rubinius. Avi gives us good reasons why Smalltalk is yet another great platform to learn, bringing back decades of evolution and maturity. So, here goes, the unabridged version of the interview.

And stay tuned! I hope to have Evan Phoenix, Hal Fulton, Peter Cooper and Adrian Holovaty as my next guests. Lots of material to begin 2008 in great style.

AkitaOnRails: Some people still try to make the case in favor of Smalltalk as it being the “purest OO language out there”. Even this being a fact, do you think that this alone is compelling enough to make a case against every other language, like Ruby which is kind of a multi-paradigm language? I hope you’re not pesky about ‘language pureness’ :-)

Avi Bryant: Actually, I am pesky about it – not from an academic or aesthetic point of view, but from a pragmatic one. “Pure” languages make a lot of things easier: development tools, VMs, and infrastructure like distributed object systems or transparent persistence all get much simpler and better when you have a small, consistent language.

One experience I had, for example, was being hired to port my client library for the GOODS object database from Smalltalk to Python; I spent a huge amount of time dealing with special cases in the Python semantics (is this a class? a type? a new-style class? implemented in python or in C?) that simply never came up in Smalltalk.

Ruby does pretty well here, better than most in the scripting language space – I wouldn’t call Ruby “multi-paradigm”, by the way – but it could stand to do better.

AkitaOnRails: I think you once said that you only consider a language “finished” when it is fast enough to extend itself. This is very true for Smalltalk, Java and other platforms. Ruby is undoubtedly lacking in its current position. That was one of the reasons we now have parallel efforts like John Lam’s IronRuby, Charles and Thomas’ JRuby and Evan’s Rubinius. One of the most criticized points is that Ruby doesn’t have a formal spec for the language or its core libraries. You also said that the Java VM is not suited enough for Ruby as Strongtalk would be, for example. Does this still hold today as we see JRuby 1.1 on the horizon?

Avi Bryant: It still holds. JRuby currently seems to benchmark on par with MRI, and the best numbers from any Ruby implementation are maybe 3X MRI on average. I think we can do 25X MRI for basic language feature benchmarks (message sends, block evaluation, etc), which should be enough to make Ruby implementations of the standard libraries feasible. Even if the net result was a similarly performing system, the side benefits would be extremely valuable.

AkitaOnRails: I remember that you started the Smalltalk.rb initiative a while ago, and if I am not mistaken it was an effort to make Ruby translate into equivalent Smalltalk code so it could run in any of the available Smalltalk VM implementations out there. Is this project still going on, do you still think it is feasible to pursue such a goal?

Avi Bryant: I still think it’s a worthwhile and realistic goal. The project is still going in the sense that I’m still talking to people about how to make this happen, but I’m not able to spend time writing code for it myself at the moment. I have some reason to hope this project will progress more concretely next year, but we’ll see.

AkitaOnRails: Some time ago I read a snippet of an IRC transcript between you, Evan, Chad and even Charles. You started talking about standardizing primitives (they are C-based in MRI today). My bet is that Rubinius is the project destined to be the “next big thing” for the Ruby community. To me it seems that you and Evan collaborate a lot, is that correct?

Avi Bryant: I wouldn’t say that, but Rubinius is certainly the Ruby project that most closely aligns with my own interests and goals at the moment, and it’s great to see how much effort is being put into it. Engine Yard also deserves a lot of kudos for their financial support of the project.

I have high hopes that the work Evan and crew are doing will be of great benefit not just to Rubinius but to all of the Ruby implementations – which is why I was advocating for a standard primitive set.

AkitaOnRails: You know a lot of people, I wonder if you ever talked with Koichi Sasada himself to input some of your ideas for the YARV project as it’s going to be the “official” Ruby virtual machine out there a few months from now.

Avi Bryant: No, I don’t believe we’ve ever spoken. I would certainly love to meet him some day.

AkitaOnRails: Then, we get one level up to applications. Your Seaside framework is pretty amazing. You’re very right about WebObjects as well. As a Java programmer I always pitied that it never got mainstream. Today it’s mostly relegated to Apple related websites, while lesser Java frameworks took the front. Tapestry and Cayenne are trying to top WebObjects’ features but seems they are nowhere near yet. Do you still program some Java? What is your opinion on current mainstream frameworks as Spring, JSF? For non-starters, what was it in WebObjects that makes it different than the rest?

Avi Bryant: I don’t do any work in Java any more (thankfully), so I can’t really give a qualified opinion on the state of frameworks there right now. However, the main thing I think WebObjects did right was to focus on the state and behavior of the application, rather than on the mechanics of URLs and HTML and HTTP requests.

So rather than worry about what a field was named, you would just say “this field is bound to this instance variable in my model”, and rather than worry about what URL a link went to, you’d say “this link triggers this method on my page object”.

Transitioning between pages was done by constructing a new page object and setting it up directly in Java, rather than constructing a URL that was going to be parsed to build a page. That directness and general style is something that very heavily influenced Seaside.

AkitaOnRails: Many people may not immediately understand it as a word that is new to most of them: “continuations”. Meaning that instead of manually controlling the flow between events, with manual marshaling and unmarshaling of objects, you program as a cohesive and continuing flow. Can you explain this in a way that a novice would understand? What are the benefits and drawbacks of this approach?

Avi Bryant: Well, it’s really all about being able to get modal behavior in a web app. In a desktop application, for example, you might occasionally have a call like “file = chooseFile()” or “color = pickColor()” that opens a modal dialog, gets a result from the user, and keeps going from there.

Seaside’s use of continuations is just to allow exactly that kind of interaction in the context of the web. The tricky bit is that because of the backbutton, the user might go pick a different color from the same dialog later, and the “pickColor()” call has to return a second time. This is confusing to think about, but mostly it Just Works.

From an implementation point of view, what we’re doing is saving a copy of the stack at the point the call is made so that we can come back to it later, more than once if need be.

AkitaOnRails: You started these concepts in the Ruby world with IOWA. Once upon a time there was a framework called Borges that was also discontinued. Then Koichi mentioned taking continuation support from the next Ruby. Do you think the Ruby world is taking a step back with lack of support for continuations?

Avi Bryant: Continuations are a great abstraction, and allow a lot of interesting experimentation with language semantics. However, I wouldn’t consider it a tragedy if they disappeared from Ruby; there are more important things to worry about. From a web perspective, I think that AJAX is starting to replace a lot of the use cases where having server-side modal logic used to be important – Dabble DB, for example, barely uses Seaside’s continuation features at all.

AkitaOnRails: Your blog has a peculiar name, HREF considered harmful. You probably got your inspiration from Edsger Dijkstra/Niklaus Wirth’s Goto Statement Considered Harmful paper where Edsger made his famous case against GOTO. At the assembler level it is impossible to not JMP all the time (at least not without some macro help) but at a C level you can avoid GOTO’s altogether. Even Basic had both GOTO and GOSUB and you could just ignore the former. I am being simplistic here, but now you’re kind of saying that URLs play a similar role in a way that they disrupt the normal flow, going back in unpredictable ways and so on. On the other hand the Internet would not be what it is without URLs. How do we deal with this conundrum?

Avi Bryant: I don’t think it’s a conundrum. I think your analogy is a good one: it’s impossible to not JMP all the time, and it’s also impossible not to use URLs all the time, but we don’t have to always be aware of either one. Back when everyone wrote in assembler, people probably obsessed about having meaningful or “pretty” JMP labels the same way they obsess about having pretty URLs today. I think we need to allow ourselves to make the same leap up in abstraction level on the web that we did with higher level programming languages.

That’s not to say we should never be thinking about URLs: any website needs to export meaningful URLs to be referenced and used externally, just like any C library needs to export meaningfully labelled function names so that it can be linked into other programs. But that’s as far as it needs to go.

AkitaOnRails: Components were a big deal in the early 90’s, with RAD tools and so forth. Today it seems that they are showing up slowly, with initiatives as JSF and ASP.NET. Rails initially packed a very crude support for ‘components’ but it was dropped very early on as a bad thing (because of the jury rigged way it was implemented). Nothing new came to replace it what feels like there wasn’t enough need for it at the Rails level. On the other hand Seaside is heavy in componentization. Can you expose a little bit about the differences in this approach?

Avi Bryant: “Component” is one of those terms like “MVC” or “open” that gets used to mean so many different things it’s hard to talk seriously about it. One concrete feature of Seaside’s approach is that each piece of UI is quite independent of what other pieces of UI may be on the page.

There’s no central controller which is responsible for request processing, choosing a template, or so on – all of those responsibilities are distributed among many smaller objects. There’s also no chance of naming conflicts because you don’t directly name links or form fields, so you can have many copies of the same form/widget/etc on the same page without any confusion.

This allows a level of modularity that’s very hard to reproduce in most traditional frameworks.

AkitaOnRails: Another thing that might shock some people is that there is no traditional templating system in Seaside, no code mangling between HTML tags all over the place. Instead all HTML is abstracted in a object hierarchy that renders itself. In the Rails world there are a few alternatives that attempt some of that, like HAML, Markaby. What do you think are the fundamental differences?

Avi Bryant: The HTML rendering API is a very core piece of Seaside, and a huge amount of the framework was designed with that underlying approach in mind. I think that’s the main difference when compared with HAML or Markaby, which are meant to be dropped into other frameworks.

Once you get rid of templates and switch to programmatic HTML generation, the design space just completely changes, and you need to re-evaluate almost every other decision that you made – this was a process that I went through with Seaside between the initial release, which used templates, and the 2.x versions, which don’t.

So I think we’ll have to see a framework that takes, say, Markaby as a starting point and asks “where can we go from here” before we can make any real comparisons.

with Adrian Holovaty, from Django fame

AkitaOnRails: You also touted the shared-nothing vs share-all dilemma. And you obviously have a strong case because this is not just techno-babble but you do have a successful product running on top of that vision. Do you still think it pays off to sacrifice a little bit of performance for more clean code and productivity? In a way it’s sort of what DHH himself said about Rails when people said it was slower than PHP or Perl. Actually this also has to do with your talk about being a “heretic” in a way that you’re doing things in a different way than we all are now. Can you elaborate on that?

Avi Bryant: “Share nothing” is great advice for making the maximum use of your server resources. For sites like yahoo.com or Facebook, that’s clearly very important because your business depends on serving millions of users every day.

“Share everything” is really hard on your servers, but it makes maximum use of your developer resources. For a startup like Dabble DB, where we have 4 developers, that’s really important. And since our revenue model only requires serving thousands of users every day, not millions, it’s ok to spend a little more per user on server hardware. It’s been a very worthwhile tradeoff for us.

AkitaOnRails: And speaking of DHH, you’ve probably met on several occasions. You’re both opinionated guys with antagonist visions. Smalltalk vs Ruby. Seaside vs Rails. Share-all vs Share-nothing. One could feel tempted to draw you two as Superman vs Batman kind of dispute. Though I know that smart people don’t fight over pesky stuff. What do you think of him and his ideas, how do you think both communities could cooperate if there’s any chance for that?

Avi Bryant: I have a lot of respect for David. The only time I’ve felt antagonistic towards him was this last summer at Foo Camp. We were playing a game of Werewolf, and I was a werewolf and he was a villager. Now, as a werewolf the best thing you can do is get the person nearest you to trust you, and that happened to be DHH. And it worked – Kevin Rose was totally on to me but David managed to persuade him not to lynch me, and I ate them both and won the game. Pn3d!

Anyway. There’s probably a lesson in there somewhere. Also, in a fight between me and DHH, Neal Stephenson would win.

AkitaOnRails: Finally, Brazil still has a young and growing community. Any closing regards for our audience??

Avi Bryant: Olá and good luck! I’ve been told Rio is even more beautiful than my home of Vancouver, and I hope to visit some day.

Don’t miss the TWiT podcast interview with Avi!

I’m buried in work (it’s 2 AM on a Sunday), but this little thought caught my attention because it’s simple and, even so, most people still fall for it. The original post is Sunk Cost for Architects, but I think it should be Sunk Cost for Managers. Here’s the translation:

Let’s say you’ve bought a movie ticket but then realise from Rotten Tomatoes that the movie stinks. Assuming you can’t sucker one of your friends to buy the ticket, you have a couple choices:

1. Since you’ve paid for the ticket, you might as well suffer through the movie anyway.
2. Throw away the ticket and do something else.

Now what?

The sunk cost fallacy is not realising that in either case, you’ve already paid for the ticket and therefore the sunk cost is not relevant for rational decision making.

Case 1: You suffer the cost of the ticket + You suffer through the movie

Case 2: You only suffer the cost of the ticket

Therefore, rationally 2 is the better option.

So instead of a movie ticket, let’s say it’s an enterprise software license… or perhaps some custom hardware.

Should how much was paid for the license or hardware have any bearing whatsoever on what is decided to be the best future architectural approach?

Well, only if the management system you’re operating in punishes rational behaviour and rewards irrational behaviour.

Total common sense. Economists are more used to studying and cataloguing behaviors like this. I recommend reading about other concepts like the good old Opportunity Cost. Parkinson’s Law, to me, is the corollary of the gas expansion law, and every developer (myself included) has suffered from it.

Nobody’s right all the time, but the daily exercise of thinking with logic and skepticism, instead of running on dogmas and precepts, improves our odds a lot. Maybe then, whoever watches A Beautiful Mind will remember more of the genius behind the Nash Equilibrium than the melodrama.