Then it became an industry. Pinecone, Weaviate, Qdrant, Chroma, Milvus, pgvector, LangChain, LlamaIndex, Haystack. Tutorials everywhere, “build your chatbot with your PDFs,” entire consultancies feeding off this. It became the “hello world” of applied LLMs: document → chunk → embed → vector DB → query.

Today, in April 2026, Claude Opus 4.6 has 1 million tokens of context. Sonnet 4.6 too. Gemini 3.1 Pro too. GPT 5.4 has a smaller window but still in the comfortable range, in the hundreds of thousands. And some models already have experimental 2M token modes. The question that keeps nagging at me: what on earth do I need a vector stack for, to solve a problem that fits inside the model’s window?

And there’s more: vector databases have real problems nobody wants to talk about. False neighbors. Arbitrary chunking that splits a definition from its usage. Embeddings that age badly. Not to mention that when the result is wrong, you have absolutely no idea why.

The thesis I’ve been chewing on is simple: in most cases, a well-aimed grep plus a generous context window beats a full RAG stack. It’s cheaper, it’s easier to maintain, and when it breaks you can actually debug it. Let’s break this down.

What the Claude Code leak showed

Before we get into the theory, let me bring up something that happened a few days ago that backs this whole argument up. On March 31, 2026, Anthropic, by accident, published version 2.1.88 of the @anthropic-ai/claude-code package on npm with a nearly 60 MB source map attached, and roughly 512,000 lines of TypeScript from their internal tool leaked into the wild. I already wrote about the incident last week, with more detail on what showed up in the code.

The part that matters for this discussion is Claude Code’s memory system. Instead of dumping everything into a vector DB, the architecture has three layers. There’s a MEMORY.md that stays permanently loaded in context, but it doesn’t hold any actual data: it’s just an index of pointers, around 150 characters per line, kept under 200 lines and about 25 KB. The real facts live in “topic files” that get pulled on demand when the agent needs them. And the raw transcripts from previous sessions are never reloaded whole, only searched with grep, hunting for specific identifiers. No embedding. No Pinecone. Just write discipline (topic file first, index after) and lexical search. That’s it.

Claude Code’s main loop also has a tiered system for handling a context that’s filling up. As I detailed in the previous post, there are five different context compaction strategies, with names like microcompact (clears old tool results based on age), context collapse (summarizes long stretches of conversation), and autocompact (which fires when the context gets close to the limit). The CLAUDE.md file, which a lot of people thought was just a convention, is first-class in the architecture: the system re-reads it at every iteration of the query.

What this tells me: the best coding agent on the market right now, built by the company selling the most expensive model out there, does not use a vector DB. It uses files on disk, a markdown index, lexical search, and smart compaction strategies for when the context overflows. They could’ve slapped embeddings on top, they have the money to run whatever they wanted, and they chose not to. The reason, in my reading, is exactly what this post is arguing: to retrieve text from files you control, with generous context available, a vector DB is dead weight. Better to invest in compacting the window you already have than indexing everything into an external store.

There’s a curious security detail that came along with the leak: people noticed the compaction pipeline has a vulnerability they’re calling “context poisoning.” Content that looks like an instruction, coming from a file the model reads (say, a CLAUDE.md from a cloned repo), can end up being preserved by the compaction model as if it were “user feedback,” and the next model takes that as a real user instruction. It’s a new attack vector. But that’s a topic for another post.

The “Dream” system and memory consolidation

But what really caught my eye for the RAG debate, which I unpacked in detail last week, is the system called autoDream. It’s a forked subagent, with read-only bash access to the project, that runs in the background while you’re not using the tool. Its job is literally to dream: to consolidate memory. The name isn’t accidental, and the obvious analogy (which I couldn’t resist) is the human brain consolidating memory during sleep, turning short-term experience into something more stable.

For a dream to actually run, three gates have to open at once: 24 hours since the last dream, at least 5 sessions since the last dream, and a consolidation lock that prevents concurrent dreams. When it fires, it goes through four phases. Orient (does an ls on the memory directory, reads the index). Gather (looks for new signals in logs, stale memories, transcripts). Consolidate (writes or updates the topic files, converts relative dates into absolute ones, deletes facts that have been contradicted). And Prune, the final cleanup that keeps the index under 200 lines.

The decision to make autoDream a forked subagent is the detail that matters here. It does not run in the same loop as the main agent. Why? Because memory consolidation is a noisy process. The model has to re-read old transcripts, compare them against what’s in MEMORY.md, decide what stays and what goes, form hypotheses about things it saw in earlier sessions. If that ran in the main context, it would pollute the “train of thought” of the agent that’s trying to help you with your current task. By forking, you keep the two separate. The main agent stays focused on what you asked for, and autoDream does the housekeeping in parallel, with no write permission on the project.

And the way it figures out what needs to be consolidated is plain old lexical search. The transcripts live as JSONL files on disk, and autoDream uses grep to look for new signals. Just grep, on text logs. Stop and think about that for a second. The memory consolidation of the most advanced agent in the world, built by one of the richest AI companies out there, is a forked subagent running grep on text logs. If a vector DB were the right answer for this kind of problem, Anthropic would’ve put a vector DB in there. They didn’t.

And there’s a detail that, to me, is the buried gold of the entire leak, and it fits this argument like a glove. In autoDream, memory is treated as a hint. The system assumes that what’s stored may be stale, wrong, contradicted by something that happened later, and the model has to verify before it trusts it. The vector DB pitch is the opposite of that: index everything, search by similarity, return the top-k, trust the result. Claude Code went the conservative route. Index little, search by word, return a hint, and stay skeptical until you’ve laid eyes on the actual fact.

The whole strategy works in two layers. Inside a single session: generous context plus grep plus smart compaction (microcompact, context collapse, autocompact). Between sessions: a subagent that consolidates memory asynchronously, using grep on the transcripts and treating the result as a tip, not as truth. Embeddings and vector DBs don’t show up in either layer. The deliberate choice was a smart reader chewing on raw text, not a dumb reader being spoonfed the top-k of an embedding.

The practical lesson for our debate is simple. The most advanced agents on the market are heading toward generous context, lexical search, and smart compaction, not toward classic RAG pipelines. If Anthropic, with all the infrastructure and talent they’ve got, picked this path for Claude Code, those of us building internal applications on a fraction of that budget should at least think about going the same way.

Where the story started turning

When the ceiling was 32k of context, retrieval was the bottleneck of the entire problem. You had to pre-filter aggressively, because anything that made it into the window was sacred space. A vector DB was the only halfway-decent way to do that semantic pre-filtering. The logic was: “the reader (LLM) is expensive and dumb, so the retriever has to be smart and selective.”

Today the equation has flipped. The reader is now the smartest one at the table, and the window grew big enough to hold an entire document. So the retriever can (and maybe should) go back to being dumb. The dumber, the better. You want high recall and low precision, and you let the model do the fine work. Grep does exactly that. So does BM25. And ripgrep flies through millions of lines without breaking a sweat.

And this isn’t just my hunch. The BEIR benchmarks have shown for a while now that BM25 matches or beats a lot of dense retrievers when the domain drifts away from where the embeddings were trained. Anthropic itself published a post on Contextual Retrieval that basically says the same thing: a lexical signal plus an LLM’s judgment beats pure embeddings on most knowledge tasks. And take a look at Claude Code, the tool I’ve been using every day for 500 hours: it navigates the repo with Glob and Grep. No vector DB, no embedding, no LangChain. It works ridiculously well.

The real problems with vector databases nobody advertises

The vector DB marketing sells the dream of perfect semantic search. Reality is messier.

False neighbors come first. Cosine similarity rewards topical similarity, not relevance. You ask “how do we handle authentication errors” and the DB returns every chunk that mentions authentication. The chunk that actually answers the question may be in tenth place, or may not have been retrieved at all because the doc author wrote “login” instead of “auth.”

Chunking is the second one, and it’s a disguised disaster. A 512-token window with a 64-token overlap sounds reasonable, until you realize your important table got cut in half, the function definition ended up separated from its usage, and the piece of documentation with the exact command got orphaned without the context of its section. The chunk boundary tends to land exactly where the answer was living.

When it fails, it fails without leaving a trace. When BM25 misses, you know why: the word isn’t there. When a vector DB returns garbage, you get a plausible-looking wrong chunk, with no diagnostic signal at all. Good luck debugging that in production at two in the morning.

The index gets stale. Every document update calls for re-embedding. If you have 10,000 docs and 200 of them change per day, that turns into a batch process, monitoring, a queue, retries, embedding API costs, and an unavoidable inconsistency window between what’s on disk and what’s in the index. Grep has none of that. File changed? The next query already sees it.

And there’s the operating cost nobody adds up. Pinecone charges per vector. Weaviate wants a cluster to maintain. pgvector saves you a new server but you still own a schema, an index, and a re-embedding pipeline. Each of those things wants engineer time, monitoring, tests, deploys. All of that to do a search that rg would often crack in 200ms.

Comparing the complexity

Look at the diagram:

On one side, eight steps, four or five services, an external index that needs to be maintained and kept up to date. On the other, four steps, zero new infrastructure. This isn’t a caricature: it is literally what you have to set up for each case.

The honest question: does the left column pay off? In 2023, yes, because the right column didn’t exist (no LLM had a 200k window). In 2026, in most cases, it doesn’t.

Pros and cons of each side

Classic RAG (vector DB)

For:

• Works for huge document bases, on the order of hundreds of GB, where even rg won’t cut it without prior indexing
• Handles heavy paraphrase and cross-lingual queries (“how do I cancel” vs. “subscription termination process”) where the user’s vocabulary doesn’t match the document’s
• Works for non-textual modalities (image, audio) where grep has nothing to look at
• Saves input tokens if you’re tight on budget or absolute latency

Against:

• Complex stack: embedding, vector DB, chunking, reranker, re-indexing pipeline
• Opaque failures, hard to debug
• Chunking destroys the context of tables, code, long definitions
• Operational overhead (index, queue, monitoring, re-embedding cost)
• The semantic search the marketing is selling rarely works the way the marketing promises

Grep + long context

For:

• Practically zero new infrastructure: ripgrep, sqlite, or a plain LIKE in Postgres
• Always fresh: file changes, the next query sees them
• Transparent failures: the word is either there or it isn’t
• Loads the document in generous chunks, the model does the fine filtering with actual semantics
• Cheaper in dev and ops, cheaper to pivot domains

Against:

• Doesn’t scale to terabytes of raw text without some kind of indexing
• Suffers when the user’s vocabulary is very different from the document’s
• Doesn’t work for non-textual modalities
• Per-query latency is higher in absolute terms (loading 100k tokens always costs more than loading 5k)
• Per-query input cost is higher if you don’t have prompt caching

But what about cost?

This is the argument I get hit with the most when I defend the “load everything into context” thesis. “It’ll get crazy expensive, 200k tokens of input per query is absurd.” Let’s actually run the numbers.

In yesterday’s LLM benchmark post I mapped out the per-token price of every model. Take Claude Sonnet 4.6: $3 per million input tokens, $15 per million output. Take GLM 5 (which I proved actually works): $0.60 input, $2.20 output. Take GPT 5.4 Pro at the top of the heap: $15 input, $180 output (yeah, that one stings, I know).

Before we turn “200k tokens” into dollars, let’s land that number on something tangible, because “100k tokens” doesn’t mean anything to anyone. A token, on average, is roughly 0.75 of a word in English (Portuguese is similar, maybe a touch heavier because of longer words). So, translating:

• 100k tokens ≈ 75,000 words ≈ a whole short novel like Hemingway’s The Old Man and the Sea with room to spare, or about three long Wikipedia articles glued together.
• 200k tokens ≈ 150,000 words ≈ a big novel, like Crime and Punishment in full, or half of the first Game of Thrones book (which clocks in around 298k words, so roughly 400k tokens).
• 400k tokens ≈ 300,000 words ≈ A Game of Thrones in full, the entire first book of the series in your window.
• 1M tokens ≈ 750,000 words ≈ the entire Lord of the Rings trilogy plus The Hobbit, or the whole Bible (King James is around 783k words, roughly 1M tokens), or about two and a half Game of Thrones books stacked on top of each other.

So when I say “throw 200k tokens of input at the model,” what that actually means in the real world is “throw the entire Crime and Punishment in as the context for your question.” That’s a lot. And that’s exactly what makes the argument of this post viable: today’s models can read an entire novel in one go and still answer a specific question about it. In 2023, this was science fiction. In 2026, it’s the base case.

So picture a query that throws 200k tokens of input at the model (there goes Crime and Punishment again) and produces 2k tokens of output (about three pages of response):

Now throw prompt caching into the mix. Claude has a cache that drops cached input to a fraction of the full price (in the ballpark of 10%, depending on the model). Gemini has a similar mechanism. When you fire a sequence of queries against the same 200k-token dump, the cost of subsequent queries plummets to pennies. With Sonnet cached, you can fairly call it about $0.10 per follow-up query without making things up.

Now compare that to the cost of running a Pinecone, or a Weaviate, or a pgvector. Setting aside the price of the subscription itself (which varies a lot), you need an engineer to wire up the pipeline, maintain it, monitor it, deal with embedding failures, redo the chunking when the domain shifts. Conservatively, you’re looking at somewhere between 40 and 80 hours of engineering to make the thing stable. At R$ 200/hour, that’s between R$ 8,000 and R$ 16,000. In USD, somewhere between $1,600 and $3,200 just to stand it up.

With $3,200, on Sonnet 4.6 with prompt caching, you can run something on the order of 30,000 queries of 200k tokens each. Thirty thousand queries, depending on the scale of the project, gives you several months or even an entire year of an average internal tool. And you didn’t pay an engineer to wire up a pipeline. There’s no vector DB server to maintain. And if the document changes, the system already sees it on the next query.

The “RAG is cheaper in tokens” argument ignores that tokens are the cheapest thing in the entire equation. Engineers cost a lot, servers cost a lot, bugs in production cost a whole lot more. Tokens have become a commodity, and they’re getting cheaper with every new model release.

The classic RAG argument was “the model is expensive, retrieval is cheap.” Today it’s the opposite: the model is the cheap part of the stack, smart retrieval is what costs a fortune to build and maintain.

Where the thesis doesn’t hold

I don’t want to come off as a fanboy. There are cases where classic RAG still wins:

1. Massive corpora. If you have 500 GB of raw text, even rg won’t solve it in acceptable time. You need some kind of indexing. It can be indexed BM25 (Tantivy, Elasticsearch), it can be a vector DB. But notice: the first option is still lexical, not vector.
2. Wildly scattered vocabulary. Customer support, where the user types “my wifi’s down” and the documentation says “loss of connectivity at the physical layer.” BM25 won’t catch that. Embedding will. Vector DB scores a point here.
3. Non-textual modalities. Image-by-image search, audio-by-audio. Embedding is mandatory.
4. Critical absolute latency. If you have to answer in 100ms with a 5k input budget, a generous dump won’t fit. Pre-filtering is necessary.
5. Compliance and audit. If you have to prove that a specific document was consulted to answer a specific query, having indexed and trackable chunks helps. A 200k-token context dump is more opaque from an audit standpoint.

For those cases, classic RAG still makes sense. But notice the size of the list. These are specific cases. The general case, things like “chat with our internal docs” or “ask the product manual,” almost all of it falls into the “grep + long context handles it better” bucket.

Lazy retrieval: the recipe I’d defend

If I were building a “chat with docs” tool today, from scratch, it would look more or less like this:

1. Keep the documents raw. Markdown, converted PDF, code, whatever. On disk, organized in folders that make sense for the domain.
2. Fast lexical filter. ripgrep with regex, or BM25 with Tantivy/SQLite FTS5, or a LIKE in Postgres if you already have one. Returns 100-300 hits.
3. Load generously. Grab not just the matching snippet, but the entire file, or a wide window around it. Throw all of it into the context.
4. Let the LLM do the fine work. Pass the original question, tell the model to find what matters, drop the rest, and answer with citations.
5. (Optional) Add embeddings only for the query classes where lexical fails, after you have real data showing that it fails.

This is the opposite of the old advice (“start with vectors, fall back to keyword”). It’s: start with keyword, and add vector only if you feel the gap. In most projects, you never will.

A toy implementation in Ruby

To make it concrete. Here’s a Ruby script using the ruby_llm gem (the same one from yesterday’s benchmark) that does exactly this flow: grep through the files, load the snippets with context, send to Claude, get the answer back. No vector DB, no chunking, no embedding, no LangChain.

#!/usr/bin/env ruby
require "ruby_llm"
require "open3"

DOCS_DIR = ARGV[0] || "./docs"
QUERY    = ARGV[1] or abort "uso: ./ask.rb <pasta> <pergunta>"

# 1. Fast lexical filter with ripgrep.
#    -i case insensitive, -l file names only, --type-add covers md/txt/extracted-pdf.
def lexical_search(dir, query)
  terms = query.downcase.scan(/\w{4,}/).uniq.first(8)  # words with 4+ letters
  pattern = terms.join("|")
  cmd = ["rg", "-l", "-i", "-e", pattern, dir]
  files, _ = Open3.capture2(*cmd)
  files.split("\n").reject(&:empty?)
end

# 2. Load entire files (up to a reasonable cap).
def load_context(files, max_chars: 600_000)
  total = 0
  files.map do |path|
    body = File.read(path)
    next if total + body.size > max_chars
    total += body.size
    "## #{path}\n\n#{body}\n"
  end.compact.join("\n---\n")
end

# 3. Send to Claude with the question and the documents.
def ask(query, context)
  chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4-6")
  prompt = <<~PROMPT
    Você tem acesso aos documentos abaixo. Responda a pergunta do usuário
    usando apenas o que está nos documentos. Cite o nome do arquivo nas
    referências. Se a resposta não estiver nos documentos, diga isso.

    --- DOCUMENTOS ---
    #{context}
    --- FIM DOS DOCUMENTOS ---

    Pergunta: #{query}
  PROMPT
  chat.ask(prompt).content
end

files = lexical_search(DOCS_DIR, QUERY)
abort "nenhum arquivo bateu" if files.empty?
puts "Encontrei #{files.size} arquivos. Carregando contexto..."
context = load_context(files)
puts ask(QUERY, context)

About 40 lines. No Pinecone dependency, no vector schema, no re-indexing pipeline. You run it as ./ask.rb ./docs "how do I configure the payment webhook" and that’s it.

That example is one-shot. You run it, it answers, done. For a real chat, with multiple questions in a row over the same documents, the design changes. Instead of running lexical_search upfront and shoving everything into the context at once, you expose the search as a tool to the model. Then it’s the agent that decides when it needs to pull more docs, what term to look for, which file is worth opening in full. That’s how Claude Code actually works: Glob, Grep and Read are tools, and the model picks the sequence. ruby_llm supports tool calling, so you can do the same thing in Ruby. It looks something like this:

require "ruby_llm"
require "open3"

DOCS_DIR = "./docs"

class SearchFiles < RubyLLM::Tool
  description "Procura arquivos cujo conteúdo casa com o padrão dado (regex). Retorna lista de paths."
  param :pattern, desc: "Padrão regex pra busca lexical (case-insensitive)"

  def execute(pattern:)
    out, _ = Open3.capture2("rg", "-l", "-i", "-e", pattern, DOCS_DIR)
    out.split("\n").reject(&:empty?)
  end
end

class ReadFile < RubyLLM::Tool
  description "Lê o conteúdo completo de um arquivo do projeto."
  param :path, desc: "Caminho relativo do arquivo"

  def execute(path:)
    File.read(path)
  rescue => e
    "erro: #{e.message}"
  end
end

chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4-6")
            .with_tools(SearchFiles, ReadFile)
            .with_instructions(<<~SYS)
              Você responde perguntas sobre os documentos em #{DOCS_DIR}.
              Use search_files pra encontrar arquivos relevantes e read_file
              pra ler o conteúdo. Sempre cite o arquivo na resposta.
            SYS

loop do
  print "> "
  msg = gets&.chomp
  break if msg.nil? || msg.empty?
  puts chat.ask(msg).content
end

The model gets the question, decides whether it needs to search, calls search_files, sees what came back, decides whether it needs to open any file, calls read_file, and only then answers. On the next question it already has the previous context in the session and can ask for more if it needs to. The context only receives what the model asked for, not the whole grep dump from the earlier example.

The same idea works for databases: swap rg for a SQL query with LIKE or tsvector (Postgres full-text), load the relevant rows, throw them in the context. If you have 10k records in an internal database, this handles it. If you have 10 million, you start needing smarter pagination or a more serious pre-filtering layer. But the mental model is the same: dumb filter + smart reader.

The point that matters

The most interesting thing in all of this isn’t even the Pinecone savings. It’s that the nature of the bottleneck has changed. In 2023, the bottleneck was retrieval: the reader was small, slow, expensive, and you needed a clever retriever to fill the window with the bare minimum. In 2026, the bottleneck is reasoning over messy context: the reader is big, relatively fast, and cheap. So it makes more sense to have a dumb retriever with high recall and let the model do the heavy lifting.

Anyone still designing systems with the 2023 mindset is paying a premium to solve a problem whose shape has changed. RAG didn’t die, the “R” got dumber and cheaper, and that’s an upgrade. The vector DB vendors aren’t going to tell you this, but it’s the path the more experienced folks have been quietly walking.

The next wave of LLM applications, in my bet, is going to be dominated by the people who got this inversion. Smaller stacks, simpler infrastructure, generous context, and a whole lot less LangChain.

What the recent literature says

Before I close out, I went and checked what the research crowd published on this. Blog hot takes age in three months in this field, so it’s better to look at the papers.

Retrieval Augmented Generation or Long-Context LLMs?, out of Google DeepMind, published at EMNLP 2024, is probably the most cited piece in the debate. Their conclusion: when the model has enough resources, long context beats RAG on average quality, but RAG is still much cheaper in tokens. They propose Self-Route, an approach where the model itself decides whether it needs retrieval or whether it can just go straight through context. The token savings are big and the quality loss is small.

Then LaRA, presented at ICML 2025, is more measured. The authors built 2326 test cases across four QA task types and three long-context types, ran them across 11 different LLMs, and the conclusion was: there is no silver bullet. The choice between RAG and long context depends on the model, the context size, the task type, and the retrieval characteristics. RAG wins on dialogue and generic queries, long context wins on Wikipedia-style QA.

Long Context vs. RAG for LLMs: An Evaluation and Revisits, from January 2025, is the one that most reinforces this post’s thesis. Long context tends to beat RAG on QA benchmarks, especially when the base document is stable. Summarization-based retrieval comes close, and chunk-based retrieval lags behind. In other words: the old way, chunk plus embed plus top-k, is the one that comes out worst.

Worth keeping on the radar too is the original Lost in the Middle (Liu et al., 2023, published in TACL in 2024). That’s the paper that showed even models with big windows have performance that depends on the position of the relevant information. Stuff at the beginning or end of the context is found easily; stuff in the middle degrades. For a long time this got used as the argument against long context, but the paper is from 2023, with 2023 models. Today’s models, the Claude 4.x and Gemini 3.x line, handle the middle a lot better. It’s not a solved problem, but it’s much smaller than it was.

On the lexical retrieval side, BEIR is still the canonical reference. The classic result is that BM25, all the way from the 90s, is still ridiculously competitive in out-of-domain scenarios. Dense models only win consistently when you have in-domain data to fine-tune the embeddings. In zero-shot scenarios, which is where most projects live, BM25 is hard to beat without serious work.

To wrap up, the Anthropic post on Contextual Retrieval, from September 2024, is the most practical piece on the list. They show that combining contextual embedding with contextual BM25 drops the top-20 failure rate from 5.7% to 2.9%. Add a reranker and it drops to 1.9%. Important detail: BM25 is the centerpiece of their result, not a sidekick. The right reading is “lexical plus vector plus reranker is the combination that works.” Anyone who can only pick one picks BM25 and still gets pretty far.

To sum up what we can actually nail down: the literature isn’t claiming “RAG is dead.” It’s saying that long context, when you can use it, tends to win on quality. It’s saying RAG’s cost is still its main argument. It’s saying lexical BM25 is much stronger than the vector DB marketing makes it sound. And it’s saying that when you really do need heavy retrieval, the robust combination is hybrid (lexical plus vector plus reranker), not pure vector. All of that lines up with what I’ve been defending in practice.

Sources

• Li, Z. et al. (2024). Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. EMNLP 2024 Industry Track.
• Yuan, K. et al. (2025). LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing. ICML 2025.
• Yu, T. et al. (2025). Long Context vs. RAG for LLMs: An Evaluation and Revisits. arXiv:2501.01880.
• Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. TACL 2024.
• Thakur, N. et al. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks 2021.
• Anthropic (2024). Introducing Contextual Retrieval. Blog post.
• Akita, F. (2026). Claude Code’s Source Code Leaked. Here’s What We Found Inside. — my coverage of the leak, with more detail on the memory architecture, KAIROS and autoDream.

If I were building a secondary Rails app connecting directly to the same database as the first, I could just add the dependency to the extracted gem:

gem 'central-support', github: 'Codeminer42/cm42-central-support', branch: 'master', require: 'central/support'

Then recreate the models including the same Concerns, and make sure I remove the ability to bin/rails db:migrate from the secondary app (by creating empty db tasks with the same name, for example).

By the way, this is one big caveat that I didn’t address in Part 1: up to this point, the schema was frozen in the central-support gem.

From now on, you must control the evolution of the tables mapped in the gem from within the gem. The best approach is to use the spec/support/rails_app and normally create new migrations with bin/rails g migration from there. Then you must move the migration to the lib/generators/central/templates/migrations folder.

The lib/generators/central/install_generator.rb will take care of making a central:install task available that will dutifully put new migrations into your application’s db/migrate folder as usual. You just have to bundle update central-support to get the newest changes, run bin/rails g central:install to create the new migrations (it will automatically skip existing ones) and run the normal bin/rails db:migrate. A migration generator code is very simple, you can do it like this:

require 'securerandom'
require 'rails/generators'
require 'rails/generators/base'
require 'rails/generators/active_record'

module Central
  module Generators
    class InstallGenerator < Rails::Generators::Base
      include ActiveRecord::Generators::Migration

      source_root File.expand_path("../templates", __FILE__)

      def create_migrations
        Dir["#{self.class.source_root}/migrations/*.rb"].sort.each do |filepath|
          name = File.basename(filepath)
          migration_template "migrations/#{name}", "db/migrate/#{name}", skip: true
          sleep 1
        end
      end
    end
  end
end

The migration_template will take care of adding the proper timestamp to the migration file, so you don’t have to add it manually and the template file name can be something plain such as migrations/add_role_field_to_user.rb.

All that having been said, there is a second challenge I added to my internal secondary app: I wanted it to have its own main database and use Central’s database as a secondary read-only source.

So migrations in the secondary app (let’s just call it Central-2) will run against its own main database, not against the main Central’s database. This adds the following problem: the test suite must be able to create and migrate both test databases, in isolation from Central. Only in production should Central-2 connect to Central’s database.

Every Rails application has a config/database.yml.sample and a db/schema.rb, so I started by creating a config/database_central.yml.sample and a db_central/schema.rb.

The config/database_central.yml.sample is already interesting:

development:
  adapter: postgresql
  encoding: unicode
  timeout: 5000
  database: fulcrum_development
  pool: 5

# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
  adapter: postgresql
  encoding: unicode
  timeout: 5000
  <% if ENV['DATABASE_CENTRAL_URL'] %>
  url: <%= ENV['DATABASE_CENTRAL_URL'] %>
  <% else %>
  database: central_test
  <% end %>
  pool: 5

production:
  adapter: postgresql
  encoding: unicode
  timeout: 5000
  url: <%= ENV['DATABASE_CENTRAL_URL'] %>
  pool: <%= ENV['DB_POOL'] || 5 %>

In production, it will use the DATABASE_CENTRAL_URL environment variable to connect to Central’s main database.

When running tests locally, it will simply connect to a local database named central_test.

Now, while running tests at Gitlab-CI (or any other CI for that matter), I have to configure the DATABASE_CENTRAL_URL to point to a secondary Postgresql test database.

For Gitlab, this is how I configure the build script:

image: codeminer42/ci-ruby:2.3

services:
  - postgres:latest

cache:
  key: central-bot
  untracked: true
  paths:
    - .ci_cache/

variables:
  RAILS_ENV: test
  DATABASE_URL: postgresql://postgres:@postgres
  DATABASE_CENTRAL_URL: postgresql://postgres:@postgres/central_test

before_script:
  - bundle install --without development production -j $(nproc) --path .ci_cache
  - cp config/database.yml.sample config/database.yml
  - cp config/database_central.yml.sample config/database_central.yml
  - bin/rails --trace central:db:create central:db:schema:load
  - bin/rails --trace db:create db:schema:load

test:
  script:
    - bundle exec rspec

Notice how I copy the “.sample” config files to make sure they exist. And then how I run tasks you know such as db:create db:schema:load to create the normal test database, but also tasks you don’t know such as central:db:create central:db:schema:load.

I defined those tasks in lib/tasks/db_central.rake like this:

task spec: ["central:db:test:prepare"]

namespace :central do

  namespace :db do |ns|

    task "environment:set" do
      Rake::Task["db:environment:set"].invoke
    end

    task :drop do
      Rake::Task["db:drop"].invoke
    end

    task :create do
      Rake::Task["db:create"].invoke
    end

    task :setup do
      Rake::Task["db:setup"].invoke
    end

    task :reset do
      Rake::Task["db:reset"].invoke
    end

    task :migrate do
      Rake::Task["db:migrate"].invoke
    end

    task :rollback do
      Rake::Task["db:rollback"].invoke
    end

    task :seed do
      Rake::Task["db:seed"].invoke
    end

    task :version do
      Rake::Task["db:version"].invoke
    end

    namespace :schema do
      task :load do
        Rake::Task["db:schema:load"].invoke
      end

      task :dump do
        Rake::Task["db:schema:dump"].invoke
      end
    end

    namespace :test do
      task :prepare do
        Rake::Task["db:test:prepare"].invoke
      end
    end

    # append and prepend proper tasks to all the tasks defined here above
    ns.tasks.each do |task|
      task.enhance ["central:set_custom_config"] do
        Rake::Task["central:revert_to_original_config"].invoke
      end
    end
  end

  task :set_custom_config do
    # save current vars
    @original_config = {
      env_schema: ENV['SCHEMA'],
      config: Rails.application.config.dup
    }

    # set config variables for custom database
    ENV['SCHEMA'] = "db_central/schema.rb"
    Rails.application.config.paths['db'] = ["db_central"]
    Rails.application.config.paths['db/migrate'] = ["db_central/migrate"]
    Rails.application.config.paths['db/seeds'] = ["db_central/seeds.rb"]
    Rails.application.config.paths['config/database'] = ["config/database_central.yml"]

    ActiveRecord::Tasks::DatabaseTasks.database_configuration = CM(Rails.root)
      .join("config", "database_central.yml")
      .File.read
      .ERB.new
      .result
      .YAML.load
      .unwrap.freeze
  end

  task :revert_to_original_config do
    # reset config variables to original values
    ENV['SCHEMA'] = @original_config[:env_schema]
    Rails.application.config = @original_config[:config]
  end
end

This is how I define a namespace of Rake tasks that start with central:, and every one of those connects to the secondary database as described in database_central.yml. The weird syntax is from my chainable_methods, don’t mind it too much.

The db_central/schema.rb is basically a copy of the spec/support/rails_app/db/schema.rb from the central-support gem, with the same tables. The spec runner of both the gem and this secondary app will just load the schema into the test database.

Now that we have the basic underpinnings for specs in place, we can focus on how the application itself can consume those external models.

We start by adding an initializer like config/initializer/db_central.rb:

DB_CENTRAL = CM(Rails.root)
  .join("config", "database_central.yml.sample")
  .File.read
  .ERB.new
  .result
  .YAML.load
  .unwrap.freeze

In this case I am reading from the sample file because, unlike the CI build, when I deploy to Heroku I don’t have a script to copy the sample to the final yaml file. This will populate the constant DB_CENTRAL with the database URL stored in the DATABASE_CENTRAL_URL environment variable that I have to set.

Then I create a new file called app/models/remote_application_record.rb that looks like this:

class RemoteApplicationRecord < ApplicationRecord
  establish_connection DB_CENTRAL[Rails.env]
  self.abstract_class = true

  unless Rails.env.test?
    default_scope -> { readonly }
  end
end

This is how you create a new connection pool for the secondary database configuration. You must only have this establish_connection in one place and have the models inherit from here. The abstract_class = true will make ActiveRecord not try to load from a table of the same name as this class.

Then we have a default_scope locking down the model as readonly. We don’t want that in the test environment because I still want to have Factory Girl populate the test database with fake data for the specs. But it’s a good idea to have it in the production environment just to make sure.

Finally, I can create all the models I need, such as app/models/central/team.rb:

module Central
  class Team < RemoteApplicationRecord
    self.table_name = 'teams'

    include Central::Support::TeamConcern::Associations
    include Central::Support::TeamConcern::Validations
    include Central::Support::TeamConcern::Scopes
  end
end

From here I can just call normal Arel queries such as Central::Team.not_archived.limit(5).

Conclusions

If you didn’t already, refer to Part 1 for more details.

This is a simple recipe to share model logic between a main read-write Rails app and a secondary read-only Rails app. They share most (not all) of the same models, they share the same logic (through some of the Concerns), and they share the same database.

In this particular case, the recommended approach is to create a Follower database, which is how Heroku calls a secondary replicated database, and make my secondary application connect to that (since it only needs a read-only source).

For more complicated scenarios, you will need a more complicated solution such as an HTTP API layer to make sure only one App manages model migrations and such. But the Rubygem approach should be “good enough” for many cases.

If I really need to go that way, it won’t be too difficult to transform this small gem into a full-blown Rails API app. If you can’t even separate the logic as a Concern, you won’t be able to separate them as APIs either, so consider this a quick exercise, a first step towards creating an Anti Corruption Layer.

And as a bonus, consider contributing to the Central and central-support open source projects. I intend to build a competitive Pivotal Tracker/Trello alternative and we are getting there!

But my major complaint with Mattermost is that the free, open source, Team Edition lacks a super important feature: not allowing users to delete private groups.

@iantien commented that the private groups are never actually “deleted”, they are just marked as deleted, audited, but all data is still in the database. Just the UI has no way to hide the “delete” option from users and there is no Administration UI to unarchive the deleted groups.

In fact, you can open a psql session in your PostgreSQL database and just do:

update channels set deleteat = 0;

This will unarchive and restore all deleted channels. But you can see how this is a hassle.

I strongly disagree when he says that only “10,000-user enterprises” would need such a feature. Even in a small team, any grumpy user can just archive a channel out of the blue and disrupt the entire team communication in private groups. Sure, the “Town Square” and other public channels will still function, but if you have just one external user participating in projects, for example, you want to use private groups to isolate your internal communication from external users.

So, not having the option for very basic permissions (such as disallowing members to delete channels or private groups) is a very big show stopper even for small teams. And sure, the USD 20/user/year fee is not expensive, but as Mattermost still has fewer features than Rocket.chat, it becomes a very hard sell.

Also, hacking the code itself and adding a flag to disallow this option, in Go, is actually quite easy, but you would have to maintain your own fork (as I think Mattermost would not accept a pull request for a feature that already is in their paid, Enterprise offering).

But after @iantien commented that nothing is deleted and it’s all audited, I quickly realized that I could use the audit metadata and devise a way to automatically restore the channels (unless it’s the system admin doing it). All without altering the source code.

One can use the many tools available in PostgreSQL itself, namely: TRIGGERS. So, without further ado, just run this in your Mattermost database:

CREATE OR REPLACE FUNCTION undelete_channel() RETURNS trigger AS $$
    DECLARE
        user_counter integer;
        channel_id character varying(26);
    BEGIN
        -- Only for channel delete operations
        IF NEW.action NOT LIKE '%/channels/%/delete' THEN
            RETURN NEW;
        END IF;

        -- Check if it is the system_admin
        SELECT count(*) INTO user_counter
        FROM users
        WHERE id = NEW.userid
        AND roles = 'system_admin';

        IF user_counter > 0 THEN
            RETURN NEW;
        END IF;

        channel_id = split_part(NEW.action, '/', 7);

        UPDATE channels
        SET deleteat = 0
        WHERE id = channel_id;

        RETURN NEW;
    END;
$$ LANGUAGE plpgsql;

DROP TRIGGER IF EXISTS undelete_channel ON audits;
CREATE TRIGGER undelete_channel AFTER INSERT ON audits
    FOR EACH ROW EXECUTE PROCEDURE undelete_channel();

That’s it, it will listen for new audit inserts, check if it is a “channel delete” action, check that it is not a ‘system_admin’, and if so it will automatically grab the channel id from the action REST URL and do the proper UPDATE to bring it back.

I tested it already and in my UI users don’t even realize something happened. Not even the offending user sees the channel go away, it instantly comes back.

So, if this was the only thing stopping you from using the free-of-charge, on-premise Team Edition, there you go. And with this you can derive functions to also prevent channels from being renamed, but I’ll leave it as an exercise for you (please share in the comments section below if you do it).

Happy Hacking!

In my previous post I forgot to mention minimal firewall and SSL configuration, but you can check it here.

One of the most difficult pieces to move away from is Slack. Don’t get me wrong, it’s a great tool and it gets the job done. The free-tier is really competent but I really want some of the paid features such as history of all conversations with search capabilities and uploads without limit.

The Cost Conundrum

But then you have to commit to pay USD 8/month per user (USD 6.67 if you pay ahead of time). If you’re a small team (around a dozen people) it’s well worth it and you should definitely pay and be done with it.

If you have more than 20 people then it starts to become a conundrum. On my GitLab cost calculations, some people argued that saving USD 15k yearly is not such a big deal, and they’re not wrong.

If you’re from outside of the US coastal cities, especially from countries in South America or Asia where currency can be more than 3 times weaker, then it quickly starts to become a big deal.

Apart from that, we are currently offloading a whole lot of our IPs and knowledge into the care of companies we don’t really know and where we don’t hold any real stakes.

If you’re a medium to big company it really starts to make sense to want to have more control over your own property. Of course, it costs some work and maintenance, so it’s not for everybody.

What I will argue is that if you have 150 people or more, and you’re still growing, it’s not such a big burden to jump into your own infrastructure.

You avoid paying around USD 12,000 a year or more, and instead you will pay USD 960 a year in Digital Ocean boxes and MMS plus a few hours every month in basic maintenance. That’s 12 times cheaper.

Again, you shouldn’t do it just for the cost savings as the maintenance burden can easily outweigh the savings if you or your team are not devops savvy and you don’t have anyone to keep monitoring it. For most companies, the conversation history is not so important and even the free-tier will be more than enough.

Cost is less relevant depending on your context, and there are many different uses for group chats. For example, if you’re an open source community, you would try Gitter instead.

There are also many other alternatives that you may want to try.

For the sake of this article, let’s assume you’re a small to medium company, in need of private rooms, many external users (for example, your clients), with at least 50 people. Then this can start to make sense. Any other scenario and we have to analyze it differently.

Why I chose Rocket.chat

With the cost reasoning out of the way, then it’s a matter of choosing which alternative to use.

I tried MatterMost first and I really wanted to like it. It has a very similar feature set to Slack and most importantly, it’s written in Go. So it’s both lightweight, very fast, and not so difficult to contribute. It uses MySQL or PostgreSQL which I like.

Unfortunately, it has an important show-stopper for my needs. You can create both public channels or private groups, but anyone can delete a private group. If you want permission control for that you must use their “Enterprise” offering and pay USD 20/year per user.

It’s still at least 3 times cheaper than Slack, so it may be a good option for you.

I really want to have full-control, not just make it cheaper. I could pay Slack already if I didn’t care, so I would not just pay an alternative and not have full control (including the ability to tweak and improve the entire code base).

Then, it took me back to Rocket.chat. Its feature set also rivals Slack, it’s good looking enough. But, it’s made in Meteor. Now, I don’t have anything against Meteor and I really think a Slack-clone is exactly the kind of use case where you could use Meteor to its full potential.

Technically I really think it’s a downside to be forced to use MongoDB (Meteor requires Mongo). For small installations I really prefer to have PostgreSQL. MongoDB is competent, but for medium to big installations there are caveats, and I will show you why below.

I strongly dislike the culture of writing software without minimal care such as having a reasonably complete test suite. You browse through the many packages that comprise Rocket.chat and several of them have no tests whatsoever. Some packages do have jasmine tests, but the majority is lacking.

This is one example I picked randomly, out of the few test files I found:

describe 'rocketchat:katex Client', ->

  it 'should exist', ->
    expect(RocketChat.katex).toBeDefined()

This is sad, sorry to say that, but it is. Not surprisingly I’ve seen katex issues while using the interface. And if you think this was just a poor choice, I picked another test file:

describe 'rocketchat:markdown Client', ->

  it 'should exist', ->
    expect(RocketChat.Markdown).toBeDefined()

Really?

So, the code itself is not so pretty, you have to keep this in mind. I’d much rather use Mattermost (which is much better structured and with enough tests), but it’s not entirely free, so it’s not an option for me. For my scenario, I’d rather have sloppy code (that minimally “works”) that I can tweak than code I can’t see, for this particular venture. Many inexperienced developers may argue about the necessity of having automated tests, but not having them also impacts this:

If we see how the project is evolving over time, we can see that interest is diminishing. The lack of good automated tests also makes it more difficult for new people to contribute without adding regressions, and makes contribution reviews more difficult and time consuming.

On the other hand the feature set is good enough as it is. There is only so much you can add to a Slack-clone. The upside is that if the code doesn’t change too much, I can tweak it more without having many things breaking all the time. It’s the complete opposite from the impressive contribution boost over at GitLab, for example.

Doing a few days of testing through the administration settings, user interface, having dozens of users doing real conversation, convinced me that even though the code quality is bad, the feature set is indeed there. I think I can live with that. Your mileage may vary though.

I am using Franz to open Slack, Rocket.chat, Hangouts, Messenger and any number of other communicators in one single app, which is great. And Rocket.chat does provide mobile apps based off PhoneGap. Again, it’s good, not great.

Many people have heavy bots or integrations into their Slack configuration. The alternatives such as Rocket.chat do support incoming and outgoing webhooks and most of the integrations should be possible (with some tweaking and hacking such as this Giphy example), so make a list of all important integrations and do some Google research on that before continuing.

Planning the Infrastructure

All that having been said, I’d recommend you try Rocket.chat’s demo server first to see if it has the minimal features you need. And also that you install a dry-run in a small Digital Ocean box with the intention of blowing it away later.

For a single-instance, stand-alone installation, temporary dry-run, I recommend you start by installing the MongoDB One-Click Application and follow this instruction to install everything in one machine.

This is the worst possible production environment, which is why I am being so repetitive in saying that you should destroy this box after you do your testing.

As I’ve said before, I am assuming a scenario where you have 50 or more people, with real clients, important communication going on. You don’t want to keep a simple installation like this for production usage.

For a minimal production installation you will need at the very least 4 boxes with 1 GB of RAM each. Now, why 4?

This is an important section for most of the beginner Javascript programmers out there playing with the so called “MEAN” stack (MongoDB, Express, Angularjs, Node.js). I’ve seen people deploy single-instance, stand-alone “MEAN” apps for production usage. And this is super bad.

MongoDB is NOT MEANT to function with just one single instance!

More than that:

DO NOT start an even number of MongoDB nodes, or at least install an Arbiter

So, the minimal setup for MongoDB is a Three-member Replica Set, no less!

The article “MongoDB Gotchas & How To Avoid Them” is a little bit old but you should be aware of several of the documented gotchas, including adding replica sets and avoiding waiting too long to start sharding your data.

The most important part of making Rocket.chat reliable is to make your MongoDB minimally reliable, and this is a hidden cost you must consider.

Installing a Three-Member Replica Set

For the sake of this article I will assume that you created 3 (three) MongoDB One-Click Applications on Digital Ocean’s, at the very least with 1 GB RAM boxes (2 GB RAM machines are a good choice if you’re in doubt, but never the 512 Mb) and another bare Ubuntu 14.04 box to be the web server. Create all machines in the same sub-region, all of them with private networking and backup enabled.

Take note of the public IPs and private IPs, for the sake of this article let’s say you have this:

Public IP     Private IP    Hostname

192.160.10.1  10.0.0.1      mongo1.yourdomain.com
192.160.10.2  10.0.0.2      mongo2.yourdomain.com
192.160.10.3  10.0.0.3      mongo3.yourdomain.com
192.160.10.4  10.0.0.4      yourdomain.com

In the commands and code snippets below, make sure to replace the fake IPs with your own private IPs.

As I said in my GitLab post, create a swap file, configure the locale, set it up for automatic security updates, and upgrade the Ubuntu packages.

The next first thing: configure the firewall in all machines (assuming ufw package is already installed):

sudo ufw reset
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow https
sudo ufw allow ssh
sudo ufw allow in on eth1 to any port 27017
sudo ufw enable

We are allowing MongoDB’s 27017 port only for machines in the same private networking (through eth1, the public IP goes through eth0). We’re also allowing port 22 for SSH and although I am not listing it here in this article you should also consider installing and configuring fail2ban. Digital Ocean also has a good post on UFW for more details.

Next thing to do on the MongoDB machines is to Disable Transparent Huge Pages (THP). For that create a file /etc/init.d/disable-transparent-hugepages, with this content:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    unset thp_path
    ;;
esac

Now you can register it as a service:

sudo chmod 755 /etc/init.d/disable-transparent-hugepages
sudo update-rc.d disable-transparent-hugepages defaults
sudo /etc/init.d/disable-transparent-hugepages start

Stop MongoDB in all 3 machines:

sudo service mongod stop

Now comes a complicated procedure that you should not do wrong. It’s only complicated because you have to do it in all 4 machines minding their correct private IP addresses. If you already have your own DNS server, you can skip this, otherwise follow through carefully.

You will have to edit both /etc/hostname and /etc/hosts. Each machine will have a particular name and they should be able to find the other machines by name. Let’s start by editing the /etc/hostname on each machine. Use the table in the beginning of this section, for example, in the machine with public IP 192.160.10.1 you must edit its hostname to be mongo1.yourdomain.com, and so on for all the machines.

Once you have it done, you have to replace the line 127.0.0.1 localhost in the /etc/hosts in each of them for the following:

127.0.0.1 localhost mongo1.yourdomain.com
10.0.0.1 mongo1.yourdomain.com
10.0.0.2 mongo2.yourdomain.com
10.0.0.3 mongo3.yourdomain.com
10.0.0.4 yourdomain.com

This snippet assumes you’re in the machine with private IP 10.0.0.1, of course. Make sure you change mongo1.yourdomain.com in the first line for the hostname of the machine you’re in. If this is too difficult for you, you definitely should not try to maintain your own infrastructure, so beware.

Finally, let’s edit the /etc/mongod.conf with these changes:

storage:
  dbPath: /mongodb-data
...
net:
  port: 27017
  bindIp: 0.0.0.0
...
replication:
  replSetName: rs0

This should be exactly the same in all machines. And you must create this /mongodb-data directory like this:

sudo mkdir /mongodb-data
sudo chown -R mongodb:mongodb /mongodb-data

Now go ahead on machine mongo1.yourdomain.com, start up with sudo service mongod start and fire up the mongo command, and you should see something like this:

# mongo
MongoDB shell version: 3.2.8
connecting to: test
>

Let’s now configure the replica set by issuing this command in the MongoDB command-line interface:

rs.initiate()
rs.add('mongo2.yourdomain.com')
rs.add('mongo3.yourdomain.com')

And you can now log in to the secondary machines and sudo service mongod start in all of them. They should all start joining the replica set and syncing.

You can check the status of the replication like this:

> rs.conf()

{
  "set" : "rs0",
  "date" : ISODate("2016-08-09T17:33:10.466Z"),
  "myState" : 1,
  "term" : NumberLong(-1),
  "heartbeatIntervalMillis" : NumberLong(2000),
  "members" : [
    {
      "_id" : 1,
      "name" : "mongo1.yourdomain.com:27017",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "uptime" : 11727,
      "optime" : Timestamp(1470763990, 3),
      "optimeDate" : ISODate("2016-08-09T17:33:10Z"),
      "electionTime" : Timestamp(1470752553, 1),
      "electionDate" : ISODate("2016-08-09T14:22:33Z"),
      "configVersion" : 6,
      "self" : true
    },
    {
      "_id" : 2,
      "name" : "mongo2.yourdomain.com:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 11693,
      "optime" : Timestamp(1470763988, 4),
      "optimeDate" : ISODate("2016-08-09T17:33:08Z"),
      "lastHeartbeat" : ISODate("2016-08-09T17:33:10.026Z"),
      "lastHeartbeatRecv" : ISODate("2016-08-09T17:33:08.887Z"),
      "pingMs" : NumberLong(0),
      "syncingTo" : "mongo3.yourdomain.com:27017",
      "configVersion" : 6
    },
    {
      "_id" : 3,
      "name" : "mongo3.yourdomain.com:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 11693,
      "optime" : Timestamp(1470763988, 4),
      "optimeDate" : ISODate("2016-08-09T17:33:08Z"),
      "lastHeartbeat" : ISODate("2016-08-09T17:33:09.994Z"),
      "lastHeartbeatRecv" : ISODate("2016-08-09T17:33:08.655Z"),
      "pingMs" : NumberLong(0),
      "syncingTo" : "mongo1.yourdomain.com:27017",
      "configVersion" : 6
    }
  ],
  "ok" : 1
}

Each box should be syncing data to another box, one of them will be marked “PRIMARY” and the others will be “SECONDARY”. If one of them “dies” for some reason, one of the others will be elected as PRIMARY until you add a replacement box. Read more details on this procedure in Digital Ocean’s post about implementing replication sets.

Adding your MongoDB environment to MMS

Another thing beginners will not realize is that it’s a good idea to monitor your services. At the very least you should install MongoDB Cloud Manager agents into your servers. Just set up an account at MongoDB Atlas.

It’s beyond the point of this article to explain Atlas, but you should have no trouble installing the agents with the Group ID and API Key provided. Then install the monitoring agent and have the Manager figure out your existing deployment and replica set.

The basic subscription for monitoring will cost you an extra USD 39 a month, but believe me when I say it’s worth it to add all your MongoDB deployments under MMS.

Installing a multi-instance Node.js service

Another mistake beginners in the MEAN stack make is to start up a single Node.js instance. Yes, Node.js’s asynchronous I/O nature makes it “concurrent enough” in 1 single thread. But you do want to maximize the rented machine so you should spin up at least one Node.js instance for each CPU core. A 1 GB RAM machine has just 1 core, but you can spin up at least 2 instances.

First of all, we must set up the yourdomain.com (192.160.10.4 or 10.0.0.4 in the example) to support Node.js, let’s do it:

sudo apt-get install -y npm curl graphicsmagick nginx git bc
sudo npm install -g n
sudo npm install -g forever
sudo npm install -g forever-service
sudo n 0.10.40

If you haven’t already, create a more restricted sudo user:

adduser rocket # add a password when asked
usermod -aG sudo rocket
su - rocket # and always ssh in with `ssh rocket@yourdomain.com`

Within rocket’s home directory let’s install the Rocket.chat codebase:

cd /home/rocket
curl -L https://rocket.chat/releases/latest/download -o rocket.chat.tgz
tar zxvf rocket.chat.tgz
mv bundle Rocket.Chat
cd Rocket.Chat/programs/server
npm install

And now let’s add a way for the machine to start the Rocket.chat whenever it reboots:

cd ~/Rocket.Chat
sudo forever-service install -s main.js -e "ROOT_URL=https://rocketchat42.com/ MONGO_URL=mongodb://mongo1.yourdomain.com:27017/rocketchat MONGO_OPLOG_URL=mongodb://mongo1.yourdomain.com:27017/local PORT=3001" rocketchat1
sudo forever-service install -s main.js -e "ROOT_URL=https://rocketchat42.com/ MONGO_URL=mongodb://mongo1.yourdomain.com:27017/rocketchat MONGO_OPLOG_URL=mongodb://mongo1.yourdomain.com:27017/local PORT=3002" rocketchat2

sudo start rocketchat1
sudo start rocketchat2

Let’s also create SSL certificates (and for that you must have a properly registered domain, of course). Again, Digital Ocean’s documentation on Let’s Encrypt is very good. In summary, all you have to do is:

sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Now edit the /etc/nginx/sites-available/default and add this inside the server block:

server {
  ...
  location ~ /.well-known {
          allow all;
  }
  ...
}

Now you can:

sudo service nginx reload

sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

cd /opt/letsencrypt
./letsencrypt-auto certonly -a webroot --webroot-path=/usr/share/nginx/html -d yourdomain.com -d www.yourdomain.com

Again, replace yourdomain.com with your own registered domain, of course. The letsencrypt command will prompt you to enter your e-mail address and accept terms of service.

Let’s Encrypt is a fantastic free SSL provider. Their certificates are short lived and are meant to expire in 90 days, so you must set up auto-renewal. Start sudo crontab -e and add the following lines:

30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
35 2 * * 1 /etc/init.d/nginx reload

This should take care of SSL. Now you can edit your /etc/nginx/sites-available/default and replace it with this template:

# Upstreams
upstream backend {
    server 127.0.0.1:3001;
    server 127.0.0.1:3002;
}

server {
  listen 443 ssl;

  server_name yourdomain.com;

  ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_dhparam /etc/ssl/certs/dhparam.pem;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;
  ssl_stapling on;
  ssl_stapling_verify on;
  add_header Strict-Transport-Security max-age=15768000;

  location / {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forward-Proto http;
    proxy_set_header X-Nginx-Proxy true;

    proxy_redirect off;
  }

}

server {
  listen 80;
  server_name yourdomain.com www.yourdomain.com;
  return 301 https://$host$request_uri;
}

Just for the sake of completeness, again: replace yourdomain.com with your domain. Usually the /etc/nginx/sites-enabled/default is a symlink to /etc/nginx/sites-available/default, just check that as well. This configuration will load balance between the 2 node.js instances we configured and started before. And you can add as many as you want following the same forever-install procedure and adding the new instances to the upstream section in the nginx configuration above.

If everything is correct, you can restart nginx again: sudo service nginx restart.

And now you should have https://yourdomain.com already up and running. The first user that signs up will be the site administrator and from there you can customize Rocket.chat internally.

Conclusion

As you can see, this is not a simple procedure to follow and I am assuming you have experience managing your own infrastructure. If you don’t, you definitely should NOT do this by yourself.

If you do it correctly, you should have a functional Slack-clone with minimal reliability (thanks in large part to Digital Ocean) and minimal cost (USD 80/month in the configuration I described that should be enough for more than 100 users). Rocket.chat also offers more documentation for other environments including Docker configurations you might want to try. But the procedure above is sufficient for my needs.

I said it already but it’s better to repeat it: don’t fall for the trap of installing everything in a single box, with single instance MongoDB, and without proper monitoring. It’s asking for trouble.

For now I am in the middle of the rollout. All developers in my company are already in the new deployment and soon half of the clients should also migrate (some will not be able to leave Slack just yet), but with Franz and the ability of full history and searchability this shouldn’t be a concern even if a developer stays offline for some period of time.

I also don’t advocate that everybody should be online and responding in real-time. It’s infeasible, unproductive, creates unnecessary tension. People should participate when they have free time, and they should be able to concentrate without worrying that they are missing something. That’s why they should opt-out of being notified in the more busy channels and just enable notification on the private groups that matter.

Coincidentally Jason Fried just posted about concerns over group chats at Signal v. Noise, but the gist is that group chat should be purposeful and not yet another tool to create tension. People should definitely get offline when they need to fully concentrate on their work and have the opportunity to catch up with interesting conversations later. And really important communication should go through e-mail or other traditional ways, a simple @all doesn’t cut it for company-wide announcements for example.

I hope this exercise gives you more perspective on what you can have and also raises awareness on the need for companies to regain more control over their own data, particularly knowledge. Erase your communication channels and you’re losing years’ worth of knowledge that can be invaluable.