And be sure to subscribe to my new newsletter The M.Akita Chronicles!

–

There’s an obsession in the market that frontend = React. Or Vue. Or Svelte. Or the framework of the week. What if I told you it’s possible to have a modern, responsive frontend with dark mode, email-compatible, multi-platform — without a single line of JavaScript framework?

I’m not talking about static sites from the 2000s. I’m talking about Tailwind CSS v4 on Rails, Hugo with the Hextra theme for the blog, and email templates that work from Gmail to Outlook 2016. All in the same project, with one developer.

Tailwind CSS v4: The End of Handcrafted CSS

I have a confession: CSS has always been my kryptonite. Not because it’s hard — but because it’s unpredictable. You change a margin-top and break the layout of a page you never touched. Cascade, specificity, inheritance — it’s a minefield.

Tailwind solves this in the most pragmatic way possible: utility classes right in the HTML.

<div class="flex items-center gap-4 p-6 bg-white dark:bg-gray-800
            rounded-xl shadow-sm border border-gray-200 dark:border-gray-700">
  <img class="w-12 h-12 rounded-full" src="/images/avatar.jpg">
  <div>
    <h3 class="font-semibold text-gray-900 dark:text-gray-100">Título</h3>
    <p class="text-sm text-gray-500 dark:text-gray-400">Descrição</p>
  </div>
</div>

“But this is just inline style with extra steps!” — No, it isn’t. It’s a design system with constraints. You don’t pick “margin: 17px” — you pick m-4 (16px) or m-5 (20px). That constraint forces visual consistency without needing a design system documented across 40 pages.

And Tailwind v4, which came to Rails via tailwindcss-rails, has a fundamental change: CSS-first configuration. No more tailwind.config.js. Everything is CSS:

@import "tailwindcss";

@theme {
  --color-brand: #8b5cf6;
  --font-sans: "Inter", system-ui, sans-serif;
}

One less config file. One less build step. Simpler.

Dark Mode: Respect Your User

Something that deeply annoys me is a site that ignores the operating system’s theme preference. The user set dark mode on the OS — respect that.

With Tailwind, dark mode is trivial:

<body class="bg-white dark:bg-gray-900 text-gray-900 dark:text-gray-100">

The dark: prefix applies the style when the operating system is in dark mode (via prefers-color-scheme: dark). No JavaScript needed. No toggle needed. No cookie needed. It works.

But “it works” is the bare minimum. The real challenge is maintaining consistency across dozens of components. Every color you use needs a dark: variant. Every background, every text, every border.

The trick: create conventions and follow them. Main background? bg-white dark:bg-gray-900. Primary text? text-gray-900 dark:text-gray-100. Secondary text? text-gray-500 dark:text-gray-400. Borders? border-gray-200 dark:border-gray-700. With 4-5 pairs defined, 90% of components are covered.

Hugo + Hextra: Blog Without a Database

The project’s blog runs Hugo with the Hextra theme. Hugo is a static site generator written in Go — absurdly fast. A full rebuild in <500ms even with dozens of pages.

What makes Hextra special is that it’s a documentation theme adapted for blogs. Navigation sidebar, integrated search, table of contents — everything ready. And customizable via partial layouts:

layouts/
├── _partials/
│   ├── custom/
│   │   └── head-end.html          # Injects analytics, fonts, custom CSS
│   └── section-box.html           # Shared partial (DRY, see below)
├── blog/
│   └── home.html                  # Custom homepage
├── newsletters/
│   └── single.html                # Individual newsletter template
├── podcast-transcripts/
│   └── list.html                  # Episode listing
└── shortcodes/                    # 17 shortcodes — components without JS
    ├── podcast-turn.html          # Speech bubble in the transcript
    ├── akita-comment.html         # Akita's comment
    ├── marvin-comment.html        # Marvin's comment
    ├── history-timeline.html      # Historical events timeline
    ├── market-ticker.html         # Financial tickers table
    ├── market-map.html            # FinViz S&P 500 map
    ├── subscribe-cta.html         # Subscription call-to-action
    ├── score.html                 # Relevance badge (high/medium/low)
    ├── holidays.html              # Holidays and commemorative dates
    ├── ref-stories.html           # Reference links
    ├── book-downloads.html        # Book download links
    ├── hacker-news.html           # ┐
    ├── anime-ranking.html         # │ Section boxes — delegate to
    ├── youtube-digest.html        # │ section-box.html (see below)
    ├── world-events.html          # │
    ├── market-news.html           # │
    └── qa.html                    # ┘

Each content type has its own template. Newsletters have a navigation sidebar and visual sections. Podcast transcripts have an embedded player and dialogue formatting. The same Markdown content is rendered differently depending on the type in the frontmatter.

Shortcodes: Components Without JavaScript

Hugo shortcodes are the equivalent of React components, but without a build step:

<!-- layouts/shortcodes/podcast-turn.html -->
{{ $speaker := .Get "speaker" }}
{{ $isAkita := eq $speaker "akita" }}

<div class="podcast-turn" style="display: flex; {{ if $isAkita }}flex-direction: row{{ else }}flex-direction: row-reverse{{ end }};">
  <img src="/images/{{ $speaker }}.jpg"
       style="width: 40px; height: 40px; border-radius: 50%;">
  <div class="bubble {{ if $isAkita }}akita-bubble{{ else }}marvin-bubble{{ end }}">
    {{ .Inner | markdownify }}
  </div>
</div>

In Markdown, the author writes:

{{< podcast-turn speaker="akita" >}}
Então, vamos falar sobre o que aconteceu essa semana no mundo tech...
{{< /podcast-turn >}}

And Hugo renders a styled chat bubble, with avatar, correct alignment, and processed markdown. Zero JavaScript. Zero framework. HTML + CSS + template logic.

DRY With a Shared Partial

With 17 shortcodes, duplication becomes a problem fast. Seven of them — the “section boxes” that wrap newsletter sections (Hacker News, Anime, YouTube, etc.) — have the same visual structure: icon, title, colored border, Markdown content. Only the color and label change.

The solution is a shared partial that takes parameters:

<!-- layouts/partials/section-box.html -->
<div class="section-box {{ .class }}">
  <div class="section-box-header">
    <span>{{ .icon }}</span> {{ .label }}
  </div>
  <div class="section-box-content">{{ .content | markdownify }}</div>
</div>

And each shortcode becomes a one-liner:

<!-- layouts/shortcodes/hacker-news.html -->
{{ partial "section-box" (dict "class" "hacker-news" "icon" "&#128274;" "label" "The Hacker News" "content" .Inner) }}

It’s the Hugo equivalent of component inheritance: a polymorphic partial parameterized by dict. The CSS in custom.css defines colors per class (.hacker-news, .anime-ranking, etc.) with light/dark variants. Result: 7 shortcodes that used to have duplicated HTML are now 7 lines + 1 partial + CSS.

But there’s a crucial catch: shortcodes are exclusive to Hugo. The same content that goes to the blog also goes out by email. And email doesn’t understand shortcodes.

The Challenge: Same Content, Two Renders

Here’s the central frontend problem in this kind of project: the same Markdown file needs to work in two completely different contexts.

On the blog (Hugo):

• Shortcodes work (\{\{< subscribe-cta >\}\})
• CSS with classes (Tailwind, Hextra)
• Dark mode via prefers-color-scheme
• Responsive images
• Flexible layout

In email (Rails → Amazon SES):

• Shortcodes are dead text — they have to be removed
• CSS must be inline (many email clients ignore <style>)
• Dark mode? Forget it for 80% of clients
• Images with absolute URLs and fixed size
• Layout with <table> (yes, in 2026)

The solution: two independent rendering pipelines. The Markdown content is the source of truth. Hugo transforms it for the web. Rails transforms it for email. Each one with its own rules.

On the Rails side, the MarkdownToHtml service needs to:

1. Process standard Markdown
2. Convert \{\{< shortcode >\}\} into equivalent HTML or strip them — via a SHORTCODE_PATTERNS hash that maps each shortcode to a regex + inline template with CSS
3. Apply the theme (light/dark based on the subscriber’s preference)
4. Inject inline CSS for email client compatibility
5. Rewrite image URLs to absolute

It’s more work than it sounds. Each shortcode needs an email-safe equivalent. Comment avatars need absolute URLs. Sections with colored backgrounds need <td style="background-color:..."> instead of CSS classes.

Email HTML: The Worst Frontend in the World

If you think cross-browser compatibility is hard, try cross-email-client compatibility. Outlook 2016 renders HTML using the Microsoft Word engine. Yes, Word. That’s not a joke.

The survival rules:

1. Layout with <table>. display: flex? grid? They don’t exist in Outlook.
2. Inline CSS on every element. <style> tags are ignored by several clients.
3. No margin on images. Use padding on the parent <td>.
4. Web fonts don’t work. Use a font stack: font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
5. Themes are a subscriber preference, not automatic detection — many email clients don’t reliably support prefers-color-scheme.

The project’s email template is an ERB that takes theme variables:

<td style="padding: 8px 28px; color: <%= @theme[:text] %>;
           background: <%= @theme[:card_bg] %>;">
  <%= raw @content %>
</td>

The theme defines all the colors:

THEMES = {
  light: {
    text: "#1a1a1a",
    card_bg: "#ffffff",
    link_color: "#7c3aed",
    # ...
  },
  dark: {
    text: "#e6edf3",
    card_bg: "#161b22",
    link_color: "#a78bfa",
    # ...
  }
}

Each subscriber picks their theme at signup time. The email is rendered with the correct colors for that person. No media queries, no automatic detection — because email clients don’t support it reliably. It’s simpler to give the user the choice.

Color Palette: HSL and the Science Behind the Choices

“But how did you pick these colors?” — Not by eyeballing it. There’s a science behind it, and it’s called HSL (Hue, Saturation, Lightness).

Most developers think about color as hex (#e8eef8) or RGB (rgb(232, 238, 248)). Those are machine formats — useful for rendering, useless for designing. When you look at #3b50a2 and #c93030 side by side, there’s no way to intuitively know if they’ll work together. They’re just numbers.

HSL solves this. Instead of mixing light channels (Red, Green, Blue), you describe color in human terms:

• Hue: the position on the color wheel, from 0° to 360°. Red = 0°, green = 120°, blue = 240°.
• Saturation: how “vivid” the color is. 0% = gray, 100% = pure color.
• Lightness: how light or dark. 0% = black, 100% = white.

With HSL, you control each dimension independently. And that’s where building the palette comes in.

Step 1: Distribute Hues Around the Color Wheel

The newsletter has 8 section types, each with its own color. For all of them to “belong” visually to the same project, the hues are distributed evenly around the 360° wheel:

Eight hues spaced around the wheel. That’s not random — it’s the same logic as a polychromatic palette in color theory. When hues are equidistant, the human eye perceives harmony even with completely different colors. It’s the same principle artists have been using with a 12-color wheel since the 18th century.

Step 2: Lock Saturation and Lightness Per Layer

Here’s the trick that makes it all work together. Each visual “layer” (background, text, border) has fixed saturation and lightness — only the hue changes:

Light theme:

• Backgrounds: ~92-95% lightness, moderate saturation. With lightness that high, even different saturations result in pastel colors with similar visual weight — no section “screams” louder than another.
• Text: ~15-20% lightness. Dark enough to guarantee >7:1 contrast ratio against the light background (above the WCAG AAA minimum of 7:1).
• Borders: ~45-55% lightness. The chromatic accent — visible but not dominant.

Dark theme:

• Backgrounds: ~10-12% lightness. Uniform depth — every section has the same “darkness”.
• Text: ~80-85% lightness. Light enough for >6:1 contrast ratio on the dark background.
• Borders: same lightness range as in the light theme — they work as accents in both contexts.

In practice, this means generating the dark variant of any color is mechanical: keep the hue, invert the lightness. A background that was 94% becomes 11%. Text that was 18% becomes 83%. That’s not design — it’s arithmetic.

Step 3: Check Contrast (WCAG)

Pretty colors nobody can read are worthless. The WCAG standard (Web Content Accessibility Guidelines) defines minimum ratios:

• AA: 4.5:1 for normal text, 3:1 for large text
• AAA: 7:1 for normal text, 4.5:1 for large text

The light backgrounds at ~95% lightness against text at ~20% lightness give a ratio >7:1 — WCAG AAA. The dark backgrounds (~11%) against light text (~82%) give >6:1 — comfortably WCAG AA, near AAA.

This isn’t altruism. Low-contrast email is harder to read, generates less engagement, and in extreme cases is flagged by email clients’ accessibility filters.

The Result

When you open the newsletter and see the Hacker News section (purple), followed by YouTube (red), followed by Anime (blue) — they look like “the same project” despite being completely different colors. That’s HSL at work: different hues, same saturation, same lightness. The eye perceives coherence without knowing how to explain why.

And the best part: adding a new section is trivial. Pick a hue that isn’t taken on the wheel, apply the same saturation/lightness rules per layer, and the new section automatically “belongs” to the existing design.

# The comment in the code sums up the rules:
# - Hues evenly spaced: blue 210, green 140, sepia 35, amber 45, royal 225, purple 270, red 0, teal 180
# - Light backgrounds: ~95% lightness, ~25% saturation
# - Light text: ~20% lightness (>7:1 contrast ratio on light bg)
# - Dark backgrounds: ~10-12% lightness
# - Dark text: ~80-85% lightness (>6:1 contrast ratio on dark bg)
# - Borders: ~45-55% lightness

Six rules. Eight hues. Two themes. 100% deterministic. Zero subjective “I think this blue looks pretty” decisions — it’s a design system derived from geometric properties of the HSL color space.

Well-Formed HTML: The Spam Filter Nobody Tells You About

Everything I described above — tables, inline CSS, font stacks — solves the rendering problem. But there’s a second reason to follow these rules that most developers ignore: email providers analyze the quality of your HTML to decide if it’s spam.

Gmail, Outlook, Yahoo — all of them run your email’s HTML through heuristics before showing it to the user. Malformed HTML is one of the classic spam signals. It makes sense: spammers don’t test templates. They spit out broken HTML generated by cheap mass-mailing tools. If your email looks like spammer output, it gets treated as spam.

The signals that trip filters:

1. Unclosed or improperly nested tags. A <div> that opens and never closes, or a </td> before the </tr>, is an immediate flag. Providers expect HTML that parses without errors — not HTML “that the browser fixes up”.
2. Unbalanced image-to-text ratio. An email that’s basically a giant image with little text? Classic spam. Providers want to see real text — not text inside images. The rule of thumb is to keep at least 60% text, 40% images.
3. Missing alt attribute on images. Beyond accessibility, an empty or missing alt is a sign of carelessness — and carelessness correlates with spam. Every <img> needs a descriptive alt.
4. CSS that hides content. display: none, visibility: hidden, font-size: 0, text the same color as the background — those are classic keyword stuffing techniques. Even if you use them legitimately for responsive email, the filter doesn’t know the difference.
5. Excessively heavy HTML. An email with 500KB of HTML raises suspicion. Keep it light — the actual content rarely needs more than 100KB of HTML.
6. Links with text that doesn’t match the href. <a href="https://site-real.com">https://site-falso.com</a> is textbook phishing. Even subtle discrepancies (domain in the text different from the domain in the link) are penalized.

The point that ties it all together: the same practices that make email render correctly in Outlook are the ones that keep your emails out of spam. Tables with clean structure? Well-formed HTML. Inline CSS on every element instead of hacks with <style>? No hidden CSS. Images with explicit dimensions and alt text? Legitimate content.

It’s no coincidence. Email providers calibrated their filters by watching what spammers do — and spammers do exactly the opposite of well-built HTML email. When you follow best practices for compatibility, for free you’re also signaling legitimacy to spam filters.

The practical lesson: treat your email’s HTML as if it were an XML document. Validate, test, and make sure every tag opens and closes in the right order. Use tools like Email on Acid or Litmus to test in multiple clients — they also report spam scoring issues. And resist the temptation to copy ready-made templates from the internet without validating them — many are a disaster of malformed HTML that slips past Gmail but gives Outlook a heart attack and gets Yahoo’s spam filter to flag you as suspicious.

RSS Feeds: The Forgotten Frontend

A detail web developers ignore: RSS is a frontend. Podcasts on Spotify and Apple Podcasts are consumed via RSS. The feed needs:

• Valid XML (podcast parsers are strict)
• Complete metadata (title, description, duration, file size)
• Image URLs that actually work (Spotify validates them)
• Cover art with minimum dimensions (1400×1400 for Spotify)
• Unique GUIDs per episode

Hugo generates RSS automatically, but the default template doesn’t have the fields podcast platforms require. A custom template with iTunes tags is necessary:

<itunes:duration>{{ .Params.duration_seconds }}</itunes:duration>
<itunes:image href="{{ .Params.cover_image | absURL }}"/>
<enclosure url="{{ .Params.audio_url }}"
           length="{{ .Params.file_size }}"
           type="audio/mpeg"/>

The Philosophy: Content First, Framework Never

The pattern that emerges from this whole setup is: content is sovereign. Markdown with YAML frontmatter is the universal format. Each frontend (blog, email, RSS, podcast player) consumes the same content and renders it its own way.

There’s no “React component that fetches data from an API”. There’s a Markdown file read by whoever needs it. This is radically simple — and radically powerful. Swap the blog engine from Hugo to something else? Change the templates, the content stays the same. Swap the email provider from SES to Sendgrid? Change the mailer, the templates stay the same.

It’s the Unix philosophy applied to frontend: each tool does one thing well, connected by a universal format (Markdown + YAML).

Conclusion

You don’t need React to have a modern frontend. Tailwind v4 gives you a design system with constraints, native dark mode, and zero JavaScript. Hugo gives you an absurdly fast blog with flexible templates. ERB with inline CSS gives you emails that work in any client.

The real investment is understanding the constraints of each platform and building rendering pipelines for each one. The payoff comes from that, not from learning the trendy framework that’ll die in 18 months.

At the end of the day, the best frontend abstraction is none at all. HTML, CSS, and Markdown. Simple, debuggable, and going to work 10 years from now.

And don’t forget to subscribe to my new newsletter The M.Akita Chronicles!

–

Ten years ago, fetching data from a website was trivial. Four lines of Ruby: Net::HTTP.get, Nokogiri parse, CSS selector, done. The whole world was open to any script with a decent User-Agent.

Today? Half the sites I need to read to put together my newsletter return “Are you a robot?” for any conventional HTTP request. The other half renders the content entirely in JavaScript — there’s no HTML at all in the server response. And the most paranoid ones combine both.

My goal was to create a daily routine so that M.arvin would deliver the top news to me in the morning on my Discord. That way I wouldn’t have to go hunting for the news of the day. Sounded simple… at first.

This post is about how I went from “one HTTP call” to “packing an entire Chromium into Docker” — and why this is the reality of any project that needs to read the web programmatically in 2026.

The Golden Age: Simple HTTP

Let’s start with what still works. Some services offer structured data with zero resistance:

# RSS feeds: the most resilient format on the web
uri = URI("https://news.google.com/rss/search?q=when:24h+allinurl:reuters.com/world")
xml = Net::HTTP.get(uri)
doc = REXML::Document.new(xml)
doc.get_elements("//item").each do |item|
  title = item.get_text("title").to_s
  url   = item.get_text("link").to_s
end

# Public APIs: clean JSON, documented, stable
response = HTTParty.get("https://api.jikan.moe/v4/top/anime?filter=airing")
animes = response.parsed_response["data"]

# GitHub trending: server-rendered HTML with no protection
html = HTTParty.get("https://github.com/trending").body
doc = Nokogiri::HTML(html)
repos = doc.css("article.Box-row")

Hacker News, GitHub Trending, anime APIs, RSS feeds, Project Gutenberg — they all work with a GET and a parser. No JavaScript, no implicit authentication, no fingerprinting. Just like the web was supposed to be.

The problem is that these are the exception. Sites with financial data, exclusive news, or premium content — exactly the data that matters most — are increasingly armored.

The First Obstacle: SPAs and JavaScript Rendering

When the server returns something like:

<html>
<body>
  <div id="root"></div>
  <script src="/app-3fa8c2.js"></script>
</body>
</html>

Your Net::HTTP.get returns exactly that: an empty div and a link to a 2 MB bundle. The real content — articles, prices, lists — is assembled in the browser by React, Vue, Angular, or the framework of the week.

For a script, this page is useless. To extract the content, you need to execute the JavaScript. That means running a real browser.

In the Ruby world, the answer is the ferrum gem — a CDP (Chrome DevTools Protocol) client that controls Chromium programmatically:

browser = Ferrum::Browser.new(headless: "new", timeout: 30)
browser.go_to("https://www.morningstar.com/markets")
browser.network.wait_for_idle  # wait for JS to finish loading
html = browser.body  # now the DOM is populated
browser.quit

That browser.body returns the HTML after the JavaScript has executed. The same data you see in the browser, now accessible to your parser.

But here’s where the headache begins: you need a Chromium binary available on the system. In Docker, that means adding ~300MB to the image. On the server, that’s one more dependency to manage. And the latency of opening a browser, navigating, waiting for JS to render, and extracting the HTML — we’re talking 3 to 10 seconds per page, instead of the 200ms of an HTTP GET.

The project uses this as the main strategy in ArticleFetcher: first it tries with a browser (to catch SPAs), then falls back to simple HTTP (to grab OpenGraph metadata when the browser fails). Most of the links that readers send in Discord are from sites that render with JavaScript, so browser-first pays off.

The Second Obstacle: Bot Detection

Opening a page with headless Chromium solves the JavaScript problem. But it solves 50% of the sites — not 100%. Because there’s an entire industry dedicated to distinguishing real browsers from automated browsers.

What Headless Chrome Reveals

Chromium in headless mode is almost identical to regular Chrome. Almost. The differences are subtle, but enough for detection:

1. navigator.webdriver — In any Chrome controlled by automation, this property returns true. In normal browsers, undefined. It’s the most obvious flag.

2. navigator.plugins — Normal Chrome has 3-5 plugins (PDF viewer, etc.). Headless returns an empty array. Bot detected.

3. navigator.userAgentData — The Client Hints API. Real Chrome returns brands, platform, mobile flag. Headless may not have that.

4. WebGL fingerprint — The WebGL renderer in a headless Chrome returns “SwiftShader” (software renderer). Real Chrome returns “NVIDIA GeForce” or “Intel Iris” or whatever. Bot detected.

5. Timing and behavior — Real browsers take time to load. They scroll. They move the mouse. They have focus events. A script that does go_to → body in 2 seconds doesn’t look like a human.

The solution is what I call StealthBrowser — a wrapper that injects JavaScript patches before any script on the page runs:

class StealthBrowser
  STEALTH_JS = <<~JS
    // Remove the automation flag
    Object.defineProperty(navigator, 'webdriver', { get: () => false });

    // Simulate real plugins
    Object.defineProperty(navigator, 'plugins', {
      get: () => [
        { name: 'Chrome PDF Plugin', filename: 'internal-pdf-viewer' },
        { name: 'Chrome PDF Viewer', filename: 'mhjfbmdgcfjbbpaeojofohoefgiehjai' },
        { name: 'Native Client', filename: 'internal-nacl-plugin' }
      ]
    });

    // Simulate a real WebGL renderer
    const getParameter = WebGLRenderingContext.prototype.getParameter;
    WebGLRenderingContext.prototype.getParameter = function(p) {
      if (p === 37445) return 'Google Inc. (NVIDIA)';
      if (p === 37446) return 'ANGLE (NVIDIA, GeForce GTX 1080 ...)';
      return getParameter.call(this, p);
    };

    // Client Hints API (absent in headless = bot signal)
    if (!navigator.userAgentData) {
      Object.defineProperty(navigator, 'userAgentData', {
        get: () => ({
          brands: [
            { brand: 'Chromium', version: '131' },
            { brand: 'Google Chrome', version: '131' }
          ],
          mobile: false, platform: 'Linux'
        })
      });
    }
  JS

  def self.fetch(url, name:, wait_for: nil)
    browser = Ferrum::Browser.new(headless: "new", timeout: 30,
      browser_options: {
        "disable-blink-features" => "AutomationControlled",
        "user-agent" => REAL_USER_AGENT
      })

    # Inject BEFORE any script on the page runs
    browser.page.command("Page.addScriptToEvaluateOnNewDocument", source: STEALTH_JS)

    browser.go_to(url)
    yield browser
  ensure
    browser&.quit
  end
end

The key trick is Page.addScriptToEvaluateOnNewDocument — a CDP command that registers JavaScript to be executed before any script on the page. That means when the bot detector runs, it sees the fake values we injected, not the real ones. And disable-blink-features=AutomationControlled removes the navigator.webdriver flag at the browser level, even before JavaScript.

This works for most sites. Morningstar, Hindenburg Research, generic news sites — they pass with the stealth. Yahoo Finance needs more — TLS fingerprinting that blocks even Chromium without crumb authentication. But the stealth browser handles the bulk.

The Third Obstacle: DataDome and Anti-Bot CDPs

And now we get to the pros. DataDome, Cloudflare Turnstile, PerimeterX, Akamai Bot Manager — services specialized in detecting automation. And they play at a completely different level.

Reuters, for example, uses DataDome. When I tried to hit reuters.com/world/ with StealthBrowser, I got back a 1,500-byte page with a JavaScript captcha. All the navigator.webdriver, WebGL, plugin patches — useless.

Why? Because DataDome doesn’t just check JavaScript properties. It detects the automation protocol itself.

The Fundamental Problem with CDP

When Ferrum (or Puppeteer, or Playwright) controls a Chrome, it does so via the Chrome DevTools Protocol — CDP. And the first thing Ferrum does when connecting is send Runtime.enable — a command that enables runtime evaluation in the browser.

The problem is that Runtime.enable changes the observable behavior of the browser. Specifically, it activates internal interceptors that can be detected via console.debug. DataDome uses this: it does a console.debug with a getter that watches whether the runtime is intercepting. If it is — and with CDP it always is — bot detected.

// What DataDome does (simplified):
let detected = false;
const el = document.createElement('div');
Object.defineProperty(el, 'id', {
  get: () => { detected = true; return 'test'; }
});
console.debug(el);
// If Runtime.enable is active, the getter is triggered
// In a normal browser, it isn't

This is not something you can patch with JavaScript. Runtime.enable is sent by Ferrum before any script runs — it’s part of the automation protocol. To fix this, you’d have to modify Ferrum to not send that command, which would break most of the gem’s functionality.

The Alternatives (and Why None Are Ideal)

I researched extensively. The options:

1. nodriver (Python): Uses Chrome via CDP without Runtime.enable. No equivalent in Ruby. And even so, it’s in a cat-and-mouse race with DataDome.

2. curl-impersonate: Does TLS fingerprinting identical to Chrome without opening a browser. Good for sites with TLS detection, useless for SPAs.

3. Scraping APIs (ScraperAPI, ZenRows): Outsource the problem. They work, but they cost money and add latency and dependency.

4. Fork Ferrum: Modify the gem to optionally not send Runtime.enable. Technically viable, but endless maintenance.

5. Google News RSS: Google has already read Reuters. The result is in a public RSS feed. No JavaScript, no DataDome, no browser.

Option 5 won. For Reuters, I traded all the headless browser complexity for:

class ReutersClient
  RSS_URL = "https://news.google.com/rss/search?q=when:24h+allinurl:reuters.com/world"

  def self.fetch_headlines(limit: 10)
    xml = Net::HTTP.get(URI(RSS_URL))
    doc = REXML::Document.new(xml)
    doc.get_elements("//item").first(limit).filter_map do |item|
      title = item.get_text("title")&.to_s&.strip
      url   = item.get_text("link")&.to_s&.strip
      next if title.blank? || url.blank?
      { title: title.sub(/\s*-\s*Reuters\s*\z/, ""), url: url, category: nil }
    end
  rescue StandardError => e
    Rails.logger.warn("ReutersClient failed: #{e.message}")
    []
  end
end

Simple. Reliable. Zero browser dependencies. The Google News link redirects to the original article when the reader clicks — works perfectly for a newsletter.

Not every scraping problem needs a scraping solution. Sometimes the data is already available in an accessible format — you just have to find it.

The Fourth Obstacle: Sites That Change Structure (Daily)

Even when scraping works, maintenance never stops. Morningstar is a textbook case — because it made me get it wrong twice before reaching a resilient solution.

Attempt 1: Select by DOM Hierarchy

Originally, the article links were inside <h2> and <h3>:

<h2><a href="/markets/article-slug">Article Title</a></h2>

One day later: zero results. Morningstar changed the structure to flat links with specific CSS classes — no heading elements:

<a class="mdc-basic-feed-item__mdc" href="/markets/article-slug">
  <header><h3>Article Title</h3></header>
  <div class="mdc-metadata__mdc">Author · Date</div>
</a>

Attempt 2: Select by CSS Class

The obvious fix: swap DOM hierarchy for CSS selectors based on the component class:

ARTICLE_CSS = "a.mdc-basic-feed-item__mdc, a.mdc-card--link__mdc"
doc.css(ARTICLE_CSS).each { |link| ... }

It worked. For exactly one day. Then: zero results again. Morningstar — like many modern sites with CSS-in-JS frameworks — generates class names dynamically. What was mdc-basic-feed-item__mdc became something else. And it may become something else tomorrow.

This is increasingly common. Sites using CSS Modules, styled-components, or Tailwind with JIT generate random hashes in class names as part of the build. Every time the site deploys, the classes change. Your hardcoded selector is a time bomb.

Attempt 3: Don’t Rely on Any Class Name

The final solution is to flip the logic: instead of telling the parser which classes to look for, let the parser discover which links are articles based on structural properties that don’t change:

1. Article links have long URLs: /news/why-stock-market-is-up-today-2026-02-17 vs. navigation links like /markets or /tools
2. Article links are grouped: they appear in lists, feeds, grids — multiple links with the same CSS class
3. Article links have descriptive titles: long text, not short labels like “Markets” or “Tools”

The algorithm:

def self.parse_html(html, limit)
  doc = Nokogiri::HTML(html)

  # 1. Collect all internal <a> with href, class, and enough text
  candidates = doc.css("a[href]").filter_map do |link|
    href = link["href"].to_s.strip
    css_class = link["class"].to_s.strip
    next if href.blank? || css_class.blank?
    next unless internal_href?(href)

    path = extract_path(href)
    segments = path.split("/").reject(&:empty?)
    next if segments.length < 2  # nav links have 1 segment: /markets

    title = extract_title(link)
    next if title.blank? || title.length < 10

    { title: title, url: absolutize(href), css_class: css_class, slug_length: path.length }
  end

  # 2. Group by CSS class
  groups = candidates.group_by { |c| c[:css_class] }

  # 3. Filter: minimum 3 links in the group (article clusters, not stray links)
  # 4. Pick the group with the highest average slug length
  best = groups
    .select { |_cls, items| items.length >= 3 }
    .max_by { |_cls, items| items.sum { |i| i[:slug_length] }.to_f / items.length }

  return [] unless best

  best[1].map { |c| c.slice(:title, :url) }.uniq { |a| a[:url] }.first(limit)
end

What saves this approach: no class name is hardcoded. Morningstar can rename every class tomorrow and the parser will keep working — because it discovers the article pattern through structure, not through names.

What’s stable on a site:

• Article URLs are always longer than navigation URLs (that’s architectural, not cosmetic)
• Articles are always grouped under the same component (that’s UX, not implementation)
• Article titles are descriptive (that’s editorial, not technical)

What’s not stable:

• CSS class names (change on every build)
• DOM hierarchy (changes on every redesign)
• data-* attributes (change as the frontend team decides)

Bottom line: don’t try to identify elements by name — identify them by behavior. Article links behave differently from navigation links in measurable ways (URL length, clustering, title), and those properties are inherent to what the element IS, not to how it was implemented.

The Reliability Hierarchy

After hours of banging my head, a hierarchy of data sources became obvious:

The rule: always prefer the simplest source that works. If the site has RSS, use RSS. If it has an API, use the API. Only go to the headless browser when there’s no alternative.

In the project, the distribution ended up like this:

Simple HTTP (Net::HTTP or HTTParty):

• Google News RSS (Reuters world headlines)
• Hacker News RSS (top stories)
• HackerNoon RSS (top stories)
• dev.to API (trending articles)
• Jikan API (anime rankings)
• YouTube Data API (videos)
• Yahoo Finance news RSS

Headless browser (StealthBrowser + Ferrum):

• Morningstar (SPA + dynamic components)
• Yahoo Finance charts (TLS fingerprinting + crumb auth)
• FinViz market map (canvas rendering)
• Generic reader articles (any URL could be an SPA)
• Hindenburg Research (JavaScript-rendered)

In other words: most data sources work without a browser. The browser is needed for a handful of problem sites — but those sites are the ones that cause the most trouble.

The Real Cost

Let’s be practical about what “packing a Chrome” means:

Docker image: +300MB. Chromium and its dependencies (fonts, graphics libs, sandbox) are huge. In an optimized Dockerfile with multi-stage build:

# Install Chromium and minimal dependencies
RUN apt-get install -y chromium fonts-liberation libnss3 libatk-bridge2.0-0 \
    libdrm2 libxkbcommon0 libgbm1

Memory: Each Chromium instance consumes 200-400MB of RAM. If you have 3 parallel workers, that’s 1.2GB just for browsers. On a 2GB VPS, that’s 60% of the memory.

Latency: A page that takes 200ms via HTTP takes 3-8 seconds via headless browser (process boot, navigation, JS execution, network idle). For 10 data sources, that’s 30-80 seconds instead of 2.

Stability: Chromium can crash, hang, or leak memory. A 30-second timeout with cleanup in ensure is mandatory:

def self.fetch(url, name:)
  browser = create(name: name)
  browser.go_to(url)
  yield browser
ensure
  browser&.quit  # ALWAYS quit, even on exception
end

If you don’t quit, the Chromium process stays hanging around burning CPU and memory. I’ve had a server with 15 zombie Chrome processes eating 3GB of RAM because an ensure was missing.

The Future (Or: It’s Going to Get Worse)

The trend is clear: sites are getting harder to access programmatically. Every quarter, the DataDomes and Cloudflares of the world add new detection techniques. Headless Chrome gets more similar to real Chrome — but the detectors keep up.

The web that was born open and machine-readable is becoming a walled garden. Even RSS, which is trivial to implement and useful to everyone, is being abandoned by many sites — because they don’t want their data read outside the context (and the ads) of the original site.

For anyone building systems that aggregate information, the playbook is:

1. Use APIs and feeds whenever they exist. More stable, faster, more respectful
2. Keep a stealth browser for SPAs that have no alternative
3. Never hardcode CSS selectors. Sites with CSS-in-JS change class names on every deploy. Identify elements by structural properties (URL length, clustering, title size) that survive redesigns
4. Detect blocks and fail gracefully — og:description as fallback, title extracted from the URL as a last resort
5. Never depend on a single access method. The site that works today may block tomorrow
6. Accept that some sites are inaccessible. If DataDome won’t allow it, the data will have to come from another source

The web in 2026 is a battlefield between those who want to read and those who want to control the reading. But in the long run, the creativity of those who want to access data wins — but it costs more effort every year. And, contrary to what I thought at the beginning: it’s never simple.

Now, you may say “another Ruby fanboy.” That’s fair, but a terrible argument, as it’s a poor and common argumentum ad hominem. And on the subject of fallacies, the click-bait article above is wrong exactly because it falls for a blatant Post hoc ergo propter hoc fallacy plus some more confirmation bias which we are all guilty of falling for all the time.

I’m not saying that the author wrote fallacies on purpose. Unfortunately, it’s just too easy to fall for fallacies. Especially when everybody has an intrinsic desire to confirm one’s biases. Even trying to be careful, I end up doing that as well.

On the specific subject of Rails, start here:

• Rails has won: The Elephant in the Room
• The Ruby Community and Reputation
• Programmers Guild - Religion and Sports

As a general rule, anyone announcing the future demise of anything is most probably wrong. It’s fun to play guessing games, but articles in high profile websites like TNW should be more careful as many beginners in the field end up trusting their opinions.

Which is why I usually tell people to stop trusting opinions. Everybody has an opinion, which is why it’s so cheap. Then people artificially grade “celebrities” or “gurus” as having opinions of higher value. I don’t buy it, neither should you.

Black Swans do happen. But it’s an exercise in futility to try to predict them. You may be lucky, once or twice. But it’s useless to attempt. Which is why I am a high proponent of Antifragility, so I will give you the general rules to understand what to do.

Am I saying that Ruby on Rails will live on forever? Don’t be absurd, of course not. Nothing lasts forever. More than 2 decades actively working in the field taught me that betting everything on one horse is a mistake. It’s valid in the financial markets, it’s valid in the tech market.

Instead of stock or bonds, your asset is your knowledge and your practice. Diversification of your portfolio is usually a better bet.

You invested all your 401k in Bitcoins. Now you will try everything to try to confirm your bias by subscribing to every news piece and opinion that confirms your bias, and you will denounce every article that may have an argument against your bet. Am I saying to not invest in Bitcoins? Hell no. Just don’t be stupid and invest everything on anything.

You invested all your time learning Google Web Toolkit in 2006. Good luck for you finding an article to confirm that bias. This is one case, an after-the-fact, that we can’t do much but consensually agree on its death.

The job of a proper professional programmer is not mastery of a single tool or even a single technology stack. Your job is to be as good as possible in learning and practicing in the technology field.

Having said all that, and even though the article’s structure is inaccurate, The feeling is not wrong. Ruby on Rails no longer represents what it was 10 years ago. Now, before you start freaking out, let’s explore it down memory lane.

2003 - A New Renaissance

As an after-the-fact exercise, one has to realize that a series of events helped unfold the Rails phenomenon between 2003 and 2011. It was not by design. It was not planned. It could’ve been something else.

• We were just recovering from the 2001 Internet dot-com Bubble Crash.
• Social networks were starting to emerge, we had Orkut, MySpace, Friendster. Still no Facebook on the radar.
• Blogs were starting to pick up steam and competing with traditional news outlets.
• Big vendors focused solely on the Enterprise market at that point and the rest were deemed amateurish.
• In the mainstream either you were doing .NET or Java on the Enterprise or doing hotsites in Flash and PHP for ad hoc agencies.
• Even open source tools like Eclipse were primarily used through enterprisey umbrellas like WebSphere or WebLogic.
• The Agile Manifesto had just been published in 2001 and the mainstream market didn’t pay much attention. PMI and RUP were the order of the day.

Then we reached 2004.

• Google successfully launched Gmail to much fanfare. Dynamic HTML and Ajax demoed how we could achieve Desktop-level applications in the Web. This is also where Javascript earns a new role beyond stupid web mini games or ads.
• Facebook skyrocketed and we were shown glimpses of what granular social networks look like.
• Apple consolidated its comeback as a relevant tech giant with the successful resurrection of OS X, proving that Unix on the consumer desktop was viable.
• Microsoft entered the 21st century after a super heavy antitrust case that almost obliterated the company. Bill Gates resigned, Steve Ballmer took the reins and they had their darkest decade
• Years of vendor lock-in finally started to wear off developers, no one actually wanted to use IBM, BEA, Oracle anymore and the promise of the open source revolution started in the 90’s finally started to solidify
• The dot-com crash left a depression in the tech market but it also left behind an infrastructure that was finally being put to good use

Ruby on Rails shows up as a very small curiosity, in an obscure tech conference in Brazil, created by a small but promising agency called 37signals. Signaling something most developers had never seen before: a flexible language - unlike Java or C# or PHP - that could be used to implement Gmail-like web applications with a flavor of the Apple vision and steering away to everything that Microsoft and IBM represented while incorporating Agile-like techniques such as Test-Driven Development in a practical way and out-of-the-box.

What else could you ask for in 2005?

Then 2006 happens: Amazon AWS releases S3 and EC2 and the term “Elastic Cloud” becomes the new hype.

Up to that point, “web development” either meant expensive and bureaucratic IT departments in big corporations or inexpensive but weak shared hosting in services such as Dreamhost, 1&1, Media Temple.

Elastic Cloud holds the key to surpass the enterprise and make it possible for independent developers to build the elusive “web scale”.

Ruby on Rails is exactly where it should be: starting to be adopted by a new crop of Silicon Valley tech start-ups trying to make a dent in the universe in this new Ajax-based Cloud-era. Remember that Rails was first released with RJS, a way to write Javascript through Ruby using Prototype and Scriptaculous. jQuery was not even mainstream yet.

Twitter shows up. Groupon shows up. Engine Yard shows up. And much more follow.

It’s all happening blistering fast, bear in mind that I am only touching the surface here.

2007 arrives and at least 3 big things happen:

• Another Rails-based start-up is born: Heroku. If AWS EC2 coined the term “Infrastructure as a Service” (IaaS) Heroku defined what a “Platform as Service” (PaaS) means. It was able to put EC2 to good use and make its power approachable.
• At the same time, developers have grown bitter of version control systems such as CVS and again, big vendor options such as Rational ClearCase, Microsoft SourceSafe, and others were just terrible. Subversion had emerged but its many shortcomings would become its demise. Linus Torvalds decides to fix the problem and release Git. Mainstream developers finally learn basic computer science terms such as DAGs (“Directed Acyclic Graphs”).
• The iPhone is released and a new mobile revolution takes place. Dominant players like Nokia and Blackberry were shaken to their core. Developers want a piece of it. Mobile Safari becomes high profile overnight. And its WebKit core rises to stardom, eclipsing old veterans like Firefox/Gecko and Opera.

2008: the birth of GitHub and the dawn of the DevOps.

Git was great but too low level. Several other contenders tried to become the new standard such as Mercurial and Bazaar. But the birth of GitHub would change all that. Again, another Rails-based start-up making the news.

But the honeymoon is showing signs of reaching an end: prominent Ruby developer, Zed Shaw, loudly leaves the community by January.

During this year we also had the cold war between Engine Yard’s Merb framework and 37signals’ Rails, resulting in the controversial Rails-Merb merge, placing another crack in the community.

But we still had a lot of work to do, so we moved on.

2009: “there’s an app for that!”

Apple finally releases a proper native SDK. Developers start building a new generation of productivity apps and games. WebKit starts growing in market share. Apple solidifies the concept of aesthetic in software development and Ruby on Rails is at its high at that point.

We begin the NoSQL era. When data storage and fetch start to really become a bottleneck, we quickly learn that traditional enterprisey databases are not “web scale” enough. MongoDB leads the pack. Web development starts an era of craze over NoSQL.

In parallel, Twitter, once the poster-child of the Rails generation, creates a crack in the Ruby community when it declares that it’s started to use yet another obscure language: Scala!

This starts the endless obsession era about “concurrency”, yet another elusive concept that many talk big and talk a lot but only a few actually grasp its meaning. It’s the start of the window of opportunity for Node.js, Scala, Erlang, Clojure, Go.

The honeymoon is over.

2010:

• The demise of Sun and Oracle as the new stewardship for Java. If things were not playing out cool for Java, that was a heavy nail in the coffin.
• Apple declares open war against Flash. Another big push for Javascript and front-end heavy web apps.

In my personal opinion, this is the year when Rails reached its peak in terms of influence. It was the go-to stack for developers in Macs wanting to create their brand new Lean Start-up.

2011: A New Era

In around 6 years the software development market did titanic shifts:

• Distributed development derived from years of Open Source workflow becomes the new standard. Git and GitHub were key players. Big vendors have to play catch-up with their stinky enterprise alternatives such as TFS.
• Agile is taken seriously in a new crop of tech startups evangelizing proper software development. Tests shift from unproductive obnoxious things into invaluable core technique. Tools like Hudson/Jenkins combined with Git workflow pave the way for what continuous integration and later continuous deployment would become.
• AWS and the Heroku example pave the way to make old IT obsolete and create a new generation of infrastructure as software, and “DevOps”. There is now the urge to deploy super web scale with zero friction. Time to market becomes a key factor: release early, release often.
• Apple releases the AppStore and helps drive the Software as Service market, with apps requiring live internet to function. The “Cloud” reaches the masses.
• By making everybody on Earth carry an always-on powerful device, a smartphone, we were never so close to the Orwellian future of 1984. Everything a person does is a data point. Big Data becomes a thing. Data Sciences becomes viable as a mainstream profession. I.A. related disciplines become a necessity to reason about so much data.

Android is getting closer to the iPhone. Microsoft is resurrecting from its ashes. In October, Steve Jobs dies, marking the end of the era of Apple’s heavy influence.

But around 2011 another thing happened: we win.

Independent developers surpassed the status quo of enterprise developers. There is a shift of power. The conventions for the software development field don’t come from the old big tool vendors anymore. We are not listening to IDE vendors like Microsoft, BEA, Oracle. We are now listening to the winners of this generation of startups: Apple, Google, Facebook.

The concept of “one language to rule them all” was shattered in the web camp by Ruby on Rails and in the native application space by Apple. 2 very unusual and strange languages, Ruby and Objective-C, reached mainstream status. They proved that developers can learn and master exotic languages turning them into profitable products.

With those foundations set, a new gold rush emerges: new languages galore, free from the notion that to become successful one had to stick to Java, C#, or C++ like languages, people finally started looking elsewhere to differentiate themselves, and thus a new generation of exotic languages arrive: Scala, Haskell, Ocaml, F#, Erlang, Clojure, Rust, Elixir, etc. The sky is the limit.

By killing Flash and accelerating HTML 5, ES6, and with all the controversies around concurrency, Javascript gained new life and fast adoption unlike nothing else before.

Epilogue

Is Ruby on Rails dead? No.

Is Ruby on Rails still the leader of the pack? No.

What is the legacy of the Ruby on Rails Generation? It played an important role in defining what we now consider the new status quo for web development: Git, CI, CD, Cloud, Metrics. It brought the Apple aesthetics thinking to the table in a big way.

That new crop of tech startups had a serious side-effect though: it put new engineers in the spotlight, elevating them to rock star levels and turning them into Steve Jobs wannabes. They just started taking everything we accomplished for granted. Recipe for disaster.

And while the big vendors’ influence became thin compared to the 90’s, they are not dead. After 2011 we ushered in a new era of chaos, an era of dissatisfied developers. Where every couple of years a new derivation emerges and we have that unsatisfying feeling that we need to rewrite everything all the time.

The Ruby on Rails pool got diluted into smaller niche languages such as Javascript itself, Go, Elixir, Scala.

The Java/Microsoft mainstream is gaining new traction. Aesthetics also got diluted. Careful engineering is getting diluted. Social networks ubiquity and the easy path of quick cloud deployment is making people less and less patient. Crafting requires patience. People crave Likes, a new generation of vanity metrics arose.

We now have a larger number of independent non-enterprise developers with little to no patience, with an exaggerated sense of confidence, and not-invented-here syndrome, taking everything for granted and caring less and less about proper value.

In conclusion, the death of Ruby on Rails is indeed greatly exaggerated, it will keep going for many years still - especially if the Ruby Core Team can actually deliver on the 3x3 goal. People like to share doomsday articles while optimistic and more realistic reports such as this Why Ruby on Rails is still the best choice? are largely ignored. Such is the way of social networks vanity.

But it will not be what it once was because the times have changed. We can now live by the “On Rails” workflow without Ruby on Rails. And it can’t go on breaking the status quo, because it became the status quo.

I wonder where that will lead us. I do have a few clues, but as I said, I am not in the game of publishing click-bait headlines.

That happens to old blog posts, including my own as I’ve been posting for several years.

In this post I will fix what I posted (in pt-BR):

• Direct Upload para S3: a Solução Definitiva!
• S3 Direct Upload + Carrierwave + Sidekiq
• Upload de Arquivos
• Enciclopédia do Heroku

They are still good posts to understand the mechanics of what I will propose here, so you should read them if you’re interested in how things work.

TL;DR is: use Cloudinary (with Attachinary) and install a CDN (Cloudfront) right now as it’s super trivial to do.

Uploading files is not a task to be taken lightly. It’s one of those tasks that you must not consider that it “works” just because it works in your development machine, for a few trivial uploads. Production level behavior changes drastically and it can bring your entire project down depending on how heavily it depends on user-generated content.

There are a lot of moving parts to understand. But in the most basic solutions you just create an HTML multipart form with a file input field and submit it to some controller endpoint directly. This controller receives the blob binary and you save it to some local directory in the server.

This is the worst possible solution and unfortunately this is the one you will find the most across the web.

There are many problems to deal with this simple implementation:

• We are in the era of 12+ megapixel cameras. Those are multi-megabyte sized files. Each upload will take a long time, especially if the user is trying to upload using unreliable or slow wireless networks.

Once the user’s browser connects to the web application it will hold that connection and block the entire process through the duration of the request-response cycle. Making this simple to understand, if you only have 1 single process in the server, it will be unable to respond to any other request until the upload finishes.

Fortunately most deployments will use NGINX in front of the web application - as they should - so this effect is minimized as NGINX will receive the entire blob before passing it to the web application process underneath, making it not so problematic in most cases.

But if you’re on Heroku, the router layer has a 30 seconds timeout limit. If the upload takes more than that (which is common with users in unreliable networks) this layer will cut the connection down before it finishes. Users will retry and this can cause your request queues to grow very fast.

Avoid uploading anything to Heroku. I consider this a feature as it forces applications deployed on Heroku to use good practices as I will explain below.

• You should also NOT save files to the local filesystem if you want to scale horizontally, as one server will not be able to see what’s in the other server unless you’re in AWS EC2 or other IaaS, with shared mounted volumes, for example. In Heroku, you can’t rely on anything in the filesystem as the machines are volatile and whenever a dyno restarts it loses whatever was not there during the deployment.

So you must upload what you received to an external storage, such as AWS S3, and this is also slow if you receive lots of uploaded files. You can rely on NGINX or Heroku’s router layer to handle the heavy receiving of the files but you will make the request-response cycle super slow if you synchronously process and upload the files to S3.

The next option you think of is to add Sidekiq or some other Async Job mechanism to do this other heavy lifting. Now you have to create front-end placeholders to show to the users while the background job is not finished sending the files to S3.

The last option, if you dig deeper, is browser-based direct uploads to S3 and just posting the URLs to your web application. This is the ideal solution, but it’s not easy to implement in your stack.

• Once you nail everything above, you still make the mistake of putting the S3 URLs directly in the IMG tags in your HTML, which is not recommended by Amazon as S3 is recommended just for “storage” purposes. This is the easy part to fix, but most projects forget about that.

Cloudinary or Carrierwave Direct

In my previous posts I explained each of the above issues in more detail and I offer some (complicated) solutions, such as using the Refile project. At that time the Carrierwave plugins for Direct Upload were not ready.

Nowadays the solution is actually super easy and this is what I recommend: go straight to Cloudinary. This is a Software as a Service solution for proper direct uploads and dynamic image processing. You should definitely explore the Attachinary gem to make the process easier.

If you have legacy Carrierwave, Paperclip, Shrine, Dragonfly, Refile or others, one jerry-rigged - but reasonable - alternative is to just add Attachinary to the mix.

Let’s say you have an old user.avatar uploader. You can just create a new user.new_avatar with Attachinary/Cloudinary. In the HTML form upload you start using just the Attachinary helpers. And in the HTML where you show the image itself, you can make a helper to check for @user.new_avatar? and show using cl_image_tag(@user.avatar.path instead, otherwise you show the old URLs.

You don’t need to totally migrate everything to Cloudinary, just keep the old assets where they are and start putting the new assets in the Cloudinary configuration. It has a Free tier that can hold up to 75 thousand images and allows traffic of up to 5GB. And with just USD 44 a month you have 10 times that. So it’s really cheap if your application is serious about file uploading.

If you don’t want to commit to an external service yet, and you’re using Carrierwave, another option you must consider is testing out CarrierwaveDirect to test direct uploading to S3 from the browser. This will at least avoid the complexities of setting up async uploads from the web application to S3.

Update: Janko Marohnić posted a very good comment that I think is important to quote directly:

Great writeup, I wholeheartedly agree that we probably don’t “just want something simple”, especially concerning synchronous uploads and CDNs.

Shrine actually has a Cloudinary integration, and it’s pretty advanced too; it supports direct uploads, setting upload options, storing Cloudinary metadata (hello responsive breakpoints) etc :). Also, the Cloudinary gem ships with a CarrierWave integration. I think it’s beneficial to use Shrine or CarrierWave with Cloudinary, because then you get to keep your file attachments library, and just change the underlying storage service.

As for CarrierwaveDirect, I don’t think it’s actually very useful, because backgrounding you need to do all by yourself anyway, so basically all it gives you is the ability to generate a direct-upload form to S3. But generating request parameters and URL to S3 is something that’s already built in into the official aws-sdk gem, CarrierwaveDirect just seems to be reimplementing that logic. And the advantage of generating this through aws-sdk is that it’s not HTML-specific, so you could setup an endpoint which returns this information as JSON, and now you have your multiple file uploads directly to S3 :)

As a disclaimer I didn’t test CarrierwaveDirect myself and I remember being quite broken 2 years ago. And I’ve heard good things about Shrine along the way but never did proper testing. I’d recommend giving it a try if you want to migrate away from Carrierwave.

Don’t serve assets from S3, use a CDN, any CDN

If you’re sticking to S3 uploading, at the very least avoid serving those files from S3. In my “Heroku Encyclopedia” post I recommend using “S3 Assets Sync”. DO NOT DO IT: use a CDN instead, it’s faster, easier and the correct solution.

The first thing you want to do is sign up to AWS’s Cloudfront CDN service. Always use a CDN for all your assets, not only the ones internal to your web application but also every user uploaded content. It’s super simple to create a new CDN endpoint pointing to your S3 bucket.

Carrierwave will default to the S3 bucket host, but once you have the Cloudfront endpoint you can easily change all the uploaded files URLs like this:

# ./config/initializers/carrierwave.rb
CarrierWave.configure do |config|
  config.fog_credentials = {
    :provider => "AWS",
    :aws_access_key_id => ENV['AWS_ACCESS_KEY_ID'],
    :aws_secret_access_key => ENV['AWS_SECRET_ACCESS_KEY'],
  }
  config.fog_directory = ENV['S3_BUCKET']

  # use only one of the following 2 settings
  # config.fog_host = "http://#{ENV['S3_BUCKET']}.s3.amazonaws.com"
  config.fog_host = ENV['S3_CDN'] # for cloudfront
end

You must obviously configure the environment variable S3_CDN to point to the Cloudfront endpoint specific to the particular S3 bucket you’re using.

The important part is: serve ALL ASSETS from a CDN, no matter what. Never directly from your web application, avoid directly serving from S3. It’s almost trivial to implement and if you don’t like AWS for any reason, you can choose Fastly, Cloudflare and many others.

They all work the same way. There is NO excuse to not use a CDN no matter the size of your application.

If you’ve been following the Elixir blogosphere, chances are that you stumbled upon Brandon Richey’s take on the classic “15 minute blog”. If you didn’t you must read at least Part 1 and Part 2. It’s a very detailed tutorial that will make it easier to get your feet wet with Phoenix goodness.

This post is intended for Rails programmers wanting to know how the Phoenix Framework compares. It’s not a completely fair comparison as this is just the good old nested resources “hello world” exercise. This will only scratch the surface but should serve as a good introduction.

For better or for worse, many still consider Ruby on Rails to be the best DSL for a web application. Rails was successful in creating a very recognizable vocabulary to describe each component of a web project. And I will argue that one of Phoenix’s strengths is to successfully borrow the same metaphor. This definitely makes the learning curve much smoother.

If you just want to clone the exercises and jump right into the code itself, I have the original Pxblog in Phoenix and the comparison Pxblog in Rails.

Getting Started: command line and folder structure

Without further ado, let’s start comparing the basic console commands:

rails new pxblog
rails g scaffold Post title:string body:string
rails g scaffold User username:string email:string password_digest:string
rake db:create
rake db:migrate
rails g migration AddUserIdToPosts
rails server

mix phoenix.new pxblog
mix phoenix.gen.html Post posts title:string body:string
mix phoenix.gen.html User users username:string email:string password_digest:string
mix ecto.create
mix ecto.migrate
mix ecto.gen.migration add_user_id_to_posts
mix phoenix.server
# iex -S mix phoenix.server to start within IEx and be able to IEx.pry

Right off the bat we feel at home. In the Rails world we have both the rails command competing with traditional rake task managers. At the Phoenix side, they fortunately concentrated everything under the Elixir built-in mix command. There seem to be discussions about moving the Rails command tasks to Rake where they belong, but it’s not coming soon.

As it is all under Mix territory, you can list Phoenix-related tasks like this:

$ mix help | grep -i phoenix

mix phoenix.digest      # Digests and compress static files
mix phoenix.gen.channel # Generates a Phoenix channel
mix phoenix.gen.html    # Generates controller, model and views for an HTML based resource
mix phoenix.gen.json    # Generates a controller and model for a JSON based resource
mix phoenix.gen.model   # Generates an Ecto model
mix phoenix.gen.secret  # Generates a secret
mix phoenix.new         # Create a new Phoenix v1.0.3 application
mix phoenix.routes      # Prints all routes
mix phoenix.server      # Starts applications and their servers

The directory structure that phoenix.new generates is slightly different from Rails though:

/_build              # binary stuff mix compiles (ignore)
/config
- config.exs         # think Rails' /config/application.rb
- dev.exs            # think Rails' /config/environments/development.rb
- prod.exs           # think Rails' /config/environments/production.rb
- prod.secret.exs    # think Rails' /config/secrets.yml
- test.exs           # think Rails' /config/environments/test.rb
/deps                # where mix deps.get put dependencies
/lib
  /pxblog
    - endpoint.ex    # a bit like config.ru and application.rb
    - repo.ex        # setup for Ecto
  - pxblog.ex        # where you can set up the OTP apps supervision tree 
/node_modules        # Phoenix integrates with Node.js
/priv
  /repo
    /migrations      # think Rails' /db/migrate
    - seeds.exs      # think Rails' /db/seeds.rb
  /static            # think Rails' /public
    /css
    /images
    /js
/test                # think Rails' /test
  /channels
  /controllers
  /models
  /support
  /views
  - test_helper.exs  # think Rails' /test/test_helper.rb
/web                 # think Rails' /app
  /channels          # think Rails 5's ActionCable channels
  /controllers
  /models
  /static            # think Rails' /app/assets
    /assets
      /images
      /css
      /js
      /vendor
  /templates         # think Rails' /app/views
    /layout
  /views             # think Rails' /app/helpers but with Presenters
    - layout_view.ex
  - router.ex        # think Rails' /config/routes.rb
  - web.ex           # macros to configure each MVC component
- .gitignore
- README.md
- brunch-config.js  # front-end dev reloading is controlled with Brunch
- mix.exs           # think Ruby's Gemfile (with extras)
- package.json      # Node.js dependencies

In Rails, the starting point is the config.ru Rackup application (as Rails has been a Rack app since 3.0). It then loads the config/environment.rb, then config/application.rb, then config/boot.rb which loads the gems declared in the Gemfile, together with config/initializers/*.rb and each file in config/environments, where we set up the Rails::Application and plug in configuration and the pipeline of Rack Middlewares.

In Phoenix, the starting point - as in any Elixir app – is the mix.exs file. In this case it points to the “Pxblog” OTP/Phoenix app defined in lib/pxblog/pxblog.ex. In turn it starts up and supervises the “Pxblog.Endpoint” and “Pxblog.Repo” apps, which are defined in lib/pxblog/endpoint.ex and lib/pxblog/repo.ex, respectively.

If you build other OTP apps, this is where you can add to the OTP Supervisor Tree. I could say that it’s something akin to a Rails Engine, although technically it is not the same thing, but this metaphor will do.

One main aspect of Rails is how it divides development, test, and production configuration in different files. Phoenix has the same thing at dev.exs, test.exs, and prod.exs. This is actually a Mix feature. Mix is a more accomplished version of Rake, and it makes sense, as José Valim also tried to push Thor to replace Rake, although it never picked up steam in the Ruby community (Rake being so ingrained everywhere). Web frameworks that don’t enforce separation of environments by default, in this day and age, are useless. The cool thing is that every Elixir app generated through Mix gets this same useful feature.

Instead of having a database.yml, the database configuration is spread through the environment configuration files and the production settings are in a separate prod.secret.exs file, which is obviously ignored in .gitignore, like Rails’ secrets.yml file.

MVC structure

You will see that each element of the MVC app in Phoenix starts like this:

# web/controllers/page_controller.ex
defmodule Pxblog.PageController do
  use Pxblog.Web, :controller
  ...
end
# web/models/post.ex
defmodule Pxblog.Post do
  use Pxblog.Web, :model
  ...
end
# web/views/post_view.ex
defmodule Pxblog.PostView do
  use Pxblog.Web, :view
end
# web/router.ex
defmodule Pxblog.Router do
  use Pxblog.Web, :router
  ...
end

This Pxblog.Web module is defined in the web/web.ex like this:

# web/web.ex
defmodule Pxblog.Web do
  def model do
    quote do
      use Ecto.Model

      import Ecto.Changeset
      import Ecto.Query, only: [from: 1, from: 2]
    end
  end

  def controller do ...

  def view do ...

  def router do ...

  def channel do ...

  @doc """
  When used, dispatch to the appropriate controller/view/etc.
  """
  defmacro __using__(which) when is_atom(which) do
    apply(__MODULE__, which, [])
  end
end

If you haven’t yet, this is a good time to learn about Elixir Macros. Think of the code in the quote block as being “injected” in each module that calls use Pxblog.Web. When you use a module it calls the using macro. Think of it like a Ruby Module Mixin calling the included callback and executing a class_eval. As there is no concept of Classes and subclasses, we include mixins in each module to acquire the desired behaviours.

MVC Router

Unlike Rails’ config/routes.rb, which defines routes, the web/router.ex defines not only the routes themselves but also transformation pipelines and routing strategies:

# web/router.ex
defmodule Pxblog.Router do
  use Pxblog.Web, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", Pxblog do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
    resources "/users", UserController do
      resources "/posts", PostController
    end
    resources "/sessions", SessionController, only: [:new, :create, :delete]
  end

  # Other scopes may use custom stacks.
  # scope "/api", Pxblog do
  #   pipe_through :api
  # end
end

You have to read it like this:

The pipeline block plugs “filters”. They are similar to Rack middlewares that we define in config/application.rb in Rails. This is very clever because one key difference between vanilla Ruby on Rails and the Rails-API project is the removal of unneeded Rack middlewares that API endpoints don’t need.

In Phoenix we can define one pipeline for web browsers and another for API clients, and you can see that the difference is the removal of Plugs.

Then we define the scopes based on the root paths. There is one scope “/” which connects to the :browser pipeline and an optional (commented out) scope “/api” which pipes through the :api pipeline.

Inside the scope block, it’s very similar to the RESTful DSL used to define routes, which again is a good adaptation from Rails Routes. Read the documentation to know the details.

Also similar to Rails Routes, it generates proper URL helpers that become available in Controllers, Views and Templates. Let’s start by seeing some Rails URL helpers in code:

<!-- app/views/posts/edit.html.erb -->
<%= link_to 'Show', [@user, post] %>
<%= link_to 'Edit', edit_user_post_path(@user, post) %>
<%= link_to 'Destroy', [@user, post], method: :delete, data: { confirm: 'Are you sure?' } %>

And in Phoenix:

<!-- web/templates/post/edit.html.eex -->
<%= link "Show", to: user_post_path(@conn, :show, @user, post) %>
<%= link "Edit", to: user_post_path(@conn, :edit, @user, post) %>
<%= link "Delete", to: user_post_path(@conn, :delete, @user, post), method: :delete, data: [confirm: "Are you sure?"] %>

The main URL helper in Rails is able to take an Array such as [@user, post] and execute it the same way as if we wrote user_post_path(@user, post). One difference from Rails is that instead of creating one helper for each HTTP verb you have just one helper per resource that accepts an extra parameter to indicate the verb. So we have post_path(@conn, :edit, post) instead of the Rails way of edit_post_path(post).

As with Rails middlewares, a pipeline receives the request connection and pipes it through, transforming its metadata so it is cleaned up and usable within our controllers.

MVC Controller

In Phoenix we start a controller like this:

# web/controllers/post_controller.ex
defmodule Pxblog.PostController do
  use Pxblog.Web, :controller
  
  alias Pxblog.Post
  
  plug :scrub_params, "post" when action in [:create, :update]
  plug :assign_user
  plug :authorize_user when action in [:new, :create, :update, :edit, :delete]

It’s similar to this Rails controller setup:

# app/controllers/posts_controller.rb
class PostsController < ApplicationController
  before_action :assign_user
  before_action :authorize_user, only: [:new, :create, :update, :edit, :destroy]

As you may have concluded, a plug call works a bit like a before_action pipeline. The scrub_params I believe is similar to Rails’ ActionDispatch::ParamParser, but I’m not sure; I know it clears out empty string values to nils so you don’t update your models unnecessarily.

But unlike Rails, where a call to redirect_to halts the pipeline, we need to explicitly halt the pipeline like this:

defp assign_user(conn, _) do
  %{"user_id" => user_id} = conn.params
  if user = Repo.get(Pxblog.User, user_id) do
    assign(conn, :user, user)
  else
    conn
    |> put_flash(:error, "Invalid user!")
    |> redirect(to: page_path(conn, :index))
    |> halt()
  end
end

In Phoenix, everything revolves around a request connection transformation pipeline that you can start configuring in the Router Plugs, Controller Plugs and Controller actions. All of them receive the connection from the previous step and return a transformed connection to the next step until it becomes a proper HTTP response. Unlike Rails, this path is much more explicit and you know that you will get this connection and you should pipe transformations through it until you render the final HTML or send back some error header.

To see how explicit, let’s start with a normal Rails controller action:

def destroy
  @user = User.find(params[:id])
  @user.destroy
  respond_to do |format|
    format.html { redirect_to users_url, notice: 'User was successfully destroyed.' }
    format.json { head :no_content }
  end
end

Once upon a time, setting the flash notice message and redirecting were 2 different methods; newer versions merged them together for convenience. Rails also has the concept of Responders, which Phoenix doesn’t have yet (#OpportunityToContribute!).

def delete(conn, %{"id" => id}) do
  user = Repo.get!(User, id)
  Repo.delete!(user)
  conn
  |> put_flash(:info, "User deleted successfully.")
  |> redirect(to: user_path(conn, :index))
end

You can see that, Responders aside, the Phoenix version is remarkably similar. And it goes like this for all RESTful actions of the controller. But more similar to the deceased Merb, each Phoenix Controller action has a proper function signature declaring the parameters it expects to receive, instead of having a global params hash that needs to go through Strong Parameters.

For different variations of the same parameters, you can declare multiple functions with the same name but different arguments to pattern match, like in the SessionController example:

# web/controllers/session_controller.ex
defmodule Pxblog.SessionController do
  use Pxblog.Web, :controller
  ...
  def create(conn, %{"user" => %{"username" => username, "password" => password}}) when not is_nil(username) and not is_nil(password) do
    user = Repo.get_by(User, username: username)
    sign_in(user, password, conn)
  end

  def create(conn, _) do
    failed_login(conn)
  end
  ...
end

Here we have the same create/2 function with pattern matching and guards. The second version receives anything, in case the first version fails to pattern match against the incoming parameters. Phoenix expects roughly the same structure of parameters as Rails, so it’s very intuitive to follow.

MVC Models

Instead of the good old ActiveRecord (ActiveModel), in Phoenix we have to deal with Ecto Models. It already supports PostgreSQL, MySQL, SQLite3, MongoDB, so you’re good to go for 99% of cases.

Ecto separates Model Logic from Model Persistence Management. Instead of using the Active Record design pattern, it favors the Data Mapper pattern. This is an old discussion among Railers. Many people dislike that persistence logic is kept together with business logic, and the heavy metaprogramming magic that makes ActiveRecord both very easy to get started with and very difficult to properly master.

You should read José Valim’s post about Ecto Associations to get started. But for now, let’s compare simple Rails and Ecto models:

# app/models/user.rb
class User < ActiveRecord::Base
  has_secure_password

  has_many :posts

  validates :username, presence: true
  validates :email, presence: true
end

In Rails we have the ActiveSupport#has_secure_password which uses BCrypt underneath to generate a proper password digest. If you’re building authentication from scratch you must use this construct.

Phoenix does not have the same feature yet (#OpportunityToContribute!) so its version is a bit more verbose to account for the BCrypt digest logic using the Comeonin password hashing library. Let’s go on in small steps:

# web/models/user.ex
defmodule Pxblog.User do
  use Pxblog.Web, :model
  import Comeonin.Bcrypt, only: [hashpwsalt: 1]

  schema "users" do
    field :username, :string
    field :email, :string
    field :password_digest, :string

    timestamps

    # Virtual Fields
    field :password, :string, virtual: true
    field :password_confirmation, :string, virtual: true

    has_many :posts, Pxblog.Post
  end
  ...
end

The first part of Ecto Models declares the database fields, virtual fields and associations. Rails ActiveRecord prefers the approach of asking the database to send the table metadata and uses metaprogramming to create all the fields later at runtime. Many people dislike this approach and this is the alternative: explicit declaration.

We are mapping the User module with the ‘users’ database table in the schema “users” do statement instead of resorting to pluralization conventions.

The last line has our well-known has_many association.

defmodule Pxblog.User do
  ...
  @required_fields ~w(username email password password_confirmation)
  @optional_fields ~w()
  ...
end

Here we have declared required fields. This is just a variable with a list of fields, not the validations themselves. This will be used in the next step to accomplish something similar to validates :username, presence: true.

defmodule Pxblog.User do
  ...
  def changeset(model, params \\ :empty) do
    model
    |> cast(params, @required_fields, @optional_fields)
    |> hash_password
  end
  ...
end

Instead of doing something like Post.create(params) we first create a changeset and then pass it to the Ecto main Repository. The Repository is then responsible for the persistence part. The Ecto Model is responsible for validating and cleaning up the changeset that the Repository receives.

The changeset/2 returns an Elixir Struct for us to work with before passing it to the Repository application for persistence.

In this function we can declare a pipeline of validations, constraints and other attribute transformations. For example, we plug in a hash_password/2 function that will get the value in password and use Comeonin.hashpwsalt/1 to transform the password string into a bcrypt digest and store it in the password_digest attribute:

defmodule Pxblog.User do
  ...
  defp hash_password(changeset) do ...
    if password = get_change(changeset, :password) do
      changeset
      |> put_change(:password_digest, hashpwsalt(password))
    else
      changeset
    end
  end
end

And we return the transformed changeset so the pipeline can pick it up and pass it to other plugs, such as validations. If we wanted to add more validations we could do it like this:

defmodule Pxblog.User do
  ...
  def changeset(model, params \\ :empty) do
    model
    |> cast(params, @required_fields, @optional_fields)
    |> validate_length(:password, min: 3, max: 100)
    |> validate_length(:username, min: 5, max: 50)
    |> validate_confirmation(:password)
    |> unique_constraint(:username)
    |> hash_password
  end
  ...
end

There you go. And in the controller, the update/2 function, for example, will use the changeset like this:

def update(conn, %{"id" => id, "user" => user_params}) do
  user = Repo.get!(User, id)
  changeset = User.changeset(user, user_params)
  case Repo.update(changeset) do
  ...
end

In the second line we use the Repository to query the ‘users’ schema as declared in the User model.

Then, we transform the user struct with the user_params map that we received from the Router pipeline, as defined in the first line.

The transformation returns a changeset, which will contain error messages. Then we pass the changeset to the Repository again so it updates the record in the table.

MVC View and Templates

In the case of the edit/2 function we call the render/3 function like this:

# web/views/user_view.ex
def edit(conn, %{"id" => id}) do
  user = Repo.get!(User, id)
  changeset = User.changeset(user)
  render(conn, "edit.html", user: user, changeset: changeset)
end

This first calls the web/views/user.ex which imports stuff like helpers, transforms the user and changeset variables into module attributes (the ones starting with “@” if you’ve been wondering what those are). And the View knows to find the edit.html template at web/templates/user/edit.html.eex because it says so in web/web.ex:

# web/web.ex
defmodule Pxblog.Web do
  ...
  def view do
    quote do
      use Phoenix.View, root: "web/templates"

      # Import convenience functions from controllers
      import Phoenix.Controller, only: [get_csrf_token: 0, get_flash: 2, view_module: 1]

      # Use all HTML functionality (forms, tags, etc)
      use Phoenix.HTML

      import Pxblog.Router.Helpers
    end
  end
  ...
end

I did not copy and paste all the other macros in web/web.ex but check them out to see what models, controllers, router, channel import in each module you create.

In Rails we have the default ERB for “Embedded Ruby” and in Phoenix we have “EEX” for “Embedded Elixir”. It’s essentially the same thing: an HTML template that accepts snippets of Elixir code enclosed between <%= … %>. So, the edit.html.eex template looks like this:

<!-- app/views/users/edit.html.erb -->
<h2>Edit user</h2>

<%= render "form.html", changeset: @changeset,
                        action: user_path(@conn, :update, @user) %>

<%= link 'Back', to: user_path(@conn, :show, @user) %> |
<%= link "Back", to: user_path(@conn, :index) %>

Which is very similar to the equivalent edit.html.erb in Rails:

<!-- web/templates/user/edit.html.eex -->
<h1>Editing User</h1>

<%= render 'form' %>

<%= link_to 'Show', @user %> |
<%= link_to 'Back', users_path %>

The Phoenix version is slightly more verbose in order to not hide as much as Rails does. One can argue whether more or less magic makes it more productive, but the Phoenix version being more explicit leaves a trail of breadcrumbs that is easier to follow, especially if you’re just getting started. Here we have no concept of “partials”; every template can render any other template, we just need to pass through the necessary variables for the template to function. But instead of passing a model instance we are passing a changeset.

The “form.html” template is also very similar, let’s check out the Phoenix version first:

<!-- web/templates/user/edit.html.eex -->
<%= form_for @changeset, @action, fn f -> %>
  <%= if @changeset.action do %>
    <div class="alert alert-danger">
      <p>Oops, something went wrong! Please check the errors below:</p>
      <ul>
        <%= for {attr, message} <- f.errors do %>
          <li><%= humanize(attr) %> <%= message %></li>
        <% end %>
      </ul>
    </div>
  <% end %>

  <div class="form-group">
    <%= label f, :username, "Username", class: "control-label" %>
    <%= text_input f, :username, class: "form-control" %>
  </div>
  ...

And now the Rails ERB version:

<!-- app/views/users/_form.html.erb -->
<%= form_for(@user) do |f| %>
  <% if @user.errors.any? %>
    <div id="error_explanation">
      <h2><%= pluralize(@user.errors.count, "error") %> prohibited this user from being saved:</h2>

      <ul>
      <% @user.errors.full_messages.each do |message| %>
        <li><%= message %></li>
      <% end %>
      </ul>
    </div>
  <% end %>

  <div class="field">
    <%= f.label :username %><br>
    <%= f.text_field :username %>
  </div>
  ...

Remarkably similar. There is some language-specific stuff like having label(f, :username) instead of f.label :username. Because in Elixir the parentheses are also optional and because Phoenix implements helpers that are very similar to the Rails version, like “form_for”, we feel very comfortable very fast.

Rails has a default layout at app/views/layouts/application.html.erb and Phoenix has a default layout at web/templates/layout/app.html.eex. The rest is pretty much the same.

The mix phoenix.gen.html creates a template structure that is similar to rails generate scaffold command.

What Phoenix calls “views” is more similar to what Rails calls “helpers”. We can use them similarly, for example, to access the current user session, we do like this in Phoenix web/views/layout_view.ex:

# web/views/layout_view.ex
defmodule Pxblog.LayoutView do
  use Pxblog.Web, :view

  def current_user(conn) do
    Plug.Conn.get_session(conn, :current_user)
  end
end

Which is almost the same as app/helpers/application.rb in Rails:

# app/helpers/application.rb
module ApplicationHelper
  def current_user
    session[:current_user]
  end
end

Finally, Phoenix’s default scaffolding already brings in Bootstrap so it looks much nicer than the 10-year-old “scaffold.css” that Rails generates by default. There are many gems that override that though.

Tests

The last thing is the testing system. Rails uses Minitest, Elixir uses ExUnit. Again, the helpers are so similar that you can translate almost directly from Phoenix to Rails and vice versa.

As a caveat, Rails tests evolved a lot in the past decade. Even without adding extra features such as Factory Girl, the standard Fixtures still don’t have an equivalent in Phoenix (#OpportunityToContribute!).

Let’s start seeing a small model test in Phoenix:

# test/models/post_test.ex
defmodule Pxblog.PostTest do
  use Pxblog.ModelCase

  alias Pxblog.Post

  @valid_attrs %{body: "some content", title: "some content"}
  @invalid_attrs %{}

  test "changeset with valid attributes" do
    changeset = Post.changeset(%Post{}, @valid_attrs)
    assert changeset.valid?
  end

  test "changeset with invalid attributes" do
    changeset = Post.changeset(%Post{}, @invalid_attrs)
    refute changeset.valid?
  end
end

And the same unit tests in Rails:

# test/models/post_test.rb
require 'test_helper'

class PostTest < ActiveSupport::TestCase
  setup do
    @valid_attr = {body: "some content", title: "some content"}
    @invalid_attr = {}
  end

  test "with valid attributes" do
    post = Post.new(@valid_attr)
    assert post.valid?
  end

  test "with invalid attributes" do
    post = Post.new(@invalid_attr)
    refute post.valid?
  end 
end

Remarkably similar. For the test data, in Phoenix we are using Elixir’s module attributes, in Rails we put it in the setup step to create instance variables. They are not the same thing, but the result is similar. Again, in Rails we test the model, in Phoenix we test the changeset.

Now let’s see some bits of a controller test. As I said before, because Phoenix does not have a Fixtures or Factory feature in place (although the nice guys at Thoughtbot just released a Factory Girl-like library for Phoenix called ExMachina) we have to do a bit more setup:

# test/controllers/session_controller.ex
defmodule Pxblog.SessionControllerTest do
  use Pxblog.ConnCase
  alias Pxblog.User

  setup do
    User.changeset(%User{}, %{username: "test", password: "test", password_confirmation: "test", email: "test@test.com"})
    |> Repo.insert
    conn = conn()
    {:ok, conn: conn}
  end

  test "shows the login form", %{conn: conn} do
    conn = get conn, session_path(conn, :new)
    assert html_response(conn, 200) =~ "Login"
  end

  test "creates a new user session for a valid user", %{conn: conn} do
    conn = post conn, session_path(conn, :create), user: %{username: "test", password: "test"}
    assert get_session(conn, :current_user)
    assert get_flash(conn, :info) == "Sign in successful!"
    assert redirected_to(conn) == page_path(conn, :index)
  end
  ...
end

The Rails controller test is getting the data from a Fixture, that’s why the setup is shorter:

# test/controllers/session_controller.rb
require 'test_helper'

class SessionsControllerTest < ActionController::TestCase
  setup do
    @user = users(:user_one)
  end

  test "shows the login form" do
    get :new
    assert_response :success
  end

  test "creates a new user session for a valid user" do
    post :create, user: {username: @user.username, password: "password"}
    assert session[:current_user]
    assert flash[:notice] == "Sign in successful!"
    assert_redirected_to user_posts_path(@user)
  end
  ...

But as I said before, Phoenix implements similar helpers, so it’s very straightforward to port from Rails to Phoenix here.

You execute the test runner in Rails with rake test and in Phoenix with mix test. Not a lot to see here. Tests load very fast, I believe they run in parallel, and it’s very fast, which is a good departure from when you have a really large Rails app where a test suite can take minutes to finish.

Conclusions

If you came this far, you may be interested in following Brandon’s tutorial and then building your own Rails copy or you can just clone from my Github repositories. This is the Phoenix exercise and this is the Rails exercise.

What you can conclude thus far is that Phoenix is already a very full-featured web framework. And this is not even touching on what I think is its crown achievement: its remarkable WebSockets support, recently tested and benchmarked with real machines from Digital Ocean and achieving a peak performance of 2 million concurrent connections.

I think this is the real use case for Phoenix: microservices to deliver on Erlang OTP’s promise of hyper-reliability and hyper-concurrency. For that goal it’s already production-ready and you can use it right now. There is still much to be learned regarding proper tuning, proper monitoring, proper production environment troubleshooting, and so on, but it’s getting there.

As a complete Ruby on Rails replacement, it is not on par yet. But it’s unfair to say that, because a new framework can’t compete with 10 years of an entire ecosystem creating all sorts of features, tools and techniques for it. Rails is a powerhouse and it will continue to be so.

In this contrived example we already got one small feature we still don’t have in Phoenix: the equivalent of ActiveSupport#has_secure_password. But this is not all, we still don’t have the equivalents for Devise, Simple Forms, Active Admin or Rails Admin, Refile, Koala, Spree, etc. But we can have those, as the ecosystem starts to fill in the blanks (#OpportunityToContribute!) in this brand new Phoenix ecosystem. And I urge people to do so, as Thoughtbot already delivered their ExMachina so those who like Factory Girl can jump right in.

As it stands right now, it feels like the time when Rails 1.2 had just been released, many libraries were just too young, we still didn’t have mature tools. But the upper hand of Phoenix and Elixir is the decades-old and battle-tested Erlang core. This is something no one else has, and the best way to capitalize on it is to go beyond what tools like Node.js have accomplished: a truly enjoyable programming environment, with a language that was truly well designed and targets programmer happiness, with a really mature core to boot.

Programmers shouldn’t have to fight over small stuff. Task management? Done, use Mix. Package Management? Done, use Hex. Promises? Don’t need them. Require semantics? Don’t need them. You get the gist.

If you’re a Rails programmer and you want to find a companion platform to augment your existing Rails solutions with highly reliable and highly concurrent microservices, Phoenix is an option ready for production right now. Jump right into it!

He also posted a great screencast about Microsoft’s new alternative MVC framework and I thought it would be great to have him at my blog to talk about technology and web frameworks. As I said before I think that we should not become alienated about what’s going on in other fronts and Scott is a very forward thinking and open minded person as well.

AkitaOnRails: First of all, let me congratulate you mr. progman.exe :-) I understand that this was something you’re very excited about. Can you tell us a bit more about your transition to Microsoft and your new duties?

Scott Hanselman: I’m working for Simon Muzio who runs www.asp.net, www.iis.net and large content-packed sites like that. I like to think my job is writing a book that will never end, where that book is blog posts, articles, screencasts, videos, talks, etc. I’m a teacher at heart, so while I took the title “Community Liaison” I like “Teacher.”

AkitaOnRails: Tell us about your history. How did you start in the computer world? Did you always develop more toward Windows or did you have a background in the Unix world as well? And before your recent departure to Microsoft you worked for Corillian. What was your role there?

Scott Hanselman: This blog posts talks about my early years in school.

See that book on the right there? That’s how I learned to program. This book was THE book in 1984. I typed in the whole book on a Commodore 64.

And a VERY influential teacher that set me on the right track.

Marianne Mayfield is my fifth-grade teacher and the reason I’m in computers today. She was at my wedding, at graduations, at family events. Twenty years later and we are still close. When I was young, she recognized that I was “at risk for trouble” and sat down with my family in order to find a way to get me back on track. She saw that I was transfixed by the Apple II in our classroom and set it up so we could sneak the computer out of the school as long as it was back by Sunday night and no one noticed.

(click links above for more details)

Scott Hanselman: I was the Chief Architect at Corillian, working with other engineers and architects throughout the company, planning, designing, coding the next generation of Corillian’s Online Banking Platform.

AkitaOnRails: I applaud your Team Hanselman campaign to help fight Diabetes. I understand that you have Diabetes as well and that, fortunately, you managed to have it under control nowadays. Can you tell us this story and the goals of your campaign?

Scott Hanselman: My personal story is here :

I currently take three shots a day of Symlin while also wearing an Insulin Pump 24-hours a day, even while I sleep. The pump saves me from an additional six shots a day, which I took for 8 years before the pump. I test my blood sugar by pricking my finger between 8 and 10 times a day – that’s about 46,500 finger pricks so far, and miles to go before I sleep.

And there’s a good explanation of diabetes here :

Here’s where the analogy gets interesting. Remember in the analogy we are flying from L.A. to New York, except we only get to check our altitude seven times. And, we only get to change altitude (take insulin) less than ten times. But, when I check my blood sugar, I’m actually seeing the past. I’m seeing a reading of what my blood sugar was 15 minutes ago. And, when I take insulin, it doesn’t start lowering my blood sugar for at least 30 minutes.

The campaign ended up raising over USD$30,000, and that money will be applied to research, education and programs surrounding diabetes.

AkitaOnRails: “What in the world were you doing at a RailsConf?” That’s what I think people from outside of the Microsoft blogosphere would ask you :-) So, are you just curious about Rails, or do you actually develop Rails-based apps? I am very interested to learn your opinions about Rails. How did you find it? What do you think is best about Rails? And more importantly, what do you think Rails could have improved?

Scott Hanselman: I’m always trying different programming languages and frameworks. I’ve been doing Rails on and off for about 18 months, but it hasn’t clicked for me yet. I’m sure it will soon. The MVC pattern is very old…I was doing it in Java over 10 years ago, and certainly the pattern goes back 30 years. I think that Rails is a good framework considering its young age. I think it’s Ruby that is the real magic though and I find the Ruby aesthetic to be very in line with my own.

AkitaOnRails: Then, following my biased preferences, the next article I like is your screencast over the CTP ASP.NET MVC framework. I see that this new framework resembles Rails a lot. Of course, it may just look like that because both implement MVC2. But on the other hand, it uses very similar Rails idioms everywhere. It lacks a little bit of ‘convention-over-configuration magic’ but from what I saw, it looks much better than doing conventional ASP.NET. What do you think about it? Can you summarize it a little bit for those that don’t know this product yet? And what do you think will still improve in it?

Scott Hanselman: If one points at Rails and says “that’s MVC” then everything else looks like Rails, sure. Any good MVC framework needs to not only follow the pattern as appropriate but also abstract away HTTP (assuming it’s a web framework) in a clean, but not complete, way. The ASP.NET MVC implementation still has some baggage that it will always have to carry because ASP.NET WebForms came first and wasn’t designed from scratch with testability in mind. That said, it’s all about alternatives. Use the framework that makes you happy and you feel most productive in. Some fraction, maybe 20%, of ASP.NET devs will prefer MVC. Now they have a MS support choice with good integration with C# 3.0

at RailsConf'07

AkitaOnRails: And what would be tech world without discussions and flame wars? I understand that most .NET developers don’t even pay attention to Rails or other frameworks. I even heard that there are some developers questioning things like “Why yet another web framework if ASP.NET is already there?” Meaning, why reinvent the wheel and divide attentions? I particularly think that this new MVC is the way to go, but it is undeniable that the old ASP.NET has much more acceptance and support. How to reconcile both point of views?

Scott Hanselman: That’s just FUD (fear, uncertainty, and doubt). Think what they like, this Framework is just about alternatives. WebForms is great, and if you’re doing Line of Business (LOB) apps with complex grids and such, it’s not going anywhere. However, if you’re an agile shop with a penchant for TDD and have been frustrated by ViewState, then check out ASP.NET MVC. There’s nothing to reconcile. Some people take a car, some a motorcycle, and others walk. As long as you get from point A to point B and feel good about the journey.

AkitaOnRails: Speaking of flame wars, there is probably one subject that gives me the creeps: the paranoid conspiracy theories regarding Miguel de Icaza and Mono. A lot of people actually believe that Miguel is ‘selling’ part of the open source world to Microsoft. Have you ever used it? What do you hear throughout the community and what’s your opinion about this matter?

Scott Hanselman: Yes, I’ve used Mono and I think it’s a great way to get your .NET application working on Linux or Mac. I think Miguel is brilliant and a fine dancer. I hear in the community that they are happy Mono exists and that it’s available as an option.

AkitaOnRails: I will probably be cursed for this one: but I am not ‘strongly’ against Microsoft as some fundamentalists are. It is public and notorious that Microsoft is a big corporation and as such it did its share of ill conceived deeds. A lot of them were already settled in court. So, it’s more complex than people usually think. I prefer to assume that inside Microsoft there is a lot of very smart people.

Scott Hanselman: I stay out of legal stuff. Microsoft is a huge company with a lot of smart people, but it’s really many medium sized companies working together (usually). I can’t speak to “ill conceived deeds.” I can only say that I do my best every day to not be evil, as do my coworkers. There’s no evil master plan in my group.

AkitaOnRails: I particularly like the roadmap for Windows Server 2008 (with it’s minimalist approach, finally having only what you need installed). Last year you were experimenting with IIS 7 and its new FastCGI support. I think it was primarily intended for PHP support but it could theoretically handle Ruby on Rails as well. Have you tried Rails there? I think Microsoft is clearly going after the low-end with Windows Server 2008 + IIS 7, against the inexpensive Linux+Apache boxes. I think they won’t be able to reach them in terms of prices but what do you think of it as an internet service contender?

Scott Hanselman: I have little power in that arena, but every time someone will listen I’m pushing for a nice inexpensive Web Server SKU. I’m a huge fan of IIS7 and I’d like as many people as possible to check it out. I haven’t tried IIS7+Rails but I’ve heard rumors of folks getting it working.

AkitaOnRails: There are a lot of new features in .NET 3.5. LINQ is the one that interests me the most. Can you give us a brief description of why it’s so interesting? And what do you think are the best improvements of the 3.5 version? Are you already using Orcas? What’s your opinion? Oh, and are you aware of the Sapphire in Steel project?

Scott Hanselman: LINQ is interesting because it’s its own Domain Specific Language for querying objects of all kinds, and it’s living right inside VB and C#. It’s a MUCH smarter compiler and it’s a joy to work with. I’ve installed Sapphire In Steel and I know a few people who are digging it. I’m looking forward to Rails on .NET.

AkitaOnRails: One of the advantages of the Microsoft .NET strategy was being prepared since its first day to be language-agnostic. Now we have the DLR that was derived from the works in IronPython and not IronRuby. But there is yet another cool language that not many people are aware of: F#, the OCaml inspired language. Have you tried it?

Scott Hanselman: Yes, and I’ve done two podcasts on it, one with Robert Pickering, author of a book on F# and one with Dustin Campbell, an enthusiast. I’m stoked it’ll be a first-class language inside VS.

with John Lam

AkitaOnRails: I am not against technical comparisons, we have to do such things to decide our steps in projects. But I am against pointless challenges. Rails is far from perfect and I am the first say it. That said, it is great for an specific niche of web development. I usually recommend that good programmers should expand their horizons and learn as much new stuff as possible. You can’t know everything perfectly but it’s always good to be aware of the possibilities. You wrote something around these lines. What do you think about it?

Scott Hanselman: Rails is good for what it does in its niche. You don’t create Windows Desktop apps with Rails and that’s fine. That’s why we don’t compare Rails to WPF, we compare it to Web Frameworks. Comparison is good and there’s always good to be found on either side. I avoid flame wars as well. It’s pointless. If the shoe pinches, don’t wear it. But, try as many shoes as you can!

AkitaOnRails: Last year you wrote an interesting post about the future. Google is this big giant, Amazon released surprisingly good products in the last few months like S3, EC2, SimpleDb. I am not well aware of the current Microsoft Live’s strategy. Can you say something about where is it heading to?

Scott Hanselman: I’m not in that group so I have no insight outside of what’s already been said. However, I do still hold to my predictions in that post!

Computing will be moved into the Cloud. It’s already happening, we’re 20% there. The idea has been around since the beginning, and it will, in my opinion, continue to come up until it actually happens and we build Skynet. One of these compute clouds will no doubt end up in orbit.

AkitaOnRails: And of course, I kind of ‘have’ to ask you this: do you use Windows Vista? How has it been for you given that there’s a huge underground campaign against it? Do you think it is as bad as everybody else says? I can’t comment because I never tried it outside of my virtual machine (I am a Mac guy), but they did sell 100 million licenses.

Scott Hanselman: Sure. I run Vista 64-bit on 3 machines, and Vista 32-bit on 2 machines. It was rough for the first 3 months while the driver support sucked, but now it’s fine. I’ve got 2 vista machines that haven’t been rebooted in months, just put to sleep and woken up. Once you get good video drivers, you’re usually cool.

Folks give Vista a hard time but they forget the wide range of hardware it has to run on. OS X only has to run on Macs, and a narrow range at that. Vista has to run on millions of combinations of hardware. Third Party Drivers are by FAR the thing that crashes Vista the most and it gets blamed. That’s why I run 64-bit and use only signed drivers. I’ve been thrilled with 64-bit. Works great.

AkitaOnRails: Ok, this is it :-) Thanks a lot!

As I promised after the Avi Bryant interview, here’s a great conversation with Adrian Holovaty, well known creator of the Django web framework written in Python.

For me this is an important piece because I always say that technology doesn’t have to be about divorce. Technology is about integration. I am a full-time Ruby on Rails developer and evangelist, but above all, I try to be a ‘good’ programmer. And good programmers acknowledge good technology and their creators’ achievements. And Adrian’s Django is such a remarkable achievement that deserves its attention and success.

So, as my very first post of the year (published at 0:01hs!), I would like to celebrate the great minds of our ‘development’ community, wishing that the good developers use their time creating great technology instead of wasting it in useless flame wars.

AkitaOnRails: As I always do, let’s start by talking a little bit about yourself. How long have you been a programmer?

Adrian Holovaty: Let’s see — I’ve been tinkering with computers since I was a little boy playing with the Commodore 64 in the mid ’80s.

One early program I wrote was for the TI-85 calculator in my high-school math class. By the time I graduated, a whole bunch of students were using it, because it had all the formulas and stuff.

I didn’t take any of this stuff seriously until college, when I minored in computer science. But I didn’t end up finishing the minor, which is something I’ve always been bummed about.

My college major was journalism, as I originally wanted to be a newspaper reporter, but after I took a job with the Web site of the campus newspaper (The Maneater – best name for a newspaper ever!), I realized I would have a lot more fun working online than with a dead-tree product.

So I ended up finding a way to combine journalism/news and computer programming, and I’ve had a number of jobs doing exactly that!

AkitaOnRails: And what led you to the computer world?

Adrian: I guess it was in my upbringing, as my dad has been a computer programmer since the ’70s, back in the punchcard days. I remember seeing all sorts of those punchcards in our house when I was growing up.

AkitaOnRails: And, of course, how did you meet and fall in love with Python?

Adrian: My Web development path was (I think) pretty typical, at least based on the experiences of my friends and colleagues. I originally learned how to write Perl CGI scripts in a college Web development course. Then I taught myself PHP because somebody recommended it to me.

After doing Web development with PHP for a few years, I got kind of tired of it. At that time, I was working with Simon Willison at a newspaper Web site in Lawrence, Kansas. Simon and I were both getting tired of PHP and decided to “dive into Python” (pun intended) by reading Mark Pilgrim ‘s excellent book of the same name, which was released around that time. This must’ve been late 2002.

We immediately fell in love with Python. And when I say “immediately,” I really do mean it. It was like a revelation, some sort of divine moment. It was the programming equivalent of love at first sight.

So Simon and I decided that, from then on, we were going to code everything in Python. That’s one of the perks of working for a small Web development shop — we could make those kinds of snap decisions! 

AkitaOnRails: I was first introduced to Python around 2000 as well. But it was through web development using Zope. I think it got a lot of attention at the time. Did you ever try it?

Adrian: No, I never tried Zope. Still, to this day, I’ve never tried it. And I’ve been told that’s a good thing, because evidently it turned some people off of Python back then, for whatever reason. (Their newest version is supposed to be a ton better, but I haven’t tinkered with it, either.)

AkitaOnRails: You’re right, and it is kind of amusing because a lot of pythonists from Brazil still use Plone (I think it evolved from Zope but I haven’t tried it either). So, We all know that Django was born out of necessity when you were developing websites for World Online. Please describe how you began there and what the everyday work is like. Maybe what led you to Django during your work day?

Adrian: Sure! Well, generally when people think of newspapers, they think of crotchety, old-fashioned editors and reporters scribbling in paper notebooks with pencils. This Kansas newspaper that I worked at was the complete opposite of that. It’s beyond the scope of this interview, but for a number of reasons that newspaper Web site attracted (and still attracts, to this day) a really great development team — one that I would stack up against any team 10 times its size.

We were doing “Web 2.0” stuff back in 2002 and 2003, and we were building Web apps in days, not weeks or months. This was mostly due to the journalism environment – newspaper people like deadlines. 

So we were in this culture of “Web development on journalism deadlines,” and we needed some tools that let us create Web applications quickly. We looked around at some existing Python libraries at the time (2003), and we ended up deciding to write our own stuff.

We didn’t set about to make a framework – it was a very classic, clichéd path, actually! What happened was, we built a site with Python. Then we built another one, and we realized the sites had a fair amount of code in common, so we did the right thing and extracted the common bits into a library.

We kept doing this – extracting and extracting, based on each new Web application that we created – and eventually we had a framework.

AkitaOnRails: As a side-note, actually this is kind of interesting! I would love to hear what do you think World Online has that attracts good developers like yourself! Just a summary would be ok. That’s the kind of insight brazilian companies have to learn better. Letting you do Python is probably one of them 

Adrian: Yeah, it kind of comes down to empowering the employees. My boss, who led the Web team, delegated all the technology decisions to me and Simon. That kind of culture really encourages quality work, because it makes everybody on the team more invested.


AkitaOnRails: You’re right. Traditional companies would rather have a more hierarchical (and bureaucratic) approach, with a dumb manager holding the whip. But I digress. So, I particularly dislike language x VS language y or framework a vs framework b kind of comparisons. Instead I’d like to hear what do you think is great about Python – the language and the platform – and Django. What are the features in Django that you hold dear to yourself, I mean, those bits of functionality that you actually think for yourself as ‘the’ greatest ideas?

Adrian: Well, I love abstractions – like, I suspect, any programmer. At its core, Django is just a set of abstractions of common Web development tasks. It gives me joy to create high-level abstractions, to encapsulate a large problem in such a way that is made simple.

What I really like about Django is the depth and breadth of abstractions that it provides. And in most cases (I hope!), they’re clean, easy to use and understandable.
Let me give you an example of an abstraction: creating an RSS feed.

When you create an RSS feed, you don’t want to have to deal with remembering angle brackets, and the exact formatting of the feed — you just care about the items in the feed. So Django provides a very simple library that lets you give it a bunch of items and creates a feed from them.

So that’s an obvious example, but what really excites me are the higher-level abstractions, like the concept of an “admin site.” Django comes with a completely dynamic application that makes a beautiful, production-ready CRUD site for your database. There’s no code to write – it’s only a small bit of optional configuration.
There’s also something called Databrowse, which is an abstraction of the concept “Show me my data, as intelligent hypertext.”

So whenever I’m explaining Django to someone, I always end up saying, “It’s just a bunch of abstractions of common Web development tasks” – from low-level HTTP wrapping to higher – and higher-level concepts. The higher you get, the more productive you can be. I apologize if this is too conceptual!

As for Python — what can I say? It’s gorgeous. It’s like poetry. It’s so clean, so logical, so regular, so obvious. And the import system is to-die-for.

A lot of people say Python code is easy to read/understand because of the consistent whitespace and simplicity of the language. I agree with that, but I think it’s also due to the elegant import system. Why? Because if you want to know how any Python module works, just look at the code, and look at the modules it imports. As the Zen of Python says, “Namespaces are one honking great idea – let’s do more of those!”

AkitaOnRails: Tech communities are great because they are so energetic and passionate. On one hand you have those that contribute and make the technology grow faster. On the other hand you have the pundits and trolls that like flame wars. Do you burn yourself sometimes, or do you tend to avoid this kind of discussion? Specifically, some time ago you and DHH were together at the Snakes and Rubies presentation. I think it was great because it didn’t end in yet another flame war. What’s your opinion on this kind of open (not flamed) discussion?

Adrian: The answer to this one is obvious – clearly constructive discussion is more productive.

At times, I’ve had my passion for Python/Django get the best of me, but I’ve gotten a lot better over the years. I’ve realized something: At the end of the day, what really matters is the sites people create with these tools, not the tools themselves. If you’re going to judge someone, judge the sites that person makes, instead of the tools that person uses.

These days, if I am involved in any sort of discussion like that, it’s usually to try to calm people down. 

AkitaOnRails: Well said. Going back to features, you mentioned ‘Databrowse’ before, I would like to know more about it. I think Django didn’t have it by the time I first tried it. Is it a new feature (or am I the one lagging behind?  Can you elaborate more on this construct?

Adrian: Databrowse is still quite under the radar! Here’s the use case.

Say you have a lot of data in your database, and all you want to do is look at it.
Here are your current options: You can drop into the psql or mysql prompts and run a bunch of SELECT queries, but that gets tiresome. You could run something like PHPMyAdmin, but that’s more of an administration tool than a tool for browsing. You could use some sort of external application that lets you browse your database tables, MS Access-style, but that’s in the realm of the desktop app.

Databrowse automatically creates a Web site that displays your data, so you can click around to your heart’s content.

The other thing it does is point out interesting, non-obvious queries. For example, if you have a table that has a DATE column in it, it will automatically create a calendar view of that table.

The point isn’t for people to use this to make public-facing sites – the point is for people to use this to explore their own data, with no effort required.

Another use case comes from the journalism world. I used to work with a guy named Derek Willis at the Washington Post newspaper. His job at the time was to acquire huge datasets and place them on the newspaper’s intranet, so that reporters could search and browse the data in their research.

Derek didn’t want to have to hand-roll a Web app each time he got a new dataset, so he used Databrowse to make intranet sites that displayed his databases — with little-to-no effort.

This comes back to what I was saying earlier about high-level abstractions. Databrowse is a particular type of abstraction, and it’s really cool that we include something like that for people to use, if they need it.

AkitaOnRails: Sounds great, I am looking forward to using it. In a degree it feels like Dabble Db – albeit inside your own app. I think you already met Avi Bryant ? I just interviewed him and it was a very insightful conversation. Have you ever tried Seaside? (out of curiosity I have a photo of you and Avi looking at his macbook sitting down in the grass, what was that?)

Adrian: Yes, it was inspired by DabbleDB – it’s essentially a very toned-down DabbleDB-ish thing for your own data. But I shouldn’t even say that, because DabbleDB is a whole universe more sophisticated than Databrowse. DabbleDB is awesome. 

Yes, I’ve met Avi a number of times and always enjoy his company. We even played some music together – he is a fantastic harmonica player!

The photo you’re referring to was taken at Foo Camp in 2007.
I haven’t tried Seaside, but it’s on my to-do list. If I were ever developing an application like DabbleDB that needed to maintain a ton of state, I would turn to Seaside first.

AkitaOnRails: I see, I feel that good musicians tend to be good programmers as well. So, my first congratulations for you is about your new book on Django that was just released. I think it is a very big milestone for the Django community. More than that you were able to make it available online under a Creative Commons license.

How did you get involved in this book? Was it difficult to convince the publisher to have the book readily available online? I ask this because at least in Brazil this is one idea that is VERY hard to push over to publishers. Tell us more about your writing experience, challenges and quirks along the way.

Adrian: Thanks very much. There’s not much to the story here – co-conspirator Jacob and I got contacted by the Apress folks, who were interested in publishing a Django book, and we did it. Well, I should amend that: It took us a long time , but we did it. 
It wasn’t difficult to convince Apress to let us make the book available online. They had previous experience doing so with Mark Pilgrim’s Dive Into Python book, and they’re generally cool people.

In retrospect, publishing the book online was a fantastic decision. Not only is the final book available online, but we made chapters available online as we wrote them, with a really nice per-paragraph commenting feature that let readers submit typos, corrections and suggestions on a very granular basis. We got a ton of fantastic feedback for which we’re immensely grateful. Having experienced this, I wouldn’t want to publish a technology book any other way.

AkitaOnRails: Now, my second congratulations is because of that Batten Award for outstanding achievement in online journalism. ChicagoCrime.org looks great and is a very good example of a mashup done right. Unfortunately our local government and public institutions have very little to almost no data available online for us to use. Where did the idea come from? How was its development?

Adrian: Thanks – and don’t let that site scare you away from visiting the beautiful city of Chicago. 

AkitaOnRails: haha, I live in São Paulo. Can’t be worse ;-)

Adrian: The idea for chicagocrime.org came when I was bumming around the official Chicago Police Department Web site and found that they publish crime data – although in an interface that’s more suited for searching than for browsing. I thought, “Wow, this is some great data!” and was writing a screen scraper within about 10 minutes.

Around that same time, I was tinkering with the Google Maps site, which had just launched, to see if I could embed Google map tiles into my own Web pages. I figured the crime data would make for a great mapping application, so I put the two together, and one of the original Google Maps mashups was born. While I was developing the site, HousingMaps.com (Craigslist + Google Maps) came out, so it beat me to the punch of being the first real Google Maps mashup – but chicagocrime.org came soon afterward.

The thing I’m most proud of is the fact that chicagocrime.org was one of the sites that influenced Google to open its mapping API. Back when HousingMaps, chicagocrime.org and the original crop of mashups came out, we were all reverse-engineering Google’s JavaScript!

AkitaOnRails: I know that you probably have 2 great passions in your career: one is Python-based development, or course, and the other is journalism (dunno the correct order here ) You’ve made yourself quite a reputation on the online media. You’re working at the Washington Post right now, is that correct? I read an article at the OJR where you explained about technology being used to empower the journalist. Do you intend to be a reporter someday or you’re more into the back-end of journalism? I think you have a strong opinion about the future of journalism in the Internet Era, don’t you?

Adrian: Well, I would say my main passion is music, but it’s hard to make a living doing that. (Django is named after the famous jazz idol Django Reinhardt)

I’m no longer working at the Washington Post. In mid 2007, I was awarded a two-year grant from the Knight Foundation to create a local-news site. So I founded EveryBlock in July. We’re a team of four people and are working hard to make a really cool application.

But, yes, going back to your question, I certainly have some strong opinions about how journalism is practiced on the Internet. This is probably out of the scope of this interview, but you can read my essay/rant awkwardly titled A fundamental way newspaper sites need to change for a taste.

AkitaOnRails: Now, that’s news to me  Didn’t know you started your own company (sorry, I still have to catch up with the Python community). Can you tell us about your new endeavor?

Adrian: Hey, I can’t blame ya – we’re staying under the radar. There’s not much to say about EveryBlock at this point other than it’s chicagocrime.org on steroids. It’s like chicagocrime, but for more cities than just Chicago and more information than just crime. 

AkitaOnRails: You should try it for Brazil :-)

Adrian: Thanks to Django’s internationalization framework, that is entirely possible. 

AkitaOnRails: Yeah, and by the way, (digressing a little) I would recommend the brazilian movie Elite Squad now that I know you’re interested in crime related data ;-)

Adrian: Noted! I’ve added “Elite Squad” to my MOVIES_TO_WATCH.TXT :-)

AkitaOnRails: Going on. Many great websites are already deployed over Django. Your own works at World Online, Washington Post being some of them. Another big name is Kevin Rose’s Pownce (the creator of Digg). You, Kevin and other influential people usually meet during conferences, don’t you? What do you think influenced him to try Django? Another great thing about Pownce has to be Leah Culver. A programmer like her is simply unheard of here in Brazil, which is a pity. You should make her the official Django cheerleader!

Adrian: I’ve never met Kevin Rose, but from what I’ve read, I believe Leah chose to use Django because either it had been recommended to her, or she liked Python, or something. I don’t know the backstory. I haven’t met Leah, either, but she was cool enough to travel to Lawrence, Kansas, for our last Django sprint a few weeks ago (which I couldn’t attend in person, unfortunately).

AkitaOnRails: It is not a rule written in stone, but in Rails community we ‘tend’ to praise Mac related tech. Of course, it is not a requirement for any language and Linux is as good. But what can I say? I am an Apple fanboy  I would like to know what’s your development environment. What tools do you use to develop your Django-based websites?

Adrian: I recently switched to a Mac after several years of using Linux on the desktop, but don’t read anything into that — I’m not a fanboy by any means, of either Linux or Mac. I miss Linux, to be honest, and in a number of ways I’ve been unimpressed by OS X’s Unix features. (Basic stuff such as readline support in the default Python is broken, etc.)

The nice things about the Mac make it worth the switch, though. Even after having used the Mac as my primary machine for 6 months, I’m still impressed that I can just close the laptop and it’ll go to sleep automatically. That never worked with Linux!

AkitaOnRails: I ask this because many people looking at Rails and Django are Java and C# programmers that are used to have a full blown IDE. I usually say that any common text editor does the job, and if you really want power vim and emacs are a perfect fit. What do you say?

Adrian: I’ve never used an IDE to program, so I’m the wrong person to ask.
Actually, I misspoke – I used an IDE-ish thing in college to program assembly. It showed the contents of the accumulators and that stuff.

AkitaOnRails: Oh, I almost forgot to ask: did you and Guido van Rossum ever met? I would like to know your opinion about the Python 3000 development. Is there anything that’s in the new version that you really anticipate?


Adrian: Yes, I’ve met Guido a number of times and was even lucky enough to have dinner with him once! It’s almost silly and unnecessary to say this, but he is incredibly smart.

I really like the new concepts and features in Python 3000, and I had the opportunity to participate in a sprint at Google Chicago a couple of months ago. My favorite feature is the migration to Unicode strings by default, because it forces the encoding issue front and center.

AkitaOnRails: Finally, what was all the fuss about Django being versioned 1.0 or jumping straight to 2.0? I didn’t follow the discussions, just read some headlines a while back. And what can people expect for the next release of Django? I understand that you don’t go big bang releases that changes the game, but what do you see getting released in the next few months in terms of new features?

Adrian: Ha – this is an amusing story. I’ve gotten worn out by people constantly asking “When is 1.0 coming out? When is 1.0 coming out?” My response always is, “Why do you need an arbitrary number assigned to the product? Many people are using the current version, so don’t hold back. Version numbers are pretty meaningless.”

So I suggested on the django-developers mailing list that we assign the version number “2.0,” as if to reinforce the fact that version numbers are indeed arbitrary. But, having given it some more thought, I’ve changed my mind on that. The amount of confusion it would cause would not be justified by the amount of rebellious pleasure I would get out of it.

As far as upcoming features, there are a number of ongoing branches of the Django codebase that will all eventually be folded into the main trunk. One is a branch called newforms-admin, which dramatically improves the amount of customization developers can make to the Django admin site. Another is queryset-refactor, which is a refactoring of our database layer, again, to make it more extensible. Other than that, there are a few small features that we need to wrap up, and it’ll be time for 1.0.

AkitaOnRails: Haha, I think you’re right, I don’t like this ‘cult of the dot-oh’ as well, it’s pretty meaningless for an open source project that’s constantly evolving. But some people do have some concerns specially in a more enterprisey-like environment. Anyway, and for those that want to see Django in action, what are the websites that you recommend people visit? Some of the Washington Post, maybe? Of course, some of the great features – as the admin site – are only for admins (duh)  But people like to see ‘living’ things instead of a bunch of tutorials.

Adrian: As an aside, I see that Rails David just wrote a rant about the same topic.

Good Django-powered sites? Let’s see – there’s Tabblo, a photo-sharing and editing site that has some really nice interactive features. There’s Curse.com, the massive online-gaming site. There’s lawrence.com, a site I helped build, which is one of the best local-entertainment sites in the world (if I may say!). And there’s a site (I forget its name) that will print a paper book from a series of Wikipedia articles that you specify – that blew my mind, the first time I saw it.

AkitaOnRails: So, I think we are done :-) That was a very pleasant conversation. I wish you have a great New Year.

Adrian: Thanks for the great questions and conversation, and happy new year!

When Yukihiro Matsumoto created Ruby as a replacement for Perl and Python, inspired by Smalltalk, in 1993, he could hardly have imagined how far his language would reach outside Japan.

But in mid-2004, when David Heinemeier Hansson released his “Rails” framework — born from the products he built at his company 37signals — I believe he already knew exactly where he was headed.

Ruby on Rails is a product of the current Web 2.0 movement. I’m sure most people still haven’t grasped it yet, but we are at the inflection point of a new generation.

It’s a fact: Sun got it wrong with J2EE. A flawed implementation of a flawed set of requirements. Those who followed that story will remember the nightmare of EJB 1 and 2, the slow and poor Web Services support, more delays in the promise of EJB 3, and so on.

We are wasting enormous amounts of time building simple things. Web applications shouldn’t be complex. The Web was created to be simple. But J2EE was built to solve 100% of “Enterprise” problems. And we all know that when you try to fix everything, you end up fixing nothing properly.

With Rails, we finally become compatible with the dynamic spirit of Web 2.0. Web Services, Ajax — none of that is hard. And the traditional stuff: persistence layers, version control, MVC, templates, caching, security — all become what they always should have been: trivial.

When someone wants to change the requirements, we cringe: it takes enormous effort to implement modifications. After all, we invented an entire family of paraphernalia — like UML — precisely to ossify and obstruct any kind of change. Let’s be honest: UML was created so consultants could charge more for their work.

With Rails, changes are welcome. Rails is the first platform truly compatible with the Agile thinking advocated in the Agile Manifesto. J2EE made the term famous, but that promise never delivered for it.

To kick off my articles, I think it’s important to clarify two concepts that Rails made famous:

• DRY – Don’t Repeat Yourself. Rails has native, simple support for Helpers, Templates, Engines, Plugins — all well integrated so that you can be productive without being messy. Copy and paste is bad; we have to stop doing it, and Rails gives us the tools to do so.

• Convention over Configuration. Instead of editing dozens of XML and properties files, Rails architecture prefers conventions: put your controller with the right name in the right directory and it will be found. Put your model in the right place and it will be located.

Two simple concepts, implemented throughout the entire framework, and they make all the difference.

And don’t let prejudice lead you astray. I’ve already heard people say: “Ruby is an uncompiled language, so it must be slow”, “everything Ruby does I can do in Java”, “Ruby doesn’t scale well”, “I don’t need to learn another language”, “Java will never go away”.

Replace “Ruby” with “Java” and “Java” with “C” — or any other language. Sounds like we’re hearing the exact same things from 10 years ago, when Java was the novelty.

It seems there’s a tendency to criticize first and only look at the good parts later — and when we finally do, we realize we’ve arrived too late to the party.

How about doing something different this time, for a change?