And make sure to subscribe to my new newsletter The M.Akita Chronicles!

I’ll talk about the podcast today; remember that the pilot episode is already live (transcripts and RSS to subscribe) and on the channel on Spotify.

–

Every AI tutorial starts with “install the model, run inference, done!”. None of them tell you what happens when you need to run this in production with real uptime, controlled cost, and consistent quality.

I’ll tell you how it was to put a TTS (Text-to-Speech) model into production on a GPU in the cloud. Because if you’re thinking about serving AI models, you need to know what you’re getting yourself into.

The Scenario

The goal: generate a ~30-minute podcast from text, with two distinct voices, broadcast quality, ready for Spotify. Every week, automatically, with no human intervention.

The chosen model: Qwen3-TTS, a text-to-speech model that supports voice cloning and fine-tuning. It runs on GPU — needs at least 6GB of VRAM for inference.

The platform: RunPod, which offers serverless GPUs with billing by the second of usage.

Sounds simple, right? Well, hold on.

Problem 1: Cold Start — The Silent Killer

Serverless GPU works like this: when there’s no request, the worker shuts down. When a request comes in, it boots up, loads the model into VRAM, and processes.

The time to “load the model into VRAM” is the cold start. For a ~4GB model, we’re talking 30 to 90 seconds. Your HTTP request sits there hanging, waiting. If your timeout is 30 seconds, you don’t even get to see the model work — you just see a timeout.

The naive solution: “bump the timeout!”. The real solution: health check with polling.

@app.get("/health")
async def health():
    if not model_loaded:
        return JSONResponse(
            status_code=503,
            content={"status": "loading", "detail": "Model loading..."}
        )
    return {"status": "ready"}

The client (in this case, a Rails job) polls /health before sending the actual request:

def wait_for_server(max_wait: 180)
  deadline = Time.current + max_wait
  loop do
    response = http_get("/health")
    return if response.code == 200
    raise "Server não iniciou a tempo" if Time.current > deadline
    sleep 5
  end
end

Sounds primitive? It is. But it works better than any sophisticated “warm pool” solution that costs 10x more.

Problem 2: Where Do the Model Weights Live?

A 4GB model needs to be available when the worker starts. Three options:

1. Bake into Docker: 4GB+ image. Slow build. Slow push. Slow pull.
2. Download from HuggingFace on boot: Network-dependent. Can fail. Adds minutes to the cold start.
3. Network Volume: Persistent disk mounted on the worker.

Option 3 is the correct one. RunPod offers Network Volumes that persist between runs. The model sits there, always ready. The cold start is just the time to load from disk to VRAM, no download.

But there’s a catch: the volume may or may not be mounted, depending on how the worker is configured. So the setup code needs to be resilient:

NETWORK_VOLUME = "/runpod-volume"
LOCAL_FALLBACK = "/workspace/models"

def resolve_model_path():
    if os.path.isdir(NETWORK_VOLUME):
        base = os.path.join(NETWORK_VOLUME, "huggingface")
        os.environ["HF_HOME"] = base
        return os.path.join(base, "hub", "models--Qwen--Qwen3-TTS")
    # Fallback: local download
    return LOCAL_FALLBACK

Auto-detection. If the volume exists, use it. If not, it works the same way (slower on the first boot). Never assume your infrastructure is perfect.

Problem 3: Two Voice Modes, Two Sets of Parameters

Here’s something no documentation will tell you: TTS models with voice cloning and fine-tuning need completely different sampling parameters depending on the mode.

With a fine-tuned voice (trained specifically on the model), you want conservative parameters:

temperature: 0.5
top_p: 0.8
repetition_penalty: 1.1

With voice cloning (base voice + reference audio), the same parameters produce white noise. Literally. White noise. Because the clone model needs more “creativity” to reconstruct the vocal characteristics:

temperature: 0.9
top_p: 0.95
repetition_penalty: 1.05

I discovered this after hours generating unusable audio. The model’s documentation doesn’t mention it. The academic papers don’t mention it. It’s the kind of knowledge that only comes from experimentation in production.

Problem 4: The Reference Audio Matters More Than the Model

Voice cloning uses an audio clip as a reference to replicate the voice. Sounds simple: record 10 seconds, feed it to the model, done.

Except if the reference audio starts with speech immediately — without even 200 milliseconds of silence at the beginning — the model learns that “the audio starts with no onset”. And every subsequent generation will clip the first syllable.

This is not a model bug. It’s the model doing exactly what you asked for: “generate audio that starts just like this reference”. If the reference starts clipped, the generation starts clipped.

The fix is not post-processing (the audio has already been generated without the syllable). It’s fixing the reference: ensuring at least 300ms of silence before the speech.

For those coming from the software world, this is counterintuitive. We’re used to treating input data as immutable and fixing things on the output. With generative models, input quality is everything.

Problem 5: Timeout and Generation Limits

A TTS model generates audio token by token. If you don’t cap it, it can generate indefinitely — and on RunPod, you pay by the second of GPU. A 30-word turn of dialogue shouldn’t generate 3 minutes of audio, but without a limit, it can.

MAX_NEW_TOKENS = 720  # ~60 seconds of audio at 12Hz
GENERATION_TIMEOUT = 180  # 3 absolute minutes

with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(model.generate, **params)
    result = future.result(timeout=GENERATION_TIMEOUT)

Double protection: token limit on generation and absolute timeout on the process. If either one trips, the turn fails and can be retried with adjusted parameters.

Problem 6: Post-Processing Is Half the Work

The model generates raw WAV. To get to podcast-quality audio, the path is long:

1. Loudness normalization: Spotify requires -14 LUFS. Without normalization, every turn has a different volume.
2. Re-encoding: The model’s WAVs can have subtly different formats (sample rate, bit depth). Concatenating with -c copy causes silent segments. You need to re-encode everything to the same format.
3. Conversion to MP3: CBR 192kbps, 44.1kHz, with ID3 metadata.

And loudness normalization needs to be two-pass: first measure, then apply. Single-pass is a guess — and guessing in production generates audio with inconsistent volume across episodes.

# Pass 1: analysis
ffmpeg -i input.wav -af loudnorm=I=-14:print_format=json -f null -

# Pass 2: applying the measured values
ffmpeg -i input.wav -af loudnorm=I=-14:measured_I=-18.5:measured_TP=-2.1:... output.wav

All of this runs on the server, after TTS generation. And it needs ffmpeg installed in the container — one more dependency to manage.

Problem 7: The Script Matters More Than the Voice Model

After solving GPU, cold start, voice cloning, sampling, normalization — the podcast comes out. And it sounds… robotic. Not because of the TTS. The TTS is excellent. It sounds robotic because the text is robotic.

This is the problem nobody mentions in TTS pipelines: audio quality starts in the script, not in the voice model. A perfect TTS reading a dry, formal text produces dry, formal audio. Garbage in, garbage out — but at a level you wouldn’t expect.

Two Passes: Draft and Refine

The solution is to generate the script in two LLM passes:

Pass 1 (Draft): Turns the newsletter into dialogue. Here the LLM gets the full content and creates the structure — who says what, in what order, with what transitions. Without changing the content, only the form!

Pass 2 (Refine): Takes the draft and makes it natural. Adds hesitations, reactions, interruptions, self-corrections. Turns “read script” into “real conversation”.

Why not do it all in a single pass? Because these are cognitively different tasks. Structuring content and adding naturalness at the same time results in a text that does neither properly. Two specialized passes consistently produce a better result.

Personality Needs Brutally Specific Instructions

“Make Akita sound confident” doesn’t work. The LLM will produce generic confidence — motivational phrases, positive tone. That’s not it. Akita is confident because he’s assertive and impatient. The difference is in the prompt details:

## Akita Delivery Style

- **Declarative, not tentative**: "Isso é óbvio" not "Eu acho que talvez isso seja..."
- **Dismissive of nonsense**: "Isso é bobagem", "Não tem nenhum mistério aqui"
- **Owns his track record**: "Eu avisei", "Quem assistiu meu canal já sabe"
- **Never hedges**: No "talvez", "quem sabe", "pode ser que"
- **Impatient with the obvious**: "Vocês já sabem como funciona"

And for Marvin (the robot co-host):

- **Filler words** (Marvin only): "bom...", "olha...", "enfim", "pois é"
- **Self-corrections** (Marvin only): "quer dizer...", "ou melhor..."
- Akita doesn't second-guess himself — if he corrects, it's
  "Não, peraí" and he restates with even MORE conviction

Without this level of specificity, both characters sound the same. The LLM needs concrete examples of how each one talks, not abstract personality descriptions.

Normalization for Speech: What Looks Right in Text Sounds Wrong in Audio

Acronyms. The invisible enemy of TTS. “EUA” in text is perfectly readable. Spoken by TTS, it comes out “éu-á” — incomprehensible. The prompt needs explicit normalization rules:

## Spoken Text Normalization (MANDATORY)

- `EUA` → `Estados Unidos`
- `UE` → `União Europeia`
- `IA` → `inteligência artificial`
- `LLM` → `modelo de linguagem`
- `API` → `interface de programação`

This needs to be in the prompt for both passes (draft and refine), because the LLM tends to reintroduce acronyms it removed in the previous pass.

The LLM Will Rewrite Your Opinions (If You Let It)

This was the most subtle lesson. The newsletter has opinion commentary from Akita and Marvin in each section. When the LLM turns it into dialogue, it substitutes the opinions with more “balanced” and “nuanced” versions. Akita, who said “isso é bobagem”, becomes Akita saying “isso tem seus méritos, mas…”. Pessimistic Marvin becomes “mildly skeptical” Marvin.

The LLM does this because it was trained to be helpful and balanced. Strong opinions seem “unhelpful” to the model, so it softens them. The original instruction — “use the newsletter comments as seeds for the conversation” — was too vague. It gave the model freedom to reinterpret.

The fix is to be explicit and inflexible:

**PRESERVE THE ORIGINAL OPINIONS**: The newsletter contains Akita and
Marvin commentary on each topic. These opinions are the author's real
voice — you MUST preserve their substance, stance, and conclusions.
Make the delivery conversational, but do NOT change what they actually
think. If Akita's comment says "isso é bobagem", the podcast version
must convey the same dismissal. Rephrase for spoken delivery, but
never substitute a different opinion.

It’s the difference between “adapt it for conversation” (LLM understands: “rewrite however you want”) and “change the HOW, never the WHAT” (LLM understands: “keep the opinion, swap the packaging”).

And this applies to any prompt engineering with authored content: the LLM will homogenize your voice if you’re not explicit about preserving it. And in a podcast where the hosts’ personality IS the product, losing the voice means losing everything.

The Real Cost of AI in Production

Let’s get practical about costs on RunPod:

The cost per episode is low. The cost of figuring out how to make it work is high. It took dozens of hours of experimentation, incomprehensible CUDA logs, failed generations, and unusable-quality audio before landing on a stable pipeline.

And the cognitive cost: keeping two ecosystems (Python for ML, Ruby for the application) in sync, with HTTP interfaces between them, and debugging that crosses language boundaries.

Anyway, for those who thought “Why not Elevenlabs?”, well, I got similar quality at a fraction of the cost, and I learned a lot in the process (and so did you, reading this).

Lessons

1. Cold start is a requirement, not a bug. Design your system expecting that the model will take time to load.

2. Network volumes are mandatory for large models. Don’t bake weights into Docker, don’t download on boot.

3. Sampling parameters are not universal. Each operation mode of the same model may need completely different configuration.

4. Input quality dominates output quality. Both for the TTS reference audio and for the script. If the reference clips the first syllable, every generation clips. If the script is generic, the audio is generic.

5. Timeouts at every layer. Token limit, process timeout, HTTP timeout. If you don’t cap it, cost scales and generation hangs.

6. Post-processing is half the pipeline. The model generates raw material. Normalization, encoding, metadata — all of that is real work.

7. Separate the ML server from the application. Python for inference, Ruby for orchestration. Trying to do everything in one language is a recipe for frustration.

8. Prompts need two passes for complex content. Structuring and naturalizing are different tasks. Forcing both into one pass produces mediocre results in both.

9. The LLM will homogenize your voice. Strong opinions turn into “nuanced takes” if you’re not explicit about preserving them. Instruct the model to change the packaging, never the content.

Conclusion

AI in production is not a 10-line HuggingFace tutorial. It’s infrastructure, it’s experimentation, it’s debugging problems that don’t exist in traditional software. But when it works — when your system generates an entire podcast automatically every week, with broadcast quality, no human intervention — the feeling is that it was worth every hour of headache.

The trick is not to underestimate the complexity. And to have a robust system (like the Rails jobs I mentioned in the first post) orchestrating it all — because AI without reliable orchestration is just an expensive script that hangs.

This way I can access https://portainer.fabioakita.dev (fake example domain) from my phone on 5G, and it hits the portainer Docker container sitting on my LAN at http://192.168.0.200:9000, which was previously only reachable from home. All through a secure tunnel.

Note that I keep saying I’m using fake example domains and IPs throughout these posts, because I don’t want to expose the real domain.

Just by sitting behind a Cloudflare secure tunnel, I’m already under several layers of protection like DDoS defense and many other well-known attack mitigations. And although I take care to configure every service with a strong, random, unique password per service, the risk isn’t zero that someone could find my services and try to brute-force them, or exploit some zero-day vulnerability in one of the many services like Prowlarr, Sonarr, and friends.

So I’m still at the mercy of undiscovered bugs that might pop up out of nowhere, particularly on each service’s login screen or API endpoints.

To avoid that risk, there’s another Cloudflare product worth configuring: Zero Trust. The idea is simple: add a new login layer on top of everything, from Cloudflare itself, to protect each service. That way, someone has to break Cloudflare before they even get a chance to try breaking one of my services.

1. Sign up for the service

This is the easy part. For few services and few users (just me), there’s a free plan, just sign up:

2. Create OAuth Credentials in Google Cloud

Since I pay for Google Workspace, I decided to use integrated login with Google OAuth, but it can be any other provider, pick whatever you prefer. For this I need to go to my Google Cloud console and enter the API & Services section. Here we create credentials for an OAuth client ID and configure it like this:

The name doesn’t matter, it’s what shows up on the login screen. What matters are the Javascript Origins fields, which must follow this format:

https://<your-team-name>.cloudflareaccess.com

And the redirect URIs field, which is where Google’s login screen will redirect to, and it has to point back to our Cloudflare:

https://<your-team-name>.cloudflareaccess.com/cdn-cgi/access/callback

Once that’s done, it gives you the option to download a JSON file with the metadata and secret. What’s important is that at the end it gives you an OAuth Client ID, which is a URL in the format xxxxxx.apps.googleusercontent.com, and most importantly, a Client Secret, which you should store in BitWarden or another secure place.

4. Creating a Login Method in Zero Trust

Now back to the Zero Trust console on Cloudflare, click “Settings” at the bottom of the menu, enter “Authentication”, and create a new Login Method:

After clicking “Add new”, just fill in the data we brought from Google Cloud:

Again, use whatever name you want, paste the OAuth Client ID as “App ID” and the Client Secret, and that’s it, now we have Google OAuth integrated as a login method in our Zero Trust.

5. Creating Policies

Now comes the part where I’m not sure if I did it entirely right. If I didn’t, let me know in the comments below!

My understanding is that we need to create at least two Policies, one for Service Auth and another for the Email I want to allow coming in from Google.

To continue, just click “Add a Policy” and edit the first policy:

The name can be anything. What matters is that under “Action” you pick “Service Auth” and under “Add Rules” you pick Any Access Service Token (this is the part I think is wrong, but I couldn’t find a better option - I didn’t quite understand how Service Token works).

Now we need to create the second policy like this:

This time what matters is having “Action” as “Allow” and adding the Rule “Emails”, picking the email you want to authenticate. In my case, just me.

6. Creating a new Application

Now we can create a new application in Zero Trust:

Click “Add an Application” and pick the type as “Self-Hosted”:

My understanding is that each application supports at most 5 hostnames. I have quite a few more than that, so I’ll have to create more than one application. The configuration in all of them is the same. Under “Basic Information” I just registered the subdomains of each container I want to protect.

Going to the “Policies” tab, just click “Select existing policies” and add the two policies we just created. That’s it.

Finally, under “Login methods”, just pick the Google one we configured earlier. And yes, these are the editing screens of an application I had already created. The new application screen has all of this on a single page, but you get the idea.

And as far as I understand, that’s all. Theoretically, everything is already working!

7. Testing the new Login

If nothing went wrong, just hit one of the hostnames we registered in the Zero Trust application, for example https://immich.fabioakita.dev, and we should see this screen first:

If this shows up, success! Zero Trust blocked access to my application until I logged in to it first.

Later you can customize the text however you want, but since it’s just for me, the “welcome” copy doesn’t matter, because actually, nobody is welcome here, just ME!

Clicking “Google” opens the familiar Google login pop-up, and only after that do I get to reach my application’s own login underneath. And this applies to every hostname I added to the Zero Trust applications.

Once you log in with Google using the correct user, the session stays valid for 24 hours (which is what we configured in the policies and application). During that window you don’t need to log in again on Zero Trust for any of the other applications on my home server. The session is shared across all applications.

8. Logging out Sessions

Once you’ve done the first login with your user, you can control that session by going to “My team” and “Users”:

Clicking on the user, you get a bunch of info and options, in particular see “Revoke Sessions”.

This way we can test the login without waiting 24 hours for the auth token to expire on its own. It’s a way to force logout your users. On the Zero Trust free plan, from what I understand, you can have up to 10 users in your organization.

Conclusion

I had never used Zero Trust before, so I might have made some mistake in the procedure. Read the documentation too. In principle, it seems to be working. I already created more applications and registered the rest of my hostnames.

Just by using CloudflareD for encrypted tunnels, all traffic reaching my home server goes through a series of Cloudflare protections that include: DDoS protection, URL normalization, Origin Rules, IP Address Rules, WAF (Web Application Firewall), Bots Mitigation, Rate Limiting, Cache, Compression and more.

These layers protect against the main well-known attack types, and they’re always adding new rules as each new vulnerability becomes known. Even if you, as a developer, don’t know how to protect against everything in your application, there’s a chance Cloudflare protects you first. That’s why I recommend keeping your domains under Cloudflare’s control.

On top of all of that, we also add Zero Trust, which guarantees that only after a login with my Google account will it start allowing access further down. And at the end of that there’s still one last barrier: the individual login screens of each application running on my home server.

Beyond that, the Cloudflare dashboard has traffic monitoring, analytics and much more. I configured just the basics, but if you want to go deeper, there are plenty of extra policies and rules you can add.

For someone to get through all of this and actually break into my home server and my LAN, they’d have to be really dedicated and really smart - I can’t picture anyone like that wasting so much time just to access a video server 🤣 but they’re welcome to try.

TL;DR: Google is actually doing a pretty good job being a counterbalancer so AWS doesn’t slack off. If you already know everything about AWS, I’d encourage you to test out Google Cloud. Possibly because of muscle memory, I would still be more comfortable with AWS, but now that I forced myself to suffer the learning process, I am pretty confident with Google Cloud and Kubernetes for most of my scenarios.

Full disclaimer that I am not an expert, so take what I say with a grain of salt. This is one of those subjects that I am super eager to talk about but I’m also very reluctant on the proper choice of words so you don’t get the wrong idea about the proposed solutions.

The goal of this exercise is mainly for me to store some snippets and thoughts for future reference. So keep in mind that this is also not a step-by-step tutorial. My first intention was to go that way, but then I realized that it would be almost like writing an entire book, so not this time.

To succeed with something like Google Cloud and Kubernetes you must be battle tested in infrastructure. If you never installed server-grade Linux boxes from scratch, if you never did server optimizations, if you’re not comfortable with bare server-side components, do not attempt a real production deployment. Your safest bet is still something like Heroku.

You have to be that kinda person that likes to tinker (as you’ve probably read me doing in previous blog posts).

I don’t know everything, but I know enough. So I only had to figure out which of the pieces would closely fit my needs. You must outline your needs before attempting to write your first YAML file. Planning is crucial.

First things first, this is what I wanted/needed:

• Scalable web application tier, where I could do both rolling updates (for zero downtime updates) and automatic and manual horizontal scaling of the servers.
• Mountable persistent storage with automatic snapshots/backups.
• Managed robust database (Postgresql) with automatic backups and easy replication to read-only instances.
• Managed solution to store secrets (such as Heroku’s ENV support). Never store production configuration in the source code.
• Docker images support without me having to build custom infrastructure to deploy.
• Static external IP addresses for integrations that required a fixed IP.
• SSL termination so I could connect to CloudFlare (CDN is mandatory, but not enough, in 2018 we need some level of DDoS protection).
• Enough security by default, so everything is - in theory - lockdown unless I decide to open it.
• High-availability in different data center regions and zones.

It’s easy to deploy a simple demo web application. But I didn’t want a demo, I wanted a production-grade solution for the long-run. Improvements to my implementation are super welcome, so feel free to comment down below.

Some of the problems for newcomers:

• The documentation is very extensive, and you will find almost everything - if you know what you are looking for. Also bear in mind that Azure and AWS also implement Kubernetes with some differences, so some documentation doesn’t apply to Google Cloud and vice-versa.
• There are many features in alpha, beta, and stable stages. The documentation kinda keeps up well, but most tutorials that are just a couple months old may not work as intended anymore (this one included - I am assuming Kubernetes 1.8.4-gke).
• There is a whole set of words that apply to concepts you already know but are called differently. Getting used to the vocabulary may get in the way at first.
• It feels like you’re playing with Lego. Lots of pieces that you can mix and match. It’s easy to mess up. This means that you can build a configuration tailored to your needs. But if you just copy and paste from tutorials you will get stuck.
• You can do almost everything through YAML files and the command line, but it’s not trivial to reuse the configuration (for production and staging environments, for example). There are 3rd party tools that deal with parameterizable and reusable YAML bits, but I’d do it all by hand first. Never, ever, try automated templates in infrastructure without knowing exactly what they are doing.
• You have 2 fat command line tools: gcloud and kubectl, and the confusing part is that they name some things differently even though they are the same “things”. At least, kubectl is close to docker, if you’re familiar with that.

Once again, this is NOT a step-by-step tutorial. I will annotate a few steps but not everything.

Scalable Web-Tier (the Web App itself)

The very first thing you must have is a fully 12-factor compliant web app.

Be it Ruby on Rails, Django, Laravel, Node.js or whatever. It must be a fully shared-nothing app, that does not depend on writing anything to the local filesystem. One where you can easily shut down and start up instances independently. No old-style session in local memory or in local files (I prefer to avoid session affinity). No uploads to the local filesystem (if you must, you will have to mount an external persistent storage), always prefer to send binary streams to managed storage services.

You must have a proper pipeline that outputs cache-busting through fingerprinting assets (and like it or not, Rails still has the best out-of-the-box solution in its Asset Pipeline). You don’t want to worry about manually busting caches in CDNs.

Instrument your app, add New Relic RPM, add Rollbar.

Again, this is 2018, you don’t want to deploy naive code with SQL (or any other input) injection, no unchecked eval around your code, no room for CSRF or XSS, etc. Go ahead, buy the license for Brakeman Pro and add it to your CI pipeline. I can wait …

As this is not a tutorial, I will assume you’re more than able to sign up to Google Cloud and find your way to set up a project, configure your region and zone.

It took me a while to wrap my head around the initial structure in Google Cloud:

• You start with a Project, which is the umbrella for everything your app needs.
• Then you create “clusters”. You can have a production or staging cluster, for example. Or a web cluster and a separated services cluster for non-web stuff, and so on.
• A cluster has a “cluster-master”, which is the controller of everything else (the gcloud and kubectl commands talk to its APIs).
• A cluster has many “node instances”, the proper “machines” (or, more accurately, VM instances).
• Each cluster also has at least one “node pool” (the “default-pool”), which is a set of node instances with the same configuration, the same “machine-type”.
• Finally, each node instance runs one or more “pods” which are lightweight containers like LXC. This is where your application actually is.

This is an example of creating a cluster:

gcloud container clusters create my-web-production \
--enable-cloud-logging \
--enable-cloud-monitoring \
--machine-type n1-standard-4 \
--enable-autoupgrade \
--enable-autoscaling --max-nodes=5 --min-nodes=2 \
--num-nodes 2

As I mentioned, it also creates a default-pool with a machine-type of n1-standard-4. Choose what combination of CPU/RAM you will need for your particular app beforehand. The type I chose has 4 vCPUs and 15GB of RAM.

By default, it starts with 3 nodes, so I chose 2 at first but auto-scalable to 5 (you can update it later if you need to, but make sure you have room for initial growth). And you can keep adding extra node-pools for differently sized node instances, let’s say, for Sidekiq workers to do heavy-duty background processing. Then you should create a separated Node Pool with a different machine-type for its set of node instances, for example:

gcloud container node-pools create large-pool \
--cluster=my-web-production \
--node-labels=pool=large \
--machine-type=n1-highcpu-8 \
--num-nodes 1

This other pool controls 1 node of type n1-highcpu-8 which has 8 vCPUs with 7.2 GB of RAM. More CPUs, less memory. You have a category of highmem which is less CPUs with a whole lot more memory. Again, know what you want beforehand.

The important bit here is the --node-labels this is how I will map the deployment to choose between Node Pools (in this case, between the default-pool and the large-pool).

Once you create a cluster, you must issue the following command to fetch its credentials:

gcloud container clusters get-credentials my-web-production

This sets the kubectl command as well. If you have more than one cluster (let’s say, one my-web-production and my-web-staging), you must be very careful to always get-credentials for the correct cluster first, otherwise, you may end up running a staging deployment on the production cluster.

Because this is confusing, I modified my ZSH PROMPT to always show which cluster I am dealing with. I adapted from zsh-kubectl-prompt:

As you will end up having multiple clusters in a big app, I highly recommend you add this PROMPT to your shell.

Now, how do you deploy your application to the pods within those fancy node instances?

You must have a Dockerfile in your application project repository to generate a Docker image. This is one example for a Ruby on Rails application:

FROM ruby:2.4.3
ENV RAILS_ENV production
ENV SECRET_KEY_BASE xpto
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get update && apt-get install -y nodejs postgresql-client cron htop vim
ADD Gemfile* /app/
WORKDIR /app
RUN gem update bundler --pre
RUN bundle install --without development test
RUN npm install
ADD . /app
RUN cp config/database.yml.prod.example config/database.yml && cp config/application.yml.example config/application.yml
RUN RAILS_GROUPS=assets bundle exec rake assets:precompile

From the Google Cloud Web Console, you will find a “Container Registry”, which is a Private Docker Registry.

You must add the remote URL to your local config like this:

git remote add gcloud https://source.developers.google.com/p/my-project/r/my-app

Now you can git push gcloud master. I recommend you also add triggers to tag your images. I add 2 triggers: one to tag it with latest and another to tag it with a random version number. You will need those later.

Once you add the registry repository as a remote on your git configuration (git remote add) and push to it, it should start building your Docker image with the proper tags you configured with the triggers.

Make sure your Ruby on Rails application doesn’t have anything in the initializers that require a connection to the database, as it’s not available. This is something you might get stuck with when your Docker build fails because of the assets:precompile task loaded an initializer that accidentally calls a Model - and that triggers ActiveRecord::Base to try to connect.

Also, make sure the Ruby version in the Dockerfile matches the one in Gemfile, otherwise it will also fail.

Notice the weird config/application.yml above? This is from figaro. I also recommend you using something to make it easy to configure ENV variables in your system. I don’t like Rails secrets, and it’s not exactly friendly to most deployment systems after Heroku made ENV vars ubiquitous. Stick to ENV vars. Kubernetes will also thank you for that.

Now, you can override any ENV variable from the Kubernetes Deployment YAML file. Now it’s a good time to show an example of that. You can name it deploy/web.yml or whatever suits your fancy and - of course - check it into your source code repository.

kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: web
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 10
  replicas: 2
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - image: gcr.io/my-project/my-app:latest
          name: my-app
          imagePullPolicy: Always
          ports:
          - containerPort: 4001
          command: ["passenger", "start", "-p", "4001", "-e", "production",
          "--max-pool-size", "2", "--min-instances", "2", "--no-friendly-error-pages"
          "--max-request-queue-time", "10", "--max-request-time", "10",
          "--pool-idle-time", "0", "--memory-limit", "300"]
          env:
            - name: "RAILS_LOG_TO_STDOUT"
              value: "true"
            - name: "RAILS_ENV"
              value: "production"
            # ... obviously reduced the many ENV vars for brevity
            - name: "REDIS_URL"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: REDIS_URL
            - name: "SMTP_USERNAME"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: SMTP_USERNAME
            - name: "SMTP_PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: SMTP_PASSWORD
            # ... this part below is mandatory for Cloud SQL
            - name: DB_HOST
              value: 127.0.0.1
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username

        - image: gcr.io/cloudsql-docker/gce-proxy:latest
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=my-project:us-west1:my-db=tcp:5432",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:

There is a lot going on here. So let me break it down a bit:

• The kind, and apiVersion is important, you have to keep an eye on the documentation if those change. This is what is called a Deployment. There used to be a Replication Controller (you will find those in old tutorials), but it’s no longer in use. The recommendation is to use a ReplicaSet.
• Name things correctly, here you have metadata:name with web. Also pay close attention to spec:template:metadata:labels where I am labeling every pod having a label of app: web, you will need this to be able to select those pods later in the Service section down below.
• Then I have spec:strategy where we configure the Rolling Update, so if you have 10 pods, it will terminate one, boot up the new one and keep doing that, without never bringing everything down at once.
• spec:replicas declares how many Pods I want at once. You will have to manually calculate the machine-type of the node-pool then divide how many total CPUs/RAM you have by how much you need for each application instance.
• Remember the Docker image we generated above with the ’latest’ tag? You refer to it in spec:template:spec:containers:image
• I am using Passenger with production configuration (check out Phusion’s documentation, do not just copy this).
• In the spec:template:spec:containers:env section I can override the ENV vars with the real production secrets. And you will notice that I can hard-code values or use this strange contraption:

- name: "SMTP_USERNAME"
  valueFrom:
    secretKeyRef:
      name: my-env
      key: SMTP_USERNAME

Now, it’s referencing a “Secret” storage that I named “my-env”. And this is how you create your own:

kubectl create secret generic my-env \
--from-literal=REDIS_URL=redis://foo.com:18821 \
--from-literal=SMTP_USERNAME=foobar

Read the documentation as you can load text files instead of declaring everything from the command line.

As I said before, I’d rather use a managed service for a database. You can definitely load your own Docker image, but I really don’t recommend it. Same goes for other database-like services such as Redis, Mongo. If you’re from AWS, Google Cloud SQL is like RDS.

After you create your PostgreSQL instance you can’t access it directly from the web application. In the end, you have a boilerplate for a second Docker image, a “CloudSQL Proxy”.

For that to work you must first create a Service Account:

gcloud sql users create proxyuser host --instance=my-db --password=abcd1234

After you create the PostgreSQL instance it will prompt you to download a JSON credential, so be careful and save it somewhere safe. I don’t have to say that you must generate a strong secure password as well. Then you must create extra secrets:

kubectl create secret generic cloudsql-instance-credentials \
--from-file=credentials.json=/home/myself/downloads/my-db-12345.json

kubectl create secret generic cloudsql-db-credentials \
--from-literal=username=proxyuser --from-literal=password=abcd1234

These are referenced in this part of the Deployment:

- image: gcr.io/cloudsql-docker/gce-proxy:latest
  name: cloudsql-proxy
  command: ["/cloud_sql_proxy", "--dir=/cloudsql",
            "-instances=my-project:us-west1:my-db=tcp:5432",
            "-credential_file=/secrets/cloudsql/credentials.json"]
  volumeMounts:
    - name: cloudsql-instance-credentials
      mountPath: /secrets/cloudsql
      readOnly: true
    - name: ssl-certs
      mountPath: /etc/ssl/certs
    - name: cloudsql
      mountPath: /cloudsql
volumes:
- name: cloudsql-instance-credentials
  secret:
    secretName: cloudsql-instance-credentials
- name: ssl-certs
  hostPath:
    path: /etc/ssl/certs
- name: cloudsql
  emptyDir:

See that you must add the database name (“my-db” in this example) in the -instance clause in the command.

And by the way, the gce-proxy:latest refers to version 1.09 at the time when this post was published. But there already was a 1.11 version. That one gave me headaches, dropping connections and adding a super long timeout. So I went back to the 1.09 (latest) and everything worked as expected. So be aware! Not everything that is brand new is good. In infrastructure, you want to stick to stable.

You may also want the option to load a separated CloudSQL instance instead of having it in each pod, so the pods could connect to just one proxy. You may want to read this thread on the subject.

It seems that nothing is exposed to anything unless you say so. So we need to expose those pods through what’s called a Node Port Service. Let’s create a deploy/web-svc.yaml file as well:

apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  sessionAffinity: None
  ports:
  - port: 80
    targetPort: 4001
    protocol: TCP
  type: NodePort
  selector:
    app: web

This is why I highlighted the importance of the spec:template:metadata:labels, so we can use it here in the spec:selector to select the proper pods.

We can now deploy these 2 like this:

kubectl create -f deploy/web.yml
kubectl create -f deploy/web-svc.yml

And you can see the pods being created with kubectl get pods --watch.

The Load Balancer

Many tutorials will expose those pods directly through a different Service, called Load Balancer. I am not so sure how well this behaves under pressure and if it has SSL termination, etc. So I decided to go full blown with an Ingress Load Balancer using the NGINX Controller, instead.

First of all, I decided to create a separated node-pool for it, for example, like this:

gcloud container node-pools create web-load-balancer \
--cluster=my-web-production \
--node-labels=role=load-balancer \
--machine-type=g1-small \
--num-nodes 1 \
--max-nodes 3 --min-nodes=1 \
--enable-autoscaling 

As when we created the example large-pool here you must take care of adding --node-labels to make the controller be installed here instead of the default-pool. You will need to know the node instance name, we can do it like this:

$ gcloud compute instances list
NAME                                             ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP      STATUS
gke-my-web-production-default-pool-123-123       us-west1-a  n1-standard-4               10.128.0.1   123.123.123.12   RUNNING
gke-my-web-production-large-pool-123-123         us-west1-a  n1-highcpu-8                10.128.0.2   50.50.50.50      RUNNING
gke-my-web-production-web-load-balancer-123-123  us-west1-a  g1-small                    10.128.0.3   70.70.70.70      RUNNING

Let’s save it like this for now:

export LB_INSTANCE_NAME=gke-my-web-production-web-load-balancer-123-123

You can manually reserve an external IP and give it a name like this:

gcloud compute addresses create ip-web-production \
        --ip-version=IPV4 \
        --global

For the sake of the example, let’s say that it generated a reserved IP “111.111.111.111”. Then let’s fetch it and save it for now like this:

export LB_ADDRESS_IP=$(gcloud compute addresses list | grep "ip-web-production" | awk '{print $3}')

Finally, let’s hook this address to the load balancer node instance:

export LB_INSTANCE_NAT=$(gcloud compute instances describe $LB_INSTANCE_NAME | grep -A3 networkInterfaces: | tail -n1 | awk -F': ' '{print $2}')
gcloud compute instances delete-access-config $LB_INSTANCE_NAME \
    --access-config-name "$LB_INSTANCE_NAT"
gcloud compute instances add-access-config $LB_INSTANCE_NAME \
    --access-config-name "$LB_INSTANCE_NAT" --address $LB_ADDRESS_IP

Once we do this, we can add the rest of the Ingress Deployment configuration. This will be kinda long but it’s mostly boilerplate. Let’s start by defining another web application that we will call default-http-backend that will be used to respond to HTTP requests in case our web pods are not available for some reason. Let’s call it deploy/default-web.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

No need to change anything here, and by now you may be familiar with the Deployment template. Again, you now know that you need to expose it through a NodePort, so let’s add a deploy/default-web-svc.yml:

kind: Service
apiVersion: v1
metadata:
  name: default-http-backend
spec:
  selector:
    app: default-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: NodePort

Again, no need to change anything. The next 3 files are the important parts. First, we will create an NGINX Load Balancer, let’s call it deploy/nginx.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-lb
    spec:
      # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
      # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
      # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
      hostNetwork: true
      terminationGracePeriodSeconds: 60
      nodeSelector:
        role: load-balancer
      containers:
        - args:
            - /nginx-ingress-controller
            - "--default-backend-service=$(POD_NAMESPACE)/default-http-backend"
            - "--default-ssl-certificate=$(POD_NAMESPACE)/cloudflare-secret"
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          image: "gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.5"
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 5
          name: nginx-ingress-controller
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
            - containerPort: 443
              name: https
              protocol: TCP
          volumeMounts:
            - mountPath: /etc/nginx-ssl/dhparam
              name: tls-dhparam-vol
      volumes:
        - name: tls-dhparam-vol
          secret:
            secretName: tls-dhparam

Notice the nodeSelector to make the node label we added when we created the new node-pool.

You may want to tinker with the labels, the number of replicas if you need to. But here you will notice that it mounts a volume that I named as tls-dhparam-vol. This is a Diffie Hellman Ephemeral Parameters. This is how we generate it:

sudo openssl dhparam -out ~/documents/dhparam.pem 2048

kubectl create secret generic tls-dhparam --from-file=/home/myself/documents/dhparam.pem

kubectl create secret generic tls-dhparam --from-file=/home/myself/documents/dhparam.pem

Also, notice that I am using version “0.9.0-beta_5” for the controller image. It works well, no problems so far. But keep an eye on release notes for newer versions as well and do your own testing.

Again, let’s expose this Ingress controller through the Load Balancer Service. Let’s call it deploy/nginx-svc.yml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
spec:
  type: LoadBalancer
  loadBalancerIP: 111.111.111.111
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
  selector:
    k8s-app: nginx-ingress-lb

Remember the static external IP we have reserved above and saved in the LB_INGRESS_IP ENV var? This is the one we must put in the spec:loadBalancerIP section. This is also the IP that you will add as an “A record” in your DNS service (let’s say, mapping your “www.my-app.com.br” on CloudFlare).

Finally, we can create the Ingress configuration itself, let’s create a deploy/ingress.yml like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/ssl-services: "web-svc"
    kubernetes.io/ingress.global-static-ip-name: ip-web-production
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
    - hosts:
      - www.my-app.com.br
      secretName: cloudflare-secret
  rules:
    - host: www.my-app.com.br
      http:
        paths:
        - path: /
          backend:
            serviceName: web-svc
            servicePort: 80

Be careful with the annotations above that hooks everything up. It binds the NodePort service we created for the web pods with the nginx ingress controller and adds SSL termination through that spec:tls:secretName. How do you create that? First, you must purchase an SSL certificate - again, using CloudFlare as the example.

When you finish buying, the provider should give you the secret files to download (keep them safe! a public dropbox folder is not safe!). Then you have to add it to the infrastructure like this:

kubectl create secret tls cloudflare-secret \
--key ~/downloads/private.pem \
--cert ~/downloads/fullchain.pem

Now that we edited a whole bunch of files, we can deploy the entire load balancer stack:

kubectl create -f deploy/default-web.yml
kubectl create -f deploy/default-web-svc.yml
kubectl create -f deploy/nginx.yml
kubectl create -f deploy/nginx-svc.yml
kubectl create -f deploy/ingress.yml

This NGINX Ingress configuration is based off of Zihao Zhang’s blog post. There are also examples in the kubernetes incubator repository. You may want to check it out as well.

If you did everything right so far, https://www.my-app-com.br should load your web application. You may want to check for Time to First-Byte (TTFB). You can do it going through CloudFlare like this:

curl -vso /dev/null -w "Connect: %{time_connect} \n TTFB: %{time_starttransfer} \n Total time: %{time_total} \n" https://www.my-app.com.br

Or, if you’re having slow TTFB you can bypass CloudFlare doing this:

curl --resolve www.my-app.com.br:443:111.111.111.111 https://www.my-app.com.br -svo /dev/null -k -w "Connect: %{time_connect} \n TTFB: %{time_starttransfer} \n Total time: %{time_total} \n"

TTFB should be in the neighborhood of 1 second or less. Anything far and above could mean a problem in your application. You must check your node instance machine types, the number of workers loaded per pod, the CloudSQL proxy version, the NGINX controller version and so on. This is a trial and error procedure as far as I know. Sign up to services such as Loader or even Web Page Test for insight.

Rolling Updates

Now, that everything is up and running, how do we accomplish the Rolling Update I mentioned in the beginning? First you git push to the Container Registry repository and wait for the Docker image to build.

Remember that I said to let a trigger tag the image with a random version number? Let’s use it (you can see it from the Build History list in the Container Registry, from the Google Cloud console):

kubectl set image deployment web my-app=gcr.io/my-project/my-app:1238471234g123f534f543541gf5 --record

You must use the same name and image that is declared in the deploy/web.yml from above. This will start rolling out the update by adding a new pod, then terminating one pod and so on and so forth until all of them are updated, without downtime for your users.

Rolling updates must be carried out carefully. For example, if your new deployment requires a database migration, then you must add a maintenance window (meaning: do it when there is little to no traffic, such as in the middle of the night). So you can run the migrate command like this:

kubectl get pods # to get a pod name

kubectl exec -it my-web-12324-121312 /app/bin/rails db:migrate

# you can also bash to a pod like this, but remember that this is an ephemeral container, so files you edit and write there disappear on the next restart:

kubectl exec -it my-web-12324-121312 bash

To redeploy everything without resorting to rolling update you must do this:

kubectl delete -f deploy/web.yml && kubectl apply -f deploy/web.yml

You will find a more thorough explanation in Ta-Ching’s blog post.

Bonus: Auto Snapshots

One item I had in my “I wanted/needed” list, in the beginning, is the ability to have persistent mountable storage with automatic backups/snapshots. Google Cloud provides half of that for the time being. You can create persistent disks to mount in your pods but it doesn’t have a feature to automatically back it up. At least it does have APIs to manually snapshot it.

For this example, let’s create a new SSD disk and format it first:

gcloud compute disks create --size 500GB my-data --type pd-ssd

gcloud compute instances list

The last command is so we can copy the name of a node instance. Let’s say it’s gke-my-web-app-default-pool-123-123. We will attach the my-data disk to it:

gcloud compute instances attach-disk gke-my-web-app-default-pool-123-123 --disk my-data --device-name my-data

gcloud compute ssh gke-my-web-app-default-pool-123-123

The last command ssh’s in the instance. We can list the attached disks with sudo lsblk and you will see the 500GB disk, probably, as /dev/sdb, but make sure that’s correct because we will format it!

sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb

Now we can exit from the SSH session and detach the disk:

gcloud compute instances detach-disk gke-my-web-app-default-pool-123-123 --disk my-data

You can mount this disk in your pods by adding the following to your deployment yaml:

spec:
  containers:
    - image: ...
      name: my-app
      volumeMounts:
        - name: my-data
          mountPath: /data
          # readOnly: true
   # ...
   volumes:
     - name: my-data
       gcePersistentDisk:
         pdName: my-data
         fsType: ext4

Now, let’s create a CronJob deployment file as deploy/auto-snapshot.yml:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: auto-snapshot
spec:
  schedule: "0 4 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: auto-snapshot
            image: grugnog/google-cloud-auto-snapshot
            command: ["/opt/entrypoint.sh"]
            env:
            - name: "GOOGLE_CLOUD_PROJECT"
              value: "my-project"
            - name: "GOOGLE_APPLICATION_CREDENTIALS"
              value: "/credential/credential.json"
            volumeMounts:
              - mountPath: /credential
                name: editor-credential
          volumes:
            - name: editor-credential
              secret:
                secretName: editor-credential

As we already did before, you will need to create another Service Account with editor permissions in the “IAM & admin” section of the Google Cloud console, then download the JSON credential, and finally upload it like this:

kubectl create secret generic editor-credential \
--from-file=credential.json=/home/myself/download/my-project-1212121.json

Also notice that, as a normal cron job, there is a schedule parameter that you might want to change. In the example, “0 4 * * *” means that it will run the snapshot every day at 4 AM.

Check out the original repository of this solution for more details.

And this should be it for now!

As I said, in the beginning, this is not a complete procedure, just highlights of some of the important parts. If you’re new to Kubernetes you just read about Deployment, Service, Ingress, but you have ReplicaSet, DaemonSet, and much more to play with.

I think this is also already too long to add a multi-region High Availability setup explanation, so let’s leave it at that.

Any corrections or suggestions are more than welcome as I am still in the learning process, and there are a ton of things that I still don’t know myself.

Update 07/20/2017: I need to clarify a few points. First, I am recommending Cloudinary’s proprietary solution, but there are caveats. You’re trading off ease of use for vendor lock-in. And I only recommend it for photos/images. If you need to upload super-large files (videos, or arbitrary binaries such as big tarballs) you need to research more and perhaps build a custom solution with Shrine or similar. Another statement I make is that if you use Heroku your only option is Direct Upload to Cloud services such as AWS S3. I delivered several apps that did hit the H12 Routing Timeout. It’s in the official documentation. But right now I deployed a simple dumb Rails 5.1 app with vanilla Carrierwave and an HTML form multipart. I was able to upload a 2+GB video file and it went through Heroku’s routing layer. It seems the router picks up the file and lets it finish then passes the finished uploaded file to the Rails app. Now, if you block the request (such as uploading from the Rails app to S3 or doing something else that is time intensive) then the routing layer will H12/timeout the request after 30sec. So you will eventually hit random disconnects if you allow any kind of large file to be uploaded. Even though you can receive it, you can’t keep it in the file system (as it’s volatile in a Heroku Dyno), so you still need to upload it somewhere and that is the part that can hit the timeout. So Direct Upload is still the way to go on Heroku.

DHH just announced a brand new feature for the upcoming Rails 5.2. This is ActiveStorage.

It should become the default solution to support file uploads. It basically supersedes Paperclip, or Carrierwave, or some features of Dragonfly and Shrine (they do a lot more).

I am writing this small post not to introduce the solution but to clarify a few criticisms I made on Twitter. But Twitter is a terrible platform for more in-depth discussions, hence this post.

The original Twitter thread can be found here.

The process has at least 3 important points to consider:

1. The upload of the file, from the end-user to your Rails controller.
2. An optional transformation step (that you should do, to resize images to be served to different devices and screen resolutions)
3. Serving the blob of the file back to the end-user as an image URL, for example.

The Upload Step

Most simple upload solutions - such as ActiveStorage, and old ones such as the original Paperclip and the vanilla install of Carrierwave - basically set the HTML form as a multipart and add a vanilla HTML file field. This will upload the form directly to a Rails controller action, which will receive it in the params hash and you can deal with the binary blob from there.

Naively deployed, this will block the MRI through the entirety of the upload. If the file is very big, it can block any other incoming request for the duration of this upload. (Technically, because Rails supports MRI threads and MRI threads are theoretically non-blocking for IO operations, it shouldn’t be as bad as it sounds.)

Fortunately, I believe no one in their right mind exposes an MRI process directly to the internet. We are usually behind a reverse proxy, such as Haproxy, NGINX, Apache HTTPD or something similar.

And usually those reverse proxies are the ones that receive the upload and they only proxy the request when the upload is finished. So the MRI/Rails application can continue to respond to requests in the meantime.

So, if you have a custom infrastructure, you’re golden.

If you’re using something like Heroku, you’re basically screwed. The Heroku routing layer has a hard timeout of 30 seconds. I consider this a good thing because you shouldn’t have requests that take this long to process. Heck, you shouldn’t have a request responding in 1 second, let alone 30 seconds.

But file upload is the exception. A big file takes time to transfer over the internet. And then the Heroku timeout kicks in and interrupts the transmission. The end-user retries and if you have a large enough crowd, you will start filling up the HTTP queue until you have cascading timeouts over each other.

Which is why the only solution available for Heroku is to do “Direct Upload” to a cloud service such as AWS S3. You can add the Carrierwave Direct add-on, or use a complete 3rd party solution such as Cloudinary, with its client library Attachinary to make things easy. And that’s it!

Active Storage, as it is right now, will work for any good enough custom deploy (NGINX + Rails/Puma) but it won’t work at all over Heroku and when the new branch is finished it will be a good choice to use over Heroku as well.

The Transformation Step

This can be done just after the upload or just before serving the file back to the user.

The first one can be done synchronously or asynchronously.

Synchronously is “bad” (I mean, in the controller action itself, because this step is CPU-intensive and takes time). It’s basically transforming the image (using something like Rmagick or MiniMagick) into other versions of different sizes (thumbnail, mobile version, high-dpi version, etc) and storing the paths to the different versions in the storage.

Asynchronously is deferring this costly transformation to ActiveJob so something like a Sidekiq worker picks it up later and does the processing. Meanwhile, you can serve a placeholder if the particular version is not ready yet.

A caveat is that if you have cloud storage and an asynchronous job transformation, you will have lots of traffic because you will spend time uploading to a cloud storage, then the job will have to download from there, do the transformation and do new uploads.

The other solution is to do no processing whatsoever and defer the processing to be on-demand. This is what 3rd party Cloudinary/Attachinary does and you can do custom transformations using URI parameters. It will make the transformation once and cache the results for future usage. An example of a Cloudinary image transformation URL looks like this:

http://res.cloudinary.com/demo/image/upload/w_400,h_400,c_crop,g_face,r_max/w_200/lady.jpg

This is also what you get if you implement Refile or Shrine. It adds a Rack endpoint to your Rails application that will fetch the image binary, perform the transformation according to the parameters received in the URI, and cache the transformation before sending the binary.

The Serving Step

This is making your Rails app serve the stored blob file. The file can be stored locally (or over a remote NFS mount) or over the internet from any cloud based storage such as AWS S3 (in which case you just link directly to their HTTP endpoint).

When your Rails app serves a local file it can just send a special Header to the reverse proxy (NGINX or Apache - X-SendFile or X-Accel-Redirect, which is the difference between send_file and send_data on ActionController::DataStreaming, by the way) and they will serve the file directly, avoiding locking the MRI for the duration of the file transfer.

If it’s in a cloud storage, it’s even easier because you just print their URL for the file directly in the HTML template and there is zero processing necessary.

You will have some processing if you make Rails read the blob and stream it directly (sometimes you need this because you have restrictive access to the files and you don’t want to risk even using a randomized URL for the file).

Conclusion

DHH is right, Basecamp does serve a lot of files and the ActiveStorage (as well as Paperclip, Carrierwave) works nicely, provided you have a proper NGINX reverse proxy in front of it and you added a proper CDN to cache the files back.

If you don’t want to have to manage your own storage, you should use a Cloud Storage (AWS S3, Google Cloud, Azure, etc). ActiveStorage or other solutions will receive the file in the Rails controller level and you SHOULD use ActiveJob to POST the blob to the cloud service in the background - not blocking your application in the process. But the trade-off is that if you add asynchronous job transformation, you will end up having to fetch the original image from the cloud storage to do the transformations.

You should perform transformations on your images to send the best-sized image back to your end-users. Again, if you do it in your application consider either an ActiveJob worker or the Rack real-time transform-and-cache solutions available from Refile or Shrine. At least as it is right now, ActiveStorage doesn’t provide a solution for the image transformations.

My recommendation of previous blog posts on the subject remains, if this is your first time or your business is just beginning, don’t sweat it. Use a complete solution like Cloudinary/Attachinary. It will take care of everything in the most optimal way.

But this is NOT a definitive recommendation. If you have custom deploys, and you know your requirements and constraints, a solution like ActiveStorage, vanilla Carrierwave, etc works well. There are always trade-offs, and having so many moving parts always adds up complexity. The Cloudinary recommendation is just so you can start with as few moving parts as possible and then move to more complex scenarios later if you need to.

Dealing with all the possible combinations of file upload management is no small feat. And you probably have most pressing concerns in your business logic than dealing with files.

Oh, and repeating the same ol’, same ol’, use a damn CDN already! Regardless of the solution you should always add a CDN to serve your assets (javascripts, stylesheets, images, etc).

For starters, Erlang does not need multiple processes like Ruby (which is limited to one connection per process, or per thread if you’re using a threaded-server like Puma). One single Erlang process will take over the entire machine, if you need to. Internally it will keep one real thread per machine-core. And each thread will have its own Scheduler to manage as many micro-processes as you need. You can read all about it in my post titled “Yocto Services”.

Moreover, Erlang has built-in capabilities to form a cluster, where each Erlang instance acts as a peer-to-peer Node, without the need for a centralized coordinator. You can read all about it in my post about Nodes. The power of Erlang is in how “easy” it is to form reliable distributed systems.

You can fire up many Phoenix instances and from one of the instances, it can broadcast messages to Users subscribed in Channels even if their sockets are connected to different instances. It’s seamless and you don’t need to do anything special in your code. Phoenix, Elixir and Erlang are doing all the heavy lifting for you behind the scenes.

No Heroku for You :-(

Because you want to take advantage of this scalability and high availability feature for distributed systems (in the small example of a real-time chat system) you will need to have more control over your infrastructure. This requirement rules out the majority of Platform as a Service (PaaS) offerings out there, such as Heroku. Heroku’s model revolves around single, volatile processes in isolated containers. Those jailed processes (dynos) are not aware of other processes or the internal networking, so you can’t fire up Dynos and have them form a cluster because they won’t be able to find each other.

If you already know how to configure Linux related stuff: Postgresql, HAproxy, etc, go ahead directly to the Phoenix-specific section.

IaaS (DigitalOcean) to the rescue!

You want long lived processes in network reachable servers (either through private networking, VPN, or plain simple - insecure! - public networks).

In this example I want to walk you through a very simple deployment using DigitalOcean (you can choose any IaaS, such as AWS, Google Cloud, Azure or whatever you feel more comfortable with).

I have created 4 droplets (all using the smallest size of 512Mb of RAM):

• 1 Postgresql database (single point of failure: it’s not the focus of this article to build a highly available, replicated database setup);
• 1 HAProxy server (single point of failure: again, it’s not the focus to create a highly available load balancing scheme);
• 2 Phoenix servers - one in the NYC datacenter and another in the London datacenter, to demonstrate how easy it is for Erlang to form clusters even with geographically separated boxes.

Basic Ubuntu 16.04 configuration

• Goals: configure locale, assure unattended updates are up, upgrade packages, install and configure Elixir and Node.
• You should also do: change SSH to another port and install [fail2ban](https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-14-04, disallow login through password).

You will want to read my post about configuring Ubuntu 16.04. To summarize, start by configuring proper UTF-8:

sudo locale-gen "en_US.UTF-8"
sudo dpkg-reconfigure locales
sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
echo 'LC_ALL=en_US.UTF-8' | sudo tee -a /etc/environment
echo 'LANG=en_US.UTF-8' | sudo tee -a /etc/environment

Make sure you add a proper user into the sudo group and from now on do not use the root user. I will create a user named pusher and I will explain in another post why. You should create a username that suits your application.

adduser pusher
usermod -aG sudo pusher

Now log out and log in again through this user. ssh pusher@server-ip-address. If you’re on a Mac copy the public key of your SSH certificate like this:

ssh-copy-id -i ~/.ssh/id_ed25519.pub pusher@server-ip-address

It creates the .ssh/authorized_keys if it doesn’t exist, sets the correct permission bits and appends your public key. You can do it manually, of course.

DigitalOcean’s droplets start without a swap file and I’d recommend adding one, especially if you want to start with the smaller boxes with less than 1GB of RAM:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo sysctl vm.swappiness=10
sudo sysctl vm.vfs_cache_pressure=50
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
echo 'vm.vfs_cache_pressure=50' | sudo tee -a /etc/sysctl.conf

Make sure you have unattended upgrades configured. You will want at least to have security updates automatically installed when available.

sudo apt install unattended-upgrades

Now, let’s install Elixir and Node (Phoenix needs Node.js):

wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb && sudo dpkg -i erlang-solutions_1.0_all.deb
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential nodejs esl-erlang elixir erlang-eunit erlang-base-hipe
sudo npm install -g brunch
mix local.hex
mix archive.install https://github.com/phoenixframework/archives/raw/master/phoenix_new.ez # this is optional, install if you want to manually test phoenix in your box

Now you have an Elixir capable machine ready. Create a image snapshot over DigitalOcean, move it to the regions where you want to create your other droplets and use this image to create as many droplets as you need.

For this example, I created a second droplet in the London region, a third droplet for postgresql in the NYC1 region and a fourth droplet in the NYC3 region for HAProxy.

I will refer to their public IP addresses as “nyc-ip-address”, “lon-ip-address”, “pg-ip-address”, and “ha-ip-address”.

Basic PostgreSQL configuration

• Goal: basic configuration of Postgresql to allow the Phoenix servers to connect.
• To do: create a secondary role just to connect to the application database and another superuser role to create the database and migrate the schema. Also lock down the machine and configure SSH tunnels or another secure way, at least private networking, rather than allowing plain 5432 TCP port connections from the public Internet.

Now you can connect to ssh pusher@pg-ip-address and follow this:

sudo apt-get install postgresql postgresql-contrib

You should create a new role with the same name of the user you added above (“pusher” in our example):

$ sudo -u postgres createuser --interactive

Enter name of role to add: pusher
Shall the new role be a superuser? (y/n) y

$ sudo -u postgres createdb pusher

Postgresql expects to find a database with the same name as the role and the role should have the same name as the Linux user. Now you can use psql to set a password for this new role:

$ sudo -u postgres psql
\password pusher

Register a secure password, take note and let’s move on.

Postgresql comes locked down to external connections. One way to connect from the outside is to configure your servers to create an SSH tunnel to the database server and keep external TCP connections through port 5432 disavowed.

But for this example, we will just allow connections from the public Internet to the 5432 TCP port. Warning: this is VERY insecure!

Edit the /etc/postgresql/9.5/main/postgresql.conf and find the listen_addresses configuration line and allow it:

listen_addresses = '*'    # what IP address(es) to listen on;

This should bind the server to the TCP port. Now edit /etc/postgresql/9.5/main/pg_hba.conf and edit the end to look like this:

# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             your-local-machine-ip-address/24        trust
host    all             all             nyc-ip-address/24       trust
host    all             all             lon-ip-address/24       trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

Save the configuration file and restart the server:

sudo service postgresql restart

See what I did there? I only allowed connections coming from the public IPs of the Phoenix servers. This does not make the server secure, just a little bit less vulnerable. If you’re behind a DHCP/NAT based network, just Google for “what’s my IP” to see your public facing IP address - which is probably shared by many other users, remember you’re allowing connections from an insecure IP to your database server! Once you make initial tests, create your new schema, then you should remove that your-local-machine-ip-address/24 line from the configuration.

From your Phoenix application, you can edit your local config/prod.secret.exs file to look like this:

# Configure your database
config :your_app_name, ExPusherLite.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "pusher",
  password: "your-super-secure-pg-password",
  database: "your-app-database-name",
  hostname: "pg-ip-address",
  pool_size: 20

Replace the information for your server and database and now you can test it out like this:

MIX_ENV=prod iex -S mix phoenix.server

If you see a :econnrefused message from postgrex, then you’re in trouble. Re-check your configuration, restart the server and try again. If everything connects, you can run MIX_ENV=prod mix do ecto.create, ecto.migrate to prepare your database.

Finally, you will want to lock down the rest of your server with UFW, at the very least. UFW should come pre-installed in Ubuntu 16, so you can just do:

ufw allow 5432
ufw allow ssh
ufw enable

That’s it. And again, this does not make your server secure, it just makes it less insecure. There is a huge difference!

And by the way, if you’re a Docker fan:

DO NOT INSTALL A DATABASE INSIDE A DOCKER CONTAINER!

You have been warned!

Basic HAProxy configuration

• Goals: Provide a simple solution to load balance between our 2 Phoenix servers.
• To do: There is something wrong with session checking or something like that as sometimes I have to refresh my browser so I am not sent back to the login form in my application. Phoenix uses cookie-based sessions so I don’t think it is missing sessions.

Now let’s ssh pusher@ha-ip-address. This one is easy, let’s just install HAProxy:

sudo apt-get install haproxy

Edit /etc/haproxy/haproxy.cfg:

...
listen your-app-name
  bind 0.0.0.0:80
  mode http
  stats enable
  stats uri /haproxy?stats
  stats realm Strictly\ Private
  stats auth admin:some-secure-password-for-admin
  option forwardfor
  option http-server-close
  option httpclose
  balance roundrobin
  cookie SERVERID insert indirect nocache
  server us your_us_server_IP:8080 check cookie us1
  server uk your_uk_server_IP:8080 check cookie uk1

You can avoid the stats lines if you have other means of monitoring, otherwise set a secure password for the admin user. One very important part is the option http-server-close as explained in this other blog post, otherwise you may have trouble with Websockets.

For some reason I am having some trouble with my application after I login and it sets the session, sometimes I have to refresh to be sent to the correct page, not sure why yet and I believe it’s something in the HAProxy configuration. If anyone knows what it is, let me know in the comments section below. For now, I am just relying on Sticky sessions (server affinity) by making HAProxy write back a cookie with the server.

Now you can restart the server and enable UFW as well:

sudo service haproxy restart
sudo ufw allow http
sudo ufw allow https
sudo ufw allow ssh
sudo ufw enable

You can easily add SSL support to your application by configuring HAProxy (and not the Phoenix nodes). The DigitalOcean documentation is comprehensive, so just follow it. At the end, my haproxy.cfg looks like this:

global
   log /dev/log    local0
   log /dev/log    local1 notice
   chroot /var/lib/haproxy
   stats socket /run/haproxy/admin.sock mode 660 level admin
   stats timeout 30s
   user haproxy
   group haproxy
   daemon
   maxconn 2048
   tune.ssl.default-dh-param 2048

defaults
   log global
   mode http
   option httplog
   option dontlognull
   option redispatch
   option forwardfor
   option http-server-close
   timeout connect 5000
   timeout client  50000
   timeout server  50000

frontend www-http
   bind your_ha_proxy_IP:80
   reqadd X-Forwarded-Proto:\ http
   default_backend www-backend

frontend www-https
   bind your_ha_proxy_IP:443 ssl crt /etc/haproxy/certs/your_domain.pem
   reqadd X-Forwarded-Proto:\ https
   acl letsencrypt-acl path_beg /.well-known/acme-challenge/
   use_backend letsencrypt-backend if letsencrypt-acl
   default_backend www-backend

backend www-backend
   redirect scheme https if !{ ssl_fc }
   # setting session stickiness
   cookie SERVERID insert indirect nocache
   server us your_us_server_IP:8080 check cookie us1
   server uk your_uk_server_IP:8080 check cookie uk1

backend letsencrypt-backend
   server letsencrypt 127.0.0.1:54321

Finally, I will assume you have a DNS server/service somewhere where you can register the IP of this HAproxy server as an A record so you can access it by a full name such as “your-app-name.mydomain.com”.

Basic Phoenix Configuration

• Goal: configure the Phoenix app to be deployable. Configure the servers to have the necessary configuration files.
• To do: find out a way to cut down the super slow deployment times.

Finally, we have almost everything in place.

I will assume that you have a working Phoenix application already in place, otherwise create one from the many tutorials out there.

I have assembled this information from posts such as this very helpful one from Pivotal about an AWS-based deployment. In summary you must do a number of changes to your configuration.

When you’re developing your application, you will notice that whenever you run it, it delta-compiles what changed. The binary bits are in the _build/dev or _build/test in the form of .beam binaries (similar to what .class are for Java).

Different from Ruby or Python or PHP, you are not deploying source-code to production servers. It’s more akin to Java, where you must have everything compiled into binary bits and packaged into what’s called a release. It’s like a “.war” or “.ear” if you’re from Java.

To create this package people usually use “exrm”, but it’s being replaced by “distillery”, so we will use it.

Then, if you’re from Ruby you’re familiar with Capistrano. Or if you’re from Python, you know Capistrano’s clone, Fabric. Elixir has a similar tool (much simpler at this point), called “edeliver”. It’s your basic SSH automation tool.

You add them to mix.exs just like any other dependency:

...
def application do
  [mod: {ExPusherLite, []},
   applications: [..., :edeliver]]
end

defp deps do
  [...,
   {:edeliver, "~> 1.4.0"},
   {:distillery, "~> 1.0"}]
end
...

From the Pivotal blog post, the important thing to not forget is to edit this part in the config/prod.exs file:

http: [port: 8080],
url: [host: "your-app-name.yourdomain.com", port: 80],
...
config :phoenix, :serve_endpoints, true

You MUST hardcode the default PORT of the Phoenix web server and the allowed domain (remember the domain name you associated to your HAProxy server above? That one). And you MUST uncomment the :serve_endpoints, true line!

For edeliver to work you have to create a .deliver/config file like this:

USING_DISTILLERY=true

# change this to your app name:
APP="your-app-name"

# change this to your own servers IP and add as many as you want
US="nyc-ip-address"
UK="lon-ip-address"

# the user you created in your Ubuntu machines above
USER="pusher"

# which server do you want to build the first release?
BUILD_HOST=$US
BUILD_USER=$USER
BUILD_AT="/tmp/edeliver/$APP/builds"

# list the production servers declared above:
PRODUCTION_HOSTS="$US $UK"
PRODUCTION_USER=$USER
DELIVER_TO="/home/$USER"

# do not change here

LINK_VM_ARGS="/home/$USER/vm.args"

# For *Phoenix* projects, symlink prod.secret.exs to our tmp source
pre_erlang_get_and_update_deps() {
  local _prod_secret_path="/home/$USER/prod.secret.exs"
  if [ "$TARGET_MIX_ENV" = "prod" ]; then
    __sync_remote "
      ln -sfn '$_prod_secret_path' '$BUILD_AT/config/prod.secret.exs'

      cd '$BUILD_AT'

      mkdir -p priv/static

      mix deps.get

      npm install

      brunch build --production

      APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phoenix.digest $SILENCE
    "
  fi
}

Remember the information we’ve been gathering since the beginning of this long recipe? These are the options you MUST change to your own. Just follow the comments in the file content above and add it to your git repository. By the way, your project is in a proper GIT repository, RIGHT??

If you like to use passphrase protected SSH private keys, then it’s going to be a huge pain to deploy because for each command, edeliver will issue an SSH command that will keep asking for you passphrase, a dozen times through everything. You’ve been warned! If you still don’t mind that, and you’re in a Mac you will have an extra trouble because the Terminal will not be able to create a prompt for you to input your passphrase. You must create an /usr/local/bin/ssh-askpass script:

#!/bin/bash
# Script: ssh-askpass
# Author: Mark Carver
# Created: 2011-09-14
# Licensed under GPL 3.0

# A ssh-askpass command for Mac OS X
# Based from author: Joseph Mocker, Sun Microsystems
# http://blogs.oracle.com/mock/entry/and_now_chicken_of_the

# To use this script:
#   Install this script running INSTALL as root
#
# If you plan on manually installing this script, please note that you will have
# to set the following variable for SSH to recognize where the script is located:
#   export SSH_ASKPASS="/path/to/ssh-askpass"

TITLE="${SSH_ASKPASS_TITLE:-SSH}";
TEXT="$(whoami)'s password:";
IFS=$(printf "\n");
CODE=("on GetCurrentApp()");
CODE=(${CODE[*]} "tell application \"System Events\" to get short name of first process whose frontmost is true");
CODE=(${CODE[*]} "end GetCurrentApp");
CODE=(${CODE[*]} "tell application GetCurrentApp()");
CODE=(${CODE[*]} "activate");
CODE=(${CODE[*]} "display dialog \"${@:-$TEXT}\" default answer \"\" with title \"${TITLE}\" with icon caution with hidden answer");
CODE=(${CODE[*]} "text returned of result");
CODE=(${CODE[*]} "end tell");
SCRIPT="/usr/bin/osascript"
for LINE in ${CODE[*]}; do
      SCRIPT="${SCRIPT} -e $(printf "%q" "${LINE}")";
done;
eval "${SCRIPT}";

Now do this:

sudo chmod +x /usr/local/bin/ssh-askpass
sudo ln -s /usr/local/bin/ssh-askpass /usr/X11R6/bin/ssh-askpass

Remember, Macs only. And now every time you try to deploy you will receive a number of graphical prompt windows asking for the SSH private key passphrase. It’s freaking annoying! And you must have XQuartz installed, by the way.

Now you must manually create 3 files in all Phoenix servers. Start with the your-app-name/vm.args:

-name us@nyc-ip-address
-setcookie @bCd&fG
-kernel inet_dist_listen_min 9100 inet_dist_listen_max 9155
-config /home/pusher/your-app-name.config
-smp auto

The /home/pusher/your-app-name is the directory where the release will be uncompressed after edeliver deploys the release tarball.

You must create this file in all Phoenix machines, by the way, changing the -name bit for the same name you declared in the .deliver/config file. The -setcookie should be any name, as long as it’s the same in all servers.

See the -config /home/pusher/your-app-name.config? Create that file with the following:

[{kernel,
  [
    {sync_nodes_optional, ['uk@lon-ip-address']},
    {sync_nodes_timeout, 30000}
  ]}
].

This is an Erlang source-code. On the NYC machine you must declare the London name, and vice-versa. If you have several machines, all of them but the one you’re in right now. Get it?

Finally, for the Phoenix app itself, you always have a config/prod.secret.exs that should never be git added to the repository, remember? This is where you put the Postgresql server information and random secret key to sign the session cookies:

use Mix.Config

config :your_app_name, YourAppName.Endpoint,
  secret_key_base: "..."

# Configure your database
config :your_app_name, YourAppName.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "pusher",
  password: "your-super-secure-pg-password",
  database: "your-app-database-name",
  hostname: "pg-ip-address",
  pool_size: 20

# if you have Guardian, for example:
config :guardian, Guardian,
  secret_key: "..."

How do you create a new random secret key? From your development machine just run: mix phoenix.gen.secret and copy the generated string into the file above.

So now you must have those 3 files in each Phoenix server, in the /home/pusher home folder:

~/your-app-name/vm.args
~/prod.secret.exs
~/your-app-name.config

As per distillery documentation, you have to set an environment variable to let it know about the vm.args file, and this is SUPER important otherwise it will generate a default one that doesn’t set the proper name and cookie, so the nodes will not find each other after booting up.

Using sudo, edit the /etc/environment and add the line:

RELEASE_CONFIG_DIR=/home/pusher/your-app-name
VMARGS_PATH=/home/pusher/your-app-name/vm.args

You have to do it in all the phoenix servers.

In your local development directory you still have to run this:

mix release.init

It will generate a standard rel/config.exs file that you must add to your git repository with the following changes to the bottom:

...
environment :prod do
  plugin Releases.Plugin.LinkConfig
  ...
end

release :your_app_name do
  set version: current_version(:your_app_name)

  set applications: [
    your_app_name: :permanent
  ]
end

The plugin should enable the release to find the local vm.args file in the server (this is super important, otherwise the remote commands such as start, stop, ping, etc will not work and the nodes will not load the correct information to boot up and form a cluster).

Finally, all set, you can issue this command:

$ mix edeliver build release production --skip-mix-clean --verbose

If it finishes without errors you will have a message like the following in the end:

...
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: _build/prod/rel/your-app-name/bin/your-app-name console
      Foreground: _build/prod/rel/your-app-name/bin/your-app-name foreground
      Daemon: _build/prod/rel/your-app-name/bin/your-app-name start
--> Copying release 0.0.1 to local release store
--> Copying your-app-name.tar.gz to release store

RELEASE BUILD OF YOUR-APP-NAME WAS SUCCESSFUL!

Now, this will take an absurdly long time to deploy. That’s because it will git clone the source code of your app, fetch all Elixir dependencies (every time!), it will have to compile everything, then it will run the super slow npm install (every time!), brunch your assets, create the so-called “release”, tar and gzip it, download it and SCP it to the other machines you configured.

In the .deliver/config file you set a BUILD_HOST option. This is the machine where all this process takes place, so you will want to have at least this machine be beefier than the others. As I am using small 512Mb droplets, the process takes forever.

The last command will download the generated tarball. It has to do it to make sure you have the NIFs compiled in the native environment where they run, because if you use a Mac, binaries from Macs won’t run on Linux.

Now we must upload and decompress this tarball in each server with the following command:

mix edeliver deploy release to production

The slower your network, the longer this will take, as it’s uploading a big tarfile across the public internet, so make sure you’re in a fast connection. And when it finishes, we can restart the servers (if you already had instances running):

mix edeliver stop production
mix edeliver start production

If you do everything right, the edeliver process finishes without any error and it leaves a daemon running in your server, like this:

/home/pusher/your-app-name/erts-8.2/bin/beam -- -root /home/pusher/your-app-name -progname home/pusher/your-app-name/releases/0.0.1/your-app-name.sh -- -home /home/pusher -- -boot /home/pusher/your-app-name/releases/0.0.1/your-app-name -config /home/pusher/your-app-name/running-config/sys.config -boot_var ERTS_LIB_DIR /home/pusher/your-app-name/erts-8.2/../lib -pa /home/pusher/your-app-name/lib/your-app-name-0.0.1/consolidated -name us@nyc-ip-address -setcookie ex-push&r-l!te -kernel inet_dist_listen_min 9100 inet_dist_listen_max 9155 -config /home/pusher/your-app-name.config -mode embedded -user Elixir.IEx.CLI -extra --no-halt +iex -- console

It will still take a long time, but it should be easier. So this is a pro-tip for you, Linux users. Follow this Gist for more details, you must emulate what’s run from the .deliver/config file’s bottom half.

Also notice that I ran the migrations manually, but you can do it using mix edeliver migrate.

Read their documentation for more commands and configurations.

Also, do not forget to enable UFW:

sudo ufw allow ssh
sudo ufw allow 8080
sudo ufw allow 4369
sudo ufw allow proto tcp from any to any port 9100:9155
sudo ufw default allow outgoing
sudo ufw enable

Debugging Production bugs

Right after I deployed, obviously it failed. And the problem is that the /home/pusher/your-app-name/log/erlang.log files (they are automatically rotated so you may find several files ending in a number), you won’t see much.

What I recommend you to do is to change the config/prod.exs file ONLY in your development machine and change the log setting to config :logger, level: :debug, use the same prod.secret.exs you edited in the servers above and run it locally with MIX_ENV=prod iex -S mix phoenix.server.

For example, in development mode I had a code in the controller that was checking the existence of an optional query string parameter like this:

if params["some_parameter"] do
 ...

That was working fine in development but crashing in production, so I had to change it to:

if Map.has_key?(params, "some_parameter") do
 ...

Another thing was that Guardian was working normally in development, but in production I had to declare its application in the mix.exs like this:

def application do
  [mod: {ExPusherLite, []},
   applications: [..., :guardian, :edeliver]]
end

I was getting :econnrefused errors because I forgot to run MIX_ENV=prod mix do ecto.create, ecto.migrate as I instructed above. Once I figured those out, my application was up and running through the http://your-app-name.yourdomain.com, HAProxy was correctly forwarding to the 8080 port in the servers and everything runs fine, including the WebSocket connections.

Conclusion

As I mentioned above, this kind of procedure makes me really miss an easy to deploy solution such as Heroku.

The only problem I am facing right now is that when I log in through Coherence’s sign in page, I am not redirected to the correct URI I am trying ("/admin" in my case), sometimes reloading after sign in works, sometimes it doesn’t. Sometimes I am inside a “/admin” page but when I click one of the links it sends me back to the sign in page even though I am already signed in. I am not sure if it’s a bug in Coherence, ExAdmin, Phoenix itself or an HAProxy misconfiguration. I will update this post if I find out.

Edeliver also takes an obscene amount of time to deploy. Even waiting for sprockets to process in a git push heroku master deploy feels way faster in comparison. And this is for a very bare-bone Phoenix app. Having to fetch everything (because Hex doesn’t keep a local global cache, all dependencies are statically vendored in the project directory). Having to run the super slow npm doesn’t help either.

I still need to research if there are faster options, but for now what I have “works”.

And more importantly, now I have a scalable cluster for real-time bi-directional WebSockets, which is the main reason one might want to use Phoenix in the first place.

If you want to build a “normal” website, keep it simple and do it in Rails, Django, Express or whatever is your web framework of choice. If you want real-time communications the easy way, I might have a better solution. Keep an eye on my blog for news to come soon! ;-)

That happens to old blog posts, including my own as I’ve been posting for several years.

In this post I will fix what I posted (in pt-BR):

• Direct Upload para S3: a Solução Definitiva!
• S3 Direct Upload + Carrierwave + Sidekiq
• Upload de Arquivos
• Enciclopédia do Heroku

They are still good posts to understand the mechanics of what I will propose here, so you should read them if you’re interested in how things work.

TL;DR is: use Cloudinary (with Attachinary) and install a CDN (Cloudfront) right now as it’s super trivial to do.

Uploading files is not a task to be taken lightly. It’s one of those tasks that you must not consider that it “works” just because it works in your development machine, for a few trivial uploads. Production level behavior changes drastically and it can bring your entire project down depending on how heavily it depends on user-generated content.

There are a lot of moving parts to understand. But in the most basic solutions you just create an HTML multipart form with a file input field and submit it to some controller endpoint directly. This controller receives the blob binary and you save it to some local directory in the server.

This is the worst possible solution and unfortunately this is the one you will find the most across the web.

There are many problems to deal with this simple implementation:

• We are in the era of 12+ megapixel cameras. Those are multi-megabyte sized files. Each upload will take a long time, especially if the user is trying to upload using unreliable or slow wireless networks.

Once the user’s browser connects to the web application it will hold that connection and block the entire process through the duration of the request-response cycle. Making this simple to understand, if you only have 1 single process in the server, it will be unable to respond to any other request until the upload finishes.

Fortunately most deployments will use NGINX in front of the web application - as they should - so this effect is minimized as NGINX will receive the entire blob before passing it to the web application process underneath, making it not so problematic in most cases.

But if you’re on Heroku, the router layer has a 30 seconds timeout limit. If the upload takes more than that (which is common with users in unreliable networks) this layer will cut the connection down before it finishes. Users will retry and this can cause your request queues to grow very fast.

Avoid uploading anything to Heroku. I consider this a feature as it forces applications deployed on Heroku to use good practices as I will explain below.

• You should also NOT save files to the local filesystem if you want to scale horizontally, as one server will not be able to see what’s in the other server unless you’re in AWS EC2 or other IaaS, with shared mounted volumes, for example. In Heroku, you can’t rely on anything in the filesystem as the machines are volatile and whenever a dyno restarts it loses whatever was not there during the deployment.

So you must upload what you received to an external storage, such as AWS S3, and this is also slow if you receive lots of uploaded files. You can rely on NGINX or Heroku’s router layer to handle the heavy receiving of the files but you will make the request-response cycle super slow if you synchronously process and upload the files to S3.

The next option you think of is to add Sidekiq or some other Async Job mechanism to do this other heavy lifting. Now you have to create front-end placeholders to show to the users while the background job is not finished sending the files to S3.

The last option, if you dig deeper, is browser-based direct uploads to S3 and just posting the URLs to your web application. This is the ideal solution, but it’s not easy to implement in your stack.

• Once you nail everything above, you still make the mistake of putting the S3 URLs directly in the IMG tags in your HTML, which is not recommended by Amazon as S3 is recommended just for “storage” purposes. This is the easy part to fix, but most projects forget about that.

Cloudinary or Carrierwave Direct

In my previous posts I explained each of the above issues in more detail and I offer some (complicated) solutions, such as using the Refile project. At that time the Carrierwave plugins for Direct Upload were not ready.

Nowadays the solution is actually super easy and this is what I recommend: go straight to Cloudinary. This is a Software as a Service solution for proper direct uploads and dynamic image processing. You should definitely explore the Attachinary gem to make the process easier.

If you have legacy Carrierwave, Paperclip, Shrine, Dragonfly, Refile or others, one jerry-rigged - but reasonable - alternative is to just add Attachinary to the mix.

Let’s say you have an old user.avatar uploader. You can just create a new user.new_avatar with Attachinary/Cloudinary. In the HTML form upload you start using just the Attachinary helpers. And in the HTML where you show the image itself, you can make a helper to check for @user.new_avatar? and show using cl_image_tag(@user.avatar.path instead, otherwise you show the old URLs.

You don’t need to totally migrate everything to Cloudinary, just keep the old assets where they are and start putting the new assets in the Cloudinary configuration. It has a Free tier that can hold up to 75 thousand images and allows traffic of up to 5GB. And with just USD 44 a month you have 10 times that. So it’s really cheap if your application is serious about file uploading.

If you don’t want to commit to an external service yet, and you’re using Carrierwave, another option you must consider is testing out CarrierwaveDirect to test direct uploading to S3 from the browser. This will at least avoid the complexities of setting up async uploads from the web application to S3.

Update: Janko Marohnić posted a very good comment that I think is important to quote directly:

Great writeup, I wholeheartedly agree that we probably don’t “just want something simple”, especially concerning synchronous uploads and CDNs.

Shrine actually has a Cloudinary integration, and it’s pretty advanced too; it supports direct uploads, setting upload options, storing Cloudinary metadata (hello responsive breakpoints) etc :). Also, the Cloudinary gem ships with a CarrierWave integration. I think it’s beneficial to use Shrine or CarrierWave with Cloudinary, because then you get to keep your file attachments library, and just change the underlying storage service.

As for CarrierwaveDirect, I don’t think it’s actually very useful, because backgrounding you need to do all by yourself anyway, so basically all it gives you is the ability to generate a direct-upload form to S3. But generating request parameters and URL to S3 is something that’s already built in into the official aws-sdk gem, CarrierwaveDirect just seems to be reimplementing that logic. And the advantage of generating this through aws-sdk is that it’s not HTML-specific, so you could setup an endpoint which returns this information as JSON, and now you have your multiple file uploads directly to S3 :)

As a disclaimer I didn’t test CarrierwaveDirect myself and I remember being quite broken 2 years ago. And I’ve heard good things about Shrine along the way but never did proper testing. I’d recommend giving it a try if you want to migrate away from Carrierwave.

Don’t serve assets from S3, use a CDN, any CDN

If you’re sticking to S3 uploading, at the very least avoid serving those files from S3. In my “Heroku Encyclopedia” post I recommend using “S3 Assets Sync”. DO NOT DO IT: use a CDN instead, it’s faster, easier and the correct solution.

The first thing you want to do is sign up to AWS’s Cloudfront CDN service. Always use a CDN for all your assets, not only the ones internal to your web application but also every user uploaded content. It’s super simple to create a new CDN endpoint pointing to your S3 bucket.

Carrierwave will default to the S3 bucket host, but once you have the Cloudfront endpoint you can easily change all the uploaded files URLs like this:

# ./config/initializers/carrierwave.rb
CarrierWave.configure do |config|
  config.fog_credentials = {
    :provider => "AWS",
    :aws_access_key_id => ENV['AWS_ACCESS_KEY_ID'],
    :aws_secret_access_key => ENV['AWS_SECRET_ACCESS_KEY'],
  }
  config.fog_directory = ENV['S3_BUCKET']

  # use only one of the following 2 settings
  # config.fog_host = "http://#{ENV['S3_BUCKET']}.s3.amazonaws.com"
  config.fog_host = ENV['S3_CDN'] # for cloudfront
end

You must obviously configure the environment variable S3_CDN to point to the Cloudfront endpoint specific to the particular S3 bucket you’re using.

The important part is: serve ALL ASSETS from a CDN, no matter what. Never directly from your web application, avoid directly serving from S3. It’s almost trivial to implement and if you don’t like AWS for any reason, you can choose Fastly, Cloudflare and many others.

They all work the same way. There is NO excuse to not use a CDN no matter the size of your application.

Think of services like Basecamp, for managing projects without the company spending on maintenance, or your favorite webmail. These are online services where you pay so you never have to think about infrastructure. A form of service outsourcing.

This week Google caused a small stir by launching its answer to Cloud Computing: Google App Engine. TechCrunch published a review here. But what is Cloud Computing? First things first, let’s go over the terms the market uses.

We developers will, at some point, want to put our application into production. That’s when the hairiest part of the process arrives: deployment!

There are several options. First, software:

• Buy a commercial application. Commodity cases like Exchange Server: you pick, buy, install, and start using it. Bigger applications sell under licensing schemes, per user or per processor. Some software can be extended and customized, but usually it’s you who has to adapt to the software.

• Get an open source solution. Download it from the internet, configure it, and use it, or hire a services company to do that work for you. The obvious advantage is zero license cost. The downside is that running that kind of setup has a cost nobody usually accounts for: TCO is not trivial. How easy the application is to modify varies case by case.

• Build the system in-house. Code the system yourself, tailored to your needs, with your own team or a hired software house. The bespoke model. It tends to cost more, but it may be necessary when no commercial software does what you need. In theory it should be the software that fits the company best; in practice the TCO is usually high and the outcome uncertain. Everything depends on the maturity of the company and of whoever was hired to build it. Your mileage will vary.

• Use an ASP. You ‘rent’ a market application, like SalesForce.com, without worrying about deployment, installation, or updates. It’s less customizable, but the total cost of ownership is usually far lower. In many cases it makes sense: all you need is a stable internet connection. The services model is a trend because today’s internet infrastructure allows it. And there’s a clear win for the developer: one fix in the central system instantly benefits the whole customer base, with no per-client installs.

As for hardware options:

• Host it yourself. Buy machines, build networks, manage everything inside your own physical perimeter. As the company grows, you buy more machines. It works best for intranets and for companies with a mature IT department. IT is normally treated as a cost, like paying taxes; running everything in-house only makes sense if your core business is those very systems.

• Host it yourself, but understanding that growth isn’t linear. Sometimes you need more resources only for a short window, and buying machines all the time just inflates your TCO. That’s what ‘on-demand’ servers are for, like some IBM lines: when you need more power, you call IBM and they unlock more processors on the machine sitting physically at your site. A limited solution.

• Co-location. The machines are yours, but they live on a third party’s infrastructure, in a data center outside your company’s perimeter. It’s a cost-benefit trade: you rarely have the expertise to keep infrastructure as stable as a third party’s, but the machines remain your responsibility. It fits companies that have the money for machines but no physical space, or branches that all need to reach the same system. The data center keeps everything running around the clock, with a stable network and 24/7 support.

• Shared hosting. You rent a piece of a machine shared with other clients. The entry cost is minimal, but you suffer when another client on the same box demands more resources, plus issues like shared system libraries. It’s a can of worms that providers keep trying to fix, but the nature of this kind of package prevents anything bulletproof. Growing in such an environment is limited. It favors freelancers and small companies that need a website online but lack the money for beefier setups.

• VPS (Virtual Private Server). You still share the machine with other clients, but you get the feel of an entire machine of your own, virtual and isolated. A ‘shared co-location’: the machine is rented in slices, using solutions like Xen or VMware, and it comes out cheaper than co-location. The isolation shields your application from the neighbors’ updates. It’s the second step for anyone who wants low cost with more flexibility. The problem remains when you need extra slices only for a short period: spinning one up takes time, and you can miss your time-to-market.

Cloud Computing

Terms settled, on to the trendy one. I don’t know if it was first, but the one making the most noise is certainly the Amazon Web Services model: Elastic Compute Cloud (EC2), Simple Storage Service (S3), and SimpleDB.

The main draw is the elastic on-demand concept. You pay for what you use, without manually reconfiguring everything at each change. Use more, pay more; scale back down, pay less. It’s payment for resource consumption, instead of a fixed monthly fee for fixed resources.

Amazon’s products cover three services:

• Storage (Amazon S3). In a traditional model you’d buy or rent a file server. S3 created a different kind of file system, based on the concept of ‘buckets’. The advantages are price and stability; the downside is a model that doesn’t map directly onto a regular file system. By now there are dozens of libraries in different languages that ease the transition.

• Database (Amazon SimpleDB). Instead of a relational Oracle or MySQL, Amazon offers an unstructured, document-based database. With no fixed schemas, the costly administration over your data domain disappears, and maintenance drops. On the other hand, there are still hardly any frameworks or libraries that migrate a relational (SQL) application to SimpleDB’s model trivially. Once again, the advantage may be price.

• VPS (Amazon EC2). Dynamic, elastic VPS instances. If your system demands more resources during a spike in visits and transactions, you spin up more slices dynamically and kill them when you no longer need them. Five slices in the morning, twenty in the afternoon, five again at night. The catch: they won’t necessarily be the same five slices. The model guarantees slices, but not which ones. Your application must store its data outside EC2, in S3 or SimpleDB, so another slice can take over without relying on local data. It’s Shared-Nothing taken to its ultimate conclusion.

What’s the advantage? You get what you need, when you need it. Instead of committing to expensive co-location or VPS plans, you leave the application in Amazon’s care and gain dynamism without worrying whether your machines ‘can take it’. It’s like paying for the gym by the hours you actually attended, instead of a fixed monthly fee. Quit mid-month, and you don’t lose the whole fee. It’s a trend in the services market.

What’s the disadvantage? With Amazon, you have to prepare your application around their limitations and requirements. Especially because having your slice killed is normal: local data is gone if it wasn’t persisted elsewhere first. Taken as a premise from day one, that stops being a problem. It’s a different paradigm from co-location or classic VPS, where local data is permanent and backups are the host’s job.

Google App Engine

Which brings us to the week’s news. In practice, App Engine maps Amazon’s services one to one:

• Amazon SimpleDB vs Google BigTable: neither is a relational database; forget the classic RDBMS.

• Amazon S3 vs Google GFS: again, no classic file system like the one on your machine.

• Amazon EC2 vs App Engine: similar, but different, as we’ll see.

Google’s marketing sold the product as “your application running inside our infrastructure.” Everyone knows that Google Search, Reader, Gmail, and Orkut run on a proprietary, highly scalable infrastructure, the one that made Google famous and is now ‘open’ (gratis, not libre) for others to enjoy. Let’s look at advantages and disadvantages:

• The downside on both sides is the paradigm break: applications must be born exclusive to one of these environments. Choosing one means locking yourself in, because supporting both costs more development.

• Amazon’s advantage is service independence: want just storage? Use S3 alone. The disadvantage is that their services demand more integration knowledge than anyone else’s.

• Google’s disadvantage is the all-or-nothing proposition: your whole application lives in there, not just pieces. The advantage is the online administration environment, which puts Google a step ahead of Amazon in usability.

Right now Google only supports applications written in Python. It ships with Django pre-available and with APIs that make handling data and storage easier. Certainly great news for the Python community.

That caused some commotion among developers of other languages, which is normal. Coming from anyone else, it would have passed unnoticed as “just another hosting provider.” Coming from Google, they know massive praise and massive criticism always follow. Praise is democratic, and so is criticism. If they didn’t see this coming, someone in Public Relations needs to be fired.

Either way, don’t pay attention to yet another round of language wars:

• they’re a flash in the pan: only fun right now

• they don’t amount to much: if Google anticipated this and has good PR, they walk away unscathed

• Google likes Python: that’s public knowledge, and there’s nothing wrong with it. Companies decide what they want, when they want, and how they want. The day you have your own company, you’ll understand this.

• and Google never said App Engine would be Python-exclusive

In the official announcement, Google presents the service as a “preview release”, the famous Beta of every self-respecting Web 2.0 application. Big news :-) The promise is to support other languages in the future. It’s not a perfect solution, but like everything coming out of a corporation the size of Google, it has huge potential. And like every Web 2.0 application, there’s a free account so you can test-drive the environment before deciding on a plan upgrade. At this Beta stage, though, it isn’t open to everyone yet.

What Does This Represent?

Once App Engine matures and starts supporting more development environments beyond Python, it can become the main threat to Amazon’s Web Services: it’s a blatant copy of their services, and Amazon is the obvious commercial competitor.

It can also threaten smaller services, like Joyent Accelerator, a hosting provider built on the same elastic-resources concept, or the newcomer Heroku, which at its core is an App Engine for Rails: a bridge over Amazon’s services, configured in a simpler way for Ruby on Rails developers. Granted, it’s not ready yet and is nowhere near Google’s scale.

Neither Google’s nor Amazon’s services are perfect. Which is an obvious thing to say: no service is. Everything depends on your requirements and on how you want to commit. The rule stays the same: the more commitment, the cheaper in the short term and the pricier in the long run; the less commitment, the pricier now and, maybe, the better later.

In short: want something cheap and flexible? Amazon and Google can help. But if one day you grow and want independence, detaching can be hard, because your application got intimately tied to their services. With a more traditional service, like co-location or VPS, you pay more now, but you stay isolated from third parties and a future migration to other hosts comes cheaper.

If App Engine succeeds, the likely effect is pressure on the hosting market to offer more creative, cheaper, higher-quality services. That’s market law: competition benefits us, the consumers. Let them fight it out while we watch. In the short term, for anyone wanting to put an application in the air, it’s just one more variable to consider.

What’s Google’s goal here? I’m not sure it’s the same as the hosting companies’: making money by outsourcing infrastructure in a commodity market. Google has specialized in turning good products into commodities, which is both good and bad, depending on your point of view. If it’s a free service, one of the goals is to help more and more websites appear: potential customers for Google’s real core business, AdSense.

Of course I wouldn’t host my Twitter-equivalent on a free service. Actually, if your data matters to you, never use free services. They’re great while they work; when they fail and you take a loss, the problem is uniquely and exclusively yours. It’s like driving a car without insurance.

(off-topic within the off-topic: I use Gmail, but every email that lands in my inbox gets automatically forwarded to my paid Apple .Mac account, every .Mac email lands on my machine, and my machine has two redundant backups. Just to illustrate. I like Gmail, but I wouldn’t bet even my emails on it.)

Understanding Google

When analyzing a Google move, never think ‘altruism’ or any such nonsense. They’re a publicly traded company whose main objective is keeping shareholders happy. Again: their business is profit, and charity pays no dividends.

Their core business happens to involve reaching people who get happy hearing stock phrases like “Don’t be evil.” Great for us, sure, but that also generates a conflict of interests.

(off-topic within the off-topic, no naivety: not doing evil is too vague, because ‘good’ and ’evil’ are relative concepts. What’s good for me can be bad for someone else, and vice versa; good and evil shift with place and time. Nobody will ever say ‘do evil’, obviously; that wouldn’t be very smart. ‘Doing good’, or at least what the majority currently perceives as ‘good’, is a very intelligent way to generate positive publicity with consumer opinion. Besides being cheap, that kind of publicity can come with tax exemptions, which for big companies is considerable savings. Brazil’s Lei Rouanet is an example: both sides win, but without laws like this there’d be little motivation for a company to ‘invest’ outside its core business. Bah, what do I know. And of course, I like Google too; so far it’s in my interest.)

To understand it better, read this. An interesting part:

“Evil,” says Google CEO Eric Schmidt, “is what Sergey says is evil.”

Wired: “moral compromise is just the cost of doing big business.”

App Engine is one more channel for AdSense, just like Android and Gmail. One reason a company like Mozilla exists is the revenue line from Google for keeping Google search as the default in its toolbar. Apple also gets paid to have Google’s search field in Safari. It doesn’t quite count as a ‘bundle deal’, since both browsers are free, but the cost is indirect: AdSense advertisers want to appear in most browsers. Yahoo! also announced a test to show AdSense ads on its own search results, which could increase Google’s market share even further.

That means indirect visits, which turn into the billions of revenue that keep Google’s shareholders happy. And geek users, who contribute the free word-of-mouth advertising, obviously love the idea that supporting Google or Firefox means ‘Microsoft losing’. I may be being pessimistic, but it happened with Rockefeller, a great philanthropist, and his Standard Oil. It happened with Bill Gates, another great philanthropist, and his Microsoft. It can happen with Google. To get an idea, read this article and an interesting excerpt:

Blogger and Mashable reader Constantine von Hoffman at Collateral Damage probably put it best when he said: “At this point, it would take a mashup of Wittgenstein, Quantum mechanics and LSD to make sense of Google’s various explanations for what it will and won’t censor and why.”

Anyway, Google App Engine by itself represents no revolution yet: it’s just one more player in the hosting market. Cloud Computing is one more marketing term for a solution derived from others that already existed: finer price granularity for services. Just as it did with Android, Google is commoditizing one more market segment.

The same old rule still applies: cost-benefit analysis. And that kind of analysis doesn’t come as a recipe, nor with the click of a button.