The pitch is strong. It’s a multimodal video generation model that takes text, up to 9 reference images, 3 reference video clips, and 3 audio tracks all in the same call. On top of that, it produces up to 15 seconds of output with native synced audio (dialogue, sound effects, ambient music), supports video extension, surgical editing of characters and objects, and prompt-driven camera planning. ByteDance is pitching it at commercial advertising, explainer videos, film production, e-commerce, and gaming. Translation: for content creators trying to climb out of the “random meme” tier and start producing things that look professional.

If you want a quick visual tour before going further, here’s a great walkthrough from Stefan, of the Stefan 3D AI Lab channel, which I already plugged in the post on AI 3D modeling:

The main interface is Dreamina’s “Photo Studio,” where you pick the generation mode and stack your references. So far I’ve only tested the standard multi-reference mode.

Test 1: lip sync with podcast audio

My first test was simple. I grabbed a few seconds of the opening bumper from my podcast The M.Akita Chronicles, which is generated by AI through an ElevenLabs v3 pipeline, passed in my new anime-style avatar as a reference image, and asked Seedance to make the character lip-sync and gesture along with that audio.

The result is decent. All of that came from a single still image. The lip sync follows the speech reasonably well, the expression has some life to it, and the hand gesture lands at the right beat. But this is a YouTube short or Instagram clip at best. Not the kind of animation you’d open a serious video with.

The feature that actually matters: video reference

And here’s what I think is the only real use of these models for actual work: feeding in video as a reference. Text alone is never going to be precise enough. You can describe a scene with whatever vocabulary you want, swap adjectives, specify lens, angle, framing, and every generation will still hand you back something subtly different. It’s a slot machine. Great for memes, terrible for production.

Most amateurs see one random clip that comes out nice on the first try and go straight to flooding their X and Instagram feeds with it. That’s not control. That’s luck. For film, for ads, for show openers, for anything where the next shot has to match the previous one, you have to be able to tell the model: “do exactly this motion, with exactly this camera.” That’s where reference video changes the game, because it replaces hours of 3D modeling, rigging, animation, and rendering that would cost a fortune in a studio.

For my test, I recycled the model I’d already generated in the post on AI 3D with Hunyuan and Nano Banana, where I tested how far you can take 3D modeling from a prompt and even hit “print” on the result. But for something more cinematic I wanted a model with a proper animation cycle. So I grabbed the Darth Talon model from Sketchfab, which has a short, well-made motion sequence, just because it looked cool.

I imported the FBX into Blender, set up a basic camera and lighting, and did a quick Cycles render. Nothing pretty about it, just a sketch to feed in as motion and framing reference.

I uploaded that render to Seedance as a video reference along with three t-pose images of the character (front, back, left side) for visual consistency:

First attempt, with a generic prompt:

Not bad. The character keeps its identity, the lighting got a serious upgrade over my raw render, and the clip is coherent. But the motion and camera position drifted noticeably from the reference. The model basically read it as “there’s a character here that looks like that doing something like that,” and improvised the rest.

Then I remembered Stefan’s tutorial: the trick is to literally drop something like Follow exact motion and camera from reference video into the prompt. Tried again:

A lot better. It’s clearly following the reference choreography, the camera angle matches in several beats, and the character looks more polished than anything I could hand-render in a quick Cycles sketch. But it’s still not a frame-for-frame match. In some shots the model is still taking liberties with timing, position, and framing. There are other prompting tricks and parameters to play with, but I stopped here to write this up while the launch is still fresh.

ComfyUI, Runway, and the rest of the ecosystem

Seedance 2.0 also already shipped as a ComfyUI cloud node, which is great for anyone who wants to drop the model into a bigger pipeline without bouncing between web UIs. If you’ve already got an image-gen workflow built in ComfyUI, this is genuinely handy.

And for anyone who wants volume without doing the per-generation credit math, Runway is offering Seedance 2.0 as part of their Unlimited plan, which runs around $76/month annual or $95/month monthly. “Unlimited” means no per-generation charge, but you do trade for lower queue priority at peak hours and an output storage cap. If you iterate a lot, the subscription model makes way more sense than buying credits one at a time.

The price question

Seedance’s own pricing is credit-based. The Standard plan is $24.90/month annual or $49.90/month monthly, and gets you 2,500 credits a month. Every 10 seconds of generated video burns 6 credits.

The math is brutal. With 2,500 credits on Standard, you get about 416 ten-second generations a month. Sounds like a lot, until you remember my own test: it took at least two tries to get close to what I wanted, and even then it wasn’t a one-to-one match. In practice, plan on 3 to 5 regenerations per usable scene. That drops your real number to something like 80 to 140 short clips per month. Maybe enough for a solo creator doing short-form content. Nowhere near enough for production that needs minutes of polished, continuous video.

The higher tiers scale credits roughly linearly, but the same problem holds: at today’s price, Seedance 2.0 is still a tool for short clips and well-made memes, or for pre-vis sketches before you send the real thing to an actual studio. It doesn’t replace continuous professional production. For that use case, Runway with Unlimited is honestly the cheaper option in practice. Full pricing table here.

What nobody wants to talk about: restrictions and deepfakes

Two things I don’t want to leave out. First: Seedance 2.0 took its sweet time getting to the public partly because ByteDance is being a lot more conservative than the competition when it comes to moderation. The model blocks generation that uses real people as reference (celebrities, politicians, public figures) and filters strong-IP characters from Disney, Marvel/DC, Nintendo, and registered trademarks generally. If you want the details of what passes, what doesn’t, and why ByteDance went that way, this MindStudio piece breaks it down. It makes sense: ByteDance operates across dozens of jurisdictions, lives under heavy regulatory scrutiny worldwide, and has the elephant in the room of Hollywood pushing the entire industry with copyright suits over training data and generation. The release got delayed and arrived with heavier filters precisely because of that.

Second point, and this is the one that matters more: for you, the average reader, the message is that deepfakes are no longer hypothetical. Photos on the internet stopped counting as proof a long time ago, because image models like Nano Banana, Hunyuan, and the rest deliver realism that fools anyone. Now video sits in the same bucket. What we just saw above, where a couple of references and two prompts get you a coherent animated scene in minutes, is just the start. The average person, with no training, isn’t going to be able to tell generated video from real video at a glance anymore. And Seedance’s filters only block above-board, legitimate use. Anyone with bad intent will always find a path through open source models, workarounds, or less strict platforms. That’s the world we’re in now, and the sooner we collectively accept it, the better off we are.

No, this doesn’t replace real artists

Let me cut off the usual speech before it shows up in the comments. No, Seedance doesn’t replace VFX artists, directors, editors, none of that. This is a hammer. The tool is the same for everybody. What comes out the other end depends entirely on who’s swinging it.

Anyone who’s actually studied framing, pacing, continuity, art direction, editing, color, composition is going to use Seedance to 10x the work they were already capable of delivering. Anyone who hasn’t studied any of that is going to crank out the same Instagram meme they were already cranking out, just 10x faster. The AI reflects who you are. If you’re a real artist, the tool multiplies you. If you only think you are, the tool multiplies the mediocrity. There’s no miracle here.

This is exactly why the whole “now everyone can be a film director with AI” wave is the same song we’ve heard with the digital camera, with Photoshop, with After Effects, with Premiere, and with every powerful tool that’s shown up in the last 30 years. The technical bar drops. The bar of taste, reference, and study sits exactly where it always has. Good people get more productive. Everyone else still needs to hire real ones to do real work.

Where this goes next

Seedance 2.0 is clearly a step in the right direction, but it’s not the tool that replaces professional production yet. Text alone stays imprecise, video reference helps but still drifts, and the price per minute of polished output is high relative to what you actually get. The good news is that this story always rhymes: Google has Veo, Runway has its own model and is also reselling Seedance, Kling and Hailuo from China keep pushing, and the open source models are gaining ground. OpenAI had Sora, but they shut it down. As competition tightens, quality goes up and price comes down. A year or two from now we’ll be looking back at this post and finding what I just showed pretty quaint. Same as it ever was.

That’s the April 6th episode of the The M.Akita Chronicles podcast, already generated with the new ElevenLabs v3 pipeline. Subscribe to the show on Spotify so you don’t miss new episodes like this one.

When Qwen3 TTS dropped, back around January this year, everyone on Twitter/X and on the AI newsletter circuit was shouting “ElevenLabs killer”. There’s a Medium piece claiming it’s the first real open source threat to ElevenLabs. There’s a post on byteiota saying 3-second voice cloning beats ElevenLabs. There’s an analysis on Analytics Vidhya calling it the most realistic open source TTS released so far. The enthusiast internet consensus was: we finally have open source that can go toe-to-toe with ElevenLabs, the tables have turned, it’s just a matter of time.

I decided to test it on my own production pipeline, as usual. I built out a whole pipeline on top of Qwen3 TTS 1.7B to generate the weekly podcast for The M.Akita Chronicles, and I documented the behind-the-scenes on the serving-AI-in-the-cloud post. Anyone who wants the detail on cold starts, voice cloning, and sampling parameters that change from one mode to another, that’s the place. I’m not repeating it all here.

This post asks a different question. After almost two months running that setup in production, shipping an episode every Monday, last night I finally pulled the plug on Qwen3 and switched the whole thing to ElevenLabs v3. Let me tell you why.

What didn’t work on Qwen3

Between February 15 and March 30, I made dozens of commits fine-tuning the podcast flow: prompts, sampling parameters, generation order, leading silence on the voice clone sample, volume normalization, pronunciation of tech acronyms. Fixing Marvin’s voice, which was clipping the first syllable because the reference audio started without any lead-in silence. Tuning Akita’s voice to sound more confident and assertive. Adding crossfades between section jingles. Sorting out the “podcast” pronunciation so it didn’t come out as “pódcast”. Making the script generator prefer Portuguese over gratuitous anglicisms so the TTS wouldn’t choke. Each of those fixes was a multi-hour session of listening to audio, regenerating, tweaking a parameter.

The result came out acceptable. “Acceptable” in the sense that I managed to ship every episode without re-recording anything by hand. But if you listen closely, the Qwen3 voice has that unmistakable AI-generating-audio quality. Flat intonation, uniform rhythm. Over long stretches you can feel it’s a machine talking. Good enough to ship, but miles away from what you’d hear on a professional podcast made by humans.

The worst problem was English pronunciation. My podcast covers tech news, so terms like “MCP”, “RAG”, “Claude Opus”, “GPT-5”, “open source” show up in every conversation. Qwen3 took those terms and pronounced them with a Brazilian accent, something like “oh-pen-ee-sohrss-eh”, that kind of thing. Unlistenable for the audience. The workaround I had to implement was manually mapping in the script-generation LLM prompt which English words to swap for Portuguese equivalents. The prompt today has a whole section split between “keep in English” (proper nouns, brands, terms already adopted into Brazilian tech lingo) and “translate to Portuguese” (gratuitous anglicisms), looking roughly like this:

**REPLACE with Portuguese** (common English words that have natural
Brazilian Portuguese equivalents):
- "update" → "atualização"
- "release" → "lançamento"
- "feature" → "recurso/funcionalidade"
- "deploy" → "implantação" or just "colocar em produção"
- "trade-off" → "dilema" or "escolha"
- "performance" → "desempenho"
- "default" → "padrão"
- "insight" → "percepção/sacada"
- "skills" → "habilidades"
- "approach" → "abordagem"
- "highlights" → "destaques"

I call this “scrubbing anglicisms so the TTS doesn’t choke”. The funny part is I didn’t want that level of restriction on my script. I wanted the voice to just pronounce “update” when “update” was the natural word. Since the model couldn’t, I had to mutilate the podcast’s vocabulary so the final result was actually listenable. It’s a band-aid, the kind you add hoping to rip it off later when the tech grows up.

The experience with ElevenLabs v3

Yesterday afternoon I opened an ElevenLabs account, bought the Pro plan ($99/month), and started experimenting with the eleven_v3 model, which shipped in February of this year. Thirty minutes later I had a proof of concept running, and about two hours later the whole podcast system was migrated. The effort gap is an abyss.

The technical details of the migration ended up in an internal project doc, so here’s the summary comparison table that actually matters:

Look at the last line. Qwen3 costs less than ten cents of a dollar per episode. ElevenLabs costs almost thirty times more. And it’s still worth it. The other lines on the table solve problems that were eating hours of my week. I no longer need to write code to scale GPUs in the cloud, I don’t need to wait for the machine to spin up every time, I don’t need to babysit a model or rebuild a Docker image when the weights change. The operation became a single line of configuration.

And here’s the interesting part: the inline emotion tags. The v3 model accepts markers like [sighs], [sarcastically], [dryly], [excited], and changes the delivery accordingly. This works in more than 70 languages, including Brazilian Portuguese. It transformed script generation, because now I can ask the LLM that writes the script to drop emotional tags at the right moments, which gives a liveliness Qwen3 couldn’t deliver in a million years. A concrete example of what comes out of the script later:

AKITA: Isso é simples. [dismissive] Quem ainda acredita que Bitcoin
vai morrer não tá prestando atenção.
MARVIN: [sighs] Mais uma semana, mais uma leva de devs confiando
cegamente em pacotes npm. Previsível.

I even have two separate tag palettes, one for Akita (expressive but controlled, uses [excited], [dismissive], [emphatic]) and another for Marvin (stoic, just [sighs], [sarcastically], [tired], [dryly]). That’s all encoded in the script generation prompts so the LLM knows which character is allowed to use what.

About Marvin’s voice

For listeners who are already used to Marvin’s voice, don’t worry: I cloned him on ElevenLabs using the same audio sample I had used to train on Qwen3. It’s the same voice. It just sounds even better now, because the ElevenLabs model captures nuance and prosody that Qwen3 couldn’t deliver.

Listen to it and tell me

To prove the talk is real, here’s Monday’s episode, April 6th, already generated with the new pipeline:

If you’re already a podcast listener on Spotify and you’ve heard previous episodes made with Qwen3, compare them and tell me in the comments whether you notice the difference or whether it’s the same to you. I’m genuinely curious how much of this is my trained ear after hours of listening to TTS audio and how much is an obvious difference for a casual listener.

Starting next week, every episode of the podcast will be generated by ElevenLabs v3. The newsletter is already plugged into the new pipeline, the scheduled jobs to pre-warm the RunPod GPU have been disabled, and the legacy Qwen3 code stays in the repo as plan B in case v3 starts acting up chronically. Two file edits and I can switch back. I probably never will.

The part about dubbing the YouTube videos

Now the hook for the second half of this post. On the anniversary piece I published earlier today, I told the story of how Claude Code translated nearly half my blog to English over a weekend. In the same spirit, I went after dubbing the videos on the Akitando channel.

The channel has 146 episodes, around 96 hours of technical content in Portuguese, and more than 500k subscribers. I already had the subtitles translated (one curated .srt per episode), and YouTube even offers automatic English dubbing. But the result is like Google Translate circa 2015: gets the idea across, nobody wants to listen for very long.

I tested ElevenLabs’ three voice approaches and only one worked. Speech-to-Speech converts a voice but doesn’t translate. The Dubbing API does translate but creates the voice on its own, with no way to force a specific clone. Only Text-to-Speech could solve this: take the English .srt I already had, send each block to the TTS endpoint with my cloned voice, and assemble the audio aligned to the original video.

But having a translated .srt is not enough. Raw subtitle translation doesn’t survive TTS. Passages with source code, URLs, hex hashes, lists of shell commands — the model switches to spelling mode and the audio comes out at twice the expected duration. Translations that run too long blow past the time window and need to be condensed so the voice doesn’t sound like an auctioneer. Truncated SRTs need to be completed. And after each fix, you re-run the pipeline, listen, find the next problem, fix it, re-run. The whole thing was a cycle of interruptions, manual corrections, and re-runs — a far cry from the “press a button and get a dub” fantasy.

The big challenge was the accent. My cloned voice was trained on Brazilian Portuguese audio, so when it tries to speak English, the thick accent comes through. ElevenLabs has an [American accent] tag that works on v3, but on top of a voice trained in another language it’s weak — the Brazilian accent still bleeds through. The fix was to train a second voice of mine, English only. I recorded a few minutes in my best American accent, uploaded it as a separate Instant Voice Clone in my account, named it “Akita English”, and set the pipeline to use that voice by default. The result comes out more natural, no tag needed, and the voice identity is still mine.

The honest ceiling on my English voice quality

Let’s be frank. I’m a Brazilian who speaks English, not a native speaker, and I don’t have the time or patience to drill pronunciation for hours. A clone can only ever be as good as the source sample — if the source is a non-native reading a script, the clone inherits every imperfection. No clone is going to make me sound like “Fabio speaking perfect English” because that sound doesn’t exist for the model to learn from.

To get to the version that’s running today, I recorded five different training samples throughout the day, each with a different rhythm and script, and ran an A/B battery against the original Portuguese. None sounded native, and that was never the goal. The realistic bar was more modest: sound recognizably like me, read technical content without stumbling, and carry a full one-hour episode without wearing the listener out. The winner ended up being the first iteration, the original “Akita English”. Far from perfect, but good enough to ship. Swapping it out later is literally one line in the config file.

What surprised me most along the way was realizing that the rhythm of the training sample matters more than the timbre. The clone learns the cadence of whoever recorded it: iteration 5 came out 25% slower than iteration 2 reading the exact same text, purely because I recorded that particular sample more slowly. For dubbing tech videos, the right target is the rhythm of a “YouTube tech explainer” — brisk, energetic, around 150 words per minute. No audiobook pacing.

How I align audio to the original video

A non-obvious point: spoken English tends to be longer than equivalent Portuguese. If you just generate the audio for each subtitle and paste it at the original timestamp, the dub drifts later and later. With 30 minutes of video you’re already several seconds out of sync.

My approach was to attack the problem in layers. The first version split the SRT into smaller blocks, called “chunks”. Each chunk maxed out at 700 characters (so v3 wouldn’t hallucinate) and could only cut at a sentence boundary, never mid-sentence. A simple algorithm accumulated subtitles into a buffer until it was about to overflow, then walked back looking for the last cue whose text ended with ., !, or ?, and flushed only up to there. The cues still in the buffer got rolled into the next chunk. Each chunk also stored the start and end timestamps of the subtitles it covered, so the assembly step knew exactly where to drop that piece of audio later.

# For every incoming cue, check whether adding it
# would push the buffer past the char cap.
for cue in cues:
    projected = buffer_char_len(buf) + len(cue.text)
    if projected > max_chars and buf:
        # Walk back for the last cue in the buffer that
        # ends in '.', '!', '?', or '…'.
        sentence_end = last_sentence_end_index(buf)
        if sentence_end is not None:
            buf = flush(buf, sentence_end, chunks)
        else:
            # Run-on sentence longer than max_chars with no
            # terminator. Rare, but happens. Warn and cut.
            log.warning("run-on sentence > %d chars — "
                        "splitting mid-sentence as a last resort",
                        max_chars)
            buf = flush(buf, len(buf), chunks)
    buf.append(cue)

    # Soft-break: if the buffer is already at least 60% full
    # AND the current cue ended a sentence, flush now. Keeps
    # chunks balanced around 60-100% of max.
    if cue.ends_sentence and buffer_char_len(buf) >= max_chars * 0.6:
        buf = flush(buf, len(buf), chunks)

Technically, this was the turning point: instead of letting the dub follow the visual structure of the subtitle file, the chunker started following the real paragraph structure of the script. I won’t retell the whole story of how I got there and why it eventually forced a mass rerender, because I come back to that later in the section about the problems that only showed up after the first upload. The technical point here is simple: once the window represented a real spoken paragraph, the audio started sounding natural, and the stretch target could move from 92% to 95%.

Once the chunking is done, there’s still the problem that English runs longer than the Portuguese equivalent when spoken. The trick here is predicting whether a chunk is going to blow past its target window before generating the audio. If the ratio characters / 16 chars-per-second already exceeds the target duration by a margin, the pipeline passes the speed parameter directly to the ElevenLabs API, asking the model to natively generate faster speech. This preserves prosody way better than compressing afterwards. The ceiling is 1.15× (beyond that the voice starts sounding rushed).

# before calling the API, predict if it will overrun
expected_sec = char_count / EXPECTED_CHARS_PER_SEC  # 16 chars/s
predicted_ratio = expected_sec / target_sec

voice_speed = 1.0
if predicted_ratio > PREEMPTIVE_SPEED_THRESHOLD:   # 1.05
    voice_speed = min(predicted_ratio * 0.98,
                      PREEMPTIVE_SPEED_MAX)         # cap at 1.15
    voice_settings["speed"] = voice_speed

Even with preemptive speed turned on, sometimes the generated audio still comes out a tiny bit longer than the target window. That’s what the second safety net is for: measure the actual audio with ffprobe after it’s generated, compare to the target window, and apply the ffmpeg atempo filter to compress in post if it exceeded the 2% tolerance, capped at 1.20×. Combining native speed (1.15×) with atempo (1.20×) gives an effective compression ceiling of 1.38×, enough to fit naturally even in the worst cases without breaking quality.

# after generating the chunk
actual = ffprobe_duration(chunk_path)
ratio = actual / target_sec

if ratio > FIT_TOLERANCE:       # 1.02
    # compress the audio without touching pitch
    ffmpeg_atempo = min(ratio, FIT_MAX_ATEMPO)   # 1.20
    apply_atempo(chunk_path, ffmpeg_atempo)

When a generated chunk comes out shorter than the target window, the pipeline slightly stretches the audio (without affecting pitch) to reduce the ugly silence after it ends. The goal isn’t to fill the whole window — a natural pause between sentences is fine — just to smooth over the worst offenders that would feel like “cut audio”.

On a big episode, 95 minutes, 194 chunks, the total cumulative drift came out to -0.7%. About 43 seconds of drift across the entire episode, imperceptible while you’re watching.

What saved the budget on this architecture is that each chunk lives on disk as its own .mp3. The pipeline keeps a manifest with the normalized text of every chunk, and before calling the ElevenLabs API, it compares the current text against the cache. If the text hasn’t changed, it reuses the existing audio without spending a single credit. If I rewrite a problematic cue, only the chunks affected by that cue get regenerated — the rest of the episode stays untouched.

This is what made the iterations viable. I’d run the batch, listen to sections, spot a problem (translation too long, a code snippet the TTS couldn’t pronounce, a chunk boundary that landed at a bad spot), fix the SRT or tweak the chunker parameters, and re-run. Each re-run consumed a fraction of the credits and time of the original run, because only the changed chunks were regenerated. Without this cache, every iteration would have cost nearly as much as the first pass, and the total batch cost would have been two or three times higher.

That chunking change also had a heavy cache and cost impact, but I’ll leave that part for the later section where I explain the actual retracing and rerender cycle. Here, the important point is just that it was the right technical move.

When atempo can’t save you: rewriting cues before TTS

Not everything is about window alignment. There’s a kind of cue that breaks the whole pipeline, where neither preemptive speed nor post-process atempo can save it, and I only ran into it once I started running the batch over the more technical episodes of the channel.

The problem is this. The v3 reads at about 16 characters per second when you hand it normal conversational text. But when you hand it a cue stuffed with a literal URL, a hex hash, a binary string, a long sorted number list, or a shell code block, the model shifts into “spelling it out letter by letter” mode and drops to about 9 characters per second. A 500-character cue that was supposed to turn into 30 seconds of audio comes out at 55. The sanity check rejects it (because it’s past 1.8×), the automatic retry tries the same 500 characters again, the five retries all fail in a row, and the chunk gets stuck.

And this did not happen by accident. I used to write those passages that way because I knew the same scripts would later become blog posts, so it made sense to leave the raw URL, the full command, the complete hash, the whole technical detail there for the reader. In Portuguese that worked fine because it matched my original workflow exactly: record the video, then publish the matching text on the blog. What I had never prepared for was the next step, turning that same material into automated English dubbing. I only realized that conflict when the pipeline started failing for real.

I hit this first on ep052, the Ubuntu beginner’s guide, where two cues carried github.com URLs, hkp://keyserver.ubuntu.com URLs, and a 40-character GPG hash. Trying to fix that in post-processing is a waste of time. The 1.20× atempo ceiling will never compress 55 seconds into 30, and even if it did, the result would be an unlistenable chipmunk. The fix has to land upstream, in the text itself.

The solution was to rewrite the cue so it describes what the command does instead of showing the literal command:

- Run: apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 \
-   --recv-keys 0A6A3E7F79F93EF8AAB9E92BAEBB74C8B5A1E44D
+ Run the full command in the video description to import the
+ signing key from the Ubuntu keyserver.

The on-screen English caption still shows the full command, so anyone reading the caption sees exactly what they need to type. The dubbed audio describes the intent in spoken language, which is what the TTS can deliver without choking. The instruction to the viewer stays intact, the characters-per-second ratio goes back to the expected 16, and the generated audio fits the original window without needing any stretch or compression in post.

I went hunting for cues like this across the whole batch. I scanned from ep052 through ep146 looking for cues with a high density of “hard” characters (digits, brackets, operators, long URLs, binary, hex) and ended up rewriting around 30 cues across 13 episodes:

• ep052 Ubuntu for devs: GitHub URLs and a 40-character GPG hash
• ep091 Hello World in C: binary examples
• ep095 640kB memory: address wrap, segment-offset
• ep106 CodeMiner: spelled-out email address
• ep113 Compression: pure binary strings, 75% hard characters
• ep115 SQL Server: long sorted number list
• ep120 Internet: dotted IP addresses
• ep121 Sockets: two JavaScript code blocks
• ep122 Proxies: Chrome User-Agent headers
• ep123 Secure networking: shell one-liners, Docker flags
• ep126 Gentoo: C chroot demo
• ep136 Containers: GitHub release URL
• ep144 Cryptography: signing key address

The pattern was always the same: keep the caption showing the code, URL, or hash verbatim, and rewrite the spoken text so the narrator describes what it does. I could have automated the rewrite by letting an LLM read each cue and propose a substitution on the fly, but it felt safer to review them all by hand. It’s exactly the kind of thing where the model decides to “improve” a command to make it cleaner and ships with broken shell. The manual sweep took about two hours. The automated version would have cost the same time in review, with the bonus anxiety of having shipped a mangled command by accident.

There was also one weird structural case on ep146, about Docker Compose. Two cues had glued themselves together because of a missing blank line in the SRT, and pysrt was treating them as one giant cue. The TTS never even got to process it — the chunker choked first. I fixed it by hand, adding the missing blank line. One-character fix, half an hour to track down.

Reconstructing truncated SRTs

Another trap showed up when I went to run the full batch: three episodes had English captions that were too short relative to the actual video length. Ep056, the Rails episode, was the most absurd case. Nineteen minutes of caption for an 80-minute video. Sixty-two minutes of content with no caption at all. Ep057 (WSL 2) was missing 14 minutes, and ep068 (Git Direito) was missing 2.

This is a leftover from my old translation workflow. At some point I started hand-revising the caption, stopped partway through, and the .en.srt file got saved truncated at the spot where I stopped. You can’t dub an 80-minute video with a 19-minute caption. The chunker produces audio up to where it can read and then simply has no idea what to do with the rest of the video.

The fix became a new script. It diffs the truncated .en.srt against the .pt-orig.srt (the raw YouTube auto-caption, which always covers the whole video), picks up the cues that only exist in Portuguese, sends them to Claude Sonnet 4.6 with a strict JSON schema asking for cue-by-cue translation, and pastes the result back at the tail of the English file. The schema is the detail that matters. When I tried asking for the response as plain text in N|text format, Claude was dropping about 20% of the cues along the way. With a strict JSON schema, the drop rate fell to zero across the three repairs.

The numbers:

• ep068 Git Direito: +50 cues / 2 minutes reconstructed
• ep057 WSL 2: +368 cues / 14 minutes
• ep056 Rails: +1445 cues / 62 minutes (plus dropping 5 fake cues the old translator had invented at the tail to plug the gap)

After the repairs, the three episodes joined the batch normally and got dubbed like any other. The kind of tool you hope you never need, but when you need it, it’s worth writing once and closing the problem for good.

Automatic emotion tags

While I was testing, I decided to add one more step to the pipeline: an “emotion tagger” that reads the English SRT before it goes to TTS and inserts tags like [sarcastic], [thoughtful], [emphatic], [deadpan] at spots where a human narrator would naturally shift tone. The idea is to mimic what a professional voice actor would do — hit the emphasis at the right moments — without turning the video into a theater of overblown emotions.

This part is tricky for two reasons. First: letting an LLM loose on your SRT runs a real risk of it “improving” the text (swapping a word here, rewriting a sentence there) and you ending up with a dub that doesn’t match the caption. Second: LLMs love to overdo it. Ask them to tag emotion and they’ll put a tag on every other line. To guard against both, I pinned a small allow-list of tags and run a round-trip validation after Claude’s response comes back:

ALLOWED_TAGS: frozenset[str] = frozenset({
    "[sarcastic]", "[thoughtful]", "[emphatic]",
    "[deadpan]", "[serious]", "[amused]",
    "[sighs]", "[exasperated]", "[confident]",
    "[matter-of-fact]",
})

def validate_tagged(original, tagged):
    """Ensure the tagged SRT preserves the original with no drift."""
    if len(original) != len(tagged):
        raise TagValidationError("cue count mismatch")

    for o, t in zip(original, tagged):
        # Index and timestamp must match byte for byte.
        if o.index != t.index or o.timestamp != t.timestamp:
            raise TagValidationError(f"cue {o.index}: header drift")

        # Any tag outside the allow-list invalidates the whole response.
        bad = find_disallowed_tags(t.text)
        if bad:
            raise TagValidationError(f"cue {o.index}: {bad}")

        # The text with the tags stripped must be byte-identical to
        # the original. If the LLM swapped a word, rewrote anything,
        # validation fails and the response is rejected.
        if strip_tags(o.text) != strip_tags(t.text):
            raise TagValidationError(f"cue {o.index}: text drift")

When validation fails, the pipeline throws the response away and re-requests (or, if it keeps failing, ships the SRT untagged). On tag density, the prompt sent to Claude explicitly says: aim for N/10 tags in a SRT with N cues, hard floor of 1 tag per 12 cues, hard ceiling of 1 tag per 7 cues, distribution balanced across the four quarters of the episode, and no two tagged cues within 4 cues of each other. That rule set was the sixth iteration — the five before it either blanketed everything in tags or loaded them all into the first quarter.

Don’t let v3 hallucinate

A critical detail that only shows up in production: ElevenLabs v3 tends to hallucinate when you send very long blocks of text. I’ve seen blocks where the input text was around 1,500 characters and the model generated nine minutes of audio when the expected duration was a minute and a half. The model just decides to keep talking on its own, inventing content.

The official ElevenLabs docs recommend keeping each call under 800 characters to avoid this. I went more conservative and cut at 700, always at sentence end (never mid-sentence, because a mid-sentence split sounds horrible when concatenated). On top of that, I bumped stability to 0.9 (Robust mode, more stable but less responsive to emotion tags), turned on apply_text_normalization so the model pronounces numbers and acronyms correctly, and added a sanity check that rejects any generated audio longer than 1.8× the expected duration.

DEFAULT_VOICE_SETTINGS = {
    "stability": 0.9,          # Robust mode
    "similarity_boost": 0.95,
    "style": 0.0,
    "use_speaker_boost": True,
}

EXPECTED_CHARS_PER_SEC = 16.0
CHUNK_SANITY_MAX_FACTOR = 1.8   # reject if actual > 1.8× expected

With those mitigations in place, a 95-minute episode (194 blocks) ran start to finish with zero hallucinations. Exactly one block failed mid-run on a transient 502 error from the API, which the automatic retry caught on the next attempt.

Mastering for YouTube

YouTube doesn’t publish an official number, but industry consensus is that its playback loudness normalization targets -14 LUFS. Deliver audio louder than that and YouTube turns it down; deliver it quieter and it leaves it alone. To land on the target exactly, the pipeline runs ffmpeg’s loudnorm filter in two passes. The first pass measures the entire audio and prints the stats (input_i, input_tp, input_lra, input_thresh) as a JSON blob on stderr. The second pass reads those stats back in and applies a linear static gain to land precisely at -14 LUFS, -1.5 dBTP:

# Pass 1: measure
ffmpeg -i final_en.mp3 \
  -af "highpass=f=80,loudnorm=I=-14:TP=-1.5:LRA=9:print_format=json" \
  -f null -

# Pass 2: apply linear gain with the measured values
ffmpeg -i final_en.mp3 \
  -af "highpass=f=80,loudnorm=I=-14:TP=-1.5:LRA=9\
:measured_I=-18.3:measured_TP=-3.1:measured_LRA=5.4\
:measured_thresh=-28.7:offset=1.2:linear=true" \
  -ar 48000 -ac 2 -c:a pcm_s16le final_en_mastered.wav

Two passes instead of one because single-pass mode runs in dynamic compression and ends up “pumping” the gain during silent stretches. With linear=true on pass 2, the gain stays static on top of the pass-1 measurements, so no pumping. The result lands within ±0.1 LU of the target, which is effectively inaudible. The highpass=f=80 filter in front kills HVAC rumble and mains hum below 80 Hz, which the human ear doesn’t catch but will move peak measurements around.

The point where I thought it was done. And it wasn’t.

This is where the most annoying, and most educational, part of the project starts.

I really thought I had nailed it on the first big pass. The pipeline closed all 146 episodes, I uploaded everything, and it looked done. The durations matched, the files were all there, and the whole process had finally crossed the complete batch of nearly 96 hours of video. But it was exactly after that upload that I started noticing problems I simply had not anticipated.

The first scare was the jingle situation. I already knew many Akitando episodes begin with that short instrumental intro before I start talking, but I underestimated how much trouble it would cause. YouTube’s .srt is almost useless here, because a jingle is not speech. Sometimes it marks [Music], sometimes it gives you nothing useful, sometimes the window is just wrong. In the first assembly pass, some episodes lost the jingle entirely, some got it doubled, and in others the source-fill from the original audio already contained the right jingle and my splice came in on top of it, pushing everything else forward. This was the kind of bug you only catch when you sit down and actually listen to the final mastered file, not when you just look at total duration.

The fix was to stop hoping the subtitle would tell me where the jingle was and go straight to the source. I built a detector using the original video audio itself plus reference jingles, with an audit and repair pass on top of the mastered outputs already generated. That eventually became a small subsystem of its own: detector, audit, cached reassembly, exact-window repair. At the end of that detour, the set of changed episodes collapsed into a well-defined reupload bucket and the jingle problem finally came under control. It was one of those classic production moments: the first solution looked good enough until it hit the whole archive.

Then came the second punch, and this one was more expensive: I had forgotten a basic fact about YouTube subtitles. SRT is built to be read on screen, not to become spoken script. So an idea that was a full paragraph in my original text showed up split into several small cues, each with its own micro time window. In the first version of the pipeline I was using those cues as the generation unit for ElevenLabs. The result was TTS audio that was coherent locally, but the final assembly inserted artificial silences in the middle of sentences. Technically aligned, humanly weird.

Fixing that hurt because it meant walking several steps back. Instead of respecting the cue structure of the SRT, I had to go back to my original scripts and rebuild chunking around the paragraphs I had actually read when recording the videos. Almost every Akitando episode has a matching blog post with a ## Script section, and that is what saved the project. Because I had been meticulous back when I made those videos — writing the scripts first, reading exactly that on camera, and archiving both the videos and the scripts — I had a source of truth. That made it possible to map the script paragraphs onto the timings in the .pt-BR.srt, regroup the .en.srt on top of that, and regenerate the chunks in the right structure.

The problem is that this destroys the previous cache. Even when the text barely changes, if the chunk boundary changes then the TTS call changes too: different context, different prosody, different beginning, different ending. So this was not a touch-up. It was a full regeneration of everything again. That was the point where the cost started escaping my original estimate.

And just when I thought now it really was over, the third hit landed: external clips. Many videos on the channel include inserted footage from other sources. The Ruby on Rails episode, for example, has 37signals clips, Apple material, commercials, external inserts in the middle of the narrative. YouTube’s .srt did not know what to do with that. In some places it simply hallucinated text, in others the timings were completely wrong, and in others the original speech had nothing to do with the subtitle I was using as ground truth. Again: if you only look at the automation, it looks fine; if you actually listen to the final master, it isn’t.

The fix, again, was to leave the subtitle behind and go back to the original material. I built an external-clip window detector based on alignment gaps between transcript, .pt-BR.srt, speechless cues and chunks that crossed those windows. Then, instead of trying to dub what made no sense to dub, the process started recovering the original audio from the source video exactly in those stretches and filling the master with the correct source audio there. That’s what fixed the 37signals, Apple Education, Mac vs PC commercial and other inserted segments I had forgotten were even there.

That was the biggest lesson from the second half of this project: automation takes you very far, but the full archive always has more dark corners than you imagine on the first pass.

The dubbing batch numbers (and why it hurt more than I expected)

I need to correct myself again. I didn’t just miscalculate the first bill. I also overestimated how confident I should have been after the first full run.

When I upgraded to ElevenLabs Business, I thought I would finally have plenty of headroom. In practice, I didn’t. The latest dashboard screenshot shows 10,046,279 credits consumed out of a limit of 11,000,000, leaving 953,721:

In plain English: I burned through more than 91% of the Business workspace limit. And that helps explain what “10 million credits” really means in this context. It does not mean “I rendered 146 episodes once”. It means rendering 146 episodes, discovering a structural problem, going back, fixing it, regenerating chunks, discovering another structural problem, going back again, rebuilding masters, auditing jingles, repairing external clip windows, uploading again. Every extra round costs money. And when the change invalidates the cache, it costs a lot.

The cost also doesn’t stop at ElevenLabs. All the subtitle curation, problematic cue rewrites, truncated SRT reconstruction, jingle audits, external clip detector work and the rest ran through Claude Code on the Claude Max 20× plan. Combined with the mass blog translation to English that I described in the anniversary post, the Claude Max 20× weekly limit hit 100%, with more than BRL 300 in extra usage on top. If you think generative AI becomes free once you’ve signed up for the plan, the invoices disagree.

Even so, I still think it’s worth it. We’re talking about 146 episodes, almost 96 hours of technical content, more than 5 years of channel archive. Doing that manually in a studio, with an actor, direction, pickups and human review on everything, would cost a completely different order of magnitude.

What changed was how I think about the project. At the start I was looking at this as “one very large batch”. In the end it was something else. It was digging through a whole archive, finding hidden exceptions, fixing old rot, rebuilding masters, rerendering, relistening.

The first full batch took a little under two days and made me think I had solved everything. I hadn’t. The second pass, with paragraph-aware chunking, was already expensive. The third, with jingle repairs and mastered-output fixes, cost more again. And the fourth hit came from the external clips I had not modeled correctly in the first automation pass.

But now, finally, I think it’s done. Or rather: I hope it’s done. Everything has been uploaded again, for the last time. It isn’t perfect in the absolute sense of the word. It’s as good as I can get it with scripts, audits and automation, without descending into manual-intervention hell episode by episode.

And if all of this was possible, it’s because years ago I had the discipline to produce the videos the right way: write first, read the script when recording, and archive the materials afterward. Without the original videos and without the original scripts, I would have had no way to discover where the .srt was lying, where the timing was wrong, where the external clip started, where the original paragraph began and ended. That old organization is what made it possible to repair the project now.

So yes: the dub for all 146 episodes is done. Not perfect. But closed. At least, I sincerely hope this time it really is.

The first test, watch it

I made a test video to prove the thing works. Two caveats up front: first, the embed below already loads with English captions on by default, so that side is solvable via URL. Second, the audio is annoying: YouTube removed the audio track selector from embedded players back around March of this year. So inside the embed you won’t find that option in the gear menu at all, it just plays in Portuguese. To actually hear the English dub, click watch directly on YouTube — the main YouTube watch page still has the audio track switcher in the gear menu.

If you’re used to YouTube’s automatic dubbing or to the AI dubbing on TikTok, compare them. The difference is glaring. The voice is mine, with a worked-over American accent, and the sync with the original video stays within 1 to 2% cumulative drift, practically imperceptible over 95 minutes of video.

And the missing piece: translating the thumbnails

While I was closing this post, the penny dropped: I’d left a loose end. All 146 thumbnails on my channel are in Portuguese, many of them with big all-caps titles like “9 DICAS PARA PALESTRANTES” or “7 RECOMENDAÇÕES DE SHOWS PARA PESSOAS DE TECH”. Perfect English audio doesn’t help if the image in the search results is an illegible block of text to anyone who doesn’t read Portuguese. YouTube lets you upload an alternative thumbnail per language, so I needed a way to generate English versions, keeping the rest of the art identical and swapping only the text.

The tool uses two pieces. yt-dlp is the modern fork of the old youtube-dl, a Python CLI that downloads anything from YouTube (videos, audio, subtitles, thumbnails) without needing an API key. Nano Banana Pro (nano-banana-pro-preview on the API) is Google Gemini’s latest image-editing model — you feed it an input image with a prompt and it returns an edited image, preserving the rest of the composition when you ask it to touch only a specific part.

The pipeline has two steps. Step 1: yt-dlp grabs the thumbnail as a .jpg, no video download:

yt-dlp --skip-download \
       --write-thumbnail \
       --convert-thumbnails jpg \
       -o "thumbnails/originals/<slug>/<video_id>" \
       "https://www.youtube.com/watch?v=<video_id>"

Step 2: send each image to Gemini Nano Banana Pro with a prompt that’s a rigid contract of hard requirements, not just a “translate this”:

TASK:
Detect every piece of Portuguese text visible on this image and
translate it to clear, natural American English. Replace the
Portuguese text with the English translation in the same visual
position.

STRICT REQUIREMENTS — follow every one:
- Preserve EVERY non-text element identically: face, pose,
  expression, background, color palette, lighting, icons, logos,
  decorative shapes, borders, layout. Only the text changes.
- The English translation must be IDIOMATIC and CONFIDENT — not
  a literal word-for-word rewrite. It's a YouTube thumbnail for a
  tech audience, so use punchy phrasing a native English-speaking
  tech YouTuber would write.
- Match the original text's font family, weight, size, color,
  stroke outline, drop shadow, and any decorative treatment.
- If there is no Portuguese text at all on the image, return the
  original image unchanged.

The “return the original image unchanged” clause is what saves the earliest episodes, which don’t have overlay text, just my face. Without it, the model would invariably try to “improve” the composition, swap the lighting, and so on.

Here’s the result on two examples (episodes 10 and 11):

Original (PT)

Translated (EN)

Original (PT)

Translated (EN)

Notice the detail on episode 10. The model didn’t translate “7 Recomendações de Shows para Pessoas de Tech” literally into “7 Show Recommendations for Tech People” the way Google Translate would. It rewrote it as “7 TV Shows You Must Watch If You’re In Tech”, the way an actual American tech YouTuber would headline the video. The rest of the image stays pixel-for-pixel identical. On episode 11, “9 DICAS PARA PALESTRANTES: venda sua caneta” becomes “9 TIPS FOR SPEAKERS: sell your pen”, with the font, all-caps treatment and position preserved. If you didn’t know the original was in Portuguese, you couldn’t guess the second image is an automatic AI edit.

Cost on the Gemini side: a few cents per image, a handful of dollars total for the whole batch. Trivial next to the $1,500+ on the audio dub. With the thumbnail sorted, the channel’s English conversion is finally buttoned up — ElevenLabs v3 cloning the voice over hand-curated English .srt files, and Nano Banana Pro editing the image so the title matches the audio.

The conclusion, which is the title of this post

When Qwen3 TTS dropped, the hype was calling it an “ElevenLabs killer”. I spent weeks trying to live with that premise in practice, with real content shipping every week. And what I found is that open source TTS is still miles behind ElevenLabs. The gap is large, and it’s a gap that shows up precisely when you step off the 5-second demo tweet and put the model to work on a real 30-minute weekly podcast.

In practice, Qwen3 doesn’t even beat ElevenLabs’ v2 model. v3, which is the current one, is still a step above v2. The prosody comes out better, the inline emotion tags work in Portuguese and English without effort, and the API stays up without you maintaining a server. The cost per character is a bit higher, but for my current volume it sits comfortably within budget and gives me back the hours I was spending on GPU babysitting.

The lesson here is the same one the LLM crowd is learning slowly. A 30-second demo tweet is one thing, production is a completely different thing. Open source has its niche, mainly when you have sensitive data that can’t leave the house, or when you have tons of idle GPU and little recurring budget. But for serious TTS use in a commercial product, here in April 2026, ElevenLabs is still untouchable. Qwen3 didn’t kill anyone.

And don’t forget: subscribe to The M.Akita Chronicles on Spotify so you don’t miss new episodes like this one, made with the new pipeline.

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

I’ll talk about the podcast today; remember that the pilot episode is already live (transcripts and RSS to subscribe) and on the channel on Spotify.

–

Every AI tutorial starts with “install the model, run inference, done!”. None of them tell you what happens when you need to run this in production with real uptime, controlled cost, and consistent quality.

I’ll tell you how it was to put a TTS (Text-to-Speech) model into production on a GPU in the cloud. Because if you’re thinking about serving AI models, you need to know what you’re getting yourself into.

The Scenario

The goal: generate a ~30-minute podcast from text, with two distinct voices, broadcast quality, ready for Spotify. Every week, automatically, with no human intervention.

The chosen model: Qwen3-TTS, a text-to-speech model that supports voice cloning and fine-tuning. It runs on GPU — needs at least 6GB of VRAM for inference.

The platform: RunPod, which offers serverless GPUs with billing by the second of usage.

Sounds simple, right? Well, hold on.

Problem 1: Cold Start — The Silent Killer

Serverless GPU works like this: when there’s no request, the worker shuts down. When a request comes in, it boots up, loads the model into VRAM, and processes.

The time to “load the model into VRAM” is the cold start. For a ~4GB model, we’re talking 30 to 90 seconds. Your HTTP request sits there hanging, waiting. If your timeout is 30 seconds, you don’t even get to see the model work — you just see a timeout.

The naive solution: “bump the timeout!”. The real solution: health check with polling.

@app.get("/health")
async def health():
    if not model_loaded:
        return JSONResponse(
            status_code=503,
            content={"status": "loading", "detail": "Model loading..."}
        )
    return {"status": "ready"}

The client (in this case, a Rails job) polls /health before sending the actual request:

def wait_for_server(max_wait: 180)
  deadline = Time.current + max_wait
  loop do
    response = http_get("/health")
    return if response.code == 200
    raise "Server não iniciou a tempo" if Time.current > deadline
    sleep 5
  end
end

Sounds primitive? It is. But it works better than any sophisticated “warm pool” solution that costs 10x more.

Problem 2: Where Do the Model Weights Live?

A 4GB model needs to be available when the worker starts. Three options:

1. Bake into Docker: 4GB+ image. Slow build. Slow push. Slow pull.
2. Download from HuggingFace on boot: Network-dependent. Can fail. Adds minutes to the cold start.
3. Network Volume: Persistent disk mounted on the worker.

Option 3 is the correct one. RunPod offers Network Volumes that persist between runs. The model sits there, always ready. The cold start is just the time to load from disk to VRAM, no download.

But there’s a catch: the volume may or may not be mounted, depending on how the worker is configured. So the setup code needs to be resilient:

NETWORK_VOLUME = "/runpod-volume"
LOCAL_FALLBACK = "/workspace/models"

def resolve_model_path():
    if os.path.isdir(NETWORK_VOLUME):
        base = os.path.join(NETWORK_VOLUME, "huggingface")
        os.environ["HF_HOME"] = base
        return os.path.join(base, "hub", "models--Qwen--Qwen3-TTS")
    # Fallback: local download
    return LOCAL_FALLBACK

Auto-detection. If the volume exists, use it. If not, it works the same way (slower on the first boot). Never assume your infrastructure is perfect.

Problem 3: Two Voice Modes, Two Sets of Parameters

Here’s something no documentation will tell you: TTS models with voice cloning and fine-tuning need completely different sampling parameters depending on the mode.

With a fine-tuned voice (trained specifically on the model), you want conservative parameters:

temperature: 0.5
top_p: 0.8
repetition_penalty: 1.1

With voice cloning (base voice + reference audio), the same parameters produce white noise. Literally. White noise. Because the clone model needs more “creativity” to reconstruct the vocal characteristics:

temperature: 0.9
top_p: 0.95
repetition_penalty: 1.05

I discovered this after hours generating unusable audio. The model’s documentation doesn’t mention it. The academic papers don’t mention it. It’s the kind of knowledge that only comes from experimentation in production.

Problem 4: The Reference Audio Matters More Than the Model

Voice cloning uses an audio clip as a reference to replicate the voice. Sounds simple: record 10 seconds, feed it to the model, done.

Except if the reference audio starts with speech immediately — without even 200 milliseconds of silence at the beginning — the model learns that “the audio starts with no onset”. And every subsequent generation will clip the first syllable.

This is not a model bug. It’s the model doing exactly what you asked for: “generate audio that starts just like this reference”. If the reference starts clipped, the generation starts clipped.

The fix is not post-processing (the audio has already been generated without the syllable). It’s fixing the reference: ensuring at least 300ms of silence before the speech.

For those coming from the software world, this is counterintuitive. We’re used to treating input data as immutable and fixing things on the output. With generative models, input quality is everything.

Problem 5: Timeout and Generation Limits

A TTS model generates audio token by token. If you don’t cap it, it can generate indefinitely — and on RunPod, you pay by the second of GPU. A 30-word turn of dialogue shouldn’t generate 3 minutes of audio, but without a limit, it can.

MAX_NEW_TOKENS = 720  # ~60 seconds of audio at 12Hz
GENERATION_TIMEOUT = 180  # 3 absolute minutes

with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(model.generate, **params)
    result = future.result(timeout=GENERATION_TIMEOUT)

Double protection: token limit on generation and absolute timeout on the process. If either one trips, the turn fails and can be retried with adjusted parameters.

Problem 6: Post-Processing Is Half the Work

The model generates raw WAV. To get to podcast-quality audio, the path is long:

1. Loudness normalization: Spotify requires -14 LUFS. Without normalization, every turn has a different volume.
2. Re-encoding: The model’s WAVs can have subtly different formats (sample rate, bit depth). Concatenating with -c copy causes silent segments. You need to re-encode everything to the same format.
3. Conversion to MP3: CBR 192kbps, 44.1kHz, with ID3 metadata.

And loudness normalization needs to be two-pass: first measure, then apply. Single-pass is a guess — and guessing in production generates audio with inconsistent volume across episodes.

# Pass 1: analysis
ffmpeg -i input.wav -af loudnorm=I=-14:print_format=json -f null -

# Pass 2: applying the measured values
ffmpeg -i input.wav -af loudnorm=I=-14:measured_I=-18.5:measured_TP=-2.1:... output.wav

All of this runs on the server, after TTS generation. And it needs ffmpeg installed in the container — one more dependency to manage.

Problem 7: The Script Matters More Than the Voice Model

After solving GPU, cold start, voice cloning, sampling, normalization — the podcast comes out. And it sounds… robotic. Not because of the TTS. The TTS is excellent. It sounds robotic because the text is robotic.

This is the problem nobody mentions in TTS pipelines: audio quality starts in the script, not in the voice model. A perfect TTS reading a dry, formal text produces dry, formal audio. Garbage in, garbage out — but at a level you wouldn’t expect.

Two Passes: Draft and Refine

The solution is to generate the script in two LLM passes:

Pass 1 (Draft): Turns the newsletter into dialogue. Here the LLM gets the full content and creates the structure — who says what, in what order, with what transitions. Without changing the content, only the form!

Pass 2 (Refine): Takes the draft and makes it natural. Adds hesitations, reactions, interruptions, self-corrections. Turns “read script” into “real conversation”.

Why not do it all in a single pass? Because these are cognitively different tasks. Structuring content and adding naturalness at the same time results in a text that does neither properly. Two specialized passes consistently produce a better result.

Personality Needs Brutally Specific Instructions

“Make Akita sound confident” doesn’t work. The LLM will produce generic confidence — motivational phrases, positive tone. That’s not it. Akita is confident because he’s assertive and impatient. The difference is in the prompt details:

## Akita Delivery Style

- **Declarative, not tentative**: "Isso é óbvio" not "Eu acho que talvez isso seja..."
- **Dismissive of nonsense**: "Isso é bobagem", "Não tem nenhum mistério aqui"
- **Owns his track record**: "Eu avisei", "Quem assistiu meu canal já sabe"
- **Never hedges**: No "talvez", "quem sabe", "pode ser que"
- **Impatient with the obvious**: "Vocês já sabem como funciona"

And for Marvin (the robot co-host):

- **Filler words** (Marvin only): "bom...", "olha...", "enfim", "pois é"
- **Self-corrections** (Marvin only): "quer dizer...", "ou melhor..."
- Akita doesn't second-guess himself — if he corrects, it's
  "Não, peraí" and he restates with even MORE conviction

Without this level of specificity, both characters sound the same. The LLM needs concrete examples of how each one talks, not abstract personality descriptions.

Normalization for Speech: What Looks Right in Text Sounds Wrong in Audio

Acronyms. The invisible enemy of TTS. “EUA” in text is perfectly readable. Spoken by TTS, it comes out “éu-á” — incomprehensible. The prompt needs explicit normalization rules:

## Spoken Text Normalization (MANDATORY)

- `EUA` → `Estados Unidos`
- `UE` → `União Europeia`
- `IA` → `inteligência artificial`
- `LLM` → `modelo de linguagem`
- `API` → `interface de programação`

This needs to be in the prompt for both passes (draft and refine), because the LLM tends to reintroduce acronyms it removed in the previous pass.

The LLM Will Rewrite Your Opinions (If You Let It)

This was the most subtle lesson. The newsletter has opinion commentary from Akita and Marvin in each section. When the LLM turns it into dialogue, it substitutes the opinions with more “balanced” and “nuanced” versions. Akita, who said “isso é bobagem”, becomes Akita saying “isso tem seus méritos, mas…”. Pessimistic Marvin becomes “mildly skeptical” Marvin.

The LLM does this because it was trained to be helpful and balanced. Strong opinions seem “unhelpful” to the model, so it softens them. The original instruction — “use the newsletter comments as seeds for the conversation” — was too vague. It gave the model freedom to reinterpret.

The fix is to be explicit and inflexible:

**PRESERVE THE ORIGINAL OPINIONS**: The newsletter contains Akita and
Marvin commentary on each topic. These opinions are the author's real
voice — you MUST preserve their substance, stance, and conclusions.
Make the delivery conversational, but do NOT change what they actually
think. If Akita's comment says "isso é bobagem", the podcast version
must convey the same dismissal. Rephrase for spoken delivery, but
never substitute a different opinion.

It’s the difference between “adapt it for conversation” (LLM understands: “rewrite however you want”) and “change the HOW, never the WHAT” (LLM understands: “keep the opinion, swap the packaging”).

And this applies to any prompt engineering with authored content: the LLM will homogenize your voice if you’re not explicit about preserving it. And in a podcast where the hosts’ personality IS the product, losing the voice means losing everything.

The Real Cost of AI in Production

Let’s get practical about costs on RunPod:

The cost per episode is low. The cost of figuring out how to make it work is high. It took dozens of hours of experimentation, incomprehensible CUDA logs, failed generations, and unusable-quality audio before landing on a stable pipeline.

And the cognitive cost: keeping two ecosystems (Python for ML, Ruby for the application) in sync, with HTTP interfaces between them, and debugging that crosses language boundaries.

Anyway, for those who thought “Why not Elevenlabs?”, well, I got similar quality at a fraction of the cost, and I learned a lot in the process (and so did you, reading this).

Lessons

1. Cold start is a requirement, not a bug. Design your system expecting that the model will take time to load.

2. Network volumes are mandatory for large models. Don’t bake weights into Docker, don’t download on boot.

3. Sampling parameters are not universal. Each operation mode of the same model may need completely different configuration.

4. Input quality dominates output quality. Both for the TTS reference audio and for the script. If the reference clips the first syllable, every generation clips. If the script is generic, the audio is generic.

5. Timeouts at every layer. Token limit, process timeout, HTTP timeout. If you don’t cap it, cost scales and generation hangs.

6. Post-processing is half the pipeline. The model generates raw material. Normalization, encoding, metadata — all of that is real work.

7. Separate the ML server from the application. Python for inference, Ruby for orchestration. Trying to do everything in one language is a recipe for frustration.

8. Prompts need two passes for complex content. Structuring and naturalizing are different tasks. Forcing both into one pass produces mediocre results in both.

9. The LLM will homogenize your voice. Strong opinions turn into “nuanced takes” if you’re not explicit about preserving them. Instruct the model to change the packaging, never the content.

Conclusion

AI in production is not a 10-line HuggingFace tutorial. It’s infrastructure, it’s experimentation, it’s debugging problems that don’t exist in traditional software. But when it works — when your system generates an entire podcast automatically every week, with broadcast quality, no human intervention — the feeling is that it was worth every hour of headache.

The trick is not to underestimate the complexity. And to have a robust system (like the Rails jobs I mentioned in the first post) orchestrating it all — because AI without reliable orchestration is just an expensive script that hangs.

I jumped to that conclusion too early. Open source options still vary a lot in quality, most aren’t great, but the commercial options are very good and quite stable. I’m not going to list everything out there, so here are my two YouTube channel recommendations to stay up to date:

• PixelArtistry_
• Stefan 3D AI Lab

In practice, there are several ComfyUI workflows to play with, like this tutorial showing how to use the recently released Trellis2 model from Microsoft. It doesn’t work perfectly like in the video every time. It depends heavily on the image you feed it. In my tests I only got deformed “blobs”.

But on the commercial side, there are 3 tools worth signing up for:

• Hunyuan Global, by Tencent. There’s the Chinese platform - which is more complete - and the global English version which is more limited. With it you can create 3D models from images with exceptional quality.
• Hitem3D - competes with Hunyuan, also excellent for generating 3D models from images. Worth testing both because depending on the images you feed them, one may turn out better than the other.
• Yvo3D - one of the pricier ones, but specifically for generating high-quality textures, it seems to be the best by far.

There are plenty of other platforms and to stay current there’s the site Top3D.ai with a ranking of the best tools. Most of them usually have free plans, trials, or similar, so it’s worth testing as many as you can. The pricing isn’t “expensive” if you’re a professional using this for client work. It’s only expensive for guys like me, who use it as a hobby.

Some others worth testing are Hyper3D Rodin, Tripo AI, Prism, and SAM 3D. The YouTube channels I recommended have videos about them, so at least go watch those.

The Ace Up the Sleeve: Nano Banana

The big trick for any platform, open source or commercial, is starting with high-quality images. Images from Google Images, or your badly taken photos, will always result in bad 3D models.

To clean up images, the easiest way is to go to Google Nano Banana. It’s by far the best AI platform for editing and generating images.

As an example, I decided to start from an image my girlfriend commissioned from an artist on Fiverr that we turned into stickers for our arcade. This image:

The first thing I asked Nano Banana was to remove the background and isolate just the character. It’s the kind of thing that used to require Photoshop. Look how it turns out now:

(ignore that it’s mirrored, I was too lazy to grab the right image.)

Anyway, that’s still not enough to generate 3D. Specifically for generating character models, the ideal is to have images in “T-Pose”, meaning the person standing upright with arms spread out like Jesus. And having photos from exactly the front, back, left, and right.

And you can ask Nano Banana to do exactly that: “generate a front photo of this character in T-Pose, for 3D modeling” and it generates this:

Half the job is done at this step. Without this you can’t generate good models. Now just ask it to generate the other angles, like the back:

Same thing, images from the left angle, the right, and done. At least 4 consistent, coherent, clean images.

Hunyuan Global

Now comes the simple part. Just pick Hunyuan or Hitem3D. In this case, I picked Hunyuan:

Just pick the new 3.0 version model and generate with 1.5 million polygons (yes, it’s quite heavy, but you can just remesh afterwards) and optionally generate textures (knowing upfront that Hunyuan isn’t great with textures).

And look at the result. Perfect! As is, it’s already usable. Like I said, it’s not great with textures, but since my goal for this test was to print on my 3D printer, I didn’t worry about textures.

Opening it in Blender, I can apply a remesh modifier to lower the polygon resolution and convert everything into quads, and it ends up like this:

And right off the bat I tried to print it and out it came:

I was particularly impressed by the details: nothing crooked, nothing out of place, hands with 5 fingers, correct proportions, perfect face details, perfect hair.

Think about this: I went from nothing, to AI, to a physical 3D print!

And now that I have this model, I can send it to Adobe Mixamo, which is an online tool to reuse pre-made poses.

I can upload my model, calibrate the armature/skeleton, and put it in whatever pose I want. Here’s an example:

In Blender I can calibrate the weight of parts of the mesh and other details to improve the pose, but 80% of the work is already done automatically. With this I can do animations and much more.

So yes, 3D modeling is already a reality with AI. There are dozens of tools for different use cases and specialties, like texture generation, skeletons, animation.

The main thing: start with a good foundation using Nano Banana like I said, that’s the main trick to make everything else work properly. And keep an eye on Tencent: Hunyuan is the most interesting model.

On the open source side there’s small Hunyuan to run locally and now Trellis2. Also keep an eye on new workflows that use both together in ComfyUI.

On my end, for doing simple things I want to 3D print on my Bambulab X1C, this is already more than excellent!

Update: this post is now obsolete. I now use akitaonrails/aitrepreneur-docker, which runs Aitrepreneur’s stacks locally in Docker with pre-loaded models and workflows. Check the README for details.

In the previous post I showed how to generate character sheets for game development. But some folks asked me “what if I want specific clothes?”.

Hold my Beer

There’s a way. It’s a separate workflow that you can later add to the previous workflow if you want (homework assignment). Just take the character generated in the first group of nodes and pass it through this other workflow that swaps clothes. (Or do the extra work of generating loras and activating them with keywords in the prompt as well, there isn’t only one way.)

Unlike Mick’s workflow - which is paid - this other one is open. I grabbed it from some Reddit post and dropped it into the workflows folder of my Docker project as “garmente-replacement-idm-vton.json”. Just load it in ComfyUI and play around. Let’s see the result. Here are two images. The result isn’t perfect, but see if you can figure out which of the two is the original and which one was generated by A.I.:

Answer at the end of the post lol

Anyway, the complete workflow looks like this:

In the current state of my setup there’s a glitch I haven’t been able to solve. I think it’s the JNodes extension that’s opening an infernal popup at the top left of the interface, which won’t close no matter what. For now I’m opening the Inspector and deleting the Node from the HTML directly lol it’s a hack but it works:

If anyone knows how to fix it, I’d really appreciate it. There’s an alert from an extension that seems to be outdated, this front-end something, but just ignore it. Of course it had to be front-end-something…

Anyway. In this first group it’s simple, just load an image as clean and clear as possible of the new garment you want to use. And then the original photo where you want this clothing to be applied. It doesn’t perform miracles. If the photo is of someone in a blazer and you want to change it to a tank top, sometimes it manages, but don’t expect the arms and torso to look good. It’s going to redraw everything as it sees fit, or it’ll end up putting the tank top on top of the blazer. Anyway, it’s A.I., the clearer the goal, the better the result. If the new garment is a blazer, ideally the original was another blazer, it’ll fit better.

Heads up that the IDM-VTON models really weigh a TON. It alone downloads no less than 30GB of models (already included in my setup) and will demand more than 20GB of VRAM, so anything below an RTX 3090 with 24GB just won’t run. You can run it online, on the Hugging Face site. Watch this tutorial that shows how:

This is the Aiconomist channel. He shows how to run it online and how to set it up on your machine in this other video which I used as reference to build my workflow. Subscribe to his channel, it’s worth it. The tutorials are very well explained and have more details that I won’t cover in this post.

That said, there’s a bug I haven’t solved yet. After running the workflow, the model stays hanging in the GPU’s VRAM and if I try to run again, it FREEZES. Then I have to shut down the Docker container and restart it to recover the memory. I tried a Clear VRAM GPU node but it didn’t work. If anyone knows how to fix it, I’d appreciate that too.

The next step is recurring across many workflows: extracting layers of information from the original photo. Getting a mask of the garment we want to replace and information about the original pose:

This is why ComfyUI workflows are powerful once you start understanding how the process is broken down. These parts are reusable and different models give different results depending on which controlnets you add.

Then we get to the main part: wiring all this information into the IDM-VTON node, which will reposition the new garment on top of the original photo:

Notice that the result isn’t very good, it looks pretty “fake”. And there are parameters you can tweak to improve it. But the use case isn’t to swap for a specific garment but rather for a similar one. I’ll explain in a bit, but first, a tangent.

The repository for the original extension of this Node is this one. But if you try to install it, it breaks. It seems to be a bit out of date with ComfyUI and needs to run on an older version. But that’s a pain. So I went ahead and made a FORK with the necessary fixes to run on the newer version (the newer versions of the Python packages diffusers, transformers, huggingface_hub, break IDM-VTON).

By the way, Python dependency management is pretty BAD. But ComfyUI’s is even worse, because different extensions (which install Python dependencies globally) can be at different stages of development. Every time ComfyUI itself changes, everyone needs to update fast. That’s why the ideal is not to install the master branch version of ComfyUI, always a stable version (marked with a tag) or two versions back, just to be safe. But then there are new extensions that require you to be on the newest version. As I said, it’s a nightmare to manage. Being a programmer, I can “hack around” when errors happen, but normal users will suffer a bit.

Rant over. In the previous step the first part of the flow ends. When it hits this Image Sender node, the process stops - I don’t know why. And you have to click the “RUN” button a second time. Then the other node, Image Receiver, will receive the new image. I think I’ll just delete those two nodes and connect the “image” output of the Run IDM-VTON Inference node directly to the next node, which are the Image Composite Masked and the VAE Encode.

Let’s go back a step:

In this next step, on the second “RUN”, it takes the image with the new garment, but low quality, and loads a new checkpoint (in this case juggernaultXL but it could be any other like SDXL itself, FLUX or dreamsharper). The model will determine the new photo, which is what gets generated at the end, on the right.

Notice that the new photo has much better garment quality, but the face is completely different. It used the information from the original photo, kept the pose, the body proportions, even the background, but it can’t preserve the face. No problem, because we can help.

Remember that in the previous step we got the mask (black and white) of the exact position of the garment? We can enlarge the mask and apply a Gaussian Blur (so it later blends nicely into the original photo) and use that to crop from the generated photo with the different face, using the IPAdapter model (which alone is another 30GB of models - I told you it’s heavy).

If I understood correctly, IP Adapter is responsible for being able to generate new images that are COHERENT with each other, with the same face, or same clothes, but in other positions, for example, a kind of conditioned generation. With it you can control the coherence of new images A LOT, generate a character in different positions, in different backgrounds and always preserving the characteristics of the original character.

IP Adapter will use the original photo, keep the face, use the masks and blend with the photo with the different face, taking only the garment, and then assemble everything in the Image Composite Masked which is the final Node of the process.

In the end, we can compare the first attempt at putting on the new garment (which came out ok but ugly, and the new garment, which came out different but prettier) and everything integrated into the original photo, preserving the same face.

As I said, the qualities are affected by the choice of model, the KSampler settings, number of steps, cfg and everything else. You have to experiment and generate several until you reach the result you like best. That’s also how you learn more. You don’t just click once and magically have the result. A good artist experiments dozens of times.

The final image might not be suitable for a Vogue cover (although I’ve already seen plenty of official covers that were VERY questionable). It depends on the need. In particular it’s good for prototyping. “I wonder how this garment looks on my model?” Instead of calling the model in, having her change clothes, or spending an hour Photoshopping, in minutes you can try several different garments and see whether the model fits the garment, without wasting time. So a photographer can just show it to the client before they spend money on studio, assistants, equipment, etc. You can visualize quickly and discuss details much faster.

This is the kind of thing where tools like this can help a lot and differentiate one artist from another, not by replacing the artist, but by creating more options. And the main thing: all these workflows are modifiable, you reprogram them and combine them with others however you want. The limit is your creativity.

I decided to document in this post one of the most complex workflows I have tested that runs on my setup: how to develop Character Sheets of random characters, in any style, whether realistic, Pixar style, cartoon style, whatever you want. The result is a sheet like the example below. In several poses, several lighting setups, several emotional expressions, with a color index, all generated from absolute scratch by “A.I.” In half an hour of processing on my RTX 4090.

Imagine you are an intern at a game studio. Your job: create 100 background characters (NPCs). In the end you have to deliver 100 of them. But just because they are secondary characters does not mean you can do them any old way, they all have to be coherent, in every pose. Imagine how many days you would take. What if you could automate it? This is the kind of scenario where A.I. is interesting. Spend more time on the main characters, do them manually, but do not waste time on the secondary ones and do not lower the final quality of everything.

I did not make this workflow, it was Mickmumpitz. Watch his entire video to learn how this workflow works, I will not repeat what he already explains:

I will also not share his workflow, because he has a paid Patreon (it is super cheap, only USD 10 per month), which gives you access to download his entire workflow pack at this link

But that alone is not enough. The requirements you need:

• a beefy machine with at least an RTX 3090 with 24GB of VRAM. The important thing here is VRAM, the models used easily exceed 20GB of usage. CPU is less important, I use a 7950X3D but a 7800 would already handle it.

• if you do not have such a machine, it takes more work but you can rent one on RUNPOD. They have several GPU configurations, but you will need to know how to configure Linux a bit or choose Windows and do Mickmumpitz’s tutorial all by hand (trust me, you will suffer if you do it that way).
• you need to have at least 500GB of available space and good internet to download. The first time my Docker runs, it automatically downloads all ComfyUI extensions and all models for everything. You can edit the models.conf file beforehand and remove models that you know you will never use. But just this workflow alone requires at least about 30GB of models, there is no way around it.
• this workflow is particularly HEAVY because it has several features (parts of which we can disable, to customize for our workflow). But for 1 character, the entire flow can take 30 MINUTES. Even on an RTX 4090 like mine.

If you have done everything right up to this point, you have ComfyUI running with all dependencies already downloaded and you bought the Workflow. When you load it you will have to fix some things manually.

The first thing is that you will see an error in several Nodes in red, all of them Ultimate SD Upscale. Just right-click on each one and choose “Fix Recreate” and it will fix automatically:

ComfyUI requires that each output slot of a Node connects to the input slot of another compatible Node, but often the workflows get so big that there are little wires connecting things scattered all over the place and it gets really hard to know what connects to what. It literally becomes spaghetti.

But there is an extension (which I have already pre-installed) called cg-use-everywhere. Every extension is a little GitHub project, cloned into the custom_nodes subdirectory. It lets you connect a Node’s output to this Anything Elsewhere node, say the “clip” output of the “Load Clip” Node. Now, every other Node that has “clip” as an input field and does not have anything explicitly connected to it will use this Anything that was created. It works super well.

Only in this Mick workflow, who knows why, all the Anywheres are broken. But it is easy to fix. Just find these Nodes, delete them, create new ones and reconnect. There are several right at the beginning, write down who connects to each one, delete them and create a new one to reconnect:

There are also some in the Upscale group 02:

Small Nodes, with only the title showing, are COLLAPSED. Just right-click and choose EXPAND to see the details. It is another way to organize the mess.

Lastly, you have to recheck the correct name of the models in all the “Load” something Nodes right at the beginning of the workflow (luckily he organized these nodes all together):

In his workflow, the Node comes looking for “FLUX/flux1-dev-fp8.safetensors” but then you go to the list (which pulls from what is actually in the “models/checkpoints” directory on your machine) and choose the one with the same name. This happens often because different people organize different subdirectories for the models. But after configuring, just save and you are done. You have to do this for the checkpoint, controlnet, ic-light, etc.

ComfyUI errors are not friendly. They are Python stacktraces, which are not friendly because they show which line of code blew up, but they do not show what was in the variables passed to the function that blew up. So there is no way to know where it went wrong. But normally on the web interface, the Node turns red, then you check it and usually it is because you forgot to download the model or pick the right one from the list.

The last thing in this workflow is to download the OpenPose image from his Patreon page, which I linked at the beginning of the post, and load it:

OpenPose is a standard file format that describes poses, an open format that can be edited in several tools. And there are several pre-trained models to extract poses in this format from images, so we do not even need to edit it by hand. We can download several pre-made open poses from this site:

I am going to repeat this several times, but watch Mick’s video tutorial where he explains how to generate characters from scratch or use your own photos with your own poses to continue the flow from there. His workflow, after running, looks like this (with zoom way down below):

In the first part, there is the manual configuration and checking of the models. You only have to fix it once and save. After that it will always work the same way. In this first menu you can also turn on and off some of the Node groups, for example, to skip the first stage of generating a character from scratch and be able to use your own photos:

If you choose to generate a character from scratch, there is a green prompt box to type how you want your character to be, in what style. Text Prompts are decoded by CLIP (Contrastive Language Image Pre-Training) models, which are specific to certain UNET categories (like SDXL for Stable Diffusion or the now best and most popular FLUX). Part of the learning is starting to figure out which sub-models (clips, clip_vision, vaes, loras, etc.) are compatible with which unets (unet, checkpoints, diffusion_models). You cannot mix just anything. Roughly speaking, it is a bit like, if you choose to do a text in German, all plugins have to be in German.

I will not show the complete workflow because it does not even fit, but you can see the prompts in green, in the middle where we choose which pose we want (it is on his Patreon), and on top the image generated by the flux1-dev-fp8 model, one of the best and most popular for having excellent quality and not being so heavy (fp8 instead of fp32, for example).

This is one of the advantages of using tools outside the Dalle or Midjourney of the world: we can control all aspects of image generation not only with text prompts, but also with image prompts, like OpenPose.

Another detail. I explained in my previous post about KSampler or Samplers in general, that they are the engine that does all the work of joining all the models, images, prompts, controlnet, vaes and produces the final image. KSampler parameters like steps, cfg, denoise, scheduler and more DEPEND on the chosen model. There are well-documented models, there are poorly documented models.

For example, say you read on a Reddit thread about this other more optimized model called “TurboVisionXL”, which is available on the Civit.ai site:

It really is a good model, but now what? What do I configure in the KSampler for it? Luckily if you scroll the page (always read the original documentation!!) you will find these recommendations:

See? He explains what to configure and what the known limitations are. Further down there will be comments from other people who can give more tips or share errors that you have already seen. It is a huge community, you need to learn how to participate in it.

But back from the tangent, in Mick’s second group of nodes, he takes the first image in “Pose T” and runs it through a “Multi View” process using Diffuser MV which does this automatically. It is the same pose, but as if the camera went 360 degrees around it, capturing different angles and, most importantly, generating new images coherent with the original (see how the model is even used to invent the character’s back).

In this step he also extracts only the face from the first image to do other treatments in parallel. Face recognition and extraction is super common. Every security system uses it, Google Photos uses it to organize dynamic albums. And it is worth learning to use in your own products. It is a reasonably light and fast process these days and there are several pre-made algorithms and models to use. In this case I do not think it even uses A.I. as such, it is just an algorithm.

Another detail: image generation models normally work in LOW RESOLUTION. Usually below 1024x1024. Do not expect to generate 4K images right off the bat. But that is no problem, because the solution is to first generate a low resolution image and then run it through another A.I. model specialized in UPSCALING like the RealESRGAN that I already talked about in my other post about upscaling old anime

Different from what Photoshop or any image editor does, if you try to enlarge an image, it just takes a pixel and tries to duplicate, quadruplicate the surrounding pixels. But that usually only makes the image bigger but blurred, because it has no way of knowing the details, and it will not even try to “invent” details.

To “invent” details, you need a pre-trained model that learned what to draw. It is also how NVIDIA’s DLSS 3 works for games, where the game renders in low resolution, like 720p, and DLSS, in real time, redraws the new screen in 4K. It is not perfect, but playing at 60fps, you will rarely notice the difference. You will only notice that it got more CRISP.

That is also why, if you use your own photo, or photos of people you know, you will see that the new photo after upscaling looks slightly different, it is no longer the same person, because the model REDRAWS details that did not exist. It is a process of interpretation and not of discovering “hidden” details (that does not exist, only in CSI fiction).

Mick’s workflow goes one step further and has a second upscaling pass, focusing on fixing only the face with a Face Detailer Node, made specifically to fix and improve faces. And here we enter plastic surgery and harmonization territory for real lol the final face will be different from the original. But for NPCs and secondary characters it does not matter, nobody will know the difference anyway, as long as it is not deformed, cross-eyed or something else that jumps out at you.

There are Nodes specialized in expressions, like the Expression Editor (PHM) which starts from a single face photo and manages to generate new photos in the most varied emotions - which we can configure exactly, numerically, however we want. I will not show it, but note that the generated images are in low quality, but the next step in the workflow is precisely to upscale them too.

This step is interesting. First the top part, with the various scenarios. They were all generated via the green prompts below them, prompts that Mick left but that we can edit however we want OR swap the generation node and put a Load Image node to load our own background images, from real photos.

The objective of this step is to generate the faces under different light configurations. Many people think that “light” in games is just “turn on the red light” and everyone will automatically turn red. In new games, with RAY TRACING yes, that is why it is SUPER HEAVY. But in games made for old machines, indie games, you cannot demand this from everyone.

So “tricks” are used to fake things like lights and shadows. It is the BAKING process where we literally “stamp the light onto the texture” of objects and characters. With global illumination on or off, they will always be “painted” with the colored light of the scene. To switch scenes, new textures are loaded with other “baked” lights. For programmers, think of baking as Caching. Why recalculate everything dynamically every time, if the scene’s light never changes? Save that step and leave it “cached”.

Going back, this step uses the IC-Light technology which is another open source project worth learning, even if you are not interested in A.I. It can manipulate the light of any image or photo. You, as a photographer, can change the light if you have changed your mind, in post-production. Adobe or BlackMagic tools, I think they already include similar technology, but it is good to know that there is open source and you can integrate it into your product.

After generating the faces with different lights, obviously there is another upscaling step - which I am skipping showing - and it goes to the final group of nodes that joins everything into a “sheet”, a “character sheet”. The game developer can use it directly in a 2D game (think of a visual novel game), or it can serve as a basis for UV Mapping textures of a simple 3D model with Armature to animate and you already have some emotional faces. Good for placing in the background scenery, pedestrians walking on the street, shopping, sitting in the square or something like that.

In addition, Mick’s tutorial describes how it is possible to create LORAS (Low-Rank Adaptation). If you want to be able to generate characters with a specific style or even specific faces (without “artistic” intervention from the A.I.), you choose a model like FLUX or SDXL and “on top” of it you add a second compatible sub-model with information about that person or character.

For this you take several clean and well-lit photos of the person, from many different angles. You do not even need much, just a dozen is enough. For each image you create prompts that describe it well, and that is it. There are tools to generate these Loras locally or sites that do this (watch the video).

Speaking of prompts, in several workflows there are steps to describe detailed prompts of dynamically generated images. Instead of requiring the user (you) to do this manually, you can use a decoding model, which decodes images into text prompts. The most used model for this is Microsoft’s FLORENCE 2.

That text box at the bottom is read-only, you do not need to type it manually. The Florence 2 Run Node will read the image and generate this text. See how it described the first image of the woman in the “T” pose. Doing tests, for your case, you determine if just this is enough; if it is not, just put a second prompt Node and concatenate the two before sending them to other nodes.

Anyway, there is A LOT to explore just in this workflow. Mick did an excellent job both on the workflow and on the video to explain it, and it is worth paying the 10 dollars to contribute. There are dozens of workflows like this available on the Web, just search. And my setup is getting more and more complex. I plan to keep adjusting it for each new interesting workflow I run into. There are already several pre-tested ones in the “workflows” folder of my project for you to play with.

Update: this post is now obsolete. I replaced the manual setup with a cleaner containerized project: akitaonrails/aitrepreneur-docker. It runs Aitrepreneur’s stacks locally in Docker, with pinned versions, persistent models, and pre-loaded workflows. Check the README for details.

In the previous post I explained my little project to spin up ComfyUI pre-configured and pre-loaded automatically using Docker Compose. Assuming you already have everything up and running, now it’s time to actually understand this beast called ComfyUI a little.

I’m no expert either, but I thought it would be cool to explain a few concepts that many people overlook.

Pre-Trained Models

Every GPT out there, or Stable Diffusion or others, loads a vector database into the GPU’s memory, into VRAM, in order to process. They are HUGE models, called 7B, 70B, 100B, etc. The “B” stands for “Billions of Parameters”. Parameters have no usable definition outside of math, they are simply numbers. Those numbers represent “something” inside millions of dimensions inside that giant file.

In a VERY ROUGH way, I think of them as compression artifacts. You take PETABYTES of books, images, texts, posts, etc, break it all into tokens and find “relationships” between one token and many other tokens, and you keep “positioning” them in space, but not in 3 dimensions, in millions of dimensions. AI models are “Vector Space Models” or VSMs, more or less like the index of an Elasticsearch if you’ve ever played with relevance-based text search, or any “Full Text Search” plugin in Postgres or MS-SQL. They’re all VSMs.

Here’s how I think about it: it’s as if the training process - the SUPER EXPENSIVE part, which takes MONTHS - takes all the information that’s fed in and compiles a new “dictionary” for a new “language”. Now, when you “talk” to a GPT, first it needs to transform your text from your language into this internal “language” of its own, which is what we call an “Embedding”.

Any data can be transformed. Audio can be converted into text and from text into embeddings. Videos can be broken down into series of images with temporal information. Images can be deconstructed into elements like luminance, chrominance, etc. In the end, everything can be deconstructed into data structures, there’s no magic.

Image generation models are trained differently. There are several focuses, focus on facial sharpness, focus on pose accuracy, focus on different styles like cartoon or painting, focus on categories like animals or objects. That’s why there are dozens of models out there and lots of “loras”.

To generate an image from a prompt, first you need to decode your prompt into an embedding, and that’s what things like a TEXT ENCODER in ComfyUI are for.

In the specific case of AI for images, the way I understand it is that they use something called a “DIFFUSION” model both to deconstruct the original image and to reconstruct a new image. It’s not intuitive, but the reconstruction doesn’t start on a blank canvas, the way it would for a human illustrator. It starts with an image filled with random “noise” like this:

The model starts from this noise and through a process called DIFFUSION it begins to redraw the image backwards, until it emerges from the chaos and arrives at an image.

What I “THINK”, what I “GUESS” is happening - without having read any paper, pure laziness - is that we try to influence this initial noise. For example, by extracting features from the image like a depth map, which looks like this:

The original photo looked like this:

.jpg)

Extracting a Depth Map is an ALGORITHM, it’s not “AI”. Any Photoshop or Premiere out there can do this with one hand tied behind its back. But the point is to explain that there’s much more information in an image than what you - untrained as you are - can see is there.

Another well-known type of ALGORITHM is Canny Edge Detection, which produces an edge map, making it easier to know where something starts and ends in the image:

So it goes more or less like this:

• just as a text is “converted” into an Embedding, vectors that capture the “essence” of the meaning of the text, dependent on the VSM of the pre-trained model, an image is “converted” internally into a LATENT, which because it’s an image is a “space” (that’s why people talk about Latent Space). We can’t “visualize” Latents, because it’s an internal representation that depends on the U-NET, the pre-trained model, the neural network that learns to “denoise” the Latent.
• An image U-NET is roughly similar to a text model like GPT or LLaMA, the pre-trained, read-only model that will be used to produce the final result.
• A GPT response is a continuation of the context of the previous conversation. It’s not “answering” you directly, it’s just “continuing” the text of the conversation and, by chance, due to training, the result “looks like” a response. In the case of images, we start with a randomly noisy image and through a process of removing and adding noise, it gradually “rearranges” the Latent into an image we can recognize.
• There is a “Scheduler” that decides how much noise is added or removed at each time step. And there is the concept of a DIFFUSION MODEL, which is more or less the processing “pipeline”.
• We can control the process by adding a text prompt, which is decoded into an “embedding” using a Text Decoder that usually has names like “CLIP” or properly a Transformer (such as LLava). These embeddings go into the U-NET via cross-attention layers, steering the “denoising” toward the meaning of the prompt.
• On top of that we have CONTROLNET, which is a second parallel network that adapts the process to extra conditions. It’s a copy of the U-NET architecture (with frozen weights) that receives extra inputs, like Canny Edges, or Depth Maps, pose skeletons (via OpenPose), etc, and learns small adjustments to INJECT into the main maps of the U-NET.
• During the denoising process, ControlNet activations are merged with those of the U-NET, forcing the process to respect the conditions we passed in.

So the U-NET is the “generator”, the Diffusion Scheduler orchestrates the process, text encoder and controlnet both influence the U-NET, and a VAE is used at the end to convert a latent space into an actual image that we humans can actually see.

VERY, VERY ROUGHLY. That’s more or less how I understand the process. And the way I see it, ComfyUI exposes several of these steps in the form of “PROGRAMMABLE NODES” that we can parameterize and tweak to influence how we want the final image to come out.

Beyond numerical parameters, these Nodes also let us choose which pre-trained model files we want to use for each stage. It’s all binary VSMs, but for different uses, which is why many of them have the same extension as I explained, but they live in different sub-directories. So each Node can find only what matters to it.

Taking my photo as an example, this isn’t exactly it but just to illustrate, it helps me think that those maps I showed influence the “noisy” image in a non-random direction, like this one I grabbed from an intermediate stage of the workflow I’m running:

If you strain your eyes, you can see the influence of the previous images, and the Diffusion model keeps doing the “denoising”, being “interrupted” by the ControlNet. It’s as if you were drawing and had a person next to you who every now and then erases what you did, or draws on top, and you keep adapting. So in the end it becomes a “collaborative” job between the two. If it were only the U-NET, it would generate a brand new image quite different from the original, but the ControlNet is the “client” that keeps nagging in your ear “that’s not how I want it, change it”, until it arrives at an acceptable result.

On top of that, a U-NET isn’t great at generating very high resolution images. The ideal is to generate small images, 512 pixels, 1024 pixels, but nothing bigger. But that would be useless for professional use. That’s what UPSCALE MODELS are for. If you read my article from yesterday about Video2K I explain that I pick a model there that’s good for temporal coherence in video. But there were other models that are better for “STATIC IMAGES”, or photos. And we can use them in ComfyUI.

After the new small image is generated by the U-NET, we can then use it as input to a second “UPSCALE” pass using a model like RealERSGAN, and go from a 1024 pixel image to 2160 pixels, essentially 4K. It’s different from enlarging the image in Photoshop, where everything ends up “soft” and blurred. Since it’s also another AI model, it’s trained to “redraw” at higher resolution. And we can add this to the ComfyUI workflow as well.

More than that, ComfyUI also supports Nodes with models like Tencent’s HunyuanVideo, which I introduced in another article from yesterday, about FramePack. In fact, we can do the same thing FramePack does but from a ComfyUI workflow: generate a video from an image, using the same model. FramePack is infinitely easier to use, but ComfyUI is the Swiss Army knife of AI: it does EVERYTHING, which is also why it’s infinitely more complicated and you need to put in serious effort to learn how to use it.

With what we’ve covered so far, take a look at the workflow I used (it’s zoomed out, you can dynamically zoom in):

You can see these intermediate steps using PREVIEW Nodes, so this is more or less how we can use ComfyUI to do a lot of advanced stuff. As I said, I myself haven’t even started scratching the surface.

Let’s now summarize some of the concepts used in ComfyUI:

1. Checkpoints (Base Models)

What they are:
These are the main Stable Diffusion models (e.g.: SD 1.5, SDXL, Flux). They contain the entire model architecture:

• Text Encoder (CLIP)

• Diffusion Model (U-Net)

• VAE (Image Decoder)

What they’re for:
Generating images from scratch. Each checkpoint has a different “style” (realistic, anime, etc).

Examples:
flux.safetensors, realisticVisionV60.safetensors, anythingV5.safetensors

2. VAE (Variational Autoencoders)

What they are:
Component that decodes images from the latent space (the format the model understands) into visible pixels.

What they’re for:

• Improving details and colors

• Some checkpoints require specific VAEs

Examples:
vae-ft-mse-840000.safetensors, animevae.pt

3. Text Encoders

What they are:
Models that convert your text prompt into numerical representations the AI understands.

Common types:

• CLIP: Standard for most models

• OpenCLIP: Alternative version used in SDXL

• T5: Larger models for complex prompts

Examples:
clip_vision.safetensors, openclip.safetensors

4. CLIP Vision

What they are:
Models specialized in visual analysis (as opposed to normal CLIP, which is text→image).

What they’re for:

• Creating descriptions of existing images

• Tools like “Image Prompt” (which use the image as input)

Example:
clip_vision_g.safetensors

5. Diffusion Models

What they are:
A specific part of the architecture (U-Net) responsible for the diffusion process (adding/removing “noise”).

What they’re for:

• Training LoRAs

• Advanced reuse workflows

Example:
unet.safetensors

6. LoRAs (Lightweight Adapters)

What they are:
Small files that modify the behavior of the main checkpoint without replacing it.

What they’re for:

• Adding specific styles (e.g.: anime)

• Creating consistent characters

• Adjusting body proportions

Example:
epiNoiseOffset.safetensors, add_detail.safetensors

7. ControlNet

What they are:
Models that impose control over the composition of the image (poses, depth, edges).

Common types:

• OpenPose (human poses)

• Canny (edges)

• Depth (depth)

Example:
control_v11p_sd15_openpose.safetensors

8. Upscale Models

What they are:
Models for increasing the resolution and detail of generated images.

Types:

• ESRGAN: For realistic details

• Anime6B: Specialized in anime art

Example:
4x_NMKD-Superscale-SP_178000_G.pth

9. Embeddings (Textual Inversions)

What they are:
Small files that add new concepts to the model’s vocabulary.

What they’re for:

• Specific artistic styles

• Custom objects

• Artifact corrections

Example:
bad_prompt.pt, easynegative.safetensors

10. Style Models

What they are:
Models specialized in applying visual styles or art “filters” on top of the generated image, without actually altering the basic structure of the content.

What they’re for:

• Thematic colorization (e.g.: watercolor palette, sepia, chroma-key)

• Simulating artistic techniques (oil, gouache, manga linework)

• Unifying aesthetics across series of images (same color and light “mood”)

Example files:

• flux1-redux-dev.safetensors (“Flux Redux” style)

• watercolor_v2.safetensors (watercolor brushwork)

• film_grain_vintage.safetensors

11. SAMs (Segment Anything Models)

What they are:
General segmentation networks capable of isolating objects or regions of interest in an image, producing binary or weighted masks.

What they’re for:

• Creating masks of objects/people for selective application of filters or compositions

• Guiding ControlNets (e.g.: applying diffusion only inside or outside a region)

• Pre-processing references (cropping background, separating layers)

Example file:

• sam_vit_b_01ec64.pth (SAM ViT-B, base segmentation model)

Visual Summary

models/
├── checkpoints/        ➔ Main models (Stable Diffusion)
│     └── *.safetensors, *.ckpt
├── diffusion_models/   ➔ Alternative diffusion models
│     └── *.safetensors, *.gguf
├── loras/              ➔ Style adapters (LoRAs)
│     └── *.safetensors
├── controlnet/         ➔ Composition controls (ControlNets)
│     └── *.pth, *.safetensors
├── vae/                ➔ VAE decoders / encoders
│     └── *.pt, *.safetensors
├── clip/               ➔ CLIP text encoders
│     └── *.safetensors, *.pt
├── clip_vision/        ➔ CLIP vision encoders
│     └── *.safetensors
├── text_encoders/      ➔ Advanced text encoders
│     └── *.safetensors, *.bin
├── upscale_models/     ➔ Super-resolution models
│     └── *.pth
├── sams/               ➔ Segment Anything (SAM) models
│     └── *.pth, *.onnx
├── style_models/       ➔ Visual style models
│     └── *.safetensors
└── embeddings/         ➔ Embeddings / custom concepts
      └── *.pt, *.safetensors

You have to download the right things into the right places. And to make matters worse, as I already said, the files usually have the same extension. So it’s very easy to get confused and drop a text encoder into the clip directory or vice versa. Once again, that’s why I encourage you to update the “models.conf” file and put the right URLs into the right lists, that way it stays much more organized and my Docker setup will pull things into the right places at startup.

Even so, you may still have a few other doubts that I had at the beginning too. For example “Why is the U-NET called a Checkpoint?”

Checkpoints ≈ A Complete Operating System

• Similarity: Just as a commit in Git captures a specific state of the code, a checkpoint is a complete “snapshot” of a model trained at a specific stage.

• Difference: Checkpoints are self-contained (they aren’t “diffs” like commits), each one is a complete model that can work independently.

• Better Metaphor: Think of checkpoints as different versions of Windows (XP, 7, 10) - each one is a complete operating system, not incremental.

So checkpoints kind of resemble, conceptually, a “git commit” - if you’re a developer. Or BTRFS or ZFS snapshots - if you’re an infra/devops person.

LoRAs ≈ Plugins/Extensions

• Similarity: Just as a PR adds functionality to a base codebase, a LoRA modifies the behavior of the base model.

• Difference: LoRAs aren’t “merged” into the original checkpoint, they are applied dynamically during inference.

• Better Metaphor: They’re like Chrome extensions (e.g.: AdBlock) - they add specific behaviors without modifying the browser itself.

Think of it this way: a model has Checkpoints, like commits, so you can “continue” training on top if you need to. At the same time, we don’t have to mess with the main model, we can create separate layers, the so-called LoRas. For example, let’s say a model is having trouble generating your face, no matter what prompt you use or what Nodes you configure.

Instead of that we can use the “dictionary” of the pre-trained model, like Flux or SDXL, and do training with our personal photos and produce a LoRa specific to me. Then we can add a LoRa Node into the workflow and that will result in much better images with your particular face.

There are entire online communities dedicated to this. One example. On the Civit AI site we’ll find lots of models based on existing models, like this Mistoon which is based on SDXL.

It’s a checkpoint. We can sign up on the site and download the file mistoonAnime_v10Noobai.safetensor and put it in the models/checkpoints directory.

Now, in any workflow that has the “Load Checkpoint” Node we have the option to choose this model. But let’s say this model isn’t able to generate a character with a school uniform like in many anime. Browsing the site, we find this LoRa:

The Mistoon model is large, around 6.6GB, but this LoRa - called Mistoon_Anime\ school\ uniform.safetensor which we should move to models/loras - is less than 290 MB, much smaller because it’s a specific training only on images of school uniforms.

Now we can wire the “Load Checkpoint” Node to this other “Load LoRa” Node and choose this LoRa that we know is compatible:

LoRas have to be picked to be used with the models they were based on. As I said before, think of the model as a “language”, if we make a LoRa in German, there’s no point mixing it with a Chinese model, roughly speaking. But this is one way to do fine-tuning so the model you like delivers results no other one can.

One problem I have with the Civit AI site is that you can only download things while logged in. That’s why I didn’t include any of their models or loras in my automatic install script. Unfortunately, in their case you have to manually download the files and move them to the right directories, write it down somewhere, so on another machine you can download and move everything all over again. It’s a pain that I haven’t sat down to fix yet. If anyone wants to contribute a solution for this (without exposing anyone’s login cookies, send a Pull Request).

A Real Example

Your day-to-day with Comfy, being a newbie like me, will go more or less like this: you start by going to Google and searching for “best workflow image to anime ComfyUI”. The first links are usually some Reddit thread:

Sub-Reddits like r/comfyui or r/StableDiffusion usually have news and workflows like this one. Then we go straight to the first comments:

Either that, or we open the workflow directly in ComfyUI and inspect the Nodes, one by one:

We may already have a problem: the Node came pre-configured to load a LoRa called “SDXL_aidma-niji_jini.safetensors” but the link in the Reddit comment downloads an “SDXL_Niji_V6_DLC_LoRa_V4.safetensors”. Searching specifically for the previous one on Google, I can’t find it. But it seems “safe” to use this other one, at least the names are very similar, both are derived from SDXL so theoretically they’re at least compatible. This is the page of that model on Civit AI, so you have to download manually and send it to the models/loras directory, same thing for the AniToon Checkpoint in the Node above. Download manually and move it to models/checkpoints.

Note that on this page it explains that this LoRa was made to work with the SDXL_Niji_V6 model and not with AniToon, which is what the Workflow I downloaded suggests in its Load Checkpoint Node. It’s worth testing with both. Either way, since we’re already on the site, let’s also download the suggested model:

File sdxlNijiSeven_sdxlNijiSeven.safetensors, 6.5GB, move it to models/checkpoints, are you all getting the process? Download, move it to the right place, reload the workflow in the browser and now it shows up, we select it:

By the way, I think it’s intuitive but for those who haven’t figured it out after opening the ComfyUI interface:

• left mouse button selects Nodes
• middle mouse button drags the canvas
• scroll wheel zooms the interface

Now then, let’s pick these two to test:

Taking advantage of this same image, notice in the left corner a Florence2 Node, which is a Microsoft LLM (and my Docker script already installs it for you). I can’t show all the related Nodes but these are the main ones explaining what it’s for:

The ideal in image generation is to write the longest and most detailed prompt about the original image that you can, but most people just write some garbage like “person smiling standing up” and think that’s enough. The Florence2 model is built to read an image and describe it in text.

Now there’s another detail: many checkpoints or Loras are trained with some activation keyword. So they only kick in if you type those words into your prompt. So this workflow doesn’t need you to type a prompt manually, Florence2 will do that, and every time its text starts with “The image … bla bla bla”, so we have a “Text Find and Replace” Node that swaps that “The Image” for the keywords of Anitoon or Niji: “aidma-niji, niji”.

Are these the right words? I HAVE NO IDEA lol

The workflow already brought these words, but on the Civit AI pages - where it should be documented, it’s not!! So we’re really GUESSING that it should be “niji” since that’s the name of the model and the lora. Pay attention to the keyword, it’s important, otherwise you add the lora and nothing changes and that’s because you forgot it in the prompt!!

There’s more, you have to check that the Nodes for things like VAE, ControlNet, Canny and everything else are populated with files that we have. Usually you use the same half-dozen and I’ve already pre-installed the main ones in Docker. In the image, see that the ControlNet is contronet-union-sdxl-1.0-promax.safetensors, compatible with SDXL, and we’re using Niji which is built on top of SDXL. We just keep guessing like that.

Also notice that it uses Zoe Depth Map to extract the Depth Map and Canny to extract the Edge Map as in the other workflow. You can configure it, in this workflow notice that the Edge Map picked up some weird artifacts in the background, you can adjust it to remove that. That’s why in the previous example the final image also had a window appearing, the model thought it was a window! But for now let’s leave it as is.

Now we can run it:

There’s a huge “RUN” button in blue down at the bottom. If everything is configured correctly, you’ll see that a job has been added to the queue. The interface is smart enough to let you work on other workflows and just keep queueing jobs while the GPU is processing. You don’t have to sit waiting, just leave several queued up and go to sleep.

When it runs, in the interface the Nodes that are processing at any given moment will get a green border, which is good so you know what stage of the process it’s in. And if any Node has an error - because a file is missing, for example - it’ll get a red border, so you have to fix it and run again.

And in the end, here’s one of the results:

Notice how the controlnet maps guaranteed the correct pose. The model interpreted those background artifacts from the edge map as lighting from some window, but in the original photo we can see it’s like the concrete brick of the wall lol but that’s how it is, without knowing, it guesses. And if you run it several times, it’ll always give you a quite different result. Look at another one:

Way off lol But we can swap the SDXL-Niji model for Anitoon. Let’s see one of the results:

Switching to the waiNSFIllustrious model:

This one came out even more off. Something in the maps is making the model assume that I have curly hair, no idea why. But that’s it: you have to tweak the parameters. The result of the workflow I showed at the beginning of the article gave a better result, using the same models.

Of all the Nodes, one of the most important is the “engine” of the process, the KSampler:

This is the guy who runs the show. We give it the model, the prompt, control-net outputs, and it’s the one that iteratively (steps) does the “denoise” of the Latent. Notice that there’s always a random factor (“seed”), a number of steps (25 to 50 is the average), “cfg” (classifier-free guidance) or “guidance_scale”, where higher values pull the image more toward your positive prompt. Denoise is how much noise the scheduler applies (1.0 is full noise, below 1.0 may produce more “artistic” results, you have to test).

Sampler Name is annoying because you’d need to understand the math behind it, but they are the step algorithms, for example euler, ddim, plms and each one has a trade-off between speed and fidelity. In practice: test them one by one and see the results. Scheduler has normal, klms, dpmsolver. Same thing: test them one by one.

As you can see, there are PLENTY of parameters you can play with. But the main flow is more or less this. From there you have to keep learning Node by Node what each one can do and download lots of pre-made workflows to see how people have been solving various problems, which models and loras are the most popular for each kind of application. Anyway, Reddit is your friend. I hope this gave you some sense of direction, I’m also still learning.

More than a year ago, when I made my last videos about A.I. and on the podcasts, I always said I ran everything on my own machine. But I never detailed how. So today I fixed that with several blog posts and little projects on GitHub with Dockerfiles so you can run it on your own machine, without configuring anything difficult.

I finally made a Dockerfile for ComfyUI, which is the best graphical interface for editing image generation workflows. If you’ve ever worked with Davinci Resolve Fusion or Blender Geometry Nodes and the like, workflows are basically node configurations to process your images, audio, video and pipe them through processing steps, not just A.I. ones. It’s like programming visually. That’s why it’s a much more complicated tool than the FramePack I showed in the previous post. You don’t just click and go.

Recently everyone went hysterical with the ChatGPT updates and how it “OH MY GOD” can generate Ghibli anime style images, and they kept spamming the world with idiotic anime-style memes. Online hysteria is really annoying. But as I said, as impressive as it is, that’s all it is: a meme generator. You can’t use it professionally.

I’m tired of trying to upload my own photos, or even generic photos, fictional characters, sometimes drawings of mine, and randomly ChatGPT just refuses to generate the image. Claiming privacy, security, copyright and whatever else it can invent as an excuse. You can’t rely on this for work. Anyone who’s a professional should have full control over the process, and that’s what I’m going to teach today.

To start, clone my GitHub repository and follow the README instructions or do this:

git clone https://github.com/akitaonrails/ComfyUI-Docker-CUDA-preloaded.git
cd ComfyUI-Docker-CUDA-preloaded
docker compose build

Since it’s more complex, this time I had to put together a Docker Compose. So to bring it up, after the build finishes, just do this:

docker compose up

Now it’s going to take a while, because by default it will download almost 500GB of pre-trained models so it can do almost anything image-related. Yes, to play with A.I. locally you need infrastructure. Otherwise, rent a beefy machine with lots of storage on a RunPod or similar and run it remotely. You can download fewer models, I’ll explain that next.

On Docker Compose. I recommend avoiding the “-d” option to bring it up as a daemon, because it’s good to reserve a terminal to keep watching the outputs (you can run docker compose logs --follow if you want, but it’s easier directly).

ComfyUI is a more complex program than average, much more so than something like FramePack. It supports installing “extensions” (plugins, addons, call them what you want). And honestly on its own it doesn’t do much, you need to install several extensions for it to start being really useful. Every now and then one of these extensions doesn’t handle errors properly and blows up in the console, something like this:

It’s always useful to keep an eye on the console to find out if it tried to load a file that doesn’t exist. Then you can grab the file name and search Google, download it and put it in the right place. Or if it’s a really buggy extension that’s better off removed from the “custom_nodes” directory, and so on. Without watching the console, it doesn’t show up in the web interface and you’ll be left in the dark.

A lot of the models and extensions I based on the Ultimate ComfyUI version 3 configuration recommended by Aitrepreneur, again. Watch this video of his to get all the details and everything that’s possible to do. He has the scripts for Linux, Windows, and the workflows in his paid Patreon version and, of course, I won’t share the workflows because they’re not mine. In theory, my config should be able to do what he teaches.

Aitrepreneur’s V3 workflow is really badass. But I can’t share it because it’s closed material on his Patreon, but here’s a screenshot of my ComfyUI with all of his stuff loaded:

But the config itself, I redid it entirely to work in this Docker. You can load any other open workflow you find on a Reddit or wherever.

Here’s what I did:

Models

In the project there has to be a “models” directory, if there isn’t one, create it, because docker compose will map it inside the container. When it starts, it’ll run the file init_models.sh. This script will load a configuration file you can edit later: models.conf. It’s huge and a piece of it looks like this:

...
[CONTROLNET]
https://huggingface.co/XLabs-AI/flux-controlnet-collections/resolve/main/flux-canny-controlnet-v3.safetensors
https://huggingface.co/XLabs-AI/flux-controlnet-collections/resolve/main/flux-depth-controlnet-v3.safetensors
https://huggingface.co/brad-twinkl/controlnet-union-sdxl-1.0-promax/resolve/main/diffusion_pytorch_model.safetensors|controlnet-union-sdxl-1.0-promax.safetensors
https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11p_sd15_canny.pth
https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11f1p_sd15_depth.pth
https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11p_sd15_openpose.pth
https://huggingface.co/Aitrepreneur/FLX/resolve/main/Shakker-LabsFLUX1-dev-ControlNet-Union-Pro.safetensors

[STYLE_MODELS]
https://huggingface.co/Aitrepreneur/FLX/resolve/main/flux1-redux-dev.safetensors

[SAMS]
https://huggingface.co/Aitrepreneur/FLX/resolve/main/sam_vit_b_01ec64.pth

Each section maps to a sub-directory, so “[SAMS]” will map to “models/sams” and download the binaries to the right places. There are some special ones in the “[GIT_REPOS]” section, for those that can be cloned via git, or “[CUSTOM]” for files you need to download later inside those cloned projects, for example. I don’t yet know if it can be simplified, but I picked this up from Aitrepreneur’s config.

The ideal is to always keep this list organized with models you find later. If you want to contribute, that would also be cool. The reason is that everything is scattered across all kinds of sites out there, you download manually, then move to the directory inside “models”. Tomorrow, if you reinstall your machine, how are you supposed to remember all the sites? There are dozens of available models.

In the next post I’ll try to summarize what the hell “checkpoints”, “diffusion models”, “loras”, “controlnet” and all the rest are. But for now, the file extension doesn’t say much. “.pt” for example only means it’s a file PyTorch can open, but it doesn’t tell you whether it’s a clip or a text encoder, for example. Same thing with “.safetensors”, it can be anything, and ComfyUI needs each file to be in the correct sub-directory in order to use it, you can’t just go around moving files randomly.

The “init_models.sh” script runs every time Docker is restarted and is smart enough to only download URLs that aren’t already downloaded in the “models” directory. That’s why it always runs when the container comes up, in the entrypoint. That’s one of the advantages of my config: it has automatic maintenance.

Extensions and Dependencies

I made a second script called init_extensions.sh, which also runs at the container’s entrypoint and which you shouldn’t touch, but it loads another configuration file called extensions.conf and that one, yes, you can mess with and add new extensions later. A piece of it looks like this:

[EXTENSIONS]
https://github.com/ltdrdata/ComfyUI-Manager.git
https://codeberg.org/Gourieff/comfyui-reactor-node.git
https://github.com/ssitu/ComfyUI_UltimateSDUpscale.git
https://github.com/kijai/ComfyUI-LivePortraitKJ.git
https://github.com/PowerHouseMan/ComfyUI-AdvancedLivePortrait.git
...

Here I declare all the extensions that add new functionality on top of ComfyUI, in particular new NODES, which is why they live in the “custom_nodes” directory, which is a volume external to Docker and therefore persistent, meaning every new extension survives container restarts.

But there’s still a BIG catch: Python Dependencies. A nightmare.

The Docker image build does a bunch of pip install and since I’m using Virtual ENV, everything gets installed in /venv inside the container, as it should.

However, the extensions that get installed outside the build - every time the container restarts - tend to be Python code and also have more dependencies. Extensions are GitHub projects and may be changing right this minute. If they change and you do a git pull to update, you need to run pip install -r requirements.txt. But if you have dozens of extensions, running pip install for each one on every Docker restart takes a long time to do almost nothing. As I said, a nightmare.

My solution - to brag a bit - fixes a good chunk of this maintenance. When the init_extensions.sh script runs, if an extension from the list is already installed, it’ll skip it. Then it’ll write to the ./custom_nodes/.last_commit/*.commit directory the latest commit of each extension. When Docker restarts, it’ll do a git fetch and check whether the latest commit hash has changed.

Only if it has changed will it run pip install for that extension that got an update, and skip all the others that haven’t changed since the last restart, saving A LOT of time and avoiding having to manually update things. Since I’m using VENV, everything will stay persisted in the “/venv” volume. Yes, it’s a VOLUME, because it changes after the build with each extension update.

This way, everything in the image gets copied into that volume, and everything the extensions install on the side also goes into that volume. Every time you restart, it all comes back the way it was. That was the solution I found and so far it seems to work just fine.

But it’s only a matter of time until some poorly made extension installs dependencies that break other dependencies, or an actual case I went through: there was an extension that crashes with no error message if you’re using Python 3.11, the Zoe Depth Map from the comfyui_controlnet_aux extension, one of the packages that seems to be popular.

I hadn’t realized I’d started using a Docker base image of ubuntu22 instead of ubuntu24, I was afraid of using something too new and breaking things, but I ended up using something too old and it broke just the same. Managing versioning is a pain. Anyway, I changed the Dockerfile to ubuntu24.

But in the external VENV volume now there were a bunch of dependencies pinned to Python 3.11. No problem, just delete the volume:

docker compose down # make sure it's stopped
docker volume rm comfyui-venv
rm ./custom_nodes/.last_commits/*.commit

And done, the VENV will start over empty, fill up with the correct Python 3.12 dependencies from the image, and when the container starts my init_extensions.sh script won’t find “.last_commits” and will force a pip install -r requirements.txt of all the extensions, now with Python 3.12. Super clean and reliable.

If even so you get an error, then it might be a bug in a specific extension, just go to its directory in “./custom_nodes” and delete it manually before restarting the container.

I think that’s the least complicated thing you can do in this case. This docker wasn’t built to run on its own on a server, it’s meant to run on your own local machine, where you’re monitoring the terminal output anyway.

Oh yes, if a new version of ComfyUI comes out then you have to rebuild the image and delete the VENV volume. The image is static, don’t forget. Inside, the Dockerfile does a new git clone and grabs the newest commit. Then it’ll install newer python packages, so you need to wipe the VENV volume and have the extensions reinstall their dependencies afterwards too. It’s a huge pain, but that’s just how it works.

Conclusion

With all that done, with a simple docker compose up I can have a stable ComfyUI environment isolated from the rest of my system, without dirtying anything with thousands of dubious dependencies. When I’m done I just do docker compose down or “ctrl-c” in its console and that’s it, my system stays clean.

On top of that, directories like “models” and their 500GB of models stay separate, you can back them up elsewhere if you need to. And if you want to bring it up on another machine, just take only those directories and bring up Docker Compose there and it has to come up exactly the same way, zero-setup. That was my goal in Dockerizing ComfyUI, FramePack, Video2K: to have everything portable, isolated and clean to bring up on any machine later without having to run sketchy installation scripts directly on my machine.

On my X I already showed my ComfyUI working. I loaded some random workflow I found on the Web and look at this:

In this workflow I upload a reference image and I can write a prompt for how I want to transform it:

With ComfyUI I have control over every step of the process, for example, this workflow can separate things like Depth Map and Edge Contour from the original image, to use as context before making the final image, ensuring a much more precise result:

And here’s a result I already got using the “waiNSFIllustrious” model that this workflow suggests, which is a model trained to generate anime-style images (there are dozens, for various different styles).

Switching the model to Mistoon, based on Stable Diffusion (SDXL), here’s another result, with the same prompt, in the same workflow:

And I can keep tuning dozens of parameters inside this workflow, swap vaes, text encoders, controlnets and much more until I get exactly the result I want. That’s the power of ComfyUI and its Custom Nodes and Workflows. That’s what I’ll try to explain the basics of in the next post.

Since I was already on a roll packaging this kind of tool in Docker, I decided to make another little project with a Dockerfile to run on my Manjaro Linux with an RTX 4090. The good thing about Aitrepreneur is that he has a Patreon you can subscribe to in order to contribute (it’s cheap) and he always uploads scripts to run on RunPod in case you don’t have a beefy machine or GPU, or .BAT scripts to run on your Windows.

My little project is on this GitHub and using it is very simple:

git clone https://github.com/akitaonrails/FramePack-Docker-CUDA.git
cd FramePack-Docker-CUDA
mkdir outputs
mkdir hf_download

# Build the image
docker build -t framepack-torch26-cu124:latest .

# Run mapping the directories outside:
docker run -it --rm --gpus all -p 7860:7860 \
  -v ./outputs:/app/outputs \
  -v ./hf_download:/app/hf_download \
  framepack-torch26-cu124:latest

This will build the Docker image and run it. And the first time you can see it will download a TON of models, HunyuanVideo, Flux and more. Get ready for over 30GB. But it’s only the first time because I map the download directory outside, so if you restart the container it will already have everything next time.

When it’s done, just go to http://localhost:7860 and you’ll see this super simple interface:

The controls are very simple:

• Disable the “Use TeaCache” option which makes it faster but then you risk getting those A.I. defects like hands with extra fingers and stuff like that.
• Total Video length can go up to 2 minutes, but only do that if you really have patience because it takes A LOT of time! I think it’s better to start by testing with short 5 to 10 second videos first.
• If your GPU has little VRAM you’ll be forced to mess with “GPU Inference Preserved Memory (GB) (larger means slower)”. In my case with 24GB of VRAM I left it as is, but look how it consumes resources while running:

Yes! 100% of the GPU and almost 100% of the VRAM, pulling more than 350W from the wall! (screw the environment! lol) The CPU stays chill, because this process is made to really MASSACRE the GPU. If you don’t have a good GPU, rent a machine on RunPod like I said before.

The 10-second video at the start of the post is a photo of one of my favorite action figures in my collection, Link from Breath of the Wild. And this is something that got me excited. This is the original photo that served as the basis for FramePack to make the video:

Making videos of photos of people is dull and tedious, but now that I’m learning to model 3D to print on my Bambulab, imagine then being able to make it animated too?

To animate in Blender and render also takes a lot of processing and time. But with this I can quickly prototype an animation and see how it looks, to decide if it’s worth making a better version in Blender afterwards. It opens up many possibilities for experimentation!

As impressive as this seems, it doesn’t do JUST ANYTHING, I had several experiments that went wrong and the movement it makes is quite limited, after all it only has a single photo to use as a reference and you can’t make movements that are too abrupt, there are limits. Even so it’s very fun to play with. The fact that it can identify more than just human beings is a big “plus” in my opinion, because that’s what got me more interested.

OH YES! Remember the other blog post I published today about doing UPSCALE TO 4K?? You can use it here too, take a video that FramePack generated that you really liked and upscale it to 4K to have even more definition! Then it’s at quality you can even use in video editing.

2-minute videos must take hours to make. 30 seconds took almost 1 hour. The best thing is to stick with 5-second videos to play and experiment, just that already takes nearly half an hour. Here are some more examples taken from photos of items in my collection (the ones you used to see in the background of my videos). Have fun!

This is an Ultraseven over 30cm tall that I found and brought directly from a used goods shop in Tokyo last year, one of my favorites especially because I watched it a lot in the early 80s when I was a kid.

Speaking of nostalgia, this is another awesome item in my collection, a super detailed Jiraya made by Iron Studios. And animated it actually turned out pretty cool. What did you think?

But now my FAVORITE, those who follow me on Instagram have seen that for the past few weeks I’ve been dedicating myself to improving my 3D techniques and also drawing and I made a concept of Mandalorian + Judge Dredd:

.png?disposition=attachment&locale=en)

Check out my other drawings on my Instagram afterwards, but anyway, I needed to know if FramePack was capable of animating drawings too and YES, IT CAN!! LOOK HOW B4D4SS!!! This one really impressed me, it gives a new dimension to my stuff!

Yes, all of this is quite heavy, but the advantage is that I can experiment INFINITELY, because everything runs locally. On someone’s commercial product in the “cloud”, I’d have to pay some subscription and they’d limit me, it would be impossible to keep experimenting whenever I wanted. Now I can do everything I want, whenever I want, however I want, without anyone or anything bothering me.

By the way, if you can, watch that “Boba Dredd” clip in full screen on a big monitor. Did you notice how sharp it is? That’s because I ran it through Video2K and upscaled to 4x the resolution that FramePack gave me. Combining these tools you can do REALLY interesting things, let me know later if you managed to use it and what cool stuff you made!

For collectors, the problem is that a lot of anime from the 90s and earlier never came out, and never will, on Blu-Ray (1080p) or UHD (4K). Only the most famous ones get the “remaster” treatment (taking the original master tapes and recapturing them at higher resolution). So a ton of old anime is stuck in the DVD era (480p) or VHS (480i).

Not every old anime can be upscaled, it can end up looking too “washed out”. Anyone who downloads torrents has probably seen badly done upscales. But I wanted to be able to test it myself, on my local machine, with my old files. Here’s a small example clip:

It’s a piece of the 1st episode of the legendary Rurouni Kenshin from 1996. Yes, today we have a remake, which they’re redoing from scratch. But for those of us who feel nostalgic about the original, there’s no replacement. The DVD quality isn’t horrendous, you can still watch it just fine today as is. But I’m curious to see how it looks in 4K.

I didn’t do any long research, but a year ago I had stumbled onto Real ERSGAN. That project has been discontinued for about 4 years, so I don’t know what the most modern option is today. But their pretrained models are still useful.

There are still projects that use this model, the one I know of is Video2K. On Windows it apparently even has a nice little graphical interface, but the advantage for Linux is that it runs in Docker. An example would be like this:

docker run --rm --gpus all \
  -v "$PWD/videos_in":/input \
  -v "$PWD/videos_out":/output \
  ghcr.io/k4yt3x/video2x:6.4.0 \
  -i /input/old_anime.mp4 -o /output/old_anime_4K.mkv \
  -p realesrgan -s 4 --realesrgan-model realesr-animevideov3

If you have your NVIDIA set up with the Container Toolkit, CUDA and all that (ask ChatGPT), Docker can map the device straight into the container. This makes it much easier to keep my Linux from getting dirty with dependencies and loose configs scattered around. I don’t do anything of this kind outside of containers anymore.

But I ran into a problem. If you’re using an AMD GPU, just doing this should already work, because this Docker image actually uses Vulkan and not CUDA. In my case I have a primary integrated AMD GPU and a secondary NVIDIA RTX 4090. I tried passing it directly:

❯ docker run --gpus all --privileged \
  --device=/dev/nvidia0 \
  --device=/dev/nvidiactl \
  --device=/dev/nvidia-uvm \
  --device=/dev/nvidia-modeset \
  --runtime=nvidia \
  -e VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json \
  -e VK_LAYER_PATH=/usr/share/vulkan/explicit_layer.d \
  -v "$HOME/Downloads/Video2K/videos_in":/input \
  -v "$HOME/Downloads/Video2K/videos_out":/output \
  --rm -it \
  ghcr.io/k4yt3x/video2x:6.4.0 \
  -i "/input/(B-A)Rurouni_Kenshin_-_01_(A5D3CB57).mkv" \
  -o "/output/(B-A)Rurouni_Kenshin_-_01_(A5D3CB57)_upscaled.mkv" \
  -p realesrgan -s 4 --realesrgan-model realesr-animevideov3

But it just won’t work. If I only say “–gpus all” it grabs the iGPU by default. Upscaling a video ends up consuming 100% of my CPU, all 32 threads, but my NVIDIA GPU just sits there watching, doing nothing. If I try to pass “/dev/dri/card0” or “/dev/nvidia0”, it errors out and can’t even start.

So from here on the solution is only for those who have NVIDIA:

CUDA Dockerfile

This time I decided not to be lazy and organize everything into a little project on Github. Clone it directly from there Batch Anime Upscaler Video2K Docker CUDA. Just do this:

git clone git@github.com:akitaonrails/Batch-Anime-Upscaler-Video2K-Docker-CUDA.git
cd Batch-Anime-Upscaler-Video2K-Docker-CUDA
mkdir input # put your old videos here
mkdir output

docker build --build-arg HOST_UID=$(id -u) --build-arg HOST_GID=$(id -g) -t anime-upscaler:latest .

If nothing goes wrong, this will build a Docker image with everything you need, then it’s just a matter of placing your videos in the subdirectory I mentioned, or any other you want, and configuring the run command:

 docker run --gpus all --rm \
  -v "./input":/input \
  -v "./output":/output \
  -e HOST_UID=$(id -u) -e HOST_GID=$(id -g) \
  anime-upscaler:latest

And that’s it. It takes a while. With my Ryzen 7950X3D plus RTX 4090, it processes at a rate of around 13 frames per second. So less than half the playback speed. If a video is 20 minutes long, it’ll take more than 40 minutes to process each one. It’s a good idea to set aside plenty of time, and that’s also why this is best reserved for videos you really want to watch.

The original Video2K script supports several parameters, which I left exposed as environment variables in the Dockerfile:

MODEL="${MODEL:-realesr-animevideov3}"
SCALE="${SCALE:-4}"
TILE="${TILE:-0}"
DENOISE="${DENOISE:-1.0}"
NUM_PROC="${NUM_PROC:-1}"

Then you can change them like this:

docker run --gpus all --rm \
  -e DENOISE=0.5 \
  -v ... \
  anime-upscaler:latest

As you can see, by default we use the pretrained model “realesr-animevideov3”, which is less detailed than the 6 billion parameter model but is much more stable between frames, which matters more for video. Most of the other models it supports, like “RealESRGAN_x4plus” or “ESRGAN_x4”, are meant for photos and images and not for video. Video demands temporal coherence, which is quite hard.

At most it’s worth trying the “realesr-general-x4v3” model, which they say is a bit lighter, in case your machine is struggling too much to run the default. But then you’ll have to modify the Dockerfile to download that model and rebuild:

# === OPTIONAL: Preload additional Real-ESRGAN models ===
# Uncomment any models you want available in the container at build time

# Anime 6-block model (better for anime stills, line art)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x4plus_anime_6B.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus_anime_6B.pth

# Original RealESRGAN 4x general-purpose model
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x4plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus.pth

# RealESRGAN 2x model (lighter upscale)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x2plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x2plus.pth

# ESRNet (non-GAN model, ultra-smooth)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRNet_x4plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRNet_x4plus.pth

# General-purpose lightweight video-friendly model (with denoise control)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.3.0/realesr-general-x4v3.pth \
#     -O /opt/Real-ESRGAN/weights/realesr-general-x4v3.pth

# Anime video model (default in your setup)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth \
#     -O /opt/Real-ESRGAN/weights/realesr-animevideov3.pth

# Original ESRGAN (2018) model
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/ESRGAN_x4.pth \
#     -O /opt/Real-ESRGAN/weights/ESRGAN_x4.pth

I haven’t tested all of them, but for anyone who wants to, I think those are the links. Change it in the Dockerfile, run the build, change the MODEL variable, and test on your videos.

What’s worth tweaking are things like DENOISE, which is set to “1.0”, the maximum, but that can leave the image too “stretched”. It’s worth trying smaller values, like “0.5”. It varies case by case, only by testing.

Another factor is the scale, SCALE, where the default is “4x”, which turns 1080p into 4K or 720p into 1440p. But sometimes you don’t need to go that high, and there’s no point because the original image already has very little detail anyway. Maybe 2x is enough. The script supports 2x, 3x, 4x but it caps at 4K resolution, so there’s no point trying to turn a video that’s already 4K into 8K, for example.

My RTX 4090 and my Ryzen 9 are quite powerful. Processing just one video, I don’t see them breaking a sweat at all:

Clearly you could run more videos in parallel. You have to be careful when it’s unbalanced. If the CPU climbs close to 100% just to separate the frames, having spare GPU capacity doesn’t help. With both having headroom, you can make the script spin up more than 1 process on the GPU and send frames in parallel with the “-e NUM_PROC=2” option. Look at how it now uses more of the GPU and the CPU still has room to spare. Maybe 3 is the maximum, but you have to test, it’s not constant the whole time.

This can dramatically reduce the total time. You can choose between increasing the GPU processes via the script or having two separate video directories and spinning up two parallel containers of this program to run at the same time, whichever works best for you.

Final Result

Let’s see how that first clip turned out with 4x upscale and 1.0 denoise:

The difference is HUGE, but whether it ended up “GOOD” is going to be VERY subjective. Personally I didn’t find the original bad to watch. Maybe tweaking DENOISE down a bit, or SCALE to 2x instead of 4x, would already be good enough, I don’t know.

For comparison, here’s another clip in the original old version:

And again, how it looks after the 4x upscale:

At least it’s working as it should, from here on it’s a matter of doing some tuning. If anyone has more tips or knows of newer open source upscaling models or projects, send them in the comments below!

Before someone asks, why not use NVIDIA’s DLSS 3, which they use to upscale game resolution? Answer: because they don’t expose that functionality as an API you can access, it’s only for their proprietary drivers in game profiles. They turn it on and tune it case by case. There’s no general API in the CUDA lib.

Post-publication Update

I decided to test the RealERSGAN_x4plus_anime_6B model, which is much heavier than the realesr-animevideov3 I uploaded to the repository.

Change the Dockerfile and add this after the wget for the other model:

RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth \
     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus_anime_6B.pth

This model can’t be run in parallel even on my RTX 4090, which stays constantly at 100%. If before it ran at around 13 frames per second, with this one it only runs at 3 frames per second. It’s 4x heavier and only for really beefy GPUs. If you don’t have something like that, stick with “realesr-animevideov3”, or with the even lighter (lower quality) “realesr-general-x4v3”. These models depend heavily on GPU power and especially on the amount of VRAM.

To run it, add “-e MODEL=RealESRGAN_x4plus_anime_6B” to the “docker run” command.

See how the same clip from before turned out using this model:

For having spent 4x more of my machine’s resources to process it, I didn’t find the final result that different, and in some parts I think it even has more defects. So that’s it, the result will vary, you have to test. It depends a lot on the quality of the original anime, but just because a bigger and theoretically “better” model exists doesn’t mean every result will turn out better.

I went poking around GitHub and there is an Awesome Image Colorization page with a bunch of research papers. Very useful for researchers but totally useless for me, since I am not going to build one from scratch lol. There were links to projects like ChromaGAN, but that one has been discontinued for about 5 years. And several other projects I saw were also discontinued ages ago. I do not know why there is so little open source still actively maintained.

Luckily I bumped into a promising one: DDColor.

Let’s clone the repo. And to keep going, I prefer to do everything inside Docker. Downloading Python dependencies always messes up my system and leaves a bunch of garbage behind, so the best thing is to isolate it all. Another thing, you need to download the binaries of the pre-trained models. Just for that I will use a bit of Python because of the “modelscope” lib that handles it:

git clone https://github.com/piddnad/DDColor.git
cd DDColor

mkdir modelscope

# create a new venv inside the DDColor project
python -m venv venv

venv/bin/pip3 install modelscope

venv/bin/python3 -c "from modelscope.hub.snapshot_download import snapshot_download; snapshot_download('damo/cv_ddcolor_image-colorization', cache_dir='./modelscope')"

Good Python practice is to do everything inside a VENV, right? Anyway, this should download the model binary, which weighs almost 900MB. Not huge.

Now we need a Dockerfile:

# Use NVIDIA CUDA base image with cuDNN 8 and Python support
FROM nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04

# Install Python and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip git libgl1 libglib2.0-0 && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Copy DDColor source code into the image
COPY . /app

# Install PyTorch (CUDA 11.8 compatible) and matching torchvision/torchaudio
RUN pip3 install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 \
    --index-url https://download.pytorch.org/whl/cu118

# Install Python dependencies, skipping dlib (not needed for inference)
RUN sed '/dlib/d' requirements.txt > temp-req.txt && pip3 install -r temp-req.txt && rm temp-req.txt

# Optional: install modelscope if you want to run snapshot_download manually inside the container
RUN pip3 install modelscope

Too lazy to open a pull request for this, if anyone wants to, send it over there.

Now we need to put a black and white image somewhere:

mkdir input
mkdir results
mv ~/Downloads/bw.jpg input/

Since my girlfriend’s photos are family and private, obviously I am not going to share them, so I grabbed a random one from Google Images:

Now we drop it in “./input” and run Docker with this command:

docker run --rm --gpus all \
  -v "$PWD/input":/workspace/input_images:ro \
  -v "$PWD/results":/workspace/results:rw \
  -v "$PWD/modelscope":/app/modelscope:ro \
  ddcolor:latest \
  python3 infer.py \
    --model_path /app/modelscope/damo/cv_ddcolor_image-colorization/pytorch_model.pt \
    --input /workspace/input_images \
    --output /workspace/results

Pay attention to the directory mapping. And note the “–gpus” option, which only works in my case because I have an RTX 4090 installed here. I have no idea if it is the same with AMD, but since the image is based on cuDNN/CUDA, I would say no. Ask ChatGPT.

If everything is right and it runs successfully, it will take just 1 second and produce this output in the terminal:

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

Output path: /workspace/results
100%|██████████| 1/1 [00:00<00:00,  4.09it/s]

And here is the result that shows up in the ./results directory:

Yes, I was impressed. Even with the image being from the Web, with watermark and all, it did everything properly and worked!!

The only problem left is being able to use a reference image to try to pull the colors. That is a harder problem and I asked ChatGPT, which answered me this:

No native support for a reference image:

The DDColor model, both in the official version hosted on ModelScope and in the piddnad/DDColor GitHub repo, was designed only for automatic colorization of black and white images, with no option to accept a reference image as input. The official description itself makes it clear that the model takes a single B&W image and returns a colored image.

The model’s forward method expects only one tensor, the B&W image converted into artificial RGB (actually the luminance expanded into 3 channels). In other words, there is no place in the model to pass a second image as a color reference.

This is also evident in the original predict.py from the Cog library, which only accepts the image parameter as input — with no support for reference, ref_image or anything of the kind.

How to solve the problem: two possible approaches

Since DDColor does not natively support a reference image to guide the colors, we have two alternatives:

🔁 1. Use a model specialized in reference-based colorization

There are models developed specifically for this, such as:

• Deep Exemplar-based Colorization (Zhang et al., CVPR 2018)

• ChromaGAN (2019)

• Exemplar-Transformer Colorization (Yin et al., ACM MM 2021)

• UniColor (Huang et al., 2022)

These models accept two images:

• The B&W image (main input)

• A colored image as a color reference

These models map semantic similarities (e.g., “skin here should have the same skin color as over there”) and do a transfer based on the image content, getting much more detailed and specific results. However, this would require:

• Downloading different weights

• Completely changing the architecture used

• Adapting the pipeline and inference

🎯 2. Use global color transfer after the DDColor output (our current approach)

The practical and lightweight solution we used was: ➡️ Run DDColor normally to generate the colored image automatically ➡️ Then globally adjust the colors of the result to look more like the reference image.

This adjustment is done with the Reinhard et al. (2001) algorithm:

1. We convert both the output image and the reference to the Lab color space.

2. We compute the mean and standard deviation of the a and b channels (the ones that represent color).

3. We adjust the result to have the same statistical distribution of color as the reference image.

4. This tends to leave the image with the same overall “mood” or tone (warmer, cooler, more vibrant).

This method is fast, lightweight, and works without altering the DDColor model.

Continuing

I used the modification it suggested to the predict.py file:

import os
import cv2
import numpy as np
import torch
import torch.nn.functional as F
from cog import BasePredictor, Input, Path
from basicsr.archs.ddcolor_arch import DDColor

def color_transfer(reference_bgr: np.ndarray, target_bgr: np.ndarray) -> np.ndarray:
    """Transfer color palette of reference image to target image using Lab mean/std."""
    # Convert BGR images to Lab color space (float32 for precision)
    ref_lab = cv2.cvtColor(reference_bgr, cv2.COLOR_BGR2LAB).astype("float32")
    tgt_lab = cv2.cvtColor(target_bgr, cv2.COLOR_BGR2LAB).astype("float32")
    # Split channels
    L_ref, a_ref, b_ref = cv2.split(ref_lab)
    L_tgt, a_tgt, b_tgt = cv2.split(tgt_lab)
    # Compute mean and std for reference and target (a and b channels)
    a_ref_mean, a_ref_std = a_ref.mean(), a_ref.std()
    b_ref_mean, b_ref_std = b_ref.mean(), b_ref.std()
    a_tgt_mean, a_tgt_std = a_tgt.mean(), a_tgt.std()
    b_tgt_mean, b_tgt_std = b_tgt.mean(), b_tgt.std()
    # Subtract target means, scale by reference/target std, add reference means
    # (Avoid division by zero in case of zero std)
    if a_tgt_std > 1e-6:
        a_tgt = ((a_tgt - a_tgt_mean) * (a_ref_std / a_tgt_std)) + a_ref_mean
    else:
        a_tgt = a_tgt - a_tgt_mean + a_ref_mean
    if b_tgt_std > 1e-6:
        b_tgt = ((b_tgt - b_tgt_mean) * (b_ref_std / b_tgt_std)) + b_ref_mean
    else:
        b_tgt = b_tgt - b_tgt_mean + b_ref_mean
    # Clip values to valid Lab range [0,255] after transfer
    a_tgt = np.clip(a_tgt, 0, 255)
    b_tgt = np.clip(b_tgt, 0, 255)
    # Merge channels back (use original L from target to preserve brightness)
    merged_lab = cv2.merge([L_tgt, a_tgt, b_tgt]).astype("uint8")
    # Convert back to BGR color space
    result_bgr = cv2.cvtColor(merged_lab, cv2.COLOR_LAB2BGR)
    return result_bgr

class ImageColorizationPipeline:
    """Helper pipeline to load DDColor model and process images."""
    def __init__(self, model_path: str, input_size: int = 256, model_size: str = "large"):
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        # Choose encoder backbone based on model size
        encoder_name = "convnext-t" if model_size == "tiny" else "convnext-l"
        # Initialize DDColor model
        self.model = DDColor(
            encoder_name=encoder_name,
            decoder_name="MultiScaleColorDecoder",
            input_size=[input_size, input_size],
            num_output_channels=2,    # model predicts ab channels
            last_norm="Spectral",
            do_normalize=False,
            num_queries=100,
            num_scales=3,
            dec_layers=9,
        ).to(self.device)
        # Load weights
        state = torch.load(model_path, map_location="cpu")
        # Some checkpoints store weights under 'params' key
        self.model.load_state_dict(state.get("params", state), strict=False)
        self.model.eval()
        self.input_size = input_size  # store for processing

    @torch.no_grad()
    def process(self, img_bgr: np.ndarray) -> np.ndarray:
        """Colorize a BGR image (numpy array) using the loaded DDColor model."""
        # Preserve original resolution L channel
        orig_h, orig_w = img_bgr.shape[:2]
        # Convert to Lab and extract L channel at original size
        img_lab = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2LAB).astype("float32")
        orig_L = img_lab[:, :, 0:1]  # shape (H, W, 1)
        # Prepare grayscale input at model resolution
        # Resize input to model expected size
        inp = cv2.resize(img_bgr, (self.input_size, self.input_size))
        inp_lab = cv2.cvtColor(inp, cv2.COLOR_BGR2LAB).astype("float32")
        L = inp_lab[:, :, 0:1]
        # Create grayscale Lab image by zeroing AB channels
        gray_lab = np.concatenate([L, np.zeros_like(L), np.zeros_like(L)], axis=2)
        # Convert back to RGB (now it's a gray RGB image of size input_size)
        gray_rgb = cv2.cvtColor(gray_lab.astype("uint8"), cv2.COLOR_LAB2RGB)
        # Prepare tensor and run model
        tensor = torch.from_numpy(gray_rgb.transpose(2, 0, 1)).float().unsqueeze(0).to(self.device)
        out_ab = self.model(tensor)  # output is (1,2,H_out,W_out) in Lab AB
        out_ab = out_ab.cpu().float()
        # Resize output AB to original image size
        out_ab_resized = F.interpolate(out_ab, size=(orig_h, orig_w), mode="bilinear", align_corners=False)
        out_ab_resized = out_ab_resized[0].numpy().transpose(1, 2, 0)  # (H, W, 2)
        # Combine original L and predicted AB, then convert to BGR
        out_lab = np.concatenate([orig_L, out_ab_resized], axis=2).astype("uint8")
        out_bgr = cv2.cvtColor(out_lab, cv2.COLOR_LAB2BGR)
        return out_bgr

class Predictor(BasePredictor):
    def setup(self):
        """Load models into memory for efficient multiple predictions."""
        # Determine paths for large and tiny model weights
        large_model_path = "checkpoints/ddcolor_modelscope.pth"
        tiny_model_path  = "checkpoints/ddcolor_paper_tiny.pth"
        # (Ensure the above files exist. If not, download from ModelScope or HuggingFace as described.)
        # Initialize pipelines for large and tiny models
        self.colorizer_large = ImageColorizationPipeline(model_path=large_model_path, input_size=512, model_size="large")
        self.colorizer_tiny  = ImageColorizationPipeline(model_path=tiny_model_path,  input_size=512, model_size="tiny")

    def predict(
        self,
        image: Path = Input(description="Grayscale input image."),
        reference: Path = Input(description="Optional reference image to guide colors", default=None),
        model_size: str = Input(description="Model size to use (large = highest quality, tiny = faster).",
                                choices=["large", "tiny"], default="large"),
    ) -> Path:
        """Run a single prediction. Colorizes the image, optionally using a reference image for color bias."""
        # Read input image
        img = cv2.imread(str(image))
        if img is None:
            raise ValueError("Failed to load input image")
        # Select model pipeline
        pipeline = self.colorizer_tiny if model_size == "tiny" else self.colorizer_large
        # Colorize the image using DDColor
        output_bgr = pipeline.process(img)
        # If a reference image is provided, transfer its color style to the output
        if reference is not None:
            ref_img = cv2.imread(str(reference))
            if ref_img is None:
                raise ValueError("Failed to load reference image")
            # Only apply transfer if reference is a color image
            if ref_img.shape[2] == 3:
                output_bgr = color_transfer(ref_img, output_bgr)
        # Save result
        out_path = "/tmp/out.png"
        cv2.imwrite(out_path, output_bgr)
        return Path(out_path)

With this modification I can run with a new parameter, note the “–ref” option that exists now and place your image in the right spot:

❯ docker run --rm --gpus all \
  -v "$PWD/input":/workspace/input_images:ro \
  -v "$PWD/results":/workspace/results:rw \
  -v "$PWD/ref_images":/workspace/ref_images:ro \
  -v "$PWD/modelscope":/app/modelscope:ro \
  -v "$PWD/infer.py":/app/infer.py:ro \
  ddcolor:latest \
  python3 /app/infer.py \
    --model_path /app/modelscope/damo/cv_ddcolor_image-colorization/pytorch_model.pt \
    --input /workspace/input_images \
    --ref /workspace/ref_images/images.jpg \
    --output /workspace/results

I passed in another colored reference photo but saw no difference in the result. Once again, I asked ChatGPT:

Why didn’t you notice a difference in the result?

The global color transfer only changes the “overall palette” — and not the colors of each object.

If the reference image has soft tones similar to what DDColor would already apply automatically, the final effect will look identical.

But if you use as reference a heavily saturated image (for example, an orange sky, or a deep blue environment), you will see DDColor’s output visually pulled toward that style.

Conclusion

I followed the suggestion and downloaded another totally unrelated image, colored, super saturated with vibrant colors to see if it made a difference. This is the reference image I passed to Docker:

And here is the new result applying it to the previous image in post-processing (this step is no longer A.I., it really is just image post-processing).

Compare it with the other re-colorized photo, I will put it down here again so you can compare:

Got it? Now you have to keep testing with different references to see how they influence the final result, but indeed, it manages to apply the Reinhard algorithm and do a global color transfer. As the name says, it is “global”, not “per object”, so it is hard to control only parts of the image — it acts on the whole image. But in theory I think you could do a manual colorization in Photoshop with saturated colors, near where you want to influence, to try to nudge it, but I have not tried that yet. Later, if anyone does, drop it in the comments.

But that is it. Now at least I can have fun grabbing old family photos and re-colorizing them.