Song to Screen: Brick-Built Music Videos on a DIY GPU Pipeline

Why Should You Care?

A few years ago, “make a music video” meant a camera crew, a set, and a budget most solo builders don’t have. Today you can write the song, generate the visuals, and cut the final edit without ever leaving a terminal, and you don’t need a data center to do it. You need a mid-range GPU or a few cents an hour of rented compute, plus a pipeline that respects what each stage is actually good at.

We build brick-built minifigure animation as a running side project: short stop-motion-style music videos made entirely from generated footage. It’s a good stress test for an “own the whole workflow” claim, because it touches every discipline at once: music generation, stem separation, word-level timing, image generation, video generation, and audio mixing. If a pipeline can carry a song from an idea to a finished, beat-synced video without a single closed black box in the middle, it can carry almost anything else you’d want to build. Here’s how ours is wired, what it costs, and the three mistakes that were expensive enough to turn into house rules.

The Pipeline Map: Song, Still, Motion, Cut

Four stages, each with a single job:

Song. Composition happens in Suno, and we’re honest about that; it’s a generation tool, not a trick we’re hiding. From there, word-level timing and a clean instrumental bed come from tools we run ourselves.
Still. Before any motion is generated, we craft a single reference image: the first frame of the clip. This is the newest addition to the pipeline, and it’s the biggest quality lever we’ve found (more on that below).
Motion. The still becomes the first frame of a short video clip, generated on a rented GPU through ComfyUI.
Cut. Clips get chained into a continuous film and mixed against the song’s actual audio, never against a guess at where the beats are.

Stage 1: song and timing

Suno handles composition. Everything downstream of the audio file is ours. We run demucs (the htdemucs_ft model) to split the master into four stems, vocals, drums, bass, and other, which gives us a clean instrumental bed independent of anything the video model generates:

$ demucs -n htdemucs_ft song.wav
# → separated/htdemucs_ft/song/{vocals,drums,bass,other}.wav

For the actual splice points, where a verse ends, where the hook lands, we don’t trust a treatment’s guessed timestamps. We re-derive them from the real audio with faster-whisper, word-level, every time:

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")
segments, _ = model.transcribe(
    "song.wav",
    language="en",
    word_timestamps=True,
    vad_filter=False,   # voice-activity filtering can silently empty music/vocals; off, always
)

vad_filter=False matters more than it looks like it should. We’ve had voice-activity detection quietly strip the transcript on vocal-heavy audio because it mistook singing for non-speech and threw the segment out. No error, no warning, just an empty result. For music and stylized vocals, turn it off.

Stage 2: still, the keyframe

This is the stage we added most recently, after an A/B test we ran ourselves. The question was simple: does a crafted start image beat pure text-to-video? We generated three test clips across four candidate compositions and two image generators, 24 stills total, at a combined cost of roughly $0.80 (about $0.02/image on one generator, about $0.04/image on the other). That’s the entire spend for the experiment. No GPU rental, no pod time; still images are a cloud API call, not a render job.

The result was decisive enough to change the house default. More on why in the next section.

Stage 3: motion, image-to-video

The still becomes the first frame (“first-frame conditioning”) for a short video clip generated through ComfyUI, running the h3 model on a rented GPU. A render job looks roughly like this:

{
  "model": "minimax_h3_fl2va_pruned_int8_convrot.safetensors",
  "width": 1152,
  "height": 640,
  "length": 124,
  "promptText": "<full shot description>",
  "imageFile": "<the crafted keyframe, pushed to the pod first>"
}

A production-tier clip at 16:9 renders at 1152x640, 124 frames, about 5.2 seconds, and takes roughly 7 minutes on a rented A40. At community pricing that’s a few cents of GPU time per clip. The render cost isn’t the expensive part of this pipeline; the iteration is.

Stage 4: cut, assembly and mix

Clips chain into one continuous film. The ambient bed from the generated footage gets ducked under the vocal track with ffmpeg’s sidechain compressor, so the bed drops out of the way whenever the vocal is present and swells back in the gaps:

[amb]volume=0.28[ambpre];[ambpre][vokey]sidechaincompress=threshold=0.02:ratio=14:attack=12:release=280:makeup=1[ambduck]

One ffmpeg gotcha worth knowing: if you place -ss/-t (seek/trim) after -i instead of before it, filters like afade will silently mistime. No error, no warning, just a slightly wrong fade that’s easy to miss on a quick listen. Always sanity-check the actual duration of a cut segment before it goes into the mix, not just whether ffmpeg exited with status 0.

What Runs Where, and What It Costs

Stage	Tool	Runs on	Rough cost
Composition	Suno	cloud	subscription
Stems	`demucs` (htdemucs_ft)	local GPU	free (local compute)
Word timing	`faster-whisper` large-v3	local GPU/CPU	free (local compute)
Keyframe stills	image-generation APIs	cloud	~$0.02 to $0.04/image
Video render	ComfyUI + h3 model	rented GPU (RunPod A40)	~$0.35/hr (community tier, at time of writing), roughly 7 min/clip
Assembly + mix	`ffmpeg`	local CPU	free (local compute)

Nothing here requires a data center. The most expensive single line item, the rented GPU, is billed by the minute, and a finished clip costs pennies of compute once you subtract iteration time.

Three Lessons We Paid For

1. Start-image quality is the biggest lever you have

Before we added the keyframe stage, we were asking a text-to-video model to compose an entire scene from a paragraph of description. Two failure patterns showed up repeatedly:

Negation doesn’t reliably work. Telling a video model “no X” in a prompt is far less effective than describing positively what does occupy that space. If you don’t want something in frame, don’t say what to exclude; say what’s there instead. We had a scene where an unwanted object kept reappearing despite an explicit negative instruction. Redesigning the composition to positively state what filled that space fixed it on the first try.
A dominant object needs its own explicit description. When a single object fills most of the frame, a video model tends to default to whatever it saw most in training for objects that shape, not necessarily what the rest of your scene’s style calls for. State the material and look directly on that surface’s own description, not just once at the top of the prompt.

The still-image stage exists because you can iterate on a single frame, checking composition, lettering, material, everything, for a couple of cents, and gate it with your own eyes before it ever reaches a paid video render. First-frame conditioning then hands the video model a scene it doesn’t have to invent; it only has to animate. That’s a fundamentally easier job, and it shows in the output.

2. Pin your engine versions

This one isn’t specific to video generation. It bit us on a different GPU lane, and it’s general enough to matter for anything you rent compute for. An unpinned install (curl install.sh | sh, grabbing “latest”) pulled in an inference-engine build with a decode regression. The failure was silent: no crash, no error code, just corrupted output that looked plausible enough at a glance to almost ship. The fix was to pin the exact engine version we’d tested against and assert it on boot, failing loud on drift rather than failing quiet in the output.

The lesson generalizes: identical code plus a drifted environment produces different behavior, and version control can’t see the difference, because nothing about your code changed. Pin the engine, the model file, and the driver/CUDA version together, and treat any of them changing as a reason to re-test, not an assumption that “it still works.”

3. Read your outputs, not your job counts

A rented GPU will happily report “100% complete” on a batch that is completely broken. Every health signal, exit code, worker count, VRAM usage, can be green while the actual output is garbage, because none of those signals measure correctness. The only thing that measures correctness is looking at the output.

Our house rule now: before scaling any batch past a small sample, someone actually looks at (or listens to) a real slice of what came back, not a summary, not a count, the actual frames or audio. It costs a few minutes and it’s caught more bad renders than every automated check combined. A done-count is a liveness signal. It is not a quality signal.

What You Need to Start

You don’t need much:

A GPU, one way or another. A mid-range card handles stem separation and transcription locally (both are CPU/GPU-friendly, no exotic hardware required). For the video render step, renting is usually more practical than owning: community-tier 48GB cards run around $0.35/hr at the pricing we’ve paid, with secure-tier and smaller-card options priced above and below that depending on the provider and moment you check. Rental marketplaces move; check current pricing before you budget, and match the card to the bottleneck, not to habit; renting more VRAM than a job can use is money spent on nothing.
demucs, faster-whisper, and ffmpeg. All open source, all free, all doing real work in this pipeline.
A ComfyUI-compatible video model and somewhere to run it. Rented, by the hour, only while it’s rendering.
Patience for the still-image stage. It’s the cheapest part of the pipeline and the highest-leverage one. Don’t skip straight to video.

If a video brought you here, welcome; this post is the full breakdown behind it, every stage, every real cost, every mistake we paid for so you don’t have to repeat it. If you found this some other way, the finished videos live on the SouthernSky channel: youtube.com/@southernskycloud, and every video description links back to the post that built it, with the exact GPU and models used for that specific render.

What You Learned

A four-stage pipeline (song, still, motion, cut) lets you own generated video end to end without any stage being a black box.
Crafting a still image first and feeding it as the first frame to a video model beats pure text-to-video, because composition problems are cheap to fix on a still and expensive to fix on a rendered clip.
Negation is a weak prompting tool for generative video and images; state what should be in frame, not what shouldn’t.
An unpinned inference engine can silently corrupt output with every health signal still green; pin your versions and assert them at boot.
A completed job count tells you the job finished, not that it worked. Look at real output before you scale.