feat(posts): publish 4-minute ai music video case study (en+zh)

2026-10-25 11:37:07 +08:00
1 parent f530311f45
commit 103ccedd63
@@ -0,0 +1,154 @@
+---
+title: "From Audio to a 4-Minute Music Video: Orchestrating AI Lyrics, Image Gen, Video Diffusion, and Face-Swapping"
+description: "A complete production breakdown of building a 4-minute 1080p anime OP music video (Crescent Bay) with 24 storyboard scenes, AI video diffusion, and FaceFusion."
+pubDate: 2026-10-25
+category: case-studies
+tags: ["ai-video", "music-video", "veo", "imagen", "face-swapping", "ffmpeg", "remotion"]
+ogImage: "/og/from-audio-to-a-4-minute-ai-music-video.jpg"
+banner: "/banners/from-audio-to-a-4-minute-ai-music-video.jpg"
+draft: false
+---
+
+Most AI video demonstrations end after five seconds: a stunning single-shot clip of a character turning their head or raindrops hitting a puddle. It looks cinematic in a tweet, but assembling a complete four-minute music video (240 seconds) across 24 consecutive scenes is an entirely different engineering challenge.
+
+The moment you scale from one shot to twenty-four, generative pipelines fall apart in predictable ways:
+1. **Character and costume drift:** Scene 3 features a maiden in pale cyan silk; by Scene 5 she is wearing dark velvet with a different hairstyle.
+2. **Context fatigue in LLM prompt generation:** Asking a language model to spit out 24 scene prompts in one prompt leads to truncated descriptions and dropped character tokens past Scene 12.
+3. **The 10-second distortion trap:** Generative video models (Kling, Runway, Minimax, Google Veo) struggle with 10-second clips; natural motions conclude in 2 seconds, and the model fills the remaining 8 seconds with distorted limbs or spinning heads.
+4. **Transition black-dips:** Chaining 24 independent clips with naive fade-to-black creates an annoying strobe effect that ruins immersion.
+
+To prove an end-to-end automated pipeline could produce a full-length, release-ready piece, I produced a complete J-Rock anime OP remake of the classic track **Crescent Bay (月牙湾)**.
+
+Here is the finished 4-minute 1080p music video:
+
+<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; border-radius: 8px; margin: 2rem 0; box-shadow: 0 8px 24px rgba(0,0,0,0.4);">
+  <iframe 
+    src="https://www.youtube.com/embed/OJC-Zdn232k" 
+    title="月牙湾 (Crescent Bay) - 1080P AI Music Video" 
+    style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border: 0;" 
+    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" 
+    allowfullscreen>
+  </iframe>
+</div>
+
+---
+
+## 1. End-to-End Pipeline Architecture
+
+The entire workflow is split into four decoupled stages rather than one monolithic script:
+
+```
+[Master Audio (240s MP3) + Lyrics]
+             │
+             ▼
+[Stage 1: LLM Screenwriter & Visual DNA]
+ ├─ 24 Time-Grid Lyric Slots (10s per slot)
+ └─ Global Visual DNA Constants (Character, Environment, Lighting, 50mm Lens)
+             │
+             ▼
+[Stage 2: 24 Atomic Variable Slots]
+ ├─ Boolean include_character switch
+ ├─ Rule-of-thirds composition (Left / Right negative space)
+ └─ Deterministic Code Assembler (Zero Prompt Drift)
+             │
+             ▼
+[Stage 3: Image-to-Video Diffusion & Face Consistency]
+ ├─ Text-to-Image (Imagen 3 / Flux) -> 24 Still Frames (16:9)
+ ├─ Image-to-Video (Veo 2 / Kling) -> 24 x 10s Fluid Clips
+ └─ High-precision Face-Swapping (FaceFusion / Insplagger) on portrait shots
+             │
+             ▼
+[Stage 4: Programmatic Video Assembly & Subtitles]
+ ├─ FFmpeg Constant-Power (P^0.78) Smooth Crossfade (Zero Black-Screen Dip)
+ ├─ Remotion React 1080p Engine (Dynamic Visualizer Ring + Subtitles)
+ └─ 1-Second Short GOP (-g 30 -bf 0) Export
+```
+
+---
+
+## 2. Decoupling Prompts: How to Stop Character Drift
+
+The classic pitfall in multi-scene generation is asking an LLM to generate 24 full Text-to-Image and Image-to-Video prompts in a single prompt.
+
+When an LLM writes 24 scenes with 150 words each, it hits token limits, and by Scene 15 it starts taking shortcuts. In initial drafts, the prompt for Scene 4 dropped the character's clothing and ethnicity, describing her merely as *"The maiden with delicate features."* The diffusion model promptly rendered a completely different person.
+
+### The Solution: Two-Stage Generation + Code Assembly
+1. **Stage 1 (Constants):** Define `visual_dna` once. This locks the protagonist's silhouette, specific hair ornaments, dual-layered garment fabrics, color names (e.g. `pale celadon-cyan and matte silver` instead of hex codes that get painted as text), and camera optics (`ARRI Alexa 65, 50mm anamorphic lens`).
+2. **Stage 2 (Variables):** The LLM outputs **only atomic variable slots** for the 24 scenes:
+   ```json
+   {
+     "scene_id": 4,
+     "include_character": true,
+     "shot_type": "Close-up Side Profile",
+     "subject_side": "right",
+     "negative_space_medium": "warm faint candle reflection on blurred rosewood screen",
+     "t2i_scene_desc": "Looking downward in gentle contemplation, eyelashes catching rain light.",
+     "i2v_camera": "Slow cinematic dolly in",
+     "i2v_subject_motion": "Slowly blinks once, subtle chest breathing.",
+     "i2v_ambient_motion": "Candle glow flickers softly in out-of-focus bokeh."
+   }
+   ```
+3. **Deterministic Assembly:** A Python script (`scripts/assemble_prompts.py`) stitches Stage 1 constants and Stage 2 variables together into the final prompts. If `include_character` is false (e.g. macro shots of a copper wind chime or misty river ripples), character tokens are completely omitted so the model doesn't hallucinate a face into a landscape.
+
+---
+
+## 3. The 10-Second Video Diffusion Rules
+
+Generating 10-second video clips from still images requires strict motion rules:
+
+* **Ban Negative Words in Positive Prompts:** Prompting `no morphing` or `no deformation` activates negative token embeddings in diffusion models like Runway and Veo, paradoxically triggering limb mutations. Instead, use positive stabilizing anchors: `consistent anatomy, temporal stability, rigid structure`. Put negative tokens strictly in the dedicated `negative_prompt` API field.
+* **The 120fps Slow-Motion Anchor:** A human gesture like turning a head takes 2 seconds. When forced to fill 10 seconds, models over-rotate necks or spawn extra arms. Appending `captured in 120fps extreme slow motion, subtle micro-movements` stretches a natural 2-second breeze or blink across the full 10-second duration.
+* **Camera Crash Guard:** If the visualizer ring and lyrics sit on the left, the subject must stay on the right. In that layout, any `Pan Right` or `Truck Right` camera motion causes the subject to drift leftward across the screen, colliding directly with the subtitles by second 6. The assembler script enforces a strict rule: if `subject_side == "right"`, rightward camera panning is blocked.
+
+---
+
+## 4. Zero-Dip Background Transitions (FFmpeg Constant-Power Crossfade)
+
+In video editing software, placing 24 clips on a timeline with cross-dissolves is trivial. In automated headless pipelines, naive transitions produce severe artifacts:
+
+1. **`fadeblack` causes strobe flicker:** Standard FFmpeg scripts often use `xfade=transition=fadeblack`, which dips through black (`A -> black -> B`). Across 24 scenes, the viewer is subjected to a disorienting black flash every 10 seconds.
+2. **Standard linear `fade` drops brightness by 16%:** In non-linear sRGB/YUV color space, linear crossfade $((1-\alpha)A + \alpha B)$ loses up to 20% perceived midpoint luminance at $\alpha = 0.5$.
+3. **The 34.5-Second Timeline Shrink:** When 24 ten-second clips overlap with a 1.5s crossfade, the total runtime is not 240 seconds:
+   $$\text{Total Duration} = 24 \times 10.0 - (23 \times 1.5) = 205.5\text{ seconds}$$
+   The video runs out 34.5 seconds before the song ends!
+
+### The Mathematical Fix
+We calculate integer frame counts per clip upfront:
+$$F_{\text{clip}} = \left\lceil \frac{F_{\text{target}} + (N - 1) \times F_D}{N} \right\rceil = \left\lceil \frac{7200 + 23 \times 45}{24} \right\rceil = 344\text{ frames}\ (11.4667\text{s})$$
+
+Each raw 10-second clip is gently stretched via `setpts=1.14667*PTS` (a subtle 12% slowdown that actually enhances fluid motion), then chained with custom constant-power curve `pow(P, 0.78)`:
+
+```bash
+ffmpeg -y -i v0.mp4 -i v1.mp4 ... \
+  -filter_complex "[0:v][1:v]xfade=transition=custom:duration=1.5:offset=9.9667:expr='clip(A*pow(P,0.78)+B*pow(1-P,0.78),0,255)'[xf1];..." \
+  -c:v libx264 -preset veryfast -crf 18 -g 30 -bf 0 dynamic_bg.mp4
+```
+
+Encoding with `-g 30 -bf 0` (1-second GOP, no B-frames) allows Remotion's multithreaded renderer to seek background frames 50% faster, eliminating render timeouts.
+
+---
+
+## 5. Subtitle Protection & Dual-Layer Typography
+
+Complex animated backgrounds with glowing audio rings create visual clutter that drowns out lyrics.
+
+Rather than running an expensive `backdrop-filter: blur` in Chromium (which forces GPU framebuffer readbacks and slows rendering by 40%), we implemented a dual-layer SVG/DOM strategy:
+1. **Base Layer:** Renders a 4.2px dark stroke (`paint-order: stroke fill`) and triple-compound drop shadows to ensure text reads clearly against bright desert or sky footage.
+2. **Top Layer:** Uses `background-clip: text` with a 12% soft-edge gradient to create smooth karaoke-style highlighting synchronized with the audio.
+3. **Visual Ducking:** When vocals are active, the audio spectrum ring automatically ducks its opacity to 50% and amplitude to 75%, allowing the sung line to take visual priority.
+
+---
+
+## 6. The Takeaway
+
+Full-length AI video generation is no longer blocked by model capabilities; it is blocked by **orchestration discipline**.
+
+By treating character traits as immutable constants, enforcing strict negative space and camera collision rules, and solving video transition mathematics in FFmpeg before final compositing, a single workstation can turn a song and text into a polished 4-minute 1080p anime music video with zero manual keyframing.
+
+---
+
+*Need custom AI video pipelines, automated media workflows, or high-throughput rendering infrastructure for your business? Let's connect:*
+
+- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
+- 📧 **Email:** [[email protected]](mailto:[email protected]?subject=AI%20Music%20Video%20Pipeline)
+- 🌐 **Website:** [hoelee.com](https://hoelee.com)
@@ -0,0 +1,154 @@
+---
+title: "从一首歌到 4 分钟完整 MV：我如何用 AI 串起作词、分镜生图、图生视频与换脸全流程"
+description: "基于《月牙湾》1080P 高清动漫 OP 制作实录：详解 24 分镜时间轴拆解、提示词防漂移两阶段架构、图生视频 10 秒慢运镜控制与无缝平滑叠化合成。"
+pubDate: 2026-10-25
+category: case-studies
+tags: ["ai-video", "music-video", "veo", "imagen", "face-swapping", "ffmpeg", "remotion"]
+ogImage: "/og/zh-from-audio-to-a-4-minute-ai-music-video.jpg"
+banner: "/banners/from-audio-to-a-4-minute-ai-music-video.jpg"
+draft: false
+---
+
+绝大多数 AI 视频演示都停留在 5 秒的片段：一个女生转头、一段雨水打在水坑里的特写。发在社交媒体上很惊艳，但要真正用 AI 拼出一整首长达 4 分钟（240 秒）、跨越 24 个连续镜头的工业级音乐视频（MV），完全是另一套工程体系。
+
+一旦把生成尺度从单镜头放大到 24 镜，现有的生成模型就会暴露出一连串致命问题：
+1. **画风突变与主角变脸**：第 3 镜是青衫抚琴少女，第 5 镜衣服换了颜色，第 7 镜连五官发型都变了；
+2. **大模型上下文衰减（Context Drift）**：如果让 LLM 一次性手写 24 个镜头的提示词，生成到第 15 镜以后，提示词往往越来越敷衍，角色锚点词被大段漏掉；
+3. **10 秒生成时长陷阱**：扩散模型（Kling、Runway、Minimax、Google Veo）通常在 2 秒内就完成了动作，为了填满剩下的 8 秒，模型会强行脑补，导致人物脖子反转 180 度或肢体畸变；
+4. **镜头切换黑屏闪烁**：把 24 段视频用传统的淡入黑场（Fade to black）串起来，整首 MV 看起来就像频繁暗下的幻灯片，极度影响观感。
+
+为了验证整套流水线能否真正跑通商业级交付，我重构改编了经典华语曲目**《月牙湾》(Crescent Bay)**，将其打造为日系热血奇幻动漫 OP 风（Oriental Fantasy J-Rock），并生成了整整 4 分钟 1080P 高清成片。
+
+以下是完整的成片视频展示（可直接全屏播放）：
+
+<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; border-radius: 8px; margin: 2rem 0; box-shadow: 0 8px 24px rgba(0,0,0,0.4);">
+  <iframe 
+    src="https://www.youtube.com/embed/OJC-Zdn232k" 
+    title="月牙湾 (Crescent Bay) - 1080P AI 音乐视频" 
+    style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border: 0;" 
+    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" 
+    allowfullscreen>
+  </iframe>
+</div>
+
+---
+
+## 1. 端到端系统架构设计
+
+整套管线没有采用耦合的单一大脚本，而是拆解为四个清晰解耦的模块：
+
+```
+[原始音频 (240秒 MP3) + 歌词]
+             │
+             ▼
+[第一阶段：编剧导演与全局视觉 DNA]
+ ├─ 24 个 10 秒时钟格歌词 (Time-Grid Lyrics)
+ └─ 锁定全局视觉 DNA 常量（角色、场景、色温、ARRI 50mm 镜头）
+             │
+             ▼
+[第二阶段：24 镜头原子变量槽位]
+ ├─ include_character 布尔开关（空镜头自动剥离人物）
+ ├─ 三分法构图（左侧 / 右侧预留负空间）
+ └─ 代码层确定性组装引擎（消除提示词偷工减料）
+             │
+             ▼
+[第三阶段：图生视频扩散与面部一致性]
+ ├─ 文生图 (Imagen 3 / Flux) 生成 24 张 16:9 高清静帧
+ ├─ 图生视频 (Veo 2 / Kling) 驱动 24 段 10 秒平滑动态
+ └─ 高精度人脸替换 (FaceFusion / Insplagger) 锁定核心特写
+             │
+             ▼
+[第四阶段：程序化无缝拼接与歌词动效合成]
+ ├─ FFmpeg 恒定功率 (P^0.78) 平滑交叉叠化（彻底消灭黑场）
+ ├─ Remotion React 1080P 引擎（动态音频圆环 + 卡拉OK流光歌词）
+ └─ 1 秒短 GOP (-g 30 -bf 0) 导出
+```
+
+---
+
+## 2. 提示词解耦：彻底杜绝角色换脸与偷工减料
+
+多镜头连续生成最大的死穴，是让大模型在一次对话里直接手写 24 个镜头的全量生图生视频提示词。
+
+当大模型连续吐出 24 个段落时，Token 消耗高达 5,000+，生成到中后段就会发生严重的“注意力衰减”。在最初的测试中，大模型在第 4 镜擅自删除了主角的服装材质与族裔设定，只写了一句 *"The maiden with delicate features"*，生图模型当场换了一张完全陌生的脸。
+
+### 工业级解法：两阶段分层 + 代码确定性组装
+1. **第一阶段（常量定义）**：在 Stage 1 仅定义一次 `visual_dna`。锁定主角的具体发型、标志性发饰道具（Hero Prop）、内外双层服装面料与工业色名（例如 `pale celadon-cyan and matte silver`，绝对不写十六进制代码，防止模型直接把 `#FFFFFF` 印在衣服上），以及统一的电影光学参数（`ARRI Alexa 65, 50mm anamorphic lens`）。
+2. **第二阶段（变量槽位）**：让大模型为 24 个镜头输出**原子级变量槽位**，杜绝重复默写常量：
+   ```json
+   {
+     "scene_id": 4,
+     "include_character": true,
+     "shot_type": "Close-up Side Profile",
+     "subject_side": "right",
+     "negative_space_medium": "warm faint candle reflection on blurred rosewood screen",
+     "t2i_scene_desc": "Looking downward in gentle contemplation, eyelashes catching rain light.",
+     "i2v_camera": "Slow cinematic dolly in",
+     "i2v_subject_motion": "Slowly blinks once, subtle chest breathing.",
+     "i2v_ambient_motion": "Candle glow flickers softly in out-of-focus bokeh."
+   }
+   ```
+3. **确定性代码组装**：由后端 Python 脚本（`scripts/assemble_prompts.py`）将 Stage 1 的常量与 Stage 2 的变量直接拼接。当 `include_character` 为 `false` 时（如远山雨景、屋檐铜铃等纯空镜），自动剥离所有人物词，从源头杜绝生图模型在风景里强塞人脸。
+
+---
+
+## 3. 图生视频 10 秒长镜头的运动控制准则
+
+将静态图片送入图生视频模型（Veo 2 / Kling / Runway）时，必须遵守严格的物理动力学规则：
+
+* **正向提示词绝对禁止否定词**：在正向 Prompt 中写 `no morphing`（不要变形）或 `no deformation`，扩散模型中的文本编码器对否定词 `no` 极不敏感，反而会激活 `morphing` 的权重导致肢体崩坏！正向必须使用正向稳定词：`consistent anatomy, temporal stability, rigid structure`；独立的负面词只放在 API 的 `negative_prompt` 参数中。
+* **120fps 极度慢动作锚点**：眨眼、转头等微动作在现实中只有 2 秒。如果强行让模型跑 10 秒，模型为了填满帧数，极易把脖子拧断或凭空多出一只手。在提示词句尾固定注入 `captured in 120fps extreme slow motion, subtle micro-movements`，将 2 秒的自然微动拉伸成凝滞而优美的 10 秒电影感镜头。
+* **运镜物理防撞锁**：我们的排版在画面左侧放置动态圆环频谱与歌词，主体人物位于右侧三分之一。如果此时运镜写了向右摇摄（`Pan Right` 或 `Truck Right`），画面中的少女在 10 秒内就会向左平移，到第 6 秒就会直接撞碎左侧的歌词！我们在组装脚本中增加了防撞硬规则：当人物在右侧时，向右运镜被程序强制拦截。
+
+---
+
+## 4. 彻底消除黑场：FFmpeg 恒定功率交叉叠化
+
+在桌面剪辑软件里拖拽两段视频做交叉溶解很简单，但在无人值守的代码管线里，常规做法有三个大坑：
+
+1. **`fadeblack` 造成频闪**：FFmpeg 常用的 `xfade=transition=fadeblack` 会让前一段画面变黑、再从黑场淡入下一段。24 个镜头拼接起来，观众每 10 秒就要忍受一次黑屏闪烁。
+2. **普通线性 `fade` 中间发暗**：在非线性的数字颜色空间中，线性加权在 50% 叠化处会产生高达 16%~20% 的感知变暗（Perceptual Dip）。
+3. **34.5 秒的时间轴缩水**：24 段 10 秒视频两两重叠 1.5 秒叠化，合成后的总时长是：
+   $$\text{总时长} = 24 \times 10.0 - (23 \times 1.5) = 205.5\text{ 秒}$$
+   比 240 秒的音频整整短了 34.5 秒！
+
+### 数学级精准对齐
+我们以**整数帧（Integer-Frame）**为基准预先计算单切片需求：
+$$F_{\text{clip}} = \left\lceil \frac{F_{\text{target}} + (N - 1) \times F_D}{N} \right\rceil = \left\lceil \frac{7200 + 23 \times 45}{24} \right\rceil = 344\text{ 帧}\ (11.4667\text{秒})$$
+
+在预处理时通过 `setpts=1.14667*PTS` 将单切片微慢放至 11.47 秒（微幅慢放反而大幅提升了 AI 视频的平稳质感），并使用自定义的恒定功率伽马曲线 `pow(P, 0.78)` 级联拼接：
+
+```bash
+ffmpeg -y -i v0.mp4 -i v1.mp4 ... \
+  -filter_complex "[0:v][1:v]xfade=transition=custom:duration=1.5:offset=9.9667:expr='clip(A*pow(P,0.78)+B*pow(1-P,0.78),0,255)'[xf1];..." \
+  -c:v libx264 -preset veryfast -crf 18 -g 30 -bf 0 dynamic_bg.mp4
+```
+
+通过 `-g 30 -bf 0` 强制输出 1 秒短 GOP，使得下游 Remotion 多线程渲染时抽帧速度提升 50%，彻底杜绝超时风险。
+
+---
+
+## 5. 防吃字双层排版与视觉闪避
+
+动态视频背景与发光的圆环动效叠加在一起，极易造成文字被背景画面吞吃。
+
+我们放弃了会严重拖慢渲染性能的实时高斯模糊遮罩（`backdrop-filter: blur` 会引发 GPU 显存回读），改用纯 SVG / DOM 双层叠印机制：
+1. **底层防吃字外铠甲**：4.2px 深色圆角外描边（`paint-order: stroke fill`）配合三重复合投影，即使底层是高亮的大漠天空，文字依然锐利立挺；
+2. **顶层卡拉OK柔光流光**：使用 `background-clip: text` 配合 12% 宽度的羽化渐变带，随演唱节奏从左向右平滑扫过；
+3. **人声期视觉闪避（Visual Ducking）**：有人声演唱时，圆环动效自动降低 50% 不透明度并将振幅压缩至 75%，确保人声与字幕占据第一视觉层级；间奏时全功率绽放。
+
+---
+
+## 6. 总结与落地思考
+
+制作一整首高质量的长视频 AI MV，真正的瓶颈早已不是单一模型的生成能力，而是**全流程工程编排与约束设计（Pipeline Orchestration）**。
+
+将角色设定抽象为不可变常量、在提示词层建立运镜防撞规则、并在底层音视频编解码上解决毫秒级时间轴与转场光学补偿，单台工作站即可将一首歌曲全自动转化为符合 YouTube 算法与观众留存率的高清工业级成片。
+
+---
+
+*如果您正在寻找企业级 AI 视频生成方案、自动化多媒体流水线定制、或想为自己的数字业务构建高吞吐量的音视频生成系统，欢迎随时联系交流：*
+
+- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
+- 📧 **邮件：** [[email protected]](mailto:[email protected]?subject=%E5%92%A8%E8%AF%A2%20AI%20MV%20%E8%87%AA%E5%8A%A8%E5%8C%96%E6%B5%81%E6%B0%B4%E7%BA%BF)
+- 🌐 **网站：** [hoelee.com](https://hoelee.com)