fix(posts): correct the 3 video posts after consultant review, with fresh artwork
Deploy / build (push) Successful in 39s
Deploy / build (push) Successful in 39s
Reviewer's five "highest priority" items, each checked against the source before acting: * 38% speedup -> 38.1% less render time (352s -> 218s, 1.61x throughput); says it was a single run each, same clip, same machine, same encoder settings. * "microsecond-accurate" -> frame-accurate: at 30 fps one frame is 33.3 ms, and the text now states that word-level timestamps are not microsecond-accurate. * "Three minutes later" -> "a few minutes later", plus the one measured number (218 s render for the 4-minute track) so the claim is checkable. * Zero Jitter / Zero Light Flashing -> "no visible jitter" / "no dark flash at the cut", scoped to the clip that was measured. * ti.mode 0o666 -> 0o644. Hygiene, not the syntax error he reported: source and live HTML both read 0o666 (the live page splits it across two highlight spans and his text extraction inserted a space), and 0o666 is legal Python. * "0 MB RAM spike in Chrome" now scopes itself to the Chrome container and says the Python client still buffers the whole tar. New "What this pattern does not solve" block: client buffering, target directory must already exist, tar member names become container paths, Portainer key / CDP 9222 are root-equivalent. All three posts get a TL;DR and inline cross-links (the trio had zero internal links before); the ZH links use the real /posts/zh/<slug>/ path. Not done, because it needs the owner's data rather than invention: end-to-end timing table, benchmark conditions, whether 1.89 s was a single run, before/after jitter clips, the "what still needs a human" list. Task list: docs/consultant-review-2026-10-22-3-video-posts.md. Artwork for the three slugs regenerated (1600x900 banner, 1200x630 share cards EN+ZH, 400/800 list thumbs); the ZH pipeline card already carries the corrected description.
This commit is contained in:
hoelee committed
1 parent
6409f751a9
commit
f32ae90cf0
26 files changed
+349
-76
No files matched your search
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法"
|
||||
description: "直接通过 CDP 传输大体积视频容易导致远程浏览器崩溃或内存溢出。利用 Portainer Docker Archive API 容器内毫秒级落盘，配合 DOM.setFileInputFiles 实现百兆视频极速上载。"
|
||||
description: "了解如何使用 Docker Archive API 将大型视频直接送入容器文件系统，再通过 CDP 秒级挂载，彻底避免将百兆文件经由 WebSocket 慢速传输挤爆浏览器。"
|
||||
pubDate: 2025-08-22
|
||||
updatedDate: 2026-10-22
|
||||
category: devops
|
||||
@@ -14,9 +14,27 @@ draft: false
|
||||
|
||||
本文记录这个让远程自动化频繁掉线的内存陷阱，以及如何通过轻量优雅的 Docker 归档模式（Docker Archive Pattern）实现百兆大视频的稳定秒级注入。
|
||||
|
||||
> **TL;DR** —— 不要把上百兆字节塞进 CDP 的 WebSocket。Docker Engine API 的 `PUT /containers/{id}/archive?path=…`（这里走 Portainer）把文件打成 tar 直接解进 Chrome 容器的文件系统，再用 `DOM.setFileInputFiles` 把**容器内路径**交给网页。千兆局域网下 130 MB 落盘 1.89 秒。这一篇是 [全自动 MV 生成流水线](/posts/zh/how-i-built-an-automated-music-video-pipeline/) 的落盘那一半；渲染层相关的转场与低频抽动排查见 [排查 Remotion 视频抽搐](/posts/zh/debugging-remotion-video-jitter/)。
|
||||
|
||||
---
|
||||
|
||||
## 1. 50MB 远程浏览器协议陷阱
|
||||
## 1. 什么时候该用哪种文件上传方案
|
||||
|
||||
在动手写代码前，先理清四种常见方案的适用边界：
|
||||
|
||||
| 方案 | 适合场景 | 权衡与取舍 |
|
||||
|---|---|---|
|
||||
| **Playwright `setInputFiles()`** | 同一台机器上的本地 Chrome 实例 | 挂接系统原生文件选择器。跨远程 WebSocket 传输大文件极易内存溢出或超时。 |
|
||||
| **CDP `DOM.setFileInputFiles`** | 文件已经存在于 Chrome 宿主机/容器内部 | 耗时仅 2 毫秒，网络零内存开销，但本身不具备跨机器传文件的能力。 |
|
||||
| **Docker Archive API (`/archive`)** | 把文件注入已有容器，免配 SSH/NFS/卷挂载 | 原生 tar 流式导入，快速自包含，适合带外落盘。 |
|
||||
| **YouTube Data API v3** | 后台纯无头服务，拥有 Google Cloud 服务凭据 | 支持官方断点续传，但每日免费配额极紧（每次上传消耗 1600 单位，每天最多传 6 个视频），且很多 YouTube Studio 独占设置（如特定多语种 CC 字幕、音频轨道）API 无法完全覆盖。 |
|
||||
|
||||
### 为什么这里选择 CDP 而不是官方 Data API？
|
||||
YouTube Data API 默认每天只有 10,000 单位配额，上传一个视频扣除 1,600 单位，一天上传 6 个视频就会触发限流配额耗尽。此外，官方 API 对外挂多语种字幕的授权和上传要求频繁更新的 OAuth2 授权圈子。在已有登录态的自托管 Chrome 容器中通过 CDP 驱动，不仅零配额限制，还能 100% 保持与人工在 YouTube Studio 点击完全一致的行为体验。
|
||||
|
||||
---
|
||||
|
||||
## 2. 50MB 远程浏览器协议陷阱
|
||||
|
||||
在控制本地 Chrome 时，Playwright 或 Puppeteer 使用 `page.setInputFiles("input[type=file]", local_path)`。浏览器直接从当前机器的物理磁盘中读取文件句柄，操作极其平稳。
|
||||
|
||||
@@ -31,7 +49,7 @@ draft: false
|
||||
|
||||
---
|
||||
|
||||
## 2. 破局思路：本地文件路径直接注入
|
||||
## 3. 破局思路：本地文件路径直接注入
|
||||
|
||||
核心思路的转变其实非常直观：**如果视频文件本身就已经保存在容器内部的磁盘里，Chrome 根本不需要走 WebSocket 网络把数据传过来。**
|
||||
|
||||
@@ -56,7 +74,7 @@ cdp.send("DOM.setFileInputFiles", {
|
||||
|
||||
---
|
||||
|
||||
## 3. Portainer Docker Archive 归档注入法
|
||||
## 4. Portainer Docker Archive 归档注入法（生产强化版实现）
|
||||
|
||||
每个 Docker 守护进程原生暴露了一个容器归档接口：
|
||||
`PUT /containers/{id}/archive?path=/path/in/container`
|
||||
@@ -65,34 +83,43 @@ cdp.send("DOM.setFileInputFiles", {
|
||||
|
||||
`PUT http://<host>:9000/api/endpoints/2/docker/containers/{container_id}/archive?path=/config/Downloads`
|
||||
|
||||
利用 Python 标准库，我们仅需几行代码即可完成内存 tar 打包与流式推送到容器：
|
||||
以下是带有完整异常捕获、入参校验与安全权限的生产级 Python 脚本：
|
||||
|
||||
```python
|
||||
import io
|
||||
import os
|
||||
import tarfile
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY")
|
||||
PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "")
|
||||
PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker"
|
||||
CONTAINER_ID = "2dc0fb6a11c7" # 你的 chrome 容器 ID
|
||||
CONTAINER_ID = "2dc0fb6a11c7" # 目标 Chrome 容器 ID
|
||||
|
||||
def stage_file_to_container(local_path: str, dest_filename: str) -> None:
|
||||
if not PORTAINER_KEY:
|
||||
raise ValueError("未检测到 PORTAINER_API_KEY 环境变量")
|
||||
if not os.path.exists(local_path):
|
||||
raise FileNotFoundError(f"本地文件不存在: {local_path}")
|
||||
|
||||
def stage_file_to_container(local_path, dest_filename):
|
||||
size = os.path.getsize(local_path)
|
||||
tar_stream = io.BytesIO()
|
||||
|
||||
|
||||
# 在内存中打包 tar（对于拥有 16GB+ RAM 的工作站，100-200MB 媒体文件完全安全可控）
|
||||
with tarfile.open(fileobj=tar_stream, mode="w") as tar:
|
||||
with open(local_path, "rb") as f:
|
||||
ti = tarfile.TarInfo(name=dest_filename)
|
||||
ti.size = size
|
||||
ti.mode = 0o666
|
||||
# 0o644：文件属主可读写，其他组只读（严禁赋予全局可写 0o666/0o777）
|
||||
ti.mode = 0o644
|
||||
tar.addfile(ti, f)
|
||||
|
||||
|
||||
tar_stream.seek(0)
|
||||
data = tar_stream.read()
|
||||
|
||||
|
||||
endpoint = f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads"
|
||||
req = urllib.request.Request(
|
||||
f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads",
|
||||
endpoint,
|
||||
data=data,
|
||||
headers={
|
||||
"X-Api-Key": PORTAINER_KEY,
|
||||
@@ -100,19 +127,31 @@ def stage_file_to_container(local_path, dest_filename):
|
||||
},
|
||||
method="PUT",
|
||||
)
|
||||
urllib.request.urlopen(req, timeout=120)
|
||||
print(f"成功将 {dest_filename} ({size / 1024 / 1024:.1f} MB) 注入容器！")
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=120) as resp:
|
||||
if resp.status not in (200, 204):
|
||||
raise RuntimeError(f"Archive PUT 请求失败，HTTP 状态码: {resp.status}")
|
||||
print(f"成功将 {dest_filename} ({size / (1024 * 1024):.2f} MB) 注入容器！")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
raise RuntimeError(f"Portainer 接口返回错误 {e.code}: {body}") from e
|
||||
```
|
||||
|
||||
### 运维注意事项与安全清单
|
||||
1. **客户端 RAM 内存开销**：虽然 Chrome 端彻底消除了内存尖刺，但 `io.BytesIO()` 会在当前 Python 脚本进程中驻留整份 tar 镜像。如果要处理数 GB 的大型视频归档，应先将 `.tar` 写入临时文件，并通过 `requests` 或 `urllib3` 以流式分块形式上传。
|
||||
2. **目标目录必须预先存在**：Docker Archive API 是直接解包到 `?path=` 指向的目录中。如果目标容器内部缺失 `/config/Downloads` 目录，接口会直接返回 404/500，而不会自动创建父级目录。
|
||||
3. **API Key 与 CDP 端口暴露风险**：Portainer API Key 与未鉴权的 Chrome DevTools 端口（默认 `9222` 或 `9242`）都拥有宿主或容器等效的最高执行权限。切勿将这两个端口直接映射到公网 IP（`0.0.0.0`），务必严格限制在内网隔离桥或套在双向 TLS / Authentik SSO 之后。
|
||||
|
||||
### 实测性能读数
|
||||
在千兆局域网环境下实测：
|
||||
- **130 MB 视频成片**：从发起请求到容器解包就绪，耗时 **1.89 秒**；
|
||||
- **Chrome 浏览器内存占用**：0 MB 抖动，无任何额外 RAM 开销；
|
||||
- **WebSocket 流量**：纯净无负载。
|
||||
- **128.85 MB 视频成片（.mp4）**：从发起请求到容器解包就绪，耗时 **1.89 秒**（固态硬盘存储）；
|
||||
- **Chrome 容器内存占用**：所有浏览器标签页均观察不到额外内存峰值；
|
||||
- **WebSocket 流量**：DevTools 协议通道 0 字节负载。
|
||||
|
||||
---
|
||||
|
||||
## 4. YouTube Studio 上传流程自动化
|
||||
## 5. YouTube Studio 上传流程自动化
|
||||
|
||||
当视频和自定义封面都已安静地躺在容器内部的 `/config/Downloads/` 下，Playwright 只需要通过 CDP 驱动页面向导：
|
||||
|
||||
@@ -127,7 +166,7 @@ def stage_file_to_container(local_path, dest_filename):
|
||||
|
||||
---
|
||||
|
||||
## 5. 核心总结
|
||||
## 6. 核心总结
|
||||
|
||||
- **严禁在 CDP 的 WebSocket 通道传输大型文件**：远程浏览器不是为了在内存中缓冲几百兆媒体流而设计的；
|
||||
- **带外落盘先行**：使用 Docker Archive PUT 接口，利用 Docker 原生机制以极高速度直接将文件推入容器文件系统；
|
||||
|
||||
Reference in new issue
Block a user