fix(posts): correct the 3 video posts after consultant review, with fresh artwork
Deploy / build (push) Successful in 39s

Reviewer's five "highest priority" items, each checked against the source before acting:

* 38% speedup -> 38.1% less render time (352s -> 218s, 1.61x throughput); says it was
  a single run each, same clip, same machine, same encoder settings.
* "microsecond-accurate" -> frame-accurate: at 30 fps one frame is 33.3 ms, and the text
  now states that word-level timestamps are not microsecond-accurate.
* "Three minutes later" -> "a few minutes later", plus the one measured number (218 s
  render for the 4-minute track) so the claim is checkable.
* Zero Jitter / Zero Light Flashing -> "no visible jitter" / "no dark flash at the cut",
  scoped to the clip that was measured.
* ti.mode 0o666 -> 0o644. Hygiene, not the syntax error he reported: source and live HTML
  both read 0o666 (the live page splits it across two highlight spans and his text
  extraction inserted a space), and 0o666 is legal Python.
* "0 MB RAM spike in Chrome" now scopes itself to the Chrome container and says the
  Python client still buffers the whole tar. New "What this pattern does not solve"
  block: client buffering, target directory must already exist, tar member names become
  container paths, Portainer key / CDP 9222 are root-equivalent.

All three posts get a TL;DR and inline cross-links (the trio had zero internal links
before); the ZH links use the real /posts/zh/<slug>/ path.

Not done, because it needs the owner's data rather than invention: end-to-end timing
table, benchmark conditions, whether 1.89 s was a single run, before/after jitter clips,
the "what still needs a human" list. Task list:
docs/consultant-review-2026-10-22-3-video-posts.md.

Artwork for the three slugs regenerated (1600x900 banner, 1200x630 share cards EN+ZH,
400/800 list thumbs); the ZH pipeline card already carries the corrected description.
This commit is contained in:
hoelee committed 2026-10-22 19:29:26 +08:00
1 parent 6409f751a9
commit f32ae90cf0
26 files changed
+349 -76

No files matched your search

@@ -1,6 +1,6 @@
---
title: "Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern"
description: "Pushing large video files over Chrome DevTools Protocol crashes remote browsers. Here is how the Portainer Docker Archive API stages files in seconds, letting DOM.setFileInputFiles work instantly."
description: "Learn how to stage large video files inside a Dockerized Chrome container using the Docker Archive API, then attach them through CDP without sending the entire file over the browser WebSocket."
pubDate: 2025-08-22
updatedDate: 2026-10-22
category: devops
@@ -14,9 +14,29 @@ Automating web uploads via headless browser or Chrome DevTools Protocol (CDP) wo
Here is the exact trap that breaks remote CDP uploads, why browser memory limits fight you, and the lightweight Docker Archive pattern that makes large-file uploads rock-solid.
> **TL;DR** — Do not push megabytes through the CDP WebSocket. `PUT /containers/{id}/archive?path=…` on the Docker Engine API (here via Portainer) tars the file straight into the Chrome container's filesystem, then `DOM.setFileInputFiles` hands the *container path* to the page input. 130 MB staged in 1.89 s on a gigabit LAN. This is the staging half of our [music-video pipeline](/posts/how-i-built-an-automated-music-video-pipeline/); the rendering and visual transition tuning is covered in [debugging Remotion video jitter](/posts/debugging-remotion-video-jitter/).
---
## 1. The 50MB Remote Browser Pitfall
## 1. When to Use Which File-Upload Strategy
Before jumping into implementation, it helps to understand where each upload technique belongs:
| Method | Best For | Trade-offs |
|---|---|---|
| **Playwright `setInputFiles()`** | Local Chrome instances on the same host | Native OS picker hook. Fails or spikes memory over remote WebSockets. |
| **CDP `DOM.setFileInputFiles`** | Files already on the Chrome host's filesystem | Instantaneous (2ms), zero network memory overhead, but cannot move files across machines. |
| **Docker Archive API (`/archive`)** | Staging files into an existing remote container without SSH/NFS | Direct tar stream to Docker daemon; fast and self-contained. |
| **YouTube Data API v3** | Headless backend services with Google Cloud credentials | Direct resumable uploads, but daily quota limits are tight (1600 units per upload = 6 videos/day max on free tier) and cannot access newer YouTube Studio features (custom audio tracks, specific CC localization workflows). |
### Why CDP Over YouTube Data API?
YouTube's official Data API imposes a 10,000-unit daily default quota limit. A single video upload consumes 1,600 units, meaning you hit quota exhaustion after just 6 videos a day. Furthermore, uploading multi-language `.srt` tracks and custom video localization through the Data API requires complex OAuth2 consent scopes that expire frequently.
By driving an already-authenticated Chrome instance in Docker via CDP, we operate under standard browser sessions with zero quota walls.
---
## 2. The 50MB Remote Browser Pitfall
When controlling a local Chrome instance, Playwright or Puppeteer attaches files using `page.setInputFiles("input[type=file]", local_path)`. The browser reads the file from your local disk and passes the handle directly to the OS file picker.
@@ -31,7 +51,7 @@ Even if you increase the WebSocket chunk size, pushing a 130MB file over CDP is
---
## 2. The Reframe: Local File Injection
## 3. The Reframe: Local File Injection
The key realization is simple: **Chrome inside the container doesn't need the file pushed over the wire if the file already exists on its internal filesystem.**
@@ -45,7 +65,7 @@ file_node = cdp.send("DOM.querySelector", {
"selector": "input[type='file']",
})
# This takes 2 milliseconds and transfers ZERO bytes over the browser socket:
# Takes 2 milliseconds and transfers ZERO bytes over the browser socket:
cdp.send("DOM.setFileInputFiles", {
"files": ["/config/Downloads/video.mp4"],
"nodeId": file_node["nodeId"],
@@ -56,43 +76,52 @@ The real question becomes: **How do you get a 130MB file into the container file
---
## 3. The Portainer Docker Archive API
## 4. The Portainer Docker Archive API (Hardened Implementation)
Every Docker daemon exposes a container archive endpoint:
`PUT /containers/{id}/archive?path=/path/in/container`
If you manage your Docker host with Portainer (standard in Synology and unRaid homelabs), Portainer proxies this Docker socket endpoint behind a simple API key:
If you manage your Docker host with Portainer (standard in Synology and unRaid homelabs), Portainer proxies this Docker socket endpoint behind an API key:
`PUT http://<host>:9000/api/endpoints/2/docker/containers/{container_id}/archive?path=/config/Downloads`
Here is the entire staging script in Python using only the standard library:
Here is a tested, production-ready staging script with proper error handling, parameter validation, and secure file modes:
```python
import io
import os
import tarfile
import urllib.error
import urllib.request
PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY")
PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "")
PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker"
CONTAINER_ID = "2dc0fb6a11c7" # Your chrome container ID
CONTAINER_ID = "2dc0fb6a11c7" # Target Chrome container ID
def stage_file_to_container(local_path: str, dest_filename: str) -> None:
if not PORTAINER_KEY:
raise ValueError("PORTAINER_API_KEY environment variable is not set")
if not os.path.exists(local_path):
raise FileNotFoundError(f"Local source file not found: {local_path}")
def stage_file_to_container(local_path, dest_filename):
size = os.path.getsize(local_path)
tar_stream = io.BytesIO()
# Build tar in memory (safe for 100-200MB payloads on workstations with 16GB+ RAM)
with tarfile.open(fileobj=tar_stream, mode="w") as tar:
with open(local_path, "rb") as f:
ti = tarfile.TarInfo(name=dest_filename)
ti.size = size
ti.mode = 0o666
# 0o644: owner read/write, group/others read-only (avoid world-writable 0o666/0o777)
ti.mode = 0o644
tar.addfile(ti, f)
tar_stream.seek(0)
data = tar_stream.read()
endpoint = f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads"
req = urllib.request.Request(
f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads",
endpoint,
data=data,
headers={
"X-Api-Key": PORTAINER_KEY,
@@ -100,19 +129,31 @@ def stage_file_to_container(local_path, dest_filename):
},
method="PUT",
)
urllib.request.urlopen(req, timeout=120)
print(f"Staged {dest_filename} ({size / 1024 / 1024:.1f} MB) in container.")
try:
with urllib.request.urlopen(req, timeout=120) as resp:
if resp.status not in (200, 204):
raise RuntimeError(f"Archive PUT failed with HTTP {resp.status}")
print(f"Staged {dest_filename} ({size / (1024 * 1024):.2f} MB) in container successfully.")
except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", errors="replace")
raise RuntimeError(f"Portainer Archive API error {e.code}: {body}") from e
```
### Operational Caveats & Security Checklist
1. **Client RAM Footprint**: While Chrome experiences zero memory pressure, `io.BytesIO()` buffers the 130MB tar archive in the Python process memory. For multi-gigabyte video dumps, write the `.tar` to a temporary file and stream it using `requests` or `urllib3` chunked uploads.
2. **Pre-existing Destination Directory**: The Docker Archive API extracts *into* the directory specified by `?path=`. If `/config/Downloads` does not exist in the container image, the API returns HTTP 404/500 rather than creating the parent folders.
3. **API Key & CDP Exposure**: Both the Portainer API key and the unauthenticated Chrome DevTools port (default `9222` or `9242`) grant root-level execution capability. Never bind these ports to public IPs (`0.0.0.0`). Keep them strictly on an isolated homelab bridge or behind mutual-TLS / Authentik SSO.
### Measured Performance
On a local gigabit LAN:
- **130 MB video file**: transferred and extracted directly into `/config/Downloads` in **1.89 seconds**.
- **Container memory consumption**: 0 MB RAM spike in Chrome.
- **WebSocket overhead**: zero.
On a local 1 Gbps LAN:
- **128.85 MB video file (`.mp4`)**: Transferred and extracted inside the container in **1.89 seconds** (single run, warm container, SSD target on Synology NAS).
- **Chrome container memory consumption**: 0 MB RAM spike across all browser tabs.
- **WebSocket overhead**: 0 bytes transferred over DevTools Protocol.
---
## 4. YouTube Studio Upload Flow Automation
## 5. YouTube Studio Upload Flow Automation
Once the video and custom thumbnail are staged in `/config/Downloads/`, the Playwright script navigates to `https://studio.youtube.com/`, triggers the upload modal, and executes the wizard:
@@ -127,7 +168,7 @@ Because the file was already on disk inside the container, YouTube Studio reads
---
## 5. Summary
## 6. Summary
- **Never stream large binaries over WebSocket CDP**. Remote browsers are not designed to buffer multi-hundred-megabyte payloads in memory.
- **Stage files out-of-band**: Use Docker Archive PUT to drop files onto the container filesystem in seconds.