post: unraid array-stop swapfile hang root cause + User Scripts fix (en+zh, OG+banner)
Deploy / build (push) Successful in 1m1s

This commit is contained in:
2026-09-15 18:44:09 +08:00
parent 53f1ec5de7
commit 0dee1ea173
6 changed files with 474 additions and 0 deletions
Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

+18
View File
@@ -358,6 +358,24 @@ const BANNERS = {
{ n: '5', label: 'fixed ✓' },
],
},
'unraid-stop-array-hangs-on-swapfile': {
titlebar: 'root@unraid — array stop incident',
lines: [
{ t: 'prompt', text: '$' }, { t: 'cmd', text: 'WebUI → Stop array · swapfile lives on /mnt/cache (btrfs RAID1)' },
{ t: 'prompt', text: 'WARN' }, { t: 'err', text: 'Retry unmounting user shares… · umount: target is busy (every 5s, forever)' },
{ t: 'prompt', text: 'WARN' }, { t: 'err', text: '/proc/swaps lists /dev/loop0 — grep swapfile never matches' },
{ t: 'prompt', text: '$' }, { t: 'cmd', text: 'fix = User Scripts: swapoff -a + losetup -j/-d at stopping_svcs · swapon at disks_mounted' },
{ t: 'prompt', text: '' }, { t: 'ok', text: '→ clean unmount on first try · swap survives stop/start ✓' },
],
flow: [
{ n: '1', label: 'stop array' },
{ n: '2', label: 'EBUSY loop', err: true },
{ n: '3', label: 'losetup -j' },
{ n: '4', label: 'swapoff hook' },
{ n: '5', label: 'clean stop ✓' },
],
},
};
const DEFAULT_BANNER = {
+5
View File
@@ -123,6 +123,11 @@ const TERMINALS = {
<div class="line"><span class="prompt">$</span><span class="cmd">CDP → shopee search "used phone" · client marketplace monitor</span></div>
<div class="line"><span class="prompt">&nbsp;</span><span class="err">/verify captcha · empty product cards</span></div>
<div class="line"><span class="prompt">$</span><span class="cmd">warm session · 7s pacing · sweep.py</span><span class="fix">→ 20+ listings ✓</span></div>`,
'unraid-stop-array-hangs-on-swapfile': `
<div class="line"><span class="prompt">$</span><span class="cmd">unraid → stop array · swapfile on /mnt/cache</span></div>
<div class="line"><span class="prompt">&nbsp;</span><span class="err">umount: target is busy · /proc/swaps says /dev/loop0</span></div>
<div class="line"><span class="prompt">$</span><span class="cmd">swapoff -a + losetup -j at stopping_svcs</span><span class="fix">→ clean stop ✓</span></div>`,
};
const DEFAULT_TERMINAL = `
@@ -0,0 +1,232 @@
---
title: "Why Unraid Hangs When You Stop the Array With a Swapfile — and the Fix"
description: "My Unraid array stop hung forever at unmounting disks because an 8 GB btrfs swapfile pinned /mnt/cache. Root cause, the /boot/config/stop trap, and two User Scripts that fix it."
pubDate: 2026-09-15
category: devops
tags: [unraid, swap, btrfs, zram, docker, user-scripts]
ogImage: /og/unraid-stop-array-hangs-on-swapfile.png
banner: /banners/unraid-stop-array-hangs-on-swapfile.png
draft: false
---
I added an 8 GB swapfile to my Unraid server — my cache pool is a btrfs RAID1,
and the kernel refuses a swapfile that lives directly on a multi-device btrfs
filesystem, so it goes through a loop device, the standard workaround. Swap
worked. Memory pressure dropped. Everything was fine… until the day I stopped
the array.
The WebUI sat on **"Retry unmounting user shares…"** forever. The cache pool
would not release. Docker was stopped, VMs were shut down, shares were
unmounted — and still, `umount /mnt/cache` failed with *target is busy*,
every five seconds, indefinitely. The only way out was a reboot.
This is the story of finding out why, the trap I fell into on the way
(`/boot/config/stop` does **not** run when you click Stop), and the two-script
fix that survives every stop/start cycle since.
## The problem, phrased the way you'd Google it
If you landed here from a search, you're probably typing one of these:
- *"unraid stop array stuck retry unmounting user shares"*
- *"umount /mnt/cache target is busy unraid"*
- *"unraid swapfile stop array hang"*
- *"unraid won't stop array after adding swap"*
Same bug. The short version: **a swapfile — or its loop device — is held open
by the kernel's swap subsystem, and Unraid's array-stop sequence has no step
that turns swap off before it unmounts disks.** The mount is busy because the
kernel itself is using it, and it will keep using it until someone runs
`swapoff`. Nobody does.
## What I tried, and why it failed
### Attempt 1: reboot, stop array again — hung again
Obviously. The swapfile was still configured in `/boot/config/go`, so every
boot recreated the loop device and enabled swap on it. Every stop hit the same
wall. At least this confirmed it was deterministic, not a one-off stuck
process.
### Attempt 2: put a `swapoff` in `/boot/config/stop`
This is the classic trap, and the Unraid docs make it look like the right
answer. The `stop` script exists precisely for "run cleanup before disks go
away" — and it does run, **but only during a full shutdown or reboot**.
I checked on my own box (Unraid 7.1.4): `/boot/config/stop` is invoked by
`rc.local_shutdown`, which is called from `rc.6`. Clicking **Stop** in the
WebUI goes through a completely different path — `emhttpd` fires a series of
*events* — and `rc.6` is never involved. No shutdown, no `stop` script, no
`swapoff`. The array stop still hung.
### Attempt 3: read what the GUI Stop actually does
This is where the real answer lives. Unraid's event dispatcher (`emhttp_event`)
runs hook scripts in a documented sequence when the array stops:
```text
stopping → stopping_libvirt → stopping_docker → stopping_svcs
→ unmounting_disks → stopping_array → stopped
```
Two facts matter:
1. The unmount happens at `unmounting_disks`. **Any swapoff must fire at
`stopping_svcs` or earlier** — one event before the unmount.
2. These events are exactly what the **User Scripts** plugin exposes as
schedule options ("At Stopping of Array" = `stopping_svcs`, "At Startup of
the array" / "At First Array Start only" = `array_started` /
`disks_mounted`). No new code needed — the automation surface already
exists.
### The subtle part: `/proc/swaps` lies about loop-backed swap
Here's the gotcha that makes even a "correct-looking" cleanup script fail.
My `/boot/config/go` set swap up like this (the standard btrfs-RAID1 recipe):
```bash
# /boot/config/go — the original (broken-on-stop) setup
mkdir -p /mnt/cache
MOUNTPOINT=/mnt/cache
SWAPFILE=${MOUNTPOINT}/swapfile
# Wait for the cache pool to be mounted
while [ ! -f "${SWAPFILE}" ]; do
sleep 1
done
# Btrfs RAID1 rejects direct swapfiles: loop device + NOCOW
chattr +C "${SWAPFILE}"
LOOPDEV=$(losetup -f)
losetup -p 100 "${LOOPDEV}" "${SWAPFILE}"
swapon -p 1 "${LOOPDEV}"
```
Now look at what the kernel reports while that's active:
```console
$ cat /proc/swaps
Filename Type Size Used Priority
/dev/loop0 partition 8388604 0 -2
$ swapon --show
NAME TYPE SIZE USED PRIO
/dev/loop0 partition 8G 0B -2
```
**The filename is `/dev/loop0`, not `/mnt/cache/swapfile`.** So a cleanup
script like this one:
```bash
# WRONG — silently does nothing for loop-backed swap
grep -q '/mnt/cache/swapfile' /proc/swaps && swapoff /mnt/cache/swapfile
```
…matches nothing, swaps nothing off, and you're back to the hang. You have to
resolve the loop device *first*`losetup -j <file>` does exactly that — then
`swapoff` the loop device, then detach it. The ZRAM Compressed Memory plugin
documents this same bug in its source comment, almost word for word: *"for
loop-backed swap, /proc/swaps lists /dev/loopN, not the image path — a plain
grep never matches, so swapoff never ran and the loop device kept the pool
busy forever."*
## The fix: two User Scripts
[User Scripts](https://forums.unraid.net/topic/48286-plugin-ca-user-scripts/)
is already installed on most Unraid boxes. Create two scripts:
**Script 1 — "Array Stop Swapoff"** — schedule: **At Stopping of Array**
(fires at `stopping_svcs`, one event before the unmount):
```bash
#!/bin/bash
# Release ALL swap, then detach only the loop device backing OUR swapfile.
swapoff -a 2>/dev/null
for l in $(losetup -j /mnt/cache/swapfile 2>/dev/null | awk -F: '{print $1}'); do
losetup -d "$l" 2>/dev/null
done
```
`losetup -j` ("which loop device is attached to this file?") is the key move —
it's immune to the `/proc/swaps` naming lie. And detaching *only* our loop
device matters: a box can have other loop devices (mounted ISOs, other images)
that must survive an array stop.
**Script 2 — "Array Start Swapon"** — schedule: **At First Array Start only**
(fires at `disks_mounted`):
```bash
#!/bin/bash
# Re-create the loop-backed swap after the cache pool mounts.
[ -f /mnt/cache/swapfile ] || exit 0
LOOP=$(losetup -f)
losetup -p 100 "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP"
```
Why "First Array Start only" and not plain "At Startup of the Array"? Because
`/boot/config/go` also runs at boot and would set swap up a second time. The
*first* array start after boot is covered by `go`; this script matters for the
**GUI stop → start cycle with no reboot in between** — which `go` never sees.
(In my setup I moved the swap block out of `go` entirely so the User Script is
the single source of truth — one place, both paths.)
After this: Stop → cache unmounts cleanly on the first try. Start → swap is
back. Verified across multiple stop/start cycles and reboots.
## What I'd do differently
**1. Don't put swap on a disk that's already dying.** My cache pool is a pair
of NVMe drives I've since confirmed are silently corrupting data (reads
returning all-zero blocks despite clean SMART). An 8 GB swapfile there meant
constant write wear on hardware I was about to replace — and it was the very
thing pinching my array stops during the migration. Match swap location to
disk health, not convenience.
**2. Prefer compressed RAM over disk swap when you have the CPU for it.** The
endgame on this box was the [ZRAM Compressed
Memory](https://forums.unraid.net/topic/196763-new-plugin-created-zram/)
plugin: Tier-1 swap lives in a zstd-compressed RAM-backed block device. Two
wins:
- **The hang becomes structurally impossible.** A zram device is not a mount.
`umount /mnt/cache` can never see it, so there's nothing to release at stop
time. (The plugin *also* ships a proper `event/stopping` hook with the
`losetup -j` fix, if you enable its Tier-2 disk swapfile.)
- **Swap-in/out is microseconds of CPU work instead of disk I/O** — on a
modern many-core CPU, compression is nearly free relative to even NVMe
latency, and you save ~3:1 on the RAM the swapped pages occupy.
One caveat I learned the hard way: the plugin's auto-size (50% of RAM) gave me
a **31.3 GB** zram device on a 62 GB host that was already 90% full. That
number is an *uncompressed ceiling* — real RAM cost is capacity ÷ compression
ratio, so a full 31.3 GB zram could eat 1030 GB of the very RAM I was short
of. On a nearly-full host, size zram fixed and modest (I went to 8 GB, the old
swapfile size, swappiness 150) so it buffers OOM instead of competing with it.
**3. Swap delays OOM; it doesn't cure it.** The honest root cause of my memory
pressure was VM RAM budgets, not a missing swapfile. Swap bought headroom and
kept the host alive during spikes — but the real fix was right-sizing guests.
## The result
Array stop went from *"hangs forever, forced reboot"* to *clean unmount on the
first attempt*, and swap now survives stop/start cycles without manual
intervention. Total fix: two shell scripts in a plugin that was already
installed. The expensive part was knowing **where** in the event sequence to
put them — and that `/proc/swaps` won't tell you the truth about loop-backed
swap.
If your Unraid array refuses to stop and you have any swap configured — file,
loop, or otherwise — `swapoff -a` from a root shell will release it
immediately, and you've confirmed the diagnosis. Then automate it before the
next reboot wipes your memory of how.
---
*Run a self-hosted stack that needs this kind of debugging — Unraid, Docker,
reverse proxies, NAS migrations? I do this for a living: [website design &
development](https://hoelee.com) is my main work, and self-hosted
infrastructure is where I go deep. Reach me on
[WhatsApp](https://wa.me/60127972969) or
[email](mailto:[email protected]?subject=Unraid%20infrastructure%20help).*
@@ -0,0 +1,219 @@
---
title: "Unraid 停止阵列时卡死的真相:Swapfile 锁住挂载点 — 附修复方案"
description: "Unraid 停止阵列时一直卡在卸载磁盘:btrfs RAID1 上的 8GB swapfile 通过 loop 设备锁住 /mnt/cache。根因分析、/boot/config/stop 陷阱、两个 User Scripts 彻底修复。"
pubDate: 2026-09-15
category: devops
tags: [unraid, swap, btrfs, zram, docker, user-scripts]
ogImage: /og/unraid-stop-array-hangs-on-swapfile.png
banner: /banners/unraid-stop-array-hangs-on-swapfile.png
draft: false
---
我给 Unraid 服务器加了 8 GB swapfile。缓存池是 btrfs RAID1,而内核不允许
swapfile 直接建在多设备 btrfs 文件系统上,所以走了 loop 设备这个标准变通方
案。Swap 生效了,内存压力降了,一切正常——直到有一天我点了「停止阵列」。
WebUI 永远停在 **"Retry unmounting user shares…"**。缓存池就是卸不掉。
Docker 已停、虚拟机已关、共享已卸载——`umount /mnt/cache` 依然每 5 秒报一
*target is busy*,无限循环。唯一出路是重启。
这篇文章讲清楚三件事:为什么会这样;我中途踩的坑(`/boot/config/stop`
你点「停止」时**根本不会运行**);以及两个脚本的修复方案——之后每次停止/启
动都干净利落。
## 问题本身(按你会搜索的方式来描述)
如果你是搜索进来的,你大概率在搜:
- *"unraid stop array stuck retry unmounting user shares"*
- *"umount /mnt/cache target is busy unraid"*
- *"unraid swapfile stop array hang"*
- *"unraid 停止阵列 卡住"*
同一个 bug。一句话版本:**swapfile(或它的 loop 设备)被内核 swap 子系统
持有打开,而 Unraid 的停止阵列流程里没有任何一步会先关掉 swap 再卸载磁
盘。** 挂载点忙,是因为内核自己在用它——除非有人执行 `swapoff`,否则它会
一直用下去。而没有人执行。
## 我试过的方案,以及为什么都失败
### 尝试 1:重启后再停阵列——照样卡死
意料之中。swapfile 配置在 `/boot/config/go` 里,每次开机都会重建 loop 设
备并启用 swap,每次停止都撞同一堵墙。至少这证明了问题是确定性的,不是偶
发进程卡住。
### 尝试 2:在 `/boot/config/stop` 里写 `swapoff`
这是最经典的陷阱,Unraid 官方文档让它看起来就是正确答案。`stop` 脚本的存
在意义就是「磁盘卸载前跑清理」——它确实会运行,**但只在完整关机或重启
时**。
我在自己的机器上验证过(Unraid 7.1.4):`/boot/config/stop`
`rc.local_shutdown` 调用,而后者在 `rc.6` 里。在 WebUI 点**停止**走的是完
全不同的路径——`emhttpd` 触发一系列 *event*——`rc.6` 根本不参与。没有关
机,就没有 `stop` 脚本,就没有 `swapoff`。阵列停止照样卡死。
### 尝试 3:搞清楚 GUI「停止」到底做了什么
真正的答案在这里。Unraid 的事件调度器(`emhttp_event`)在阵列停止时按文
档化顺序执行钩子脚本:
```text
stopping → stopping_libvirt → stopping_docker → stopping_svcs
→ unmounting_disks → stopping_array → stopped
```
两个关键事实:
1. 卸载发生在 `unmounting_disks`。**任何 swapoff 必须在 `stopping_svcs`
或更早触发**——比卸载早一个事件。
2. 这些事件正是 **User Scripts** 插件暴露出来的调度选项(「At Stopping of
Array」= `stopping_svcs`,「At First Array Start only」=
`disks_mounted`)。不需要写新代码——自动化接口本来就在那里。
### 隐蔽的部分:`/proc/swaps` 对 loop swap 说谎
这是让「看起来正确」的清理脚本也会失败的坑。我的 `/boot/config/go` 原来
是这样配置 swap 的(btrfs RAID1 标准写法):
```bash
# /boot/config/go — 原始(停止时会卡的)配置
mkdir -p /mnt/cache
MOUNTPOINT=/mnt/cache
SWAPFILE=${MOUNTPOINT}/swapfile
# 等待缓存池挂载
while [ ! -f "${SWAPFILE}" ]; do
sleep 1
done
# btrfs RAID1 拒绝直接 swapfileloop 设备 + NOCOW
chattr +C "${SWAPFILE}"
LOOPDEV=$(losetup -f)
losetup -p 100 "${LOOPDEV}" "${SWAPFILE}"
swapon -p 1 "${LOOPDEV}"
```
现在看内核在活动状态下报告的内容:
```console
$ cat /proc/swaps
Filename Type Size Used Priority
/dev/loop0 partition 8388604 0 -2
$ swapon --show
NAME TYPE SIZE USED PRIO
/dev/loop0 partition 8G 0B -2
```
**文件名是 `/dev/loop0`,不是 `/mnt/cache/swapfile`。** 所以这样的清理脚
本:
```bash
# 错误 — 对 loop swap 静默失效
grep -q '/mnt/cache/swapfile' /proc/swaps && swapoff /mnt/cache/swapfile
```
……什么都匹配不到,什么都关不掉,然后你又一次卡在停止阵列。必须*先*解析
loop 设备——`losetup -j <文件>` 就是干这个的——再 `swapoff` 那个 loop 设
备,最后卸载它。ZRAM Compressed Memory 插件的源码注释几乎一字不差地记录
了同一个 bug*"for loop-backed swap, /proc/swaps lists /dev/loopN, not
the image path — a plain grep never matches, so swapoff never ran and the
loop device kept the pool busy forever."*loop 型 swap 在 /proc/swaps 里显
示为 /dev/loopN 而不是镜像路径——普通 grep 永远匹配不到,swapoff 从未执
行,loop 设备让缓存池永远忙碌。)
## 修复方案:两个 User Scripts
大多数 Unraid 机器已经装了 [User
Scripts](https://forums.unraid.net/topic/48286-plugin-ca-user-scripts/)
插件。创建两个脚本:
**脚本 1 —「Array Stop Swapoff」**——调度选 **At Stopping of Array**(在
`stopping_svcs` 触发,比卸载早一个事件):
```bash
#!/bin/bash
# 先释放所有 swap,再只卸载【我们的】swapfile 背后的 loop 设备。
swapoff -a 2>/dev/null
for l in $(losetup -j /mnt/cache/swapfile 2>/dev/null | awk -F: '{print $1}'); do
losetup -d "$l" 2>/dev/null
done
```
`losetup -j`(「哪个 loop 设备挂在这个文件上?」)是关键一步——它不受
`/proc/swaps` 命名谎言的影响。而且只卸载*我们自己的* loop 设备很重要:机
器上可能还有别的 loop 设备(挂载的 ISO、其他镜像),它们必须在停止阵列时
存活下来。
**脚本 2 —「Array Start Swapon」**——调度选 **At First Array Start
only**(在 `disks_mounted` 触发):
```bash
#!/bin/bash
# 缓存池挂载后重建 loop swap。
[ -f /mnt/cache/swapfile ] || exit 0
LOOP=$(losetup -f)
losetup -p 100 "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP"
```
为什么选「First Array Start only」而不是普通的「At Startup of the
Array」?因为开机时 `/boot/config/go` 也会运行并配置一次 swap。开机后的
*第一次*阵列启动由 `go` 负责;这个脚本真正管的是**不重启、纯 GUI 停止 →
启动的循环**——而 `go` 永远看不到这种场景。(在我的配置里,我把 swap 那段
`go` 里整个移除了,让 User Script 成为唯一事实来源——一处配置,两条路
径都覆盖。)
之后:停止 → 缓存池第一次尝试就干净卸载。启动 → swap 恢复。多次停止/启动
循环和重启后都验证通过。
## 我会做得不一样的地方
**1. 不要把 swap 放在已经快死的盘上。** 我的缓存池是一对后来确认在静默损
坏数据的 NVMe(SMART 全绿,读取却返回全零块)。8 GB swapfile 放在那里意
味着对我即将更换的硬件持续写入磨损——而且它正是在迁移期间卡住我阵列停止
的元凶。swap 的位置要匹配磁盘健康状况,而不是图方便。
**2. CPU 够用时,压缩内存 swap 优于磁盘 swap。** 这台机器的最终方案是
[ZRAM Compressed
Memory](https://forums.unraid.net/topic/196763-new-plugin-created-zram/)
插件:Tier-1 swap 放在 zstd 压缩的 RAM 块设备里。两个好处:
- **卡死在结构上变得不可能。** zram 设备不是挂载点,`umount /mnt/cache`
永远看不见它,停止时无需释放任何东西。(如果你启用它的 Tier-2 磁盘
swapfile,插件也自带正确的 `event/stopping` 钩子,用的正是
`losetup -j` 这个修复。)
- **swap 换入换出是微秒级 CPU 操作而不是磁盘 I/O**——在现代多核 CPU 上,
压缩相对 NVMe 延迟几乎免费,而且被换出的页面在 RAM 里省约 3:1 空间。
一个我用惨痛教训学到的注意事项:插件的自动大小(RAM 的 50%)在我 62 GB
内存、已用 90% 的机器上给了一个 **31.3 GB** 的 zram 设备。那个数字是*未压
缩上限*——实际 RAM 消耗 = 容量 ÷ 压缩比,所以装满的 31.3 GB zram 可能吃掉
10–30 GB 我本来就稀缺的空闲内存。在接近满载的机器上,zram 要固定且保守地
设置大小(我改成 8 GB——原来 swapfile 的大小——swappiness 150),让它缓冲
OOM 而不是和 OOM 抢内存。
**3. swap 只能推迟 OOM,治不了 OOM。** 我内存压力的诚实根因是虚拟机内存
分配,不是缺 swapfile。swap 买来了余量、让宿主机在峰值时活了下来——但真正
的修复是给虚拟机合理分配内存。
## 结果
停止阵列从「永远卡死、强制重启」变成「第一次尝试就干净卸载」,swap 在停止/
启动循环之间自动存活,无需人工干预。修复总量:两个 shell 脚本,跑在一个早
就装好的插件里。贵的部分是知道该把它们放在事件序列的**哪个位置**——以及
`/proc/swaps` 不会告诉你 loop swap 的真相。
如果你的 Unraid 阵列拒绝停止,而你配置过任何形式的 swap——文件、loop 或其
他——在 root shell 里执行 `swapoff -a` 会立刻释放它,诊断就此确认。然后在
下次重启抹掉你的记忆之前,把它自动化。
---
*你也有需要这种深度调试的自托管环境吗——Unraid、Docker、反向代理、NAS 迁
移?这是我的本行:[网站设计与开发](https://hoelee.com)是主业,自托管基础
设施是我钻研最深的领域。欢迎通过
[WhatsApp](https://wa.me/60127972969) 或
[邮件](mailto:[email protected]?subject=Unraid%20infrastructure%20help)联系
我。*