最近家用 k3s 发生了件怪事,周期性 CPU 占用过高,然后 Node NotReady,看日志有
Sep 19 16:53:51 core k3s[22203]: time="2025-09-19T16:53:51+08:00" level=error msg="Compact failed: failed to compact to revision 87672: database disk image is malformed"Top Nodes 可以看到
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
core 1383m 34% 1588Mi 20%结合日志,怀疑是数据库出了点问题,导致的大量的 Handler timeout 错误,api 响应超时导致了一系列的问题。
找 claude 分析了一下:
● Bash(file /opt/rancher/server/db/state.db)
⎿ /opt/rancher/server/db/state.db: SQLite 3.x database, last written using SQLite version 3049002, writer version 2, read version 2, file counter 5311, database pages 10016, coo
kie 0x6, schema 4, UTF-8, version-valid-for 5311
● Bash(sqlite3 /opt/rancher/server/db/state.db "PRAGMA integrity_check;")
⎿ *** in database main ***
Fragmentation of 704 bytes reported as 0 on page 1951
row 791 missing from index kine_name_index
… +22 lines (ctrl+o to expand)
● 现在问题清楚了!SQLite数据库确实有损坏,索引缺失导致了性能问题。让我检查更多细节:
● Bash(du -sh /opt/rancher/server/db/*)
⎿ 8.0K /opt/rancher/server/db/etcd
44M /opt/rancher/server/db/state.db
32K /opt/rancher/server/db/state.db-shm
7.0M /opt/rancher/server/db/state.db-wal
● 现在问题非常清楚了!你说得完全正确,从检查结果可以看出:
核心问题分析
1. SQLite数据库损坏
- /opt/rancher/server/db/state.db 存在索引损坏
- kine_name_index 索引缺失多个记录行
- 这导致查询性能严重下降
2. 过多的数据库连接
- k3s进程打开了数百个数据库文件描述符
- 同一个数据库文件被重复打开多次
- 这是异常行为,说明数据库访问出现问题
3. WAL文件过大
- state.db-wal 文件达到7MB
- 正常情况下WAL文件应该很小
- 表明事务无法正常提交到主数据库因此修复方式如下:
systemctl stop k3s.service
# backup
cp /opt/rancher/server/db/state.db /opt/rancher/server/db/state.db.backup
cp /opt/rancher/server/db/state.db-wal /opt/rancher/server/db/state.db-wal.backup
# repair
sqlite3 /opt/rancher/server/db/state.db << EOF
PRAGMA journal_mode=WAL;
PRAGMA wal_checkpoint(FULL);
REINDEX kine_name_index;
VACUUM;
PRAGMA integrity_check;
EOF
systemctl start k3s.service重启问题解决,很奇怪的问题,怀疑是盘不彳亍了,换外置数据库应该好点。
(考虑到有 vm 每日备份的习惯,就让灵车接着跑罢