最近家用 k3s 发生了件怪事,周期性 CPU 占用过高,然后 Node NotReady,看日志有

Sep 19 16:53:51 core k3s[22203]: time="2025-09-19T16:53:51+08:00" level=error msg="Compact failed: failed to compact to revision 87672: database disk image is malformed"

Top Nodes 可以看到

NAME   CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
core   1383m        34%      1588Mi          20%

结合日志,怀疑是数据库出了点问题,导致的大量的 Handler timeout 错误,api 响应超时导致了一系列的问题。

找 claude 分析了一下:

● Bash(file /opt/rancher/server/db/state.db)
  ⎿ /opt/rancher/server/db/state.db: SQLite 3.x database, last written using SQLite version 3049002, writer version 2, read version 2, file counter 5311, database pages 10016, coo
    kie 0x6, schema 4, UTF-8, version-valid-for 5311


● Bash(sqlite3 /opt/rancher/server/db/state.db "PRAGMA integrity_check;")
  ⎿  *** in database main ***
     Fragmentation of 704 bytes reported as 0 on page 1951
     row 791 missing from index kine_name_index
     … +22 lines (ctrl+o to expand)

● 现在问题清楚了!SQLite数据库确实有损坏,索引缺失导致了性能问题。让我检查更多细节:

● Bash(du -sh /opt/rancher/server/db/*)
  ⎿  8.0K       /opt/rancher/server/db/etcd
     44M        /opt/rancher/server/db/state.db
     32K        /opt/rancher/server/db/state.db-shm
     7.0M       /opt/rancher/server/db/state.db-wal


● 现在问题非常清楚了!你说得完全正确,从检查结果可以看出:

  核心问题分析

  1. SQLite数据库损坏
  - /opt/rancher/server/db/state.db 存在索引损坏
  - kine_name_index 索引缺失多个记录行
  - 这导致查询性能严重下降

  2. 过多的数据库连接
  - k3s进程打开了数百个数据库文件描述符
  - 同一个数据库文件被重复打开多次
  - 这是异常行为,说明数据库访问出现问题

  3. WAL文件过大
  - state.db-wal 文件达到7MB
  - 正常情况下WAL文件应该很小
  - 表明事务无法正常提交到主数据库

因此修复方式如下:

systemctl stop k3s.service

# backup
cp /opt/rancher/server/db/state.db /opt/rancher/server/db/state.db.backup
cp /opt/rancher/server/db/state.db-wal /opt/rancher/server/db/state.db-wal.backup

# repair
sqlite3 /opt/rancher/server/db/state.db << EOF
PRAGMA journal_mode=WAL;
PRAGMA wal_checkpoint(FULL);
REINDEX kine_name_index;
VACUUM;
PRAGMA integrity_check;
EOF

systemctl start k3s.service

重启问题解决,很奇怪的问题,怀疑是盘不彳亍了,换外置数据库应该好点。

(考虑到有 vm 每日备份的习惯,就让灵车接着跑罢

Last modification:September 21, 2025
如果觉得我的文章对你有用,请随意赞赏