跳转至

故障处理汇总

磁盘故障带来 unrecoverable error when merging small parts in the partition

2022-11-11T10:27:47.164+0800 panic VictoriaMetrics/lib/storage/partition.go:931 FATAL: unrecoverable error when merging small parts in the partition "/vm-data/data/small/2022_11": error when merging parts to "/vm-data/data/small/2022_11/tmp/172666BA849D06B1": cannot merge 14 streams: [/vm-data/data/small/2022_11/91224_60084_20221102130050.739_20221102130104.527_17234CDD753C6D4E /vm-data/data/small/2022_11/164243_87952_20221102125913.974_20221102125934.530_17234CDD753C6D47 /vm-data/data/small/2022_11/180484_104605_20221102125941.709_20221102125959.194_17234CDD753C6D49 /vm-data/data/small/2022_11/174886_130095_20221102130011.709_20221102130100.680_17234CDD753C6D4C /vm-data/data/small/2022_11/326346_158974_20221102125950.420_20221102130021.499_17234CDD753C6D4A /vm-data/data/small/2022_11/318353_133387_20221102125927.092_20221102125949.397_17234CDD753C6D48 /vm-data/data/small/2022_11/383411_146258_20221102130013.524_20221102130102.913_17234CDD753C6D4D /vm-data/data/small/2022_11/434374_151242_20221102125959.45...
panic: FATAL: unrecoverable error when merging small parts in the partition "/vm-data/data/small/2022_11": error when merging parts to "/vm-data/data/small/2022_11/tmp/172666BA849D06B1": cannot merge 14 streams: [/vm-data/data/small/2022_11/91224_60084_20221102130050.739_20221102130104.527_17234CDD753C6D4E /vm-data/data/small/2022_11/164243_87952_20221102125913.974_20221102125934.530_17234CDD753C6D47 /vm-data/data/small/2022_11/180484_104605_20221102125941.709_20221102125959.194_17234CDD753C6D49 /vm-data/data/small/2022_11/174886_130095_20221102130011.709_20221102130100.680_17234CDD753C6D4C /vm-data/data/small/2022_11/326346_158974_20221102125950.420_20221102130021.499_17234CDD753C6D4A /vm-data/data/small/2022_11/318353_133387_20221102125927.092_20221102125949.397_17234CDD753C6D48 /vm-data/data/small/2022_11/383411_146258_20221102130013.524_20221102130102.913_17234CDD753C6D4D /vm-data/data/small/2022_11/434374_151242_20221102125959.453_20221102130059.194_17234CDD753C6D4B /vm-data/data/small/2022_11/853827_...
goroutine 56 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage({0x8fbdd5, 0x5}, {0xc0018dc000, 0x8216}, 0x2?)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:269 +0x975
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevelSkipframes(0x1, {0x8fbdd5, 0x5}, {0x91e5e7?, 0x0?}, {0xc0007f9f88?, 0x0?, 0x0?})
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:137 +0x1d6
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(...)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:129
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:125
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*partition).smallPartsMerger(0xc000914000)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:931 +0xee
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*partition).startMergeWorkers.func1()
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:910 +0x25
created by github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*partition).startMergeWorkers
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:909 +0x27

清理对应的损坏文件