etcd 故障排查之 `snapshotting is taking more than x seconds to finish`

查看 etcd 运行日志,如果看到如下日志:

snapshotting is taking more than x seconds to finish …  


etcd sends a snapshot of its complete key-value store to refresh slow followers and for backups. Slow snapshot transfer times increase MTTR; if the cluster is ingesting data with high throughput, slow followers may livelock by needing a new snapshot before finishing receiving a snapshot. To catch slow snapshot performance, etcd warns when sending a snapshot takes more than thirty seconds and exceeds the expected transfer time for a 1Gbps connection.


etcd会把kv snapshot发送给一些比较慢的follow或者进行数据备份。慢的snapshot发送会拖慢系统的性能,其自身也会陷入一种活锁状态:在很慢地收完一个snapshot后还没有处理完,又因为过慢而接收新的snapshot。当发送一个snapshot超过30s并且在1Gbps(千兆)网络环境下使用时间超过一定时间时,etcd就会打印这个日志进行告警。