убедитесь, что в логах vmselect и vmstorage нод нет ошибок на интервале времени, когда обновлялся второй график
кластерная, вот я могу связать это с тем, что у меня 3 узла, на всех я установил -replicationFactor=3 флаг для vminsert и для vmselect (+-dedup.minScrapeInterval=1ms как в доке), потом остановил весь стек контейнеров кластера ВМ (docker-compose down), чуть подождал и вновь запустил, ошибки, конечно, были в vmselect и vmstorage на этот период времени.
можете посмотреть, какие ошибки были в vminsert, когда был недоступен один узел?
2021-08-03T17:22:09.320+0300 info VictoriaMetrics/app/vminsert/netstorage/netstorage.go:210 successfully dialed -storageNode="192.168.3.52:8400" 2021-08-03T17:22:09.321+0300 info VictoriaMetrics/app/vminsert/netstorage/netstorage.go:210 successfully dialed -storageNode="192.168.3.53:8400" 2021-08-03T17:22:09.323+0300 info VictoriaMetrics/app/vminsert/netstorage/netstorage.go:210 successfully dialed -storageNode="192.168.3.51:8400" 2021-08-04T09:37:40.320+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:236 cannot send 77530 bytes with 417 rows to -storageNode="192.168.3.52:8400": cannot read ack from vmstorage: cannot read data in 0.000 seconds: EOF; closing the connection to storageNode and re-routing this data to healthy storage nodes 2021-08-04T09:37:40.322+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:167 cannot make a copy #3 out of 3 copies according to -replicationFactor=3 for 83624 bytes with 450 rows, since a part of storage nodes is temporarily unavailable 2021-08-04T09:37:40.322+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:167 cannot make a copy #3 out of 3 copies according to -replicationFactor=3 for 75144 bytes with 412 rows, since a part of storage nodes is temporarily unavailable 2021-08-04T09:37:40.323+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:167 cannot make a copy #3 out of 3 copies according to -replicationFactor=3 for 77530 bytes with 417 rows, since a part of storage nodes is temporarily unavailable 2021-08-04T09:37:40.520+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:206 cannot dial storageNode "192.168.3.52:8400": dial tcp4 192.168.3.52:8400: connect: connection refused 2021-08-04T09:37:41.320+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:167 cannot make a copy #3 out of 3 copies according to -replicationFactor=3 for 77668 bytes with 381 rows, since a part of storage nodes is temporarily unavailable 2021-08-04T09:37:41.320+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:167 cannot make a copy #3 out of 3 copies according to -replicationFactor=3 for 56118 bytes with 280 rows, since a part of storage nodes is temporarily unavailable 2021-08-04T09:37:41.321+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:167 cannot make a copy #3 out of 3 copies according to -replicationFactor=3 for 30140 bytes with 145 rows, since a part of storage nodes is temporarily unavailable 2021-08-04T09:37:42.320+0300 warn VictoriaMetrics/app/vminsert/netstorage/netstorage.go:167 cannot make a copy #3 out of 3 copies according to -replicationFactor=3 for 33861 bytes with 223 rows, since a part of storage nodes is temporarily unavailable и тд и тд Имитировал отказ узла с ip 192.168.3.52
Обсуждают сегодня