10000
http://10.0.226.130:3100/loki/api/v1/query_range?direction=BACKWARD&end=1619568000000000000&limit=1000&query=%7Bnamespace%3D%22kong-free%22%2C+app_kubernetes_io_name%3D%22kong%22%7D+%7C%3D%22World%22&start=1611792000000000000
Query failed: Get "http://10.0.226.130:3100/loki/api/v1/query_range?direction=BACKWARD&end=1619568000000000000&limit=1000&query=%7Bnamespace%3D%22kong-free%22%2C+app_kubernetes_io_name%3D%22kong%22%7D+%7C%3D%22World%22&start=1611792000000000000": EOF
real 0m35.386s
user 0m0.103s
sys 0m0.054s
Включил loglevel debug, по traceID грепнул, вот что выдало (последние строчки):
ts=2021-04-28T08:00:58.014561013Z caller=spanlogger.go:79 org_id=fake traceID=2752d80790f3b57c method=query.Exec level=debug Ingester.TotalReached=1 Ingester.TotalChunksMatched=9 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Store.TotalChunksRef=52957 Store.TotalChunksDownloaded=12250 Store.ChunksDownloadTime=2.287528377s Store.HeadChunkBytes="0 B" Store.HeadChunkLines=0 Store.DecompressedBytes="18 GB" Store.DecompressedLines=30429153 Store.CompressedBytes="1.7 GB" Store.TotalDuplicates=0
ts=2021-04-28T08:00:58.014632581Z caller=spanlogger.go:79 org_id=fake traceID=2752d80790f3b57c method=query.Exec level=debug Summary.BytesProcessedPerSecond="496 MB" Summary.LinesProcessedPerSecond=862544 Summary.TotalBytesProcessed="18 GB" Summary.TotalLinesProcessed=30429153 Summary.ExecTime=35.27833914s
level=info ts=2021-04-28T08:00:58.014686674Z caller=metrics.go:91 org_id=fake traceID=2752d80790f3b57c latency=slow query="{namespace=\"kong-free\", app_kubernetes_io_name=\"kong\"} |=\"World\"" query_type=filter range_type=range length=2160h0m0s step=8h38m24s duration=35.27833914s status=200 limit=1000 returned_lines=1000 throughput=496MB total_bytes=18GB
level=warn ts=2021-04-28T08:00:58.016364878Z caller=logging.go:60 traceID=2752d80790f3b57c msg="GET /loki/api/v1/query_range?direction=BACKWARD&end=1619568000000000000&limit=1000&query=%7Bnamespace%3D%22kong-free%22%2C+app_kubernetes_io_name%3D%22kong%22%7D+%7C%3D%22World%22&start=1611792000000000000 35.280294446s, error: write tcp 10.0.226.130:3100->10.0.123.172:55264: i/o timeout ws: false; User-Agent: loki-logcli/2.2.1; "
Удалось починить?
Не удалось. Скорей всего там идет закрытие websocket соединения, отваливается по таймауту. Нужно реализовать keep-alive/ping-pong websocket connection в logcli.
Обсуждают сегодня