Remove Exists call in meta fetcher #6474
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <[email protected]>
fpetkovski
force-pushed
the
reduce-compactor-api-calls
branch
from
09a2483
to
ffe357d
Compare
GiedriusS
reviewed
Jun 27, 2023
test/e2e/store_gateway_test.go
Outdated
| // thanos_blocks_meta_synced: 2x loadedMeta 1x labelExcludedMeta 1x TooFreshMeta 1x noMeta. | ||
| testutil.Ok(t, s1.WaitSumMetrics(e2emon.Equals(5), "thanos_blocks_meta_synced")) | ||
| // thanos_blocks_meta_synced: 2x loadedMeta 1x labelExcludedMeta 1x. | ||
| testutil.Ok(t, s1.WaitSumMetrics(e2emon.Equals(4), "thanos_blocks_meta_synced")) |
There was a problem hiding this comment.
Why did this change? 🤔 Also, the comment above this line seems to be missing a part.
There was a problem hiding this comment.
Good catch, looks like it's a leftover artefact of my debugging. I reverted the changes and the test is still passing.
There was a problem hiding this comment.
How did it work before? 😂 this should fail with 4. What the hell 😄
There was a problem hiding this comment.
I think it's because it waits for the sum to become equal to the asserted value. So in theory it should fail if we set it to 6, but anything bellow 5 should pass with enough retries :)
Signed-off-by: Filip Petkovski <[email protected]>
GiedriusS
approved these changes
Jun 27, 2023
|
Maybe this change worth a changelog? Though its an implementation detail only |
3 tasks
fpetkovski
added a commit
to fpetkovski/thanos
that referenced
this pull request
Jul 10, 2023
Signed-off-by: Filip Petkovski <[email protected]>
|
I think it makes sense to do it. Added one here: https://github.com/thanos-io/thanos/pull/6518/files. |
squat
pushed a commit
that referenced
this pull request
Jul 10, 2023
Signed-off-by: Filip Petkovski <[email protected]>
coleenquadros
pushed a commit
to coleenquadros/thanos
that referenced
this pull request
Jul 11, 2023
* Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <[email protected]> * Revert e2e test change Signed-off-by: Filip Petkovski <[email protected]> --------- Signed-off-by: Filip Petkovski <[email protected]>
2 tasks
GiedriusS
pushed a commit
to vinted/thanos
that referenced
this pull request
Jul 27, 2023
* Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <[email protected]> * Revert e2e test change Signed-off-by: Filip Petkovski <[email protected]> --------- Signed-off-by: Filip Petkovski <[email protected]>
subbarao-meduri
added a commit
to stolostron/thanos
that referenced
this pull request
Sep 7, 2023
* Updates busybox SHA (#6283) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <[email protected]> * Reverted test code Signed-off-by: Alex Le <[email protected]> * Updated comment Signed-off-by: Alex Le <[email protected]> * docs: mismatch in changelog Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: Alex Le <[email protected]> * Updates busybox SHA (#6283) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> Signed-off-by: Alex Le <[email protected]> * trigger workflow Signed-off-by: Alex Le <[email protected]> * trigger workflow Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: GitHub <[email protected]> Co-authored-by: Etienne Martel <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <[email protected]> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <[email protected]> * sort ProxyResponseHeap comparing only diff labels Signed-off-by: Walther Lee <[email protected]> * proxy heap: remove store labels before comparing response series Signed-off-by: Walther Lee <[email protected]> * tests: Remove custom Between test matcher (#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <[email protected]> * Run go mod tidy Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * Remove testingRespSet and update comparison func Signed-off-by: Walther Lee <[email protected]> * Fix tests Signed-off-by: Walther Lee <[email protected]> * query frontend, query UI: Native histogram support (#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <[email protected]> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <[email protected]> Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> Add chnagelog Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed changelog Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed qfe Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <[email protected]> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <[email protected]> Updated Readme Signed-off-by: Sebastian Rabenhorst <[email protected]> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <[email protected]> trigger tests Signed-off-by: Sebastian Rabenhorst <[email protected]> Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> * Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <[email protected]> * fixed imports Signed-off-by: Sebastian Rabenhorst <[email protected]> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <[email protected]> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]> * store: add streamed snappy encoding for postings list (#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <[email protected]> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <[email protected]> * store: fix estimation Signed-off-by: Giedrius Statkevičius <[email protected]> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <[email protected]> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <[email protected]> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <[email protected]> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: atomically replace no compact marked map (#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <[email protected]> * Reuse buffers for label comparison Signed-off-by: Filip Petkovski <[email protected]> * Use len instead of cap Signed-off-by: Filip Petkovski <[email protected]> * Use len instead of cap Signed-off-by: Filip Petkovski <[email protected]> * Fixed Jaeger exporter specification link (#6324) Signed-off-by: Sebastian Rabenhorst <[email protected]> * receive: Make max backoff configurable (#6163) Signed-off-by: Philip Gough <[email protected]> * Replace summary in extprom metrics with histogram (#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <[email protected]> * Added changelog Signed-off-by: Sebastian Rabenhorst <[email protected]> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <[email protected]> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <[email protected]> * Add test case for proxy store Signed-off-by: Filip Petkovski <[email protected]> * Pass engine factory by reference Signed-off-by: Sebastian Rabenhorst <[email protected]> Cleanup Signed-off-by: Sebastian Rabenhorst <[email protected]> * Use NewQueryEngineFactory in tests Signed-off-by: Sebastian Rabenhorst <[email protected]> * Avoid expensive log.Valuer evaluation for disallowed levels (#6322) Signed-off-by: Xiaochao Dong (@damnever) <[email protected]> * Fix inconsistent error for series limits in Store API (#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <[email protected]> * update changelog Signed-off-by: Thibault Mange <[email protected]> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <[email protected]> Signed-off-by: Thibault Mange <[email protected]> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <[email protected]> Signed-off-by: Thibault Mange <[email protected]> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <[email protected]> --------- Signed-off-by: Thibault Mange <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> * *: remove unmaintained gzip library (#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <[email protected]> * Traces sampler env var (#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <[email protected]> * Test correction Signed-off-by: shayyxi <[email protected]> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <[email protected]> * added the changelog. Signed-off-by: shayyxi <[email protected]> * ran make docs to fix the build failure. Signed-off-by: shayyxi <[email protected]> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <[email protected]> * fixed review comments. Signed-off-by: shayyxi <[email protected]> --------- Signed-off-by: shayyxi <[email protected]> Signed-off-by: Shazi <[email protected]> Co-authored-by: shayyxi <[email protected]> * query: use storepb.SeriesServer (#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <[email protected]> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <[email protected]> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345) Signed-off-by: aimuz <[email protected]> * Return grpc code resource exhausted for byte limit error (#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <[email protected]> * fix lint Signed-off-by: Ben Ye <[email protected]> * update partial response strategy Signed-off-by: Ben Ye <[email protected]> * fix limit Signed-off-by: Ben Ye <[email protected]> * try to fix tests Signed-off-by: Ben Ye <[email protected]> * fix test error message Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349) Signed-off-by: Alban HURTAUD <[email protected]> * Optimize generating store match reason (#6355) Removes reason generation when debug logging is not enabled. Signed-off-by: Filip Petkovski <[email protected]> * Updates busybox SHA (#6365) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * Query: Add +Inf bucket to query duration metrics (#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <[email protected]> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <[email protected]> --------- Signed-off-by: Jacob Baungard Hansen <[email protected]> * Check context when expanding postings (#6363) * check context when expanding postings Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * ui: only keep name in store_matches param (#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <[email protected]> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <[email protected]> * Optimizing "grafana generated" regex matchers (#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <[email protected]> * fixing native histogram tests Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * Cache: various index cache client improvements (#6374) * e2e(query): Reproduce dedup issue from #6257 Signed-off-by: Douglas Camata <[email protected]> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <[email protected]> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <[email protected]> * Improve query dedup test Signed-off-by: Douglas Camata <[email protected]> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <[email protected]> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <[email protected]> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <[email protected]> * Fix receive drawing Signed-off-by: Douglas Camata <[email protected]> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <[email protected]> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <[email protected]> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <[email protected]> * Query Explanation (#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <[email protected]> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <[email protected]> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <[email protected]> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <[email protected]> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <[email protected]> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <[email protected]> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <[email protected]> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <[email protected]> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <[email protected]> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <[email protected]> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <[email protected]> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <[email protected]> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <[email protected]> * Update UI assets Signed-off-by: Pradyumna Krishna <[email protected]> --------- Signed-off-by: Pradyumna Krishna <[email protected]> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <[email protected]> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Put back the correct makefile Signed-off-by: Douglas Camata <[email protected]> * add de-cix as adopter (#6386) Signed-off-by: Raul Garcia Sanchez <[email protected]> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <[email protected]> * Add back line break at end of makefile Signed-off-by: Douglas Camata <[email protected]> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <[email protected]> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <[email protected]> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <[email protected]> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <[email protected]> * Reuse nginx image name Signed-off-by: Douglas Camata <[email protected]> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * [chore] Updating Query Engine and Prometheus (#6392) * Updating Query Engine Signed-off-by: Alan Protasio <[email protected]> * fix prometheus breaking change Signed-off-by: Alan Protasio <[email protected]> * Update prometheus with https://github.com/prometheus/prometheus/pull/12387 Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777) Signed-off-by: haanhvu <[email protected]> * check context cancel when doing posting batches (#6396) Signed-off-by: Ben Ye <[email protected]> * Expose store gateway query stats in series response hints (#6352) * expose query stats hints Signed-off-by: Ben Ye <[email protected]> * update Signed-off-by: Ben Ye <[email protected]> * add query stats hints in result Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * add merge method Signed-off-by: Ben Ye <[email protected]> * fix unit test Signed-off-by: Ben Ye <[email protected]> modify hints proto Signed-off-by: Ben Ye <[email protected]> fix unit test Signed-off-by: Ben Ye <[email protected]> update format Signed-off-by: Ben Ye <[email protected]> * update comments Signed-off-by: Ben Ye <[email protected]> * try again Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * receive: make az aware ketama hashring (#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <[email protected]> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <[email protected]> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <[email protected]> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <[email protected]> --------- Signed-off-by: Alexander Rickardsson <[email protected]> Signed-off-by: Michael Hoffmann <[email protected]> Co-authored-by: Michael Hoffmann <[email protected]> * Reformat asciiflow chart Signed-off-by: Douglas Camata <[email protected]> * Proposal: query path tenancy (#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <[email protected]> * Update proposal after initial feedback Signed-off-by: Douglas Camata <[email protected]> * Add cool picture Signed-off-by: Douglas Camata <[email protected]> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <[email protected]> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <[email protected]> * Improve writing in "How" section Signed-off-by: Douglas Camata <[email protected]> * Fix owner profile link Signed-off-by: Douglas Camata <[email protected]> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <[email protected]> Signed-off-by: Douglas Camata <[email protected]> * Address few more PR review comments Signed-off-by: Douglas Camata <[email protected]> * Address feedback on flag name text Signed-off-by: Douglas Camata <[email protected]> * Update diagram Signed-off-by: Douglas Camata <[email protected]> * Improve non-goals text Signed-off-by: Douglas Camata <[email protected]> * Update diagram Signed-off-by: Douglas Camata <[email protected]> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <[email protected]> Signed-off-by: Douglas Camata <[email protected]> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <[email protected]> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <[email protected]> * Add an action plan Signed-off-by: Douglas Camata <[email protected]> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <[email protected]> * Fix typo Co-authored-by: Giedrius Statkevičius <[email protected]> Signed-off-by: Douglas Camata <[email protected]> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <[email protected]> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <[email protected]> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <[email protected]> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <[email protected]> * Improve action plan structure and writing Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> Co-authored-by: Filip Petkovski <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]> * Fix double-counting bug in http_request_duration metric (#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <[email protected]> * Update Changelog Signed-off-by: 4orty <[email protected]> --------- Signed-off-by: 4orty <[email protected]> * Replace group with resolution in compact metrics. Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: https://github.com/thanos-io/thanos/issues/5841 Signed-off-by: SuperQ <[email protected]> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <[email protected]> * Run go mod tidy Signed-off-by: Douglas Camata <[email protected]> * Updates busybox SHA (#6403) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * Fix series stats merge (#6408) * fix series stats merge Signed-off-by: Ben Ye <[email protected]> * update license header Signed-off-by: Ben Ye <[email protected]> * use reflect Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Receive: allow unlimited head_series_limit tenants (#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: https://github.com/thanos-io/thanos/issues/6393 Signed-off-by: Jacob Baungard Hansen <[email protected]> * expose downloaded data size in query hints (#6409) Signed-off-by: Ben Ye <[email protected]> * maintainers: add myself to triagers (#6414) Signed-off-by: Michael Hoffmann <[email protected]> * Add `@douglascamata` to triagers (#6418) Signed-off-by: Douglas Camata <[email protected]> * Add Blog (#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <[email protected]> * Add Headers to blog Signed-off-by: Pradyumna Krishna <[email protected]> * Lint blog Signed-off-by: Pradyumna Krishna <[email protected]> --------- Signed-off-by: Pradyumna Krishna <[email protected]> * blog: Fix images for LFX post (#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <[email protected]> * fix lint Signed-off-by: Saswata Mukherjee <[email protected]> --------- Signed-off-by: Saswata Mukherjee <[email protected]> * Index Cache: Change cache key for postings (#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <[email protected]> * add changelog Signed-off-by: Ben Ye <[email protected]> * update code back Signed-off-by: Ben Ye <[email protected]> * add colon Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * fix another test Signed-off-by: Ben Ye <[email protected]> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <[email protected]> * address required comments Signed-off-by: Ben Ye <[email protected]> * fix compression scheme name Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Receive: upgrading logs for failed uploads to error (#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <[email protected]> * docs: added changelog entry Signed-off-by: Victor Fernandes <[email protected]> --------- Signed-off-by: Victor Fernandes <[email protected]> * fix postings test Signed-off-by: Ben Ye <[email protected]> * Add aiven as adopter... more soon! Signed-off-by: Jonah Kowall <[email protected]> * First draft of the Aiven adoption story Signed-off-by: Jonah Kowall <[email protected]> * Add aiven as adopter... more soon! (#6430) Signed-off-by: Jonah Kowall <[email protected]> * Report gRPC connnection errors to the caller (#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <[email protected]> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <[email protected]> * docs: add changelog Signed-off-by: Eng Zer Jun <[email protected]> --------- Signed-off-by: Eng Zer Jun <[email protected]> * Expose estimated chunk and series size as configurable options (#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <[email protected]> * fix lint Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Receive: make tsdb stats limit configurable (#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <[email protected]> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <[email protected]> --------- Signed-off-by: Michael Hoffmann <[email protected]> * *: wire new Engine/Explain fields in query-frontend (#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <[email protected]> * index cache: Cache expanded postings (#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * fix Signed-off-by: Ben Ye <[email protected]> * fix lint Signed-off-by: Ben Ye <[email protected]> * rebase main and added compression name to key Signed-off-by: Ben Ye <[email protected]> * update key Signed-off-by: Ben Ye <[email protected]> * add e2e test for memcached Signed-off-by: Ben Ye <[email protected]> * fix cache config Signed-off-by: Ben Ye <[email protected]> * address review comments Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * add approximate series size to index stats (#6425) Signed-off-by: Ben Ye <[email protected]> * index stats: fix chunk size calculation (#6424) Signed-off-by: Ben Ye <[email protected]> * Remove some unused Cortex vendored code and metrics (#6440) * Avoid direct cortex vendor dependency in promclient (#6443) * use own explanation struct Signed-off-by: Ben Ye <[email protected]> * omit empty Signed-off-by: Ben Ye <[email protected]> * fix e2e test Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351) Fixes: #6280 Signed-off-by: Jonathan Davies <[email protected]> * Updates busybox SHA (#6448) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * store: read postings directly into delta encoded format (#6442) Instead of allocating bytes for raw postings, let's read them directly into diff varint format to save memory. Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: Add Index Stats to block metadata (#6441) * compactor set index stats on block Signed-off-by: Ben Ye <[email protected]> * comment Signed-off-by: Ben Ye <[email protected]> * change field Signed-off-by: Ben Ye <[email protected]> * update downsample Signed-off-by: Ben Ye <[email protected]> * add index stats into compaction Signed-off-by: Ben Ye <[email protected]> * fix tests Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * estimate block chunk and series size from metadata (#6449) Signed-off-by: Ben Ye <[email protected]> * Adding graphic for blog showing architecture Signed-off-by: Jonah Kowall <[email protected]> * Fixing mermaid graph Signed-off-by: Jonah Kowall <[email protected]> * Fixing mermaid graph again Signed-off-by: Jonah Kowall <[email protected]> * Remove image and fix mermaid for other flowchart Signed-off-by: Jonah Kowall <[email protected]> * Fixing prod flowchart Signed-off-by: Jonah Kowall <[email protected]> * Fixing prod flowchart title Signed-off-by: Jonah Kowall <[email protected]> * Fixing md table Signed-off-by: Jonah Kowall <[email protected]> * Fixing md table again Signed-off-by: Jonah Kowall <[email protected]> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <[email protected]> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <[email protected]> * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <[email protected]> * Some corrections on text and links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing diagams Signed-off-by: Jonah Kowall <[email protected]> * Fixing currencies Signed-off-by: Jonah Kowall <[email protected]> * Fixing diagams Signed-off-by: Jonah Kowall <[email protected]> * Fixing images Signed-off-by: Jonah Kowall <[email protected]> * Website: Fix blog images width (#6450) * Set blog post images width to 100% Blog Post images width isn't set and hence the images doesn't get fit to the page. Setting width to 100% make sure images fit to the web page and screen size. Signed-off-by: Pradyumna Krishna <[email protected]> * Remove width constraint from blog images Signed-off-by: Pradyumna Krishna <[email protected]> --------- Signed-off-by: Pradyumna Krishna <[email protected]> * Cache calculated mint and maxt for each remote engine (#6458) The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times on each remote engine. These calculations can get expensive when an engine covers a large amount of TSDBs. This commit introduces caching of the calculated values for a remote engine's mint, maxt and label sets. Signed-off-by: Filip Petkovski <[email protected]> * cache empty expanded postings (#6464) Signed-off-by: Ben Ye <[email protected]> * mixin/receive: add limits alerting (#6466) * add some thanos receive limits alerts Signed-off-by: Thibault Mange <[email protected]> * update changelog Signed-off-by: Thibault Mange <[email protected]> * fix and generate examples Signed-off-by: Thibault Mange <[email protected]> * quick fix Signed-off-by: Thibault Mange <[email protected]> * quick fix Signed-off-by: Thibault Mange <[email protected]> * fix test Signed-off-by: Thibault Mange <[email protected]> * fix invalid changelog Signed-off-by: Thibault Mange <[email protected]> --------- Signed-off-by: Thibault Mange <[email protected]> * Update docs/blog/2023-06-08-thanos-at-aiven.md Co-authored-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Jonah Kowall <[email protected]> * Add retention for M3 Signed-off-by: Jonah Kowall <[email protected]> * Add image versus mermaid diagram. Couldn't comment it out unfortunately Signed-off-by: Jonah Kowall <[email protected]> * Adding text around link to fix linter Signed-off-by: Jonah Kowall <[email protected]> * make docs Signed-off-by: Jonah Kowall <[email protected]> * add alert for tenant reaching head series limit (#6467) Signed-off-by: Thibault Mange <[email protected]> * go.mod: bump rest of otel libs (#6447) * go.mod: bump rest of otel libs Bump these two OTEL libs to be like the rest of the OTEL libs. We had a strange issue where `service.name` was not set even though it was in environment variables. Bumping these libs fixes the problem. Signed-off-by: Giedrius Statkevičius <[email protected]> * otlp: only set service.name if provided Signed-off-by: Giedrius Statkevičius <[email protected]> * e2e: add test case Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * store: disable pooling for postings benchmarks (#6473) Pooling hides the real allocations in postings benchmarks because the allocation only happens once really. In reality this is different when multiple readers are created at once. Disable pooling in benchmarks to understand what is happening. Signed-off-by: Giedrius Statkevičius <[email protected]> * Fixing date and author for Aiven blog (#6476) * Fixing authors Signed-off-by: Jonah Kowall <[email protected]> * make docs and fixing date Signed-off-by: Jonah Kowall <[email protected]> * Fixing authors Signed-off-by: Jonah Kowall <[email protected]> --------- Signed-off-by: Jonah Kowall <[email protected]> * Check context when expanding postings 2nd attempt (#6471) * check context when expanding postings Signed-off-by: Ben Ye <[email protected]> * import Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * [FEAT] Add thanos ruler blog post (#6477) Signed-off-by: Hélia Barroso <[email protected]> * blog: Small fixes to Aiven article (#6481) * Correct some links and typos in 2023-06-08-thanos-at-aiven.md The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links. Also fixed a few other typos I spotted. Signed-off-by: Hugh Cole-Baker <[email protected]> * Update 2023-06-08-thanos-at-aiven.md Clarify a couple of things in the final paragraph. Signed-off-by: Hugh Cole-Baker <[email protected]> --------- Signed-off-by: Hugh Cole-Baker <[email protected]> * Rename series and postings fetch duration metrics (#6479) * fix wrong metric names for series and postings fetch duration Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * note breaking change Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Update to prometheus/common v0.44.0 (#6483) Signed-off-by: Dan Bason <[email protected]> * go.mod: update rueidis client (#6485) The latest improvements have great perf improvements. Let's update it. Thank you to [rueian](https://github.com/rueian) and all of the contributors to that awesome Redis client! Signed-off-by: Giedrius Statkevičius <[email protected]> * Introduce tenancy package (#6482) This commit introduces a tenancy package in preparation for introduction of tenancy in the query path. Tenancy related code which can be common between components are moved out of the receive component into the tenancy package. Signed-off-by: Jacob Baungard Hansen <[email protected]> * Ignore nginx.com links in mdox (#6489) Signed-off-by: Saswata Mukherjee <[email protected]> * blog: Update Aiven blog post (#6487) Wording changes and better formatting for the tables. Signed-off-by: Simon Brüggen <[email protected]> * store: optimized snappy streamed reading (#6475) Optimize snappy streamed reading by traversing through the byte slice in advance to determine how big of a buffer we will need. I have opted to rewrite snappy streamed format decoding because it is a straightforward format. Complex parts were deferred to klauspost/compress. Signed-off-by: Giedrius Statkevičius <[email protected]> * Remove Exists call in meta fetcher (#6474) * Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <[email protected]> * Revert e2e test change Signed-off-by: Filip Petkovski <[email protected]> --------- Signed-off-by: Filip Petkovski <[email protected]> * e2e/store: use now instead of time.Now() each time (#6493) Consistently use the same timestamp instead of the current time each time during a test. Signed-off-by: Giedrius Statkevičius <[email protected]> * [FIX] Documentation issue (#6494) Signed-off-by: Hélia Barroso <[email protected]> * Updates busybox SHA (#6497) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * chore: pkg imported more than once (#6499) * chore: pkg imported more than once Signed-off-by: guoguangwu <[email protected]> * fix: remove pkg alias Signed-off-by: guoguangwu <[email protected]> --------- Signed-off-by: guoguangwu <[email protected]> * *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496) * Fix and improve PathContentReloader tests Signed-off-by: Douglas Camata <[email protected]> * Run tests in parallel Signed-off-by: Douglas Camata <[email protected]> * Add changelog entry Signed-off-by: Douglas Camata <[email protected]> * Fix table tests Signed-off-by: Douglas Camata <[email protected]> * Fix typo Co-authored-by: Filip Petkovski <[email protected]> Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> Co-authored-by: Filip Petkovski <[email protected]> * add context check when decoding cached postings (#6506) Signed-off-by: Ben Ye <[email protected]> * *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503) * Add fallback config reload for symlinks Signed-off-by: Douglas Camata <[email protected]> * Improve pooling config reload Signed-off-by: Douglas Camata <[email protected]> * Make tests more reliable Signed-off-by: Douglas Camata <[email protected]> * Improve tests again Assert on runutil.Repeat output Signed-off-by: Douglas Camata <[email protected]> * Pass the debounce/reload time to each test Signed-off-by: Douglas Camata <[email protected]> * Add comments and ensure interfaces are implemented Signed-off-by: Douglas Camata <[email protected]> * Make the limit configuration reload time customizable Signed-off-by: Douglas Camata <[email protected]> * goimports file Signed-off-by: Douglas Camata <[email protected]> * Fix lint warning Signed-off-by: Douglas Camata <[email protected]> * Improve log for polling engine Signed-off-by: Douglas Camata <[email protected]> * Fix tests Signed-off-by: Douglas Camata <[email protected]> * Make linter happy Signed-off-by: Douglas Camata <[email protected]> * Extract symlink identification to a separate function Signed-off-by: Douglas Camata <[email protected]> * Update changelog Signed-off-by: Douglas Camata <[email protected]> * Fix links to go-grpc-middleware after v2 merge into main Signed-off-by: Douglas Camata <[email protected]> * Add period to make linter happy. Signed-off-by: Douglas Camata <[email protected]> * Update changelog entry Signed-off-by: Douglas Camata <[email protected]> * Replace the fsnotify based engine from `PathContentReloader` with the polling one Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Remove check for empty filePath Signed-off-by: Douglas Camata <[email protected]> * Unexpose pollingEngine.Start Signed-off-by: Douglas Camata <[email protected]> * Fix pollingEngine doc-comment Signed-off-by: Douglas Camata <[email protected]> * Log file path when config is reloaded Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * receive: add float histogram support (#6323) * Added receive float histogram support Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed imports Signed-off-by: Sebastian Rabenhorst <[email protected]> * Added comments for fns copied from Prometheus Signed-off-by: Sebastian Rabenhorst <[email protected]> Improved comment Signed-off-by: Sebastian Rabenhorst <[email protected]> * Removed unnecessary if Signed-off-by: Sebastian Rabenhorst <[email protected]> * Fixed native histogram proto conversion in remote engine Signed-off-by: Sebastian Rabenhorst <[email protected]> * Removed unused histogram conversion from remote engine Signed-off-by: Sebastian Rabenhorst <[email protected]> * Fix and renaming in native_histograms_test.go Signed-off-by: Sebastian Rabenhorst <[email protected]> * Trigger Build Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]> * go.mod: bump e2e framework version (#6516) Bump e2e framework version and add a note about E2E_DOCKER_CPUS. Signed-off-by: Giedrius Statkevičius <[email protected]> * Add changelog for #6474 (#6518) Signed-off-by: Filip Petkovski <[email protected]> * cortex/redisclient: use rueidis client (#6520) * cortex/redisclient: use rueidis client Use the same rueidis client in query-frontend. Solves https://github.com/thanos-io/thanos/issues/6094. Signed-off-by: Giedrius Statkevičius <[email protected]> * cortex/cache: gofumpt + fix errors.Errorf Signed-off-by: Giedrius Statkevičius <[email protected]> * cacheutil/docs: clean up more old stuff Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * index header: Remove memWriter from fileWriter (#6509) * index header: remove memWriter from fileWriter Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * refactor Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> * update comment Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Wrap object store Prometheus registry (#6152) * Wrap object store Prometheus registery In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26. Signed-off-by: Douglas Camata <[email protected]> * Update objstore to the latest version This version removes the `thanos_` prefix from metrics, which is the reason for wrapping the objstore's metrics registry in the first place. Signed-off-by: Douglas Camata <[email protected]> * Fix modules Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Wrap metrics registerer for objstore bucket Signed-off-by: Douglas Camata <[email protected]> * Remove prefix from Thanos Store metrics Signed-off-by: Douglas Camata <[email protected]> * Fix goimports Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Put back upgraded objstore dep Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Move to more recent ref of thanos/objstore Signed-off-by: Douglas Camata <[email protected]> * Ignore OBS in objstore tests Signed-off-by: Douglas Camata <[email protected]> * Fix linting error on main test after objstore upgrade Signed-off-by: Douglas Camata <[email protected]> * Skip OCS objstore test in circle ci Signed-off-by: Douglas Camata <[email protected]> * Fix echo in makefile Signed-off-by: Douglas Camata <[email protected]> * Fix typo Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * e2e/store: try to fix Series() limit test again (#6522) I finally managed to reproduce this failure locally with https://github.com/efficientgo/e2e/commit/c316eb95ae5b0fa65b0be110f8cfc3c48e8810bb. The added t.Logf() showed that is the problem is that with a lower bytes limit, it might hit the series or chunks part first. I have bumped the bytes limit. I calculated the new bytes limit by checking how much bytes are allocated before sending the last chunk. I have also noticed that one block is created without a delay. Update it so that it would be like the others. Include objstore@main update with https://github.com/thanos-io/objstore/pull/62/files so that Iter() would always return an error on a timeout. Signed-off-by: Giedrius Statkevičius <[email protected]> * pkg/reloader: use watchInterval timeout for initial apply (#6519) * pkg/reloader: use watchInterval timeout for initial apply Signed-off-by: Craig Peterson <[email protected]> * changelog Signed-off-by: Craig Peterson <[email protected]> * use distinct context for initial sync Signed-off-by: Craig Peterson <[email protected]> --------- Signed-off-by: Craig Peterson <[email protected]> * Standardize index cache metrics (#6523) * Add histogram metrics for index cache item size (#6528) * add histogram metrics for index cache item size Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Upgrade objstore (#6507) * Upgrade objstore Signed-off-by: Kemal Akkoyun <[email protected]> * Update with the latest APIs Signed-off-by: Kemal Akkoyun <[email protected]> --------- Signed-off-by: Kemal Akkoyun <[email protected]> Co-authored-by: Matej Gera <[email protected]> * Make compact lifecycle more flexible to be overridden for sharded compaction (#5964) * Updates busybox SHA (#6534) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * BucketedBytes to buffer byte slices when decoding postings from cache (#6531) * BucketedBytes to buffer byte slices when deconding postings from cache Signed-off-by: Alan Protasio <[email protected]> * Fixing Lint Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * Shipper: change upload compacted type from bool to a function (#6526) * change shipper upload compacted type from bool to a function Signed-off-by: Ben Ye <[email protected]> * add default to false Signed-off-by: Ben Ye <[email protected]> * reset uploadedCompacted to 0 Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Query: Forward tenant information via StoreAPI (#6530) * Querier: Forward tenant information downstream With this commit we attach tenant information to each query request and forward it via the StoreAPI to any downstream Store Gateways and Queriers. We add the following command lines options which mimics the tenant functionality in Receive. The options are currently hidden, as they provide no real functionality yet. This will come in future steps. --query.tenant-header --query.default-tenant --query.tenant-certificate Signed-off-by: Jacob Baungard Hansen <[email protected]> * Receive: Use CertificateField from Tenancy pkg These consts are now defined in the Tenancy package, so we should use those instead. Signed-off-by: Jacob Baungard Hansen <[email protected]> --------- Signed-off-by: Jacob Baungard Hansen <[email protected]> * Deduplicate matchers in posting group (#6532) * deduplicate matchers in posting group Signed-off-by: Ben Ye <[email protected]> * don't use map for deduplication Signed-off-by: Ben Ye <[email protected]> * lint Signed-off-by: Ben Ye <[email protected]> * address comments Signed-off-by: Ben Ye <[email protected]> * fi…
subbarao-meduri
added a commit
to stolostron/thanos
that referenced
this pull request
Sep 21, 2023
* fix 0.31 changelog (#6278) Signed-off-by: junot <[email protected]> * Query: Switch Multiple Engines (#6234) * Query: Switch engines using `engine` param Thanos query has two engine, prometheus (default) and thanos. A single engine runs through thanos query command at a time, and have to re run the command to switch between. This commit adds a functionality to run multiple engines at once and switch between them using `engine` query param inq query api. To avoid duplicate matrics registration, the thanos engine is provided with a different registerer having prefix `tpe_` (not been finalized yet). promql-engine command line flag has been removed that specifies the query engine to run. Currently this functionality not implemented on GRPCAPI. Signed-off-by: Pradyumna Krishna <[email protected]> * Add multiple engine support to GRPCAPI Fix build fail for thanos, adds support for multiple engine in GRPCAPI. Signed-off-by: Pradyumna Krishna <[email protected]> * Create QueryEngineFactory to create engines QueryEngineFactory makes a collection for all promql engines used by thanos and returns it. Any engine can be created and returned using `GetXEngine` method. It is currently limited to 2 engines prometheus and thanos engines that get created on the first call. Signed-off-by: Pradyumna Krishna <[email protected]> * Use QueryEngineFactory in query API thanos query commands pass `QueryEngineFactory` to query apis that will use engine based on query params. It will provide more flexibility to create multiple engines in thanos. Adds `defaultEngine` CLI flag, A default engine to use if not specified with query params. Signed-off-by: Pradyumna Krishna <[email protected]> * Update Query API tests Fixes breaking tests Signed-off-by: Pradyumna Krishna <[email protected]> * Minor changes and Docs fixes * Move defaultEngine argument to reduce diff. * Generated Docs. Signed-off-by: Pradyumna Krishna <[email protected]> * Add Engine Selector/ Dropdown to Query UI Engine Selector is a dropdown that sets an engine to be used to run the query. Currently two engines `thanos` and `prometheus`. This dropdown sends a query param `engine` to query api, which runs the api using the engine provided. Provided to run query using multiple query engines from Query UI. Signed-off-by: Pradyumna Krishna <[email protected]> * Move Engine Selector to Panel Removes Dropdown component, and renders Engine Selector directly. Receives defaultEngine from `flags` API. Updates parseOptions to parse engine query param and updates test for Panel and utils. Signed-off-by: Pradyumna Krishna <[email protected]> * Upgrade promql-engine dependency Updates promql-engine that brings functionality to provide fallback engine using enigne Opts. Signed-off-by: Pradyumna Krishna <[email protected]> * Add MinT to remote client MinT method was missing from Client due to updated promql-engine. This commits adds mint to the remote client. Signed-off-by: Pradyumna Krishna <[email protected]> * Use prometheus fallback engine in thanos engine Thanos engine creates a fallback prometheus engine that conflicts with another prometheus engine created by thanos, while registering metrics. To fix this, provided created thanos engine as fallback engine to thanos engine in engine Opts. Signed-off-by: Pradyumna Krishna <[email protected]> * Use enum for EngineType in GRPC GRPC is used for communication between thanos components and defaultEngine was a string before. Enum makes more sense, and hence the request.Enigne type has been changed to querypb.EngineType. Default case is handled with another default value provided in the enum. Signed-off-by: Pradyumna Krishna <[email protected]> * Update query UI bindata.go Compile react app using `make assets`. Signed-off-by: Pradyumna Krishna <[email protected]> --------- Signed-off-by: Pradyumna Krishna <[email protected]> * docs: mismatch in changelog Signed-off-by: Etienne Martel <[email protected]> * Updates busybox SHA (#6283) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <[email protected]> * Reverted test code Signed-off-by: Alex Le <[email protected]> * Updated comment Signed-off-by: Alex Le <[email protected]> * docs: mismatch in changelog Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: Alex Le <[email protected]> * Updates busybox SHA (#6283) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> Signed-off-by: Alex Le <[email protected]> * trigger workflow Signed-off-by: Alex Le <[email protected]> * trigger workflow Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: GitHub <[email protected]> Co-authored-by: Etienne Martel <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <[email protected]> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <[email protected]> * sort ProxyResponseHeap comparing only diff labels Signed-off-by: Walther Lee <[email protected]> * proxy heap: remove store labels before comparing response series Signed-off-by: Walther Lee <[email protected]> * tests: Remove custom Between test matcher (#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <[email protected]> * Run go mod tidy Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * Remove testingRespSet and update comparison func Signed-off-by: Walther Lee <[email protected]> * Fix tests Signed-off-by: Walther Lee <[email protected]> * query frontend, query UI: Native histogram support (#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <[email protected]> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <[email protected]> Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> Add chnagelog Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed changelog Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed qfe Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <[email protected]> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <[email protected]> Updated Readme Signed-off-by: Sebastian Rabenhorst <[email protected]> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <[email protected]> trigger tests Signed-off-by: Sebastian Rabenhorst <[email protected]> Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> * Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <[email protected]> * fixed imports Signed-off-by: Sebastian Rabenhorst <[email protected]> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <[email protected]> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]> * store: add streamed snappy encoding for postings list (#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <[email protected]> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <[email protected]> * store: fix estimation Signed-off-by: Giedrius Statkevičius <[email protected]> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <[email protected]> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <[email protected]> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <[email protected]> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: atomically replace no compact marked map (#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <[email protected]> * Reuse buffers for label comparison Signed-off-by: Filip Petkovski <[email protected]> * Use len instead of cap Signed-off-by: Filip Petkovski <[email protected]> * Use len instead of cap Signed-off-by: Filip Petkovski <[email protected]> * Fixed Jaeger exporter specification link (#6324) Signed-off-by: Sebastian Rabenhorst <[email protected]> * receive: Make max backoff configurable (#6163) Signed-off-by: Philip Gough <[email protected]> * Replace summary in extprom metrics with histogram (#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <[email protected]> * Added changelog Signed-off-by: Sebastian Rabenhorst <[email protected]> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <[email protected]> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <[email protected]> * Add test case for proxy store Signed-off-by: Filip Petkovski <[email protected]> * Pass engine factory by reference Signed-off-by: Sebastian Rabenhorst <[email protected]> Cleanup Signed-off-by: Sebastian Rabenhorst <[email protected]> * Use NewQueryEngineFactory in tests Signed-off-by: Sebastian Rabenhorst <[email protected]> * Avoid expensive log.Valuer evaluation for disallowed levels (#6322) Signed-off-by: Xiaochao Dong (@damnever) <[email protected]> * Fix inconsistent error for series limits in Store API (#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <[email protected]> * update changelog Signed-off-by: Thibault Mange <[email protected]> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <[email protected]> Signed-off-by: Thibault Mange <[email protected]> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <[email protected]> Signed-off-by: Thibault Mange <[email protected]> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <[email protected]> --------- Signed-off-by: Thibault Mange <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> * *: remove unmaintained gzip library (#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <[email protected]> * Traces sampler env var (#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <[email protected]> * Test correction Signed-off-by: shayyxi <[email protected]> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <[email protected]> * added the changelog. Signed-off-by: shayyxi <[email protected]> * ran make docs to fix the build failure. Signed-off-by: shayyxi <[email protected]> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <[email protected]> * fixed review comments. Signed-off-by: shayyxi <[email protected]> --------- Signed-off-by: shayyxi <[email protected]> Signed-off-by: Shazi <[email protected]> Co-authored-by: shayyxi <[email protected]> * query: use storepb.SeriesServer (#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <[email protected]> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <[email protected]> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345) Signed-off-by: aimuz <[email protected]> * Return grpc code resource exhausted for byte limit error (#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <[email protected]> * fix lint Signed-off-by: Ben Ye <[email protected]> * update partial response strategy Signed-off-by: Ben Ye <[email protected]> * fix limit Signed-off-by: Ben Ye <[email protected]> * try to fix tests Signed-off-by: Ben Ye <[email protected]> * fix test error message Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349) Signed-off-by: Alban HURTAUD <[email protected]> * Optimize generating store match reason (#6355) Removes reason generation when debug logging is not enabled. Signed-off-by: Filip Petkovski <[email protected]> * Updates busybox SHA (#6365) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * Query: Add +Inf bucket to query duration metrics (#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <[email protected]> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <[email protected]> --------- Signed-off-by: Jacob Baungard Hansen <[email protected]> * Check context when expanding postings (#6363) * check context when expanding postings Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * ui: only keep name in store_matches param (#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <[email protected]> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <[email protected]> * Optimizing "grafana generated" regex matchers (#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <[email protected]> * fixing native histogram tests Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * Cache: various index cache client improvements (#6374) * e2e(query): Reproduce dedup issue from #6257 Signed-off-by: Douglas Camata <[email protected]> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <[email protected]> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <[email protected]> * Improve query dedup test Signed-off-by: Douglas Camata <[email protected]> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <[email protected]> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <[email protected]> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <[email protected]> * Fix receive drawing Signed-off-by: Douglas Camata <[email protected]> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <[email protected]> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <[email protected]> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <[email protected]> * Query Explanation (#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <[email protected]> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <[email protected]> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <[email protected]> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <[email protected]> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <[email protected]> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <[email protected]> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <[email protected]> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <[email protected]> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <[email protected]> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <[email protected]> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <[email protected]> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <[email protected]> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <[email protected]> * Update UI assets Signed-off-by: Pradyumna Krishna <[email protected]> --------- Signed-off-by: Pradyumna Krishna <[email protected]> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <[email protected]> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Put back the correct makefile Signed-off-by: Douglas Camata <[email protected]> * add de-cix as adopter (#6386) Signed-off-by: Raul Garcia Sanchez <[email protected]> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <[email protected]> * Add back line break at end of makefile Signed-off-by: Douglas Camata <[email protected]> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <[email protected]> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <[email protected]> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <[email protected]> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <[email protected]> * Reuse nginx image name Signed-off-by: Douglas Camata <[email protected]> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * [chore] Updating Query Engine and Prometheus (#6392) * Updating Query Engine Signed-off-by: Alan Protasio <[email protected]> * fix prometheus breaking change Signed-off-by: Alan Protasio <[email protected]> * Update prometheus with https://github.com/prometheus/prometheus/pull/12387 Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777) Signed-off-by: haanhvu <[email protected]> * check context cancel when doing posting batches (#6396) Signed-off-by: Ben Ye <[email protected]> * Expose store gateway query stats in series response hints (#6352) * expose query stats hints Signed-off-by: Ben Ye <[email protected]> * update Signed-off-by: Ben Ye <[email protected]> * add query stats hints in result Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * add merge method Signed-off-by: Ben Ye <[email protected]> * fix unit test Signed-off-by: Ben Ye <[email protected]> modify hints proto Signed-off-by: Ben Ye <[email protected]> fix unit test Signed-off-by: Ben Ye <[email protected]> update format Signed-off-by: Ben Ye <[email protected]> * update comments Signed-off-by: Ben Ye <[email protected]> * try again Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * receive: make az aware ketama hashring (#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <[email protected]> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <[email protected]> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <[email protected]> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <[email protected]> --------- Signed-off-by: Alexander Rickardsson <[email protected]> Signed-off-by: Michael Hoffmann <[email protected]> Co-authored-by: Michael Hoffmann <[email protected]> * Reformat asciiflow chart Signed-off-by: Douglas Camata <[email protected]> * Proposal: query path tenancy (#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <[email protected]> * Update proposal after initial feedback Signed-off-by: Douglas Camata <[email protected]> * Add cool picture Signed-off-by: Douglas Camata <[email protected]> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <[email protected]> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <[email protected]> * Improve writing in "How" section Signed-off-by: Douglas Camata <[email protected]> * Fix owner profile link Signed-off-by: Douglas Camata <[email protected]> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <[email protected]> Signed-off-by: Douglas Camata <[email protected]> * Address few more PR review comments Signed-off-by: Douglas Camata <[email protected]> * Address feedback on flag name text Signed-off-by: Douglas Camata <[email protected]> * Update diagram Signed-off-by: Douglas Camata <[email protected]> * Improve non-goals text Signed-off-by: Douglas Camata <[email protected]> * Update diagram Signed-off-by: Douglas Camata <[email protected]> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <[email protected]> Signed-off-by: Douglas Camata <[email protected]> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <[email protected]> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <[email protected]> * Add an action plan Signed-off-by: Douglas Camata <[email protected]> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <[email protected]> * Fix typo Co-authored-by: Giedrius Statkevičius <[email protected]> Signed-off-by: Douglas Camata <[email protected]> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <[email protected]> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <[email protected]> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <[email protected]> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <[email protected]> * Improve action plan structure and writing Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> Co-authored-by: Filip Petkovski <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]> * Fix double-counting bug in http_request_duration metric (#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <[email protected]> * Update Changelog Signed-off-by: 4orty <[email protected]> --------- Signed-off-by: 4orty <[email protected]> * Replace group with resolution in compact metrics. Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: https://github.com/thanos-io/thanos/issues/5841 Signed-off-by: SuperQ <[email protected]> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <[email protected]> * Run go mod tidy Signed-off-by: Douglas Camata <[email protected]> * Updates busybox SHA (#6403) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * Fix series stats merge (#6408) * fix series stats merge Signed-off-by: Ben Ye <[email protected]> * update license header Signed-off-by: Ben Ye <[email protected]> * use reflect Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Receive: allow unlimited head_series_limit tenants (#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: https://github.com/thanos-io/thanos/issues/6393 Signed-off-by: Jacob Baungard Hansen <[email protected]> * expose downloaded data size in query hints (#6409) Signed-off-by: Ben Ye <[email protected]> * maintainers: add myself to triagers (#6414) Signed-off-by: Michael Hoffmann <[email protected]> * Add `@douglascamata` to triagers (#6418) Signed-off-by: Douglas Camata <[email protected]> * Add Blog (#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <[email protected]> * Add Headers to blog Signed-off-by: Pradyumna Krishna <[email protected]> * Lint blog Signed-off-by: Pradyumna Krishna <[email protected]> --------- Signed-off-by: Pradyumna Krishna <[email protected]> * blog: Fix images for LFX post (#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <[email protected]> * fix lint Signed-off-by: Saswata Mukherjee <[email protected]> --------- Signed-off-by: Saswata Mukherjee <[email protected]> * Index Cache: Change cache key for postings (#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <[email protected]> * add changelog Signed-off-by: Ben Ye <[email protected]> * update code back Signed-off-by: Ben Ye <[email protected]> * add colon Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * fix another test Signed-off-by: Ben Ye <[email protected]> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <[email protected]> * address required comments Signed-off-by: Ben Ye <[email protected]> * fix compression scheme name Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Receive: upgrading logs for failed uploads to error (#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <[email protected]> * docs: added changelog entry Signed-off-by: Victor Fernandes <[email protected]> --------- Signed-off-by: Victor Fernandes <[email protected]> * fix postings test Signed-off-by: Ben Ye <[email protected]> * Add aiven as adopter... more soon! Signed-off-by: Jonah Kowall <[email protected]> * First draft of the Aiven adoption story Signed-off-by: Jonah Kowall <[email protected]> * Add aiven as adopter... more soon! (#6430) Signed-off-by: Jonah Kowall <[email protected]> * Report gRPC connnection errors to the caller (#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <[email protected]> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <[email protected]> * docs: add changelog Signed-off-by: Eng Zer Jun <[email protected]> --------- Signed-off-by: Eng Zer Jun <[email protected]> * Expose estimated chunk and series size as configurable options (#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <[email protected]> * fix lint Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Receive: make tsdb stats limit configurable (#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <[email protected]> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <[email protected]> --------- Signed-off-by: Michael Hoffmann <[email protected]> * *: wire new Engine/Explain fields in query-frontend (#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <[email protected]> * index cache: Cache expanded postings (#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * fix Signed-off-by: Ben Ye <[email protected]> * fix lint Signed-off-by: Ben Ye <[email protected]> * rebase main and added compression name to key Signed-off-by: Ben Ye <[email protected]> * update key Signed-off-by: Ben Ye <[email protected]> * add e2e test for memcached Signed-off-by: Ben Ye <[email protected]> * fix cache config Signed-off-by: Ben Ye <[email protected]> * address review comments Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * add approximate series size to index stats (#6425) Signed-off-by: Ben Ye <[email protected]> * index stats: fix chunk size calculation (#6424) Signed-off-by: Ben Ye <[email protected]> * Remove some unused Cortex vendored code and metrics (#6440) * Avoid direct cortex vendor dependency in promclient (#6443) * use own explanation struct Signed-off-by: Ben Ye <[email protected]> * omit empty Signed-off-by: Ben Ye <[email protected]> * fix e2e test Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351) Fixes: #6280 Signed-off-by: Jonathan Davies <[email protected]> * Updates busybox SHA (#6448) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * store: read postings directly into delta encoded format (#6442) Instead of allocating bytes for raw postings, let's read them directly into diff varint format to save memory. Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: Add Index Stats to block metadata (#6441) * compactor set index stats on block Signed-off-by: Ben Ye <[email protected]> * comment Signed-off-by: Ben Ye <[email protected]> * change field Signed-off-by: Ben Ye <[email protected]> * update downsample Signed-off-by: Ben Ye <[email protected]> * add index stats into compaction Signed-off-by: Ben Ye <[email protected]> * fix tests Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * estimate block chunk and series size from metadata (#6449) Signed-off-by: Ben Ye <[email protected]> * Adding graphic for blog showing architecture Signed-off-by: Jonah Kowall <[email protected]> * Fixing mermaid graph Signed-off-by: Jonah Kowall <[email protected]> * Fixing mermaid graph again Signed-off-by: Jonah Kowall <[email protected]> * Remove image and fix mermaid for other flowchart Signed-off-by: Jonah Kowall <[email protected]> * Fixing prod flowchart Signed-off-by: Jonah Kowall <[email protected]> * Fixing prod flowchart title Signed-off-by: Jonah Kowall <[email protected]> * Fixing md table Signed-off-by: Jonah Kowall <[email protected]> * Fixing md table again Signed-off-by: Jonah Kowall <[email protected]> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <[email protected]> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <[email protected]> * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <[email protected]> * Some corrections on text and links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing links Signed-off-by: Jonah Kowall <[email protected]> * Fixing diagams Signed-off-by: Jonah Kowall <[email protected]> * Fixing currencies Signed-off-by: Jonah Kowall <[email protected]> * Fixing diagams Signed-off-by: Jonah Kowall <[email protected]> * Fixing images Signed-off-by: Jonah Kowall <[email protected]> * Website: Fix blog images width (#6450) * Set blog post images width to 100% Blog Post images width isn't set and hence the images doesn't get fit to the page. Setting width to 100% make sure images fit to the web page and screen size. Signed-off-by: Pradyumna Krishna <[email protected]> * Remove width constraint from blog images Signed-off-by: Pradyumna Krishna <[email protected]> --------- Signed-off-by: Pradyumna Krishna <[email protected]> * Cache calculated mint and maxt for each remote engine (#6458) The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times on each remote engine. These calculations can get expensive when an engine covers a large amount of TSDBs. This commit introduces caching of the calculated values for a remote engine's mint, maxt and label sets. Signed-off-by: Filip Petkovski <[email protected]> * cache empty expanded postings (#6464) Signed-off-by: Ben Ye <[email protected]> * mixin/receive: add limits alerting (#6466) * add some thanos receive limits alerts Signed-off-by: Thibault Mange <[email protected]> * update changelog Signed-off-by: Thibault Mange <[email protected]> * fix and generate examples Signed-off-by: Thibault Mange <[email protected]> * quick fix Signed-off-by: Thibault Mange <[email protected]> * quick fix Signed-off-by: Thibault Mange <[email protected]> * fix test Signed-off-by: Thibault Mange <[email protected]> * fix invalid changelog Signed-off-by: Thibault Mange <[email protected]> --------- Signed-off-by: Thibault Mange <[email protected]> * Update docs/blog/2023-06-08-thanos-at-aiven.md Co-authored-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Jonah Kowall <[email protected]> * Add retention for M3 Signed-off-by: Jonah Kowall <[email protected]> * Add image versus mermaid diagram. Couldn't comment it out unfortunately Signed-off-by: Jonah Kowall <[email protected]> * Adding text around link to fix linter Signed-off-by: Jonah Kowall <[email protected]> * make docs Signed-off-by: Jonah Kowall <[email protected]> * add alert for tenant reaching head series limit (#6467) Signed-off-by: Thibault Mange <[email protected]> * go.mod: bump rest of otel libs (#6447) * go.mod: bump rest of otel libs Bump these two OTEL libs to be like the rest of the OTEL libs. We had a strange issue where `service.name` was not set even though it was in environment variables. Bumping these libs fixes the problem. Signed-off-by: Giedrius Statkevičius <[email protected]> * otlp: only set service.name if provided Signed-off-by: Giedrius Statkevičius <[email protected]> * e2e: add test case Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * store: disable pooling for postings benchmarks (#6473) Pooling hides the real allocations in postings benchmarks because the allocation only happens once really. In reality this is different when multiple readers are created at once. Disable pooling in benchmarks to understand what is happening. Signed-off-by: Giedrius Statkevičius <[email protected]> * Fixing date and author for Aiven blog (#6476) * Fixing authors Signed-off-by: Jonah Kowall <[email protected]> * make docs and fixing date Signed-off-by: Jonah Kowall <[email protected]> * Fixing authors Signed-off-by: Jonah Kowall <[email protected]> --------- Signed-off-by: Jonah Kowall <[email protected]> * Check context when expanding postings 2nd attempt (#6471) * check context when expanding postings Signed-off-by: Ben Ye <[email protected]> * import Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * [FEAT] Add thanos ruler blog post (#6477) Signed-off-by: Hélia Barroso <[email protected]> * blog: Small fixes to Aiven article (#6481) * Correct some links and typos in 2023-06-08-thanos-at-aiven.md The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links. Also fixed a few other typos I spotted. Signed-off-by: Hugh Cole-Baker <[email protected]> * Update 2023-06-08-thanos-at-aiven.md Clarify a couple of things in the final paragraph. Signed-off-by: Hugh Cole-Baker <[email protected]> --------- Signed-off-by: Hugh Cole-Baker <[email protected]> * Rename series and postings fetch duration metrics (#6479) * fix wrong metric names for series and postings fetch duration Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * note breaking change Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Update to prometheus/common v0.44.0 (#6483) Signed-off-by: Dan Bason <[email protected]> * go.mod: update rueidis client (#6485) The latest improvements have great perf improvements. Let's update it. Thank you to [rueian](https://github.com/rueian) and all of the contributors to that awesome Redis client! Signed-off-by: Giedrius Statkevičius <[email protected]> * Introduce tenancy package (#6482) This commit introduces a tenancy package in preparation for introduction of tenancy in the query path. Tenancy related code which can be common between components are moved out of the receive component into the tenancy package. Signed-off-by: Jacob Baungard Hansen <[email protected]> * Ignore nginx.com links in mdox (#6489) Signed-off-by: Saswata Mukherjee <[email protected]> * blog: Update Aiven blog post (#6487) Wording changes and better formatting for the tables. Signed-off-by: Simon Brüggen <[email protected]> * store: optimized snappy streamed reading (#6475) Optimize snappy streamed reading by traversing through the byte slice in advance to determine how big of a buffer we will need. I have opted to rewrite snappy streamed format decoding because it is a straightforward format. Complex parts were deferred to klauspost/compress. Signed-off-by: Giedrius Statkevičius <[email protected]> * Remove Exists call in meta fetcher (#6474) * Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <[email protected]> * Revert e2e test change Signed-off-by: Filip Petkovski <[email protected]> --------- Signed-off-by: Filip Petkovski <[email protected]> * e2e/store: use now instead of time.Now() each time (#6493) Consistently use the same timestamp instead of the current time each time during a test. Signed-off-by: Giedrius Statkevičius <[email protected]> * [FIX] Documentation issue (#6494) Signed-off-by: Hélia Barroso <[email protected]> * Updates busybox SHA (#6497) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * chore: pkg imported more than once (#6499) * chore: pkg imported more than once Signed-off-by: guoguangwu <[email protected]> * fix: remove pkg alias Signed-off-by: guoguangwu <[email protected]> --------- Signed-off-by: guoguangwu <[email protected]> * *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496) * Fix and improve PathContentReloader tests Signed-off-by: Douglas Camata <[email protected]> * Run tests in parallel Signed-off-by: Douglas Camata <[email protected]> * Add changelog entry Signed-off-by: Douglas Camata <[email protected]> * Fix table tests Signed-off-by: Douglas Camata <[email protected]> * Fix typo Co-authored-by: Filip Petkovski <[email protected]> Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> Co-authored-by: Filip Petkovski <[email protected]> * add context check when decoding cached postings (#6506) Signed-off-by: Ben Ye <[email protected]> * *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503) * Add fallback config reload for symlinks Signed-off-by: Douglas Camata <[email protected]> * Improve pooling config reload Signed-off-by: Douglas Camata <[email protected]> * Make tests more reliable Signed-off-by: Douglas Camata <[email protected]> * Improve tests again Assert on runutil.Repeat output Signed-off-by: Douglas Camata <[email protected]> * Pass the debounce/reload time to each test Signed-off-by: Douglas Camata <[email protected]> * Add comments and ensure interfaces are implemented Signed-off-by: Douglas Camata <[email protected]> * Make the limit configuration reload time customizable Signed-off-by: Douglas Camata <[email protected]> * goimports file Signed-off-by: Douglas Camata <[email protected]> * Fix lint warning Signed-off-by: Douglas Camata <[email protected]> * Improve log for polling engine Signed-off-by: Douglas Camata <[email protected]> * Fix tests Signed-off-by: Douglas Camata <[email protected]> * Make linter happy Signed-off-by: Douglas Camata <[email protected]> * Extract symlink identification to a separate function Signed-off-by: Douglas Camata <[email protected]> * Update changelog Signed-off-by: Douglas Camata <[email protected]> * Fix links to go-grpc-middleware after v2 merge into main Signed-off-by: Douglas Camata <[email protected]> * Add period to make linter happy. Signed-off-by: Douglas Camata <[email protected]> * Update changelog entry Signed-off-by: Douglas Camata <[email protected]> * Replace the fsnotify based engine from `PathContentReloader` with the polling one Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Remove check for empty filePath Signed-off-by: Douglas Camata <[email protected]> * Unexpose pollingEngine.Start Signed-off-by: Douglas Camata <[email protected]> * Fix pollingEngine doc-comment Signed-off-by: Douglas Camata <[email protected]> * Log file path when config is reloaded Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * receive: add float histogram support (#6323) * Added receive float histogram support Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed imports Signed-off-by: Sebastian Rabenhorst <[email protected]> * Added comments for fns copied from Prometheus Signed-off-by: Sebastian Rabenhorst <[email protected]> Improved comment Signed-off-by: Sebastian Rabenhorst <[email protected]> * Removed unnecessary if Signed-off-by: Sebastian Rabenhorst <[email protected]> * Fixed native histogram proto conversion in remote engine Signed-off-by: Sebastian Rabenhorst <[email protected]> * Removed unused histogram conversion from remote engine Signed-off-by: Sebastian Rabenhorst <[email protected]> * Fix and renaming in native_histograms_test.go Signed-off-by: Sebastian Rabenhorst <[email protected]> * Trigger Build Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]> * go.mod: bump e2e framework version (#6516) Bump e2e framework version and add a note about E2E_DOCKER_CPUS. Signed-off-by: Giedrius Statkevičius <[email protected]> * Add changelog for #6474 (#6518) Signed-off-by: Filip Petkovski <[email protected]> * cortex/redisclient: use rueidis client (#6520) * cortex/redisclient: use rueidis client Use the same rueidis client in query-frontend. Solves https://github.com/thanos-io/thanos/issues/6094. Signed-off-by: Giedrius Statkevičius <[email protected]> * cortex/cache: gofumpt + fix errors.Errorf Signed-off-by: Giedrius Statkevičius <[email protected]> * cacheutil/docs: clean up more old stuff Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * index header: Remove memWriter from fileWriter (#6509) * index header: remove memWriter from fileWriter Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> * refactor Signed-off-by: Ben Ye <[email protected]> * fix test Signed-off-by: Ben Ye <[email protected]> * update comment Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Wrap object store Prometheus registry (#6152) * Wrap object store Prometheus registery In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26. Signed-off-by: Douglas Camata <[email protected]> * Update objstore to the latest version This version removes the `thanos_` prefix from metrics, which is the reason for wrapping the objstore's metrics registry in the first place. Signed-off-by: Douglas Camata <[email protected]> * Fix modules Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Wrap metrics registerer for objstore bucket Signed-off-by: Douglas Camata <[email protected]> * Remove prefix from Thanos Store metrics Signed-off-by: Douglas Camata <[email protected]> * Fix goimports Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Put back upgraded objstore dep Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * Move to more recent ref of thanos/objstore Signed-off-by: Douglas Camata <[email protected]> * Ignore OBS in objstore tests Signed-off-by: Douglas Camata <[email protected]> * Fix linting error on main test after objstore upgrade Signed-off-by: Douglas Camata <[email protected]> * Skip OCS objstore test in circle ci Signed-off-by: Douglas Camata <[email protected]> * Fix echo in makefile Signed-off-by: Douglas Camata <[email protected]> * Fix typo Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * e2e/store: try to fix Series() limit test again (#6522) I finally managed to reproduce this failure locally with http…
|
@fpetkovski Seeing how a bunch of issues cropped up, how complicated would it be to revisit this change and introduce a hidden flag to be able to switch between the original and the revised behaviour? |
|
@mfoldenyi I would be okay with reverting this since it caused more issues than I expected. It also solves a problem which many people do not seem to have. I will raise a PR next week. |
|
@fpetkovski We don't have to revert the behavior entirely. Can we add a flag to select between current behavior and the old behavior? With https://github.com/thanos-io/thanos/blob/main/pkg/block/fetcher.go#L174 interface it should be possible to do so. |
|
Sounds good, we can also do that. I'll raise a PR. |
2 tasks
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers.
This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file.
Changes
Verification