(Calculated fields) Add calculated-field predicates to QueryAst - #6777
fulmicoton wants to merge 3 commits into
Conversation
fulmicoton
changed the title
5e6e6c7 to
7695065
Compare
7695065 to
2187a7a
Compare
6a00966 to
478a291
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 478a291907
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| boost: NotNaNf32, | ||
| }, | ||
| Cache(CacheNode), | ||
| CalcField(CalcFieldQuery), |
There was a problem hiding this comment.
This adds a new publicly serialized QueryAst variant (type: calc_field), but a repository-wide search of docs/, config/, and the README finds no user-facing description of its expression grammar, supported field types, or missing-value behavior. Downstream callers therefore cannot reliably construct or reason about this new query behavior; add it to the native-query/search API documentation.
AGENTS.md reference: AGENTS.md:L23-L24
Useful? React with 👍 / 👎.
adb1594 to
4510c0d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e42805d961
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| &self, | ||
| _context: &BuildTantivyAstContext, | ||
| ) -> Result<TantivyQueryAst, InvalidQuery> { | ||
| let predicate = JitExprPredicate::new(self.expression.clone()) |
There was a problem hiding this comment.
When an expression references an unmapped field in Quickwit's default dynamic mode, such as (GT duration 2i64), the physical fast-field column is _dynamic.duration: normal query paths use find_field_or_hit_dynamic to perform this fallback. This implementation ignores the schema context and passes duration unchanged to Tantivy, while the warmup visitor also requests the unchanged name, so the predicate binds a missing column and returns no matches for valid dynamically mapped data. Rewrite calculated-field variables to their resolved split-schema column names before constructing and warming the predicate.
Useful? React with 👍 / 👎.
| const DEFAULT_EXPR_COMPILATION_CACHE_CAPACITY: usize = 256; | ||
| let capacity: usize = quickwit_common::get_from_env( | ||
| "QW_EXPR_COMPILATION_CACHE_CAPACITY", | ||
| DEFAULT_EXPR_COMPILATION_CACHE_CAPACITY, |
There was a problem hiding this comment.
This introduces the operator-facing QW_EXPR_COMPILATION_CACHE_CAPACITY setting and its default of 256, but a repository-wide search of docs/ and config/ finds no entry describing it, its units, or when it is read. Operators therefore cannot discover or safely tune the process-wide JIT cache without reading the source; add this environment variable to the searcher/node configuration documentation.
AGENTS.md reference: AGENTS.md:L23-L24
Useful? React with 👍 / 👎.
e42805d to
73c08ba
Compare
This adds the capability in Quickwit's QueryAST to match a jitexpr predicate within the query (so-called calculated fields in pomsky). The PR just plugs in a new feature from tantivy.
Add QueryAst::CalcField(CalcFieldQuery) with a jitexpr expression serialized as a string, including visitor/transformer support and user-query parsing passthrough. The list like jitexpr language is part of tantivy.
The query ast then converts to Tantivy's JitExprPredicate / DocPredicateQuery
Collect referenced fast fields for warmup, including string dictionaries through the existing whole-column warmup path. Preserve exact field paths and skip cache-hit subtrees.
Ast part of that effort, I refactored the different visitor collecting fast fields that needed to be downloaded for query matching into a single visitor.
Example:
{
"type": "calc_field",
"expression": "(GT custom.duration 1i64)"
}
73c08ba to
8416e8b
Compare
3 participants
Description
This adds the capability in Quickwit's QueryAST to match a jitexpr predicate within the query (so-called calculated fields in pomsky). The PR just plugs in a new feature from tantivy.
QueryAst::CalcField(CalcFieldQuery)with a jitexpr expression serialized as a string, including visitor/transformer support and user-query parsing passthrough. The list like jitexpr language is part of tantivy.JitExprPredicate/DocPredicateQueryAst part of that effort, I refactored the different visitor collecting fast fields that needed to be downloaded for query matching into a single visitor.
Example:
{ "type": "calc_field", "expression": "(GT custom.duration 1i64)" }