fix(postgres): back fulltext indexes with a GIN tsvector index - #993
Open
HarshMN2345 wants to merge 1 commit into
Open
HarshMN2345 wants to merge 1 commit into
HarshMN2345 wants to merge 1 commit into
Conversation
Fulltext indexes were created as plain btrees on the column, which search queries could never use and which reject values above 2704 bytes. The search expression called to_tsvector without a config, so it was not IMMUTABLE and could not be indexed either. Use an explicit 'english' config (the default_text_search_config of the official images) in both the search query and a GIN expression index, so the index matches the query and search results stay the same.
Contributor
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 46 minutes. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs appwrite/appwrite#14081
On Postgres,
INDEX_FULLTEXTwas created as a plain btree on the column, soQuery::searchcould not use it. The search expression calledto_tsvectorwithout a config, which is not IMMUTABLE, so it could not be indexed at all. The btree also rejects values above 2704 bytes, which is why saving users with many labels fails (_key_search).This change uses an explicit config in both
search/notSearchand a GIN expression index built from the same helper, so the two expressions always match. A multi-column fulltext index gets one expression per column. Searches only ever use single-attribute fulltext indexes, and a multi-column GIN index can serve a condition on any one of its columns.Why
englishrather thansimple: the search used to rely ondefault_text_search_config, which ispg_catalog.englishinappwrite/postgresand the official postgres images. Pinningenglishkeeps the same stemming and stop words, so results don't change (for example,roomstill matchesrooms; withsimpleit matched 0 of 50k rows). Servers set up with a different default would now get English matching. A per-index config could come later.Migration: fulltext indexes that already exist stay btrees, and from now on
searchwill not use them. They need to be dropped and recreated, ideally withCREATE INDEX CONCURRENTLYon large tables. This PR doesn't include migration tooling.Verification (appwrite/postgres:0.1.0, PG 18.3, 50k rows; plans captured with auto_explain from
Database::find):USING btree (note).search('note','zebra')ran as a pkey index scan that filtered every row (24,975 rows removed, 83 ms). Inserting a 3.7 KB value failed withindex row size 3712 exceeds btree version 4 maximum 2704.USING gin (to_tsvector('english'::regconfig, regexp_replace(note, ...))). Same search: Bitmap Index Scan on the fulltext index (0.2 ms). The 3.7 KB insert succeeds.search('note','room')still matches rows containingrooms.PostgresTeste2e: 692 tests, the same single pre-existingtestCacheFallbackfailure before and after.