Repository navigation
Fix LIKE/GLOB in GTMSQLite to backtrack after a wildcard - #616
Open
IncognitoQuack wants to merge 1 commit into
Open
IncognitoQuack wants to merge 1 commit into
IncognitoQuack wants to merge 1 commit into
Conversation
The CFString implementations of LIKE and GLOB match each pattern
element at the first place it fits and never reconsider. When the rest
of the pattern then fails, the matchAll ("%" for LIKE, "*" for GLOB)
ahead of it is never allowed to consume more of the target string, so
matches that need a later occurrence are reported as non-matches:
SELECT 'banana' LIKE '%na'; -- CFAdditions 0, SQLite 1
SELECT 'abab' LIKE 'a%b'; -- CFAdditions 0, SQLite 1
GTMSQLite.h:64 says SQLite semantics are retained wherever reasonable,
and GTMSQLite.m:1020 describes LikeGlobCompare() as a reimplementation
of patternCompare() in func.c, which recurses at matchAll precisely to
get this backtracking. Only false negatives are produced; nothing that
matched before stops matching.
Record the pattern and string position at each matchAll, and on failure
hand it one more composed character sequence and retry the rest of the
pattern from there. Failures that backtracking cannot help -- the
string running out, or an unanchored search that already covered the
whole remainder -- stay final, which is the same pruning SQLite's
SQLITE_NOWILDCARDMATCH gives it. Because the recorded position only
ever moves forward, a walk backtracks at most as many times as the
target string is long.
Diffed against the built-in SQLite LIKE/GLOB over 555,982 exhaustively
enumerated pattern/target pairs: 19,143 change, every one of them from
a wrong answer to the answer SQLite gives, and no answer that was
already correct changes. A further 7.2M randomised pairs agree
exactly. Existing testLikeGlobCFAdditions queries return identical
rows before and after.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The contract
Foundation/GTMSQLite.h:62-65says the CFString-based LIKE and GLOB givebetter handling of case and composed character sequences, but that
"whereever reasonable, SQLite semantics have been retained", and then
enumerates the specific places they deliberately differ. Greedy matching
is not one of them.
Foundation/GTMSQLite.m:1019-1021is more explicit still:LikeGlobCompare()is described as "essentially a reimplementation of
patternCompare()infunc.c of the SQLite sources".
patternCompare()recurses at the matchAllcharacter, trying every position at which the rest of the pattern could
start, and that recursion is the whole reason it is correct.
What the code does instead
LikeGlobCompare()walks the pattern once and never reconsiders. Eachpattern element is matched at the first place it fits (
CFStringFindWithOptions/
CFStringFindCharacterFromSetreturn the earliest match), the walkcommits to that position, and if the rest of the pattern then fails it
reports a non-match. The matchAll ahead of it (
%for LIKE,*for GLOB)is never given the chance to consume more of the target string, so any
match that needs a later occurrence of what follows the wildcard is missed.
'banana' LIKE '%na'is the whole bug in one line:%matches nothing,nais found at offset 2, the walk ends at offset 4 withbanananotconsumed to the end, and the answer is 0. SQLite answers 1, with
%absorbing
bana.Repro
At
491a17e:With this change all six agree with SQLite.
These are not exotic inputs. The condition is just "the text the pattern
looks for after the wildcard occurs more than once in the value", which is
ordinary for suffix queries over paths, hostnames, filenames and any
repetitive text.
%is in the pattern of essentially every LIKE queryanyone writes.
The reason it went unnoticed is that the existing coverage cannot see it.
Every row in
t1intestLikeGlobCFAdditionscontains what the patternssearch for at most once --
'%bcd'is tested againstabcdandbcd,'ab%d'againstabcdandabd-- so the first match is always also theonly match, and the greedy shortcut is indistinguishable from a correct
matcher.
The fix
At each matchAll, record the pattern index just past it and how much of the
target string it has consumed. When the rest of the pattern fails, hand the
matchAll one more composed character sequence and resume the walk from the
recorded pattern index. That is what
patternCompare()gets from recursing.Failures that backtracking provably cannot help stay final:
of it -- every position in a retry is at or past where it was, so these
can only get worse;
rest of the string, so no later starting point can do better.
This is the same pruning SQLite gets from
SQLITE_NOWILDCARDMATCH, and itis what keeps non-matching multi-wildcard patterns cheap. On a successful
unanchored search the skipped region is also handed to the matchAll
immediately, since no element can start there, rather than being rediscovered
one character at a time on each backtrack.
The recorded position only ever moves forward, so a walk backtracks at most
as many times as the target string is long: no exponential blowup.
Evidence
Differential testing against this same database's built-in LIKE/GLOB
(
initInMemoryWithCFAdditions:NO), which is the reference implementationthe header says is being followed.
Exhaustive. 555,982 pattern/target pairs, enumerating all patterns up to
length 5 over
{a, b, %, _}and{a, b, *, ?}, patterns over alphabetsincluding
[ab],[^a],[a-c]and[]a], an ESCAPE-clause suite, aprecomposed/decomposed suite, and both the UTF-8 (
Like8/Glob8) andUTF-16 (
Like16/Glob16) entry points:Before the change every one of those 19,143 was a false negative; there
were no false positives, which is the expected signature of a sound but
incomplete matcher.
Randomised. 7.2M further pairs over random patterns (length 0-8,
including character sets) and random targets (length 0-11), across LIKE,
GLOB and both text encodings: 0 mismatches. 0 mismatches under
-fsanitize=address,undefinedas well.No regression in the existing tests. Replaying every query in
testLikeGlobCFAdditionsoperation-for-operation, including thesetLikeComparisonOptions:/ index create/drop sequence, on both the UTF-8and UTF-16 databases, produces byte-identical result sets before and after
(72 result lines).
Failing before, passing after. The nine cases added to
testLikeGlobCFAdditionsall fail at491a17eand pass with this change,on both the UTF-8 and UTF-16 databases. They use a separate table so no
existing expectation moves.
Cost. No measurable change over the 555,982-pair corpus (1.77s / 1.80s
before, 1.78s / 1.80s after). Worst case I could construct is a 64,000-
character target with a pattern like
%athat has to walk the whole string:2.5ms, against 0.24ms for built-in SQLite -- and the old code was returning
the wrong answer. Without the two pruning rules above the same case took
5.1s, which is why they are there.
Caveats
the Command Line Tools, so there is no XCTest:
xcodebuild ... build testfor
GTM.xcodeprojandGTMiPhone.xcodeprojandpod lib lintwere allimpossible.
GTMSQLiteis not inPackage.swiftorBUILD, so the Xcodeworkflow is the only one that covers it, and I am relying on CI for it.
Everything above was produced by compiling
Foundation/GTMSQLite.mdirectly with clang into standalone harnesses; the modified
GTMSQLiteTest.mwas checked withclang -fsyntax-only -Wall -Wextraagainst a stub XCTest header, and everything was re-verified by applying
the patch to a fresh clone of
main.than a stack. That is the standard greedy-wildcard argument: each earlier
segment is already matched at the earliest position it can be, so moving
one later never helps. I have exercised that empirically rather than
proved it here, which is what the exhaustive multi-wildcard suites above
(patterns up to length 5 over alphabets containing two wildcard
characters) are for.
LikeGlobCompare(). Two other places where theCF implementations differ from SQLite are visible in the corpus and are
deliberately left alone as separate concerns: a malformed character set
(
'a[') raises an error where SQLite returns 0, which the existing testsassert; and the UTF-16 entry points reject an empty pattern or empty
target. Both are byte-for-byte unchanged by this patch.