Skip to content

Latest commit

 

History

History
305 lines (265 loc) · 40.7 KB

PARITY.md

File metadata and controls

305 lines (265 loc) · 40.7 KB

Parity with the JS client

What the Rust client (zero-client) has ported from the JS client, how it is checked, and where it deliberately differs. The conformance runner (cargo run -p zero-conformance) prints the live numbers; the table below is updated with each milestone.

Ported

Area JS source Rust Vectors
JSON parse and stringify JSON.parse, JSON.stringify, property order json json-parse, json-stringify-strings, sweep-json, harden-json, generated/harden-json-fuzz
JS numbers Number#toString, Number(string) json::number number-format, number-parse
String orders compareUTF8, JS < strings string-compare
Hashes packages/shared/src/hash.ts hash::xxhash32 hash
Value comparison packages/zql/src/ivm/data.ts compare compare-values
Row keys packages/zero-client/src/client/keys.ts keys primary-key-string, aggregate-row-key
LIKE and ILIKE packages/zql/src/builder/like.ts like like
TTL packages/zql/src/query/ttl.ts ttl ttl
AST, normalization, wire schema packages/zero-protocol/src/ast.ts ast ast, ast-features, generated/ast-fuzz
Query hashes packages/zero-protocol/src/query-hash-visitor.ts hash::query ast, query-internals-hash, name-args-hash, sweep-name-args-hash
Query builder packages/zql/src/query/query-impl.ts, expression.ts query sweep-query-hash
Name mapping packages/zero-types/src/name-mapper.ts schema::NameMapper ast-features
Wire messages packages/zero-protocol/src (up.ts, down.ts) protocol protocol-upstream, protocol-downstream, sweep-downstream, protocol-client-schema, generated/protocol-*-fuzz, generated/harden-downstream-text
Binary poke chunks decodePokeChunks protocol::poke_chunk protocol-poke-chunks, generated/protocol-poke-chunks-fuzz, generated/harden-poke-bytes
Sec-WebSocket-Protocol encodeSecProtocols, decodeSecProtocols protocol::sec_protocols protocol-sec-protocols
Connect URL createConnectionURL in zero.ts protocol::ConnectionUrl a unit test against URLSearchParams; tests/walking_skeleton.rs
Client schema clientSchemaFrom schema::Schema::client_schema tests/walking_skeleton.rs (zero-cache rejects a wrong one)
Protocol version PROTOCOL_VERSION (57) protocol::PROTOCOL_VERSION meta
IVM operators and memory source packages/zql/src/ivm ivm ivm, ivm-features, bughunt-ivm, restored-ivm, restored-ivm-rebaselined, generated/ivm-fuzz, generated/ivm-features-fuzz, generated/sweep-ivm-*, generated/harden-ivm-*, generated/restored-chinook-decpush
Pipeline builder packages/zql/src/builder/builder.ts builder the IVM vectors above
Views and change application array-view.ts, view-apply-change.ts ivm::ArrayView the IVM vectors above
Typed queries and codegen packages/zql/src/query/query.ts, packages/zerogen typed, zero-rust codegen the query vectors through AnyQuery, which typed queries build (rust/codegen-tests)
Typed CRUD in mutators tx.mutate.<table> (crud.ts) typed::TypedTableMutator, generated mutate rust/codegen-tests/tests/typed.rs; the CRUD itself is client::Transaction's, which the client vectors check
Client engine packages/zero-client/src/client (zero.ts, connection-manager.ts, query-manager.ts, zero-poke-handler.ts, mutation-tracker.ts, context.ts, ivm-branch.ts, crud-impl.ts) client, on memory storage client, client-harden-connection, client-harden-mutations, harden-queries, bughunt-client, generated/client-aggregates-fuzz
Synced aggregates and groups packages/zql/src/builder/builder.ts, crud-impl.ts builder, client::Transaction the client vectors above
WebSocket and timers the browser WebSocket, setTimeout, microtasks tokio_transport (tokio-transport feature) tests/tokio_transport.rs; tests/client_e2e.rs against zero-cache
Storage and the client lifecycle Replicache's persistence (kv/, persist/), mem-fallback-store.ts, zero-swift's ReplicaDatabase client::{ZeroStorage, StorageBackend}, SQLite behind sqlite every client file again as (sqlite); tests/sqlite_storage.rs; tests/client_e2e.rs
Inspector and query metrics packages/zero-client/src/client/inspector, metrics.ts, query-manager.ts, packages/zero-protocol/src/inspect-up.ts, inspect-down.ts, packages/zql/src/planner/planner-debug.ts client::Inspector, client::Metrics client-inspector, also as (sqlite); tests/inspector.rs
TDigest packages/shared/src/tdigest.ts tdigest tdigest
AST to ZQL packages/zero/src/ast-to-zql.ts ast::ast_to_zql ast-to-zql

Not ported yet

Within what the conformance vectors cover, nothing. zero-swift's hardening end-to-end tests are ported too: a chaos proxy between the client and zero-cache, offline replays, outages, and a seeded chaos fuzz (zero-client/tests/client_e2e), and its WebSocket transport probes (zero-client/tests/transport_probes.rs). What the JS client has that the Rust one doesn't is listed below, under "Not ported".

Coming from the JS client

What each part of @rocicorp/zero is called in Rust. Names follow JS in snake case. The client runs on one thread: Zero is neither Send nor Sync, and an app drives it on its event loop (with the tokio-transport feature, a tokio LocalSet).

Options

ZeroOptions has JS's options, in snake case. schema, context, and mutators are arguments of Zero::new, with the scheduler and the socket factory.

JS option Rust
cacheURL, auth, userID, storageKey cache_url, auth, user_id, storage_key
logLevel, logSink log_level, log_sink (a LogSink; the default is StderrLogSink)
mutateURL, mutateHeaders, queryURL, queryHeaders mutate_url, mutate_headers, query_url, query_headers
onUpdateNeeded, onClientStateNotFound, onStorageFailure on_update_needed, on_client_state_not_found, on_storage_failure
pingTimeoutMs, disconnectTimeoutMs, queryChangeThrottleMs, maxHeaderLength the same, in snake case
hiddenTabDisconnectDelay hidden_disconnect_delay_ms, with Zero::set_visible
kvStore ('idb' or 'mem') storage: ZeroStorage::Memory (the default), Sqlite, or Custom
getTraceparent get_traceparent
onOnlineChange (deprecated) connection().subscribe(listener)
server, getQueriesURL (deprecated) cache_url, query_url
batchViewUpdates batch_view_updates
slowMaterializeThreshold slow_materialize_threshold_ms (5,000, as in JS)
maxRecentQueries (deprecated) not ported: JS keeps no recent queries by default either

One default differs from JS, on purpose: storage is ZeroStorage::Memory, where JS keeps its data in IndexedDB. SQLite storage is the opt-in sqlite feature, so the default build has no third-party dependencies, and an app chooses where its replica lives (ZeroStorage::Sqlite { directory }); an app that wants its data across restarts sets it. With directory: None, each app gets a folder of its own, named for its bundle identifier or executable (on iOS and Apple's other platforms, its container is its own already), as IndexedDB keeps each origin's databases apart. hidden_disconnect_delay_ms is JS's 5 minutes; a mobile app, which the system suspends soon after it goes to the background, can set 0 to disconnect as soon as it says it is hidden, as Swift does.

The client

JS Rust
new Zero(options) Zero::new(schema, context, mutators, options, scheduler, socket_factory)?
zero.clientID, await zero.clientGroupID client_id(), client_group_id(), known at once
zero.userID, zero.schema, zero.context, zero.closed user_id(), schema(), context(), closed()
zero.materialize(query) materialize(&query, &ttl)?, a MaterializedView with data, add_listener, destroy, update_ttl
zero.run(query, {type}) run(&query, RunType, ttl), a Deferred<ViewValue> to await or read once settled
zero.preload(query) preload(&query, &Ttl::DEFAULT_PRELOAD), a Preload with cleanup, cached, and complete
zero.mutate(request) mutate(name, args), a MutatorResult whose client and server are Deferred<MutationOutcome>
zero.connection.state, .subscribe, .connect({auth}) connection().state(), .subscribe(listener), .connect(ConnectAuth)?; see below
zero.close(), zero.delete(), dropAllDatabases() close(), delete(), client::drop_all_databases(&storage), which return the DroppedDatabases
escapeLike(text) like::escape_like
zero.mutate.issue.insert(...) (CRUD mutators) not ported: write a mutator
zero.inspector inspector(), an Inspector; see below
zero.mutateBatch, zero.online, zero.server, zero.version not ported; protocol::PROTOCOL_VERSION is the sync protocol's version
none (Swift's addMutationOutcomeListener) add_mutation_outcome_listener(listener), which returns the function that removes it
none (Swift's Zero.prepare) Zero::prepare(..), then Zero::from_prepared(..)?: the SQLite store read on another thread at launch
none (Swift's pendingSettled(timeout:)) pending_settled(timeout_ms), a Deferred<bool>: whether no mutations are pending; see below

Rust adds reconnect_now (the network may be back, so skip the backoff), when_pending_settled, and flush_storage (wait for the store's queued writes, such as before the app exits).

connect({auth}) is connect(ConnectAuth::Keep) without auth, ConnectAuth::Set(token) with a token, and ConnectAuth::Clear with null. Where JS's promise rejects, such as for a token on a client without a user_id, it returns the error and changes nothing. It returns before the state changes, where JS's promise waits for the next one: subscribe follows it.

add_mutation_outcome_listener is Swift's API, with no JS counterpart: the listener hears each mutation's server outcome once, as a MutationServerOutcome, including that of a mutation replayed after its server result settled with a ZeroError when the client gave up connecting. Swift's mutationOutcomes stream isn't ported: the listener can feed a channel.

pending_settled is Swift's API too, with no JS counterpart: it waits for the server to confirm every pending mutation, as before set_visible(false), and resolves with whether none are left. It resolves at once when the server can't confirm them (the client isn't connected, it's closed, or its client group is disabled), and at timeout_ms on the client's scheduler. when_pending_settled calls a function instead, with no timeout.

When the server no longer has the client's state, JS asks the app to reload. Without on_client_state_not_found, Rust replaces the client inside the same Zero, as Swift does: a new client ID and client group on the same store, with the views and preloads moved to the new client.

The inspector

JS Rust
inspector.client, .clientGroup client(), client_group(), an InspectorClient and an InspectorClientGroup
await inspector.metrics(), .serverVersion() metrics(), server_version(), a Deferred<Metrics> and a Deferred<String>
await inspector.clients(), .clientsWithQueries() clients(), clients_with_queries(), a Deferred<Vec<InspectorClient>>
await client.queries(), .map(), .rows(table) queries(), map(), rows(table); a query is an InspectorQuery, with JS's fields
await inspector.analyzeQuery(query, options) analyze_query(&query, AnalyzeQueryOptions), a Deferred<AnalyzeQueryResult>
inspector.analyzeQuery with a server AST, or a name and arguments analyze_server_ast(&ast, options), analyze_named_query(name, args, options)
the password prompt authenticate(password), a Deferred<bool>
formatPlannerEvents(events) format_planner_events(&events)
the TDigest of query-materialization-client and the other metrics tdigest::TDigest, which reads and writes JS's JSON form

Inspector takes the context's type (Inspector<C>, () by default), since analyze_query builds a query with it. The client records JS's metrics, query-update-client sampled as JS samples it, and logs a query's end-to-end materialization as JS does, as one line: the message, the query ID, its AST as JSON, and the milliseconds, a warning past slow_materialize_threshold_ms. At the debug level it asks zero-cache for its timings (debugPerf), as JS does.

zero-cache can require its admin password. Where JS prompts for it, an RPC the server refuses fails with an error that starts with "Authentication failed" and says to call authenticate(password). A password the server accepted is presented again before the next RPC on a new connection, as JS presents its adminPassword, and forgotten when the server refuses it. An RPC waits for the connection, and fails with an error when the connection drops before the answer or the client closes, where JS's would wait for an answer that never comes.

Queries

Generated code (zero-rust codegen) has a row struct per table with Column and Relationship constants, and zql::<table>() for its typed Query. AnyQuery is the untyped builder, named by strings; a typed query wraps one (any(), into_any(), Query::from_any). materialize, run, and preload take any ZeroQuery: an AnyQuery, a typed Query, or a named query's QueryRequest, which they build with the client's context.

JS Rust
defineQueries(...), defineQuery(...) define::define_query(name, |args, ctx| query) (serde feature), called as Q.with(args); untyped, AnyQuery::name_and_args(name, args)
.where('open', true) .where_eq(Issue::OPEN, true)
.where('title', 'LIKE', '%x%') .where_(Issue::TITLE, SimpleOperator::Like, "%x%".into())
.where(({cmp, and, or, not, exists}) => ...) .where_expr(|e| e.and(vec![...])), with RowExpression
.where(({cmp, json}) => cmp(json('metadata', 'tags', 0), 'bug')) .where_expr(|e| e.json(User::METADATA, path).eq("bug".into()))
.whereExists('labels', q => ...) .where_exists(Issue::LABELS, ExistsOptions::default(), |q| ...)
.related('comments', q => q.limit(10)) .related(Issue::COMMENTS, |q| q.limit(10))
.orderBy('created', 'desc'), .limit(n), .start(row), .one() .order_by(Issue::CREATED, Direction::Desc), .limit(n), .start(&row, inclusive), .one()
.count(), .sum, .avg, .min, .max the same, with _json forms for a JSON path; count is an i64
.related('comments', q => q.count()) .related(Issue::COMMENTS, |q| q.count()), read as row.aggregates.count(Issue::COMMENTS)
.distinct(...), .groupBy(...).count() .distinct(vec![...]), .group_by(vec![...]).count()

A view's data is a ViewValue; a typed query reads it as its rows with query.read(&view.data()) (or Vec::<Issue>::from_view). An unnamed query only sees local data, as in JS without enableLegacyQueries: the server never confirms it, so a run that waits for Complete or Cached never settles, and preloading it does nothing.

Mutators

JS Rust
defineMutator(async ({tx, args, ctx}) => ...) define::define_mutator(name, |tx, args, ctx| ...) (serde feature), a synchronous function over typed arguments; untyped, Mutator::new(name, |tx, args, ctx| ...) over JSON
zero.mutate(mutators.issue.create(args)) zero.mutate_request(CREATE_ISSUE.with(args)), or zero.mutate(name, args) with JSON arguments
await tx.mutate.issue.insert(row), .upsert, .update, .delete mutate::issue(tx).insert(&row)? (generated), or tx.table("issue").insert(&object)?
await tx.run(query) tx.run(&query)?
tx.location, tx.reason, tx.clientID, tx.mutationID, tx.token location(), reason(), client_id(), mutation_id(), token()
throw new ApplicationError(message, {details}) Err(ApplicationError { message, details }.into())

A mutator runs once optimistically and again on every rebase while it is pending, so it must depend only on the transaction, its arguments, and the context. A definition is registered with its mutator() in Zero::new; its arguments are read back as its type before the body runs, and arguments that don't read fail it with an ApplicationError, where JS validates them with the definition's schema. Calling a mutator that isn't registered throws in JS; in Rust both outcomes are a ZeroError saying so, and nothing is written or sent.

React and SolidJS

No UI bindings. useQuery's typed equivalent is LiveView, after the Swift client's LiveQuery: Zero::live takes a typed Query or a named query's QueryRequest and keeps its value read as the query's Rust type, reading only the rows whose entries changed, with its result type and an error that is the server's query error or a row that didn't read (which keeps the last value). It follows the client when the client replaces itself, and has listeners a UI wakes itself from; the untyped equivalent is a MaterializedView and its listener. apps/zbugs-rune shows how a Rune app binds live views to its UI, on the shared zbugs model (apps/zbugs-model), whose queries and mutators are define_query and define_mutator definitions; apps/zbugs-gpui is a GPUI app on the same model.

How some vectors are compared

  • ast-features checks that the wire schema rejects what JS rejects, but not valita's error messages. Its name mapping uses the JS test schema as JS wrote it into ivm.json's shared fixtures.

  • Protocol messages are compared as JSON values, not text: Rust re-encodes a message from its fields, in an order of its own (sweep-downstream compares text with keys sorted). The header encoder is checked by decoding what it writes, as Swift checks it, so its exact escaping isn't pinned by a vector.

  • sweep-query-hash replays each program of builder calls and compares both the AST and the hash, and requires the query to be valid, since JS built every one without throwing. The builder's misuse messages are checked by ivm-features' builder errors (M2); none of the vectors reach limit or orderBy inside a junction relationship, so a unit test in query.rs pins those two messages.

  • The client vectors (client/harness.ts) are scenarios: an app and a server take steps against one client, on a manual clock and mock sockets. Rust replays each on a Zero with zero_client::testing's scheduler and socket factory and compares, after every step, what the app saw (views, listener calls, mutation and run settlements, connection states) and every message the server received, as Swift's ClientConformanceTests does.

  • client-inspector stubs JS's password prompt to answer nothing, so an RPC the server refuses fails with "Authentication failed"; the Rust error goes on to say to call authenticate(password), and the runner compares its part before the colon. A query's milliseconds come from the manual clock, so its metrics and log lines are compared exactly.

  • The IVM vectors (ivm/harness.ts) are programs of query builder calls. Rust replays each with its own builder, must build JS's AST and format or fail with JS's message, then runs JS's AST over memory sources, as Swift does: JS runs the AST with each ordering naming a column once, and Rust must give that answer to the AST as written. Each push (or, for harden-ivm, each batch) compares the changes the view received and the view itself; harden-ivm also compares whether listeners were notified, and tags numbers JSON can't carry. Errors are compared by message, including JS's internal assertions: Rust reports "Unexpected early return prevented full hydration" where JS's Take and Cap assert it, which the Swift client skips.

  • The restored zql vectors (restored-ivm, restored-ivm-rebaselined, generated/restored-chinook-decpush) carry JS's AST and format instead of a program, with the case's own sources (or the chinook fuzzer's mini data) and builder settings, so Rust replays only the pipeline and the pushes, as Swift's RestoredIVMTests does. A test fails a doctored copy of restored-ivm and of the chinook corpus, so a runner that stopped comparing would show.

Documented differences

Each is counted separately in the conformance report, never as an exact match. Both come from what Rust's types can hold, not from the client's logic, and are kept on purpose.

Difference Why it stays
Lone surrogate A Rust String can't hold a lone surrogate, which a JS string can. The parser reads an escaped one ("\ud800") as U+FFFD, as the Swift client does; such a case matches when JS's output, with each such escape read as U+FFFD, is what Rust writes. Holding one would take a string type of Rust's own (WTF-8) throughout the client and in every app's code. Postgres text and jsonb can't store one: only a json (not jsonb) column, which keeps its text as written, could bring one to the client, and it shows as U+FFFD, as a browser renders it. Nearly all such cases come from the JSON fuzz corpus.
Unwritable path A JSON path index is a u32, so a program with a negative or fractional index (json('metadata', -1)) can't be written in Rust. JS rejects it with "Invalid JSON path segment …"; such a case matches when JS rejects it that way. Rust's types rule out at compile time what JS throws for at run time. An index past MAX_JSON_PATH_INDEX can be written, and is rejected with JS's message.

JSON nesting is no longer a difference. JSON nested up to json::MAX_DEPTH (5,000) levels deep, more than the about 4,100 zero-cache can write, is parsed, cloned, compared, written, and dropped within a 2 MB thread stack, even in a debug build, so the client takes any value the server sends. Deeper text is rejected ("JSON nested too deeply"), where JS takes whatever fits its stack. The Swift client has the same limit. Formatting a value with Debug still recurses.