Feature proposal: support asymmetric embedding models (input_type / query & passage prefixes)

[ankekod]

Many popular embedding models are asymmetric: they need a signal telling
them whether a text is being embedded for storage (a document/passage) or
for retrieval (a query). Embedding both the same way measurably degrades
retrieval accuracy. This is indicated by the input_type field in the OpenAI API specification.

rag_api currently embeds documents and queries identically and doesn't use the ìnput_type`field, so these models
can't be used correctly today:

Provider / model	Mechanism	Storage value	Query value
NVIDIA NIM (NV-Embed, nv-embedqa-e5-v5)	input_type body field	passage	query
Cohere embed	input_type body field	search_document	search_query
Voyage AI	input_type body field	document	query
Jina v3	task body field	retrieval.passage	retrieval.query
E5 / BGE / nomic (self-hosted)	text prefix	"passage: "	"query: "

Proposal

No API or LibreChat change is needed. Internally the store-vs-retrieve intent
is already unambiguous: /embed flows through Embeddings.embed_documents()
and /query through embed_query(). The change is confined to embeddings
initialization in app/config.py plus one small wrapper implementing the
LangChain Embeddings interface.

• When enabled, construct the configured embedding client and route storage vs.
  query embeds through an AsymmetricEmbeddings wrapper that sends the right
  signal per operation (embed_documents/embed_query and async variants).
• Body-field mode (OpenAI-compatible providers): inject the field via
  model_kwargs={"extra_body": {...}}. Verified against the pinned
  langchain-openai / openai — the field lands at the top level of the
  /embeddings request body, exactly what NIM and friends read.
• Prefix mode (E5/BGE/nomic): prepend a per-operation string. Provider-
  agnostic — works with Ollama, HF, TEI, etc., where body-field injection can't.

Two separate client instances back the two operations rather than mutating one
client's model_kwargs per call: /query embeds run in the thread pool while
/embed batches run concurrently, so a shared mutable instance would be a race
condition. In prefix-only mode the two instances would be byte-identical, so a
single client is shared between both slots — this avoids loading a local
sentence-transformers model into memory twice.

New env vars, all optional and fully backward compatible (default behavior is
unchanged — a plain client, no wrapper):

EMBEDDINGS_ASYMMETRIC=true              # default false; master switch for body-field mode
EMBEDDINGS_INPUT_TYPE_FIELD=input_type  # "task" for Jina
EMBEDDINGS_INPUT_TYPE_STORAGE=passage
EMBEDDINGS_INPUT_TYPE_QUERY=query
EMBEDDINGS_STORAGE_PREFIX=              # prefix mode; works with any provider
EMBEDDINGS_QUERY_PREFIX=

EMBEDDINGS_ASYMMETRIC=true on a provider that can't inject body fields
(anything but openai/azure) fails fast at startup with an error pointing to
the prefix vars, rather than silently embedding every query as a passage.

Because configuration is env-vars-only, this is drop-in compatible with
existing deployment methods, including LibreChat's Helm chart (rag.configEnv)
and the lite image — no chart or Dockerfile changes needed.

Scope of changes

• New: app/services/embeddings.py (~40 lines, AsymmetricEmbeddings)
• Modified: app/config.py — the new env vars, an extra_body param on
  init_embeddings, and a build_embeddings() helper holding the
  wrapper/validation logic
• Tests: wrapper unit tests, a config-branching matrix, env parsing, and an
  in-process mock-server test asserting input_type reaches the request body
• README: the env vars documented with an NVIDIA NIM example

Status

Implemented on my fork, on a branch off current main:
https://github.com/ankekod/rag_api/tree/new/feature/asymmetric-embeddings

Verified end-to-end in a live LibreChat deployment — LibreChat running
against this build of rag_api, pointed at an NVIDIA NIM asymmetric embedding
model. Uploading files (LibreChat → /embed) sends input_type=passage and
file search (→ /query) sends input_type=query, both reaching NIM correctly,
with retrieval working. This confirms the headline claim in practice: no
LibreChat-side change is needed — the store-vs-retrieve direction is fully
carried by which rag_api endpoint LibreChat already calls.

pytest passes (246 tests). Happy to open a PR against main if this direction
is acceptable.

Caveat we'd document

Enabling this changes how documents are embedded, so vectors stored beforehand
(without an input_type / prefix) should be re-embedded — querying old vectors
with the query-side setting compares two different input modes and degrades
results. Called out in the README and the PR description.

[ahmedbellaaj10]

Hi @ankekod, thanks for this.

I read new/feature/asymmetric-embeddings. It's close to what we would have written, and it still merges cleanly onto current main. Two additions I'd like to propose:

1. Naming aligned with 🔒 fix: close /query and /query_multiple authorization holes; add /v1/embeddings and /v1/rerank #318. The maintainer's search-stack PR already has query/document prefixes for /v1/embeddings (RAG_CHAT_EMBEDDING_QUERY_PREFIX / RAG_CHAT_EMBEDDING_DOCUMENT_PREFIX). Using EMBEDDINGS_DOCUMENT_PREFIX and EMBEDDINGS_INPUT_TYPE_DOCUMENT instead of STORAGE would keep one vocabulary across the codebase.
2. A startup check, not only the README warning. Turning prefixes on over a collection that already holds chunks mixes two kinds of vectors, and nothing errors; results just get worse. Recording the document-side setting on the collection (langchain_pg_collection.cmetadata) and refusing to start on a mismatch would make that visible. Collections created before the change would count as "no prefix", so upgrading without enabling anything changes nothing.

Since this discussion hasn't had a maintainer reply yet, I've opened #344 to get it triaged, linking here and to #318.

Are you planning to open a PR? If not, would you be OK with me opening one that builds on your commit (keeping you as the author), with the rename and the check as separate commits? Happy to go whichever way you prefer.