embedding-runtime (Tier-2 vector runtime)

OpenAI-compatible embeddings + rerank server for the Vector Inference line. KServe owns the lifecycle (registered as ClusterServingRuntime embedding-runtime-cpu / embedding-runtime-cuda, modelFormat embedding-runtime); this container owns only the serving code. Same pattern as llama-cpp-runtime.

Endpoints

• POST /v1/embeddings — {input: str|[str], model?, task?, instruction?}
• POST /v1/rerank — {query, documents: [str], top_n?, task?, instruction?}
• GET /health

Env

• EMBEDDING_MODEL_DIR (default /mnt/models) — KServe model mount.
• EMBEDDING_MODEL_NAME (default = model dir) — overrides the ST model id.
• EMBEDDING_TASK_INSTRUCTIONS — JSON {task: instruction-prefix} (E5/Qwen3/Instructor class).
• EMBEDDING_TASK_ADAPTERS — JSON {task: lora-adapter-path-or-repo} (Jina-v5-base class).

Local run (smoke)

docker build -t smartmlops/embedding-runtime:dev apps/embedding-runtime
docker run --rm -p 8080:8080 \
  -e EMBEDDING_MODEL_NAME=BAAI/bge-small-en-v1.5 \
  smartmlops/embedding-runtime:dev
curl -s localhost:8080/v1/embeddings -H 'content-type: application/json' \
  -d '{"input":["hello world"],"task":"retrieval.query"}' | head -c 200

Local tests

The endpoint tests install only the FastAPI test surface plus pytest/httpx, then monkeypatch the transformer, torch, and NumPy surfaces before importing server.py. They do not install transformer runtimes, download model weights, or run real model inference.

python3 -m pip install -r requirements-test.txt
python3 -m pytest tests