Ship a provider node

One installer for vLLM or SGLang, llm-d prefix-aware routing, signed quotes, canaries, conformance, and scaling.

Owner

Supply

Version

beta-2026.07

Verified

2026-07-13

The Omnious Provider Node is the supported Kubernetes package for a self-hosted provider: vLLM or SGLang, llm-d cache-aware routing, the signed quote loop, metrics, canaries, conformance tests, circuit breaking, and optional autoscaling in one installer.

Install the baseline

PROVIDER_ID=acme-qwen PROVIDER_KEY=0x... \
MODEL_ID=Qwen/Qwen3-32B MODEL_CLASS=qwen3-32b \
FLOOR_IN=210000 FLOOR_OUT=680000 HF_TOKEN=hf_... \
./provider-node/install.sh

Set ENGINE=sglang for SGLang. The installer pins llm-d, installs the Gateway API Inference Extension CRDs, creates the Secret without putting private keys in Helm values, waits for the deployment, and runs the conformance hook. Mainnet and any private validation environment need separate releases, identities, keys, logs, and budgets.

Tool calling is part of the serving contract

Omnious agent requests can include OpenAI tools even when a turn does not ultimately call one. A backend that rejects tool_choice: "auto" will win, fail, and hand the request to another provider. Provider Node therefore keeps the quote signer stopped until ordinary completion, model identity, a real streamed automatic tool call, required tool choice, and a forced named function all pass through the exact serving URL.

# Resolve the reviewed parser from the exact model family
TOOL_PROFILE=auto ./setup.sh

# Explicit override for a newly reviewed family
TOOL_PROFILE=custom VLLM_TOOL_CALL_PARSER=<official-parser> ./setup.sh

The versioned registry maps reviewed families such as Qwen3-Coder, Qwen2.5/Hermes, GPT-OSS, Kimi K2, Hunyuan A13B, GLM 4.5/4.7, Jamba, Mistral 7B v0.3, and OLMo 3. It configures vLLM's model-native parser, while the public contract remains OpenAI-compatible tools. Unknown families fail closed until an official parser is reviewed; managed endpoints do not need a local parser but must pass the same checks. The family entries live in config/tool-profiles.tsv, so a new reviewed profile is a versioned data change rather than another installer-specific condition.

The generated config/omnious-model-tool-profiles.jsonassigns the Omnious-owned tool contract to every approved model and keeps self-hosted parser configuration separate. Runtime support comes only from the exact provider/model model_capabilities row after conformance. Aggregator metadata is translated by an adapter; it is never the platform contract or the source of auction eligibility.

Two routing layers, two jobs

Omnious chooses the provider using signed price and measured service. Inside the winning provider, llm-d chooses the replica using real KV allocation and eviction events, queue depth, and KV pressure. The engine block size, SGLang page size, and llm-d token processor block size must match; the installer derives all three from one setting.

The quote signer reads the metrics sidecar, publishes live capacity, and stops refreshing quotes when its local health breaker opens. Old quotes then expire within the signed TTL, so a broken backend removes itself from the market instead of selling fictional headroom.

Defaults are a benchmark starting point

Both engines use continuous request scheduling, prefix caching, and chunked prefill. The initial token budget is 8,192 with a 90% GPU-memory target. Tune against your actual model, quantization, hardware, prompt/output distribution, and p95 TTFT/ITL objectives. Watch queue depth, KV occupancy, preemption or retraction, cache hits, tokens per second per GPU, errors, and quote headroom together.

Tests and scaling are part of the node

The Helm test verifies model identity, non-streaming and streaming OpenAI compatibility, and automatic plus forced tool calling through the production proxy. The quoter runs a shorter version before it can publish its first quote. A scheduled local canary repeats the serving-path check. Optional KEDA scaling uses llm-d running-request and flow-control queue metrics, not raw GPU utilization, and requires a Prometheus installation.

The full runbook covers secrets, network exposure, allowlist data, tuning, metrics, autoscaling, benchmark format, upgrades, rollback, and removal in provider-node/README.md.