#Inference Guide

Ready-to-deploy recipes for validated open-weight LLMs on Alauda AI. Each model in this guide has been deployed end-to-end on a real cluster and benchmarked, so you get a known-good deployment manifest, the runtime image that serves it, and the throughput you can expect.

The models here were validated on Huawei Ascend NPUs (910B4 and 910B3) with the community vLLM-Ascend engine, deployed through Alauda AI's InferNex surface — a KServe LLMInferenceService reconciled by the InferNex-Bridge into a load-aware router (hermes-router / EPP) in front of the vLLM-Ascend instances. Most were run through the same InferNex aggregation surface and the same two benchmark scenarios (the two Qwen models share an identical 2 × TP=4 topology and are directly comparable; DeepSeek-V4-Flash uses larger topologies, up to a 16-card DP2 × TP=8 cross-node aggregation). For the runtime model (KServe, ModelCar storage, scheduling) see Model Deployment & Inference.

#Validated models

The two Qwen models and DeepSeek-V4-Flash (W4A8) were validated on Ascend 910B4 (32 GB/card), driven through KServe LLMInferenceService with load-aware routing (InferNex-Bridge + hermes-router). The two Qwen models run an 8-card aggregation — 2 instances × TP=4; DeepSeek-V4-Flash (~151 GB W4A8) fills all 8 cards as 1 instance × TP=8, and additionally validated the MaaS gateway (API-key) ingress next to the internal KServe ingress.

DeepSeek-V4-Flash (W8A8, ~280 GB) was validated on Ascend 910B3 (64 GB/card) across two nodes = 16 cards as a single DP2 × TP=8 cross-node aggregation with a mooncake cross-rank KV store, driven through both the internal KServe ingress and the product MaaS gateway.

#Runtime images

#Benchmark scenarios

These models were measured with aiperf against the same two scenarios, modelled on real serving patterns. Output is pinned to 128 tokens and load is closed-loop, concurrency 4 (4 in-flight requests, fixed); each scenario ran 240 requests. DeepSeek-V4-Flash (W8A8) is the exception — it was swept at concurrency 8 / 16 / 32 (480 requests per tier).

#Deploy a validated model

Each model page links self-contained YAMLs under assets/ that hold the real InferNex deployment — a KServe LLMInferenceService (infernex.io/runtime: true) plus the two LLMInferenceServiceConfig objects (engine template + hermes-router/EPP template) that the InferNex-Bridge reconciles into the running instances.

base=https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/inference_guide/assets

# 1. Edit the file first: set the namespace and the image tag (match your CANN /
#    host driver). model.uri already points at the public ModelCar on Docker Hub
#    (oci://docker.io/alaudadockerhub/...) — repoint it only if you mirror locally.
# 2. Apply — the InferNex-Bridge reconciles the LLMInferenceService into the
#    hermes-router (EPP) + the vLLM-Ascend instances.
kubectl apply -f $base/qwen3-32b/qwen3-32b-agg-base-llmisvc.yaml

# 3. Watch it come up (first start loads weights + compiles graphs — can take minutes).
kubectl -n <your-namespace> get llminferenceservice -w

# 4. Call the OpenAI-compatible endpoint through the gateway.
curl -s http://<gateway>/<namespace>/qwen3-32b-agg-base/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-32B","messages":[{"role":"user","content":"hello"}]}'

#Caveats

• These manifests deploy through InferNex (LLMInferenceService + InferNex-Bridge
  • hermes-router). The two LLMInferenceServiceConfig objects live in the kserve namespace; the LLMInferenceService lives in your deployment namespace.
• Resource keys are for Ascend 910B4 (huawei.com/Ascend910). Adjust the resource key, image, and version fields for your actual NPU model.
• The ModelCar images are public on Docker Hub under alaudadockerhub — the manifests pull them with no credentials. Mirror them to your own registry and repoint model.uri if you prefer; the modelcar pull secret in the manifest is only needed for a private registry.
• The benchmark numbers were measured closed-loop (concurrency 4) on 8 cards. Treat them as the per-instance operating point under steady load, not a saturation ceiling.

#Verify the ModelCar signature

The ModelCar images are signed with Cosign. Verify an image against the published public key (cosign.pub) before deploying:

curl -sO https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/inference_guide/cosign.pub

cosign verify --key cosign.pub --insecure-ignore-tlog=true \
  docker.io/alaudadockerhub/modelcar-qwen3-32b:v0.1.0

The three signed images and their digests:

--insecure-ignore-tlog=true is required because these were signed with --tlog-upload=false (no public transparency-log entry); verification relies on the public key alone.