One ModelDeployment, two serving stacks

A Modelplane InferenceCluster now chooses the serving stack it runs. Standard is the stack Modelplane composes itself. Dynamo uses NVIDIA's components instead: Grove and the KAI Scheduler place a multi-node engine as a gang, and a ModelExpress server moves model weights GPU to GPU between replicas.

The ModelDeployment an ML team writes doesn't change. The same manifest runs on either stack. Which stack a cluster runs is a platform decision, made per cluster, so a fleet can run both at once.

It's in main today and will be part of the next release. We validated it on EKS with one Standard cluster and one Dynamo cluster.

A serving stack owns one cluster#

Modelplane operates a fleet. It provisions clusters and node pools, schedules each replica onto hardware that fits, stages weights once per cluster, scales replicas, and fronts the whole fleet with one OpenAI-compatible endpoint. It isn't a serving layer itself.

A serving stack owns what happens inside one cluster. It places a multi-node engine's pods and gets the model's weights into GPU memory. Dynamo does both, with a frontend and router ahead of the engines, and it reaches into territory Modelplane's own stack doesn't, like keeping those weights resident across an engine crash.

A ModelDeployment describes an engine#

The ML team writes one container named engine with its image, command, and args, and what they write is what runs. An engine is either Standalone, or a Leader and a Worker whose command spans nodes. A ModelDeployment describes that engine and says nothing about the stack underneath it.

So a serving stack has to run two pod specs with distinct commands, and give a worker a way to find its leader. Grove does both.

Opting a cluster in#

apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-h200-us-east
spec:
  # Standard (the default) or Dynamo. Immutable.
  stack: Dynamo
  cluster:
    source: EKS
    eks:
      region: us-east-1
  nodePools:
  - name: gpu
    className: eks-h200-8x
    nodeCount: 2

On a Dynamo cluster Modelplane installs Grove, the KAI Scheduler, and one ModelExpress server. On a Standard cluster it installs the LeaderWorkerSet controller. Everything else about a cluster, from how it fronts requests to how it stages model weights, is the same on both.

The choice is immutable, which makes adoption incremental. A platform team stands up a Dynamo cluster next to the ones it already runs and moves deployments over cluster by cluster.

Gang scheduling with Grove and the KAI Scheduler#

A Standalone engine is a Deployment on both stacks. A Leader and Worker gang is a LeaderWorkerSet on Standard, and on Dynamo a Grove PodCliqueSet with a leader clique and a worker clique, scheduled by KAI. KAI schedules the gang all or nothing, so it either gets every node it asked for or waits, rather than half-placing and holding GPUs it can't serve from.

Here's a 480B model across two nodes, tensor-parallel within each node and pipeline-parallel across them. $(MODELPLANE_LEADER_ADDRESS) is the address the leader is reachable at, and it resolves on both stacks:

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-coder
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      modelCacheRef:
        name: qwen3-coder
      engines:
      - name: qwen3-coder
        members:
        - role: Leader
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("120Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                command:
                - /bin/sh
                - -c
                - >-
                  exec vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct
                  --served-model-name=qwen3-coder
                  --tensor-parallel-size=8
                  --pipeline-parallel-size=2
                  --distributed-executor-backend=mp
                  --nnodes=2 --node-rank=0
                  --master-addr=$(MODELPLANE_LEADER_ADDRESS)
                  --max-model-len=32768
                  --port=8000
        - role: Worker
          worker:
            nodes: 1
          # nodeSelector is the same as the leader's. Omitted for brevity.
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                command:
                - /bin/sh
                - -c
                - >-
                  exec vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct
                  --served-model-name=qwen3-coder
                  --tensor-parallel-size=8
                  --pipeline-parallel-size=2
                  --distributed-executor-backend=mp
                  --nnodes=2 --node-rank=1
                  --master-addr=$(MODELPLANE_LEADER_ADDRESS)
                  --headless
                  --max-model-len=32768

The modelCacheRef names a ModelCache, which stages a model's weights once per cluster on shared storage. Both members name the model by its Hugging Face repo id, and Modelplane points the engine's HF_HUB_CACHE at the mount, so the engine resolves that repo id against the staged snapshot instead of downloading it.

Cold starts with ModelExpress#

Loading weights is the slowest part of bringing a replica online, and it gets slower when several replicas scale up at once and read the same storage.

A Dynamo cluster runs one ModelExpress server. It brokers which replica holds a model in GPU memory, and never touches the weight bytes itself. The first replica loads from the cache volume and publishes itself as a source, and later replicas pull the weights from a peer's GPU. That transfer uses a fast fabric where the cluster has one, such as EFA on EKS, and TCP where it doesn't.

An engine opts in through its own command, with --load-format modelexpress. Modelplane doesn't add a load format of its own. The same command runs on a Standard cluster, where ModelExpress finds no server and the engine loads the weights from the mount.

What's next#

The end state is the same cluster opt-in composing a full DynamoGraphDeployment, so a fleet gets Dynamo's frontend and router while the API an ML team writes stays what it is. That needs two things from the Dynamo operator: a worker pod spec that can differ from its leader's, and a switch to leave the user's command alone rather than generating launch flags. NVIDIA has both in flight in dynamo#12696.

Try it#

The getting-started guide covers standing up a fleet, and how it works covers what a serving stack installs. Modelplane is Apache 2.0 and moving fast at github.com/modelplaneai/modelplane, and questions are welcome in Slack.