#GPU Slicing with Dynamic Resource Allocation (DRA)

The classic device-plugin model hands out whole GPUs: a Pod asks for nvidia.com/gpu: 1 and gets an entire card, even for a LoRA fine-tune that never touches more than a few GiB. On a 24 GB A30 that is up to 20× the memory the job needs, sitting idle.

Dynamic Resource Allocation (DRA) — a resource.k8s.io/v1 API that reached GA in Kubernetes 1.34 — lets a Pod request a slice of a GPU instead. With the NVIDIA DRA driver that slice can be:

• a MIG partition — a hardware-isolated fraction of the card (its own SMs and VRAM, enforced by the GPU), or
• a time-sliced / MPS-shared full GPU — several Pods packed onto one card with no memory isolation.

This guide requests a MIG slice through a ResourceClaimTemplate, then runs a supervised fine-tune inside it with Kubeflow Trainer v2. Every asset is under assets/dra/, and the flow is exercised end-to-end by the c15_dra_gpu_slice.sh case in the repo's e2e harness.

#Device plugin vs. DRA

DRA doesn't replace your scheduler or Kueue — it changes how the device is described and claimed. Quota, priority, and preemption still work (see Combine slices with Kueue quota).

#Prerequisites

#The three objects in a slice request

  cluster (driver/admin)                    namespace (you)
  ┌────────────────────┐        ┌──────────────────────────────────┐
  │ DeviceClass         │◄───────│ ResourceClaimTemplate            │
  │  mig.nvidia.com     │  refs  │  request: 1× mig.nvidia.com      │
  │  gpu.nvidia.com     │        │  selector: profile == "1g.6gb"   │
  └────────────────────┘        └──────────────┬───────────────────┘
  ┌────────────────────┐                        │ per-Pod instance
  │ ResourceSlice(s)    │  scheduler matches     ▼
  │  advertised by the  │◄──────────  Pod.spec.resourceClaims ──► container.resources.claims
  │  kubelet-plugin     │  claim & allocate a slice, inject it into the Pod
  └────────────────────┘

• DeviceClass (created by the driver) names a kind of device and a base selector. This cluster's NVIDIA driver ships gpu.nvidia.com (full GPUs) and mig.nvidia.com (MIG slices).
• ResourceClaimTemplate (you) says "give each Pod one device from this class, matching this CEL selector". The scheduler stamps out a per-Pod ResourceClaim from it.
• The Pod references the template in spec.resourceClaims, and the container opts in with resources.claims. No nvidia.com/* limit is used at all.

#Step 1 — Confirm the driver is advertising slices

DeviceClasses exist as soon as the driver is installed, but a Pod can only be scheduled once the kubelet-plugin publishes ResourceSlices for the node's GPUs:

kubectl get deviceclasses
# NAME              ...
# gpu.nvidia.com
# mig.nvidia.com

kubectl get resourceslices
# NAME                       NODE              DRIVER           ...
# <node>-gpu.nvidia.com-...  192.168.143.59    gpu.nvidia.com   ...

If get resourceslices is empty, the driver is installed but not advertising devices on any node — jump to Enabling the DRA driver before continuing.

Inspect the advertised devices to learn the exact attribute keys and values your CEL selectors can match on — do not assume the profile strings; read them from your cluster:

kubectl get resourceslices -o yaml | grep -A20 'attributes:'
# ...
#   attributes:
#     gpu.nvidia.com/profile: {string: "1g.6gb"}
#     gpu.nvidia.com/productName: {string: "NVIDIA A30"}
#     gpu.nvidia.com/type: {string: "mig"}

#Step 2 — Smoke-test a slice

Before wiring DRA into a training job, prove the driver hands you a slice with dra-smoke-pod.yaml. It claims one MIG slice and prints what the container sees:

base=https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/training_guides/assets/dra
NS=my-namespace   # namespace where your Pods run

# 1. A ResourceClaimTemplate for one 1g.6gb MIG slice.
curl -fsSL $base/mig-slice-resourceclaimtemplate.yaml | sed "s/kubeflow-admin-cpaas-io/$NS/" | kubectl apply -f -

# 2. A Pod that claims it and runs nvidia-smi + a matmul on the slice.
curl -fsSL $base/dra-smoke-pod.yaml | sed "s/kubeflow-admin-cpaas-io/$NS/" | kubectl create -f -

# 3. Watch it, then read the logs.
kubectl -n "$NS" get pods -l app=dra-smoke -w
kubectl -n "$NS" logs -l app=dra-smoke --tail=40

You should see the slice's reduced memory ceiling — around 6 GiB for 1g.6gb, not the card's full 24 GiB — which is the whole point:

===== nvidia-smi -L =====
GPU 0: NVIDIA A30 (UUID: GPU-...)
  MIG 1g.6gb  Device  0: (UUID: MIG-...)
cuda_available: True
device: NVIDIA A30 MIG 1g.6gb
slice_total_mem_GiB: 5.75
matmul_on_slice: OK

Check the allocation from the control plane:

kubectl -n "$NS" get resourceclaims
# NAME                 STATE                ...
# dra-smoke-...-gpu-slice   allocated,reserved

#Step 3 — Fine-tune inside the slice with Kubeflow Trainer v2

The dra-sft-trainingruntime.yaml TrainingRuntime is an ordinary Trainer v2 runtime with exactly two DRA hooks and no nvidia.com/* limit:

spec:
  template:
    spec:
      replicatedJobs:
        - name: node
          template:
            spec:
              template:
                spec:
                  resourceClaims:                         # (1) Pod-level: bind a per-Pod claim
                    - name: gpu-slice
                      resourceClaimTemplateName: mig-1g-6gb
                  containers:
                    - name: node
                      resources:
                        claims:                           # (2) Container-level: consume it
                          - name: gpu-slice
                        # NOTE: no nvidia.com/gpu here — the GPU comes from the claim

Its training script is self-contained — it builds a small causal-LM, wraps it in a LoRA adapter, and trains on a synthetic dataset, so the run needs no model or dataset download and finishes in about a minute. That makes it safe on air-gapped GPU nodes and ideal for validating the slice. Swap in a real base model and dataset to make it a production fine-tune (see Turn this into a real fine-tune); the DRA plumbing does not change.

Apply the runtime and submit the job:

base=https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/training_guides/assets/dra
NS=my-namespace

# ResourceClaimTemplate (skip if you already applied it in Step 2).
curl -fsSL $base/mig-slice-resourceclaimtemplate.yaml | sed "s/kubeflow-admin-cpaas-io/$NS/" | kubectl apply -f -
# TrainingRuntime that consumes the slice.
curl -fsSL $base/dra-sft-trainingruntime.yaml         | sed "s/kubeflow-admin-cpaas-io/$NS/" | kubectl apply -f -
# TrainJob.
curl -fsSL $base/dra-sft-trainjob.yaml                | sed "s/kubeflow-admin-cpaas-io/$NS/" | kubectl create -f -

Follow it to completion:

kubectl -n "$NS" get trainjob,pods
pod=$(kubectl -n "$NS" get pods -l jobset.sigs.k8s.io/replicatedjob-name=node -o jsonpath='{.items[0].metadata.name}')
kubectl -n "$NS" logs -f "$pod"

Success looks like:

[slice] device=NVIDIA A30 MIG 1g.6gb total_mem=5.75 GiB
trainable params: 147,456 || all params: 4,...  || trainable%: ...
{'loss': ..., 'step': 10}
[slice] peak_mem=1.83 GiB
DRA LoRA SFT on GPU slice: OK

peak_mem well under the slice's total_mem confirms the fine-tune ran entirely within its partition. Run a second TrainJob while the first is going: on a card partitioned into four 1g.6gb slices, up to four fine-tunes run concurrently and isolated on one physical GPU — each blind to the others' memory.

#Sizing the slice / choosing a profile

Change the MIG profile in the CEL selector of mig-slice-resourceclaimtemplate.yaml:

selectors:
  - cel:
      expression: device.attributes["gpu.nvidia.com"].profile == "2g.12gb"

Typical profiles (confirm the exact strings against your ResourceSlices — geometry is per-GPU):

Pick the smallest profile whose memory comfortably holds your model + LoRA + optimizer state + activations. A 0.5–1.5B model LoRA fits 1g.6gb; a 7B QLoRA usually wants 2g.12gb+.

#No MIG? Share a full GPU by time-slicing

If the GPU isn't MIG-capable, or you don't want to partition it, shared-gpu-resourceclaimtemplate.yaml requests a full GPU shared by time-slicing via an opaque GpuConfig:

spec:
  spec:
    devices:
      requests:
        - name: gpu-slice
          exactly:
            deviceClassName: gpu.nvidia.com
            allocationMode: ExactCount
            count: 1
      config:
        - requests: ["gpu-slice"]
          opaque:
            driver: gpu.nvidia.com
            parameters:
              apiVersion: resource.nvidia.com/v1beta1
              kind: GpuConfig
              sharing:
                strategy: TimeSlicing   # or MPS for concurrent sharing

Point the runtime at this template instead (resourceClaimTemplateName: shared-gpu-timeslice) — nothing else changes. The e2e case takes DRA_SLICE_MODE=shared to run exactly this path.

#Combine slices with Kueue quota

DRA is about the shape of the device; Kueue is about who may use how much. Kueue understands DRA DeviceClasses, so a ClusterQueue can quota mig.nvidia.com devices the same way it quotas nvidia.com/gpu — admitting TrainJobs against a slice budget and preempting borrowers when a higher-priority job needs a slice back. The Preemptible TrainJobs with Kueue guide's cohort pattern applies unchanged; swap the covered resource in the ResourceFlavor for the DRA device class. (DRA support in Kueue is evolving — verify against your Kueue version before relying on it in production.)

#Wiring the same claim into other orchestrators

The DRA hooks live in the Pod spec, so anything that lets you shape a Pod can request a slice — Trainer v2 is just one caller:

• Kubeflow Pipelines (KFP v2): attach resourceClaims / resources.claims to a task's Pod through the kfp-kubernetes platform config (or a pod-spec patch), so a pipeline component runs on a slice.
• Volcano / plain Job / Pod: the two hooks are byte-for-byte identical to the smoke Pod — set spec.resourceClaims and the container's resources.claims.

The smoke Pod is the minimal template for any of these.

#Turn this into a real fine-tune

The bundled script trains a synthetic model so the slice can be validated offline. To fine-tune a real model on the slice, keep the two DRA hooks and swap the container for a real recipe — for example the LlamaFactory runtime from Fine-Tuning with Kubeflow Trainer v2, with its dataset-initializer / model-initializer steps downloading a base model and dataset onto a shared PVC. The only change to that runtime is deleting the nvidia.com/gpualloc / gpucores / gpumem limits and adding spec.resourceClaims + resources.claims. Size the MIG profile to the model (a 0.6B LoRA fits 1g.6gb; a 7B QLoRA wants 2g.12gb+).

#Enabling the DRA driver (admin)

The NVIDIA DRA driver installs a controller plus a kubelet-plugin DaemonSet gated on a node label. Until a GPU node carries that label, the plugin publishes no ResourceSlices and DRA Pods stay Pending.

# The kubelet-plugin DaemonSet is scheduled by a node label (check its nodeSelector):
kubectl -n kube-system get ds nvidia-dra-driver-gpu-kubelet-plugin \
  -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}'
# {"nvidia-device-enable":"pgpu-dra"}

# Label the GPU node so the plugin lands and starts advertising devices:
kubectl label node <gpu-node> nvidia-device-enable=pgpu-dra

# For MIG slices, the GPU must be in MIG mode. Either let the driver manage it,
# or enable it out of band (requires the GPU to be idle; may need a node drain):
#   nvidia-smi -i <gpu> -mig 1

#Troubleshooting

For repeatable coverage of this whole flow, the c15_dra_gpu_slice.sh case applies the template, runtime, and TrainJob, and asserts the fine-tune finishes inside the slice. It self-skips when no gpu.nvidia.com ResourceSlices are present, so it stays green on device-plugin-only clusters.