GPU scheduling on Kubernetes

The device-plugin model exposes a GPU as an indivisible resource: a pod either asks for a whole card or none. A small inference service might use maybe 8 GB of an 80 GB A100 and a sliver of its compute units (SMs), yet it reserves the entire card. Everything else sits idle 'just in case.'

Because several small jobs can't share one card, growing the cluster looks like the only option. It isn't. The fix is to split the same card safely and place work intelligently.

GPUs are advertised to Kubernetes through a device plugin. The plugin counts the cards on the node and reports to the kubelet; the kubelet records it as node capacity; the API server keeps it as allocatable; and the scheduler finds a node with as many whole units as the pod requests, then binds the pod there.

The catch: this resource is an integer. There is no half a GPU; the scheduler can't hand out '0.3 GPU'. Splitting a card means either changing what that integer means (MIG) or sharing one unit across several pods over time (time-slicing).

On A100/H100, Multi-Instance GPU (MIG) carves the card into hardware-isolated slices. Each slice has its own compute partitions, its own memory slice and its own memory bandwidth. One A100 can be split into up to seven independent instances; the most common profile is 1g.5gb (1/7 of compute, 5 GB of memory).

What matters is isolation: slices can't affect each other's performance, there are no 'noisy neighbours,' latency stays predictable. Each slice appears to Kubernetes as a separate resource, so the scheduler can place seven different pods on a single card, and they all run at once. MIG splits space.

For bursty, intermittent jobs that don't need isolation, there's a cheaper path: time-slicing. One GPU is shared by several pods, but in time rather than space: the card runs the jobs in turn, in small quanta. As the cursor moves along the time axis below, you can see that at any instant only one job is on the GPU.

The cost: there's no real isolation. If one job abuses the card it slows the others down, and every switch costs a small context switch. Great for bursts, risky for latency-sensitive production traffic. MIG splits space, time-slicing splits time.

Both open one card to many jobs, but they promise different things. The decision comes down to one question: can these jobs trust each other?

You've split the card; but how does a pod reach the right machine? Two mechanisms work together. A taint hangs a 'no entry' sign on an expensive GPU node; only pods that tolerate it (carry a toleration) may land there. node affinity pulls the pod toward GPU-labelled nodes.

So taint is a push, affinity is a pull. The result: the GPU pod (with toleration + affinity) settles onto the GPU node; the CPU pod, lacking a toleration, can't enter and lands on the CPU node. That keeps stray CPU work off expensive cards.

For multi-GPU jobs, how the cards connect to each other decides throughput. NVLink-connected cards talk extremely fast; talking over PCIe is far slower. A training job's per-step all-reduce flows over exactly this link.

That's why topology-aware scheduling matters: you co-locate multi-GPU jobs on NVLink-connected cards (the same island). Below, the same four-GPU job is placed two ways: in the right one the data flies within one island over NVLink; in the wrong one it splits across islands and is condemned to PCIe.

Split the card, put the work on the right node, respect the topology. All three moves are configuration, not code, and together they multiply your capacity without ordering a single new GPU.