Skip to content
Posts en inglés. Usá el traductor del navegador para leerlos en tu idioma.
Featured

Build a Resilient GPU Kubernetes Cluster with Spot Instance Failover

Yammbo
· 8 min read
Copyright: CC BY-NC-SA 4.0
spot instance failover digitalocean kubernetes gpu workloads cluster autoscaler taints tolerations
Build a Resilient GPU Kubernetes Cluster with Spot Instance Failover

Running compute-intensive workloads like AI inference, machine learning training, or batch image processing on Kubernetes often involves a trade-off between cost and reliability. Spot GPU instances offer significant cost savings, but they come with the risk of being reclaimed by the cloud provider. This tutorial guides you through setting up a Kubernetes cluster designed to gracefully handle these interruptions, ensuring your GPU workloads remain available by automatically failing over to On-Demand GPU instances when Spot capacity is reclaimed.

Step 1: Understanding Spot GPU Instances and Reclaim Behavior

Spot GPU instances are a cost-effective way to access compute resources. Cloud providers offer these instances at a discounted price because they represent unused capacity. The key characteristic of Spot instances is their ephemeral nature: the cloud provider can reclaim them at any time, typically with a short notification period. For example, on DigitalOcean Kubernetes Service (DOKS), Spot GPU Droplets can be reclaimed, and importantly, DOKS reclaims the entire Spot node pool, not just individual nodes. This means your infrastructure must be designed to withstand the sudden loss of an entire pool of compute resources, rather than assuming a single node failure.

To maximize the value of Spot GPU instances, they are best suited for:

  • Stateless, decoupled workloads: Tasks like API inference or processing items from a queue that don't maintain persistent connections or data on the node.
  • Fault-tolerant processes: Workloads that can tolerate interruptions and resume from a known state.
  • Short-lived processes: Jobs that complete quickly, minimizing the chance of interruption.

For longer-running tasks, such as machine learning model training, implementing checkpointing is crucial. This technique involves periodically saving the state of your work to persistent storage (like block storage or object storage) so that if an interruption occurs, your workload can restart from the last saved checkpoint instead of beginning anew.

Step 2: Designing Your Resilient Kubernetes Cluster Architecture

To achieve resilience against Spot instance reclaims, we'll configure a Kubernetes cluster with three distinct node pools. This architecture ensures that core services are stable, GPU workloads benefit from Spot savings, and a fallback mechanism is in place for interruptions.

The Three Node Pools:

  1. Dedicated System Pool: This is an On-Demand node pool reserved exclusively for critical cluster services. This includes Kubernetes system components, ingress controllers, monitoring tools, and any other operational tooling. System processes should never run on ephemeral Spot hardware to ensure the stability of your cluster's control plane and essential infrastructure. This pool should be untainted.
  2. Spot GPU Pool: This pool consists of Spot GPU instances. It's designed for your primary GPU workloads, offering significant cost savings. To ensure only appropriate workloads land here, this pool will be tainted. A taint prevents pods from being scheduled on nodes unless they have a corresponding toleration. For example, a taint like gpu-pool-type=spot:NoSchedule will ensure only pods explicitly configured to tolerate this taint can run on these nodes.
  3. On-Demand GPU Pool (Fallback): This is your safety net. It's a second node pool composed of On-Demand GPU instances, matching the hardware specifications of your Spot GPU pool as closely as possible. This pool is initially provisioned with zero nodes, meaning it incurs no cost until it's actually needed. It will be untainted, allowing any GPU workload to run on it. When the Spot GPU pool is reclaimed, pending GPU workloads will trigger the cluster autoscaler to scale up this On-Demand pool, providing the necessary compute resources.

This design leverages Kubernetes taints and tolerations to direct workloads and the Cluster Autoscaler to manage resource provisioning dynamically.

Step 3: Implementing Node Pools in DigitalOcean Kubernetes

Let's walk through the conceptual steps to create these node pools using doctl, DigitalOcean's command-line interface. While specific commands are shown, the underlying principles apply to any Kubernetes environment supporting Spot instances and node pools.

First, ensure you have a DOKS cluster running and your kubectl context is configured correctly. You will also need doctl installed and authenticated.

1. Create the Dedicated System Pool

This pool hosts your core cluster services. It should be an On-Demand pool with sufficient resources for your control plane and essential add-ons. It should not have any taints.

doctl kubernetes cluster node-pool create <your-cluster-name> \ --name system-pool \ --size s-2vcpu-4gb \ --count 3

This command creates a node pool named system-pool with three s-2vcpu-4gb Droplets. Adjust the size and count based on your cluster's needs.

2. Create the Spot GPU Pool

This pool will house your GPU-intensive workloads. It uses Spot instances and is configured with a specific label and taint to ensure only your GPU applications run here.

doctl kubernetes cluster node-pool create <your-cluster-name> \ --name spot-gpu-pool \ --size g-2vcpu-16gb \ --count 1 \ --labels gpu-pool-type=spot \ --taints gpu-pool-type=spot:NoSchedule \ --spot

Here, we create a Spot node pool with a GPU-enabled Droplet size (e.g., g-2vcpu-16gb). The --labels gpu-pool-type=spot adds a label for easy identification, and --taints gpu-pool-type=spot:NoSchedule applies a taint that prevents pods without a matching toleration from being scheduled on these nodes. The --spot flag designates it as a Spot pool.

3. Create the On-Demand GPU Pool (Fallback)

This is your fallback pool. It uses On-Demand GPU instances and is initially scaled to zero nodes. It should match the hardware specifications of your Spot GPU pool and remain untainted.

doctl kubernetes cluster node-pool create <your-cluster-name> \ --name ondemand-gpu-pool \ --size g-2vcpu-16gb \ --count 0 \ --labels gpu-pool-type=on-demand

Notice the --count 0, which means this pool will not consume resources (and incur cost) until it's needed. It's labeled gpu-pool-type=on-demand for identification. Because it's untainted, any pod that tolerates the Spot taint (and thus can run on a GPU node) can also run here.

After creating the pools, you can verify their status using kubectl get nodes and doctl kubernetes cluster node-pool list <your-cluster-name>.

Step 4: Configuring Your GPU Workloads for Failover

To make your GPU workloads resilient, you need to configure their Kubernetes pod definitions to interact correctly with the tainted Spot GPU pool and allow failover to the untainted On-Demand GPU pool.

Pod Configuration:

Your GPU workload's pod specification needs two key elements:

  1. Toleration for the Spot Taint: This allows your pod to be scheduled on the Spot GPU nodes.
  2. Node Selector or Affinity: While not strictly necessary for failover if the On-Demand pool is untainted, a nodeSelector or nodeAffinity can be used to explicitly prefer or require GPU nodes. For primary scheduling on Spot, you'd typically target the gpu-pool-type=spot label. For failover, simply tolerating the taint is sufficient, as the Cluster Autoscaler will bring up On-Demand nodes for pending pods.

Here's an example of a pod definition that tolerates the gpu-pool-type=spot:NoSchedule taint:

apiVersion: apps/v1 kind: Deployment metadata: name: gpu-inference-app spec: replicas: 3 selector: matchLabels: app: gpu-inference template: metadata: labels: app: gpu-inference spec: containers: - name: inference-container image: your-gpu-image:latest resources: limits: nvidia.com/gpu: 1 tolerations: - key: "gpu-pool-type" operator: "Equal" value: "spot" effect: "NoSchedule"

In this example, the tolerations section explicitly allows this deployment's pods to be scheduled on nodes with the gpu-pool-type=spot:NoSchedule taint. When the Spot pool is reclaimed, these pods will become unschedulable on the remaining nodes (system pool). The Cluster Autoscaler, observing these pending pods, will then scale up the ondemand-gpu-pool (since it's untainted and matches the resource requirements), and the pods will be rescheduled there.

Application Best Practices:

  • Stateless Design: Design your applications to be stateless. If state is required, externalize it to a database or object storage.
  • Checkpointing: For long-running tasks like ML training, implement regular checkpointing to persistent storage. This allows your application to resume from the last saved state after an interruption, avoiding complete restarts.
  • Matching Hardware Specs: Ensure your On-Demand fallback pool uses GPU Droplets that meet or exceed the minimum requirements of your workload, ideally matching the Spot GPU pool's specifications.

Step 5: Testing Your Resilient Setup

Validating your failover mechanism is crucial. The best way to test is to simulate a Spot instance reclaim and observe your workloads gracefully migrating to the On-Demand fallback pool.

Deploy Your GPU Workload:

First, deploy your GPU-intensive application (e.g., the gpu-inference-app from the previous step) to your cluster. Verify that the pods are running on the Spot GPU nodes:

kubectl get pods -o wide

Look at the NODE column to confirm pods are on nodes belonging to your spot-gpu-pool.

Simulating a Reclaim:

While cloud providers have specific mechanisms for simulating reclaims, a practical way to test the failover is to manually delete the Spot GPU node pool. This mimics the sudden removal of resources.

doctl kubernetes cluster node-pool delete <your-cluster-name> --node-pool-id <spot-gpu-pool-id>

You can find the <spot-gpu-pool-id> using doctl kubernetes cluster node-pool list <your-cluster-name>.

Observing Failover:

After deleting the Spot GPU pool, monitor your cluster's state:

  1. Pod Status: Initially, your GPU pods will enter a Pending state as their nodes are removed.
  2. Cluster Autoscaler Activity: The Kubernetes Cluster Autoscaler will detect the pending pods that cannot be scheduled due to insufficient resources. Since your ondemand-gpu-pool is configured to scale from zero and is untainted, the autoscaler will begin provisioning new On-Demand GPU nodes in that pool.
  3. Node Provisioning: Observe the new nodes appearing in your cluster:
    4. kubectl get nodes --watch
  4. Workload Rescheduling: Once the On-Demand GPU nodes are ready, your pending GPU pods will be scheduled onto these new nodes and transition to a Running state.
    5. kubectl get pods -o wide --watch

This sequence confirms that your cluster successfully handles a Spot instance reclaim by automatically provisioning On-Demand resources and rescheduling your workloads, maintaining service availability.

Building a resilient Kubernetes cluster with Spot instance failover is a powerful strategy for optimizing cost while ensuring the availability of your critical GPU workloads. By carefully designing your node pools, configuring taints and tolerations, and implementing application best practices like checkpointing, you can leverage the economic benefits of Spot instances without compromising reliability. Learn more about building and hosting your online presence with Yammbo at https://yammbo.com.