Compute Cluster

Overview

A Compute Cluster connects DDOE to your Kubernetes environment and provides the infrastructure required to run applications, services, pipelines, preprocesses, and AI workloads. It serves as the execution layer where DDOE performs data processing, automation, and machine learning tasks.

The Compute Clusters page provides a centralized interface for viewing and managing Kubernetes clusters at the organization level. Accessible from the left-side navigation menu, it enables administrators to configure, monitor, and manage the compute resources available to the organization.

DDOE supports integration with an on-premises self-managed Kubernetes (vanilla) clusters, allowing organizations to leverage their existing infrastructure for running FaaS (Functions-as-a-Service) workloads and related services. Once connected, compute clusters can be used to deploy applications, execute pipelines, access connected storage integrations, and process data across projects.

Important

The Compute Cluster feature is supported only in Hybrid environments, and is not enabled by default.

It helps users:

• Monitor Cluster Status: Quickly view the status of all provisioned clusters (e.g., Failed, Ready, Validating) to identify any issues immediately

• View Cluster Details: Access essential information for each cluster including Provider (GCP, Azure, Local), Cluster Name, Number of Node Pools, Creation date and creator, Endpoint, Kubernetes Version, Default Namespace, and Status

• Manage Node Pools: View and manage node pools within selected clusters

• Add New Clusters: Create new clusters through a popup interface that allows configuring cluster details and node pools (available only to Org-admin and Org-owner roles)

• Edit Clusters: Modify existing cluster configurations and node pool settings

• Filter and Search: Easily find specific clusters using search and filter options.

Kubernetes Clusters

Kubernetes Clusters provide the underlying infrastructure that powers DDOE Compute Clusters. They enable organizations to deploy, manage, and scale applications, services, pipelines, and AI workloads without directly managing Kubernetes resources.

This section explains the key Kubernetes-related components available within a DDOE Compute Cluster, including Node Pools, Kubernetes Volumes, and Storage Integrations.

DDOE supports:

• Multi-Cloud Flexibility: Deploy workloads across AWS, Azure, GCP, and on-premises Kubernetes environments.

• Resource Isolation: Separate workloads across projects and organizations.

• High Availability: Leverage Kubernetes redundancy and failover capabilities.

• Resource Optimization: Use dedicated node pools, volumes, and storage integrations to efficiently manage compute and storage resources.

Organizations interact with Kubernetes through the Compute Clusters interface, allowing them to manage data operations, ML workflows, and applications without directly administering the Kubernetes infrastructure.

Learn here to create your compute cluster.

Node Pools

A Node Pool is a logical group of Kubernetes worker nodes within a Compute Cluster that share similar characteristics, such as CPU, memory, GPU, operating system, or workload type.

Kubernetes Volumes

Use Kubernetes Volumes to provide storage for applications and services running within the compute cluster. These volumes can be mounted by FaaS applications to store configuration files, secrets, models, logs, temporary files, and other application-specific resources.

DDOE supports the following Kubernetes volume types:

• Config Map: Stores non-sensitive configuration data as key-value pairs and makes it available to applications running in the cluster. Mount configuration data into containers.

• Secret: Stores sensitive information securely and provides it to applications when needed, such as credentials, tokens, and certificates.

• Host Path: Mount files or directories from the Kubernetes node's local file system.

• Empty Dir: Provides temporary storage that is created when a pod starts and removed when the pod is deleted.

• Persistent Volume Claim (PVC): Connects applications to persistent storage managed by Kubernetes.

• NFS: Mounts a shared storage from an external NFS server.

Note

Kubernetes Volumes are intended for application workloads only and cannot be used to store, manage, or synchronize dataset items. To access dataset data, use supported storage drivers such as NFS, FS, or S3.

Learn here to add Kubernetes Volume to your compute cluster.

Storage Integrations

Storage Integrations are a key feature of Compute Clusters in DDOE. They enable compute clusters to securely connect to external storage systems, allowing applications, services, pipelines, and datasets to access data directly from customer-managed storage locations.

Using Storage Integrations, you can connect your compute cluster to on-premises storage solutions such as:

• NFS

• S3 API-compatible storage

• Host Path

Once a storage integration is attached to a compute cluster, it becomes available for dataset creation through Compute Cluster Sync. This allows DDOE to access files directly from the connected storage without requiring data uploads or duplication.

Learn here to add storage integrations in your compute cluster.

Create Clusters

Prerequisites

Before you get started, make sure you have the following:

• Organization Admin or higher permissions

• The Organization ID - Found in your console

• Kubernetes cluster with API access

• Service Account with appropriate permissions in your cluster

• Python 3.8+ installed

• dtlpy SDK installed (minimum version 1.115.44): pip install "dtlpy>=1.115.44"

Service Account Permissions (RBAC)

DDOE needs permissions to manage workloads in your cluster. In most setups, namespace-scoped permissions are sufficient to create, update, delete, and monitor resources like pods, services, deployments, jobs, HPA, config maps, secrets, and PVCs. Optionally, pod execution access can be granted.

For a minimal namespace-scoped setup, you'll need:

• Namespace

• ServiceAccount

• Token Secret

• Role

• RoleBinding

Important: From Kubernetes 1.24 onward, ServiceAccounts no longer automatically generate a token Secret. You'll need to create an explicit kubernetes.io/service-account-token Secret (same as provided below) to get a stable JWT for authentication.token in your compute config.

Token Secret:

apiVersion: v1
kind: Namespace
metadata:
  name: <NAMESPACE>
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: <SERVICE_ACCOUNT_NAME>
  namespace: <NAMESPACE>
---
apiVersion: v1
kind: Secret
metadata:
  name: <SERVICE_ACCOUNT_NAME>
  namespace: <NAMESPACE>
  annotations:
    kubernetes.io/service-account.name: <SERVICE_ACCOUNT_NAME>
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: dataloop-minimal-role
  namespace: <NAMESPACE>
rules:
  - apiGroups: [""]
    resources:
      - pods
      - pods/exec
      - services
      - endpoints
      - configmaps
      - events
      - secrets
      - persistentvolumeclaims
      - serviceaccounts
    verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]

  - apiGroups: ["apps"]
    resources:
      - deployments
      - deployments/scale
      - daemonsets
    verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]

  - apiGroups: ["autoscaling"]
    resources:
      - horizontalpodautoscalers
      - horizontalpodautoscalers/status
    verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]

  - apiGroups: ["rbac.authorization.k8s.io"]
    resources:
      - roles
      - rolebindings
    verbs: ["create", "get", "list", "watch", "delete", "patch"]

  - apiGroups: ["batch"]
    resources:
      - jobs
    verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dataloop-minimal-rolebinding
  namespace: <NAMESPACE>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: dataloop-minimal-role
subjects:
  - kind: ServiceAccount
    name: <SERVICE_ACCOUNT_NAME>
    namespace: <NAMESPACE>

When Do You Need Cluster-Wide Permissions?

• Multiple namespaces: If you want DDOE to manage resources across multiple namespaces, use a ClusterRole + ClusterRoleBinding (or create the Role/RoleBinding in each namespace).

• Creating namespaces: If you want DDOE to create namespaces, you must grant access to the cluster-scoped namespaces resource.

Set Up Your Cluster

This section provides step-by-step instructions for setting up your Kubernetes clusters.

1. Open Compute Cluster from the left-side menu.

2. Click Create Cluster. A New Cluster Details section is displayed. Fill in all necessary details.    

   1. Cluster Name: Enter a unique name to identify your cluster.

   2. Endpoint: Enter the Kubernetes API server URL. The web address where your Kubernetes cluster can be accessed.

   3. CA Certificate: Enter CA Certificate details. Security certificate to verify the cluster's identity.

   4. Token: Enter service account JWT token. Authentication key to securely connect to the cluster.

   5. Namespace: Provide a namespace for the workload. A specific workspace within the cluster for organizing your applications.

3. Service Exposure Setting:

   1. Service Endpoint: Provide the IP/DNS address for accessing the service. The address where applications running on the cluster can be accessed.

   2. Public: Accessible from anywhere on the internet.

   3. Private: It restricts exposure and can use an internal gateway and load balancer. Only accessible within your internal network.

      1. Service Gateway: Provide an internal cluster IP used to route the traffic to the serve agent.

      2. Use Internal Load Balancer: By default, it is selected.

4. Node Pools: Groups of machines that run your applications. If not configured yet, click Add Node Pool configure it for you cluster.

5. Volumes: Use Kubernetes Volumes to provide persistent or shared storage for applications and services running within the compute cluster. If not configured yet, click Add Volume configure it for you cluster.

6. Environment Variables: In a DDOE Compute Cluster, the Environment Variables section is used to inject key-value pairs into the workloads (jobs, containers, or applications) running on the cluster. Environment variables allow you to provide configuration settings to your application without hardcoding them in code.

   1. Select a variable from the list. If not available, click Add Variable.

      1. Name: Enter a name for the variable.

      2. Value: Enter the value details assigned to that variable.

      3. Click Apply. The variable will be created and added to the list.

7. Storage Integrations: Configure Storage Integration for your cluster. If not available, click Add Integration. A panel appears on the right side where you can create your integration.

8. After configuring the Storage Integration, click Connect Cluster.

Add Node Pools for Your Cluster

While creating or editing your cluster in DDOE, you can configure Node Pools for your cluster.

1. Click Add Node Pool. A new node pool panel is displayed on the right-side, where you can start configuring it by fill in the following fields.

2. Node Pool Name: Enter a unique name to identify your node pool.

3. Instance Catalog: Select an instance catalog from the following DDOE list:

4. Tolerations (List): Kubernetes tolerations for tainted nodes. For example;  {"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}.

5. NodeSelector (JSON): Kubernetes node selector labels. For example; {"gpu": "true"},

6. Description: Enter a description for your node pool.

7. Preemptible: Enable whether to use preemptible/spot instances.

8. Run: AI Cluster

9. After configuring, click Create Node Pool.

Add Kubernetes Volumes

While creating or editing your cluster in DDOE, you can configure the Volume for your cluster.

1. Click Add Volume. A Volume Configuration panel is displayed on the right-side, where you can start configuring it by fill in the following fields.

2. Volume Name: Enter a unique name to identify your volume.

3. Volume Type: Select the volume type to be configured from the list:

   5. Config Map: A Config Map volume allows applications to access configuration information stored in Kubernetes.

      11. Config Map Name: Enter the Config Map Name.

      12. Items: In Items, specify the key-to-file mappings to be mounted inside the container.

          13. Key: Specify the Config Map key that contains the configuration value you want to mount. For example, application.properties.

          14. Path: Specify the file path where the selected key should be mounted inside the container. For example, config/application.properties.

   6. Secret: A Secret volume securely provides sensitive information to applications.

      15. Secret Name: Enter the name of the Kubernetes Secret that contains the data you want to make available to the application.

      16. Items: Use Items to specify which secret values should be mounted and where they should appear inside the container.

          17. Key: Specify the secret key that contains the value to be mounted. For example, api-key, tls-crt, etc.

          18. Path: Specify the file path where the selected secret key should be mounted inside the container. For example, secrets/api-key.

   7. Host Path: A Host Path volume mounts a file or directory from the Kubernetes node into the container.

      19. Read Only: Enable Read Only if write access should be restricted.

      20. Path: Enter the absolute path of the file or directory on the Kubernetes node that you want to mount. For example, /var/log/application.

      21. Type: Select the type of host path that Kubernetes should expect or create.

          22. Directory Or Create: Specifies that the path should be a directory. If the directory does not exist on the host machine, Kubernetes automatically creates it before mounting.

          23. Directory: Specifies that the path must already exist as a directory on the host machine. If the directory does not exist, the pod will fail to start.

   8. Empty Dir: An Empty Dir volume provides temporary storage that exists for the lifetime of a pod. Data stored in an Empty Dir volume is deleted when the pod is removed.

      24. Size Limit: Specify the Size Limit, if required.

   9. Persistent Volume Claim (PVC): A PVC volume attaches existing persistent storage managed by Kubernetes.

      25. Claim Name: Enter the Claim Name of the existing PVC. Claims must exist in the same namespace as the Pod using the claim.

   10. NFS: An NFS volume mounts storage from an external Network File System (NFS) server.

       26. NFS Server Address: Enter the NFS Server Address.

       27. Export Path: Enter the Export Path that should be mounted.

4. Once you configure the volume, click Add to Cluster. The newly created Volume will be listed under the Volumes section of the new cluster creation window.

Add Storage Integrations

While creating or editing your cluster in DDOE, you can configure the Storage Integrations for your cluster.

1. Integration Name: Enter a unique name to identify your integration.

2. Provider: By default, On-Prem is selected.

3. Integration Type: DDOE supports three types of On-Premises integration:  

   1. NFS: Select NFS from the list follow the steps:

      1. NFS Server Address: Enter the IP address or the hostname of your NFS server.

      2. Export Path: Enter the directory path on the NFS server that is exported or shared.

   2. Host Path: Select Host Path from the list and follow the steps:

      1. In Path, enter the directory path on the host machine that you want to expose to the compute cluster.

      2. In Type, select the host path mount type:

         • Directory: The specified directory must already exist on the host.

         • DirectoryOrCreate: Creates the directory automatically if it does not already exist.

      3. Enable Read Only if the mounted path should be accessible for viewing only and should not allow modifications.

   3. S3 API: Select S3 API from the list and follow the steps:

      1. In Access Key ID, enter the access key provided by your S3-compatible storage administrator.

      2. In the Secret Access Key, enter the secret key associated with the access key.

      3. In the Endpoint URL, enter the URL of the S3-compatible storage service.

      4. In Region, enter the storage region configured for the S3 service. If your storage administrator has not specified a region, use the default value provided for your environment.

4. After configuring, click Create Integration.

Learn more about storage Integrations in DDOE.

Difference between Kubernetes Volume and Storage Integrations

DDOE supports both Storage Integrations and Kubernetes Volumes, but they serve different purposes.

• Storage Integrations connect external storage systems to a compute cluster, allowing datasets and applications to access customer-managed data.

• Kubernetes Volumes provide storage or configuration resources that can be attached to DDOE-managed deployments, such as FaaS applications, services, and platform agents.

Make Your Cluster as Default

You can make your cluster as default in DDOE.

1. Open Compute Cluster from the left-side menu.

2. Find your compute cluster from the list.

3. Click the Three-dots and select the Set as default from the list. A confirmation message is displayed.

Edit Compute Clusters

You can make changes to your existing Kubernetes clusters in DDOE.

1. Open Compute Cluster from the left-side menu.

2. Find your compute cluster from the list.

3. Click the Three-dots and select the Edit from the list. An Edit Cluster popup is displayed.

4. You can only make changes in the following fields:

   1. CA / Token

   2. Namespace

   3. Service Endpoint

   4. Make the service Public or Private

   5. Add or change node pools

   6. Add or change Volume

   7. Add or change Storage Integrations.

5. Click Save Changes. The changes will reset the cluster and update all integrations, even if there are connected services and datasets.

Delete Compute Clusters

• You can delete only the compute clusters that belong to your active organization. Global Compute Clusters cannot be deleted.

• Before deleting a compute cluster, ensure that no active services, applications, or workloads depend on it.

• If services are currently running on the cluster, move them to another compute cluster or delete them before proceeding. Deleting a compute cluster may affect the operation of services deployed on that cluster.

1. Open Compute Cluster from the left-side menu.

2. Find your compute cluster from the list.

3. Click the Three-dots and select the Delete from the list. A confirmation message is displayed.

4. Click Delete.