Documentation Index

Fetch the complete documentation index at: https://docs.dataloop.ai/llms.txt

Use this file to discover all available pages before exploring further.

Create Datasets

  • Updated On Jul 22, 2026
  • Published on Apr 21, 2025
Prev Next

Overview

DDOE enables you to create and manage datasets while giving you flexibility in choosing your preferred storage location. You can upload data directly into DDOE as a local upload, ingest data using the SDK, or connect external storage such as cloud storage services or on‑premises storage systems.

Now the Data page, when you click on Create Dataset will enable you choose all these options.

Learn more about the supported data formats.

Upload Your Data in DDOE (Local Upload)

DDOE storage is the platform’s internal dataset storage, where files are securely stored on DDOE’s GCP‑hosted cloud storage. This internal storage enables you to manage digital files—such as images, videos, audio files, text files, and other data—used for the annotation process.

  1. Log in to the DDOE platform.

  2. From the left‑side navigation panel, select Data. Alternatively, use the Dashboard → Data Management widget.

  3. In the Datasets tab, click Create Dataset, or click on the down-arrow and select Create Dataset from the list. The New Dataset popup is displayed.

  4. Select Local Upload tile from list. The Dataset Details section is displayed.

  1. Dataset Name: Enter a Name for the dataset.

  2. Recipe: By default, the Project Recipe is selected. Select a recipe based on your data type. If suitable recipe is not available, click Create New to create a recipe for your dataset.

Information

Refer to the What you need to know on the right-side panel for additional information.

  1. Click Create Dataset. The new dataset will be created, and the Dataset Browser page is displayed. You can now start uploading your data.

Connect and Sync Your Cloud Storages

Cloud storage services are online platforms that allow organization to store and manage their data. DDOE supports the following cloud storage services:

  • Amazon Web Services (AWS) S3: Amazon S3 (Simple Storage Service) is a highly scalable, object storage service offered by AWS.

  • Microsoft Azure Blob Storage: Microsoft Azure provides Blob Storage for storing and managing unstructured data. It integrates well with other Azure services.

  • Google Cloud Storage: Google Cloud Storage is part of the Google Cloud Platform and offers object storage, archival storage, and data transfer services. It's often used alongside other GCP services.

  1. Log in to the DDOE platform.

  2. From the left‑side navigation panel, select Data. Alternatively, use the Dashboard → Data Management widget.

  3. In the Datasets tab, click Create Dataset, or click on the down-arrow and select Create Dataset from the list. The New Dataset popup is displayed.

  4. Select Cloud Storage Sync tile from list.

  5. Select your storage drivers from the list.

Enables DDOE to securely connect to AWS services (such as S3), allowing you to sync, manage, and process datasets stored in AWS cloud environments.

  1. Select AWS S3 Bucket tile from the list.

  2. Select the Integration from the list. If suitable integration is not available, click Add Integration.

    1. Integration Name: Enter the name of the integration.

    2. Integration Type: Select the integration type from the list. Refer to the links for more information.

      1. Cross Account

        1. DDOE IAM User ARN

        2. IAM Role ARN

      2. STS

        1. AWS Access Key ID

        2. AWS Secret Access Key

        3. IAM Role ARN

      3. Access Key

        1. AWS Access Key ID

        2. AWS Secret Access Key

    3. Click Create Integration. The Integration will be created and listed.

  3. Dataset Name: Enter a Name for the dataset.

  4. Recipe: By default, the Project Recipe is selected. Select a recipe based on your data type. If suitable recipe is not available, click Create New to create a recipe for your dataset..

  5. Bucket Name: Enter your S3 bucket name.

  6. Path Prefix: Specify the directory path prefix within the bucket

  7. Region: Select the AWS region from the list.

  8. Storage Class: Enter your S3 storage class.

  9. Allow Deletion of Items from Storage:

    1. If Yes, deleting items in DDOE will permanently delete them from your external storage.

    2. If No, items deleted in DDOE will remain in your storage and won't be restored during re-sync.

  10. Run initial sync: Enable to start the initial sync. You can also set up Automatic sync.

  11. Click Create Dataset. The new dataset will be created, and the Dataset Browser page is displayed. You can now start syncing your data.

Allows DDOE to access and synchronize data from Google Cloud Platform services (such as GCS), supporting seamless dataset ingestion and management within DDOE workflows.

  1. Select GCP GCS Bucket tile from the list.

  2. Select an Integration from the list. If suitable integration is not available, click Add Integration.

    1. Integration Name: Enter the name of the integration.

    2. Integration Type: Select the integration type from the list. Refer to the links for more information.

      1. Cross Project

        1. Service Account ID

        2. Bucket Name

      2. Workload Identity Federation: It allows DDOE to securely access cloud resources without storing long‑lived credentials (like service account keys or access keys).

        1. Azure Token URL

        2. Azure Client ID

        3. Azure Client Secret

        4. Azure Scope

        5. GCP Credential Configuration File

      3. Private Key

        1. JSON Private Key

    3. Click Create Integration. The Integration will be created and listed.

  3. Dataset Name: Enter a Name for the dataset.

  4. Recipe: By default, the Project Recipe is selected. Select a recipe based on your data type. If suitable recipe is not available, click Create New to create a recipe for your dataset.

  5. Bucket Name: Enter your GCS bucket name.

  6. Path Prefix: Specify the directory path prefix within the bucket

  7. Allow Deletion of Items from Storage:

    1. If Yes, deleting items in DDOE will permanently delete them from your external storage.

    2. If No, items deleted in DDOE will remain in your storage and won't be restored during re-sync.

  8. Run initial sync: Enable to start the initial sync. You can also set up Automatic sync.

  9. Click Create Dataset. The new dataset will be created, and the Dataset Browser page is displayed. You can now start syncing your data.

Connects DDOE with Azure storage services (such as Blob Storage and Data Lake Gen2), enabling secure data synchronization and management directly from Azure cloud storage.

  1. Select Azure tile from the list.

  2. Select an Integration from the list. If suitable integration is not available, click Add Integration.

    1. Integration Name: Enter the name of the integration.

    2. Tenant ID: Go to Azure Active Directory → Overview, Copy the Tenant ID

    3. Client ID: The Client ID is the Application (App Registration) ID.

    4. Client Secret: The Client Secret is generated under the app registration.

    5. Storage Account Name: A Storage Account Name is the unique name assigned to an Azure Storage Account, which is used to identify and access storage resources such as Blob Storage and Data Lake within Azure. For example, https://<storage-account-name>.blob.core.windows.net

    6. Click Create Integration. The Integration will be created and listed.

  3. Dataset Name: Enter a Name for the dataset.

  4. Recipe: By default, the Project Recipe is selected. Select a recipe based on your data type. If suitable recipe is not available, click Create New to create a recipe for your dataset.

  5. Storage Type: Select the Azure storage type you want to configure:

    1. Blob Storage: Select this option to store data in an Azure Blob container by specifying the container name.

      1. Container Name: Enter the name of the blob container.

    2. Data Lake Storage Gen2: Select this option to store data in an Azure Data Lake Gen2 file system by specifying the file system name.

      1. File System Name: Enter the name of the file system.

  6. Path Prefix: Specify the directory path prefix within the bucket

  7. Allow Deletion of Items from Storage:

    1. If Yes, deleting items in DDOE will permanently delete them from your external storage.

    2. If No, items deleted in DDOE will remain in your storage and won't be restored during re-sync.

  8. Run initial sync: Enable to start the initial sync. You can also set up Automatic sync.

  9. Click Create Dataset. The new dataset will be created, and the Dataset Browser page is displayed. You can now start syncing your data.

Information

Refer to the What you need to know on the right-side panel for additional information.

Connect and Sync Your On-Premise Data

Enables to connect to storage hosted in your own data center or infrastructure and automatically sync data.

Important

The On-Prem Sync feature is available only in Hybrid and On-Premises environments.

  1. Log in to the DDOE platform.

  2. From the left‑side navigation panel, select Data. Alternatively, use the Dashboard → Data Management widget.

  3. In the Datasets tab, click Create Dataset, or click on the down-arrow and select Create Dataset from the list. The New Dataset popup is displayed.

  4. Select On-Prem Sync tile from list.

  5. Select your storage type Network File System (NFS), NFS with MetadataIQ, S3-compatible API, and S3 API with MetadataIQ.

Enables DDOE to connect to on‑premises file storage using standard NFS, allowing datasets to be synced and managed directly from local file systems.

NFS Share Mounting

Make sure your NFS share is mounted before creating the dataset. Contact our support team for setup assistance.

  1. Select Network File System tile from the list.

  2. Select the NFS Integration from the list. If not available, click Add Integration to create one.

    1. Integration Name: Provide a name for your NFS Integration.

    2. NFS Server Address: Enter the IP address or the hostname of your NFS server.

    3. Export Path: Enter the directory path on the NFS server that is exported or shared.

    4. Click Create Integration. The NFS type integration will be created and listed in the Integration page.

  3. Dataset Name: Enter a Name for the dataset.

  4. Recipe: By default, the Project Recipe is selected. Select a recipe based on your data type. If suitable recipe is not available, click Create New to create a recipe for your dataset.

  5. Path Prefix: Specify the directory path prefix within the bucket.

  6. Allow Deletion of Items from Storage:

    1. If Yes, deleting items in DDOE will permanently delete them from your external storage.

    2. If No, items deleted in DDOE will remain in your storage and won't be restored during re-sync.

  7. Run initial sync: Enable to start the initial sync. You can also set up Automatic sync.

  8. Click Create Dataset. The new dataset will be created, and the Dataset Browser page is displayed. You can now start syncing your data.

Dell PowerScale OneFS: Enables DDOE to connect to on‑premises NFS storage while leveraging MetadataIQ to accelerate data discovery, indexing, and metadata‑driven dataset management.

NFS Share Mounting

Make sure your NFS share is mounted and MetadataIQ is properly configured before creating the dataset. Contact our support team for setup assistance.

  1. Select NFS with MetadataIQ tile from the list.

  2. Select the Integration from the list. If not available, click Add Integration.

    1. Integration Name: Provide a name for your NFS Integration.

    2. NFS Server Address: Enter the IP address or the hostname of your NFS server.

    3. Export Path: Enter the directory path on the NFS server that is exported or shared.

    4. Click Create Integration. The NFS type integration will be created and listed in the Integration page.

  3. Dataset Name: Enter a Name for the dataset.

  4. Recipe: By default, the Project Recipe is selected. Select a recipe based on your data type. If suitable recipe is not available, click Create New to create a recipe for your dataset.

  5. Elastic Index Name: Enter the name of the Elasticsearch index created in by MetadataIQ. The Elastic Index Name refers to the Elasticsearch index created and maintained by MetadataIQ for the target NFS data source.

  6. Elastic Index Path: Specify the directory path.

  7. Allo Deletion of Items from Storage:

    1. If Yes, deleting items in DDOE will permanently delete them from your external storage.

    2. If No, items deleted in DDOE will remain in your storage and won't be restored during re-sync.

  8. Run initial sync: Enable to start the initial sync. You can also set up Automatic sync.

  9. Click Create Dataset. The new dataset will be created, and the Dataset Browser page is displayed. You can now start syncing your data.

Allows DDOE to integrate with on‑premises object storage that exposes an S3‑compatible API, enabling cloud‑like data access and synchronization.

  1. Select S3-Compatible API tile from the list.

  2. Select an Integration from the list. If suitable integration is not available, click Add Integration.

    1. In Integration Name, enter a name for the S3 API integration.

    2. In Access Key ID, enter the access key provided by your S3-compatible storage administrator.

    3. In the Secret Access Key, enter the secret key associated with the access key.

    4. In the Endpoint URL, enter the URL of the S3-compatible storage service (for example, Dell ECS, MinIO, or Ceph).

    5. In Region, enter the storage region configured for the S3 service. If your storage administrator has not specified a region, use the default value provided for your environment.

    6. Click Create Integration. The S3 API integration is created and appears on the Integrations page.

  3. Dataset Name: Enter a Name for the dataset.

  4. Recipe: By default, the Project Recipe is selected. Select a recipe based on your data type. If suitable recipe is not available, click Create New to create a recipe for your dataset.

  5. Bucket Name: Enter your S3 bucket name.

  6. Path Prefix: Specify the directory path prefix within the bucket

  7. Allo Deletion of Items from Storage:

    1. If Yes, deleting items in DDOE will permanently delete them from your external storage.

    2. If No, items deleted in DDOE will remain in your storage and won't be restored during re-sync.

  8. Run initial sync: Enable to start the initial sync. You can also set up Automatic sync.

  9. Click Create Dataset. The new dataset will be created, and the Dataset Browser page is displayed. You can now start syncing your data.

Combines S3‑compatible object storage access with MetadataIQ to provide enhanced metadata indexing, faster data ingestion, and efficient dataset management in DDOE.

MetadataIQ Configuring

Make sure MetadataIQ is properly configured before creating the dataset. Contact our support team for setup assistance.

  1. Select S3-Compatible API tile from the list.

  2. Select an Integration from the list. If suitable integration is not available, click Add Integration.

    1. In Integration Name, enter a name for the S3 API integration.

    2. In Access Key ID, enter the access key provided by your S3-API storage administrator.

    3. In the Secret Access Key, enter the secret key associated with the access key.

    4. In the Endpoint URL, enter the URL of the S3-API Integration.

    5. In Region, enter the storage region configured for the S3 service. If your storage administrator has not specified a region, use the default value provided for your environment.

    6. Click Create Integration. The S3 API integration is created and appears on the Integrations page.

  3. Dataset Name: Enter a Name for the dataset.

  4. Recipe: By default, the Project Recipe is selected. Select a recipe based on your data type. If suitable recipe is not available, click Create New to create a recipe for your dataset.

  5. Elastic Index Name: Enter the name of the Elasticsearch index created in by MetadataIQ. The Elastic Index Name refers to the Elasticsearch index created and maintained by MetadataIQ for the target NFS data source.

  6. Elastic Index Path: Specify the directory path.

  7. Bucket Name: Enter your S3 bucket name.

  8. Path Prefix: Specify the directory path prefix within the bucket

  9. Run initial sync: Enable to start the initial sync. You can also set up Automatic sync.

  10. Click Create Dataset. The new dataset will be created, and the Dataset Browser page is displayed. You can now start syncing your data.

Information

Refer to the What you need to know on the right-side panel for additional information.

Connect Your Compute Cluster

The Compute Cluster Sync option allows you to create datasets directly from storage integrations connected to a compute cluster. When you select this option, DDOE retrieves the compute clusters available in your active organization and displays only those that have one or more configured storage integrations.

Each compute cluster contains a storage-integrations plugin that maintains information about the storage integrations attached to that cluster. A compute cluster can be connected to multiple storage integrations, and a single storage integration can be associated with multiple compute clusters.

Important

The Compute Cluster Sync feature is available only in Hybrid and On-Premises environments.

Note:

Global Compute Clusters are not displayed in the Compute Cluster Sync list. Only organization-specific compute clusters with configured storage integrations are available for dataset creation.

Prerequisites

Before creating a dataset using Compute Cluster Sync, ensure that:

  • A compute cluster has been created within your organization.

  • The compute cluster is active and available.

  • At least one storage integration is attached to the compute cluster.

  • You have the required permissions to create datasets.

  • The target storage location (bucket, container, file share, directory, etc.) exists and is accessible.

  • The required recipe is available in the project.

  1. Log in to the DDOE platform.

  2. From the left‑side navigation panel, select Data. Alternatively, use the Dashboard → Data Management widget.

  3. In the Datasets tab, click Create Dataset, or click on the down-arrow and select Create Dataset from the list. The New Dataset popup is displayed.

  4. Select Compute Cluster Sync tile. The platform displays all eligible compute clusters in the organization that have configured storage integrations. Learn here to create a new compute cluster.

  5. Select a Compute Cluster from the list. You can use the search or filter option to find a relevant one. The Dataset Details section is displayed.

  6. Dataset Name: Enter a unique name for the dataset.

  7. Recipe: Select a recipe from the list. By default, the Project recipe is selected.

  8. Bucket Name: Provide your bucket name of your storage.

  9. In Path Prefix, optionally specify a path within the storage location to limit access to a specific folder or directory.

  10. Click Create Dataset. The dataset is created and linked directly to the selected storage integration, enabling DDOE to access data from the connected storage location without requiring manual uploads or data synchronization.

Create Datasets with Sample Data

Enables to browse and install datasets from DDOE’s Marketplace.

  1. Log in to the DDOE platform.

  2. From the left‑side navigation panel, select Data. Alternatively, use the Dashboard → Data Management widget.

  3. In the Datasets tab, click Create Dataset, or click on the down-arrow and select Create Dataset from the list. The New Dataset popup is displayed.

  4. Select Sample Datasets tile from list. The Marketplace page is displayed.

  1. Select a dataset based on your requirement. You can use the search and filter on the left-side panel.

  2. After making a selection, a Details panel appears on the right side. Click Install to install the selected dataset. Once installation is complete, the Dataset Browser page is displayed.

    1. Installed datasets are marked with a green checkmark ✅ icon.

    2. Auto-Update: In the Details panel on the right, enable Auto‑Update to automatically update installed datasets when a newer version becomes available in the Marketplace.

Upload Data in DDOE Using the SDK

Enables programmatic data import into DDOE using the SDK.

  1. Log in to the DDOE platform.

  2. From the left‑side navigation panel, select Data. Alternatively, use the Dashboard → Data Management widget.

  3. In the Datasets tab, click Create Dataset, or click on the down-arrow and select Create Dataset from the list. The New Dataset popup is displayed.

  4. Select Import via SDK tile from list. The developer guide page is displayed, start following the steps as per your requirement.

Copyright © 2026 Dell Inc.