Overview
Storage Integrations act as the bridge between DDOE and your storage systems. They allow DDOE datasets, Compute Clusters, applications, and pipelines to securely access data stored in cloud or on-premises environments without requiring manual uploads or data duplication.
They provide a centralized way to configure and manage storage connections while handling authentication, authorization, and secure credential management.
DDOE supports integrations with:
NFS
S3 API-compatible storage
Host Path
Why Use Storage Integrations?
Storage Integrations eliminate the need to manually upload and manage large volumes of data inside DDOE. Instead of copying data into the platform, DDOE accesses the data directly from its original storage location.
Common Uses of Storage Integrations
Dataset Creation
Create datasets directly from connected storage sources such as:
AWS S3 buckets
Azure Blob Storage containers
GCP Storage buckets
On-premises NFS shares
S3-compatible object storage
This allows you to work with existing data without re-uploading it.
Cloud Storage Synchronization
Connect cloud storage providers and synchronize data between DDOE and your cloud environment.
Examples:
AWS S3
Azure Blob Storage
Google Cloud Storage
This keeps datasets aligned with the storage source.
Compute Cluster Storage Access
Storage Integrations are tightly integrated with Compute Clusters. Once attached to a Compute Cluster, the storage becomes available for:
Dataset creation through Compute Cluster Sync
AI applications
Services
Pipelines
FaaS workloads
This enables workloads running inside Kubernetes to access data directly from connected storage systems.
Hybrid and Multi-Cloud Environments
Organizations can connect and manage storage across:
Multiple cloud providers
On-premises infrastructure
Kubernetes environments
AWS: Why Cross-Account Integration is the Best Choice?
AWS Cross-Account Integration is the recommended approach for securely connecting AWS resources to DDOE.
Key benefits:
Least-Privilege Access: Grants only the permissions required for the integration, reducing security risks.
Credential Rotation: Uses IAM roles instead of long-lived access keys, making credential management more secure and easier to maintain.
Improved Auditing: Provides detailed logs of all actions performed through the integration, improving visibility and compliance.
Separation of Responsibilities: Allows DDOE to access only the required AWS resources without exposing AWS account credentials.
Overall, AWS Cross-Account Integration provides a more secure, flexible, and manageable alternative to Access Keys or STS-based integrations.
For configuration instructions, see the AWS Cross Account Integration article.
GCP: Why Cross-Project Integration is the Better Choice?
GCP Cross-Project Integration is the recommended approach for securely connecting Google Cloud resources to DDOE.
Key benefits:
Least-Privilege Access: Grants only the permissions required for the integration, reducing security risks.
Credential Rotation: Uses IAM roles and service accounts instead of long-lived private keys, improving security and simplifying credential management.
Improved Auditing: Provides detailed activity logs, making it easier to monitor access and investigate issues.
Separation of Responsibilities: Allows organizations to delegate access to specific resources without exposing project credentials.
Overall, GCP Cross-Project Integration provides a more secure, flexible, and manageable alternative to Private Key–based integrations, helping organizations maintain stronger control over their cloud resources while simplifying access management.
For configuration instructions, see the GCP Cross Project Integration article.
Important Considerations
When you set up an external cloud storage:
Choose a nearby storage region to improve file loading and annotation performance.
Provide write access so DDOE can create thumbnails, converted files, and manage the hidden
dataloopfolder.Synchronize file deletions. If files are removed from external storage, configure Upstream Sync or manually update DDOE to keep data and metadata consistent.
Create Storage Integrations
This integration allows DDOE to connect to external storage services, such as cloud storage (AWS S3, Google Cloud Storage, Azure Blob Storage), and on-premises systems, for seamless access to datasets and files.
Who can create?
Only users with the Organization Admin or Owner roles are authorized to create integrations.
You can manage and import/export datasets between DDOE and external storage systems.
This is especially important for handling large volumes of data used in machine learning and AI projects.
Clicking on the Create Integration -> Create Storage Integration allows you to create the following types of integrations:
View Your Integrations
Access the Integrations page by clicking on the Integrations from the left-side menu. The page displays the list of Integrations available in your organization.
Search and Filter Integrations
The following list provides the specific criteria of search and filters for Integrations:
To search: Search Integrations by Integrations Name.
To Filter: Filter the listed integrations by the following criteria:
Provider: The available storage providers for the datasets.
AWS
Azure
GCP
Type: The type of the Integrations.
AWS: Cross Account, STS, and Access Key
GCP: Cross Project, and Private Key
Azure: Client Secret
Untrusted Integrations
Filter integrations based on the email ID of the creator.
Integration Details
Integrations page displays available Integrations in your organization in a list view. The column values are populated according to the available Integrations.
Column Name | Description |
|---|---|
Provider icon | The icon of the cloud storage provider. |
Integration Name | The name of the integrations. |
Integration Type | The type of the integration depends on the cloud storage provider. |
Completed | It displays whether the necessary steps are completed to establish trust on the Integration. A green-tick mark is displayed if the integration is trusted. If not, a warning icon with a tool-tip explaining the current status is displayed. |
Created At | The creation date of the Integration. |
Created By | The Avatar of the user who created the Integration. You can see the email ID of the user when you hover. |
Click on the More actions (three dots) to view and perform the following actions:
Rename Integration
Edit Integration: When you click, you can modify Access Key integrations' Key and Secret values.
Container Registry integration is editable only from DDOE SDK.
You cannot edit cross integration.
Copy Integration ID
Delete Integration