S3 API

Overview

S3 API is a standard set of web-based operations used to interact with object storage systems. It was originally introduced by Amazon S3 (Simple Storage Service), and many storage platforms now support the same API standard.

An S3 API Integration allows DDOE to connect to an S3-compatible object storage system that is hosted within your organization's environment (on-premises) instead of using a public cloud provider.

This means applications can use the same API to work with different storage systems, such as:

AWS S3
Dell ECS
MinIO
Ceph Object Storage
Scality
Cloudian

Purpose

The primary purpose of the S3 API is to provide a standardized way to store, retrieve, and manage files (objects) in object storage. Instead of accessing files through mounted folders like NFS, applications communicate with storage using API requests.

How S3 API Works

Unlike traditional file systems, S3 stores data as objects inside a bucket.

Traditional File Storage

Folder
├── Images
│   ├── image1.jpg
│   └── image2.jpg

S3 Object Storage

Bucket: training-data

Objects:
images/image1.jpg
images/image2.jpg

When an application needs a file: Application → S3 API Request → Object Storage → Return File

Instead of opening a local file path, the application sends an HTTP request to retrieve the object.

How S3 API Works in DDOE

Dataset → Storage Driver → Streamer Service → S3 API → Object Storage

Example

A customer stores images in MinIO:

Bucket:
autonomous-driving

Files:
camera/frame001.jpg
camera/frame002.jpg

DDOE accesses these files through:

S3 Integration
Compute Cluster
Storage Driver
Streamer Service

The files remain in MinIO and are not copied into DDOE.

Advantages of S3 API

Highly Scalable: Object storage can handle. This makes it ideal for AI and enterprise workloads.
- Millions of files
- Billions of objects
- Petabytes of data
No Data Duplication: Files remain in storage and are accessed directly by DDOE.
Cloud and On-Prem Support: The same API works with:
- AWS S3
- Dell ECS
High Availability: Object storage platforms are designed to:
- Replicate data
- Prevent data loss
- Support large-scale deployments
Easy Integration: Most modern tools and applications support S3 APIs.

When Should You Use S3 API?

Use S3 API when:

You have large datasets
You need highly scalable storage
You use Dell ECS, MinIO, Ceph, or AWS S3
You want a Direct Data architecture
Multiple applications need access to the same storage
You are building AI/ML platforms

Create an On-Premises S3 API Integration

Prerequisites
Roles and Permissions: Only users with the Organization Admin or Owner roles are authorized to create integrations. Learn more
Access Key ID: The Access Key ID acts like a username that identifies your application or user account when connecting to the S3 storage. To get it for example, AWS S3: IAM User → Security Credentials → Access Keys.
Secret Access Key: The Secret Access Key acts like a password associated with the Access Key ID. Generated together with the Access Key ID.
Endpoint URL: The Endpoint URL is the network address of the S3-compatible storage service.
Region: The Region identifies the geographical or logical location of the storage. For example, AWS S3: Available in the bucket properties.

Log in to the DDOE platform.
From the left-side panel, select Integrations.
Click Create Integration -> Create Storage Integration. A pop-up window is displayed on the right-side.
In Integration Name, enter a name for the S3 API integration.
From the Provider dropdown, select On-Prem.
From the Integration Type dropdown, select S3 API.
In Access Key ID, enter the access key provided by your S3-compatible storage administrator.
In the Secret Access Key, enter the secret key associated with the access key.
In the Endpoint URL, enter the URL of the S3-compatible storage service (for example, Dell ECS, MinIO, or Ceph).
In Region, enter the storage region configured for the S3 service. If your storage administrator has not specified a region, use the default value provided for your environment.
Click Create Integration. The S3 API integration is created and appears on the Integrations page, where it can be attached to a compute cluster and used for dataset creation through Compute Cluster Sync.

Documentation Index