---
title: "S3 API"
slug: "s3-api"
updated: 2026-08-04T06:45:52Z
published: 2026-08-04T06:45:52Z
canonical: "docs.dataloop.ai/s3-api"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dataloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# S3 API

## Overview

**S3 API** is a standard set of web-based operations used to interact with **object storage systems**. It was originally introduced by Amazon S3 (Simple Storage Service), and many storage platforms now support the same API standard.

An **S3 API Integration** allows DDOE to connect to an **S3-compatible object storage system** that is hosted within your organization's environment (on-premises) instead of using a public cloud provider.

---

## Purpose

The primary purpose of the S3 API is to provide a **standardized way to store, retrieve, and manage files (objects)** in object storage. Instead of accessing files through mounted folders like NFS, applications communicate with storage using API requests.

---

## How S3 API Works

Unlike traditional file systems, S3 stores data as **objects** inside a **bucket**.

**Traditional File Storage**

```plaintext
Folder
├── Images
│   ├── image1.jpg
│   └── image2.jpg
```

S3 Object Storage

```plaintext
Bucket: training-data

Objects:
images/image1.jpg
images/image2.jpg
```

When an application needs a file: ***Application → S3 API Request → Object Storage → Return File***

Instead of opening a local file path, the application sends an HTTP request to retrieve the object.

---

## How S3 API Works in DDOE

Dataset → Storage Driver → Streamer Service → S3 API → Object Storage

**Example**

A customer stores images in MinIO:

```plaintext
Bucket:
autonomous-driving

Files:
camera/frame001.jpg
camera/frame002.jpg
```

DDOE accesses these files through:

1. S3 Integration
2. Compute Cluster
3. Storage Driver
4. Streamer Service

The files remain in MinIO and are not copied into DDOE.

---

## Advantages of S3 API

- **Highly Scalable**: Object storage can handle. This makes it ideal for AI and enterprise workloads.
  - Millions of files
  - Billions of objects
  - Petabytes of data
- No Data Duplication: Files remain in storage and are accessed directly by DDOE.
- Cloud and On-Prem Support: The same API works with:
  - AWS S3
  - Dell ECS
- High Availability: Object storage platforms are designed to:
  - Replicate data
  - Prevent data loss
  - Support large-scale deployments
- Easy Integration: Most modern tools and applications support S3 APIs.

---

## When Should You Use S3 API?

**Use S3 API when:**

- You have large datasets
- You need highly scalable storage
- You use Dell ECS, MinIO, Ceph, or AWS S3
- You want a Direct Data architecture
- Multiple applications need access to the same storage
- You are building AI/ML platforms

---

## Create an On-Premises S3 API Integration

> [!WARNING]
> **Prerequisites**
> 
> - **Roles and Permissions**: Only users with the Organization **Admin** or **Owner** roles are authorized to create integrations. [Learn more](https://docs.dataloop.ai/docs/org-members-groups#org-role-permissions)
> - **Access Key ID:** The **Access Key ID** acts like a username that identifies your application or user account when connecting to the S3 storage. To get it for example, **AWS S3**: IAM User → Security Credentials → Access Keys.
> - **Secret Access Key**: The Secret Access Key acts like a password associated with the Access Key ID. Generated together with the Access Key ID.
> - **Endpoint URL:** The **Endpoint URL** is the network address of the S3-compatible storage service.
> - **Region:** The **Region** identifies the geographical or logical location of the storage. For example, **AWS S3**: Available in the bucket properties.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/image(739).png)

1. Log in to the DDOE platform.
2. From the left-side panel, select **Integrations**.
3. Click **Create Integration** -> **Create Storage Integration**. A pop-up window is displayed on the right-side.
4. In **Integration Name**, enter a name for the S3 API integration.
5. From the **Provider** dropdown, select **On-Prem**.
6. From the **Integration Type** dropdown, select **S3 API**.
7. In **Access Key ID**, enter the access key provided by your S3-compatible storage administrator.
8. In the **Secret Access Key**, enter the secret key associated with the access key.
9. In the **Endpoint URL**, enter the URL of the S3-compatible storage service (for example, Dell ECS, MinIO, or Ceph).
10. In **Region**, enter the storage region configured for the S3 service. If your storage administrator has not specified a region, use the default value provided for your environment.
11. Click **Create Integration**. The S3 API integration is created and appears on the **Integrations** page, where it can be attached to a compute cluster and used for dataset creation through **Compute Cluster Sync**.