Contents x
    Create Storage Drivers
    • 28 Apr 2025
    • Print
    • Dark
      Light
    • PDF
    Contents

    Create Storage Drivers

    • Updated On 28 Apr 2025
    • Print
    • Dark
      Light
    • PDF
    Article summary

    Create AWS S3 Storage Drivers

    If you're using the Amazon S3 cloud storage service, storage drivers are used to establish the connection between your applications and the cloud storage infrastructure. The storage drivers are an abstract representation of the bucket in the S3 service. You can make use of your existing buckets and folder paths to allow Dataloop to read and write data. Learn more

    Prerequisites: Ensure you complete one of the following AWS integrations:

    1. Log in to the Dataloop platform.
    2. Select Data from the left-side panel.
    3. Select the Storage Drivers tab.
    4. Click Create Storage Driver. The Data Management Resource Creation right-side panel is displayed.
    5. Storage Driver Name: Enter a Name for the storage driver.
    6. Provider: Select AWS from the list.
    7. Resource: Ensure the S3 Bucket is selected by default.
    8. Integration: Select the relevant AWS Integration from the list.
    9. Bucket Name: Enter the S3 bucket name.
    10. Region: Select the region where the S3 bucket is located from the list.
    11. Path (Optional): Enter the folder path, if required.
    12. Storage Class (Optional): Enter the storage class, if required.
    13. Allow Delete Items (Optional): Select the checkbox, if required. This option allows Dataloop to remove items from the storage driver when those items are deleted from Dataloop's dataset.
    Allow Delete Items
    • If Not Enabled, the deleted items in Dataloop will not be restored when re-syncing the storage driver.
    • If Enabled, it does not delete items from the Dataloop dataset when you delete from the storage driver.
    • Items in the bucket are deleted only when the last reference (pointer) to them is removed. As long as at least one pointer exists, the item remains.
    1. Click Create Storage Driver. A confirmation message is displayed.

    Learn how to Create a Dataset Based on an External Cloud Storage.

    Create GCS Storage Drivers

    If you're using Google Cloud Storage (GCS) service, storage drivers are used to establish the connection between your applications and the cloud storage infrastructure. The storage drivers are an abstract representation of the bucket in the GCS service. Learn more

    Prerequisites: Ensure you complete the following GCP integrations:

    1. Log in to the Dataloop platform.
    2. Select Data from the left-side panel.
    3. Select the Storage Drivers tab.
    4. Click Create Storage Driver. The Data Management Resource Creation right-side panel is displayed.
    5. Storage Driver Name: Enter a Name for the storage driver.
    6. Provider: Select GCP from the list.
    7. Resource: Ensure the GCS Bucket is selected by default.
    8. Integration: Select the name of Cross Project or Private Key integration type from the list.
    9. Bucket Name: Enter the GCS bucket name.
    10. Path (Optional): Enter the folder path, if required.
    11. Allow Delete Items (Optional): Select the checkbox, if required. This option allows Dataloop to remove items from the storage driver when those items are deleted from Dataloop's dataset.
    Allow Delete Items
    • If Not Enabled, the deleted items in Dataloop will not be restored when re-syncing the storage driver.
    • If Enabled, it does not delete items from the Dataloop dataset when you delete from the storage driver.
    • Items in the bucket are deleted only when the last reference (pointer) to them is removed. As long as at least one pointer exists, the item remains.
    1. Click Create Storage Driver. A confirmation message is displayed.

    Learn how to Create a Dataset Based on an External Cloud Storage.

    Create Azure Blob Storage Drivers

    If you're using Azure Blob Storage service, storage drivers are used to establish the connection between your applications and the cloud storage infrastructure. The storage drivers are an abstract representation of the container in the Azure Blob. Learn more

    Prerequisites: Ensure you complete Azure Secret Key integration.

    1. Log in to the Dataloop platform.
    2. Select Data from the left-side panel.
    3. Select the Storage Drivers tab.
    4. Click Create Storage Driver. The Data Management Resource Creation right-side panel is displayed.
    5. Storage Driver Name: Enter a Name for the storage driver.
    6. Provider: Select Azure from the list.
    7. Resource: Select the Blob Storage from the list.
    8. Integration: Select the Azure (Client Secret) integration from the list.
    9. Bucket Name: Enter the Azure bucket name.
    10. Path (Optional): Enter the folder path, if required.
    11. Allow Delete Items (Optional): Select the checkbox, if required. This option allows Dataloop to remove items from the storage driver when those items are deleted from Dataloop's dataset.
    Allow Delete Items
    • If Not Enabled, the deleted items in Dataloop will not be restored when re-syncing the storage driver.
    • If Enabled, it does not delete items from the Dataloop dataset when you delete from the storage driver.
    • Items in the bucket are deleted only when the last reference (pointer) to them is removed. As long as at least one pointer exists, the item remains.
    1. Click Create Storage Driver. A confirmation message is displayed.

    Learn how to Create a Dataset Based on an External Cloud Storage.

    Create Azure Data Lake Gen2 Storage Drivers

    Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob Storage.
    Data Lake Storage Gen2 provides file system semantics with Hierarchical directory structure.
    The storage drivers are an abstract representation of the container in the Azure. Learn more

    Prerequisites: Ensure you complete Azure Secret Key integration.

    1. Log in to the Dataloop platform.
    2. Select Data from the left-side panel.
    3. Select the Storage Drivers tab.
    4. Click Create Storage Driver. The Data Management Resource Creation right-side panel is displayed.
    5. Storage Driver Name: Enter a Name for the storage driver.
    6. Provider: Select Azure from the list.
    7. Resource: Select the Data Lake Storage Gen2 from the list.
    8. Integration: Select the Azure (Client Secret) integration from the list.
    9. Bucket Name: Enter the Azure bucket name.
    10. Path (Optional): Enter the folder path, if required.
    11. Allow Delete Items (Optional): Select the checkbox, if required. This option allows Dataloop to remove items from the storage driver when those items are deleted from Dataloop's dataset.
    Allow Delete Items
    • If Not Enabled, the deleted items in Dataloop will not be restored when re-syncing the storage driver.
    • If Enabled, it does not delete items from the Dataloop dataset when you delete from the storage driver.
    • Items in the bucket are deleted only when the last reference (pointer) to them is removed. As long as at least one pointer exists, the item remains.

    Learn how to Create a Dataset Based on an External Cloud Storage.

    1. Click Create Storage Driver. A confirmation message is displayed.
    What's Next