Elastic Container Registry

Overview

The Dataloop platform supports integration with AWS Elastic Container Registry (ECR) for managing and deploying Docker container images. This section outlines the steps, features, and limitations of ECR integration within the Dataloop platform.

Integration Features

• Private Registry Support: ECR integration allows the platform to securely access private Docker container registries hosted on AWS.

• Integration:

  • Creating an AWS ECR integration can be done programmatically using the Dataloop UI or SDK.
  • This ensures secure and flexible setup tailored to the user’s environment.

• Service Deployment: Once an integration is established, users can deploy services and pipelines using Docker images stored in their private AWS ECR.

• Integration Visibility: Created integrations are listed in the Integrations section of the platform’s UI, with “AWS” displayed as the provider.

Integration Process

The Integration process involves the following steps:

1. Prerequisites
2. Integrate ECR Using UI
3. Integrate ECR Using SDK

Prerequisites

Ensure you have the following details ready to start the integration process:

• Access Key ID: The Access Key ID for AWS ECR is a unique identifier used with the Secret Access Key to securely authenticate and authorize access to AWS resources, for example, (e.g., AKIAIOSFODNN7EXAMPLE). The Access Key ID from your AWS IAM user. Ensure the IAM user has scoped permissions for ECR. If you do not have the access key ready, follow the steps:
  • Create an S3 Bucket
  • Create an IAM Policy
  • Create an IAM User
  • Create an Access Key for the IAM User
• Secret Access Key: The Secret Access Key for AWS ECR is a confidential credential used with the Access Key ID to securely authenticate and authorize programmatic access to AWS services. If you do not have the access key ready, follow the steps:
  • Create an S3 Bucket
  • Create an IAM Policy
  • Create an IAM User
  • Create an Access Key for the IAM User
• Account ID: The Account ID in AWS is a unique 12-digit number that identifies your AWS account across all AWS services. To view it, follow the steps:
  • Sign in to the AWS Management Console.
  • Click on your account name or profile icon in the top-right corner.
  • Your 12-digit Account ID will be displayed.
• Region: An AWS Region is a geographically isolated area where Amazon Web Services (AWS) hosts its data centers to provide cloud services with low latency and compliance with local regulations. For example, us-east-1 → N. Virginia, USA. To view it, follow the steps:
  • Sign in to the AWS Management Console.
  • In the top-right corner, next to your account name, you'll see a dropdown with the current Region (e.g., N. Virginia, us-east-1).
  • Click the dropdown to view and select other available Regions.

Integrate ECR Using UI

AWS Elastic Container Registry (ECR) is a managed Docker container registry that makes it easy to store, manage, and deploy container images. Below are the steps to create and configure an AWS ECR:

1: Set Up AWS ECR

1. Log in to AWS Management Console.
2. In the search bar, type Elastic Container Registry (ECR).
3. Select ECR from the services list.

4. Create Repository: Click Create.
5. Repository name: Enter a repository name (e.g., dataloop-integration).
6. Leave other settings as default or customize as needed.
7. Click Create.
8. Set IAM Permissions:
   • Attach necessary policies to the IAM user/role to access ECR.
     1. Go to the Repositories section in the AWS ECR console.
     2. Select your repository and click on Permissions.
     3. Add a repository policy to control access for specific users, roles, or services.
   • Minimum permissions include:
     • ecr:GetAuthorizationToken
     • ecr:BatchCheckLayerAvailability
     • ecr:GetDownloadUrlForLayer
     • ecr:PutImage
     • ecr:InitiateLayerUpload
   • IAM Role Configuration: Attach necessary permissions to the IAM role or user accessing the repository. For example, use the AmazonEC2ContainerRegistryFullAccess policy.

2: Retrieve AWS Credentials

1. Access Keys:
   1. Go to IAM → Users.
   2. Select your user and click Security Credentials.
   3. Create and download an Access Key ID and Secret Access Key.
2. Account ID: Click your account name → Account to find your 12-digit Account ID.
3. Region: Note the Region where the ECR repository was created (e.g., us-east-1).

Refer to the Prerequisites section for the details.

3: Integrate AWS ECR

1. Log in to Dataloop Platform.
2. Select the Integrations from the left-side menu.
3. Click Create Integration -> Create Registry Integration.
   1. Integration Name: Enter a name for the ECR Integration.
   2. Provider: Select AWS from the list. Refer to the [Prerequisites] for the details:
      1. Access Key ID: Enter the Access Key ID. .
      2. Secret Access Key: Enter the Access Key ID.
      3. Account: Enter the AWS account ID.
      4. Region: Enter the AWS Account Region.
4. Click Create Integration.

Integrate ECR Using SDK

Access the Repository from Dataloop

To use the ECR repository in the Dataloop platform, configure an integration using the Dataloop SDK:

1. Install the Dataloop SDK if not already installed:

pip install dtlpy

2. Use the SDK to programmatically create the integration in Dataloop:

import dtlpy as dl
org_id = 'your_organization_id'  # Replace with your organization ID
org = dl.organizations.get(organization_id=org_id)
integration = org.integrations.create(
    integrations_type=dl.IntegrationType.PRIVATE_REGISTRY,
    name='aws-ecr-integration',
    options={
        "name": "AWS",
        "spec": {
            "accessKeyId": "your_access_key_id",
            "secretAccessKey": "your_secret_access_key",
            "account": "your_aws_account_id",
            "region": "your_aws_region"
        }
    },
    metadata={"provider": 'aws'}
)

For the accessKeyId, secretAccessKey, account, and region, refer to the Prerequisites section for the details.

3. Verify that the integration is listed in the platform’s UI under Integrations.

Limitations and Restrictions

1. Non-editable Integration

• Container Registry integrations cannot be edited through the UI.
• A tooltip for the Edit Integration action in the UI will state: “Container Registry integration is editable only from Dataloop SDK.”

2. Usage Restrictions: ECR integrations cannot be used to create storage drivers. This restriction is enforced in the UI.

3. Deletion and Its Impact

• Deleting an integration will:
  • Immediately revoke permissions for any connected pipeline or service.
  • Prevent access to cached or pulled images during the next update, pod initialization, or similar actions.
• A confirmation dialog will prompt users to confirm the deletion, displaying the message: “Removing the integration will result in the loss of access to any connected private registry.”