Container Registry

Overview

The Dataloop platform supports integration with AWS Elastic Container Registry (ECR) for managing and deploying Docker container images. This section outlines the steps, features, and limitations of ECR integration within the Dataloop platform.

Integration Features

• Private Registry Support: ECR integration allows the platform to securely access private Docker container registries hosted on AWS.

• SDK-Only Integration:

  • Creating an AWS ECR integration can only be done programmatically using the Dataloop SDK.
  • This ensures secure and flexible setup tailored to the user’s environment.

• Service Deployment: Once an integration is established, users can deploy services and pipelines using Docker images stored in their private AWS ECR.

• Integration Visibility: Created integrations are listed in the Integrations section of the platform’s UI, with “AWS” displayed as the provider.

Create AWS Container Registry Integration

To integrate AWS Container Registry (GCR) with the Dataloop platform, follow these steps:

• Create and Configure an AWS Container Registry
• Create an AWS ECR Integration in Dataloop (Using SDK)

Create and Configure an AWS Container Registry

AWS Elastic Container Registry (ECR) is a managed Docker container registry that makes it easy to store, manage, and deploy container images. Below are the steps to create and configure an AWS ECR:

Step 1: Create an ECR Repository

1. Log in to AWS Management Console:
   1. Navigate to the ECR service page.
2. Create a New Repository:
   1. Click on Create repository.
   2. Provide the following details:
      1. Repository name: Enter a unique name for your repository.
      2. Visibility settings: Choose Private for restricted access.
      3. Tag immutability: (Optional) Enable to prevent overwriting of image tags.
      4. Scan on push: (Optional) Enable to scan images for vulnerabilities.
   3. Click Create repository.

Step 2: Authenticate Docker to ECR

1. Install AWS CLI: Ensure the AWS Command Line Interface (CLI) is installed on your machine. You can download it from AWS CLI documentation.

2. Login to ECR:

   1. Use the AWS CLI to authenticate Docker with your ECR:

   aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<your-region>.amazonaws.com
   
   

   2. Replace <your-region> with your AWS region (e.g., us-east-1) and <aws_account_id> with your AWS account ID.

Step 3: Push Docker Images to ECR

1. Tag Your Docker Image: Tag your local Docker image with the ECR repository URI:

docker tag <image-name>:<tag> <aws_account_id>.dkr.ecr.<your-region>.amazonaws.com/<repository-name>:<tag>

2. Push the Image: Push the tagged image to the ECR repository:

docker push <aws_account_id>.dkr.ecr.<your-region>.amazonaws.com/<repository-name>:<tag>

Step 4: Configure Permissions for the ECR Repository

1. Set Repository Policies:

   1. Go to the Repositories section in the AWS ECR console.
   2. Select your repository and click on Permissions.
   3. Add a repository policy to control access for specific users, roles, or services.

2. IAM Role Configuration: Attach necessary permissions to the IAM role or user accessing the repository. For example, use the AmazonEC2ContainerRegistryFullAccess policy.

Create an AWS ECR Integration in Dataloop (Using SDK)

Access the Repository from Dataloop

To use the ECR repository in the Dataloop platform, configure an integration using the Dataloop SDK:

1. Install the Dataloop SDK if not already installed:

pip install dtlpy

2. Use the SDK to programmatically create the integration in Dataloop:

import dtlpy as dl
org_id = 'your_organization_id'  # Replace with your organization ID
org = dl.organizations.get(organization_id=org_id)
integration = org.integrations.create(
    integrations_type=dl.IntegrationType.PRIVATE_REGISTRY,
    name='aws-ecr-integration',
    options={
        "name": "AWS",
        "spec": {
            "accessKeyId": "your_access_key_id",
            "secretAccessKey": "your_secret_access_key",
            "account": "your_aws_account_id",
            "region": "your_aws_region"
        }
    },
    metadata={"provider": 'aws'}
)

3. Verify that the integration is listed in the platform’s UI under Integrations.

Limitations and Restrictions

1. Non-editable Integration

• Container Registry integrations cannot be edited through the UI.
• A tooltip for the Edit Integration action in the UI will state: “Container Registry integration is editable only from Dataloop SDK.”

2. Usage Restrictions: ECR integrations cannot be used to create storage drivers. This restriction is enforced in the UI.

3. Deletion and Its Impact

• Deleting an integration will:
  • Immediately revoke permissions for any connected pipeline or service.
  • Prevent access to cached or pulled images during the next update, pod initialization, or similar actions.
• A confirmation dialog will prompt users to confirm the deletion, displaying the message: “Removing the integration will result in the loss of access to any connected private registry.”