Google Artifacts Registry

Overview

The Google Artifact Registry (GAR) offers a more robust and scalable solution for Dataloop integrations by supporting multiple artifact formats, including Container images. This allows users to manage not only containerized workloads but also custom plugins, scripts, and machine learning models more securely and efficiently.

• GAR's fine-grained IAM controls and VPC Service Controls enhance security, ensuring that data workflows in Dataloop remain compliant and secure.
• As Google’s recommended and actively developed service, GAR provides a future-proof foundation for managing complex data and AI pipelines on the Dataloop platform. Learn more.

To integrate Google Artifacts Registry (GAR) with the Dataloop platform, follow these steps:

1. Create and Configure a GCP GAR
2. Integrate GAR with Dataloop
3. Integrate GAR with SDK

Create and Configure GAR

1. Enable Artifact Registry API on GCP

1. Go to the GCP Console
2. Search and select the APIs & Services > Library.
3. Search for Artifact Registry API.

5. Click Enable.

Use this command to run in your terminal:

gcloud services enable artifactregistry.googleapis.com

This enables the GAR API for your GCP project.

2. Configure IAM Permissions

Grant appropriate permissions to the user or service account that Dataloop will use:

1. Go to IAM & Admin -> IAM.
2. Identify the user and click on the Edit principal ✏️ pencil icon.
3. Role:
   1. Select the Artifact Registry -> Artifact Registry Administrator from the list.
   2. Click Add another role, and select the Storage Admin (optional, for managing storage).
4. Click Save.

Use this command to assign the role:

gcloud projects add-iam-policy-binding [PROJECT-ID] \
  --member="user:[USER-EMAIL]" \
  --role="roles/artifactregistry.admin"

3. Authenticate Docker with GAR

Before pushing or pulling images, configure Docker to authenticate with GAR:

gcloud auth configure-docker [REGION]-docker.pkg.dev

Replace [REGION] with your repository’s region (e.g., us-central1).

4. Tag and Push Container Images to GAR

Tag your Docker image:

docker tag [IMAGE_NAME] [REGION]-docker.pkg.dev/[PROJECT_ID]/[REPOSITORY]/[IMAGE_NAME]:[TAG]

Push it to Artifact Registry:

docker push [REGION]-docker.pkg.dev/[PROJECT_ID]/[REPOSITORY]/[IMAGE_NAME]:[TAG]

Integrate GAR with Dataloop

1. Generate a Service Account Key

1. Create a service account:
   1. Navigate to IAM & Admin > Service Accounts.
   2. Click Create Service Account.
   3. Provide a Name (e.g., dataloop-access) and click Create.
   4. Assign the following roles:
      1. Storage Object Viewer.
      2. Container Registry Viewer.
   5. Click Done.
2. Generate a key for the service account:
   1. Find your service account in the list, and click to open.
   2. Select the Keys tab.
   3. Click Add Key > Create New Key.
   4. Select JSON and click Create.
   5. Save the JSON key file securely.

3. Upload the Service Account Key to Dataloop

1. Log in to your Dataloop Platform.
2. Navigate to the Integrations page.
3. Click Create Integration -> Create Registry Integration.
   1. Integration Name: Provide a name for the integration.
   2. Provider: Select GCP from the list.
      1. Service: Select Google Artifacts Registry from the list.
      2. Click Import JSON file.
      3. Location: Enter the full location of your Google Artifact Registry (GAR), including the region, (e.g., us-central1-docker.pkg.dev).
4. Click Create Integration.

4. Test the Configuration

• In the Dataloop platform, try pulling or referencing a Container image hosted in your GCR.
• Ensure the integration is working without any errors.