Contents x
    Cross Project Integration
    • 14 May 2025
    • Print
    • Dark
      Light
    • PDF
    Contents

    Cross Project Integration

    • Updated On 14 May 2025
    • Print
    • Dark
      Light
    • PDF
    Article summary

    Overview

    Dataloop allows users with a GCP project to integrate their GCS bucket with the Dataloop platform and create datasets. The GCP Cross Project integration stands as the best choice for GCP due to its significant advantages.

    Important
    • We recommend selecting Cross-Project as the integration type. It is GCP’s most secure option and provides enhanced protection for your data.
    • Organizations can create up to 3 Cross Projects integrations. To increase the number of Cross Projects integrations for your organization, please contact our Customer support team.


    The GCP cross-project integration process involves:

    1. Create a Cloud Storage Bucket

    1. Log in to Google Cloud Console.
    2. From the left portal menu, go to Cloud Storage > Buckets.
    3. Click Create bucket.
    4. Enter a Name for the storage bucket, and click Continue.
    5. In the Choose where to store your data, select a location for the bucket, and click Continue.
    6. Click Create bucket.
    Note: For all other optional settings, use the default values.

    2. Create an IAM Role

    1. Log in to the Google Cloud Console.
    2. From the left portal menu, select the IAM & admin > Roles.
    3. Click +Create Role.
    4. Enter a role Title.
    5. (Optional) Enter a Description.
    6. (Optional) Enter a role ID. By default, the role ID is generated.
    7. Click +Add Permissions, then search for and add the following permissions:
      1. storage.objects.create
      2. storage.objects.delete
      3. storage.objects.get
      4. storage.objects.list
      5. storage.buckets.get
      6. storage.buckets.getIamPolicy
    Note:
    • The storage.objects.delete permission allows the Dataloop platform to delete dataset items. For more information, see the Downstream section.
    • The storage.buckets.getIamPolicy permission allows the Dataloop platform to validate that the integration was created successfully.
    1. Click Create.
    Tip
    • See Create and manage custom roles for more information on creating an IAM role in GCP.
    • To display roles information, you must select a project in the Google Cloud Console. If not available, create a project.

    3. Create the GCP Cross Project Integration

    Maximum GCP Cross Project Integrations

    You can create only maximum of 3 GCP Cross Project Integrations.

    1. Log in to the Dataloop platform.
    2. From the left-side panel, select Integrations.
    3. Click Create Integration -> Create Storage Integration. A pop-up window is displayed on the right-side.
    4. Integration Name: Enter a Name for the integration.
    1. Provider: Choose GCP from the list.
    2. Integration Type: Select the Cross Project from the list.
    3. Click Get New Service Account. Copy and add the Service Account ID as a principal in your project, and assign it with the relevant role. Assign a role.
    1. Or, click Existing Service Account in case you already created one, you can choose from the list of created Service Accounts that have not been assigned to an integration.
    1. Service Account ID: Copy the Service Accounts ID (Email).
    2. Resource Name: Follow this steps and enter the resource name.
    3. Click Create Integration.

    4. Grant Permissions to the Dataloop Service Account

    To grant Dataloop service account permissions to access the Cloud Storage Bucket:

    1. Log in to Google Cloud Console.
    2. From the left portal menu, go to Cloud Storage > Buckets.
    3. Select the Storage bucket for which you want to add permissions, and then click on it.
    4. Select the Permissions tab.
    5. Click Grant access. The Add Principals dialog box appears.
    6. Under the Add principals, add the Service account ID provided by the Dataloop platform.
    7. Under the Assign roles, choose custom and choose the role you recently created.
    8. Click Save. A confirmation message is displayed.
    Tip: See Use IAM with buckets for more information on adding permissions to a storage bucket in GCP.

    5. Complete the GCP Cross Project Integration

    1. Log in to the Dataloop platform.
    2. Under the Resource name, provide the name of the bucket you wish to integrate its data with the Dataloop.
    3. Click Create Integration. A confirmation message is displayed.
    ⚠️ Troubleshooting Permission Errors

    If you receive a permission denied error, even though the user's email is listed in the bucket’s IAM roles, it could be due to a stale or improperly propagated IAM configuration.

    ✅ Recommended Fix:

    • Remove the user’s email from the bucket’s IAM permission list.
    • Re-add the same email with the appropriate role (e.g., Storage Object Viewer, Editor).

    This action refreshes the IAM policy propagation, ensuring that the access rights take effect properly.

    6. Create GCS Storage Drivers

    For more information, see the Create a GCS Storage Drivers topic.

    What's Next