Create and Manage Pipelines

Overview

Creating pipelines using a template (one of your organization templates or one of Dataloop’s templates) from the Marketplace or creating them from scratch offers flexibility and efficiency in developing and deploying data processing and machine learning workflows.

How to Create Pipelines Using a Template from the Marketplace?

The Marketplace is a comprehensive repository that contains a wide variety of pre-defined pipeline templates. These templates are designed to cover common use cases and scenarios across different industries and applications, providing a quick and efficient way to get started with pipeline creation. Here’s how it works:

1. Open the Pipelines page from the left-side menu.
2. Click Create Pipeline and select the Use a Template from the list. It opens the Select Pipeline Template popup window.
3. Select a template that is already installed.
4. Click Create Pipeline. The selected pipeline template will be displayed, and you can configure the available nodes as needed.

How to Create Pipeline from the Scratch?

Creating pipelines from scratch is an approach suited for users with specific requirements that cannot be fully met by existing templates, or for those who prefer to have complete control over every aspect of their pipeline design. Here’s how it works:

1. Open the Pipelines page from the left-side menu.
2. Click Create Pipeline and select the Start from Scratch from the list. It opens the Set Pipeline Name popup window.
3. Enter a name for the new pipeline.
4. Click Create Pipeline. The Pipeline window is displayed, and you can configure the available nodes as needed.

How to Create a Pipeline Using the SDK?

To learn more about creating pipelines in the SDK, click here.

How to Save a Pipeline as a Template?

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Pipeline Name. The pipeline page is displayed.
3. Click on the Actions and select Save as new template from the list.
4. (Optional) Update the Pipeline name.
5. (Optional) Provide a description for the new pipeline template.
6. Check the following options if needed:

• Configure DPK Variables for Solution Mapping: Set up your template by defining $var(x) in the pipeline variables (limited to datasets and models). These variables will be applied during the solution installation to assign Model or Dataset IDs in the newly created pipeline. For example, to map a solution: dpk.variables = {"x": dpkA.models.yolo8"}.
• Retain Entity-Related Fields (Recommended for Internal Projects): Ensure that fields related to entities, such as dataset ID, model ID, recipe ID, etc., are preserved in the template. This is applicable for nodes related to Tasks, Datasets, and Models. It is recommended for templates intended for use within the same project where the entities originate.
• Preserve Service Configuration and Initialization Input Values: Include any changes to service settings or initialization input values in the template. If not selected, these changes will not be saved, and the settings will revert to their default values. Note that this option does not cover secrets or integrations.

How to Reset a Pipeline?

Resetting a pipeline clears all counters, as well as any pending items and cycles, without deleting existing logs or executions.

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Pipeline Name. The pipeline page is displayed.
3. Click on the Actions and select Reset pipeline from the list.

How to Add Secrets to a Pipeline?

Users can configure storage integrations with popular cloud storage services and set up private-registry integrations for more secure and efficient container management. The available types are Key-Value, AWS S3, Google Cloud Storage, Azure Blob, and Google Container Registry.

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Action (three dots) icon.
3. Click Manage Secrets from the list.
4. Select an existing secrets and integrations from the list, or
5. Click the + icon. The New Integration popup is displayed.
   1. Enter a unique name for your secret.
   2. Enter the secret value that you created at the Data Governance > Secrets page.
   3. Click Create.
6. Click Confirm.

For more information about Dataloop secret, see the Secrets.

How to Copy Pipeline ID?

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Action (three dots) icon.
3. Click Copy Pipeline ID from the list. The pipeline ID will be copied to the clipboard.

How to Rename a Pipeline?

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Action (three dots) icon.
3. Click Rename Pipeline from the list.
4. Make the update and click Confirm. A confirmation message is displayed.

How to View Pipeline's Executions?

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Action (three dots) icon.
3. Click View Executions from the list. The CloudOps → Executions tab is displayed.

How to View Pipeline's Logs?

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Action (three dots) icon.
3. Click View Logs from the list. The CloudOps → Logs tab is displayed.

How to View Pipeline's Audit Logs?

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Action (three dots) icon.
3. Click View Audit Logs from the list. The Audit Logs page is displayed.

How to Delete Pipelines?

1. Go to the Pipelines from the left-side navigation panel.
2. Identify the Pipeline and click on the Action (three dots) icon.
3. Click Delete Pipeline from the list. A confirmation message is displayed.
4. Click Yes. A confirmation message is displayed.