---
title: "Manage Pipelines"
slug: "manage-pipelines"
updated: 2025-07-27T05:59:06Z
published: 2025-07-27T05:59:06Z
canonical: "docs.dataloop.ai/manage-pipelines"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dataloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage Pipelines

This article provides a detailed guide on managing pipelines within the Dataloop platform. It covers essential aspects such as add secrets, reset a pipeline, etc.

---

## Filtering Data Between Nodes

Hover over a connection between nodes and click the + sign to add a filter. Adding a filter means that only data assets (i.e., items or annotations) that comply with the filter condition will be passed onto the next pipeline node.

Filters can be selected from previously selected filters (saved in the [Dataset browser](https://docs.dataloop.ai/docs/schema-based-search#save-a-search-query)) or written directly into the [DQL](https://docs.dataloop.ai/docs/api-dql-query) editor.

For example, the following filter will only pass items whose "size" attribute in the "system" in the "metadata" is less than 1MB:

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/image%28541%29.png)

Important

- Notice that unlike the DQL editor in the Dataset Browser, in the pipeline DQL editor you do not need to include the attributes you filter by within a "filter" property.
- The DQL property JOIN is not supported in the pipeline DQL filter.

Remove a filter by hovering over a connection and clicking the X icon (this will sever the connection between the output/input points, and you will need to reconnect the nodes). Alternatively, you remove the filter by setting the filter to an empty JSON { }.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/pipeFilter.gif)

---

## Reset Pipelines

Resetting a pipeline clears all counters, as well as any pending items and cycles, without deleting existing logs or executions.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/pipelines_more_actions.png)

1. Go to the **Pipelines** from the left-side navigation panel.
2. Identify the Pipeline and click on the Pipeline Name. The pipeline page is displayed.
3. Click on the **⋮** **three dots** and select **Reset pipeline** from the list. A successful message is displayed.

---

## Add Secrets to a Pipeline

Users can configure storage integrations with popular cloud storage services and set up private-registry integrations for more secure and efficient container management. The available types are Key-Value, AWS S3, Google Cloud Storage, Azure Blob, and Google Container Registry.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/pipelines_add_secrets.png)

1. Go to the **Pipelines** from the left-side navigation panel.
2. Identify the Pipeline and click on the **⋮** **three dots** icon.
3. Click **Manage Secrets** from the list.
4. Select an existing secrets and integrations from the list, or
5. Click the **+** icon. The **New Integration** popup is displayed.
  1. Enter a unique name for your secret.
  2. Enter the secret value that you created at the Data Governance > **Secrets** page.
  3. Click **Create**.
6. Click **Confirm**.

For more information about Dataloop secret, see the [Secrets](https://docs.dataloop.ai/docs/secrets-overview).

---

## Node Actions

Once you select a node and clicking on the **Actions** feature in the right-side panel allows you to carry out various tasks that are specific to the function you have selected. The list of actions displayed on each Node are according to the Node's Type.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/pipelines_node_actions.png)

---

### Add New Model Nodes

Dataloop allows the installations for AI/ML models by allowing them to be hosted and executed on:

- Dataloop's **Managed Compute** (internal infrastructure): The Models run on the Dataloop's Compute.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/mrkplc_models_dataloop.png)
- **External Compute Providers** (e.g., OpenAI, Azure, GCP, IBM, NVIDIA) via **API Service** Integration: The Models run on external provider's compute, which requires secret credentials to complete the installation.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/mrkplc_models_servicebyapi1.png)

1. Open the **Pipelines** page.
2. Click on the pipeline from the list. The pipeline page is displayed.
3. In the Node Library on the left-side panel, click on the ➕ icon. The **Add Pipeline Nodes** page of the Marketplace is displayed.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/node_library_plus_icon.png)
4. In the **Models** tab, select the required node from the list. By default, the model nodes are sorted by Installation Status.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/add_node_to_node_library.png)
5. Click **Add Node** from the right-side panel. The **Install Model Application** popup is displayed.
6. Select the required model version from the list and click **Proceed**. A popup is displayed to select the API key to complete the integration process if needed.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/pipelines_select_api_key.png)
7. Select an API Key, Secret or an Integration, as required. If not available:
  1. If there is no secret, click [Add New Secret](https://docs.dataloop.ai/docs/secrets-overview#create-secrets-and-api-keys) and follow the steps.
  2. To set the integration later, click **Set Up Later**.
8. Click **Install Model**. A confirmation message is displayed, and the model node will be added to the Node Library.
  1. To view application details, click on the **View Application** link. It navigates you to the [CloudOps](https://docs.dataloop.ai/docs/cloud-ops-overview) page.
  2. To view the model, go to the [Models](https://docs.dataloop.ai/docs/models-overview) page.

---

### Add Application Nodes

Dataloop allows the installations of applications by allowing them to be hosted and executed on:

- Dataloop's **Managed Compute** (internal infrastructure): The applications run on the Dataloop's Compute.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/mrkplc_models_dataloop.png)
- **External Compute Providers** (e.g., OpenAI, Azure, GCP, IBM, NVIDIA) via **API Service** Integration: The applications run on external provider's compute, which requires secret credentials to complete the installation.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/mrkplc_models_servicebyapi1.png)

1. Open the **Pipelines** page.
2. Click on the pipeline from the list. The pipeline page is displayed.
3. In the Node Library on the left-side panel, click on the ➕ icon. The **Add Pipeline Nodes** page of the Marketplace is displayed.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/node_library_plus_icon.png)
4. Select the **Applications** tab.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/add_app_node_to_node_library.png)
5. Select the required application node from the list. By default, the application nodes are sorted by Installation Status.
6. Click **Add Node** from the right-side panel.
7. Click **Proceed**. A popup is displayed to select the secret key, API key, or integration to complete the integration process, if needed.
  1. If there is no secret key, click [Add New Secret](https://docs.dataloop.ai/docs/secrets-overview#create-secrets-and-api-keys) and follow the steps.
  2. To set the integration later, click **Set Up Later**.
8. Click **Install Application**. A confirmation message is displayed, and the application node will be added to the Node Library.
9. To view application details, click on the **View Application** link. It navigates you to the [CloudOps](https://docs.dataloop.ai/docs/cloud-ops-overview) page.

---

### Install Trainable Models

When you see there are no trainable models available in the **Train** model node, follow the instructions to install them:

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/image-1753595647079.png)

1. Click on the **Install Foundation Model**. The **Select a Foundation Model** page is displayed.

1. Select a trainable model, and click **Install**. The **Install Model Application** popup is displayed.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/install_model_application.png)

1. Select the required model variation from the list and click **Install Model**. A successful message is displayed, click **View Model** to view the newly installed model in the **Model Management** page. Also, the newly installed model will be selected in the node configuration section.

---

### Install Models

When you see there are no models available in the **Evaluate** Model Node, follow the instructions to install them:

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/image-1753595862115.png)

1. Click on the **Install Foundation Model**. The **Select a Foundation Model** page is displayed.
2. Select a model and click **Install**. The **Install Model Application** popup is displayed.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/install_model_application.png)

1. Select the required model variation from the list and click **Install Model**. A successful message is displayed and the newly installed model is available in the **Model Management** page.

Once installed, it will be available for selection for the evaluation.

---

### Set Pipeline Triggers

Dataloop allows you to set triggers on pipeline nodes.

- [Event Based Triggers](https://docs.dataloop.ai/docs/pipeline-trigger#set-event-based-triggers)
- [Cron Based Triggers](https://docs.dataloop.ai/docs/pipeline-trigger#set-cron-based-triggers)
- [Set Manual Triggers](https://docs.dataloop.ai/docs/manage-your-items#run-items-with-pipelines)
- [Set Triggers via SDK](https://docs.dataloop.ai/docs/pipeline-trigger#%F0%9F%A7%A0-programmatic-invocation)

---

### Remove Model Nodes

Important

The node will be removed only from the Node Library but will remain available in the Marketplace under the Models tab for [re-adding](/docs/manage-pipelines#readd-a-pipeline-node-to-the-node-library).

1. Open the **Pipelines** page.
2. Click on the pipeline from the list. The pipeline page is displayed.
3. In the Node Library on the left-side panel, identify the node from the list and hover over.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/pipelines_remove_node.png)

1. Click on the **X** icon. A confirmation message is displayed.
2. Click **Remove**. The removal success confirmation message is displayed.

---

### Re-add Ppipeline Nodes

Follow these steps to restore a removed node to the Node Library:

1. Open the **Pipelines** page.
2. Click on the pipeline from the list. The pipeline page is displayed.
3. In the Node Library on the left-side panel, click on the **+** icon **Add new nodes**. The Models tab of the marketplace is displayed.
4. Identify the model that you have to re-add to the library and click **Add Model** -> **Restore Pipeline Node**. A confirmation message is displayed.

Re-adding pipeline nodes scenarios

- For uninstalled applications: The button will display **Add Node**.
- For installed applications:
  - If a node shortcut exists, the button will display **Add Model**.
  - If there is no node shortcut (removed), the button will display **Add Model** with a dropdown offering:
    - **Add Model**
    - **Restore Pipeline Node**

---

## Undo/Redo Pipeline Editing Steps

While structuring the pipeline and making adjustments, you can use the Undo/Redo to trace back editing steps back and forth as needed, without having to manually change back the pipeline configuration. This includes nodes added or removed, nodes connectivity, and nodes settings.

> [!WARNING]
> Undo/Redo Limitation
> 
> Undo/Redo actions do not track code changes in code nodes. Manual edits are not currently versioned or traceable.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/pipelines_undo_redo(1).png)

---

## Copy Pipeline ID

1. Go to the **Pipelines** from the left-side navigation panel.
2. Identify the Pipeline and click on the **⋮** **three dots** icon.
3. Click **Copy Pipeline ID** from the list. The pipeline ID will be copied to the clipboard.

---

## Rename Pipelines

1. Go to the **Pipelines** from the left-side navigation panel.
2. Identify the Pipeline and click on the **⋮** **three dots** icon.
3. Click **Rename Pipeline** from the list.
4. Make the update and click **Confirm**. A confirmation message is displayed.

---

## Update Pipeline Settings: On Pause Action

Each pipeline has its own independent settings. Configuring the desired behavior for one pipeline will not impact any of your other available pipelines.

To configure pipeline behavior when paused:

1. Go to the **Pipelines** from the left-side navigation panel.
2. Open the Pipeline from the list.
3. Click on the **⋮** **three dots** and select the **Settings** from the list. The Settings popup is displayed.

![](https://cdn.document360.io/53f32fe9-1937-4652-8526-90c1bc78d3f8/Images/Documentation/pipelines_more_actions.png)

1. Choose the desired action from the dropdown list:
  1. **Always ask me what to do**:
  2. **Enable: Keep pipeline triggers activated**:
  3. **Disable: Deactivate pipeline triggers**:
2. Click **Save Changes** to the settings to apply them permanently.

## View Executions Data

1. Go to the **Pipelines** from the left-side navigation panel.
2. Identify the Pipeline and click on the **⋮** **three dots** icon.
3. Click **View Executions** from the list. The CloudOps → Executions tab is displayed.

---

## View Logs Data

1. Go to the **Pipelines** from the left-side navigation panel.
2. Identify the Pipeline and click on the **⋮** **three dots** icon.
3. Click **View Logs** from the list. The CloudOps → Logs tab is displayed.

---

## View Audit Logs

1. Go to the **Pipelines** from the left-side navigation panel.
2. Identify the Pipeline and click on the **⋮** **three dots** icon.
3. Click View Audit Logs from the list. The Audit Logs page is displayed.

---

## Delete Pipelines

Warning

All the applications created by the Pipeline will be deleted.

1. Go to the **Pipelines** from the left-side navigation panel.
2. Identify the Pipeline and click on the **⋮** **three dots** icon.
3. Click **Delete Pipeline** from the list. A confirmation message is displayed.
4. Click **Yes**. A confirmation message is displayed.