Pipeline Nodes - Overview

Overview

Dataloop pipelines provides a user-friendly interface for building, managing, and monitoring end-to-end machine learning workflows. Dataloop's pipeline can be made up of different nodes, each with a different role in the pipeline, such as storing data, executing functions, training models, or sending data to annotation or QA tasks.

Node Library

The Node Library in Dataloop's Pipeline offers a comprehensive list of available nodes, including both default and installed options. You can easily search for specific nodes using the search bar to help build your pipeline. Additionally, the Node Library allows you to add new model nodes and application nodes, from the Marketplace by clicking on the Plus (+) icon.

By default, the following node categories are available on the Dataloop's Node Library:

• Data Nodes
• Workflow Nodes
• Automation Nodes
• Model Nodes
• Image Utils Nodes

• The Search nodes field allows you to search for nodes based on their category and name.
• Each node elements will serve a unique support to the pipeline to accomplish its task. Once you place a node on the canvas and clicking it will bring up the node’s Config in the right-side panel.

Pipeline Node's Actions

Once you select a node and clicking on the Actions feature in the right-side panel allows you to carry out various tasks that are specific to the function you have selected. The following table lists the actions available on pipelines nodes.

How to Add a New Model Node to the Node Library in Pipelines?

1. Open the Pipelines page.
2. Click on the pipeline from the list. The pipeline page is displayed.
3. In the Node Library on the left-side panel, click on the Plus icon. The Add Pipeline Nodes page of the Marketplace is displayed.
4. In the Models tab, select the required node from the list. By default, the model nodes are sorted by Installation Status.
5. Click Add Node from the right-side panel. A confirmation message is displayed, and the model node will be added to the Node Library.
   1. To view application details, click on the View Application link. It navigates you to the CloudOps page.
   2. To view the model, go to the Models page.

How to Add an Application Node to the Node Library in Pipelines?

1. Open the Pipelines page.
2. Click on the pipeline from the list. The pipeline page is displayed.
3. In the Node Library on the left-side panel, click on the Plus icon. The Add Pipeline Nodes page of the Marketplace is displayed.
4. Select the Applications tab.
5. Select the required application node from the list. By default, the application nodes are sorted by Installation Status.
6. Click Add Node from the right-side panel. A confirmation message is displayed, and the application node will be added to the Node Library.
   1. To view application details, click on the View Application link. It navigates you to the CloudOps page.
   2. To view the model, go to the Models page.

How to Remove a Node from the Node Library in Pipelines?

1. Open the Pipelines page.
2. Click on the pipeline from the list. The pipeline page is displayed.
3. In the Node Library on the left-side panel, identify the node from the list and hover over.
4. Click on the X icon. A confirmation message is displayed.
5. Click Remove. The removal success confirmation message is displayed.

How to Open the Pipeline Dataset Node's Dataset Browser?

This option allows you to browse the Dataset that you have selected in the Dataset Node.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the Dataset node.
3. Click on the Action from the right-side panel.
4. Click Open Dataset from the list. The Dataset Browser page is displayed.

How to Copy Service ID from a Pipeline Node?

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Copy Service ID from the list. You can copy the service ID only from the Dataset, Update Variable, FaaS Node, and Labeling nodes.

How to Copy Node ID of a Pipeline Node?

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Copy Node ID from the list. You can copy the node ID only from the Dataset, Update Variable, Data Split, FaaS Node, Code, Labeling, Train Model, Predict Node, Create New Model, Evaluate Model, and Model Data Split nodes.

How to View Pipeline Node's Service Analytics?

It allows you to view the Monitoring tab of the Application Service.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click View Analytics from the list. The Monitoring tab of the Application Service page is displayed.

How to View Pipeline Node's Service Execution Status?

It allows you to view the Executions page of the Application Service.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click View Executions from the list. The Executions tab of the CloudOps Service page is displayed.

How to Install a Trainable Model from the Marketplace for the Train Model Node?

When you see there are no trainable models available in the Train Model Node, follow the instructions to install them:

1. Click on the Install Foundation Model. The Select a Foundation Model page is displayed.
2. Select a trainable model and click Install. The Install Model Application popup is displayed.
3. Select the required model variation from the list and click Install Model. A successful message is displayed and the newly installed model is available in the Model Management page.

Once installed, it will be available for selection for the training.

How to Install a Model from the Marketplace for Evaluate Model Node?

When you see there are no models available in the Evaluate Model Node, follow the instructions to install them:

1. Click on the Install Foundation Model. The Select a Foundation Model page is displayed.
2. Select a model and click Install. The Install Model Application popup is displayed.
3. Select the required model variation from the list and click Install Model. A successful message is displayed and the newly installed model is available in the Model Management page.

Once installed, it will be available for selection for the evaluation.

How to View Service Logs of a Pipeline Node?

It allows you to view the Logs page of the Application Service.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click View Logs from the list. The Logs tab of the CloudOps page is displayed.

How to View Task Details of a Labeling Task Node?

It allows you to open and view the Tasks page, where you view the list view of the selected task.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the Labeling node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Open in Task Management from the list. The Tasks tab of the Labeling page is displayed.

How to Browse Task Details of a Labeling Task Node?

It allows you to open and view the task browser page, where you see the items available for the selected task.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the Labeling node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Browse Task from the list. The task browser page is displayed.

How to Open Service Page of a Pipeline Node?

It allows you to open the service page of the function where you can see Monitoring, Executions, Logs, Triggers, and Instances information of the service. It will be available once you set a model.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Open Service Page from the list. The Monitoring tab of the service page is displayed.

How to Download Codebase File of the FaaS Node?

It allows you to download the Codebase file. To learn how to download, see the Codebase.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the FaaS node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Download Codebase File from the list. The codebase package file will be downloaded, if you have the permission.

How to Open Model's Page of the Model Nodes?

It allows you to open the model that has been referenced. It will be unavailable if there is no model reference.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Open Model from the list. The Model's Details page is displayed.

How to Open GitHub Code for the Selected Model?

It allows you to open the GitHub Code reference for the selected model. It will be unavailable if there is no fixed model selected.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Open GitHub Code from the list. The GitHub page of the selected model is displayed.

How to View Service Settings of the Model?

It allows you to view the Service Settings based on the model you have selected while running the pipeline.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click View Service Settings from the list. The View Service Setting popup window of the service is displayed.

How to Edit Service Settings of the Model?

It allows you to make changes to the Service Configuration based on the model you have selected. This option is not available for Train and Validation.

1. Open the Pipelines page and click on the pipeline from the list.
2. Select the node from the pipeline.
3. Click on the Action from the right-side panel.
4. Click Edit Service Settings from the list. The Edit Service Setting popup window of the service is displayed.
5. Hover over the section and make changes, if required.
6. Click Save Changes.