Contents x
    Automation Nodes
    • 14 May 2025
    • Print
    • Dark
      Light
    • PDF
    Contents

    Automation Nodes

    • Updated On 14 May 2025
    • Print
    • Dark
      Light
    • PDF
    Article summary

    Overview

    The Automation elements of the Dataloop pipeline consist of Application (FaaS) and Code nodes, which enable efficient, scalable automation within machine learning workflows.

    FaaS Node

    The FaaS Node allows you to invoke functions defined in the Marketplace -> Applications. It's ideal for handling larger code scopes or reusable code across multiple pipelines, centralizing maintenance, version control, and providing control over computing settings and resource allocation. Learn more the Application.

    When you select a FaaS node, its details, such as Configuration, Executions, Logs, Instances, and available actions are shown on the right-side panel.

    🔧 Config

    • Node Name: By default, FaaS Node is displayed as name. You can change it accordingly.
    • Function: Select a function from the list. The FaaS Node’s Output and Input channels are determined by the function chosen for the node.
    • Node Input: It is generated automatically according to the selected FaaS.
    • Node Output: It is generated automatically according to the selected FaaS.
    • Trigger (Optional): An Event/Cron Triggers can be set on this node, enabling you to initiate the pipeline run from this specific point.

    📌 Executions

    When a node is selected, the Executions tab provides details about the execution of each item processed through that node. Choose a specific execution from the list to access its detailed information.
    • Use the filter to display executions based on its status.
    • Click Rerun Executions to rerun the selected executions. Before you confirm the rerun, you have two types of executions cycles:
      • Resume Cycle: Resume the cycles of the selected executions starting from the current node onwards.
      • Restart Cycle: Restart the cycles of the selected executions starting from the beginning.
    • View Execution details such as Information, Input, and Output details.

    📋 Logs

    The Logs tab provides a granular view of what happens behind the scenes when a node executes. It showcases all log entries, including function and service level logs, application installation processes (from initialization to completion), generated during executions of the corresponding node within the selected time-frame. Dataloop stores logs for a maximum of 14 days. Click on the log to view log details in the bottom.

    Utilize the following options to navigate through the logs:

    • Search: Enter free text to search through the logs.
    • Log Level Filter: Filter logs by selecting a level (critical, error, warning, info, debug) to find logs at that level and above.
    • Show Service Logs: Show/hide service-level logs, aiding in the debugging and resolution of service-level errors and issues.

    ⚙️ Instances

    The Instances tab is active specifically for Custom nodes such as FaaS and Code nodes, providing information about the service instances (machine replicas) connected to the pipeline. This tab shows: - Running instances (replicas of the service/machine) - Resource usage (CPU, memory) - Health status and uptime - Logs for each instance, if needed

    The FaaS node allows you to perform the following actions:

    Open the Service Page

    Open the service page of a pipeline node where you can see Monitoring, Executions, Logs, Triggers, and Instances information of the service. It will be available once you set a model.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click Open Service Page from the list. The Monitoring tab of the service page is displayed.

    View Service Settings

    It allows you to view the Service Settings based on the model you have selected while running the pipeline.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click View Service Settings from the list. The View Service Setting popup window of the service is displayed.

    Download the Codebase File

    It allows you to download the Codebase file for a FaaS node. To learn how to download, see the Codebase.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the FaaS node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click Download Codebase File from the list. The codebase package file will be downloaded, if you have the permission.

    📋 Copy the Service ID

    Copy service ID from a pipeline node.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click Copy Service ID from the list. You can copy the service ID only from the Dataset, Update Variable, FaaS Node, and Labeling nodes.

    📋 Copy the Node ID

    Copy node ID of a pipeline node

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click Copy Node ID from the list. You can copy the node ID only from the Dataset, Update Variable, Data Split, FaaS Node, Code, Labeling, Train Model, Predict Node, Create New Model, Evaluate Model, and Model Data Split nodes.

    View Log Details

    It allows you to view the Logs page of the Application Service from a pipeline node.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click View Logs from the list. The Logs tab of the CloudOps page is displayed.

    View Execution Details

    It allows you to view the Executions page of the Application Service.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click View Executions from the list. The Executions tab of the CloudOps Service page is displayed.

    📊 View Analytics Details

    It allows you to view the Monitoring tab of the Application Service.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click View Analytics from the list. The Monitoring tab of the Application Service page is displayed.

    Code Node

    The Code Node allows you to execute custom functions directly within a pipeline. It is ideal for handling lightweight tasks such as uploading annotations in JSON format, cropping annotations from items, generating bounding boxes, and more.

    When the pipeline is triggered, a dedicated service is deployed to run the code. This service may require monitoring, maintenance, and scaling depending on the workload and operational needs.

    When you click on a Code Node, its details, such as Configuration, Executions, Logs, Instances, and available Actions are shown on the right-side panel. The Code Node details are presented in four tabs as follows:

    🔧 Config

    • Node Name: By default, code is displayed as name. You can change accordingly.
    • Application Name: Provide a name for the application that will be created to execute this code node. This name will be used as both the package and service names.
    • Function Name: Provide a name for the function that will run your code. Make sure to use the same name as defined in the code, following the Python convention for function names. By default, run is displayed as name.
    • Source Code Type: Select a source code type from the following list.
      • Code Editor (by default): The Code Editor allows you to enter the code for the code node.
      • Upload ZIP: It allows you to upload your code in a .zip file.
      • Connect GitHub: It allows you to enter the URL of your code's Git repository and Git Tag.
    • Requirements (Optional): It allows you to upload the requirements file in .txt format.
      • This file is necessary when your code relies on additional Python libraries for execution, and it helps specify which Python libraries need to be installed.
      • Alternatively, a docker image can be specified in docker image URL while editing a service. Also, see the Custom Docker Image article for more information about the docker registry.
    • Node Inputs: Input channels are set to be of type: item by default. A Code node can have multiple input and output channels. You can perform the following:
      • Add new inputs by clicking on the plus icon. It allows you to add a new name, new input type, and description.
      • Click on the Pencil icon to update Name and Description. You cannot edit the type.
      • Click Set Parameter to set input parameter for the Code node.
      • For more information, see the Node Inputs article.
    • Node Outputs: Output channels are set to be of type: item by default. You can perform the following:
      • Add new outputs by clicking on the plus icon. It allows you to add a new name, new output type, description, and action.
      • Click on the Pencil icon to update Name, Description, and Actions. You cannot edit the type.

    When adding output channels, you may need to choose a different type for your output. For example, if a code node connects to another code node that expects a boolean as an input, you will need to define the output of the first code node to be of type boolean.

    Recommended uploading your code as a package to our Application, and using it as a FaaS node rather than code. This will enable you to better manage versions, reuse the service, and more.

    📌 Executions

    When a node is selected, the Executions tab provides details about the execution of each item processed through that node. Choose a specific execution from the list to access its detailed information.
    • Use the filter to display executions based on its status.
    • Click Rerun Executions to rerun the selected executions. Before you confirm the rerun, you have two types of executions cycles:
      • Resume Cycle: Resume the cycles of the selected executions starting from the current node onwards.
      • Restart Cycle: Restart the cycles of the selected executions starting from the beginning.
    • View Execution details such as Information, Input, and Output details.

    📋 Logs

    The Logs tab provides a granular view of what happens behind the scenes when a node executes. It showcases all log entries, including function and service level logs, application installation processes (from initialization to completion), generated during executions of the corresponding node within the selected time-frame. Dataloop stores logs for a maximum of 14 days. Click on the log to view log details in the bottom.

    Utilize the following options to navigate through the logs:

    • Search: Enter free text to search through the logs.
    • Log Level Filter: Filter logs by selecting a level (critical, error, warning, info, debug) to find logs at that level and above.
    • Show Service Logs: Show/hide service-level logs, aiding in the debugging and resolution of service-level errors and issues.

    ⚙️ Instances

    The Instances tab is active specifically for Custom nodes such as FaaS and Code nodes, providing information about the service instances (machine replicas) connected to the pipeline. This tab shows: - Running instances (replicas of the service/machine) - Resource usage (CPU, memory) - Health status and uptime - Logs for each instance, if needed

    The Code node allows you to perform the following actions:

    ✏️ Edit Service Settings

    It allows you to make changes to the service settings like Integrations, Service Configuration, and Execution Configuration based on the model you have selected. This option is not available for Train and Validation.

    Edit Service Configuration while running a pipeline

    You can now edit the service configuration of a node even while running a pipeline, providing greater flexibility.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click Edit Service Settings from the list. The Edit Service Setting popup window of the service is displayed.


    Integrations

    The secret values are displayed, if the secret is defined in the dpk/dpkConfig. Click on the Edit Secret to search and select a secret from the list.

    1. To create a new secret, click Create New -> Create Secret.
    2. To create a new Integration, click Create New -> Create Integration.
    3. Click Save Changes, if no additional modifications are needed.


    Service Configuration

    The service configuration section allows making changes to the following parameters, to change, click on the Edit Configuration.

    1. Machine Types: The machine type on which the service will run (refers to CPU, GPU, RAM).
      1. Change the machine type from the list.
      2. Click on the Machine Settings to enable Preemptible Machine, the reclaimable compute instances.
    2. Docker Image: A container image encapsulating essential dependencies and code for service execution. By default, the docker image based on the SDK version is displayed. You can modify the URL as required.
    3. Concurrency: Number of concurrent executions per instance. By default, the value 10 is displayed.
    4. Scaling Type: Choose the scaling type based on your workload.
      1. Autoscale: Recommended by default. Autoscaling dynamically adjusts the number of service instances based on the incoming execution load, suitable for changing workloads. When autoscaler reduces instances, it waits for execution timeout before termination.
        1. Min Instances: By default, 0. If you set the min instances to 0, the service will not run until it has something in the queue, it will be suspended.
        2. Max Instances: By Default, 2.
        3. Queue Size: By default, 10.
      2. Fixed size: Use fixed scaling when you need a predetermined number of service instances, suitable for consistent workloads. Compute hours counts also when there is no workload. By default, 1 instance is selected.
    5. Click Save Changes, if no additional modifications are needed.


    Execution Configuration

    The execution configuration section allows making changes to the following parameters, to change, click on the Edit Configuration.

    1. SDK Version: The SDK version is displayed. Click to select a new version from the list.
    2. Execution Timeout: The maximum execution duration before service restarts (in seconds). By default, 3600 is displayed.
    3. Drain Time: Delays service restart to complete ongoing executions (in seconds). By default, 600 is displayed.
    4. On Reset Action: Action taken when a service is reset or restarted (e.g. timeout). Select one of the following when you reset the execution.
      1. Fail Execution
      2. Rerun Execution
    5. Max Attempts: Configures to automatically retry failed executions up to x attempts. Maximum number of execution attempts allowed. By default, 3 is displayed.
    6. Bot: A dummy user used for running the service ('developer' role). A default Bot is displayed. Select a new one from the list, if needed.
    7. Rerun executions as process: Defines if each execution will start a new thread or a process. Enable to make the execution as a process.
    8. Click Save Changes.

    📋 Copy the Node ID

    Copy node ID of a pipeline node

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click Copy Node ID from the list. You can copy the node ID only from the Dataset, Update Variable, Data Split, FaaS Node, Code, Labeling, Train Model, Predict Node, Create New Model, Evaluate Model, and Model Data Split nodes.

    View Execution Details

    It allows you to view the Executions page of the Application Service.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click View Executions from the list. The Executions tab of the CloudOps Service page is displayed.

    View Log Details

    It allows you to view the Logs page of the Application Service from a pipeline node.

    1. Open the Pipelines page and click on the pipeline from the list.
    2. Select the node from the pipeline.
    3. Click on the Action from the right-side panel.
    1. Click View Logs from the list. The Logs tab of the CloudOps page is displayed.

    What's Next