Pipeline triggers

Overview

This document provides an overview of pipeline triggers, their execution modes, and configuration options within the Dataloop system. 🚀

The Dataloop Pipeline allows you to configure one or more triggers, enabling automatic execution of the pipeline flow in response to specific events (event trigger) or at predefined intervals (cron trigger). When a trigger is linked to a pipeline, the pipeline runs whenever the trigger is activated.

• The triggered resource object is passed as an input parameter to the pipeline.
• A trigger can be assigned to multiple pipeline nodes, each executing independently when the trigger fires.

Event-based triggers

It allows you to invoke a function in response to specific system events, using the triggering resource as the function's input. Event triggers are linked to a combination of resources and actions:

• Resources: Entities within the Dataloop system (e.g., Item, Dataset, Annotation, Task).
• Actions: Events that occur on these resources (e.g., Create, Update, Delete, Clone).

A DQL filter can be applied to filter specific resources before triggering the pipeline.

Resources

The following table lists available resources and their corresponding actions. Event triggers support only the following types of function inputs: Item, Dataset, Model, Task, Assignment, and Annotation:

• Created: Triggers the pipeline when a new item is added (uploaded or synced to a dataset).
• Updated: Triggers the pipeline when an item is modified. See ** for more details regarding itemStatus.Updated.
• Deleted: Triggers the pipeline when an item is removed.
• Clone: Triggers the pipeline for items generated through a "clone" operation.
• Task Assigned: Triggers the pipeline when an item is assigned to a task. To use this event, set the trigger to ItemStatus.TaskAssigned.
• Status Changed: Triggers the pipeline when a task or assignment status updates (e.g., task marked as "completed", assignment marked as "done").

** Use the trigger "itemStatus.Updated" to run your pipeline over items once their status is changed within a task context ('approved', 'discard', etc.).

Actions

Triggers respond to specific actions taken on resources:

Execution modes

Certain events, such as item updates, may occur multiple times for the same entity. The execution mode determines whether the pipeline runs every time the event occurs or only the first time.

• Once – The pipeline executes only the first time an event is triggered for a specific resource. For example, if the resource is an "item" and the action is "Updated," the pipeline will run only for the item's first update.
• Always – The pipeline runs every time the event is triggered for a resource. For example, if the resource is an "item" and the action is "Updated," the pipeline will execute for each update. In this mode, every occurrence triggers the full pipeline flow.

Cron triggers - scheduled execution

The Cron trigger allows pipelines to execute at scheduled time intervals using Cron syntax. For more details on Cron syntax, visit: Wikipedia - Cron.

Trigger types in pipeline nodes

This table outlines which pipeline nodes in Dataloop support different trigger types—either based on events (such as item uploads or task completions) or scheduled using cron expressions (time-based automation).

• ✅ Supported: The node supports the specified trigger type.
• ❌ Not Supported: The node does not support the specified trigger type.

Event-based triggers respond to specific actions or changes in the function based on the node category, while cron-based triggers allow pipelines to run at scheduled intervals (e.g., daily, hourly).

If function inputs are not supported by event triggers, then cron trigger will be the default selection, and you cannot select “event” trigger.

Set an event-type trigger for a pipeline

Allows you to configure a trigger that automatically starts the pipeline based on defined conditions or events.

1. Open Pipelines from the left-side menu.
2. Click on the pipeline from the list to add a trigger.
3. Identify and select the node to add a trigger. A right-side panel will be displayed.
4. In the Config tab, click on the Trigger (optional) toggle icon. A trigger ⚡ (lightning) icon will be added on top of the selected node.

5. Type: Select the type Event trigger, if it is not selected by default. It allows you to invoke a function in response to specific system events, using the triggering resource as the function's input.

6. Execution mode: By default, Once is selected. Read more.

7. Event Resource: Select an event resource from the list. Event triggers support only the following types of function inputs: Item, Dataset, Model, Task, Assignment, and Annotation. Read more.

8. Event Action: Select an event action (created, updated, deleted, and clone.) that will trigger the function for the selected resource. It allows you to filter any items to target specific resources or events, ensuring the trigger responds only to these specified conditions. Read more.

9. Type DQL:

   1. Write or Edit the DQL in the editor: Use DQL operators like $and, $or, $eq, $gt, etc.

   2. Example Use Cases:

      1. Only trigger on images: { "metadata.mimetype": "image/jpeg" }
      2. Trigger when a specific tag exists: { "tags": { "$in": ["for_review"] } }
      3. Trigger on items in a specific folder: { "dir": "/my-data/images" }

          {
          "$and": [
              { "hidden": false },
              { "type": "file" },
              {
              "metadata.mimetype": {
                  "$in": ["image/jpeg", "image/png"]
              }
              }
          ]
          }
          ```
      

   • "hidden": false — The trigger will exclude hidden items, ensuring only visible ones are considered.
   • "type": "file" — Only items of type file (excluding folders, videos, etc.) will trigger this node.

   Together, this DQL ensures the trigger activates only when a new visible file-type item is created (as seen from the Event Type: Item, Action: Created).

Set a cron-type trigger for a pipeline

Allows you to configure a trigger that automatically starts the pipeline based on the scheduled intervals.

1. Open Pipelines from the left-side menu.
2. Click on the pipeline from the list to add a trigger.
3. Identify and select the node to add a trigger. A right-side panel will be displayed.
4. In the Config tab, click on the Trigger (optional) toggle icon. A trigger ⚡ (lightning) icon will be added on top of the selected node.

5. Type: Select the type Cron trigger, if it is not selected by default. It allows you to schedule the selected function to run at specific times by using a cron expression to define the execution intervals. Set or select a cron schedule:

   1. Set Cron Schedule: Enter a time interval based on days, hours, minutes, and seconds. For example,

      Minute	Hour	Day of Month	Month	Day of Week
      0	*/2	*	C*	*

      Result is Schedule: At 0 minutes past the hour, every 2 hours.

   2. Type Cron Schedule: Enter one of the following cron type:
      Cron expressions format:

      Value	Schedule
      0 * * * *	Every hour
      */15 * * * *	Every 15 mins
      0 */2 * * *	Every 2 hours
      0 18 * * 1-6	Every week Mon-Sat at 6pm
      10 2 * * 6,7	Every Sat and Sun on 2:10am
      0 0 * * 0	Every Sunday midnight

Add triggers for pipelines using SDK

Add event or cron triggers to start nodes using the SDK code.

Trigger status

Pipeline triggers have three statuses:

1. Created - A new trigger is set but not yet active.
2. Active - The trigger is running and responding to events.
3. Inactive - The trigger is paused and will not respond to events.

Trigger activation & paused pipelines

• When a pipeline is paused, its triggers can remain active to queue events.
• Any new event during pause will be added as a pending cycle.
• Cron triggers do not remain active when a pipeline is paused.

When pausing a running pipeline with active triggers (excluding cron triggers), a popup will appear, allowing you to choose whether to keep the pipeline triggers active or deactivate them.

Update pipeline settings: On pause action

Each pipeline has its own independent settings. Configuring the desired behavior for one pipeline will not impact any of your other available pipelines.

To configure pipeline behavior when paused:

1. Open Settings via the three-dot menu.
2. Choose the desired action from the dropdown list.
3. Click Save Changes to the settings to apply them permanently.

Resuming pipelines with active triggers

• When resuming a paused pipeline, all triggers reactivate.
• A dialog will appear to select whether to process pending cycles.
• Click here for more details.