Pipeline Triggers

Overview

Triggers in Dataloop's Pipeline system determine how and when data flows between nodes. They initiate execution in response to scheduled times, platform events, or manual actions, enabling dynamic and flexible data processing.

This guide provides an overview of trigger types, their behavior, and how to configure them across different interfaces, including the UI and SDK.

• The triggered resource object is passed as an input parameter to the pipeline.
• A trigger can be assigned to multiple pipeline nodes, each executing independently when the trigger fires.

🔄 What are Triggers?

• A trigger is a configuration that initiates a pipeline flow.
• It impacts the specific node to which it is assigned.
• Triggers pass the relevant resource object (e.g., item, dataset) as input to the pipeline.
• A single trigger can be assigned to multiple pipeline nodes, each executed independently when the trigger is fired.

Invoking Data to Pipelines

You can invoke your data to pipeline by using Automatic, Manual, and via SDK invocation.

🔁 Automatic Invocation

Dataloop supports automatic triggers that execute pipeline flows without manual input. These can be configured to respond to time-based schedules or data-driven events.

⏰ Cron Based Triggers

• Schedule-based: runs the pipeline at fixed time intervals.
• Use cases: daily retraining jobs, hourly scans, periodic clean-up tasks.
• Configured using CRON expressions.

Learn more

⚡ Event Based Triggers

• Triggered by specific events within the Dataloop system.
• Events include:
  • Item created, updated, or deleted
  • Annotation added or modified
  • Dataset or folder activity
• Ideal for real-time, reactive automation.

Learn more

🧑‍💻 Manual Invocation

Users can also manually trigger pipeline flows directly from the dataset browser. You can trigger:

• Specific individual items
• All items matching a filter/query
• An entire dataset or folder

Learn more

🧠 Programmatic Invocation

For advanced control, Dataloop SDK allows you to define and trigger pipeline executions via code.

Key Functions:

• add_trigger(): Assign a trigger to a specific pipeline node.
• pipeline.update(): Save the updated pipeline configuration.

Execution Options:

You can use the SDK to manually invoke the pipeline from any custom script or application. Define precise input using:

• Item IDs
• Dataset IDs
• DQL filters for bulk operations.

Set Event Based Triggers

Configure an event-based trigger on a pipeline node to automatically initiate a pipeline in response to specific system events. These events are tied to actions occurring in the data source and can include Create, Update, Clone, and/or Delete operations.

An example, if a task node’s trigger is set to Created and Updated, the pipeline will be activated whenever an item is added or modified in the source dataset.

Event-based triggers enable functions to be invoked dynamically based on changes within the system, using the resource that triggered the event as the function’s input.

Triggers consist of:

• Resources: Core entities in the Dataloop platform, such as Item, Dataset, Annotation, or Task.
• Actions: Events associated with these resources, like Create, Update, Delete, or Clone.

You can also apply a DQL (Dataloop Query Language) filter to refine which resources trigger the pipeline, allowing for precise and controlled automation.

To set an event based trigger on a node:

1. Open Pipelines from the left-side menu.
2. Click on the pipeline from the list to add a trigger.
3. Identify and select the node to add the trigger. A right-side panel will be displayed.
4. In the Config tab, click on the Trigger (optional) toggle icon. A trigger ⚡ (lightning) icon will be added on top of the selected node.

5. Type: Select the type Event trigger, if it is not selected by default. It allows you to invoke a function in response to specific system events, using the triggering resource as the function's input.

6. Execution mode: By default, Once is selected. Read more.

7. Event Resource: Select an event resource from the list. Event triggers support only the following types of function inputs: Item, Dataset, Model, Task, Assignment, and Annotation. Read more.

8. Event Action: Select an event action (created, updated, deleted, and clone.) that will trigger the function for the selected resource. It allows you to filter any items to target specific resources or events, ensuring the trigger responds only to these specified conditions. Read more.

9. Type DQL:

   1. Write or Edit the DQL in the editor: Use DQL operators like $and, $or, $eq, $gt, etc.

   2. Example Use Cases:

      1. Only trigger on images: { "metadata.mimetype": "image/jpeg" }
      2. Trigger when a specific tag exists: { "tags": { "$in": ["for_review"] } }
      3. Trigger on items in a specific folder: { "dir": "/my-data/images" }

          {
          "$and": [
              { "hidden": false },
              { "type": "file" },
              {
              "metadata.mimetype": {
                  "$in": ["image/jpeg", "image/png"]
              }
              }
          ]
          }
          ```
      

   • "hidden": false — The trigger will exclude hidden items, ensuring only visible ones are considered.
   • "type": "file" — Only items of type file (excluding folders, videos, etc.) will trigger this node.

   Together, this DQL ensures the trigger activates only when a new visible file-type item is created (as seen from the Event Type: Item, Action: Created).

Resources

The following table lists available resources and their corresponding actions. Event triggers support only the following types of function inputs: Item, Dataset, Model, Task, Assignment, and Annotation:

• Created: Triggers the pipeline when a new item is added (uploaded or synced to a dataset).
• Updated: Triggers the pipeline when an item is modified. See ** for more details regarding itemStatus.Updated.
• Deleted: Triggers the pipeline when an item is removed.
• Clone: Triggers the pipeline for items generated through a "clone" operation.
• Task Assigned: Triggers the pipeline when an item is assigned to a task. To use this event, set the trigger to ItemStatus.TaskAssigned.
• Status Changed: Triggers the pipeline when a task or assignment status updates (e.g., task marked as "completed", assignment marked as "done").

** Use the trigger "itemStatus.Updated" to run your pipeline over items once their status is changed within a task context ('approved', 'discard', etc.).

Actions

Triggers respond to specific actions taken on resources:

Execution Modes

Certain events, such as item updates, may occur multiple times for the same entity. The execution mode determines whether the pipeline runs every time the event occurs or only the first time.

• Once – The pipeline executes only the first time an event is triggered for a specific resource. For example, if the resource is an "item" and the action is "Updated," the pipeline will run only for the item's first update.
• Always – The pipeline runs every time the event is triggered for a resource. For example, if the resource is an "item" and the action is "Updated," the pipeline will execute for each update. In this mode, every occurrence triggers the full pipeline flow.

Set Cron Based Triggers

Allows you to configure a trigger that automatically starts the pipeline based on the scheduled intervals. Cron triggers initiate the flow of data based on a time interval (e.g., every hour, day, etc.) or at a specific time set by the cron expression. Read more about setting cron expressions. Cron triggers work with nodes that either do not receive input or have a default input value.

An example of using a cron trigger in a node is a FaaS node that trains a model once a day; this node does not have inputs, but simply runs a service once a day to train the model. Similarly, you may create a Code node that will run as defined by the cron trigger, provided that the Code node either has a default input value or does not receive input.

The Cron trigger (scheduled execution) allows pipelines to execute at scheduled time intervals using Cron syntax. For more details on Cron syntax, visit: Wikipedia - Cron.

To set a cron based trigger on a pipeline node:

1. Open Pipelines from the left-side menu.
2. Click on the pipeline from the list to add a trigger.
3. Identify and select the node to add the trigger. A right-side panel will be displayed.
4. In the Config tab, click on the Trigger (optional) toggle icon. A trigger ⚡ (lightning) icon will be added on top of the selected node.

5. Type: Select the type Cron trigger, if it is not selected by default. It allows you to schedule the selected function to run at specific times by using a cron expression to define the execution intervals. Set or select a cron schedule:

   1. Set Cron Schedule: Enter a time interval based on days, hours, minutes, and seconds. For example,

      Minute	Hour	Day of Month	Month	Day of Week
      0	*/2	*	C*	*

      Result is Schedule: At 0 minutes past the hour, every 2 hours.

   2. Type Cron Schedule: Enter one of the following cron type:
      Cron expressions format:

      Value	Schedule
      0 * * * *	Every hour
      */15 * * * *	Every 15 mins
      0 */2 * * *	Every 2 hours
      0 18 * * 1-6	Every week Mon-Sat at 6pm
      10 2 * * 6,7	Every Sat and Sun on 2:10am
      0 0 * * 0	Every Sunday midnight

Add Triggers via SDK

Add event or cron triggers to start nodes using the SDK code.

Trigger Status

Pipeline triggers have three statuses:

1. Created - A new trigger is set but not yet active.
2. Active - The trigger is running and responding to events.
3. Inactive - The trigger is paused and will not respond to events.

Trigger Activation & Paused Pipelines

• When a pipeline is paused, its triggers can remain active to queue events.
• Any new event during pause will be added as a pending cycle.
• Cron triggers do not remain active when a pipeline is paused.

When pausing a running pipeline with active triggers (excluding cron triggers), a popup will appear, allowing you to choose whether to keep the pipeline triggers active or deactivate them.

Update Pipeline Settings: On Pause Action

Each pipeline has its own independent settings. Configuring the desired behavior for one pipeline will not impact any of your other available pipelines.

To configure pipeline behavior when paused:

1. Open Settings via the three-dot menu.
2. Choose the desired action from the dropdown list.
3. Click Save Changes to the settings to apply them permanently.

Resuming Pipelines with Active Triggers

• When resuming a paused pipeline, all triggers reactivate.
• A dialog will appear to select whether to process pending cycles.
• Click here for more details.