Overview
Preprocesses are automated services and pipelines that run when new items are created in project datasets. They help prepare, transform, enrich, or analyze data before it is used in downstream workflows such as annotation, training, and data processing.
A preprocess becomes available on the Preprocesses page after a trigger is configured for a service or pipeline. Once enabled, the trigger automatically executes the associated service or pipeline whenever a new item is added.
The Preprocesses page provides a centralized, scope-aware view for managing these automations across the platform.
.png)
The Preprocesses table provides a centralized view of all configured preprocess triggers available within the current scope.
Each row represents a preprocess trigger associated with a pipeline or service (DPK application). When enabled, the trigger automatically runs on the customer's compute environment whenever a new item is created.
The table displays key information about each preprocess, including:
Preprocess Name
Version
Type
Function Name
Scope
Trigger Status
Preprocess Scopes
Preprocesses can be configured at the following scopes:
Project – Applies only to the current project.
Organization – Applies to all projects within the organization.
System – Applies across all organizations and projects.
Managing Preprocesses
Management permissions vary by scope:
Project preprocesses can be managed from the Preprocesses tab of a project by Project Developers and above.
Organization preprocesses can be managed from the System Project by Organization Admins and above.
System preprocesses can be managed from the System Project by System Administrators (Superusers).
Scope Visibility
The table displays preprocesses from the current scope and any higher scopes.
For example:
In a Project, users can view Project, Organization, and System preprocesses.
In an Organization, users can view Organization and System preprocesses.
In the System Project, users can view System preprocesses.
Preprocesses configured at higher scopes are displayed for visibility but can only be managed from their respective scope.
Trigger Status
Each preprocess can be:
Active: Runs automatically when new items are created.
Inactive: The preprocess does not run automatically when new items are created. While a preprocess is inactive, item creation events are not collected for that preprocess. As a result, items created during this period will not be processed automatically and must be processed later through a manual execution.
Users can enable or disable preprocesses using the Play/Pause controls. Enabling a preprocess does not apply it to previously uploaded items. Existing items must be processed manually.
Add Preprocess to a Project
The Marketplace in DDOE allows you to add preprocesses to your project to automatically run services or pipelines whenever new items are created.
When adding a preprocess:
Applications are created with their trigger Active by default.
Pipelines appear in the Preprocesses table only after they have been published.
To do so, perform the below steps:
Navigate to the Data page from the left-side menu.
In the Data Management page, select the Preprocesses tab.
Click Add Preprocess to open the Marketplace popup window.
Select one of the following tabs based on your requirement:
Pipelines: Predefined workflow pipelines
Applications: Services that run as standalone applications.
Select the desired pipeline or application and click Install.
Use Case
Automated Image Processing for an AI Training Project
Scenario
A customer (e.g., an automotive company) is building an AI model to detect road signs from images.
They continuously upload thousands of images into DDOE from:
Cameras
Data collection vehicles
External datasets
Problem Without Preprocesses
Every time an image is uploaded, the team must manually:
Resize images to standard dimensions
Normalize formats (PNG/JPG)
Extract metadata (timestamp, GPS info)
Organize files into folders
Solution Using Preprocesses
The customer installs a Preprocess (pipeline/service) from the Marketplace.
What the Preprocess Does Automatically:
When a new image is uploaded:
Resizes the image (e.g., 1024x1024)
Converts format to a standard type (JPEG)
Extracts metadata (GPS, timestamp)
Adds labels or tags (e.g., “daytime”, “rainy”)
Stores it in the correct structure
Default System Preprocesses
DDOE includes several system preprocesses that are active by default:
Image Preprocess
Video Preprocess
These preprocesses automatically generate thumbnails and extract metadata when new items are added.