Filter Items

Overview

The DDOE Dataset Browser is a powerful interface that enables users to efficiently search, filter, and refine datasets to identify required items with ease. It provides multiple levels of search functionality, ranging from simple searches to advanced queries, ensuring flexibility for diverse user needs.

The Dataset Browser is a versatile tool that supports:

• Filtering by item, annotation, and task properties.

• Natural language semantic search via CLIP.

• Multi-level search approaches (Basic, Advanced, and DQL).

• Task-specific filtering for assignment management.

Filter Modes

The Dataset Browser supports three primary search modes: Basic Search, Advanced Search, and DQL Query Editor Search.

Also supports, Saved Filters search, Semantic Search, etc.

1. Basic Search

2. Advanced Search

3. DQL

4. Semantic Search

5. Collections

6. ML Subsets

7. Tasks and Statuses

8. Saved Filters (if available)

Basic Search

The Basic Search feature enables quick filtering and discovery of items using predefined parameters. It helps you easily locate items by applying simple conditions such as labels, attributes, status, users, and more.

Available pre-defined search filters are:

• Annotated: allows to search annotated items or not annotated items.

• Attributes: allows to search annotated items with attributes.

• Labels: allows to search annotated items with specific labels.

Clicking on the allows you to Add more search types into the query.

You can clear a single filter by pressing X or the Clear button inside the filter config dialog, or all filters by pressing Clear.

Advanced Search

Filters are integral components of the Dataset Browser, providing users with the capability to refine and narrow down the displayed items based on specific criteria. These filters offer a powerful tool for managing and exploring large datasets efficiently.

Advanced Search uses a search schema that allows users to query items and annotations data with higher granularity:

• Search Items using structured schema fields.

• Explore diverse use cases of search schemas within the Dataset Browser.

• View available schemas directly in the interface for guidance.

Search Query Variables

DQL Search

The DDOE Query Language (DQL) provides a powerful and flexible way to search, filter, and retrieve items within a dataset. Unlike Basic Search, which uses predefined filters, DQL enables users to build custom, fine-grained queries that can combine multiple conditions and operators.

DQL Syntax Highlights

DQL queries are written in a structured JSON-like format. Some of the common elements include:

• Field Operators:

  • =, != (equals / not equals)

  • >, <, >=, <= (range comparisons)

  • in, not in (match against lists)

• Logical Operators:

  • $and, $or, $not (combine multiple conditions)

• Wildcard & Regex Support:

  • Search partial text patterns or match flexible strings.

Learn more

Use the DQL Query Editor to Search

In addition to data search using the basic UI, it is possible to filter data using the Dataloop Query Language.

1. In the Dataset Browser, click on the DQL tab.

2. Click on the DQL Editor icon. The DQL Search window is displayed.

3. Edit the query as required, or select a saved query from the list.

4. Once edited, click Search to run the search query, or click Save As to save it.

Copy the DQL Filter Query

1. In the Dataset Browser, click on the DQL tab.

2. Click on the Copy Filter icon. A confirmation message is displayed, and the query will be copied in DQL format.

Expand the DQL Filter Query

1. In the Dataset Browser, click on the DQL tab.

2. Click on the Expand icon. The query field will be expanded.

Filters Operands

The following operators are applied within and between filters, unless otherwise specified when manually modifying a DQL query.

Cross Filter Operand

A relationship between multiple filters in a single query is based on the AND operand.
For example, filtering by status Annotated AND the user john@doe.ai AND the annotation type: Box, will return items that are annotated, have Box annotation, and have john@doe.ai as an annotator in one or more of the annotations.

Inter-Filter Operand

The operand relationship between multiple values in a specific filter is based on the OR operator. For example, filtering by labels with the values Person and Dog, the filter will return all items with annotations of either of these labels, not necessarily both at the same time.

Move Between Search Filter Modes

DDOE provides three filter modes for searching data:

• Basic: Simple filter-based searches

• Advanced: More detailed filtering options

• DQL: Query-based filtering using DQL syntax

You can switch between these modes using the filter mode toggle.

Important Notes

• Switching filter modes does not run a search automatically.

• Any changes that have not been applied are discarded when switching modes.

• If a query cannot be converted to another mode, that mode is disabled.

Supported Mode Transitions

When Switching to Basic Mode Is Not Allowed

You cannot switch to Basic mode if the query contains:

• Wildcards (for example, Dog*)

• Complex logic or nested conditions

• Unsupported operators such as:

  • !=

  • EXCEPT

  • INTERSECT

• Invalid queries

Supported, example:

Label = 'Dog' AND Annotated = true

Label IN ['Dog', 'Cat']

Not supported, example:

Label = 'Dog' OR Annotated = true
Label != 'Cat'

When Switching to Advanced Mode Is Not Allowed

You cannot switch from DQL to Advanced mode if the query contains:

• EXCEPT or INTERSECT operators

• Complex nested logic

• Invalid DQL syntax

• Queries that cannot be represented using Advanced filters

Loading Saved or URL Filters

When DDOE loads filters from Static filters, URLs, Saved task filters; it automatically selects the most suitable mode:

1. Basic Mode

2. Advanced Mode (if Basic is not supported)

3. DQL Mode (if Advanced is not supported)

DQL Queries in Advanced Mode

If an Advanced task filter uses a Match operator, DDOE keeps you in the current mode.

When a DQL query is active in Advanced mode, the Items filter section is locked and displays the following message: DQL Query is active, edit the query using the DQL editor.

Semantic Search - CLIP Based

NLP (Natural Language Processing) with CLIP (Contrastive Language-Image Pre-Training) refers to the integration of language understanding capabilities with visual recognition. CLIP is a neural network model that leverages NLP techniques to process and understand text while simultaneously analyzing and comprehending images. This enables the model to interpret and align textual descriptions with corresponding visual content effectively.

You can input any descriptive text, and the model will retrieve images or data entries that best match the query, significantly enhancing the user experience by making the search process more intuitive and flexible. Additionally, you can use it alongside other search fields such as Annotations and Items to further refine and improve your search results.

Prerequisites: Install a CLIP Model.

1. In the Data Browser, click on the Semantic Search.

2. Select the NLP (CLIP) from the list. A new field NLP (CLIP) is added.

3. Enter the required text in the NLP (CLIP) field. For example, "Image with red cars".

4. Click Search to view the search result.

Searchable and Unsearchable Metadata

• Searchable metadata refers to key-value pairs stored under metadata.user that are indexed by DDOE. Learn more.

• Unsearchable metadata in DDOE refers to custom key-value pairs stored under alternative subpaths like metadata.user.unsearchable, which are not indexed by the platform. Learn more.

Filter by Collections

The Collections feature in DDOE's Data Browser helps organize data by allowing you to group specific sets of items based on task needs (e.g., annotation, review, training) into a collection folder. You can create up to 10 collection folders.

1. Open the  Dataset Browser: Go to the Data → Datasets → Your Dataset → Data Browser.

2. Select the Collections from the top-bar. The available items are displayed in folder wise.

3. Select a collection from the list. Once selected, only items included in that collection will be displayed.

Filter Items without Collections

Click on the Items without collections

Learn more about Collections.

Filter by ML Subsets

The ML Subsets view in DDOE’s Data Browser is a dedicated feature that streamlines machine learning workflows by helping you organize and manage datasets more efficiently. It enables you to classify and filter items according to their ML subset assignments—such as train, validation, and test—which are essential stages in the ML lifecycle for model development and evaluation.

1. Open the  Dataset Browser: Go to the Data → Datasets → Your Dataset → Data Browser.

2. Select the ML Subsets from the top-bar. The available items are displayed in folder wise.

3. Select a subset. For example, Train, etc. and click outside of the ML Subsets pop-up. The items are added to the train subset will be filtered and displayed.

A search query for the train subsets will be also added and visible on the Items search field.

If there are no items added to ML Subsets, click Split Into Subsets.

Learn more about ML Subsets.

Filter by Tasks & Statuses

The Tasks & Statuses filter in DDOE's Data Browser allows you to filter and search for items based on task-related attributes, such as task status or lack of assignments. This feature offers advanced filtering options, enabling better data management.

Prerequisites: Create a Task.

Filters Items by Tasks & Statuses

1. Open the  Dataset Browser: Go to the Data → Datasets → Your Dataset → Data Browser.

2. Select the Tasks & Statuses from the top.

3. Select the following options to filter items:

   1. Items in task:

      1. Search tasks: Select a task to filter items included as part of this task.

      2. Filtering by Status:

         1. Use the Timestamp to select a time period to view the items. Use the filter to view items based on the following task status options.

         2. Items with Status [= (equals)]:  Filter items that have one of the following assigned task status: Complete or Discard

         3. Items with Status [!= ( not equals)]: Filter items that do not have one of the following assigned task status: Complete or Discard.

         4. Items without Status: Filter items that do not have a status. The DQL (Data Query Language) query will update accordingly.

   2. Items Not in task: Filter items that are not part of any tasks.

Item Status

An item earns the status annotated if it receives any form of annotation, such as a classification, a note, or any other tool-generated annotation, regardless of whether this occurs in the dataset browser or during a task. However, being annotated doesn't necessarily mean an item is completed.

• This situation may arise if the item is annotated via the dataset browser, where annotations can be saved without assigning a status, or if it's annotated during a task without the complete button being engaged. Perhaps because the annotator plans to return to it later.

• Completed status is assigned to an item when an annotator finalizes their work on it by clicking the complete button during a task.

• An approved status is granted after an item undergoes a QA task and the QA tester decides to approve it by clicking the respective button.

• An item is deemed not annotated if it lacks any annotations. Interestingly, an item can be marked as completed and yet be considered not annotated if the complete button was clicked without any actual annotation work being done on it.

Saved Filters

The DDOE platform allows you to customize and save search queries within data querying or search systems, designed to streamline and optimize user interactions with large datasets.

Save Search Queries

DDOE platform offers the ability to save specific search filter criteria, allowing for efficient and consistent future searches.

1. Open the  Dataset Browser: Go to the Data → Datasets → Your Dataset → Data Browser.,

2. Create a search query.

3. Click on Saved Filters.

4. Select Save Filter. A dialogue window is displayed.

5. Enter a name for the new filter query.

6. Click Save. A confirmation message is displayed.

Filter Items Using Saved Queries

DDOE platform offers the ability to reuse specific search filter criteria.

1. In the Dataset Browser, click on the Saved Filters.

2. Select Saved Filters and choose the saved filter from the list. Clicking on the saved filter allows the search query to run and displays the result.

View Saved Filters

Up to three saved filters are displayed in the top menu, above of the search bar. Clicking on any of these filters will automatically run the corresponding query in the Smart Search field, instantly applying the saved conditions to your dataset.

Delete Saved Search Queries

DDOE platform offers the ability to reuse specific search filter criteria.

1. In the Dataset Browser, click on the ⋮ Three Dots.

2. Select the Saved Filters.

3. Find the query to be deleted and click on the Delete icon when you hover over.

4. Click Delete Query. A confirmation message is displayed.