Getting Started

The DQL - The Dataloop Query Language

While the Dataloop platform gives you different capabilities to organize your data in datasets, folders, and versioning systems, you still need the ability to query your data. This is where our Dataloop Query Language becomes handy.

The Dataloop Query Language allows you to navigate through and sort massive amounts of data and can be used in the SDK as part of your code, as well as in the user interface in the Dataset Browser and in triggers and filters in the FaaS and Pipelines.

DQL Structure

Every DQL query has the following components:

• Resource : The target resource for the query. The resource can be items or annotations.
• Filter: The filter includes attributes and logical operators to filter items.

When using the DQL in the SDK, additional fields such as sort, page, and pageSize can be defined to sort the data that is returned from the query.

• sort: The properties and order by which results are sorted. Sort is set by default to include the attributes "type": “ascending” and "createdAt": “descending” (i.e., sort results by item type in ascending order and sort each item type by the date in which it was created in descending order – from newest to oldest).
• page: The page offset of the results (i.e., how many pages of the results should be skipped before presenting the results). The default value is 0.
• pageSize: The number of items per page. In the SDK, the default value is 1000 items per page.
  
  

For example, the following query executed in the DQL Editor in the Dataset Browser will return only items that

• are of type file
• are not hidden
• have the attribute “createdAt'' greater than Jan 1, 2022
• and have the attribute “mimetype” set to “video*” (the wildcard character * makes this query search for any format of video – video/webm, video/mkv, etc.).

{
	"filter": {
		"$and": [{
			"createdAt": {
				"$gt": "2022-01-01T00:00:00.000Z"
			}
		}, {
			"metadata.system.mimetype": {
				"$eq": "video*"
			}
		}, {
			"hidden": false
		}, {
			"type": "file"
		}]
	}
}

In this case, as the query was executed in the DQL Editor in the Dataset Browser, the system sets the "resource" to be “items”, "page": 0, "pageSize": 100, "sort" : {type: “ascending”, createdAt: “descending”}.

The "$and" array tells the system to apply the AND logical operator to the query fields within the array.

DQL Format

The query is written in JSON format: { }

Data Types

The DQL uses the following data types:

• string: surround string constants with double quotes, e.g., "hello world"
• number: floating point and integer
• boolean: boolean variables are true or false
• array: collection of elements, e.g., [ "item1", "item2" ]
• null: no value

Operators

DQL uses the following logical operators in the following format:

• AND: “$and”
• OR: “$or”
• Exists: "$exists": true or false
• Equal: "$eq": "value"
• Equal not case sensitive (for strings): "$ieq": "value"
• Not equal: "$ne": "value"
• Greater than: "$gt": "value"
• Greater than or equal: "$gte": "value"
• Less than: "$lt": "value"
• Less than or equal: "$lte": "value"
• Field List Matching: "$in": ["option-1", "option-2"]
• Wildcard characters: * (representing zero or more characters) and ? (representing 1 character)

More on operators here.