- 03 Jun 2024
- Print
- DarkLight
- PDF
Getting Started
- Updated On 03 Jun 2024
- Print
- DarkLight
- PDF
The DQL - The Dataloop Query Language
While the Dataloop platform gives you different capabilities to organize your data in datasets, folders, and versioning systems, you still need the ability to query your data. This is where our Dataloop Query Language becomes handy.
The Dataloop Query Language allows you to navigate through and sort massive amounts of data and can be used in the SDK as part of your code, as well as in the user interface in the Dataset Browser and in triggers and filters in the FaaS and Pipelines.
DQL Structure
Every DQL query has the following components:
- Resource : The target resource for the query. The resource can be items or annotations.
- Filter: The filter includes attributes and logical operators to filter items.
When using the DQL in the SDK, additional fields such as sort, page, and pageSize can be defined to sort the data that is returned from the query.
- sort: The properties and order by which results are sorted. Sort is set by default to include the attributes "type": “ascending” and "createdAt": “descending” (i.e., sort results by item type in ascending order and sort each item type by the date in which it was created in descending order – from newest to oldest).
- page: The page offset of the results (i.e., how many pages of the results should be skipped before presenting the results). The default value is 0.
- pageSize: The number of items per page. In the SDK, the default value is 1000 items per page.
For sorting and pagination of data in the UI, use the details view.
For example, the following query executed in the DQL Editor in the Dataset Browser will return only items that
- are of type file
- are not hidden
- have the attribute “createdAt'' greater than Jan 1, 2022
- and have the attribute “mimetype” set to “video*” (the wildcard character * makes this query search for any format of video – video/webm, video/mkv, etc.).
{
"filter": {
"$and": [{
"createdAt": {
"$gt": "2022-01-01T00:00:00.000Z"
}
}, {
"metadata.system.mimetype": {
"$eq": "video*"
}
}, {
"hidden": false
}, {
"type": "file"
}]
}
}
In this case, as the query was executed in the DQL Editor in the Dataset Browser, the system sets the "resource" to be “items”, "page": 0, "pageSize": 100, "sort" : {type: “ascending”, createdAt: “descending”}.
The "$and" array tells the system to apply the AND logical operator to the query fields within the array.
DQL Format
The query is written in JSON format: { }
JSON is a minimal text-based data exchange format that is used primarily to transmit data between a server and web application.
To learn more about JSON click here.
Data Types
The DQL uses the following data types:
- string: surround string constants with double quotes, e.g., "hello world"
- number: floating point and integer
- boolean: boolean variables are true or false
- array: collection of elements, e.g., [ "item1", "item2" ]
- null: no value
Operators
DQL uses the following logical operators in the following format:
- AND: “$and”
- OR: “$or”
- Exists: "$exists": true or false
- Equal: "$eq": "value"
- Equal not case sensitive (for strings): "$ieq": "value"
- Not equal: "$ne": "value"
- Greater than: "$gt": "value"
- Greater than or equal: "$gte": "value"
- Less than: "$lt": "value"
- Less than or equal: "$lte": "value"
- Field List Matching: "$in": ["option-1", "option-2"]
- Wildcard characters: * (representing zero or more characters) and ? (representing 1 character)
More on operators here.