Tasks

Overview

The Tasks tab on the labeling page displays the list of annotation and quality assurance tasks created using the Labeling Tasks page or Pipeline, along with their respective information.

Labeling Tasks Features

The Labeling Tasks in Dataloop allow you to perform the following functions:

• Creating any number of tasks.
• Assigning statuses to items per task, with predefined and custom statuses alike.
• Editing tasks, such as adding items to a task, adding team members, changing configurations, etc.
• Using the distribution data-allocation method to plan individuals' workload within a task.
• Utilizing the pulling data allocation method for agility and scalability, adding the required workforce to meet deadlines and quality goals.
• Assigning assignments to ensure work continuity for inactive team members.
• Redistributing work to balance workloads.
• Using consensus tasks for a majority vote and high-quality data.

Main Sections of Tasks Tab

The Tasks tab can be broken down into the following sections (marked on the screenshot), as outlined below:

Section 1: Create Task and Refresh Buttons

• Create Task: To create a Labeling or a Review (QA) task, click on the Create Task button.
• Create Labeling Workflow: To create a Labeling or a Review (QA) task using the pipeline, click on the dropdown arrow -> Create Labeling Workflow option.
• Refresh button: To refresh the list of tasks, assignments, or issues, click on the Refresh button.

Section 2: Search and Filters

The Labeling Tasks page enables you to search and filter the tasks. You have the flexibility to choose multiple filter criteria.

Search

The Labeling Tasks page enables you to enter the following search criteria to search tasks, assignments, and issues in the Search field:

• Tasks: To search for a task, utilize the following criteria:
  • Task Name: This displays tasks with the matching task name.
  • Task ID: It lists the tasks based on their unique task ID. You can either use the Copy Task ID icon next to the Task Name or use the URL to get the Task ID.
  • Dataset name: This displays tasks with the matching dataset name.

After entering the search criteria, you can begin the search by either pressing the Enter (or Return) key or by clicking outside the Search field.

Filter

The Labeling Tasks page enables you to use the following filter criteria to filter tasks when you click on the Select filters:

Filter Using the Task Owner or Username: Select the Project Owner's name from the Select owner dropdown list to filter the tasks.

After you select the filter criteria, the system initiates the search function.

Clear Filters: To clear the search or filter criteria, click on the Clear Filters button.

Section 3: List of Tasks

The Tasks tab displays a table containing a list of tasks, with their respective details presented in individual columns. Clicking on the Task Name or double-clicking on the task row will navigate you to the assignments page for that specific task.

• Task Name: This displays the name of the task. You can click on it to view the task's browser page. Also, you click on the Copy icon to Copy the Task ID.
• Task Related*: This displays the name of the parent task.
• Status*: This displays the status of the tasks.
• Progress*: This displays the item's completion status.
• Type*: This displays the task type, labeling or QA tasks.
• Remaining*: This displays the number of remaining items to be completed in the labeling task.
• Assignments*: This displays the number of assignments associated to the tasks.
• Open Issues*: This displays the number of items with annotations that have open issues.
• For Review*: This displays the number of items with annotations that have issues corrected for review.
• Groups: This displays the group name of the task owner.
• Task Owner: This displays the email ID and avatar of the task owner.
• Priority: This displays the tasks with a High, Medium, or Low priority.
• Channel: This displays the channel type used to create the tasks, Pipeline or Workflow.
• Dataset: This displays the name of the dataset used to create the task.
• Updated At: This displays the task update date, for example, Sep, 15 2023.
• Quality: This displays the tasks with a quality task types, Consensus, Quality, and Honeypot.
• Created At: This displays the task creation date, for example, Sep, 15 2023.
• Due Date: This displays the due date for the task set by the task creator. It helps the annotation team to prioritize tasks and plan their schedule accordingly.
• Allocation Method: This displays the task allocation method - Distribution or Pulling type.

Show/Hide Columns: You can customize the visibility of the column fields in the table view by clicking on Show/Hide Columns.

The default column fields are marked with "*".

Section 4: Task Actions

When you click on a task, a Task Actions button is displayed on the right-side panel of the labeling tasks page. Also, the following list of actions displayed:

• Edit Task: Edit the task allows you to change task details, such as the assignees, due date, and more. For more information, see the Edit an Annotation article.
• Open Task Browser: This allows you to open and view the task browser page.
• Open Task Analytics: This takes you to open and view the Analytics page of the task. For more information, see the Analytics article.
• Delete Task: This allows you to delete the selected task.

Section 5: Task Details

When you select a task, the right-side panel of your Labeling Task page displays comprehensive information regarding the selected task, offering the following details for tasks:

General Details

The General Details section of the task provides you the following general information:

• Browser icon: This allows you to open and view the task browser page.
• Analytics icon | This takes you to open and view the Analytics page of the task.
• Task Owner: This displays the email ID and avatar of the task owner.
• Priority: This displays the tasks with a High, Medium, or Low priority.
• Channel: This displays the channel type used to create the tasks, Pipeline or Workflow.
• Created At: This displays the task creation date, for example, Sep, 15 2023.
• Updated At: This displays the task update date, for example, Sep, 15 2023.
• Pipeline: This displays the pipeline name and link if the task is channel is pipeline.
• Description: The description provided for the selected task. To add a description, click on the Edit icon. You can also use SDK to add or edit description.

Use the Copy Task ID icon next to the task name to copy the task ID.

Status

The Status section of the tasks provides you the status of the item as follows:

Progress: This displays the item's completion status.
Open Issues: This displays the number of items with annotations that have open issues.
For Review: This displays the number of items with annotations that have issues corrected for review.

Data Source, Instructions & Statuses

This section of the task provides you the following information, related to the dataset and recipe used to create the task:

• Dataset: The name of the dataset used to create the selected task.
• Recipe: The name of the recipe selected for this task.
• Statuses: The annotation task status of the items in the task.

Assignments

This section of the task provides you the following information, related to the task assignment:

• Allocation Method: This displays the type of allocation method used in the task.
• Assignments: This displays the number of assignments available for the task.
• Groups: This displays the number of groups assigned.

Tasks - Metadata (Read Only)

This section of the task provides you the metadata information of the selected task. You have the option to copy the metadata, but editing it is not possible.

For example,

{
  "system": {
    "nextIndex": 3,
    "batchSize": 5,
    "maxBatchWorkload": 7,
    "allowedAssignees": [
      "name@dataloop.ai"
    ],
    "groups": [],
    "relatedTasks": [],
    "disableWebm": false,
    "groupsMembers": []
  }
}

Section 6: View the Number of Tasks, Assignments, Issues, and Reviews

The Labeling Tasks page provides you the number of Tasks, Assignments, Issues, and Reviews available. You can view the following information on the top-right side of the Labeling Tasks page.

Labeling Workflow Using Pipelines

Dataloop allows you to create a Labeling Task Workflow Using a Pipeline Template to streamline and manage the labeling process efficiently.

How to Automate a Labeling Task Workflow Using Pipelines?

To create the workflow, follow the steps:

1. Click on the Labeling from the left-side menu.
2. Click on the Dropdown arrow next to the Create Task, or click Automate Using Pipelines from the empty state.
3. Select the Create Labeling Workflow from the list. The Select Labeling Workflow page is displayed.
4. Select a Workflow widget and verify the details displayed on the right-side panel. A preview of the template with the available nodes is displayed.
5. Click on Create Workflow. A new pipeline workflow page is displayed, and you start building your workflow by using various pipeline nodes.

Labeling Tasks

A task of labeling contents, such as text, audio, images, video, etc. to efficiently organize the dataset. The machine learning models are used to recognize this annotation and make predictions.

How to Create a Labeling Task?

To create a labeling task, you can use one of the following options to create an annotation task:

• Labeling: You can click on the Labeling from the left-side menu, and click Create Task.
• Data: Open the Data from the left side panel.

1. Open the Data from the left side panel.
2. Select and open the dataset that you need to create an annotation task.
3. Click Dataset Actions.
4. Select Labeling Tasks > Create New Task. Enter required data as given in the following sections. Select a particular section from the left-side menu, if required.
   :::

Section 1. General

4. Enter or select the required details in the General section:
   1. Task Name: Enter a name for the new task. By default, the project name + (total number of tasks + 1) is displayed.
      For example, if the project name is abc and the total number of tasks you have already is 5, then the new task name is abc-6.
   2. Task Type: Select the Labeling type.
   3. Owner: By default, the current user's email ID is displayed. Click on it to select a different owner from the list.
   4. Status: By default, the To Do status is displayed, and it cannot be changed.
   5. Priority: Select a priority from the list. By default, Medium is selected.
   6. (Optional) Completion Due Date: Select a task's due date from the calendar.
5. Click Next: Data Source.

Section 2. Data Source

6. Enter or select the required details in the Data Source section.

   1. Select Dataset: By default, the selected dataset name is displayed, click on it to select a different dataset. The Dataset field is disabled, if you select any particular item(s) in the Dataset.

      Note:

      You cannot create a task with a dataset that contains items 80,000 or above. To use this dataset, sampling must be done or replaced with another dataset. You can view the number of total items on the top-right side of the page.

   2. (Optional) Filters: Refine data selection by selecting specific folders, using DQL filters, or sub-sampling (randomly and equally distributed). The Folder or DQL field is Active only if you do not select any items in the Dataset.

      1. Folders: Select a folder from the dataset, if available.
      2. Selected Filters / Saved DQL Query: Select a filter or saved DQL query from the list, if available.
      3. Data Sampling: Enter the Percentage or Number of Items for the task. Data sampling does not give an exact number of items.
         1. Percentage: The option selects the items randomly. For example, if the percentage is 100% for four items, then 75% is for three items (It can be 1/4, 3/4, or 4/4) from the selected dataset. *
         2. Number of Items: The allows you to select the items sequentially from the start of the dataset, not randomly.
      4. Collections: Choose a collection from the list to filter and display items within the selected collection.

7. (Optional) WebM Conversion: By default, Enforce WEBM conversion of video items for frame-accurate annotations is selected.

8. Click Next: Instructions.

Section 3. Instructions

9. Enter or select the required details in the Instructions section. The number of Labels and Attributes is displayed on the top-right side of the page.

   1. Recipe: By default, the default recipe is displayed. Select a recipe from the list, if needed.
   2. Labeling Instructions (.pdf): The labeling instruction document is displayed, if available. Go to the Recipe section to upload a PDF instruction. You can select the page range accordingly.

10. Click Next: Statuses. The Statuses section is displayed.

Section 4. Statuses

11. By default, the Completed status is selected. Click Add New Status to add a new status.
12. Click Next: Assignments.

Section 5. Assignments

13. Enter or select the required details in the Assignments section.
    1. Allocation Method: Select one of the following allocation methods:
       1. Pulling: The pulling distribution method means that annotators only pull a batch of items at a time and the maximum number of items in an assignment. You can make changes in the following fields if required: Pulling batch size (items) and Max items in an assignment.
       2. Distribution: The distribution allocation method means that the items will be distributed in advance among users, equally or based on a custom percentage. The Auto Distribution option distributes the task equally among the users. By default, it is checked.
    2. Available Users: Search for or select users from the list, and click the Forward arrow icon to add them to the Assigned users list.
    3. Assigned Users:
       1. Search for or view the assigned users from the list. The allocation percentage is equally distributed if you select Auto Distribution.
       2. Select and click the Backward arrow icon to remove them from the Assigned Distribution list.

14. Click Next: Quality.

Section 6. Quality (Optional)

Enable advanced quality monitoring options to ensure data quality and review performance.

15. Select the quality task type and proceed to customize its properties. By default, None is selected. Select the following types as needed:

    1. Consensus: The Consensus task is to create replicas of the items for simultaneous work by multiple annotators and generate majority-vote datasets. For more information, see Consensus.
    Not available for Pipeline tasks

    Qualification and Honeypot are not available for Pipeline tasks.

    b. Qualification: The Qualification task is used to create a dataset from multiple annotators. For more information, see Qualification.
    c. Honeypot: The Honeypot task is used to create a dataset from multiple annotators. For more information, see Honeypot.

16. Click Create Task. A confirmation message is displayed.

How to Edit an Annotation Task?

1. Open the Labeling page from the left-side menu.
2. Identify the annotation task that is to be edited, and click on the 3-dots icon.
3. Select Edit Task from the list. The Edit Task page will be displayed, indicating whether it originated from Pipeline or Workflow through the respective tag.

4. Select the required section and make the changes. For more information about each section, see Create an Annotation Task. You can modify only the following fields:
   1. General:
      1. Task Name
      2. Owner
      3. Priority
      4. Completion Due Date
   2. Instructions: Recipe
   3. Assignments:
      1. Edit the Item Workload
      2. Reassign
5. Click Save Changes.

How to Add Items to an Existing Annotation Task?

Before adding items to an existing task, you may select the items you wish to add by clicking on them (CTRL+click to select multiple items). If you do not select any items, you can choose to filter the items with a DQL query or add all items that are not already included in the task.

1. Open the Data page.
2. Identify and open the Dataset.
3. Select the required items from the dataset.
4. Click Dataset Actions.
5. Select Labeling Tasks > Add to an existing task from the list.
6. In the Select Task section, select the task to which you need to add items.
7. Click Next: Data Source.
8. In the Data Source section, edit the Filters (Optional) details.
9. Click Next: Assignments.
10. In the Assignments section, add contributors from the Available Contributors list to the Assigned Contributors list.
11. Click Add Items.

Review (QA) Tasks

The purpose of the Review (QA) task is to increase the quality of annotations by reviewing annotation work and triggering problematic ones for correction by their original creator.

The task of reviewing annotation work is completed as a Review (QA) task. It has the option to flag annotations as having an 'issue' and send them to the original annotator for correction.

You can create Review (QA) tasks based on the following two scenarios:

• A Review (QA) Task from a Labeling task: To validate annotations created by assignees.
• A Standalone Review (QA) Task: To validate annotations that are uploaded to the platform, for example, Annotations created by your model.

To learn about the Review (QA) process, see QA Process.

How to Create a Review (QA) Task for a Labeling Task?

To create a Review (QA) task, follow the instructions for each section:

1. Open the Labeling page from the left-side menu.
2. Click Create Task.

Section 1. General

3. Enter or select the required details in the General section:
   1. Task Name: By default, your task name - QA is displayed. Modify, if needed.
   2. Task Type: Select the Review (QA) as task type.
   3. Owner: By default, the current user's email ID is displayed. Click on it to select a different owner from the list.
   4. Status: By default, the To Do status is displayed, and it cannot be changed.
   5. Priority: Select a priority from the list. By default, Medium is selected.
   6. (Optional) Completion Due Date: Select a task's due date from the calendar.
4. Click Next: Data Source.

Section 2. Data Source

6. Enter or select the required details in the Data Source section.

   1. Select Dataset: By default, the dataset used to create the task is displayed.
   2. (Optional) Filters: Refine data selection by selecting specific folders, using DQL filters, or subsampling (randomly and equally distributed). The Folder or DQL field is Active only if you do not select any items in the Dataset.
      1. Folders: Select a folder from the dataset.
      2. Selected Filters / Saved DQL Query: Select a filter or saved DQL query from the list.
      3. Data Sampling: Enter the Percentage or Number of Items for the task. Data sampling does not give an exact number of items.
         1. Percentage: The option selects the items randomly. For example, if the percentage is 100% for four items, then 75% is for three items (It can be 1/4, 3/4, or 4/4) from the selected dataset. *
         2. Number of Items: The allows you to select the items sequentially from the start of the dataset, not randomly.
      4. Collections: Choose a collection from the list to filter and display items within the selected collection.

7. Click Next: Instructions.

Section 3. Instructions

8. Enter or select the required details in the Instructions section. The number of Labels and Attributes is displayed on the top-right side of the page.

   1. Recipe: By default, the default recipe is displayed. Select a recipe from the list, if needed.
   2. QA Instructions (.pdf): The QA Instruction document is displayed, if available. Go to the Recipe section to upload a PDF instruction.

9. Click Next: Statuses. The Statuses section is displayed.

Section 4. Statuses

10. By default, the Approved status is selected. Click Add New Status to add a new status.
11. Click Next: Assignments.

Section 5. Assignments

12. Enter or select the required details in the Assignments section.

    1. Allocation Method: Select one of the following allocation methods:
       1. Pulling: The pulling distribution method means that annotators only pull a batch of items at a time and the maximum number of items in an assignment. You can make changes in the following fields if required: Pulling batch size (items) and Max items in an assignment.
       2. Distribution: The distribution allocation method means that the items will be distributed in advance among users, equally or based on a custom percentage.
          1. The Auto Distribution option distributes the task equally among the users. By default, it is checked.
          2. The Show only unassigned users to any labeling task option allows existing users to complete their task.
    2. Available Users: Search for or select users from the list, and click the Forward arrow icon to add them to the Assigned Users list.
    3. Assigned Users:
       1. Search for or view the assigned users from the list. The allocation percentage is equally distributed if you select Auto Distribution.
       2. Select and click the Backward arrow icon to remove them from the Assigned Distribution list.

13. Click Create Task.

How to Edit a Review (QA) Task?

1. Open the Labeling page from the left-side menu.
2. Identify the Review (QA) task that is to be edited and click on the 3-dots icon.
3. Select Edit Task from the list. The Edit Task page will be displayed, indicating whether it originated from Pipeline or Workflow through the respective tag.

4. Select the required section and make the changes. For more information, see the Create a Review (QA) task topic.
5. Click Save.

How to Delete a Task?

1. Open the Labeling page from the left-side menu.
2. Identify the Annotation or QA task that is to be deleted and click on the 3-dots icon.
3. Select Delete Task from the list. A confirmation message is displayed.
4. Click Yes. A confirmation message is displayed.

Archive Tasks

Once you delete tasks or datasets:

Deleting Tasks: When you delete a task, if the task has been part of analytics activities, instead of deleting, Dataloop platform will automatically archive it for you. This means all the usual steps of deletion will take place, except the task will be safely stored in our archive rather than being permanently removed. This also applies to any assignments within the task that have analytics data – they'll be preserved in the archive as well.

Dealing with Datasets: When it comes to datasets, if any tasks within the dataset have been active or have analytics data, rest assured they won't be deleted. This is to ensure that no significant information or insights are inadvertently lost.

Archive Flag: If a task or assignment is archived due to associated activity after the deletion, a status indicating it is archived (archive= true) will be added to its metadata.system. This status can be used to specifically search for archived tasks or assignments.

Analytics Performance Tab: For a comprehensive view, our Analytics Performance Tab will now include data from archived assignments. You'll be able to see details like the assignment name and the annotator involved, even for tasks that have been archived instead of deleted. Ensure you have Annotation Manager or above to view archived assignments.