Labeling

Overview

Labeling page provides you the comprehensive list of annotation and QA tasks, assignments, and task's annotation issues for your project in three different tabs.

• Tasks: The Tasks tab on the Labeling page displays the list of annotation and quality assurance tasks created using the Labeling Tasks page or Pipeline, along with their respective information.
• Assignments: The Assignments tab on the Labeling page displays the list of task's assignments along with their respective information.
• Issues: The Issues tab on the Labeling page displays the list of Items with Annotation issues created during the quality assurance task along with their respective information.

Upon selecting a task, assignment, or issue from the list, the right-side panel presents specific details, including general details, item's and task's status, data source and recipe, assignments, and metadata. Additionally, the labeling tasks page provides a summary of the total count of tasks (not in archive), assignments (not in Archive), issues, and reviews on the top right-hand side of the page.

Labeling in Dataloop

Labeling tasks in Dataloop allow you to annotate data, review annotations, label images, perform quality assurance checks, or any other data-related task requiring human input.

Annotation workflows on the Dataloop platform are created using any combination of two different task types:

• Labeling Task: The task of creating annotations of any type over data items. Task managers distribute work, monitor progress and performance, and allocate resources to meet deadlines and budget requirements.

• QA Task: The task of reviewing annotation work done in an annotation task. It has the option to flag annotations as having an 'issue' and send them for correction by the original annotator.

Tasks are created by managers who define the requirements for each task, such as:

• General Settings: Task priority, due date, and other parameters.
• Data Source: The scope of data to be annotated or reviewed.
• Instructions: A recipe that includes the labels/attributes to use (ontology/taxonomy), the labeling tools to use (e.g., classification, bounding-box, polygon), and work instructions (a PDF attachment).
• Status: Status of the task. By default, Completed is displayed. You can add a new status if needed.
• Assignments: Selected annotators, groups of annotators, reviewers, or domain experts.
• Allocation Method: Items can be distributed in advance, providing a known workload for team members, or pulled on-demand according to individual progress.
• Quality: Configuring the task with quality features, such as consensus tasks.

Upon creation, tasks are fragmented into assignments. Each assignment has subtasks assigned to individual annotators. Work progress is monitored through analytic metrics, from high-level task progress and quality to individual assignment performances.

Labeling Tasks Features

The Labeling Tasks in Dataloop allow you to perform the following functions:

• Creating any number of tasks.
• Assigning statuses to items per task, with predefined and custom statuses alike.
• Editing tasks, such as adding items to a task, adding team members, changing configurations, etc.
• Using the distribution data-allocation method to plan individuals' workload within a task.
• Utilizing the pulling data allocation method for agility and scalability, adding the required workforce to meet deadlines and quality goals.
• Assigning assignments to ensure work continuity for inactive team members.
• Redistributing work to balance workloads.
• Using consensus tasks for a majority vote and high-quality data.

Workflow Pipelines

Items can be included in any number of annotation and QA tasks. Utilizing different recipes (work instructions), users can simplify complex annotation tasks into smaller ones, making them easier to complete and monitor, and incorporating automation steps and domain experts. Annotation workflows are best created, managed, and visualized in the Dataloop pipelines.

To learn more about creating and managing tasks, read here.

Actions and Statuses

Work on items is declared done by setting a status on the item. Statuses are defined when creating a task. There are default statuses, but the task creator can define custom ones.

• Labeling tasks:

  • Complete: The default status to declare that annotation work is completed.
  • Discard: Disqualified item (cannot be annotated).

• QA tasks:

  • Approve: The default status to declare that QA review work is done.
  • Discard: Disqualified item (cannot be reviewed).

An item will consequently have several statuses, one for every task it was added to. The status of an item in a task can be changed. The original annotator/reviewer, or other project users with privileges, such as the task owner, can open the item again and set a different status.

Users and Roles

These are the main users that typically take part in a task, correlated with their role in a project:

• Task Owner: A user with the role of a project manager responsible for arranging the data, preparing and delivering it to tasks, and creating them.
• Task Manager: A user with the role of an annotation manager in the project responsible for task execution, workforce management, and accuracy assurance.
• Annotator: A user with the role of an annotator in the project responsible for actual labeling or quality assurance but limited to labeling/QA work.

Task and Assignment Status

Tasks and assignments have a status set by the system that cannot be changed by the users:

• To-Do: A task or assignment that is not started yet.
• In-Progress: An assignment is being worked on by assignees. In a task, at least one of its assignments is being worked on.
• Completed: All items in the task or assignment are completed.
• Completed with issues: All items had statuses, but one or more items have an issue on at least one of the annotations; therefore, the task or assignment is not completed.

Status Overlay on Completed Items

Setting a status as Completed on items in annotation or QA tasks flags the item as done and often triggers further automation steps in Pipelines and FaaS. It is important to ensure that no changes to items/annotations are made after setting the status, otherwise, those changes won't be in effect for the automation steps. For example, setting a status may trigger a simple step for downloading the annotations JSON. If changes are made to the annotations afterward, there won't be a trigger for a new download process.

Now, the Dataloop platform forces all users to view an overlay of the status and must remove the status from the item before making any changes, even as part of fixing issues opened during a QA task.

To enable or disable this feature:

1. Open the Dashboard from the left-side menu.
2. Click on the Settings.
3. In the General section, click on the Enable force all users to remove status before editing annotations.

The Tasks page can be broken down into the following sections (marked on the screenshot), as outlined below:

Section 1: Create Task and Refresh Buttons

• Create Task: To create a Labeling or a QA task, click on the Create Task button. For more information, see the Annotation and QA tasks article.
• Create Labeling Workflow: To create an Annotation or a QA task using the pipeline, click on the dropdown arrow -> Create Labeling Workflow option.
• Refresh button: To refresh the list of tasks, assignments, or issues, click on the Refresh button.

Section 2: Search and Filters

The Labeling Tasks page enables you to search and filter the tasks, assignments, or issues. You have the flexibility to choose multiple filter criteria.

Search

The Labeling Tasks page enables you to enter the following search criteria to search tasks, assignments, and issues in the Search field:

• Tasks: To search for a task, utilize the following criteria:
  • Task Name: This displays tasks with the matching task name.
  • Task ID: It lists the tasks based on their unique task ID. You can either use the Copy Task ID icon next to the Task Name or use the URL to get the Task ID.
  • Dataset name: This displays tasks with the matching dataset name.
• Assignments: To search an assignment, utilize the following criteria:
  • Assignment Name: This displays the assignment with the matching name.
  • Task ID: This lists assignments based on the task ID of the assignment. You can either use the Copy Task ID icon next to the Task Name or use the URL to get the Task ID.
• Issues: To search an issue, use the Annotation Name. This displays the issues based on the annotation name.

After entering the search criteria, you can begin the search by either pressing the Enter (or Return) key or by clicking outside the Search field.

Filter

The Labeling Tasks page enables you to use the following filter criteria to filter tasks, assignments, and issues:

Filter Criteria for Tasks

When you click on the Select filters on the Tasks tab, the following filter criteria will be listed:

Filter Criteria for Assignments

When you click on the Select filters on the Assignments tab, the following filter criteria will be listed:

Filter Criteria for Issues

When you click on the Select filters on the Issues tab, the following filter criteria will be listed:

Filter Using the Project Owner or Username

• Select the Project Owner's name from the Select owner dropdown list to filter the tasks.
• Select the Assignment's Username from the Select user dropdown list to filter the assignments and issues.

After you select the filter criteria, the system initiates the search function.

Clear Filters: To clear the search or filter criteria, click on the Clear Filters button.

Section 3: List of Tasks, Assignments, and Issues

List of Tasks

The Tasks tab displays a table containing a list of tasks, with their respective details presented in individual columns. Clicking on the Task Name or double-clicking on the task row will navigate you to the assignments page for that specific task.

• Task Name: This displays the name of the task. You can click on it to view the task's browser page. Also, you click on the Copy icon to Copy the Task ID.
• Task Related*: This displays the name of the parent task.
• Status*: This displays the status of the tasks.
• Progress*: This displays the item's completion status.
• Type*: This displays the task type, labeling or QA tasks.
• Remaining*: This displays the number of remaining items to be completed in the labeling task.
• Assignments*: This displays the number of assignments associated to the tasks.
• Open Issues*: This displays the number of items with annotations that have open issues.
• For Review*: This displays the number of items with annotations that have issues corrected for review.
• Groups: This displays the group name of the task owner.
• Task Owner: This displays the email ID and avatar of the task owner.
• Priority: This displays the tasks with a High, Medium, or Low priority.
• Channel: This displays the channel type used to create the tasks, Pipeline or Workflow.
• Dataset: This displays the name of the dataset used to create the task.
• Updated At: This displays the task update date, for example, Sep, 15 2023.
• Quality: This displays the tasks with a quality task types, Consensus, Quality, and Honeypot.
• Created At: This displays the task creation date, for example, Sep, 15 2023.
• Due Date: This displays the due date for the task set by the task creator. It helps the annotation team to prioritize tasks and plan their schedule accordingly.
• Allocation Method: This displays the task allocation method - Distribution or Pulling type.

Show/Hide Columns: You can customize the visibility of the column fields in the table view by clicking on Show/Hide Columns.

The default column fields are marked with "*".

List of Assignments

The Assignments tab displays a table listing all assignments, with detailed information in each column. Clicking on an assignment's name directs you to its corresponding item's page. When filtered by username, the selection persists even after reloading or refreshing the platform.

• Users: This displays the avatar of the assignment's users.
• Assignments: This displays the name of the assignments. Use the Copy Assignment ID icon next to the assignment name to copy the Assignment ID.
• Type: This displays the task type, labeling or QA tasks.
• Status: This displays the status of the assignments.
• Priority: This displays the assignments' with a High, Medium, or Low priority.
• Progress: This displays the item's completion status.
• Open Issues: This displays the number of items with annotations that have open issues.
• For Review: This displays the number of items with annotations that have issues corrected for review.

You can customize the visibility of the column fields in the table view by clicking on Show/Hide Columns.

List of Issues

The Issues tab presents a table listing items with annotation issues, and it provides details for each item in the respective columns.

• Item: This displays the thumbnail view of the item.
• Annotation: This displays the label name used to annotate the item.
• Assignment: This displays the name of the item's assignment.
• Dataset: This displays the name of the dataset.
• Status: This displays the current status of the issue.
• Open Since: This displays the creation date of the issue.

Also,

• Click on the More Details to view the details including, File Name, Item ID, Dataset ID, etc.
• Clicking on the Annotation label, Assignment Name, and Dataset takes you to the respective page.
• Double-clicking on the issue takes you to the respective type of annotation studio of the item.

Section 4: Task Actions and Assignment Actions

When you click on a task or an assignment, a Task Actions or Assignment Actions action button is available on the right-side panel of the labeling tasks page.
When you click on the Task Actions or Assignment Actions button, the following list of actions becomes available:

Task Actions

• Edit Task: Edit the task allows you to change task details, such as the assignees, due date, and more. For more information, see the Edit an Annotation article.
• Open Task Browser: This allows you to open and view the task browser page.
• Open Task Analytics: This takes you to open and view the Analytics page of the task. For more information, see the Analytics article.
• Delete Task: This allows you to delete the selected task.

Assignment Actions

• Clear Batch | This allows you to clear the assignment batch of the tasks which are pulling tasks. |
• Load Batch | This allows you to load the assignment batch of the tasks which are pulling tasks. |
• Open Assignment Browser | This allows you to open and view the assignment browser page. |
• Open Assignment Analytics | This allows you to open and view the Analytics page of the assignment. For more information, see the Analytics article. |
• Reassign: This allows you to reassign the task to a new user.

Section 5: Task and Assignment Details

When you select a task or an assignment, the right-side panel of your Labeling Task page displays comprehensive information regarding the selected task or assignment, offering the following details for both tasks and assignments:

Tasks - General Details

The General Details section of the task provides you the following general information:

• Browser icon: This allows you to open and view the task browser page.
• Analytics icon | This takes you to open and view the Analytics page of the task.
• Task Owner: This displays the email ID and avatar of the task owner.
• Priority: This displays the tasks with a High, Medium, or Low priority.
• Channel: This displays the channel type used to create the tasks, Pipeline or Workflow.
• Created At: This displays the task creation date, for example, Sep, 15 2023.
• Updated At: This displays the task update date, for example, Sep, 15 2023.
• Pipeline: This displays the pipeline name and link if the task is channel is pipeline.
• Description: The description provided for the selected task. To add a description, click on the Edit icon. You can also use SDK to add or edit description.

Use the Copy Task ID icon next to the task name to copy the task ID.

Tasks - Status

The Status section of the tasks provides you the status of the item as follows:

Progress: This displays the item's completion status.
Open Issues: This displays the number of items with annotations that have open issues.
For Review: This displays the number of items with annotations that have issues corrected for review.

Tasks - Data Source, Instructions & Statuses

This section of the task provides you the following information, related to the dataset and recipe used to create the task:

• Dataset: The name of the dataset used to create the selected task.
• Recipe: The name of the recipe selected for this task.
• Statuses: The annotation task status of the items in the task.

Tasks - Assignments

This section of the task provides you the following information, related to the task assignment:

• Allocation Method: This displays the type of allocation method used in the task.
• Assignments: This displays the number of assignments available for the task.
• Groups: This displays the number of groups assigned.

Tasks - Metadata (Read Only)

This section of the task provides you the metadata information of the selected task. You have the option to copy the metadata, but editing it is not possible.

For example,

{
  "system": {
    "nextIndex": 3,
    "batchSize": 5,
    "maxBatchWorkload": 7,
    "allowedAssignees": [
      "name@dataloop.ai"
    ],
    "groups": [],
    "relatedTasks": [],
    "disableWebm": false,
    "groupsMembers": []
  }
}

Assignments - General Details

The General Details section of the assignment provides you the following general information:

• Browser icon: This allows you to open and view the assignment browser page.
• Analytics icon | This takes you to open and view the Analytics page of the assignment.
• Task Name: This displays the assignment name. You can click on it to navigate to the assignment browser page.
• User: This displays the avatar of the assignment's users.
• Priority: This displays the assignments with a High, Medium, or Low priority.
• Channel: This displays the channel type used to create the tasks, Pipeline or Workflow.
• Created At: This displays the assignment creation date, for example, Sep, 15 2023.
• Updated At: This displays the assignment update date, for example, Sep, 15 2023.
• Description: The description provided for the selected assignment's task. To add a description, click on the Edit icon. You can also use SDK to add or edit description.

Use the Copy Assignment ID icon next to the assignment name to copy the Assignment ID.

Assignments - Status

The Status section of the assignment provides you the following information:

Progress: This displays the item's completion status.
Open Issues: This displays the number of items with annotations that have open issues.
For Review: This displays the number of items with annotations that have issues corrected for review.

Assignments - Metadata (Read Only)

This section of the assignment provides you the metadata information of the selected assignment. You have the option to copy the metadata, but editing it is not possible.

For example,

{
  "system": {
    "datasetId": "64a671e0d854daf23d6ed2b9",
    "taskId": "65091cc7d93bfc635d4f1878",
    "active": true
  },
  "datasetId": "64a671e0d854daf23d6ed2b9"
}

Section 6: View the Number of Tasks, Assignments, Issues, and Reviews

The Labeling Tasks page provides you the number of Tasks, Assignments, Issues, and Reviews available. You can view the following information on the top-right side of the Labeling Tasks page.