Overview

Overview

Labeling page provides you the comprehensive list of annotation and QA tasks, assignments, and task's annotation issues for your project in three different tabs.

• Tasks: The Tasks tab on the labeling page displays the list of annotation and quality assurance tasks created using the Labeling Tasks page or Pipeline, along with their respective information.
• Assignments: The Assignments tab on the labeling page displays the list of task's assignments along with their respective information.
• Issues: The Issues tab on the labeling page displays the list of Items with Annotation issues created during the quality assurance task, along with their respective information.

Upon selecting a task, assignment, or issue from the list, the right-side panel presents specific details, including general details, item's and task's status, data source and recipe, assignments, and metadata. Additionally, the labeling tasks page provides a summary of the total count of tasks (not in archive), assignments (not in Archive), issues, and reviews on the top right-hand side of the page.

Labeling in Dataloop

Labeling tasks in Dataloop allow you to annotate data, review annotations, label images, perform quality assurance checks, or any other data-related task requiring human input.

Annotation workflows on the Dataloop platform are created using any combination of two different task types:

• Labeling Task: The task of creating annotations of any type over data items. Task managers distribute work, monitor progress and performance, and allocate resources to meet deadlines and budget requirements.

• QA Task: The task of reviewing annotation work done in an annotation task. It has the option to flag annotations as having an 'issue' and send them for correction by the original annotator.

Tasks are created by managers who define the requirements for each task, such as:

• General Settings: Task priority, due date, and other parameters.
• Data Source: The scope of data to be annotated or reviewed.
• Instructions: A recipe that includes the labels/attributes to use (ontology/taxonomy), the labeling tools to use (e.g., classification, bounding-box, polygon), and work instructions (a PDF attachment).
• Status: Status of the task. By default, Completed is displayed. You can add a new status if needed.
• Assignments: Selected annotators, groups of annotators, reviewers, or domain experts.
• Allocation Method: Items can be distributed in advance, providing a known workload for team members, or pulled on-demand according to individual progress.
• Quality: Configuring the task with quality features, such as consensus tasks.

Upon creation, tasks are fragmented into assignments. Each assignment has subtasks assigned to individual annotators. Work progress is monitored through analytic metrics, from high-level task progress and quality to individual assignment performances.

Labeling Tasks Features

The Labeling Tasks in Dataloop allow you to perform the following functions:

• Creating any number of tasks.
• Assigning statuses to items per task, with predefined and custom statuses alike.
• Editing tasks, such as adding items to a task, adding team members, changing configurations, etc.
• Using the distribution data-allocation method to plan individuals' workload within a task.
• Utilizing the pulling data allocation method for agility and scalability, adding the required workforce to meet deadlines and quality goals.
• Assigning assignments to ensure work continuity for inactive team members.
• Redistributing work to balance workloads.
• Using consensus tasks for a majority vote and high-quality data.

Workflow Pipelines

Items can be included in any number of annotation and QA tasks. Utilizing different recipes (work instructions), users can simplify complex annotation tasks into smaller ones, making them easier to complete and monitor, and incorporating automation steps and domain experts. Annotation workflows are best created, managed, and visualized in the Dataloop pipelines.

To learn more about creating and managing tasks, read here.

Actions and Statuses

Work on items is declared done by setting a status on the item. Statuses are defined when creating a task. There are default statuses, but the task creator can define custom ones.

• Labeling tasks:

  • Complete: The default status to declare that annotation work is completed.
  • Discard: Disqualified item (cannot be annotated).

• QA tasks:

  • Approve: The default status to declare that QA review work is done.
  • Discard: Disqualified item (cannot be reviewed).

An item will consequently have several statuses, one for every task it was added to. The status of an item in a task can be changed. The original annotator/reviewer, or other project users with privileges, such as the task owner, can open the item again and set a different status.

Users and Roles

These are the main users that typically take part in a task, correlated with their role in a project:

• Task Owner: A user with the role of a project manager responsible for arranging the data, preparing and delivering it to tasks, and creating them.
• Task Manager: A user with the role of an annotation manager in the project responsible for task execution, workforce management, and accuracy assurance.
• Annotator: A user with the role of an annotator in the project responsible for actual labeling or quality assurance but limited to labeling/QA work.

Task and Assignment Status

Tasks and assignments have a status set by the system that cannot be changed by the users:

• To-Do: A task or assignment that is not started yet.
• In-Progress: An assignment is being worked on by assignees. In a task, at least one of its assignments is being worked on.
• Completed: All items in the task or assignment are completed.
• Completed with issues: All items had statuses, but one or more items have an issue on at least one of the annotations; therefore, the task or assignment is not completed.

Status Overlay on Completed Items

Setting a status as Completed on items in annotation or QA tasks flags the item as done and often triggers further automation steps in Pipelines and FaaS. It is important to ensure that no changes to items/annotations are made after setting the status, otherwise, those changes won't be in effect for the automation steps. For example, setting a status may trigger a simple step for downloading the annotations JSON. If changes are made to the annotations afterward, there won't be a trigger for a new download process.

Now, the Dataloop platform forces all users to view an overlay of the status and must remove the status from the item before making any changes, even as part of fixing issues opened during a QA task.

To enable or disable this feature:

1. Open the Dashboard from the left-side menu.
2. Click on the Settings.
3. In the General section, click on the Enable force all users to remove status before editing annotations.