Overview

Labeling tasks in Dataloop allow you to annotate data, review annotations, perform quality assurance checks, or any other data-related task requiring human input.

Create Your Labeling Tasks

Annotation managers are responsible for creating tasks by specifying the labeling requirements. On the Dataloop platform, you can create labeling tasks directly from either the Dataset browser or the Labeling page.

Clicking on the Create Task opens a popup where you can choose to create one of the following task types:

• Labeling Tasks
• Review (QA) task
• Consensus task
• Honeypot task
• Qualification task
• Task from a pipeline template

Task Structure and Monitoring

Once a task is created, it is automatically divided into smaller assignments. Each assignment contains subtasks that are distributed to individual annotators. Progress is tracked using analytic metrics, providing insights from overall task completion and quality levels to the performance of individual assignments.

The Labeling page offers a comprehensive overview of your project’s tasks. It is organized into three tabs, displaying all labeling tasks (including consensus, honeypot, and qualification tasks), review (QA) tasks, individual assignments, and any annotation-related issues.

• Tasks
• Assignments
• Issues

Workflow Pipelines

Items can be included in any number of labeling and review (QA) tasks. Utilizing different recipes (work instructions), users can simplify complex annotation tasks into smaller ones, making them easier to complete and monitor, and incorporating automation steps and domain experts. Annotation workflows are best created, managed, and visualized in the Dataloop pipelines.

Actions and Statuses

Work on items is declared done by setting a status on the item. Statuses are defined when creating a task. There are default statuses, but the task creator can define custom ones.

• Labeling tasks:

  • Complete: The default status to declare that annotation work is completed.
  • Discard: Disqualified item (cannot be annotated).

• Review (QA) tasks:

  • Approve: The default status to declare that QA review work is done.
  • Discard: Disqualified item (cannot be reviewed).

An item will consequently have several statuses, one for every task it was added to. The status of an item in a task can be changed. The original annotator/reviewer, or other project users with privileges, such as the task owner, can open the item again and set a different status.

Users and Roles

These are the main users that typically take part in a task, correlated with their role in a project:

• Task Owner: A user with the role of a project manager responsible for arranging the data, preparing and delivering it to tasks, and creating them.
• Task Manager: A user with the role of an annotation manager in the project responsible for task execution, workforce management, and accuracy assurance.
• Annotator: A user with the role of an annotator in the project responsible for actual labeling or quality assurance but limited to labeling/review (QA) work.

Task and Assignment Status

Tasks and assignments have a status set by the system that cannot be changed by the users:

• To-Do: A task or assignment that is not started yet.
• In-Progress: An assignment is being worked on by assignees. In a task, at least one of its assignments is being worked on.
• Completed: All items in the task or assignment are completed.
• Completed with issues: All items had statuses, but one or more items have an issue on at least one of the annotations; therefore, the task or assignment is not completed.

Bulk Classifications

Bulk Classification in Dataloop's Assignment Browser is designed to help annotators quickly apply classification labels to multiple items at once—directly from the Cards View.

Status Overlay on Completed Items

Setting a status as Completed on items in annotation or QA tasks flags the item as done and often triggers further automation steps in Pipelines and FaaS. It is important to ensure that no changes to items/annotations are made after setting the status, otherwise, those changes won't be in effect for the automation steps. For example, setting a status may trigger a simple step for downloading the annotations JSON. If changes are made to the annotations afterward, there won't be a trigger for a new download process.

Now, the Dataloop platform forces all users to view an overlay of the status and must remove the status from the item before making any changes, even as part of fixing issues opened during a QA task.

To enable or disable this feature:

1. Open the Dashboard from the left-side menu.
2. Click on the Settings.
3. In the General section, click on the Enable force all users to remove status before editing annotations.