Overview

Labeling page provides you the comprehensive list of labeling and review (QA) tasks, assignments, and task's annotation issues for your project in three different tabs.

• Tasks: The Tasks tab on the labeling page displays the list of annotation and quality assurance tasks created using the Labeling Tasks page or Pipeline, along with their respective information.
• Assignments: The Assignments tab on the labeling page displays the list of task's assignments along with their respective information.
• Issues: The Issues tab on the labeling page displays the list of Items with Annotation issues created during the quality assurance task, along with their respective information.

Upon selecting a task, assignment, or issue from the list, the right-side panel presents specific details, including general details, item's and task's status, data source and recipe, assignments, and metadata. Additionally, the labeling tasks page provides a summary of the total count of tasks (not in archive), assignments (not in Archive), issues, and reviews on the top right-hand side of the page.

Labeling in Dataloop

Labeling tasks in Dataloop allow you to annotate data, review annotations, label images, perform quality assurance checks, or any other data-related task requiring human input.

Annotation workflows on the Dataloop platform are created using any combination of two different task types:

• Labeling Task: The task of creating annotations of any type over data items. Task managers distribute work, monitor progress and performance, and allocate resources to meet deadlines and budget requirements.

• Review (QA) Task: The task of reviewing annotation work done in an annotation task. It has the option to flag annotations as having an 'issue' and send them for correction by the original annotator.

Tasks are created by managers who define the requirements for each task, such as:

• General Settings: Task priority, due date, and other parameters.
• Data Source: The scope of data to be annotated or reviewed.
• Instructions: A recipe that includes the labels/attributes to use (ontology/taxonomy), the labeling tools to use (e.g., classification, bounding-box, polygon), and work instructions (a PDF attachment).
• Status: Status of the task. By default, Completed is displayed. You can add a new status if needed.
• Assignments: Selected annotators, groups of annotators, reviewers, or domain experts.
• Allocation Method: Items can be distributed in advance, providing a known workload for team members, or pulled on-demand according to individual progress.
• Quality: Configuring the task with quality features, such as consensus tasks.

Upon creation, tasks are fragmented into assignments. Each assignment has subtasks assigned to individual annotators. Work progress is monitored through analytic metrics, from high-level task progress and quality to individual assignment performances.

Workflow Pipelines

Items can be included in any number of labeling and review (QA) tasks. Utilizing different recipes (work instructions), users can simplify complex annotation tasks into smaller ones, making them easier to complete and monitor, and incorporating automation steps and domain experts. Annotation workflows are best created, managed, and visualized in the Dataloop pipelines.

To learn more about creating and managing tasks, read here.

Annotator - Get started

1. Annotators are usually joined by an invitation to a project. If you're expecting an invitation, check with the project manager. Our customers also have the option to manage their workforce in Organizations, and from there assigned to different projects. In such cases, expect an invitation to the organization, but you'll need to be added to a project before you begin annotating.
2. The project you are currently working on is referred to as the active project. It is indicated as a breadcrumb at the header of the platform. It is simple if you have only one, otherwise, click My dashboard to see available projects and switch between them.
3. In your active project there's a list of My assignments where your work assignment with progress indication, number of items, and more. Double-click an assignment to start working on it.

Annotation Work Process

The following are the main concepts of annotation work in the Dataloop platform:

1. Your work in a project is contained in Assignments. A manager can create multiple assignments for you in the same project.

2. You work on files or items. The assignment can have all the files allocated to you in advance, or it can contain only a few, and more are added when those are completed.

3. You need to declare finishing work on a file by setting a status on it from the buttons in the bottom-right. The default ones for annotation work are:

   1. Complete - Annotation work is completed.
   2. Discard - The file is disqualified for annotation work.

4. You can browse between files in your assignment using the arrow keys at the bottom of the annotation studio. This can help in learning about edge cases or when waiting for information from your managers, reflecting new knowledge on uncompleted items. However, you are still expected to either Complete or Discard all the files.

Actions and Statuses

Work on items is declared done by setting a status on the item. Statuses are defined when creating a task. There are default statuses, but the task creator can define custom ones.

• Labeling tasks:

  • Complete: The default status to declare that annotation work is completed.
  • Discard: Disqualified item (cannot be annotated).

• Review (QA) tasks:

  • Approve: The default status to declare that QA review work is done.
  • Discard: Disqualified item (cannot be reviewed).

An item will consequently have several statuses, one for every task it was added to. The status of an item in a task can be changed. The original annotator/reviewer, or other project users with privileges, such as the task owner, can open the item again and set a different status.

Users and Roles

These are the main users that typically take part in a task, correlated with their role in a project:

• Task Owner: A user with the role of a project manager responsible for arranging the data, preparing and delivering it to tasks, and creating them.
• Task Manager: A user with the role of an annotation manager in the project responsible for task execution, workforce management, and accuracy assurance.
• Annotator: A user with the role of an annotator in the project responsible for actual labeling or quality assurance but limited to labeling/review (QA) work.

Task and Assignment Status

Tasks and assignments have a status set by the system that cannot be changed by the users:

• To-Do: A task or assignment that is not started yet.
• In-Progress: An assignment is being worked on by assignees. In a task, at least one of its assignments is being worked on.
• Completed: All items in the task or assignment are completed.
• Completed with issues: All items had statuses, but one or more items have an issue on at least one of the annotations; therefore, the task or assignment is not completed.

Status Overlay on Completed Items

Setting a status as Completed on items in annotation or QA tasks flags the item as done and often triggers further automation steps in Pipelines and FaaS. It is important to ensure that no changes to items/annotations are made after setting the status, otherwise, those changes won't be in effect for the automation steps. For example, setting a status may trigger a simple step for downloading the annotations JSON. If changes are made to the annotations afterward, there won't be a trigger for a new download process.

Now, the Dataloop platform forces all users to view an overlay of the status and must remove the status from the item before making any changes, even as part of fixing issues opened during a QA task.

To enable or disable this feature:

1. Open the Dashboard from the left-side menu.
2. Click on the Settings.
3. In the General section, click on the Enable force all users to remove status before editing annotations.