Annotator Getting Started

Overview

Dataloop annotation platform (also referred to as annotation-studio and data-application) is designed to provide annotators with a smooth and streamlined experience, and to ensure pixel-accurate work at high velocity. The platform's performance, responsiveness, and user-friendliness are highly important to us, as it all directly reflects on annotation work throughput.

Getting started

1. Annotators are usually joined by an invitation to a project. If you're expecting an invitation, check with the project manager. Our customers also have the option to manage their workforce in Organizations, and from there assigned to different projects. In such cases, expect an invitation to the organization, but you'll need to be added to a project before you begin annotating.
2. The project you are currently working on is referred to as the active project. It is indicated as a breadcrumb at the header of the platform. It is simple if you have only one, otherwise, click My dashboard to see available projects and switch between them.
3. In your active project there's a list of My assignments where your work assignment with progress indication, number of items, and more. Double-click an assignment to start working on it.

Annotation Work Process

The following are the main concepts of annotation work in the Dataloop platform:

1. Your work in a project is contained in Assignments. A manager can create multiple assignments for you in the same project.

2. You work on files or items. The assignment can have all the files allocated to you in advance, or it can contain only a few, and more are added when those are completed.

3. You need to declare finishing work on a file by setting a status on it from the buttons in the bottom-right. The default ones for annotation work are:

   1. Complete - Annotation work is completed.
   2. Discard - The file is disqualified for annotation work.

4. You can browse between files in your assignment using the arrow keys at the bottom of the annotation studio. This can help in learning about edge cases or when waiting for information from your managers, reflecting new knowledge on uncompleted items. However, you are still expected to either Complete or Discard all the files.

QA Process

QA Process Overview

Many projects make use of Dataloop QA Workflow, which allows managers to take the work done by annotators and have it reviewed. If reviewers decide that corrections need to be made, they have several options:

1. Reviewer can do the corrections by themselves.
2. Reviewer can raise an Issue on an annotation.
   1. Raising an Issue on an annotation in a QA assignment will automatically remove the Complete status and the item will reappear in the annotation assignment.
   2. Annotators are expected to review all items with issues, correct them, and flag them with For-review. It allows reviewers to decide, if the correction is good, and the item can be finalized.
3. Reviewer can create an Annotation-Note.
   1. If an annotator misses an annotation, for example, placing a bounding box over three cars and skipping the fourth car, then the reviewer sees a problem but doesn't have the annotation to flag with an Issue.
   2. The tool for the above issue is the annotation note. For example, an annotation tool that puts an icon of a bug over the image/video, with a label Car and an optional free-text comment you missed this car.
   3. This annotation note is created with an Issue, and will trigger the same process.
   4. The annotator must fix the issue (for example, create a box over the fourth car), flag the annotation note with For-review*, and press Complete.

Working With QA

Assignments List

From the assignments page, annotators can view assignments that have issues, and address those by double-clicking the assignment.

Annotations List

In the annotation studio, annotations with issues are marked with the red exclamation mark.

Marking For-Review

After correcting the annotation, click the magnifier's icon from the annotation list, or select all the annotations with issues and do a bulk operation from a similar icon at the top icon line.