QA/QC Workflow

Overview

The quality assurance process ensures that annotations are compliant with certain quality standards or requirements, as well as identifying and addressing issues, as well as verifying that the desired level of quality has been achieved.

The QA process consists of the following steps:

1. Reviewers can use two tools to indicate quality issues during QA.
   1. Create an Issue with an annotation. It is used when there is a quality problem with an annotation, such as a wrong position, label, attribute, etc.
   2. Create a Note annotation. It is used to indicate that an annotation is missing. The note annotation is created with an issue on it.
2. Having an issue with an item in a QA task removes its status from the original task where the annotation was created.
   1. For note annotations, since they are created in the QA task, the issue is assigned by default to the last person who sets a status on the item. However, the note reporter can manually change the note assignee. For more information on the Note annotation tool, see the Note article.
3. Annotators will see open issues on assignments, and the task owner or manager can see them on the task level.
4. Annotators can start correcting the annotations.
5. After correcting an annotation, the annotator flags it For review and sets the status to Complete. This will make the item appear again in the QA task.

6. The reviewer can view how many For review annotations are pending on their assignments and open them for review.
7. If corrections are accepted, the reviewer can flag them as Approved. If all corrections are approved, the reviewer can set the item's status to Approve.

To gain a clearer understanding of the QA process, see the QA Process Example section.

Roles of Users in the QA Process

• Create QA Tasks: An Annotation Manager creates QA tasks, redistributes and reassigns these tasks to annotators, reviews their tasks, etc.
• QA Tasks Assignees: QA tasks can be assigned to annotators, annotation managers, developers, and project owners.
• Report Issues: Issues can be reported only by annotation manager role and above.

QA Process Example

The following example gives you an idea of steps involved in the QA process.

Step 1: Task Creation

A task or assignment is created for an annotator. For more information regarding task creation, see the Create an Annotation Task article.

Step 2: Work on the Annotation Task

The annotator completes the annotation task.

Step 3: Create a QA Task

The annotation manager views the completed annotation task and creates a QA task (in this example, the annotation manager assigns the QA task to self and acts as a QA reviewer, but you can assign it to roles). For more information regarding QA task creation, see the QA task article.

Step 4: Create Issues During the QA

During the QA task, the QA tester (in this case, the annotation manager):

• Discovers issues with a couple of annotations and opens an issue with them.
• In addition, the QA tester discovers a few objects that are not annotated, and since there is no annotation, adds an issue for them.
• Creates a note (a ladybug icon) over the unannotated objects with instructions for the annotator. For more information about the Note, see the QA Note Annotation article.

When an issue or note is created, the Completed status is removed from the item, and the assignment becomes active again.

On the Task page, the progress bar shows that one item is not completed, and there are issues remaining to be resolved.

As you can see in the above screenshot, the annotation manager can view tasks, issues, and pending review items.

Step 5: View Issues Reported During the QA

The annotator can see that there are open issues with the task on the the Tasks page.

• Hovering over the red exclamation displays the number of open issues.
• Double-clicking on the exclamation opens the item with issues.

Step 6: Correct the Issues

1. The annotator corrects the annotations and marks those annotations For Review in the Annotations tab by clicking on the hourglass icon.

2. The annotator clicks Complete. On the Tasks page, the task is shown with items for review.

Similarly, the Tasks page on the QA tester’s platform shows that the annotator has submitted items for review.

Step 7: Review QA Corrections and Approve

The QA tester clicks the Browse Pending Review to view the corrected annotations on the items.

If the issues have been corrected satisfactorily, the QA tester approves them by:

1. Selecting the annotations that are marked for review.
2. Clicking on the Approve checkmark icon.

At this point, the For review icons disappear, and the QA tester can click Approve.

Both the annotation task and the QA task are now complete.

Set a Status

When working on a QA assignment, the assignee or QA tester is expected to review annotations and set the item's status to:

• Approved, if it is annotated satisfactorily.
• Discarded, if the item is unsuitable for the task.
• or raise issues if there are problems with the annotation of the item that the QA tester wants the annotator to fix.

Setting the status on an item will trigger the studio to move on to the next item.

View QA Status

You can view the QA status of a task on the following pages, depending on the user role and the type of information requested.

Task's Page

When you double-click on a task within the task page, it displays assignments, especially if there are items with open issues or those pending review. Additionally, on this page, you have the options to Browse Tasks and Browse Issues.

Task Assignments' Page

Double-click on a task to see all its assignments. Each assignment line shows an indication of whether there are open issues or items pending review.

Issues' Page

On the Workflows > Issues page, you can see the full list of the issues in a project. For more information about the issue page, see the Issues article.