Create Qualification Tasks

Qualification tasks create an assignment over entire selected data for each of the assignees. By having ground-truth annotations, a score is calculated and provided for each of the assignees.

Qualification tasks provide a tool to evaluate annotators' skills and performance by letting them work on a 'test' task that has ground-truth answers hidden from them. After completing their work (the assignment-completed event), scores can be calculated by comparing annotations in items in their assignment with ground-truth annotations from the original data.

How qualification works

Qualification tasks provide a tool to evaluate annotators' skills and performance by letting them work on a 'test' task that has ground-truth answers hidden from them. After completing their work (the assignment-completed event), scores can be calculated by comparing annotations in items in their assignment with ground-truth annotations from the original data.

Unlike consensus tasks, the annotator's work is not merged back into the original item, maintaining it as a clean qualification-test copy. By creating multiple qualification tests in the project context, managers can obtain insights into annotators' performances with various data (for example, image and video) and tools (box, polygon, etc.).

Continuous qualification task

Qualification tasks by nature never ends. Any new user added to them as an assignee receives an assignment that includes all items. This allows for creating a qualification test once in a project and using it for testing new annotators along the project's lifecycle.

Qualification score

When selecting to enable qualification score, a Pipeline is created with 2 nodes:

1. Task node - the qualification task itself
2. Application (FaaS) node - running Dataloop default score function.

Dataloop default score function is in our GIT repository, and includes documentation of how score are calculated and saved for the different annotation types.

As such, you can fork our GIT repo and customize the score function to facilitate your custom logic, then add it as a new Application (FaaS) and place it in the Pipeline instead of the default Dataloop score function.

To learn more, contact the Dataloop team.

Create a qualification task

1. Open the Labeling from the left-side menu.
2. Click Create Task. The task type section popup is displayed.
3. Select the qualification from the popup.

4. Click Continue.

1. General

Enter or select the required details in the General section:

1. Task Name: Enter a name for the new task. By default, the project name + (total number of tasks + 1) is displayed.
   For example, if the project name is abc and the total number of tasks you have already is 5, then the new task name is abc-6.
2. Owner: By default, the current user's email ID is displayed. Click on it to select a different owner from the list.
3. Priority: Select a priority from the list. By default, Medium is selected.
4. Completion Due Date (Optional): Select a task's due date from the calendar.
5. Click Next: Data Source.

2. Data source

1. Enter or select a Dataset from the list.
2. Click Next: Instructions.

3. Instructions

1. Enter or select the required details in the Instructions section. The number of Labels and Attributes is displayed on the top-right side of the page.
   1. Recipe: By default, the default recipe is displayed. Select a recipe from the list, if needed.
   2. Labeling Instructions (.pdf): The labeling instruction document is displayed, if available. Go to the Recipe section to upload a PDF instruction. You can select the page range accordingly.
2. Click Next: Statuses. The Statuses section is displayed.

4. Statuses

1. By default, the Completed status is selected. Click Add New Status to add a new status.
2. Click Next: Assignments.

5. Assignments

1. Enter or select the required details in the Assignments section.
   1. Allocation Method: Select one of the following allocation methods: Distribution is selected by default.
      1. Pulling: The pulling distribution method means that annotators only pull a batch of items at a time and the maximum number of items in an assignment. You can make changes in the following fields if required: Pulling batch size (items) and Max items in an assignment.
      2. Distribution: The distribution allocation method means that the items will be distributed in advance among users, equally or based on a custom percentage. The Auto Distribution option distributes the task equally among the users. By default, it is checked.
   2. Available Users: Search for or select users from the list, and click the Forward arrow icon to add them to the Assigned users list.
   3. Assigned Users:
      1. Search for or view the assigned users from the list. The allocation percentage is equally distributed if you select Auto Distribution.
      2. Select and click the Backward arrow icon to remove them from the Assigned Distribution list.

2. Click Next: Qualification.

6. Qualification

Enable advanced quality monitoring options to ensure data quality and review performance.

1. Apply score function, if required.
2. Click Create Task. The qualification task will be created and is displayed in the tasks list.

To edit a qualification task, see the Edit a task section.