Create Consensus Tasks

Overview

The Consensus is an important quality control feature, allowing you to compare annotations by different users on a specific item and generating majority-vote based, high-quality data.

• Purpose: To ensure quality control by comparing annotations from multiple contributors.

• Functionality:

  • Facilitates the comparison of annotations made by different contributors on the same item.

  • Ensures consistency and reliability by calculating consensus metrics.

  • Allows for more items to be annotated within a single task.

• Use Case: Ideal for tasks requiring alignment among multiple annotators working on the same data. The task calculator helps estimate the scope and scale of the task, providing clarity on workload distribution.

How Consensus Works

When consensus is enabled for a labeling task, it is configured with the percentage of items to cover and the number of annotators. Dataloop automatically creates copies of the items and assigns them randomly to annotators.

Key Points:

• Task Browsing: Only the original items are visible when browsing the task.

• Assignment Browsing: Users can view the specific copies assigned to them within their assignments.

• Annotation Merging: Once all copies of an item are assigned a status (e.g., "Completed"), the system merges the annotations back onto the original item. Until then, the original item will not display the annotations from its copies.

• Data Download: When downloading consensus data, the JSON file includes all annotations made by different users, along with their usernames. This allows users to calculate their own scores and determine which annotations are of the highest quality.

• Syncing Datasets with Consensus Task: Cloned external datasets (AWS, GCP, Azure) cannot be synced with a consensus task.

Consensus Score

Quality task scoring is now available for items, annotations, and users. When creating a task with Consensus enabled, a scoring function is activated to evaluate all users, items, and annotations. Detailed scoring insights can be accessed through the new Scores tab on the Analytics page.

Benefits of Consensus Scoring with Application Integration:

• Customizable Scoring: Calculate consensus scores using your preferred methods and thresholds (e.g., IOU).

• Dataset Management: Move or clone items and annotations to a majority-vote dataset for use in training or testing.

• Pipeline Integration: Trigger further processing for annotations with high scores in the pipeline workflow.

To learn more about the Dataloop Scoring Function, see the Scoring and metrics app README file.

Create Consensus Tasks

A labeling task assigned to multiple annotators for the same items, used to compare their results and measure agreement to ensure label quality and reduce subjectivity.

1. Open the Labeling from the left-side menu.

2. Click Create Task. The task type section popup is displayed.

3. Select the Consensus from the popup.

4. Click Continue.

1. General

Enter or select the required details in the General section:

1. Task Name: Enter a name for the new task. By default, the project name + (total number of tasks + 1) is displayed.
   For example, if the project name is abc and the total number of tasks you have already is 5, then the new task name is abc-6.

2. Owner: By default, the current user's email ID is displayed. Click on it to select a different owner from the list.

3. Priority: Select a priority from the list. By default, Medium is selected.

4. Completion Due Date (Optional): Select a task's due date from the calendar.

5. Click Next: Data Source.

2. Data Source

1. Search or select a Dataset from the list.

2. Click Next: Instructions.

3. Instructions

1. Enter or select the required details in the Instructions section. The number of Labels and Attributes is displayed on the top-right side of the page.

   1. Recipe: By default, the default recipe is displayed. Select a recipe from the list, if needed.

   2. Labeling Instructions (.pdf): The labeling instruction document is displayed, if available. Go to the Recipe section to upload a PDF instruction. You can select the page range accordingly.

2. Click Next: Statuses. The Statuses section is displayed.

4. Statuses

1. By default, the Completed status is selected. Click Add New Status to add a new status.

2. Click Next: Assignments.

5. Assignments

1. Allocation Method: Distribution is selected by default, and the Pulling method is not supported for Consensus tasks. The distribution allocation method means that the items will be distributed in advance among users, equally or based on a custom percentage.

2. Consensus Setup:

   1. Items for Multi-Labeling (by default, 100%): Use the slider to set the percentage of items to be labeled by multiple annotators.

      1. These items are randomly selected from the task, and the rest will be labeled by a single annotator. As you change the percentage of items, the exact number of consensus items is calculated and shown in the below Summary.

      2. When 100% is selected, the Distribute Equally will be disabled.

   2. Annotators Per Item (min 2): Use this slider to define how many annotators will label each consensus item. Annotators are randomly chosen from the task's assignee pool, and the items are distributed evenly among them.

   3. Summary: This section provides a breakdown of the labeling allocation:

      • % for Consensus: Indicates the number of items selected for consensus labeling, multiplied by the number of assigned annotators.

      • % No Consensus: Indicates the number of items assigned to a single annotator (i.e., not included in consensus labeling).

      • Total Items: Displays the total number of items in the task, including both consensus and non-consensus allocations.

3. Available Users in the Project: It displays the users available in the project. Search for or select users from the list, and click the Forward arrow → icon to add them to the Assigned users list.

4. Assigned Users for Task: It displays the user assigned for task from the project users list. Search for or view the assigned users from the list. The allocation percentage is equally distributed if you select Distribute Equally.

   1. Distribute Equally (Default: Enabled): When enabled, the task is evenly divided among all assigned users. Consensus data is always distributed equally. If you're working with only a subset of the data for consensus labeling, the remaining items can be distributed unequally by disabling this option.

   2. Removing Users: To remove a user from the assigned users list, select the user and click the Backward Arrow ← icon.

5. Click Next: Consensus.

6. Consensus

Enable advanced quality monitoring options to ensure data quality and review performance.

1. Score Function (Optional): Select a score function to calculate quality score. Clicking on the Apply Score Function, a pipeline will be created facilitating a score function running on items completed in the quality task.

2. Click Create Task. The consensus task will be created and is displayed in the tasks list.

Create Pipeline Consensus Tasks

A method of automating consensus tasks within a pipeline, enabling data to flow through multiple annotation stages with built-in agreement validation between annotators. To create a consensus task, you can use one of the following options to create an annotation task.

1. Open the Pipelines from the left side panel.

2. Select and drag the Consensus task node from the node library to the pipeline canvas.

3. Select the Consensus task node and click Create Consensus task from the right-side panel.

4. The Create New Task page is displayed and perform the following steps:

1. General

1. Enter or select the required details in the General section:

   1. Task Name: Enter a name for the new task. By default, the project name + (total number of tasks + 1) is displayed.
      For example, if the project name is abc and the total number of tasks you have already is 5, then the new task name is abc-6.

   2. Owner: By default, the current user's email ID is displayed. Click on it to select a different owner from the list.

   3. Pipeline Name: The name of the pipeline is displayed.

   4. Priority: Select a priority from the list. By default, Medium is selected.

2. Click Next: Data Source.

For the following steps, refer to the below links:

2. Data Source

3. Instructions

4. Statuses

5. Assignments

6. Consensus

The consensus task will be created and is displayed on the pipeline node.

Redistribute \ Reassign Consensus Tasks

Dataloop consensus tasks currently have the following limitations:

• Redistribution Restriction: To ensure consistency in quality evaluation, consensus tasks cannot be redistributed once created.

• Reassignment Rules: Consensus assignments can only be reassigned to users who have never had an assignment in the task. This prevents scenarios where a single annotator works on multiple consensus copies of the same item.

To reassign:

1. Open the Labeling > Tasks page.

2. Click on the consensus task. The Assignments tab is displayed.

3. Select the assignment to be reassigned, and click on the Assignment Actions.

4. Select the Reassign from the list. The Reassign Assignment window of the Edit Task will be displayed.

5. Select a new user from the list and click Apply Reassignment.

Refer to the Edit a task section to edit tasks.