Tasks Overview

Overview

The Tasks tab on the labeling page displays the list of annotation and quality assurance tasks created using the Labeling Tasks page or Pipeline, along with their respective information.

Features

The Labeling Tasks in Dataloop allow you to perform the following functions:

• Creating any number of tasks.
• Assigning statuses to items per task, with predefined and custom statuses alike.
• Editing tasks, such as adding items to a task, adding team members, changing configurations, etc.
• Using the distribution data-allocation method to plan individuals' workload within a task.
• Utilizing the pulling data allocation method for agility and scalability, adding the required workforce to meet deadlines and quality goals.
• Assigning assignments to ensure work continuity for inactive team members.
• Redistributing work to balance workloads.
• Using consensus tasks for a majority vote and high-quality data.

Task details

This section provides detailed information about a task once you select it. It includes all relevant metadata and configuration settings related to the task.

List of tasks

The Tasks tab displays a table containing a list of tasks, with their respective details presented in individual columns. Clicking on the Task Name or double-clicking on the task row will navigate you to the assignments page for that specific task.

• Task: This displays the name of the task. You can click on it to view the task's browser page. Also, you click on the Copy icon to Copy the Task ID.
• Related Task*: This displays the name of the parent task.
• Status*: This displays the status of the tasks.
• Progress*: This displays the item's completion status.
• Task Owner: This displays the email ID and avatar of the task owner.
• Priority: This displays the tasks with a High, Medium, or Low priority.
• Type*: This displays the task type, labeling or QA tasks.
• Quality: This displays the tasks with task types, Consensus, Qualification, and Honeypot.
• Remaining*: This displays the number of remaining items to be completed in the labeling task.
• Allocation Method: This displays the task allocation method - Distribution or Pulling type.
• Assignments*: This displays the number of assignments associated to the tasks.
• Pipeline: The name and link of the pipeline to which the task belongs.
• Groups: This displays the group name of the task owner.
• Open Issues*: This displays the number of items with annotations that have open issues.
• For Review*: This displays the number of items with annotations that have issues corrected for review.
• Dataset: This displays the name of the dataset used to create the task.
• Updated At: This displays the task update date, for example, Sep, 15 2023.
• Created At: This displays the task creation date, for example, Sep, 15 2023.
• Due Date: This displays the due date for the task set by the task creator. It helps the annotation team to prioritize tasks and plan their schedule accordingly.

Show/Hide Columns: You can customize the visibility of the column fields in the table view by clicking on Show/Hide Columns.

The default column fields are marked with "*".

General details

When you select a task, the right-side panel of your Labeling Task page displays comprehensive information regarding the selected task, offering the following details for tasks:

The General Details section of the task provides you the following general information:

• Browser icon: This allows you to open and view the task browser){target="_blank"} page.
• Analytics icon | This takes you to open and view the Analytics page of the task.
• Owner: This displays the email ID and avatar of the task owner.
• Priority: This displays the tasks with a High, Medium, or Low priority.
• Channel: This displays the channel type used to create the tasks, Pipeline or Workflow.
• Created At: This displays the task creation date, for example, Sep, 15 2023.
• Updated At: This displays the task update date, for example, Sep, 15 2023.
• Quality: This displays the Qualification task type.
• Pipeline: This displays the pipeline name and link if the task is channel is pipeline.
• Description: The description provided for the selected task. To add a description, click on the Edit icon. You can also use SDK to add or edit description.

Use the Copy Task ID icon next to the task name to copy the task ID.

Status

The Status section of the tasks provides you the status of the item as follows:

• Progress: This displays the item's completion status.
• Open Issues: This displays the number of items with annotations that have open issues.
• For Review: This displays the number of items with annotations that have issues corrected for review.

Data source, instructions & statuses

This section of the task provides you the following information, related to the dataset and recipe used to create the task:

• Dataset: The name of the dataset used to create the selected task.
• Recipe: The name of the recipe selected for this task.
• Statuses: The annotation task status of the items in the task.

Assignments

This section of the task provides you the following information, related to the task assignment:

• Allocation Method: This displays the type of allocation method used in the task.
• Assignments: This displays the number of assignments available for the task.
• Groups: This displays the number of groups assigned.

Metadata (read only)

This section of the task provides you the metadata information of the selected task. You have the option to copy the metadata, but editing it is not possible.

For example,

{
  "system": {
    "nextIndex": 4,
    "relatedTasks": [],
    "disableWebm": false,
    "allowedAssignees": [],
    "archive": false,
    "markForDeletion": false,
    "consensusTaskType": "consensus",
    "consensusPercentage": 52,
    "consensusAssignees": 3,
    "scoring": true,
    "nodeId": "68ef5a4b-2960-4076-82f4-52887e7785d4",
    "pipelineId": "67f7731a586dd221f6036f50",
    "workloadDistribution": [
      {
        "assigneeId": "name@gmail.com",
        "load": 33.33
      },
      {
        "assigneeId": "name@dataloop.ai",
        "load": 33.33
      },
      {
        "assigneeId": "name@gmail.com",
        "load": 33.34
      }
    ],
    "consensusAssignmentId": "67f7731cbf685453abc1433b",
    "consensusTotalItems": 73,
    "consensusRemaining": 73,
    "consensusIssues": 0,
    "consensusForReview": 0,
    "groupsMembers": []
  }
}

Task scoring process

A new Function called Score function will handle annotation scoring for items in tasks. The default function can be replaced with custom functions by users.
Scoring functionality is tied to specific task types: Consensus, Qualification, and Honeypot tasks.

• Task Setup:
  • Scoring is enabled by default for these task types.
  • A scoring pipeline is created with the same name as the task.
  • The pipeline runs automatically, and users can manage it (stop, edit, or replace the scoring function).

• Function Deployment:
  • Scoring functions use project resources and affect billing.
  • Scoring resets if the item's status changes.

• Analytics:
  • A new scores tab provides insights:
    • Metrics for contributors, items, annotations, and labels.
    • Average IOU (Intersection Over Union) and labels consensus scores.
    • Histograms and bar charts for annotation and item scores.
    • Drill-down capabilities for detailed exploration.
  • Confusion Matrices:
    • Contributors-Agreement matrix.
    • Labels confusion matrix with clickable cells to explore respective items.
• Behavioral Notes:
  • Scoring resets when an item's status is updated.
  • Tasks created via SDK enable scoring by default.