Score Analytics

Overview

When running Consensus, Qualification, or Honeypot tasks and choosing the score-function option, completed items are processed by the function to have a corresponding score.

The analytics score dashboard displays the score information in the respective context:

• Consensus task: The consensus or agreement in the task for all annotations
• Qualification task: The individual annotator score vs. ground truth
• Honeypot task: See the quality scores for an annotator in a task for honeypot items having ground truth.

Metrics

• Assignees: Number of users assigned to the consensus, qualification, or honeypot task. In consensus tasks, not all users in the task necessarily work on consensus items.
• Items: Number of consensus, qualification, or honeypot items. In consensus and honeypot tasks, not all items in the task are necessarily used for quality control.
• Annotations: Number of annotations from all users, created in consensus, qualification, or honeypot items.
• Average Geometry Score: Average score between all annotations' geometry (Intersection Over Union for box or polygon annotations, and distance for point annotations).
• Avg. Labels Consensus: Average annotation-label consensus between annotations. Label agreement is either 0 or 1.

Graphs

Graphs related to annotation labeling on datasets are crucial for visualizing and assessing the quality and effectiveness of the annotation process in machine learning and data science projects.

Annotations Consensus

• The Annotation Consensus graph provides insight into the level of agreement among annotators, helping you gauge the quality of the annotations more effectively.
• The x-axis measures the number of annotations that fall within each consensus score range. The y-axis represents consensus score ranges (from 0-10 to 90-100), indicating the level of agreement among annotators.
• Distribution of annotations according to their quality score.
• Number of annotations by their average agreement score range (IOU, Labels, and Attributes consensus). For example, all annotations with an average score between 90 and 100.

Items Consensus

The Items Consensus chart is a visualization used to understand how consistently different evaluators or models agree on the classification or labeling of items within a dataset.

• Distribution of items according to the average quality score of their annotations. Number of items by average agreement score range (IOU, Labels, and Attributes consensus) of all annotations in them.
• The chart measures consensus or agreement levels among various assessors regarding specific items. This could be used in scenarios where multiple annotators label data, and it's important to measure how often they agree on their assessments.
• It is presented as a histogram, where the x-axis represents ranges of consensus percentages (e.g., 0-10%, 10-20%, etc.), and the y-axis represents the number of items that fall into each consensus range.
• For example, a high bar in the 90-100% range would indicate that many items received nearly unanimous agreement among annotators, suggesting high reliability in those annotations.
• Click on the Download icon to download the data in a CSV file.

The Items Consensus and Annotations Consensus Graphs Display Similar Scores

This occurs, for example, when an item is annotated by three different annotators, and each annotator assigns the same single label using the classification tool.

• In such cases, each item has exactly one annotation, resulting in the number of items being equal to the number of annotations.
• This alignment leads to similar results on both graphs.

The Items Consensus and Annotations Consensus Graphs Display Different Scores

The Items Consensus and Annotations Consensus graphs will display different results when an item has more than one annotation.

For example, consider a scenario where:

• There are 3 items, each annotated by three assignees.
• The 1st and 2nd assignees each annotated 2 annotations.
• The 3rd assignee provided 3 annotations.
• This results in a total of 7 annotations.

In this scenario:

• The Items Consensus graph will show that there is only one item being compared across the three different users, focusing on the item-level agreement.
• The Annotations Consensus graph, on the other hand, will display these 7 annotations and will provide a consensus agreement score for each annotation, showing the level of agreement on the annotations themselves rather than the items.

Assignees Confusion Matrix

• A confusion matrix showing the level of agreement (IOU, Labels, and Attributes consensus) between all annotations of assignees. The X-axis shows the base list of assignees, and the Y-axis shows the reference list of assignees in the task.
• The assignees' confusion matrix measures the agreement between each pair of assignees by comparing their annotations. For every pair, the cell value is determined by calculating the average of all annotation scores, relative to the other assignee in the pair.
• Click on the Download icon to download the data in a CSV file.

Labels Confusion Matrix

A confusion matrix showing the level of agreement over the different annotation labels. The X-axis shows the list of labels as ground truth, and the Y-axis shows their confusion with other labels. In consensus tasks, where there’s no ground truth, the X-axis is the first assignees in the task, as a baseline.

• Each cell in our label confusion matrix provides valuable insights into how different labels are compared during the annotation process.
• The numbers you see in each cell indicate the frequency with which two annotators either agree on a label (e.g., both labeling an item as "dog") or when an annotation compares with a ground truth standard.
• For instance, in our dog/cat dataset, if both annotators label an item as "dog," this agreement is recorded as '1' in the corresponding cell.
• Currently, if the same item is labeled differently by annotators (even if it's the same bounding box), it is not counted as a match. This approach ensures clarity in identifying areas of consensus and disagreement.
• Furthermore, the unlabeled category in the matrix indicates situations where one annotator has labeled an item, but another has missed it entirely.
• These instances are also counted, enhancing our understanding of the annotation consistency and completeness.
• Normalized and unnormalized confusion matrices: Click on the Normalized and unnormalized to view the confusion scores.

For example, the following image presents a confusion matrix with decimal values, possibly reflecting normalized data based on probabilities or percentages. The matrix includes categories for "cat," "dog," "other," and "unlabeled," indicating how often these labels are correctly or incorrectly applied.

Here’s an analysis:

This is a confusion matrix used to visualize the performance of a classification model. Here’s a breakdown of its components:

1. Labels: These indicate the categories used by the model. In this case, the categories are "cat," "dog," "other," and "unlabeled."
2. Matrix Cells: Each cell in the matrix represents the count of instances for a predicted label versus the actual label.
   1. The rows represent the actual labels.
   2. The columns represent the predicted labels.

• For example, the value 3.2 in the top left cell under "cat" and in the row for "cat" indicates that there were 3.2 instances where the model correctly predicted the actual label as "cat."

3. Off-diagonal Cells: These cells show where the model has made errors. For instance, the cell in the first row, second column (0.4) shows that there were 0.4 instances where the model incorrectly predicted "dog" when the actual label was "cat."
4. Color Intensity: Typically, the darker the color, the higher the count of instances for that cell, which helps in quickly identifying the most frequent predictions.

The matrix helps in assessing the model's ability to correctly classify each category and to see where it may be confusing one label for another.

Unnormalized and Normalized Confusion Matrices

Unnormalized Confusion Matrix

Unnormalized scores are the raw values generated directly by the scoring function without any adjustments. The values can vary widely based on the scale and units of the data, making comparisons difficult if the ranges differ significantly.

For example, as shown in the above screenshot:

• The cells contain the number of instances for each combination of actual and predicted labels.
• The cell corresponding to the actual label cat and predicted label cat shows 86, indicating that the model correctly identified 86 instances of cats.
• The cell corresponding to the actual label dog and predicted label dog shows 56, meaning the model correctly identified 56 instances of dogs.

This matrix helps in understanding the absolute number of correct and incorrect predictions made by the model. It is useful for evaluating the model's performance in terms of raw counts.

Normalized Confusion Matrix

Normalized scores are adjusted values that have been scaled to a fixed range between 0 and 1. This process facilitates easier and more meaningful comparisons, and allows you to compare them across different datasets.
The normalization allows for a comparison of model performance across different classes without being influenced by the class sizes.

For example, as given in the above screenshot:

• The cells display the proportion of instances for each combination of actual and predicted labels. To normalize these values, the score function divides each value by the highest value in the matrix, which is a score of 86.
• The cell corresponding to the actual label cat and predicted label cat shows 1, meaning the model perfectly identified all instances of cat when normalized.
• The cell corresponding to the actual label dog and predicted label dog shows 0.7, indicating that 70% of the actual dog instances were correctly identified when normalized.
• This matrix helps in understanding the performance of the model in a standardized way, facilitating comparisons between different classes or models.