Contents x
    Performance Dashboard
    • 15 Aug 2024
    • Print
    • Dark
      Light
    • PDF
    Contents

    Performance Dashboard

    • Updated On 15 Aug 2024
    • Print
    • Dark
      Light
    • PDF
    Article summary

    The Performance Dashboard on the Analytics page helps you to understand better how your team works and performs.

    On this page, you can find meaningful data on time spent On this page, you can find On this page, you can find meaningful data on time spent on labels, annotations, and items. As well as data on each annotator’s productivity and performance in the task and assignment context.

    This allows you to gain insight that can help you increase your team’s performance and productivity.

    The Performance dashboard page is described in two sections as follows:

    Section 1: Metrics

    The Metrics section within the performance dashboard furnishes the following details pertaining to annotations and items.

    • Total Working Time: Activity time in annotation studios and when performing classification from task-browser (does not include all time spent in task-browser, only time spent on classification).
      • Idle Time: A tab is classified as idle immediately when it is not in focus.
      • Active Time: A tab remains classified as active when it is opened on a different screen but still visible.
      • Metric Variance: Total working time and active time metrics may vary based on the duration of activities performed within the browser.
    Net Annotation Time vs. Total Annotation Time
    • Net and total annotation times differ; net time excludes any annotations not saved to the server, while total time includes all annotation activities, even unsaved ones.
    • The system stops counting time after 5 minutes of inactivity in a tab.
    • The assignment stats do not currently show net or total annotation time; they include idle time, which can inflate the total time shown.
    • Total Annotation Time: The overall duration devoted to working on annotations, encompassing tasks such as creation, editing, undoing (Ctrl+Z), and deletion. This also includes classification time from the task-browser.
    • Net Annotation Time: Time spent on creating annotations (defers from total working - it doesn't include editing/deleting/undoing time).
    Average Labels per Item
    • An item with zero seconds is unlikely, but possible if the time is very small. The system might show it as zero seconds, depending on rounding rules.
    • Labels time will be zero if there are no annotations, unless the annotations were subsequently deleted.
    • Avg. Item Time: Total annotation time, divided by the number of items in context, gives you the average annotation time per item.
    • Avg. Annotation Time: The average time it takes to create a single annotation.
    • Avg. Annotations per Item: Average number of annotations in an item.
    Bulk Classification Impact

    Bulk classification is monitored through a specific analytic metric, which contributes to total working time but isn't displayed separately on the dashboard.

    Section 2: Graphs

    The Graph section of the performance dashboard provides graphical representation of the following details.

    Annotations

    The count of annotations generated within a specified time frame. The display resolution for this information can be adjusted, ranging from hourly to daily or weekly intervals.

    Items Status

    The number of items marked with a status over a time interval. One graph line per status.

    Avg. Time/Class

    The average time spent on annotation work for each label: sort the list in ascending or descending order to investigate the extremes. This helps identify labels that are either too difficult or have unclear instructions for annotators, in contrast to labels that are completed too quickly and may not meet your requirements and standards.

    Time/Item

    A detailed list of the time spent on each item. Sort in ascending/descending order to investigate the items that took too long to complete or those that were perhaps completed too quickly and possibly not thoroughly enough.

    Total Items/Time

    It shows the distribution of item annotation time. The resolution can be selected (in minutes, hours, days), as well as steps (for example every 5 minutes or every 2 days).

    Avg. Annotation Time/Annotator

    The average time spent on annotation work per annotator. Sort in ascending/descending order to investigate the slowest and fastest annotators.

    Assignment Stats

    Provides in-depth details of each assignment within the selected scope (project/dataset/task). Utilize the table's sorting options to gain additional insights into both the assignment itself and the annotator's performance.

    The table includes the following details:

    • Assignment Name: The name of the assignment.
    • User: The assigned user's avatar is displayed. It's important to note that if actions within this assignment are carried out by someone other than the assigned user (such as a manager), that person and their corresponding actions are also documented in the table.
    • Labeling Company: It displays the user's labeling company.
    • Total Working Time: The time spent actively engaged in Studio, and the time dedicated to bulk classification in the task browser.
    • Active Time: The total time a user was active on workflows pages (task/assignment/issues table, task browser, task annotation studio).
    • Studio Annotation Time: The total time working on annotations through the annotation studio.
    • Task Browser Annotation Time: The total labeling time in the Task Browser view (such as bulk classification), allowing you to monitor team efficiency and productivity.
    • Start Time: The start time signifies the day and time when work commenced with the first item, marked by either the creation of an annotation or the assignment of a status (e.g., “Complete”). Browsing items in an assignment without performing any substantive work does not qualify as initiating the task.
    • Total Completed Items: The total number of items on which the user clicked “completed.”
    • Total Items Worked On: The total items that were opened by the user in the annotation studio.
    • Total Items with Status: The total number of items with a status in a particular assignment.
    • Avg. Item Active Time: The user's “Active Time” divided by the number of items.
    • Overall Items Time: The time the annotation studio was open.
    • Total Annotations: The number of annotations in the assignment.
    • Avg. Annotation Time: The sum of all annotation time divided by the number of annotations created in the assignment.

    Download Graph Data

    Click on the Download icon to download the selected graph data in CSV format.

    What's Next