Create Honeypot Tasks

A Honeypot task is a quality control mechanism used during data annotation to evaluate annotator accuracy and reliability. It works by embedding pre-annotated (ground-truth) items also called "honeypots"—within the regular labeling workflow. These items must be placed in a folder named /honeypot inside the dataset. Dataloop uses this folder as the source of trusted items for quality testing.

How honeypot works

While Qualification tasks assess annotators' skills through initial testing, Honeypot tasks continuously monitor their performance, quality, and accuracy during real-world annotation. This is achieved by embedding items with known ground truth within tasks containing raw data. The annotators' scores are calculated based on their accuracy on these honeypot items, providing an ongoing estimate of the quality and reliability of their annotations on raw images.

Honeypot Score

Honeypot scores can be integrated as part of a FaaS (with an event trigger or in a pipeline) and provide the annotator's score. Dataloop provides a default score function, but any custom function that runs over the assignment items (taking only the relevant honeypot ones) can be used for that purpose.

To learn more about the Dataloop Scoring Function, see the Scoring and metrics app README file.

Create a honeypot task

A quality control task where pre-labeled data is hidden among regular tasks to test annotators’ accuracy and reliability without their knowledge.

1. Open the Labeling from the left-side menu.
2. Click Create Task. The task type section popup is displayed.
3. Select the Honeypot from the popup.

4. Click Continue.

1. General

Enter or select the required details in the General section:

1. Task Name: Enter a name for the new task. By default, the project name + (total number of tasks + 1) is displayed.
   For example, if the project name is abc and the total number of tasks you have already is 5, then the new task name is abc-6.
2. Owner: By default, the current user's email ID is displayed. Click on it to select a different owner from the list.
3. Priority: Select a priority from the list. By default, Medium is selected.
4. Completion Due Date (Optional): Select a task's due date from the calendar.
5. Click Next: Data Source.

2. Data source

1. Enter or select a Dataset from the list.
2. Click Next: Instructions.

3. Instructions

1. Enter or select the required details in the Instructions section. The number of Labels and Attributes is displayed on the top-right side of the page.
   1. Recipe: By default, the default recipe is displayed. Select a recipe from the list, if needed.
   2. Labeling Instructions (.pdf): The labeling instruction document is displayed, if available. Go to the Recipe section to upload a PDF instruction. You can select the page range accordingly.
2. Click Next: Statuses. The Statuses section is displayed.

4. Statuses

1. By default, the Completed status is selected. Click Add New Status to add a new status.
2. Click Next: Assignments.

5. Assignments

1. Enter or select the required details in the Assignments section.
   1. Allocation Method: Select one of the following allocation methods: Distribution is selected by default.
      1. Pulling: The pulling distribution method means that annotators only pull a batch of items at a time and the maximum number of items in an assignment. You can make changes in the following fields if required: Pulling batch size (items) and Max items in an assignment.
      2. Distribution: The distribution allocation method means that the items will be distributed in advance among users, equally or based on a custom percentage. The Auto Distribution option distributes the task equally among the users. By default, it is checked.
   2. Available Users: Search for or select users from the list, and click the Forward arrow icon to add them to the Assigned users list.
   3. Assigned Users:
      1. Search for or view the assigned users from the list. The allocation percentage is equally distributed if you select Auto Distribution.
      2. Select and click the Backward arrow icon to remove them from the Assigned Distribution list.

2. Click Next: Honeypot.

6. Honeypot

Enable advanced quality monitoring options to ensure data quality and review performance.

1. Modify the Honeypot items values as required.
2. Apply score function, if required.
3. Click Create Task. The Honeypot task will be created and is displayed in the tasks list.

To edit a qualification task, see the Edit a task section.