Task Allocation Methods

Overview

Tasks can be configured to operate using one of the following two allocation methods, which dictate how items are distributed to the assignments.

• Distribution: In this method, when tasks are created, dataset items are initially allocated to assignments. Any modifications to assignments, such as the addition or removal of assignees or the rebalancing of items, necessitate a recalibration of the assignments.
• Pulling: In this method, dataset items are initially queued in a concealed assignment and subsequently distributed to assignees in small batches based on their progress. This method makes it quite straightforward to make changes to the team since no assignee is burdened with many items in their assignment.

Distribution Method

Items within a task are distributed among all users. The automatic option allocates an equal portion of items to each user, whereas the manual option provides greater control. As the task is actively being worked on by annotators, you have the flexibility to redistribute and balance the remaining items among annotators once more.

You can redistribute items to the assignees based on the specific context of their workload by using a Metadata Key or Value, or by employing a DQL query.

Pulling Method

When creating an annotation or QA task and choosing the Pulling allocation method, the Dataloop system will assign a batch of X items to each assignee at a time.
As assignees work on their tasks, the system will allocate new batches, each of the same size (batch size), whenever their pool of items requires replenishment. It's important to note that an assignee cannot have more than Y items pending in their assignment, as determined by the "Max items" setting.

Items that have not yet been assigned are maintained in a queue, and this queue cannot be interacted with or viewed by users.

Managing Pulling Tasks

Pulling-based tasks offer greater control and flexibility during their execution, making it easier to adapt to changing requirements and needs.

Changing Team Size

Use the Edit task option from a task's ellipsis icon (3-dots) to add or remove users.

• New users show as new assignments.
• Removed users show as zeroed and locked assignments.

It allows you to track past activity and analytics.

Clear an Assignment

With Pulling-based tasks, you can clear a specific assignment from its items and put them back into the general queue.
For example, if an assignee is not working on a specific day, you can clear the respective assignment from items without removing the user from the task.

1. Open Workflows > Tasks page from the left portal menu.
2. Double-click on the task and click on the Ellipsis icon.
3. Select Clear from the list. A confirmation message is displayed.

Load an Assignment

When the assignee is available, you can just load their assignment with a new batch, and they continue working with the new items they receive to the assignment.

1. Open Workflows > Tasks page from the left portal menu.
2. Double-click on the task and click on the Ellipsis icon.
3. Select Load Batch from the list. A confirmation message is displayed.

QA Considerations

Items that have been completed by a user may undergo a Quality Assurance (QA) task at a later stage, and there could be an issue identified in one of the annotations. In such a scenario, the status of the item will be removed, and the item will be returned to the queue of the annotation task.

Items with issues will be specifically routed back to the original user who created the annotations, allowing them to address and rectify their work. These items with issues will receive priority in the queue and will be the first to be assigned again with the next batch pulled by the assignee.

Working in Annotation Studio

When first entering the annotation studio, whether it's as part of a labeling or a QA task, the studio loads the first batch of items and the counter will show that we're on the first item of 5.

After setting a status for 4 items (e.g. Complete/Approve), the counter will show we're on the 5th item of 5 in total.

Competing the 5th item will cause the studio to pull the next batch of 5 items, and we will be back as the start point - item 1 of 5.

However, if at the point of being at 5/5 items we will not complete the 5th item, and just try to move to the next one, we will find ourselves on the 1st item our of 6. That happens because we have 1 item from the previous batch, and the studio was able to load 5 more items, because 6 is still less than the "Max items" settings (which is 7 in our example).

Similarly, if we skip 2 items from a batch, the studio will be able to load 5 more, and will be back at 1 out of 7 items.

If for example we have 5 items in our batch, and we skip 3 of them, we will be left with these 3 items, and won't be able to load more, because in our example loading 5 more items will take us over the "Max-Items" limit of 7. We will be forced to complete at least 1 more item from the 3 we have, before we can pull the next batch of 5.

As we browse through our batch of items and complete/approve them, we can use the back button to review items we already passed through. However, after the studio reloads a new batch of items, we cannot go back to items we already completed/approved by using the back button - we can still get to them by going to the assignment browser, finding the item and double-clicking it.

Bucket Assignment - Tasks Without Assignees

Distribution Type Tasks

You can create distribution-type tasks without any assignees using SDK.

• Once the task is created, an icon labeled Bucket Assignment will be displayed on the task's assignment, indicating that the entire task's workload will be assigned to the task owner as a bucket.
• The User Interface does not allow the creation of distribution-type tasks without any assignees.

Pulling Type Tasks

You can use either the SDK or the UI to create pulling-type tasks without any assignees. Once the task is created, no Bucket Assignment icon will be displayed on the task's assignment.

SDK Example

An example of task creation without any assignees - bucket assignment creation from the SDK.

import dtlpy as dl
import datetime

if dl.token_expired():
    dl.login()

project = dl.projects.get(project_id='ENTER YOUR PROJECT ID')
dataset = project.datasets.get(dataset_name='ENTER YOR DATASET NAME')

filters = dl.Filters()
task = dataset.tasks.create(task_name='ENTER YOUR TASK NAME',
                            due_date=datetime.datetime(day=15, month=7, year=2023).timestamp(),
                            filters=filters)

For more information about how to create a task using the SDK, see the Creating Tasks article.