Annotations Validation

Overview

Annotations validation enables project managers to enforce annotation rules by loading a JavaScript file in the annotation studio and running it when annotators click the Action button to assign the item with a status. Developers can build and enforce any restrictions, such as:

• Labels that cannot co-exist (for example, you cannot label both genders in an image with one person)
• Number of polygon points
• Area or segmentation masks (avoid large masks)
• Minimum number of labels required

Using Annotations Validation

1. Open Recipe and change to the Instructions tab.
2. Locate the Annotation Verification section and click Upload to select and upload your JS validation script file. You can remove the file to allow changing versions, but only a single file can work in any recipe at a time.
3. Create an annotation task with this recipe as its instructions set.
4. Open the annotation studio from the task/assignment. The JS file is loaded in the background, and ready to be run.
5. Create annotations and press the actions button to set COMPLETED or any other status. The JS file is being executed.
6. As a user with Developer or Project-owner role, you will see a code-editor button.

7. Open your JS file in a code editor, edit it, and be able to run it.

8. Once the validation script finish running, a message with a success/error, as defined in the function's returned object.
9. In case of an error, specific annotations that failed to comply with the validation scripts' rules will be flagged as problematic. An annotation-specific error message will be displayed in a tooltip on mouse hover.

Script Guidelines

Your JS validation script must follow these guidelines:

1. Validation must occur in a function named validateAnnotations.
2. The function must get the Annotations object as an argument, which is an array of Annotation objects.
3. The function must return an object with the following interface:

{
    ok: boolean
    errorMessage?: string
    errors?: { [annotationId: string]: string }
}

Debugging

To debug the validation script while in the annotation-studio, use JavaScript debugging - console.log/debugger and inspect in your DevTools.

To see coordinates of annotations in the studio, which can help analyze the scripts' work/behavior, use the _def property.

Annotation Object

When interpret rating and referring to annotations received in the Annotations object, their properties comply with Dataloops' standard annotation JSON format, as documented here.

Sample Validation Scripts

Sample 1 - Only a single annotation can be on the item

 function validateAnnotations(annotationsArr) {
    //get all annotations in image
    let result = {}
    result.ok = true //or false
    result.errorMessage = "some message to display to user"
    result.errors = []
	console.log(annotationsArr.length)
    if (annotationsArr.length != 1){
        result.errorMessage = `there are ${annotationsArr.length} annotations. should only be one.`
        result.ok = false
    }

    return result
}

Sample 2 - A Polygon must have 6 points

function validateAnnotations(annotations) {
    const result = {
        ok: true,
        errors: {}
    }
    for (const a of annotations) {
        if (a.type === `segment` && a._def[0].length !== 6) {
            result.ok = false
            result.errors[a.clientId] = `Polygon must have 6 points`
        }
    }
    if (!result.ok) {
        result.errorMessage = `Some annotations have to be fixed`
    }
    return result
}

Sample 3 - Box annotations must have a parent annotation

function validateAnnotations(annotations) {
    const result = {
        ok: true,
        errors: {}
    }
    for (const a of annotations) {
        if (a.type === 'box') {
            if (!a.metadata.system.parentId) {
                result.ok = false
                result.errors[a.clientId] = 'Box annotation must have a parent'
            }
            }
        }
        if (!result.ok) {
              result.errorMessage = `Some annotations have to be fixed`
        }
        return result
}