KeyFrames and Advanced Tracking Tools

Dataloop platform offers the following three interpolation methods while annotating videos:

1. Linear
2. AI tracker
3. None

Linear Interpolation

Video Interpolation automatically calculates the linear change of size and position between two keyframes, allowing users to speed up work by eliminating the need to make small changes on every frame. By default, Linear interpolation is selected in the video studio.

How to Use the Linear Tracker?

1. Create an annotation while Linear Interpolation is selected.
2. Move forward to a later frame and adjust the annotation position.
3. Move backward and play the video to see how the annotation changes automatically between frames.

AI Tracker

The AI Tracker is a machine-learning-based application that can track objects automatically, potentially saving time over manual corrections.
Activate the AI Tracker by switching the interpolation method.

Once you create an annotation, the AI Tracker calculates size/position only for the next few seconds of the video. Do not move annotations or play the video during this calculation to avoid having the AI Tracker perform the calculation again before it finishes the previous one, as this may lead to errors.

None - No Tracker

In case where the task requires only human labeling, use the none interpolation method option. The annotations will be affected only by the human labelers’ annotations, with no automation assistance.

Frame Types

KeyFrames

Video annotations include keyFrames – frames in which the annotation was created (start frame), ended (end frame), or manually edited with actions, such as changing size, changing position, changing the label, setting/updating attributes, or turning occlusion on/off (hidden/visible).

KeyFrames are listed in the Annotations tab in the right panel, indicating ranges between one keyFrame and the next one, and allowing clicking to jump directly to the desired keyFrame. KeyFrames are also indicated over the annotation bar panel, as a full circle over the frame bar. You can delete a keyframe and remove it from the list by hovering over it and click Trash icon.

KeyFrames in JSON File

KeyFrames are recorded in the JSON file under Annotations.metadata.system.snapshots_. It specifies the annotation data relevant from this point forward, including position, label, and attributes.

Setting an End-Frame

By default, when creating a new annotation it has its first frame, and it lasts until the end of the video. If not specifically setting an end-frame at the end of the video file, which will be recorded as a fixed frame, the annotation will only have the first frame specified.

Automated Frames

Automated frames are generated automatically between two KeyFames when the Linear or AI Tracker is used since they can automatically adjust annotation position/size.

Automated Frames in JSON File

Automated frames are represented as additional frames recorded as snapshots at Annotations.metadata.system.snapshots_. They can be seen as type Linear.

Reflected Frames

Reflected frames exist between two keyframes. Since reflected frames are not automated, it means that nothing changed in the annotation in these frames.

For example, setting an annotation as hidden in frame 10 and visible again in frame 20 means that we have keyframes in frames 10 and 20, and reflected frames between 11 and 19, having the same information as frame 10.

Reflected Frames in JSON File

Reflected frames have no representation in the annotations JSON file.

Scope of Changes – Label & Attributes

Any changes made to an annotation’s label/attribute, for example, changing the annotation’s label, will automatically take effect in all following frames, until the next keyFrame. Any automated/reflected frame between the frame of change and the next keyFrame will be automatically adjusted.

This enables making corrections/changes to annotations much easier, eliminating the need for changing all frames one by one.

Scope of Changes – Size and Position

Changing the size and position of an annotation in an existing frame, where size and position have been previously determined, will only affect that specific frame. Changes are not calculated forward to the next frame or next keyFrame.

Hidden Annotations (Occlusion)

During video playback, an annotation might be hidden for a certain period. For example, a person walking in the street while being hidden by a passing car. Once visible again, the person is the same and must be annotated with the same annotation; hence, the annotation needs to be flagged as Hidden in a certain range of frames.

How to Hide Annotation?

1. Annotate the object you wish to annotate.

2. Set the annotation as hidden using the Set Visible/Set Hidden icon, either from the annotation controller or the annotation toolbar. While hidden, annotations are still visible on the canvas.

3. When the object is visible again, set it accordingly using the Set Visible/Set Hidden icon, either from the annotation controller or the annotation toolbar.

   
   

An object can have multiple ranges of hidden frames. The hidden frames are shown in the video annotations bar, making it easier to understand the annotation’s sequence.