Instance Segmentation Studio

Overview

Dataloop offers an Instance Segmentation Studio, a sophisticated form of image segmentation in computer vision that identifies and outlines each unique instance of objects within an image, differentiating between multiple occurrences of the same type of object. With the instance segmentation, each element has its own annotation.

Key Features

• Precise Object Differentiation: Automatically segments and distinguishes between individual instances of objects within an image.
• Unique Annotations for Each Instance: Ensures that every object instance has its own annotation, providing detailed and accurate segmentation.
• Enhanced Accuracy for Complex Scenes: Ideal for scenarios with overlapping or multiple objects of the same type.

Main Sections of the Instance Segmentation Studio

The main sections of an Instance Segmentation Studio are described below:

• Section 1: Label Picker and Annotation Tools (left-side panel)
• Section 2: Annotation Lists, Attributes, and Item Details (right-side panel)
• Section 3: Instance Segmentation Canvas
• Section 4: Item Info & Controls (top-side panel)

Instance Segmentation Canvas

The Instance Segmentation Studio Canvas provides a versatile platform for segmentation using the available annotation applicators.

Before You Begin

Supported Data Types

The Instance Segmentation Studio supports images.

• Refer to the Supported Image formats section to view the supported image formats in the Instance Segmentation Studio.
• To view the Instance Segmentation output JSON format, refer to the JSON format.

Supported Annotation Tools

The Instance Segmentation Studio supports various Semantic Instance Tool Applicators to collectively enhance the flexibility and efficiency of the Instance Segmentation Studio, enabling users to accurately label and segment objects within images for training machine learning models.

Brush Applicator

the Brush Applicator enables manual annotation by allowing you to draw directly over regions of interest within an image. It lets you select specific pixels for annotation, making it ideal for detailed and precise tasks where objects have complex shapes that require careful outlining.

Polygon Applicator

The Polygon Applicator is a tool designed for creating precise polygonal outlines around objects. It allows you to select a pixel area to start the annotation and click to place vertices along the object's perimeter.

Once all vertices are placed, the tool automatically connects them to form a closed shape. This applicator is particularly effective for annotating objects with simple, straight-edged shapes or when precision is required to define the edges accurately.

Freehand movement: Additionally, you can select pixels using freehand movement by holding down the left mouse button and dragging.

Bucket Applicator

The Bucket Applicator is a tool designed for quickly filling unsegmented regions within an image, making the annotation process more efficient. It works by filling areas where no pixels have been segmented until it reaches an existing segmentation boundary.

When you click on a specific area, the tool automatically fills the region with a chosen label, using color or texture similarities to detect boundaries. This applicator is particularly useful for painting backgrounds or filling large, contiguous areas with clear, well-defined edges that can be easily detected and segmented.

Eraser

The Eraser tool allows users to remove annotation's pixels or correct mistakes. You can use it for fine-tuning the segmented regions, correcting over-segmented areas, or removing incorrectly annotated pixels.

SAM Hover & Click

SAM Hover & Click provides an interactive approach to instance segmentation, allowing users to hover over and click on specific regions to generate segmentation masks.

When a user hovers over an object, the tool displays a suggested mask generated by a computer vision model for object detection. The user can then segment the suggested pixels with a simple click.

To learn more about the SAM, see the Segment Anything of Meta.

SAM Box

It allows users to quickly draw bounding boxes around objects, which the SAM then analyzes and suggests precise segmentation masks for the object.
This method is especially useful for defining the general area of an object before refining its boundaries.

Segment Anything Model (SAM)

The Segment Anything Model (SAM) in the Instance Segmentation Studio is a powerful tool designed to make the process of annotating images for instance segmentation both efficient and precise. SAM uses advanced machine learning algorithms to automate and assist with the segmentation process. For more information, refer to the Segment Anything.

The Segment Anything Model (SAM) in the Instance Segmentation Studio includes two key features: SAM Hover & Click and SAM Box. These features enhance the process of instance segmentation, making it more intuitive and efficient.

Work in the Instance Segmentation Studio

How to Perform the Instance Segmentation Using the Brush Applicator?

1. Select the Label from the Label Picker.
2. Click Add Instance. It activates the Semantic Instance Tools and by default, the Brush Applicator is selected. Also, it creates an annotation in the Annotation list.
3. In the Brush Setting, set the Brush Size and Active Opacity value to control the transparency of the filled regions. A smaller brush size is useful for precise work, while a larger brush can cover bigger areas more quickly.
4. Start painting over the object you want to segment. Carefully trace the boundaries of the object, making sure to cover all relevant parts. The painted area will be highlighted to show the segment being created.
5. If required, use the Eraser tool to remove any incorrect annotations.
6. Click Save to save the changes.

How to Perform the Instance Segmentation Using the Bucket Applicator?

1. Select the Label from the Label Picker.
2. Click Add Instance. It activates the Semantic Instance Tools and by default, the Brush applicator is selected. Also, it creates an annotation in the Annotation list.
3. Select the Bucket from the Semantic Instance Tools.
4. In the Bucket Setting, adjust the Active Opacity value to control the transparency of the filled regions.
5. Click on the area of the image that you want to segment. The Bucket Applicator will automatically fill the region with the selected label.
6. If required, use the Eraser tool to remove any incorrect annotations.
7. Click Save to save the changes.

How to Perform the Instance Segmentation Using the Polygon Applicator?

1. Select the Label from the Label Picker.
2. Click Add Instance. It activates the Semantic Instance Tools and by default, the Brush applicator is selected. Also, it creates an annotation in the Annotation list.
3. Select the Polygon from the Semantic Instance Tools.
4. In the Polygon Setting, set the Active Opacity value to control the transparency of the filled regions.
5. Click on the image to place the first vertex at the starting point of the object's boundary.
6. Continue clicking along the edge of the object to place additional vertices. Each click adds a new vertex and a connecting line segment, forming the outline of the object.
7. To close the polygon and complete the outline, click on the starting vertex (green color) or press Enter key. The area within the polygon will be filled to indicate the segmented region.
8. If required, use the Eraser tool to remove any incorrect annotations.
9. Click Save to save the changes.

How to Erase the Instance Segmentation Using the Eraser Applicator?

1. Identify the annotation to remove parts of the segmentation by erasing them.
2. Select the Eraser from the Semantic Instance Tools.
3. In the Eraser Setting, set the Eraser Size according to the detail required. A smaller eraser size is ideal for detailed corrections, while a larger size can be used for broader areas.
4. Click and drag the eraser over the areas of the segmentation that you want to remove. The erased regions will be cleared from the segmentation mask.
5. Click Save.

How to Perform the Instance Segmentation Using the SAM Hover & Click?

1. Select the Label from the Label Picker.
2. Click Add Instance. It activates the Semantic Instance Tools and by default, the Brush applicator is selected. Also, it creates an annotation in the Annotation list.
3. Select the SAM-Hover and Click from the Semantic Instance Tools.
4. In the SAM-Hover and Click Setting, set the Active Opacity value to control the transparency of the filled regions.
5. Move the cursor over different parts of the object. SAM will highlight the regions it detects as part of the object.
6. Click on the highlighted regions to create segmentation masks. Each click refines and adds to the segmentation.
7. Continue hovering and clicking to adjust the segmentation. For complex shapes, click multiple times to ensure all parts are accurately segmented.
8. If required, use the Eraser tool to remove any incorrect annotations.
9. Click Save.

How to Perform the Instance Segmentation Using the SAM Box?

1. Select the Label from the Label Picker.
2. Click Add Instance. It activates the Semantic Instance Tools and by default, the Brush applicator is selected. Also, it creates an annotation in the Annotation list.
3. Select the SAM-Box from the Semantic Instance Tools.
4. In the SAM - Box Setting, set the Active Opacity value to control the transparency of the filled regions.
5. Click and drag to draw a rectangular box around the object you want to segment. Ensure the box tightly encloses the object for the best results. Once the box is drawn, SAM will automatically analyze the contents within the box and generate an initial segmentation mask for the object.
6. If required, use the Eraser tool to remove any incorrect annotations.
7. Click Save.

How to Find the Unmasked or Not Annotated Pixels in the Instance Segmentation Studio?

Enforce 100% segmentation annotation: It is a feature in the Instance Segmentation Studio designed to help users identify unsegmented or unannotated areas within an image. This tool ensures the completeness and accuracy of annotations by highlighting any unmarked regions. It can detect up to a maximum of 500 unmasked pixels, aiding in thorough and precise annotation. If the number of pixels to be identified exceeds 500, a popup message is displayed to notify the user.

To find the Unmasked pixels:

After completing the annotation segmentation in the Instance Segmentation Studio, click on the Unmasked Pixels toggle option. This will display a popup message indicating the number of pixels that remain unmasked.

How to Change the Labels for an Annotation in the Instance Segmentation Studio?

1. In the Annotation list, select the annotation from the annotations list.
2. Click on either the Change label & attributes icon above the Search Annotations field. A pop-up window is displayed.
3. Make sure you are on the labels tab, and select the desired new label to switch to.
4. Click Save Changes.

How to Hide/Reveal an Annotation in the Instance Segmentation Studio?

1. In the Annotation list, select the annotation from the annotations list.
2. Click on the hide (eye) icon while hover over, or the eye icon above the Search Annotations field. An "Annotation is hidden. Show All" message is displayed.
3. Click Show All to reveal the annotation.

How to Set an Attribute for an Annotation in the Instance Segmentation Studio?

1. In the Annotation list, select the annotation. The Attributes section is displayed below.
2. Set the attributes and click Save icon.

How to Delete Instance Segmentation Annotations in the Instance Segmentation Studio?

1. Select the relevant annotation(s) from the annotation list.
2. Click on the trash icon to delete the annotation.
3. Click Yes to confirm the deletion.

Keyboard Shortcuts

General Shortcuts

Annotation Tool Specific Shortcuts