Model Versions

Overview

A model version can be initialized as a baseline within the Marketplace catalog by selecting a specific foundation model architecture, which is usually in an untrained state initially.

This allows for the flexibility to begin training the model with datasets of your choice. Once the model has been trained, it can undergo evaluation to assess its performance and effectiveness.

Following evaluation, the model can be deployed to make further predictions and annotation tasks. This process enables a streamlined approach to developing, testing, and utilizing AI models for various predictive tasks.

Version table details

The list of models and its details are available in the following table:

Search and filter

Dataloop platform allows you to search models using Model Name, Model ID, Application Name, Dataset Name, and Creator Email ID, providing users with the capability to refine and narrow down the displayed models.

Use the Filter by Status field to filter models by status. To learn more, refer to the Model Version Status section.

Model Version Status

A model version status is indicated in the versions table, and at the Version-Details page. The status on a model version can be one of the following:

• Cloning: The model version is currently in the cloning process.
• Created: The model version was created but was not yet trained.
• Pre-Trained: The model (entity) was trained already.
• Pending: The model version is pending training. Training will begin when there are available resources on the training service (begins with 'mgmt-train' and ends with the model-id of the respective model-architecture. For example, a single service handles training for all ResNet model versions, etc.). Increase the auto-scaling factor to enable immediate training.
• Training: The model version is currently training.
• Trained: The model version finished the training process.
• Deployed: The model version is running as an application service.
• Failed: Model training failed. This could be because of a problem with the model adaptor, the training method, configuration or compute resources issues. Refer to logs for more information.

Install Foundation Models

The Marketplace's Models tab showcases foundation model architectures, encompassing organization and project scope-level models. These foundational models serve as the basis for initiating model management, installing models, versioning, and training with diverse datasets.

To install, follow these steps:

1. Open the Models from the left-side menu. The Versions tab is displayed.
2. Click on the Install Foundation Model. The Select Foundation Model window of the Marketplace is displayed.
3. For remaining instructions, follow the steps.

Clone a Model

You can clone an existing model version to create a new one and train it with additional data.

1. Open the Models page.
2. In the Versions tab, find the model version you wish to use as a basis for a new version, and click the 3-dots action button.
3. Select the Clone Model Version option. The Clone Model Version window will be displayed.
4. Enter a new name for the model and a description (optional).
5. Add a Tag for the new model if needed. Tags from the base model will be displayed, if available.
6. In the Advanced section, select the Training Artifact:
   1. Local (default): Training data (W&B) are saved in the Dataloop file system. Provide the folder path info in the Local Path field.
   2. Item: Provide an item-id, and training data will be saved on it.
   3. Link: Provide a URL and the file name to save the training data on a remote server.
7. Make changes in the Model Configuration.
8. Click Clone Model. A confirmation message is displayed, and the new model version will be displayed.

Compare Models

Dataloop platform allows you to compare two or more models to analyze the parameters.

1. Open the Models page.
2. In the Versions tab, select the model versions you wish to compare.
3. Click Compare Models available on the top-bar. The Multiple Models page is displayed.
4. Analyze the details and parameters as required.

Train a Model

1. From the Models -> Versions tab, search and locate the model-version you would like to start training. You can train models that are with status created or pre-trained.

2. Select the Train option from the 3-dots actions button of the model. A Model Version Training window is displayed.

3. Data & Instructions: Review and make changes to the following fields in the Data & Instructions. You can select a dataset, recipe, and determine the subsets to be used as the model's training and validation sets.

   1. Dataset: Select the dataset from the list.
   2. Labels:
      1. Predefined labels: Click View Labels to use the model's predefined labels. If the model is not trained yet, then use the Predefined labels to train it.
      2. Recipe: Select a recipe from the list.
   3. Train Subset:
      1. Tagged Items: Select items marked with Train tag. Click Show Filter to view the filter.
      2. DQL Filter: Select a saved DQL filter.
      3. Folder: Select a specific folder to define the data scope for the train subset.
      4. Collections: Select collections from the list. It allows you to select items from the collections.
   4. Validation Subset:
      1. Tagged Items: Select items marked with Validation tag. Click Show Filter to view the filter.
      2. DQL filter: Select a saved DQL filter.
      3. Folder: Select a specific folder to define the data scope for the validation subset.
      4. Collections: Select collections from the list. It allows you to select items from the collections.
   Default Filter Selection Logic

   If the model has predefined filters: Compare them (without spaces/line breaks) with the ‘Tagged Items’ filter.

   • If it matches: Select ‘Tagged Items’ by default.
   • If it does not match: Default to ‘DQL Filter’.

   If there are no predefined filters, select ‘Tagged Items’ by default.

4. Click Next button. The Model Configuration section is displayed.

5. Make required changes and click Next: Service Settings.

6. Make required changes in the Service Settings section. Refer to the Edit Service Settings section for detailed information about the available fields.

7. Once you complete, click Train Model. A confirmation message is displayed.

Evaluate a Model

To evaluate a Pre-Trained, Trained, or Deployed model version:

1. From the Models -> Versions tab, search and locate the model-version you would like to start evaluating.
2. Select the Evaluate option from the 3-dots actions button of the model. A Model Version Evaluation window is displayed.
3. In the Evaluation:
   1. Dataset: Select a Dataset from the list to use it for the evaluation. By default, the model's dataset is displayed.
   2. Test Subset:
      1. Tagged Items: Select items marked with Test tag. Click Show Filter to view the filter.
      2. DQL Filter: Select a saved DQL filter.
      3. Folder: Select a specific folder to define the data scope for the test subset.
      4. Collections: Select collections from the list. It allows you to select items from the collections.
4. Click Next button. The Service Settings section is displayed.
5. Make required changes in the Service Settings section. Refer to the Edit Service Settings section for detailed information about the available fields.
6. Once you complete, click Evaluate Model. A confirmation message is displayed.

Deploy a Model

1. Open Models from the left-side menu.
2. In the Versions tab, select the model that you intend to deploy. You can deploy models that are pre-trained, trained, or deployed.
3. Click on the 3-dots action icon and select the Deploy option. The Model Version Deployment window is displayed.
4. Make required changes in the Service Settings section. Refer to the Edit Service Settings section for detailed information about the available fields.
5. Once you review the above parameters, click Deploy. A message will pop up when the model is deployed and ready as a service.

Model Details

Model-Version has an internal details view, with more information about the model-version.

To access the model-version details view, click a model version from the Versions table.

The model-version details view includes four tabs:

• Details
• Explore
• Precision-Recall
• Test

Details

Contains information about the model, such as:

• Model creation date and creator details.
• Tags: shows tags currently on the model version, allowing you to add additional tags or remove existing ones.
• Labels used in the model - review and search for labels.
• Model configuration.

Explore

The Explore tab displays the metrics generated during the training process, or metrics uploaded by SDK to store on the model-version. To learn more, refer to the Training Metrics article.

Precision-Recall

Precision and Recall are two critical metrics used to evaluate the performance of an AI model, especially in tasks like classification where the outcomes are categorized as positive (relevant) or negative (irrelevant). These metrics are particularly useful in scenarios where the classes are imbalanced (i.e., one class is significantly more frequent than others).

You can measure precision and recall values after setting up a model evaluation. To set up a model evaluation, refer to the Model Evaluation article.

Visualizing accuracy

Visualizing accuracy, recall (aka sensitivity), and precision, which are the common performance measures for classification tasks. Given samples from two categories, we can refer to samples as positive or negative (two classes). The left rectangle represents positive samples, and the right rectangle represents negative samples. The circle contains all samples predicted as positive. Given the model predictions, each sample can be considered as TP (true positive), TN (true negative), FP (false positive), or FN (false negative).

Precision recall curve

The Precision-Recall Curve is a plot that illustrates the trade-off between precision and recall for different thresholds. A higher area under the curve (AUC) represents both high recall and high precision, where high precision relates to a low false positive rate, and high recall relates to a low false negative rate. This curve is particularly useful for evaluating models on imbalanced datasets.

Precision by confidence threshold

Demonstrates the performance of your model for the highest-ranked label across the entire spectrum of confidence threshold levels. Raising the confidence threshold leads to a reduction in false positives, enhancing precision. Conversely, lowering the confidence threshold results in a decrease in false negatives, thereby improving recall.

Precision

Precision, also known as positive predictive value, measures the accuracy of the positive predictions made by the model. It is the ratio of true positive predictions to the total positive predictions (including both true positives and false positives). In simpler terms, precision answers the question: "Of all the instances the model labeled as positive, how many are actually positive?"

High precision indicates that the model is reliable in its positive classifications, making few false positive errors.

Recall

Recall, also known as sensitivity or true positive rate, measures the model's ability to correctly identify all relevant instances. It is the ratio of true positive predictions to the total actual positives (sum of true positives and false negatives). Essentially, recall answers the question: "Of all the actual positive instances, how many did the model correctly identify?"

High recall indicates that the model is good at detecting positive instances, minimizing the number of false negatives.

Test

The Test tab is to allow users to directly interact with a deployed model to assess its inferencing capabilities on new, unseen data. This is crucial for understanding how the model performs outside a controlled training environment.

The Test tab is to allow users to directly upload images and click Test to start the model prediction process.

Testing a deployed model by uploading a file for inferencing involves evaluating the model's performance and behavior on new data that it hasn't seen during the training phase. This process helps verify that the model accurately interprets and analyzes the data, providing expected outcomes based on its learned patterns.

Why Test a Deployed Model?

Testing a deployed model is crucial for several reasons:

• Accuracy Verification: Ensures that the model performs as expected on real-world data.
• Performance Evaluation: Helps identify any latency or computational performance issues that may not have been evident during the development phase.
• Feedback Loop: Provides valuable information that can be used to further refine and improve the model.

Uploading a File for Inferencing: Users can upload a file that the model will use to make predictions. The file type and content depend on the model's purpose (for example, images for a vision model, text files for natural language processing, etc.).

Archive a Model

As you experiment and create a model version, choosing your path forward can sometime mean that some model versions are no longer required. Removing them from the model-versions list will simplify it and improve your ability to filter and search for the versions you really need.

1. Open the Models page.
2. In the Versions tab, find the model version you wish to use as a basis for a new version, and click on the 3-dots action button.
3. Select the Move to Archive option from the list. A confirmation message is displayed.
4. Click Yes. A confirmation message is displayed.

Restore an Archived Version

1. Open the Models page.
2. In the Versions tab, and click on the Archive button. The Archived Models page is displayed.
3. Review archive model versions, search and locate the model-version you wish to restore.
4. Click on the 3-dot action-button on the right-side, select the Restore option and confirm the operation in the pop-up window.
5. The model-version is now restored and can be found in the main Versions tab.