RLHF JSON Format

Updated On 03 Jun 2024

Print
Dark
Light
PDF

Article summary

Did you find this summary helpful?

Thank you for your feedback

This page describes RLHF JSON, a data format for describing RLHF data in JavaScript Object Notation (JSON).

For more information on RLHF Studio, see the RLHF Studio.

The RHLF Studio Item Format

It is built from two layers: a file layer and an annotation layer.
The file contains a list of prompts, and the annotations layer contains a list of responses per prompt.

File or Prompt Layer Format

The following format is a data structure example of a file that contains two prompts. The first prompt contains text and an image, and the second prompt contains text only.

{
	"shebang": "dataloop",
	"metadata": {
		"dltype": "prompt"
	},
	"prompts": {
		"prompt1": [
			{
				"mimetype": "application/text",
				"value": "What animal is in this image?"
			},
			{
				"mimetype": "image/jpeg",
				"value": "https://gate.dataloop.ai/api/v1/items/6489600c8d5a1c350e55116a/stream"
			}
		],
		"prompt2": [
			{
				"mimetype": "application/text",
				"value": "What is the eye color of this cat?"
			}
		
		]
	}
}

Annotations or Response Layer Format

Data structure of a response.

{
  "id": "64899218661aa36a1744112c",
  "datasetId": "64885c4d71e85c4f03c3758c",
  "url": "https://rc-gate.dataloop.ai/api/v1/items/64899218661aa36a1744112c",
  "dataset": "https://rc-gate.dataloop.ai/api/v1/datasets/64885c4d71e85c4f03c3758c",
  "createdAt": "2023-06-14T10:10:32.863Z",
  "dir": "/image_prompts",
  "filename": "/image_prompts/flyingdog.json",
  "type": "file",
  "hidden": false,
  "metadata": {
    "system": {
      "encoding": "7bit",
      "isBinary": false,
      "mimetype": "application/json",
      "originalname": "flyingdog.json",
      "refs": [],
      "shebang": {
        "dltype": "prompt"
      },
      "size": 161,
      "taskStatusLog": []
    }
  },
  "name": "flyingdog.json",
  "creator": "lior@dataloop.ai",
  "stream": "https://rc-gate.dataloop.ai/api/v1/items/64899218661aa36a1744112c/stream",
  "thumbnail": "https://rc-gate.dataloop.ai/api/v1/items/64899218661aa36a1744112c/thumbnail",
  "annotations": [
    {
      "id": "64899219bf9b191be0acdd2c",
      "datasetId": "64885c4d71e85c4f03c3758c",
      "itemId": "64899218661aa36a1744112c",
      "url": "https://rc-gate.dataloop.ai/api/v1/annotations/64899219bf9b191be0acdd2c",
      "item": "https://rc-gate.dataloop.ai/api/v1/items/64899218661aa36a1744112c",
      "dataset": "https://rc-gate.dataloop.ai/api/v1/datasets/64885c4d71e85c4f03c3758c",
      "type": "binary",
      "label": "q",
      "coordinates": "https://rc-gate.dataloop.ai/api/v1/items/648992159b8b5e823eda9972/stream",
      "metadata": {
        "system": {
          "automated": true,
          "promptId": "first"
        },
        "user": {
          "annotation_type": "prediction",
          "model": {
            "confidence": 0.9,
            "name": "model1"
          },
          "stream": true
        }
      },
      "creator": "lior@dataloop.ai",
      "createdAt": "2023-06-14T10:10:33.092Z",
      "updatedBy": "lior@dataloop.ai",
      "updatedAt": "2023-06-14T10:10:33.092Z",
      "hash": "64885c4d71e85c4f03c3758c_64899218661aa36a1744112c_q_lior@dataloop.ai",
      "source": "sdk"
    }
  ],
  "annotationsCount": 1,
  "annotated": true
}

RLHF JSON Fields Description

Key Name	Definition	Parent Key
id	Annotation Id	annotations
datasetId	Dataset Id	annotations
url	API URL of the annotation	annotations
dataset	API URL of the dataset	annotations
createdAt	Date and time when annotation was created	annotations
dir	The folder location where the JSON file is available	annotations
filename	Name of the file	annotations
type	Annotation type	annotations
hidden	Whether the annotation is hidden or not	annotations
metadata	Information about the annotation	annotations
system	Annotation system information	metadata
encoding		system
mimetype	The mime type of the file	system
originalname	The filename of the JSON	system
refs	The references of the task	system
shebang dltype	It allows UI to present the annotation as LiDAR instead of a JSON type.	system
size	File size	system
taskStatusLog	Logs from the task.	system
name	Name of the file	annotations
creator	User who created the annotation	annotations
stream	The URL pointing to the item stream	annotations
thumbnail	The URL pointing to the item thumbnail	annotations
item	The item ID	annotations
label	the label ID	annotations
coordinates	Annotation position coordinates	annotations
automated	True = Annotation created by automation process	system
promptId	An unique value to represent the response	system
user	The user who added the metadata via SDK	metadata
annotation_type	The type of annotation	user
confidence	Model confidence	model
name	name of the model	model
stream	Whether it is a stream data or not	user
creator	User who created the annotation	annotation
createdAt	Date and time when annotation was created	annotations
updatedBy	User who updated the annotation	annotations
updatedAt	Date and time when annotation was last updated	annotations
hash	Used to map annotation’s data to a string	annotations
source	Where the annotation was created: UI/SDK	annotations
annotationCount	Number of annotation	annotations

What's Next

Recipe JSON Format

Table Of Contents

The RHLF Studio Item Format
RLHF JSON Fields Description