RLHF JSON Format

This page describes RLHF JSON, a data format for describing RLHF data in JavaScript Object Notation (JSON).

For more information on RLHF Studio, see the RLHF Studio.

The RHLF Studio Item Format

It is built from two layers: a file layer and an annotation layer.
The file contains a list of prompts, and the annotations layer contains a list of responses per prompt.

File or Prompt Layer Format

The following format is a data structure example of a file that contains two prompts. The first prompt contains text and an image, and the second prompt contains text only.

{
	"shebang": "dataloop",
	"metadata": {
		"dltype": "prompt"
	},
	"prompts": {
		"prompt1": [
			{
				"mimetype": "application/text",
				"value": "What animal is in this image?"
			},
			{
				"mimetype": "image/jpeg",
				"value": "https://gate.dataloop.ai/api/v1/items/6489600c8d5a1c350e55116a/stream"
			}
		],
		"prompt2": [
			{
				"mimetype": "application/text",
				"value": "What is the eye color of this cat?"
			}
		
		]
	}
}

Annotations or Response Layer Format

Data structure of a response.

{
  "id": "64899218661aa36a1744112c",
  "datasetId": "64885c4d71e85c4f03c3758c",
  "url": "https://rc-gate.dataloop.ai/api/v1/items/64899218661aa36a1744112c",
  "dataset": "https://rc-gate.dataloop.ai/api/v1/datasets/64885c4d71e85c4f03c3758c",
  "createdAt": "2023-06-14T10:10:32.863Z",
  "dir": "/image_prompts",
  "filename": "/image_prompts/flyingdog.json",
  "type": "file",
  "hidden": false,
  "metadata": {
    "system": {
      "encoding": "7bit",
      "isBinary": false,
      "mimetype": "application/json",
      "originalname": "flyingdog.json",
      "refs": [],
      "shebang": {
        "dltype": "prompt"
      },
      "size": 161,
      "taskStatusLog": []
    }
  },
  "name": "flyingdog.json",
  "creator": "lior@dataloop.ai",
  "stream": "https://rc-gate.dataloop.ai/api/v1/items/64899218661aa36a1744112c/stream",
  "thumbnail": "https://rc-gate.dataloop.ai/api/v1/items/64899218661aa36a1744112c/thumbnail",
  "annotations": [
    {
      "id": "64899219bf9b191be0acdd2c",
      "datasetId": "64885c4d71e85c4f03c3758c",
      "itemId": "64899218661aa36a1744112c",
      "url": "https://rc-gate.dataloop.ai/api/v1/annotations/64899219bf9b191be0acdd2c",
      "item": "https://rc-gate.dataloop.ai/api/v1/items/64899218661aa36a1744112c",
      "dataset": "https://rc-gate.dataloop.ai/api/v1/datasets/64885c4d71e85c4f03c3758c",
      "type": "binary",
      "label": "q",
      "coordinates": "https://rc-gate.dataloop.ai/api/v1/items/648992159b8b5e823eda9972/stream",
      "metadata": {
        "system": {
          "automated": true,
          "promptId": "first"
        },
        "user": {
          "annotation_type": "prediction",
          "model": {
            "confidence": 0.9,
            "name": "model1"
          },
          "stream": true
        }
      },
      "creator": "lior@dataloop.ai",
      "createdAt": "2023-06-14T10:10:33.092Z",
      "updatedBy": "lior@dataloop.ai",
      "updatedAt": "2023-06-14T10:10:33.092Z",
      "hash": "64885c4d71e85c4f03c3758c_64899218661aa36a1744112c_q_lior@dataloop.ai",
      "source": "sdk"
    }
  ],
  "annotationsCount": 1,
  "annotated": true
}

RLHF JSON Fields Description