3. File Specification: Trial Schema (trial.jsonl)

This document details the JSON schema for defining individual experimental trials. In the directory structure, this file is named trial.jsonl.

Each JSON object represents one "trial" which can present one or more stimuli (e.g., text, image, or video) and then ask the user one or more questions. The trial.jsonl file contains one JSON object per line.

As shown in the image above, the interface is divided into two panels: the left panel displays the stimuli, and the right panel displays the queries. The trial object consists of two main arrays of objects that correspond to these components: the stimuli array for the left panel and the queries array for the right panel.

3.1. Top-Level Fields (Single Trial)

The root object of a single trial contains the following key properties:

3.1.1. Full Trial Object Example

Here is an example of a complete trial object that shows an image, followed by a short text blurb, then asks two questions: one multiple-choice and one slider.

{
  "stimuli_id": "trial_balls_01",
  "stimuli": [
    {
  "input_type": "img",
      "media_url": ["assets/balls.jpg"]
    },
    {
      "input_type": "text",
      "text": "Observe the agent's action in the image above."
    }
  ],
  "queries": [
    {
      "prompt": "Which ball appears bigger?",
      "type": "multi-choice",
      "tag": "ball_size",
      "option": [
        "Blue Ball",
        "Red Ball",
        "Equal Size"
      ],
      "randomize_order": false,
      "required": true
    },
    {
      "prompt": "How confident are you in your answer?",
      "type": "single-slider",
      "tag": "goal_confidence",
      "slider_config": {
        "min": 0,
        "max": 100,
        "default_value": 50,
        "labels": [
          { "value": 0, "label": "Not at all confident" },
          { "value": 100, "label": "Completely confident" }
        ],
        "show_label_values": true
      },
      "randomize_order": false,
      "required": true
    }
  ]
}

For an example with multiple media URLs and dimensions, see Appendix A.1.

Live Demo:

3.2. Query Types and Arguments

The queries array defines each prompt or question. All query objects include at least prompt, type, and optional required. The full argument list is specified per query type below.

3.2.1. text-instruction

Displays informative text only (no response is collected). Use this to insert explanatory copy between questions.

JSON Example:

{
  "prompt": "In the next questions, you'll evaluate the agent's beliefs.",
  "type": "text-instruction"
}

3.2.2. multiple-choice

Multiple-choice queries allow participants to select from a list of options. This query type has two variants: single-select (radio buttons) and multi-select (checkboxes).

3.2.2.1. single-select

Presents radio buttons; user selects exactly one choice.

JSON Example:

{
  "prompt": "What is the capital of France?",
  "type": "multi-choice",
  "tag": "capital_choice",
  "option": [
    "London",
    "Berlin",
    "Paris",
    "Madrid"
  ],
  "randomize_order": true,
  "required": true
}

Live Demo:

3.2.2.2. multi-select

Presents checkboxes; user can select one or more choices.

JSON Example:

{
  "prompt": "Which numbers are greater than 10?",
  "type": "multi-select",
  "tag": "numbers_gt_10",
  "option": [
    "5",
    "12",
    "8",
    "15",
    "3",
    "20"
  ],
  "randomize_order": false,
  "required": true
}

Live Demo:

3.2.3. slider

Single or multiple sliders for ratings.

3.2.3.1. single-slider

Presents one slider (no option list). Requires slider_config.

JSON Example:

{
  "prompt": "How confident are you?",
  "type": "single-slider",
  "tag": "confidence_rating",
  "slider_config": {
    "min": 0,
    "max": 100,
    "default_value": 50,
    "labels": [
      { "value": 0, "label": "Not at all" },
      { "value": 100, "label": "Completely" }
    ],
    "show_label_values": true
  },
  "required": true
}

Live Demo:

3.2.3.2. multi-slider

Presents one slider per item in option. Requires slider_config. Optionally show a random subset via sampling_size.

JSON Example:

{
  "prompt": "Please rate how likely each of the following statements is to be true",
  "type": "multi-slider",
  "tag": "statement_ratings",
  "option": [
    "I am more easily convinced by data and reasoning than by emotional appeals.",
    "I feel stressed when my environment is disorganized or messy.",
    "I usually think out loud and test ideas by talking them through."
  ],
  "slider_config": {
    "min": 0,
    "max": 100,
    "default_value": 50,
    "labels": [
      { "value": 0, "label": "Strongly disagree" },
      { "value": 50, "label": "Neutral" },
      { "value": 100, "label": "Strongly agree" }
    ],
    "show_label_values": true
  },
  "randomize_order": false,
  "required": true
}

Live Demo:

3.2.4. textbox

Provides a free-response text area for the user to type in an answer.

JSON Example:

{
  "prompt": "Please describe the agent's behavior.",
  "type": "textbox",
  "tag": "agent_description",
  "required": true
}

Live Demo:

3.2.5. ranking

Presents a list of items from the option array that the user can drag and drop to rank in order. The user must rank all items.

JSON Example:

{
  "prompt": "Rank the following goals in order of importance (drag to reorder):",
  "type": "ranking",
  "tag": "goal_ranking",
  "option": [
    "Finding the key",
    "Avoiding obstacles",
    "Reaching the exit",
    "Collecting coins"
  ],
  "randomize_order": true,
  "required": true
}

Live Demo: