2. File Specification: Config Schema (config.json)

The config.json file provides high-level metadata about the experiment and defines the overall structure (or "flow") of the trials presented to the participant.

2.1. Metadata

These top-level keys describe the study and are used for dataset indexing and citation.

Key Type Description
experimentName String The full title of the experiment or study.
description String A brief paragraph describing the task.
paperDOI String The DOI link to the original publication (e.g., "https://doi.org/...").
taskType Array of Strings Keywords describing the cognitive task (e.g., "Social Cognition", "Decision Making").
responseType Array of Strings All response types used, e.g., ["multi-choice", "slider", "free-text"].
contributors Array of Strings List of researcher names who prepared the data.
stimuli_count Integer Total number of unique stimuli/trials defined for this experiment (i.e., the number of entries in trial.jsonl with id values that are referenced by config.json).

The stimuli_count field provides a quick summary of dataset size. Participant demographics (participants_info) and the total number of ratings (judgment_count) are now specified in the Human Data files; see the Human Data Schema.

2.2. Experiment Flow (experimentFlow)

To control the order and grouping of multiple trials, an experimentFlow array is used. Each entry in this array is a dictionary describing a full flow definition (e.g., for a specific experimental condition). The list should enumerate every condition included in the experiment. If conditions differ so substantially that they require separate instructions, assets, or config logic, consider splitting them into separate experiment folders instead. Make this decision on a case-by-case basis.

Key Type Description
blocks Array of Arrays of Strings A nested array. Each inner array is a "block" of experiment components. Each string is an id that can reference:
  • A trial from trial.jsonl (the id field of a trial object)
  • An instruction module from instruction.jsonl (the id field of an instruction module)
  • A test trial module from instruction.jsonl (the id field of a test_trial module)
  • A comprehension quiz module from instruction.jsonl (the id field of a comprehension_quiz module)
To include instructions, test trials, or quizzes in the experiment, their id values must be specified in the blocks array.
block_randomization Array of Booleans (optional) An optional array of booleans, where each boolean corresponds to a block in the blocks array. If true for a block, that block's order can be randomized relative to other blocks. If not specified, defaults to all false (no randomization). The length of this array should match the number of blocks.
experimental_condition String (optional) An optional label identifying the experimental condition associated with this flow definition.

2.2.1. Example experimentFlow

"experimentFlow": [
  {
    "experimental_condition": "condition_1",
    "blocks": [
      [ "instruction_01", "test_trial_01"],
      [ "trial_1_1", "trial_1_2", "trial_1_3" ]
    ],
    "block_randomization": [false, true],
  },
  {
    "experimental_condition": "condition_2",
    "blocks": [
      [ "instruction_02", "test_trial_02"],
      [ "trial_2_1", "trial_2_2", "trial_2_3" ]
    ],
    "block_randomization": [false, true],
  }
]

In this example:

  • There are two experimental conditions. The interface will randomly present one of the two conditions to the participant. You do not need to specify the experimental condition field if there is only one condition.
  • Within the first experimental condition, there are two blocks. The first block includes an instruction module ("instruction_01") and a test trial ("test_trial_01"). The second block includes three regular trials ("trial_1_1", "trial_1_2", "trial_1_3").
  • The block_randomization array is [false, true], meaning the first block (index 0) will not be randomized, while the second block (index 1) can be randomized relative to other blocks. If block_randomization is not specified, it defaults to all false (no randomization).
  • All IDs (for instructions, test trials, quizzes, and regular trials) must match the id fields defined in their respective files (instruction.jsonl or trial.jsonl).

2.3. Full config.json Example

Here is what a complete config.json file looks like, combining the metadata and the experiment flow. The id strings in the blocks array can refer to:

All IDs must match the id fields defined in their respective source files.

{
  "experimentName": "Epistemic Language Understanding",
  "description": "Participants observe short animations and answer questions about the agent's goals and beliefs.",
  "paperDOI": "https://doi.org/10.1111/example.doi.12345",
  "taskType": [
    "Social Cognition", 
    "Theory of Mind"
  ],
  "responseType": [
    "multi-choice",
    "single-slider"
  ],
  "contributors": [
    "Jane Doe",
    "John Smith"
  ],
  "stimuli_count": 48,
  "experimentFlow": [
    {
      "blocks": [
        [ "instruction_01", "test_trial_01", "trial_1_1", "trial_1_2", "trial_1_3" ],
        [ "trial_2_1", "trial_2_2", "trial_2_3", "trial_2_4", "comprehension_quiz_01" ]
      ],
      "block_randomization": [false, true]
    }
  ]
}