CodeBook Lab
CodeBook Lab is an LLM annotation pipeline for computational social science. It is designed to help researchers run, compare, and evaluate annotation experiments without writing custom annotation infrastructure for each new project.
Used alongside CodeBook Studio, it allows researchers to define an annotation task once and then use that same codebook plus a human-annotated ground-truth.csv to run local or HPC-based LLM workflows. Human annotations serve as the validation benchmark, and CodeBook Lab strips the codebook-defined label columns from ground-truth.csv before sending text to the LLM, making it possible to compare model outputs against a researcher-defined gold standard.
How Studio and Lab Fit Together
The relationship between the two tools is straightforward: CodeBook Studio is where the annotation task is defined, while CodeBook Lab is where the LLM experiment is run and evaluated.
CodeBook Studio
- Define the annotation task
- Annotate texts with humans
- Export
codebook.json
CodeBook Lab
- Run LLM annotation experiments
- Compare models, prompts, and settings
- Evaluate outputs against human labels
codebook.json and a human-annotated ground-truth.csv
Accuracy, agreement, runtime, energy, and emissions tradeoffs
CodeBook Studio defines the task; CodeBook Lab runs and evaluates the experiment.
CodeBook Lab supports the main annotation types commonly used in text-as-data workflows:
- Binary labels
- Categorical labels
- Likert-scale ordinal labels
- Open-ended text responses
The pipeline is designed to make it easier to compare tradeoffs across:
- Models available through Ollama
- Prompt styles and example use
- Hyperparameter settings such as temperature and top-p
- Runtime, energy use, and estimated carbon emissions
The repository includes a synthetic starter task and a configuration-driven workflow so that most experiments can be controlled through a single param_grid.yaml file rather than by editing pipeline code.