CodeBook Lab

Python Package Documentation

CodeBook Lab is a Python package for running validation-first LLM annotation experiments against human-coded benchmark data. It takes a codebook.json and labelled ground-truth.csv exported from CodeBook Studio, removes human labels before annotation, and scores model outputs against the held-out labels.

Researchers can evaluate local Ollama models systematically across the dimensions that matter for text-as-data research:

Model choice and model size: compare any locally available Ollama models
Prompt style and example use: compare standard, persona, and chain-of-thought wrappers with or without codebook examples
Chat and reasoning modes: test fresh chats per query, per-text chats, continuous chats, and model reasoning settings
Sampling settings: sweep over temperature, top-p, and related generation parameters

Experiments are controlled from Python, so users can start with a single run, scale up to multiple experiment specifications, or iterate through larger parameter grids when needed. Because the codebook and labelled data stay constant across runs, each configuration can be compared against the same human labels.

CodeBook Studio

Define the annotation task
Annotate texts with humans
Export codebook.json and ground-truth.csv

CodeBook Lab

Run LLM annotation experiments
Isolate model, prompt, chat, reasoning, and sampling effects
Score outputs against human labels

Shared inputs
codebook.json + human-annotated ground-truth.csv

Experiment outputs
Accuracy, agreement, runtime, energy, and emissions metrics

CodeBook Lab also supports human-reliability workflows: it can validate coder CSVs, calculate inter-coder agreement, identify unresolved disagreements, and build an adjudication queue that can be completed in CodeBook Studio. Final outputs include run-level annotation quality, runtime, prompt and response length, energy consumption, and estimated carbon emissions, with aggregate metrics logged for cross-experiment comparison.

The package ships with a bundled starter task so you can test the workflow before plugging in your own data. For installation, experiments, human reliability, and the full API, see the CodeBook Lab documentation. Citation details for both CodeBook Lab and CodeBook Studio are available on the CodeBook citation page.

Full package documentation is available at lorcanmclaren.com/codebook-lab.