ibl_bwb_eval#

Evaluation contract for the IBL BrainWideBench benchmark.

Everything an external consumer needs to produce a valid submission and to score one, and nothing else: the task vocabulary and readout specs, the reported metrics, the evaluation protocol (seeds, selection metric, eval sessions), the on-disk prediction format, the per-suite scorers and the cross-suite ranking.

Only the stdlib-only names are re-exported here, because importing any submodule runs this file first: ibl_bwb_eval.predictions.PredictionsWriter needs torch, so it is imported from its own module rather than from the package root.

Every readout name carries its suite, the way TS1Task and TS3Task do, so all three suites’ specs are exported side by side and no call site depends on context to say which one it meant.

This package imports nothing from core, pretrain or the task suites, and its dependency set is the scoring extra: numpy, scipy, scikit-learn, torch, torchmetrics, safetensors and rich. No torch_brain, hydra, ray or wandb. That isolation is enforced by tests/test_ibl_bwb_eval_isolation.py.

Tasks#

`TS1Task`	The eight scored TS1 tasks: three sequence-level, five timestep-level.
`TS2Task`	The two scored TS2 tasks, holding out units and trailing timesteps respectively.
`TS3Task`	The scored TS3 tasks, named `<entity>_<target>`: what is classified, at which atlas level.
`task_id`	The flattened task id a submission is filed under, e.g. `"ts2-co_smoothing"`.
`is_task_of`	Whether a flattened task id belongs to `suite`.
`get_ts1_supported_tasks`	Returns a list of all supported tasks.

Readout types#

The suite-neutral pieces: the slice a model sizes its readout head from, and the two enums a spec describes its target with.

`ReadoutSpec`	What a model needs to size a readout head, and nothing more.
`TargetResolution`	How many targets a task has along the input's time axis: one, or one per timestep.
`DataType`	What kind of value each target is, independent of `TargetResolution`.

TS1 readouts#

`TS1ReadoutSpec`	Specification for a single decoding readout head.
`get_ts1_readout_spec`	What `task` predicts, and the metrics it is scored on.

TS2 readouts#

TS2 sizes no readout head, so its spec names the axis the held-out mask varies along instead, and the metrics the task is scored with.

`TS2ReadoutSpec`	What one TS2 task predicts.
`get_ts2_readout_spec`	What `task` holds out, and the metrics it is scored on.

TS3 readouts#

`TS3ReadoutSpec`	What one TS3 task predicts.
`get_ts3_readout_spec`	What `task` classifies, its label vocabulary, and the metrics it is scored on.
`COSMOS_LABELS`	Built-in immutable sequence.
`check_ts3_label_order`	Raise unless `labels` is the task's vocabulary, in order.