score_dir

ibl_bwb_eval.scoring.ts3.score_dir(pred_dir, gt_dir)

Score all prediction files found under pred_dir.

Expects prediction files written by PredictionsWriter (metadata carries label, task, seed). pred_dir may point at any subtree.

Ground truth files are looked up as: {gt_dir}/{task}/ground_truth.safetensors. Prediction files whose task doesn’t start with ts3- (e.g. other task suites picked up under a shared pred_dir) are skipped silently. Among ts3- tasks, one that doesn’t resolve to an existing ground truth file is skipped with a warning.

Returns a dict keyed by (label, seed) mapping metric name to scalar value.

Return type:

dict[tuple[str, int], dict[str, float]]