rank#

ibl_bwb_eval.scoring.aggregation.rank(summary, primary_metric, alpha=0.05, method='step_down')[source]#

Competition ranking, averaged over recording_id within each task.

For each (task, recording_id), ranks labels by primary_metric[task]. With method="step_down" (default), a candidate is only promoted past the current anchor if a one-sided Welch’s t-test finds it significantly better at alpha. Ties otherwise. With method="naive", ranking is by sorted mean alone, no significance test, ties only on exact equality. Ranks are then averaged over recording_id within each task. For TS3, whose rows all share a single recording_id sentinel, that average is a no-op over one value.

Warns (UserWarning) per task where some recording_id has fewer than 2 labels scored on it (no comparison to make there, so that recording’s rank is trivial). This typically means the labels being ranked weren’t evaluated on the same sessions.

Parameters:

  • summary (dict[tuple[str, str, str], dict[str, tuple[float, float | None, int]]]) – output of aggregate().

  • primary_metric (Mapping[str, str]) – task id -> metric name to rank that task on (e.g. from each suite’s own readout spec / metric convention).

  • alpha (float) – significance threshold for the pairwise comparisons, used only when method="step_down".

  • method (Literal['step_down', 'naive']) – "step_down" or "naive", see above.

Return type:

dict[str, dict[str, float]]

Returns:

Dict keyed by task, mapping label -> avg rank for that task. This is the row-oriented dict-of-dicts shape pd.DataFrame(result) expects directly (columns=task, index=label, NaN where a label has no score for a task); overall rank across tasks is then a simple .mean(axis=1) on that frame.

On this page