Burna AI measures inter-rater reliability from the signed blind-grading submissions themselves. It computes weighted Cohen's kappa, a binary kappa at the grade 3 threshold, agreement matrices that show where raters diverge grade by grade, each rater's agreement with the reference, how well the AI suggestion's stated confidence matches how often it agrees with the reference, grade distributions, and the rate at which clinicians override the AI suggestion, week by week. A reference grade comes from established ground truth, an adjudicator's ruling or two raters who agree. An event only one rater graded is left out rather than scored as a miss, and because the figures come from the signed source of record rather than analytics telemetry, a reviewer can trace each one back to the submissions behind it.
Interactive demo
Try Burna right now, no sign up required.