Evaluating AI adverse event grading software: four questions about the architecture

Adverse event grading software reads clinical documentation and proposes a CTCAE grade, and usually an attribution, for each adverse event in a trial. To evaluate it, test four parts of the architecture: whether every grade cites the source text and criterion behind it, whether the output stays inside the CTCAE version the protocol names, how causality is scored, and what signed record remains after a clinician reviews it. Agreement data and a security review come next.

A demonstration on a clean note shows that a system can produce a grade, not what it does with the notes that generate queries: the ambiguous negation, the lab value with no baseline, the symptom copied forward from a visit three weeks earlier. Architecture decides that, and it changes far less often than a feature list.

Four questions for evaluating adverse event grading software

Question	What a strong answer shows	How to test it
1. Can a clinician independently review the basis for every grade?	The CTCAE criterion and the exact line in the record, attached to the grade	Open any grade and trace it to the note without asking the vendor
2. Is the output constrained, and what happens when the record does not support a grade?	Grades drawn only from the CTCAE version the protocol names; thin evidence returns "cannot grade" with the gap named	Run a deliberately incomplete note
3. How is attribution decided, and can a medical monitor check the working?	Named causality methods applied per suspect drug, with the inputs visible	Run a combination-regimen case and read the working
4. What record survives until an inspection?	A signed record: who signed, when, what the signature means, what was accepted or overridden, linked to its evidence	Ask for a grade signed last month, with its history

A worked example: one note through the four questions

A patient on an investigational agent plus capecitabine is seen on day 12 of cycle 2. The note reads: "Nine loose stools yesterday. Tolerating oral fluids. Not admitted." It records no usual stool frequency and nothing about daily activities.

• Basis. A strong answer shows that sentence and the full criterion, because stool count is one of several descriptors at each grade. Under CTCAE v6.0, Diarrhea Grade 2 includes "Increase of 4 - 6 stools per day over baseline" and "limiting instrumental ADL". Grade 3 lists "Increase of >=7 stools per day over baseline", "hospitalization indicated", "severe increase in ostomy output compared to baseline", "requires IV intervention" and "limiting self-care ADL".
• Constraint. Nine stools a day is Grade 3 if the patient's baseline was two or fewer, Grade 2 if it was three to five, and Grade 1 if it was six to eight. "Not admitted" and "Tolerating oral fluids" weigh against the hospitalization and IV descriptors but say nothing about daily activities, so a strong answer returns "cannot grade" and names both gaps: usual stool frequency and ADL impact. The version decides what the record cites: v6.0 Grade 1 reads "Change in consistency or frequency" where v5.0 reads "Increase of <4 stools per day over baseline", and only v6.0 Grade 3 includes "requires IV intervention".
• Attribution. Each drug gets its own assessment. The capecitabine label lists diarrhea under Warnings and Precautions, which counts as previous experience with that drug; the investigational agent has far less. Day 12 is read against each drug's dosing schedule, and infection or a new concomitant medication is weighed as an alternative cause, with every input shown.
• Record. The signed record holds the sentence, criterion and version, the baseline the clinician supplied, the causality inputs, the suggestion and what the investigator signed.

Can a clinician independently review the basis for every grade?

This is a regulatory question as much as a usability one. Section 520(o)(1)(E) of the Federal Food, Drug, and Cosmetic Act sets four criteria that keep clinical decision support software outside the device definition. One is that the software enables the health care professional to "independently review the basis for such recommendations."

FDA lists the criteria in its Digital Health Policy Navigator and interprets them in its Clinical Decision Support Software guidance, revised in January 2026. What FDA's clinical decision support guidance asks of grading software takes them one at a time.

A paragraph of reasoning beside the grade can be plausible and still wrong. A citation attached to the grade, and checked for completeness before anyone signs, cannot exist without the text it points to.

How Burna AI answers. Every grade traces to the line in the medical record that produced it, and a grade whose citation set is incomplete is flagged before it is signed. The public record of what Burna AI does today lists this capability.

Is the output constrained when the record does not support a grade?

Constraint has two parts, and the first is the version. NCI released CTCAE v5.0 in 2017 and v6.0 in 2025, and the worked example shows how far the cited text can move between them. A site can carry trials on both versions at once, so the software should grade against the version the protocol names, and nothing else. ctcae.ai lists every CTCAE v6.0 change, term by term, and running both CTCAE versions side by side covers the site's side.

The second is the boundary. When a finding lacks the qualifier the criterion requires, or a lab value lacks its baseline, the system either estimates a grade, or declines and names what is missing. Know which you are buying before the first monitoring visit.

How Burna AI answers. For each term, the engine retrieves the governing criterion from CTCAE v5.0 or v6.0, matched to the study's version. Where the evidence runs thin, it returns a "cannot grade" state and names the gap. CTCAE grading in Burna AI shows each step from note to signed grade.

How is attribution decided, and can a medical monitor check the working?

Attribution judges how likely it is that each drug in the regimen caused an event, and for a sponsor it is where the label starts. FDA's labeling rule counts an adverse event as an adverse reaction only where "there is some basis to believe there is a causal relationship" with the drug, and those reactions fill the Warnings and Precautions and Adverse Reactions sections (21 CFR 201.57(c)(6) and (c)(7)). How attribution shapes the drug label follows that path.

The software should apply named, published causality methods to each suspect drug in a combination regimen, rather than to the regimen as a whole. Three methods are common (the Naranjo scale, WHO-UMC and Kramer compared), and each works differently:

• The Naranjo scale (Clinical Pharmacology and Therapeutics, 1981) asks ten weighted questions and sums them into definite, probable, possible or doubtful.
• The WHO-UMC system sorts a case into categories from certain and probable through possible and unlikely to conditional and unassessable, weighing time relationship, other explanations, response to withdrawal and rechallenge (Uppsala Monitoring Centre).
• The Kramer algorithm (JAMA, 1979) scores six axes: previous experience with the drug, alternative causes, timing, drug levels and evidence of overdose, dechallenge, and rechallenge. The sum rates the case definite, probable, possible or unlikely.

A monitor who sees the inputs behind a score can disagree with one step; a monitor shown only a label can only accept or reject it.

How Burna AI answers. Each event is scored for attribution against every drug in the regimen using WHO-UMC and Kramer, with the reasoning shown and comorbidities and concomitant medications carried into the scoring. The investigator sees the reasoning and decides.

What record survives until an inspection?

A grade is defended later, in front of someone who was not there. An investigator keeps study records for 2 years after a marketing application is approved for the indication or, if no application is filed or it is not approved for that indication, until 2 years after the investigation is discontinued and FDA is notified (21 CFR 312.62(c)). What a BIMO inspection asks of adverse event records covers that visit.

21 CFR Part 11 makes the record specific:

• 11.10(e): secure, computer-generated, time-stamped audit trails of actions that create, modify or delete records.
• 11.50: the signer's printed name, the date and time, and the meaning of the signature.
• 11.70: signatures linked to their records so they cannot be excised, copied or transferred.

The test is whether the clinician's approval is an electronic signature with those attributes, or an on-screen acknowledgement that resembles one. A Part 11 checklist for AI-assisted grading goes requirement by requirement.

How Burna AI answers. A grade becomes a record when a qualified investigator signs it with a 21 CFR Part 11 electronic signature, which carries what was accepted, what was overridden and the citation set as signed. The audit trail records what the AI suggested and what the human decided.

How should you judge an AI grading vendor's agreement claims?

Careful human reviewers vary, so an agreement figure needs a baseline. Two reviewers identifying CTCAE toxicities in the same 100 on-treatment radiation therapy notes agreed at an unweighted kappa of 0.68 and a weighted kappa of 0.59 (Fairchild et al., International Journal of Radiation Oncology, Biology, Physics, 2020). Attribution is less stable still: in the placebo arms of two phase III trials, when the same adverse event was reported for the same patient at several visits, its attribution category changed at least once 36% and 31% of the time (Hillman et al., Journal of Clinical Oncology, 2010). Ask for the reference panel's own inter-rater agreement, so the software's figure can be read against how often two experts agree.

The reference standard comes next: who graded, how disagreements were adjudicated, whether the comparison was blinded, which cases and which statistic, and where the results are published. Agreement matters most between Grade 2 and Grade 3, where CTCAE moves to "severe or medically significant but not immediately life-threatening". That is a severity line, separate from regulatory seriousness, as serious versus severe adverse events explains.

FDA's January 2025 draft guidance on AI in drug regulatory decision-making assesses model credibility for a stated context of use. AI-assisted grades become safety data behind regulatory decisions, so a vendor should state its own.

Then run your own comparison:

1. Pull de-identified notes from your own trials, weighted toward hard cases: ambiguous negations, syndrome-level grading and thin evidence.
2. Have two of your own graders grade them blind, and adjudicate their disagreements to set a reference.
3. Run the software on the same notes, scoring the Grade 2 to Grade 3 line separately.
4. Put every disagreement through the four questions: citation, version, attribution working, signed record.

How Burna AI answers. Burna AI is in ongoing internal testing demonstrating strong agreement with expert clinicians. The testing uses blind, adjudicated comparison against expert clinician panels on synthetic and de-identified cases, scored with weighted kappa and per-grade agreement, including the Grade 3+ boundary, and the validation methodology is published in full. A concordance study is running across three campuses against an adjudicated two-clinician reference panel, on 1,600 cases drawn from a de-identified dataset of 3 million patients, and reads out in December 2026. Burna AI is a Mayo Clinic Platform_Accelerate participant.

Which security and integration questions should you send an AI grading vendor?

Send these six questions with your security questionnaire:

1. Which model providers process our data, and will we be told before that changes?
2. Where does protected health information live, and how is it separated from identity, billing, analytics and other customers?
3. Who can reach a record, and is every access logged?
4. What is retained, for how long, and how is it deleted?
5. Can instructions hidden in an uploaded document change what the model does?
6. What independent security assessments exist, and when did they run?

Add the integration questions alongside them: which systems the software reads notes from, and whether its output reaches your EDC with the MedDRA term and version recorded (moving adverse events from the EHR to the EDC).

How Burna AI answers. Protected health information (PHI) lives in an isolated data plane, separate from identity and billing, with access audited row by row and business associate agreements signed. Each subprocessor is published on the Burna AI subprocessor list, with its purpose and whether it can reach PHI, and new engagements carry 30 days' advance notice. Our SOC 2 Type 2 audit is in progress. The architecture a security review asks about covers SSO and SAML sign-in, role-based access and organization scoping.

Common questions about adverse event grading software

Is AI adverse event grading software a medical device?

It depends on intended use and the four criteria in section 520(o)(1)(E): software that meets all four, including independent review of the basis for its output, falls outside the device definition. Burna AI is built as non-device clinical decision support. Burna AI has submitted a Q-Submission to FDA, and its application to the CDER Emerging Drug Safety Technology Program (EDSTP) is in progress. The determination rests with FDA; is Burna AI a medical device has the short answer.

Can a general-purpose chatbot grade CTCAE adverse events?

A general-purpose chatbot can return a CTCAE grade, and the same four questions apply to it. A pasted note usually arrives without the protocol's CTCAE version, the patient's baseline or the regimen, and a chat reply is not a signed record. ctcae.ai sets out how to check an AI-suggested CTCAE grade.

Run the four questions on your own notes

Bring a de-identified note from your own trial and watch it graded, cited, attributed and signed. AI suggests, clinicians decide. Book a 15-minute conversation.