AI in insurance: admissibility, accountability, and legal integrity
As Artificial Intelligence (AI) becomes increasingly integrated into the insurance claims process, a key question emerges: can AI-generated insights or outputs stand up in court?
While the term “AI evidence” is frequently used in industry discussions, legal experts caution against oversimplification. Tristan Marot, Senior Associate at Norton Rose Fulbright, offers crucial insight into the real legal considerations for insurers navigating this evolving terrain.
“Courts don’t admit ‘AI’,” says Marot. “They admit reliable, relevant, fairly obtained evidence.” He notes that AI is better viewed as a tool to organise information or derive insights, not a standalone source of admissible evidence. “AI outputs either help point to primary records (telematics, photos, logs) or amount to analysis/opinion that a human expert must then adopt, explain, and quantify.”
Building a legally defensible AI framework
To ensure that AI-enhanced claims decisions can hold up under legal scrutiny, insurers need to embed evidentiary safeguards into their operating models from the outset.
“Insurers need to build the evidentiary foundations into the operating model, to ensure provenance and chain of custody are maintained within claims systems, treat AI flags as investigative leads to be corroborated with primary records, and keep a human decision maker who records reasons,” explains Marot.
He adds that insurers should also maintain “a compact model governance pack for each tool (purpose, validation/error rates, limits, monitoring, and change logs),” and ensure “authenticity through hashing and immutable audit trails.”
Legal challenges include issues such as hearsay, data integrity, explainability, and potential bias. But these are not new, Marot says - they're just being applied to modern tools.
“The main legal challenges are the familiar ones applied to a new toolset: hearsay when a model synthesises third-party data; authenticity and integrity of electronic records under ECTA; explainability for complex models; privilege over development materials; and vendor or outsourcing control.”
To mitigate these, Marot recommends that “where inferential analysis is relied upon, [insurers should] route it through a human expert who adopts the method and can speak to error rates and limitations.” He further suggests demonstrating integrity with “standard system evidence, hashes, metadata and SOPs,” and ensuring contractual rights around “audit, cooperation and incident reporting” with third-party providers.
Transparency through governance, not litigation
Transparency and auditability in AI systems begin long before a legal dispute arises. “Transparency and auditability are achieved less through disclosures at litigation and rather through disciplined housekeeping,” Marot asserts.
He recommends the use of model cards at the development stage that clearly articulate “purpose, inputs, training sources, performance and limits,” and stresses the importance of versioning, reproducibility, and logging.
“For higher impact systems, it is prudent to periodically review the system by an internal audit function or an independent assessor to attest that controls operate as described.”
On regulatory compliance, Marot underscores the importance of POPIA compliance - particularly section 71, which governs automated decisions - and the need for a board-approved AI governance policy. This should include “ownership and reporting to risk/compliance, prudential outsourcing oversight... and an auditable controls library aligned to recognised norms.”
Maintaining fairness requires an end-to-end approach: “Has the organisation completed an AI or data protection impact assessment, do they test for disparate error or impact pre-launch and on a monitoring cadence, what thresholds and remediation triggers are in place, and what is the procedure to escalate edge cases or sensitive matters for senior human review?”
Preparing for the legal landscape ahead
Despite the rapid adoption of AI tools, Marot doesn’t expect radical changes to the law of evidence in response.
“I don’t foresee there being major changes to the Law of Evidence in light of AI. It will always remain a question of provenance... and what is the probative value of the evidence to the court.”
However, that doesn’t mean insurers can afford to be passive. “Insurers need to ensure that as they adopt new technologies and modernise processes, they plan to be in the best position as possible to adduce the evidence they rely upon to courts.”
He lists several best practices to ensure defensibility:
- Be crystal clear on each use case’s purpose;
- Preserve end-to-end data lineage;
- Keep a human in the loop and require written reasons for adverse outcomes;
- Validate performance, monitor for drift and act on thresholds;
- Test and document fairness;
- Enforce change control and versioning;
- Secure vendor audit and litigation cooperation rights; and
- Maintain POPIA-aligned privacy and ISO grade security with incident readiness.
Ultimately, Marot emphasises a principle that cuts through the hype and complexity: “The real evidence contest in court is about provenance, method and fairness, not the marketing label of the tool.”
Writer’s Thoughts
As AI continues to shape the future of insurance claims, the key takeaway for insurers is that integrating robust legal safeguards from the outset is crucial for defensible claims decisions. By ensuring transparency, accountability, and human oversight, insurers can position themselves to meet both legal and regulatory challenges, ultimately safeguarding the integrity of their claims processes. Do you agree? Please comment below, interact with us on Twitter at @fanews_online or email me your thoughts.
