Several organizations are thinking about the right way to regulate AI and the idea of assurance labs which would test and validate AI solutions in the US healthcare is taking shape. This was the topic we discussed with Brian Anderson - CEO of the coalition for Health AI or CHAI: how will assurance labs look like, how much will assessments cost, who will pay for them, and how will AI “nutrition labels” look like.
Summary:
Assurance Labs in Healthcare AI
The Coalition for Health AI (CHI) is developing a network of quality assurance labs to evaluate AI models in healthcare.
These labs aim to provide independent, transparent assessments of AI models' performance across different populations.
By the end of 2024, CHI plans to have two certified labs operational, with more to follow in 2025.
Model Cards and Evaluation
CHAI has introduced "model cards" or "nutrition labels" for AI models, describing their training data, methodology, indications, and limitations.
Model cards are created by developers, while assurance labs provide independent evaluation reports.
CHAI is working on technical specifications for model cards to ensure consistency and transparency.
Goals and Benefits
Assurance labs aim to balance innovation with safety in AI development.
They can help identify model performance issues across different populations and accelerate improvements.
The process is intended to build trust in AI among healthcare providers and patients.
Implementation and Challenges
CHAI is creating a competitive marketplace of quality assurance labs to keep costs reasonable.
Labs must be free from conflicts of interest with AI vendors.
Evaluation reports will be published in a public registry for transparency.
The cost of evaluations is expected to be in the range of thousands of dollars, not millions.
Future Plans
CHAI is exploring partnerships with health systems and NGOs to establish quality assurance labs in the EU.
The initiative aims to be scalable and adaptable to different geographic regions and populations.
www.facesofdigitalhealth.com
Newsletter: https://fodh.substack.com/
