#464: Why LLM Unpredictability is a Liability in MedTech - Global Medical Device Podcast powered by Greenlight Guru | Lyssna här

Artificial intelligence has officially entered the mainstream cultural zeitgeist, creating a wave of excitement—and a fair share of fatigue—across the medical device industry. In this episode, host Etienne Nichols sits down with Tyler Harmon, biomedical engineer and CEO of Iaso Automated Medical Systems, to cut through the marketing buzzwords. Together, they explore the technical realities behind the technology stack, shifting the conversation away from generic AI toward specific, actionable engineering frameworks.

The discussion highlights a critical distinction between traditional machine learning models and consumer-oriented Large Language Models (LLMs). Harmon explains that while technologies like convolutional neural networks (CNNs) have successfully processed medical imaging for years, modern LLMs introduce an intentional element of randomness to mimic human conversation. This lack of predictability presents unique challenges for medical device developers who operate in a deterministic, safety-critical environment where reproducibility is paramount.

Looking toward practical deployment, the episode addresses how companies can responsibly govern these tools both within their software architectures and their internal Quality Management Systems (QMS). From classifying external AI models as Software of Unknown Provenance (SOUP) under IEC 62304 to leveraging machine learning for early detection of Acute Respiratory Distress Syndrome (ARDS) in the ICU, this conversation serves as an essential guide for innovators looking to build the next generation of safe, compliant, and effective medical technologies.

Key Timestamps

00:05 – Introduction to the dual nature of AI in MedTech: embedded clinical algorithms versus internal process optimization.
02:14 – Demystifying the math: Breaking down artificial intelligence into linear and non-linear algorithmic transformations.
04:30 – The Turing Test, Markov chains, and why consumer LLMs are mathematically designed to be unpredictable.
07:15 – Real-world success stories: How convolutional neural networks (CNNs) revolutionized emergency stroke triage.
09:42 – Inside Iaso Automated Medical Systems: Using non-LLM machine learning to identify Acute Respiratory Distress Syndrome (ARDS) in critical care.
12:10 – AI Governance in the QMS: Designing specialized Standard Operating Procedures (SOPs) and Machine Learning Management Systems (AIMS).
15:35 – Evaluating recent FDA 510(k) clearances for LLM-adjacent technologies and managing third-party stacks as SOUP.

Quotes"If we as innovators can't explain things to a more general audience, we generally don't understand them ourselves. And if you can't do that, it's probably not the best idea to be implementing it into your products." - Tyler Harmon"I am probably going to be the biggest advocate you'll ever talk to about 'doctors need enablement, not replacement.' We need to give them the tools, the force multipliers to tackle the challenges they're going to face this century." - Tyler HarmonTakeaways

Classify External AI as SOUP: Treat third-party language models and external tech stacks as Software of Unknown Provenance (SOUP) under IEC 62304 frameworks, implementing rigorous risk management boundaries to isolate the core medical device logic.
Engineer Out Randomness: Recognize that consumer LLMs purposefully integrate randomness layers to maximize user engagement. For clinical safety, developers must utilize architectural harnesses or alternative machine learning methods (like CNNs or random forests) to force more deterministic outcomes.
Establish an AI Management System: Expand your organizational compliance beyond standard Quality Management Systems (QMS) and Information Security Management Systems (ISMS). Implement specific AI standard operating procedures and work instructions to govern internal token usage and data handling.
Prioritize Clinical Enablement Over Automation: Focus clinical software engineering on clearing workflow bottlenecks and flagging early-stage critical conditions (such as ARDS) to allow bedside clinicians to deploy their hands-on expertise faster.

References

Berlin Criteria: The formal, quantitative medical classification standard used by clinicians to diagnose and grade the severity of Acute Respiratory Distress Syndrome.
IEC 62304: The international standard governing medical device software lifecycle processes, specifically detailing the management of Software of Unknown Provenance (SOUP).
Connect with Etienne Nichols on LinkedIn to stay updated on the latest episodes and industry insights.

MedTech 101 SectionUnderstanding Non-Linear Math and LLMs

Think of a traditional medical device software algorithm like a standard thermometer tracking a fever. It follows a straight, predictable line: if the temperature input increases by one degree, the reading on the screen changes by exactly one degree. This is a linear system.

Modern AI, like Large Language Models (LLMs), works more like a seasoned doctor trying to diagnose a complex case by listening to a patient's story. The human brain doesn't just look at variables in a straight line; it connects random pieces of past experiences, reads between the lines, and notes subtle shifts in tone. To replicate this mathematically, software engineers introduce non-linearity.

Instead of a straight line, the math behaves like a web of thousands of intersecting pathways. To make the system feel even more human, creators add a controlled "randomness layer" (similar to a digital coin flipper) so the software doesn't always choose the most obvious, predictable word next. While this makes chatting with a computer feel incredibly natural, it presents an engineering challenge for medical device developers who require identical, reproducible results every single time.

Feedback Call-to-Action

We love hearing from our community of MedTech professionals. Do you have thoughts on how AI governance should evolve, or is there a specific industry topic you want us to tackle next? We read every message and pride ourselves on providing personalized responses to our listeners. Share your feedback, reviews, and topic suggestions directly with our production team at podcast@greenlight.guru.