AI in medical devices: BlueBridge positions and current developments

Each line is a short statement. Tap it for supporting detail.

The 20-second version

Most AI health startups can build the model. The hard part is proving it’s safe enough to clear.

We have twenty years’ experience in regulated medical software, taking AI imaging devices through design controls and clinical validation.

We’re now developing new methodologies for LLM-based devices.

We’ve built our own design-control platform to maintain the traceability and rigour while letting us move as fast as AI tools write code, so documentation never lags behind the product.

Where we’re heading
AI-enabled devicesWe already deliver AI imaging devices through design controls, risk management and clinical validation.

So far our regulated AI work has been computer vision rather than generative AI. That’s where most of the market is: of about 1,500 AI devices on FDA’s list, roughly three quarters are radiology.

An AI-enabled device has AI somewhere in it, for example in the interface. With AI as the device, the AI function itself makes the medical claim. Regulators care about the second much more.

LLMs in devicesWe’re building an evaluation harness for LLM-based devices, because nobody has a settled answer yet for how you show a generative model is safe enough.

Our starting assumptions: the model is locked, we don’t fine-tune, and the things that vary are the prompt and the context. That turns the problem into one we can measure: for a given input, how repeatable and how safe is the output?

We bring evaluation experience from two non-regulated LLM projects (RFX and Propylon, see Projects) and from regulated imaging AI. There are no LLM device clients yet.

Our own platformWe’ve built our own design-control platform so our process can keep up with AI-speed development without dropping traceability.

See “Our platform” for detail. Short version: we decided to build rather than buy, rewrote our SOPs at the same time, and the platform is those SOPs in software form. We build it with Claude, inside our ISO 13485 system.

Our platform (QDP)
WhyCode is fast now. Design controls are the bottleneck.
  • SOPs overhauled alongside the platform.
The ideaThe software is a manifestation of our SOPs.
  • Process is the path of least resistance.
  • Human in the loop makes the best choices.
  • Required steps happen by default.
TraceabilityTraceability is the default.
  • V-model links and traceability matrix.
  • Upstream change flags suspect links.
  • Impact analysis before a change goes in.
  • Gap list: no test, failed test, open risk.
  • Signed revisions. Full audit trail.
AI’s roleAI drafts and checks. A named human approves.
  • Platform is the system of record.
  • Approval on record: ID, version, reason, author, approver.
  • AI inputs kept so outputs can be reproduced.
Non-determinismLLMs get narrow jobs, never open-ended ones.
  • One task per call.
  • Specific prompt and context, version-controlled.
  • A rubric for each task.
  • Easier to test, repeat and review.
ResultsTime savings being measured on past and mock projects. No figures yet.

Current developments
AgentsAgents raise the bar for evaluation, because the route to the answer has to be checked as well as the answer itself.

More detail under “Agents” in Evaluation. UpDoc, the first cleared device described as an agent, kept the regulated function deterministic.

Recent referencesDevelopments from the last six months.
  • UpDoc clearance (announced June 2026).
  • EU AI Omnibus deal (May 2026) and the 2028 date for devices.
  • FDA’s January 2026 CDS and wellness changes; QMSR in force since February.
  • CDRH’s FY2027 guidance agenda (October 2026).
  • OpenAI’s HealthBench Professional (April 2026), aimed at clinician workflows: care consultation, documentation, research.
  • “Emerging Trends in AI Agent Evaluation”, Princeton, Stanford, UK AISI and others (October 2026).
Open problemThere is no settled method yet for validating generative AI in a medical device.

FDA has said traditional performance metrics may not be enough for generative AI and that new metrics may be needed.

Questions for AI startups
If you run the same case ten times, how different are the answers?

Leads into repeatability, worst of n and our evaluation harness.

What happens to your evidence when your model provider releases a new version?

Leads into locked versions, PCCPs and treating the vendor as a supplier.

Have you tested whether clinicians actually catch the model’s mistakes?

Seeded errors in a usability study show whether the human in the loop is a real control.

Which patient data goes to the model provider, and where is it processed?

Hosted models mean PHI may leave their environment. Leads into our de-identification and data pipeline work.

Regulation as of November 2026
FDA · first LLM deviceThe first patient-facing LLM device FDA cleared kept the LLM away from the dose. The LLM turns what the patient says into data, and a rules-based engine does the calculation.
  • UpDoc, 510(k) K253281, cleared 23 December 2025 and announced in June 2026. Insulin management for adults with type 2 diabetes.
  • The insulin instructions come from a deterministic service using clinician-set parameters. The predicate was an existing dose calculator, so no De Novo was needed.
  • Where the LLM sits in the architecture determines the regulatory path.
  • UpDoc was on our list of possible contacts.

Sources: Innolitics, IntuitionLabs

FDA · lifecycle guidanceFDA’s main AI lifecycle guidance has been in draft since January 2025, and finalising it is at the top of the device centre’s list for this year.
  • It applies a total product lifecycle approach: data management, bias, user interface and what clinicians need to understand, post-market monitoring.
  • Also on CDRH’s FY2027 priority list: a draft guidance on generative AI conversational devices for mental health.
  • One trade article describes this guidance as already final. RAPS reported it as still to be finalised on 2 October. FDA’s own guidance page is the reference.

Source: RAPS, 2 Oct 2026

FDA · decision supportIn January FDA widened the room for decision-support software that isn’t regulated as a device. That gets startups to market faster, and it doesn’t give them evidence a hospital or an acquirer will accept.
  • 6 January 2026: revised CDS and general wellness guidance, published without a comment period.
  • FDA will use enforcement discretion for CDS that gives a single recommendation, if it is the only clinically appropriate option.
  • Most of the “time-critical means it’s a device” language came out, and the transparency expected of CDS can be less detailed.
  • Patient-facing tools and opaque models are still treated as devices. Generative AI isn’t addressed directly.

Sources: FDA Law Blog, McDermott

FDA · quality systemSince February, FDA’s quality system rule lines up with ISO 13485. We’ve run a certified 13485 system for years, so for our clients nothing has to change.
  • QMSR took effect on 2 February 2026.
  • FDA also finalised updated cybersecurity guidance in February 2026. Most AI devices that connect to the internet count as cyber devices.

Sources: Ballard Spahr, Congressional Research Service

CanadaHealth Canada has its own route for pre-approving model updates, and this year it tightened what it expects after launch.
  • Pre-market guidance for machine-learning-enabled medical devices, Class II to IV, including PCCPs, good machine learning practice and sex- and gender-based analysis (SGBA Plus).
  • March–April 2026: five new or revised guidances, including terms and conditions that can be put on a licence at any point, and clinical evidence across the lifecycle.
  • Relevant to the Canada outreach, and TomoVerse is Canadian.

Sources: BLG, NSF

AI terms you’ll hear
Foundation model / LLMA large general-purpose model, such as GPT, Claude or Gemini, trained by a vendor and used by others as a component.

For a device company it’s usually off-the-shelf software (SOUP) from a supplier that releases new versions on its own schedule. FDA has said it wants to start tagging devices that use foundation models on its AI device list.

AgentAn LLM that plans over several steps and acts by calling tools, rather than answering one prompt.

Examples: looking things up in an EHR, running a calculation, writing to a system, messaging a patient. More autonomy means more ways to go wrong, and evaluation has to check each step as well as the result.

Scaffold / harnessThe code, tools and prompts wrapped around a model to make it an agent. An evaluation harness is the setup that runs the tests.

Change the scaffold and performance changes, even with the same model, so the model and scaffold are evaluated as a combination.

TrajectoryThe full record of what an agent did: every step, tool call and intermediate result.

Reviewing trajectories is how you spot shortcuts or unsafe actions that a correct final answer hides.

Tool use / MCPLetting a model call external functions. MCP (Model Context Protocol) is the emerging standard for plugging tools and data sources into models.

Each connected tool is something the model can misuse or be tricked into misusing, so it belongs in the risk analysis.

RAG / groundingRetrieval-augmented generation: the system fetches relevant documents and the model answers from them, ideally citing them.

The usual way to get current or product-specific knowledge without retraining. Two things to test: did retrieval find the right documents, and does the answer actually match them.

Context and promptThe prompt is the instructions; the context is everything the model sees for a given request, including retrieved documents and conversation history.

With a locked model these are the things that vary, so they need version control like any other design input.

HallucinationOutput that is fluent and confident but not supported by the input or by fact.

Measure the rate, and grade each one by how much harm it could do.

Non-determinismThe same input can give different outputs from one run to the next.

Settings such as temperature reduce it but rarely remove it, and hosted models can change behind the same name. Hence repeated runs and pinned model versions.

Fine-tuning vs promptingFine-tuning retrains the model on your data; prompting and retrieval steer a model that stays unchanged.

Fine-tuning brings training-data obligations with it. Our default is a locked model with controlled prompts and context.

Reasoning modelsModels that work through a problem in visible intermediate steps before answering, often called chain of thought.

Useful for monitoring, but the visible reasoning isn’t guaranteed to reflect what actually drove the answer.

LLM as judgeUsing one model to grade another’s output. It needs calibrating against human experts before you trust it.

Also called an autograder. Its own error should show up in the reported uncertainty.

pass@k vs pass^kpass@k means it succeeds at least once in k tries; pass^k means it succeeds every time. Devices need the second.

See “Worst case” under Evaluation for the worked numbers.

Red-teamingDeliberately trying to make the system fail or misbehave before users do.

For LLMs that includes jailbreaks (talking the model past its safeguards) and adversarial phrasing of clinical questions.

Prompt injectionInstructions hidden in content the model reads, such as a document or web page, that hijack what it does.

The main security risk for agents that read external data. It’s a cybersecurity issue as well as an AI one.

ContaminationWhen test questions or answers were in the model’s training data, so a high score reflects memory rather than ability.

One reason public benchmark scores say little about a specific device.

Reward hackingThe system meets the letter of the test without doing the task as intended.

For example, an agent that finds the answer key instead of solving the problem. Trajectory review catches it.

Evaluation
The core positionYou don’t validate an LLM. You validate a bounded system that contains one: this task, for these users, in this workflow.

FDA defines an AI device function by what it’s intended to do, and the EU AI Act classifies risk the same way. The same model writing a meeting summary and recommending a cancer treatment sits in completely different risk classes. The model is identical; the system around it is what changed.

No single scoreWith generative AI there isn’t one right answer, so one accuracy number won’t do. We report a set of metrics, and we weight each error by how much harm it could do.

Typical set: clinically significant hallucination rate, critical omission rate, consistency when the question is reworded, refusal rate, and performance by subgroup and site.

Example: inventing a patient’s middle name and inventing a contraindication are both hallucinations, and only one of them can hurt someone. FDA’s advisory committee asked for error frequency and severity for this reason.

Worst caseAverages hide the dangerous cases. We run the same case many times and look at the worst answer, because any one of those answers could be the one a patient gets.

The Princeton, Stanford and UK AI Security Institute survey of agent evaluation (October 2026) has a clean illustration. Give an agent four tasks, five tries each. It succeeds on a single try 45% of the time. It succeeds at least once on 75% of tasks. It succeeds every time on only 25%. Companies tend to quote the 75%. For a device, the 25% is the number that matters.

OpenAI’s HealthBench reports “worst of n” for the same reason.

LLM as judgeAn LLM grader is a measuring instrument. Before we trust its scores, we calibrate it against clinicians, the same way you’d calibrate a lab assay.
  • Check the grader against expert-labelled cases from the actual intended use.
  • Put the grader’s own error into the error bars. UK AISI uses a panel of LLM graders and reports the median.
  • Have experts review flagged cases plus a random sample.

It isn’t a new idea, but most startups skip it.

Use the right toolWherever a deterministic check exists, we use it. Expert judgement is for the parts that genuinely need a judgement.

Doses, units, required fields, citations and structured output can all be checked by code. Clinical quality and tone need rubrics graded by experts or a calibrated LLM. Questions about how people use the tool need human studies. Google’s health-agent work splits the evaluation in this way.

Locked modelWe lock the model and treat the prompt and context as the variables. The question becomes: for a given input, how repeatable is the output?

We record model version, prompt version and the identity of every source the model saw, so any output can be reproduced. A new model version is treated as a design change and re-run against the full test set.

AgentsWith agents, how they got there matters as much as the answer. Two runs can reach the same result and one of them took an unsafe shortcut on the way.

From the October agent-evaluation survey:

  • Performance depends on the model and on the scaffolding and tools around it. Swap either and the results change, so you evaluate the combination.
  • Read the logs, not just the final answers. That’s where shortcuts, cheating and contamination show up.
  • The test harness is itself something that can be wrong, and it needs checking.
  • Agents that use tools expand the security surface: prompt injection through a document the agent reads is a real hazard.
Human in the loopA human in the loop counts as a risk control only if you’ve tested whether the human actually catches the errors.

Automation bias is well documented: people approve confident-looking output. FDA’s January 2026 decision-support guidance discusses automation bias explicitly. So validation should include the clinician and the AI together, with seeded errors, to measure how many get caught.

BenchmarksA high score on a medical exam benchmark tells you something about the model and nothing about your device.

Example: researchers took USMLE questions and swapped brand drug names for generic names. The models often gave different answers, even though they knew the two names meant the same drug. That suggests memorisation rather than understanding. HealthBench is a good benchmark, and its authors would agree it isn’t a validation framework for a specific device.

SummaryDone properly, LLM validation looks like regulatory science again: a test harness and a set of metrics in place of sensitivity and specificity.

Worked example. Intended use: write patient discharge instructions from a clinician-approved summary. Take about 2,000 representative cases, several prompt variants and repeated runs: tens of thousands of outputs. Code checks catch medication and dose errors. A calibrated LLM grader scores clarity and completeness. Clinicians review everything flagged plus a sample. You report hallucination rate, critical omission rate, medication transcription errors, consistency and subgroup performance.

Opinions for debate
ArchitectureDiscussDesign LLM devices so the AI never makes the medical decision itself.
  • An LLM (large language model) is the type of AI behind ChatGPT and Claude. Its answers can vary from run to run and can be wrong.
  • Use it for language: taking in what the patient says, summarising, explaining.
  • Leave the medical step, such as a dose calculation or a classification, to ordinary software. It gives the same result every time and can be tested the usual way.
  • UpDoc, the first cleared LLM device, was built like this.
  • Counter: if the AI’s clinical reasoning is the whole value of the product, this doesn’t work, and the evidence needed is much harder to produce.
DeregulationDiscussLess regulation doesn’t remove the need to prove the product works.
  • FDA is easing oversight of some software, so some products now need less regulatory review.
  • Buyers of the company will still want proof that the product does what it says.
  • Hospitals and other customers want that proof too, before they trust it with patients.
  • Testing and quality records are good practice whether or not a regulator demands them.
  • A later feature change can bring a product back under device rules.
Model vendorsDiscussTreat the company supplying the AI model like any other critical supplier.
  • Most startups build on a model made by another company, such as OpenAI, Anthropic or Google. This is called a foundation model.
  • Those companies update their models regularly, which can change how the product behaves.
  • ISO 13485 already covers this for suppliers: assess them, control what you buy, track their changes.
  • So a model update is a supplier change: re-test before accepting it.
  • That’s why we pin each product to one specific model version instead of letting it update itself.
The bottleneckDiscussBuilding a medical AI product is cheap now. Showing it’s safe and works is the expensive part.
  • AI tools let a small team build a working prototype quickly.
  • The slow, costly work is the testing, documentation and clinical evidence that regulators and buyers need.
  • So the advantage goes to companies that can produce that proof efficiently, not to those that build fastest.

Limits of current claims
No time-saving figures for the platform. We’re still measuring.

Testing is under way on past and mock projects.

The LLM evaluation harness is in development.

The methods are well thought through; the tooling isn’t finished. What it’s designed to do, in short:

  • Worst of n: run each test case several times and score the worst answer, not the average. Any one of those answers could be the one a patient gets.
  • Repeat and compare: run the same input many times and measure how much the answers vary. Low variation is evidence in itself.
  • Look at the spread: report the distribution of outputs, not one headline number, so rare bad answers are visible.
  • Reliability as a metric: succeeding every time counts for more than succeeding at least once.
  • Deterministic wherever possible: fixed model version, controlled settings, and code-based checks for anything that can be checked by code.
  • Calibrated grading: where an LLM grades outputs, its scores are checked against expert judgement first.
Our LLM projects so far were not medical devices.

RFX and Propylon show evaluation skill, not regulatory experience with LLMs. Our regulated AI is computer vision.

Projects
TomoVerse Canada · cardiology · AI imaging SaMD tvCT: AI-assisted CT planning for TAVR. The AI proposes segmentations and measurements of the aortic root; a clinician reviews and adjusts.
  • Locked, clinician-supervised model.
  • AI development practice follows ISO/IEC 5338, with FDA AI guidance as the reference.
  • Model development plan and data management plan, plus a clinical validation plan built around a multi-centre comparison of automated measurements with expert readers.
  • Data pipeline: DICOM ingestion, de-identification, pre- and post-processing, enforced governance.
  • Evaluated at three levels: the model, the derived measurements, and the locked end-to-end workflow.
Entia
  • AI computer vision in a regulated diagnostic setting.
  • Finds cell layers in centrifuged blood samples, more robustly than the previous approach.
Novus
  • AI computer vision in a regulated diagnostic instrument.
  • Detects fluid fill state in assay cartridges, robust to lighting, exposure and camera alignment.
RFX Not regulated · LLM scoring LLMs score tender responses against a rubric.
  • Scored against a rubric, with reasons and quoted source text.
  • Tested answers of different strength: strong, medium, deliberately weakened.
  • Checked scores fall as quality falls (monotonicity).
  • Measured variance across runs, prompts and models.
  • Compared with expert judgement. Reported model disagreement.
  • Transfers to devices: same problem as validating an LLM judge.
Propylon Not regulated · legal tech LLM search and version comparison over legislation.
  • Compares two versions of a law. Classifies each change: minimal, minor or material.
  • Several LLMs compared against expert judgement.
  • Rubrics and a defined evaluation process.
  • Transfers to devices: material-change classification is change-impact analysis for requirements, risk controls and tests.

Nothing matches that search. Try a shorter word.