Digit Health. 2026 Aug 14;12:20552076261478506. doi: 10.1177/20552076261478506. eCollection 2026 Jan-Dec.
ABSTRACT
OBJECTIVE: To compare ten large language models (LLMs) from seven AI companies, across free and paid tiers, on inhaler technique instruction accuracy for ten devices, benchmarked against GINA 2025 and GOLD 2026 strategy reports.
METHODS: In a cross-sectional, blinded evaluation, ten LLMs (ChatGPT Free/Plus, Claude Free/Pro, Gemini, Google AI Pro, DeepSeek, Microsoft Copilot, Perplexity AI, Meta AI) were queried between 10-17 March 2026 via each vendor's official web interface. Fifty standardised prompts (10 devices × 5 question types) were submitted in triplicate on different days, yielding 1500 outputs. Two pulmonologists and one allergist scored seven metrics step-completion rate, critical and non-critical error counts, step-sequencing accuracy, safety-warning score, and Likert-scaled overall accuracy and patient comprehensibility against a gold standard derived from GINA 2025, GOLD 2026, the ERS/ISAM Task Force consensus, and manufacturer leaflets.
RESULTS: Paid tiers outperformed free tiers on five of seven metrics (Mann-Whitney U; all p < 0.001), with a 24-fold reduction in critical errors (0.48 → 0.02) and a 12.9-point gain in safety-warning coverage (62.4% → 75.3%). Google AI Pro achieved the highest overall accuracy (4.98 ± 0.13); Microsoft Copilot (3.23) and Perplexity AI (2.83) ranked lowest. Claude Free surpassed Claude Pro on safety warnings (99.2% vs. 66.7%; r = 0.86). Test-retest reliability met ICC ≥ 0.75 for only two metrics.
CONCLUSIONS: Paid subscriptions yield meaningful but non-uniform gains in inhaler-instruction accuracy. Safety-warning omissions and test-retest variability argue against autonomous LLM use; these tools are best deployed as adjuncts to clinician- and pharmacist-led teach-back education.
PMID:42605352 | PMC:PMC13477548 | DOI:10.1177/20552076261478506