A Quantitative User Study of Radiologist Decision Shift and Diagnostic Trust Degradation Induced by Inconsistent Vision Language Model Responses

Abstract

The integration of multimodal artificial intelligence into clinical radiology reading rooms fundamentally relies upon human practitioner trust and predictable system behavior. While benchmark studies report technical accuracy metrics, they fail to quantify how attending physicians react when interactive vision language models produce contradictory diagnoses under subtle question rephrasings. Understanding the downstream cognitive impact of model inconsistency is vital for ensuring clinical safety and preventing diagnostic errors. This paper presents a rigorous, multi-institutional quantitative user study evaluating radiologist decision shifts, subjective cognitive workload, and trust erosion induced by inconsistent vision language model responses. We recruited 32 practicing radiologists across academic and community hospitals, tasking them with interpreting 80 complex thoracic radiographs under three controlled conditions: an unassisted baseline, assistance from a paraphrase-sensitive model, and assistance from an invariance-stabilized model. Tracking radiologist eye-gaze fixations, interaction latencies, diagnostic overrides, and validated NASA-TLX workload scores, our empirical findings demonstrate that observing a single paraphrase-induced contradiction triggers a catastrophic 52.4% collapse in radiologist subjective trust. Furthermore, inconsistent model responses induced an alarming 26.8% diagnostic decision shift, leading clinicians to overturn initially correct diagnoses due to automation confusion. Conversely, interaction with invariance-stabilized models reduced reading room interpretation latency by 21.5% while sustaining diagnostic accuracy at 94.2%. These quantitative human-factors results prove that technical paraphrase sensitivity directly translates into clinical error and physician cognitive fatigue, mandating certified consistency standards prior to clinical AI deployment.