Towards empathic medical conversation in Narrative Medicine: A visualizatio

• Intelligence augmentation tool enhances the quality of empathic medical conversation. • EOPD enables empathic opportunity recognition, tailors personalized communication. • Multi-modal emotion data helps physicians understand patients’ emotion and intention. • Empathic opportunity reminding facilitate physicians to express warmth and concern. • Physician-AI collaboration ensures physician's autonomy and mitigate algorithm biases. Empathic medical conversation is central to patient-centered care within Narrative Medicine. However, difficulties, such as physicians’ limited empathic capabilities and lack of time, impede the practice. Research on real-time, on-site empathic medical exchanges has been limited in exploring technology to assist and enhance physicians’ capabilities. This paper proposed the Empathic Opportunity Perception and Distinction (EOPD) framework for building physician-AI collaboration based on Intelligence Augmentation (IA) for empathic conversations. The EOPD integrates two multi-modal machine learning (ML) models based on facial and verbal cues, presenting a physician-AI interaction framework and three distinctive visualization components: emotional reference, opportunity reminding and keyword collection, and situation understanding. To assess EOPD's effectiveness and gauge physicians’ and patients’ receptiveness, a prototype system named EMVIS ( EM otional VIS ualization ) was designed and developed. Results from the study demonstrated improvements in physicians’ empathy efforts and perceived empathy performance when using EMVIS, particularly for junior physicians. Physicians and patients held positive attitudes towards EMVIS, with patients expressing a high expectation that EMVIS would improve the physician-patient relationship. The research showed the efficacy of the multi-modal ML models in supporting complex affective empathy and EMVIS in facilitating and complementing empathy concerns. It highlighted the tailored support to junior and senior physicians and emphasized physician-AI collaboration to maintain user autonomy and mitigate potential biases. Future research should explore extensive system applications, tailor visual and interactive support for physicians, and implement adaptive and reflective ML models to improve the effectiveness and efficiency of empathy communications.