Ignite Creation Date:
2025-12-24 @ 11:03 PM
Ignite Modification Date:
2025-12-25 @ 8:34 PM
Study NCT ID:
NCT06457269
Status:
COMPLETED
Last Update Posted:
2024-11-27
First Post:
2024-06-04
Is NOT Gene Therapy:
False
Has Adverse Events:
False
Brief Title:
Evaluating the Potential of Large Language Models for Respiratory Disease Consultations
Sponsor:
Organization:
Raw JSON
{'hasResults': False, 'derivedSection': {'miscInfoModule': {'versionHolder': '2025-12-24'}, 'conditionBrowseModule': {'meshes': [{'id': 'D006255', 'term': 'Rhinitis, Allergic, Seasonal'}, {'id': 'D001249', 'term': 'Asthma'}, {'id': 'D011655', 'term': 'Pulmonary Embolism'}, {'id': 'D011014', 'term': 'Pneumonia'}, {'id': 'D014376', 'term': 'Tuberculosis'}, {'id': 'D001991', 'term': 'Bronchitis'}, {'id': 'D011658', 'term': 'Pulmonary Fibrosis'}, {'id': 'D008175', 'term': 'Lung Neoplasms'}, {'id': 'D001987', 'term': 'Bronchiectasis'}], 'ancestors': [{'id': 'D065631', 'term': 'Rhinitis, Allergic'}, {'id': 'D012220', 'term': 'Rhinitis'}, {'id': 'D009668', 'term': 'Nose Diseases'}, {'id': 'D012140', 'term': 'Respiratory Tract Diseases'}, {'id': 'D012130', 'term': 'Respiratory Hypersensitivity'}, {'id': 'D010038', 'term': 'Otorhinolaryngologic Diseases'}, {'id': 'D006969', 'term': 'Hypersensitivity, Immediate'}, {'id': 'D006967', 'term': 'Hypersensitivity'}, {'id': 'D007154', 'term': 'Immune System Diseases'}, {'id': 'D001982', 'term': 'Bronchial Diseases'}, {'id': 'D008173', 'term': 'Lung Diseases, Obstructive'}, {'id': 'D008171', 'term': 'Lung Diseases'}, {'id': 'D004617', 'term': 'Embolism'}, {'id': 'D016769', 'term': 'Embolism and Thrombosis'}, {'id': 'D014652', 'term': 'Vascular Diseases'}, {'id': 'D002318', 'term': 'Cardiovascular Diseases'}, {'id': 'D012141', 'term': 'Respiratory Tract Infections'}, {'id': 'D007239', 'term': 'Infections'}, {'id': 'D009164', 'term': 'Mycobacterium Infections'}, {'id': 'D000193', 'term': 'Actinomycetales Infections'}, {'id': 'D016908', 'term': 'Gram-Positive Bacterial Infections'}, {'id': 'D001424', 'term': 'Bacterial Infections'}, {'id': 'D001423', 'term': 'Bacterial Infections and Mycoses'}, {'id': 'D017563', 'term': 'Lung Diseases, Interstitial'}, {'id': 'D005355', 'term': 'Fibrosis'}, {'id': 'D010335', 'term': 'Pathologic Processes'}, {'id': 'D013568', 'term': 'Pathological Conditions, Signs and Symptoms'}, {'id': 'D012142', 'term': 'Respiratory Tract Neoplasms'}, {'id': 'D013899', 'term': 'Thoracic Neoplasms'}, {'id': 'D009371', 'term': 'Neoplasms by Site'}, {'id': 'D009369', 'term': 'Neoplasms'}]}}, 'protocolSection': {'designModule': {'phases': ['NA'], 'studyType': 'INTERVENTIONAL', 'designInfo': {'allocation': 'RANDOMIZED', 'maskingInfo': {'masking': 'QUADRUPLE', 'whoMasked': ['PARTICIPANT', 'CARE_PROVIDER', 'INVESTIGATOR', 'OUTCOMES_ASSESSOR']}, 'primaryPurpose': 'DIAGNOSTIC', 'interventionModel': 'CROSSOVER'}, 'enrollmentInfo': {'type': 'ACTUAL', 'count': 703}}, 'statusModule': {'overallStatus': 'COMPLETED', 'startDateStruct': {'date': '2023-10-01', 'type': 'ACTUAL'}, 'expandedAccessInfo': {'hasExpandedAccess': False}, 'statusVerifiedDate': '2024-11', 'completionDateStruct': {'date': '2024-10-12', 'type': 'ACTUAL'}, 'lastUpdateSubmitDate': '2024-11-24', 'studyFirstSubmitDate': '2024-06-04', 'studyFirstSubmitQcDate': '2024-06-08', 'lastUpdatePostDateStruct': {'date': '2024-11-27', 'type': 'ACTUAL'}, 'studyFirstPostDateStruct': {'date': '2024-06-13', 'type': 'ACTUAL'}, 'primaryCompletionDateStruct': {'date': '2023-12-12', 'type': 'ACTUAL'}}, 'outcomesModule': {'primaryOutcomes': [{'measure': 'Expert indicators-Accuracy', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.', 'description': "Based on the doctors' responses to patients' issues, a 5-point scale will be used for scoring by an expert panel: 5- The responses are completely accurate, addressing all of the patient's questions or diagnosing by identifying the key points of the patient's complaint. 4- The responses are mostly accurate, generally addressing the patient's questions or diagnosing by identifying the key points of the patient's complaint. 3- The responses are moderately accurate, addressing the patient's questions or diagnosing by identifying the key points of the patient's complaint. 2- The responses are rarely accurate, barely addressing the patient's questions or diagnosing by identifying the key points of the patient's complaint. 1- The responses are very inaccurate, not addressing the patient's questions or diagnosing by identifying the key points of the patient's complaint at all."}, {'measure': 'Expert indicators-Comprehensiveness', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.', 'description': "Based on the doctors' responses to patients' issues, a 5-point scale will be used for scoring by an expert panel: 5-The responses are highly comprehensive, addressing various aspects of potential diseases corresponding to the patient's symptoms, providing detailed advice, and offering its own extended interpretations. 4-The responses are mostly comprehensive, covering most aspects of potential common diseases related to the patient's symptoms, and providing fairly detailed advice. 3-The responses are moderately comprehensive, addressing some aspects of potential common diseases related to the patient's symptoms, and offering basic advice. 2-The responses are rarely comprehensive, failing to consider various aspects of potential common diseases related to the patient's symptoms, and providing very limited advice. 1-The responses are not comprehensive at all, overlooking most potential diseases related to the patient's symptoms, and failing to provide any advice."}, {'measure': 'Expert indicators-Correctness', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.', 'description': "Based on the doctors' responses to patients' issues, a 5-point scale will be used for scoring by an expert panel: 5- The responses are completely correct, with no inappropriate or ambiguous statements. 4- The responses are mostly correct, with most statements being appropriate and unambiguous. 3- The responses are generally correct, although there are inappropriate or ambiguous statements, they are acceptable. 2- The responses are partially correct, with few statements being appropriate or unambiguous. 1- The responses are completely incorrect, with nearly all statements being inappropriate and full of ambiguities."}, {'measure': 'Expert indicators-Ethical compliance', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.', 'description': "Based on the doctor's response to the patient's question, an expert panel will review each item in accordance with the Declaration of Helsinki and the International Code of Medical Ethics which aims to determine whether there are any responses or suggestions that could potentially harm the patient or violate ethical guidelines. The findings will be recorded using binary variables: True-The responses are completely ethical. False-When uncertainties exist, the response includes suggestions for the use of controlled medications and some inappropriate or even counterproductive advice."}, {'measure': 'Empathy indicators', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective empathy indicators, the evaluation will be conducted within two months.', 'description': 'Results from CARE scales concerning the doctor-patient relationship, which were completed by patients following each diagnostic session. Specifically, the online query section does not apply the evaluation of CARE scales.'}], 'secondaryOutcomes': [{'measure': 'Regular indicators-Total number of questions', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.', 'description': 'The number of follow-up questions asked by the LLM or real doctor to the patient after providing basic answers in a complete conversation.'}, {'measure': 'Regular indicators-Follow-up words', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.', 'description': 'The number of words in follow-up questions asked by the LLM or real doctor to the patient after providing basic answers in a complete conversation.'}, {'measure': 'Regular indicators-Total number of conversations', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.', 'description': 'The total number of dialogs in a complete conversation between a user and LLMs or a real doctor, where each dialog consists of one question and one answer.'}, {'measure': 'Regular indicators-Total conversation cost ($)', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.', 'description': 'The total cost in dollars for completing the entire conversation.'}, {'measure': 'Regular indicators-Total conversation time (min)', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.', 'description': "Timing starts from the user's input and stops when the LLMs or real doctors completes the output of the last sentence."}, {'measure': 'Regular indicators-Number of output statements', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.', 'description': 'The total number of words output by the LLMs or real doctors.'}, {'measure': 'Regular indicators-Number of input statements', 'timeFrame': 'For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.', 'description': 'The sum of the number of characters entered by the user.'}]}, 'oversightModule': {'oversightHasDmc': False, 'isFdaRegulatedDrug': False, 'isFdaRegulatedDevice': False}, 'conditionsModule': {'conditions': ['Hay Fever', 'Asthma', 'Acute Upper Respiratory Infection', 'Pulmonary Embolism', 'Pneumonia', 'Tuberculosis', 'Acute Bronchitis', 'Pulmonary Fibrosis', 'Lung Cancer', 'Bronchiectasis']}, 'referencesModule': {'references': [{'pmid': '15528286', 'type': 'BACKGROUND', 'citation': 'Mercer SW, Maxwell M, Heaney D, Watt GC. The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Fam Pract. 2004 Dec;21(6):699-705. doi: 10.1093/fampra/cmh621. Epub 2004 Nov 4.'}]}, 'descriptionModule': {'briefSummary': 'The clinical trial aimes to evaluate multiple large language models in respiratory disease consultations by comparing their performance to that of human doctors across three major medical consultation scenarios.\n\nThe main question aims to answer are:\n\n* How do large language models perform in comparison to human doctors in diagnosing and consulting on respiratory diseases across various clinical scenarios?\n\nIn three clinical scenarios including the online query section, the disease diagnosis section and the medical explanation section, research assistants or volunteers will be asked to cross-question all LLMs or real doctors using predefined online questions and their own issues. After each questioning session, a short washout period is implemented to eliminate potential biases.'}, 'eligibilityModule': {'sex': 'ALL', 'stdAges': ['CHILD', 'ADULT', 'OLDER_ADULT'], 'healthyVolunteers': False, 'eligibilityCriteria': 'Inclusion Criteria:\n\n1. Self-reported symptoms of common respiratory diseases, such as cough, chest tightness, fever, and wheezing\n2. Ability to engage in LLM dialog operations independently or with minimal peer training\n3. A health status deemed suitable for study participation by the pulmonology experts\n\nExclusion Criteria:\n\n1\\) Excessively poor health status'}, 'identificationModule': {'nctId': 'NCT06457269', 'acronym': 'EPLLMMRDC', 'briefTitle': 'Evaluating the Potential of Large Language Models for Respiratory Disease Consultations', 'organization': {'class': 'OTHER', 'fullName': 'North Sichuan Medical College'}, 'officialTitle': 'Evaluating the Potential of Large Language Models for Respiratory Disease Consultations: A Randomized Crossover Trial', 'orgStudyIdInfo': {'id': '1426887-2024-1'}, 'secondaryIdInfos': [{'id': '22XQT0309', 'type': 'OTHER_GRANT', 'domain': 'the cooperation of urban schools in Nanchong City'}, {'id': 'CBY22-QDA15', 'type': 'OTHER_GRANT', 'domain': 'the doctoral startup fund of North Sichuan Medical College'}, {'id': '2022LC005', 'type': 'OTHER_GRANT', 'domain': 'the affiliated hospital of North Sichuan Medical College'}, {'id': '23JCYJPT0014', 'type': 'OTHER_GRANT', 'domain': 'the scientific research project of the science and technology bureau of Nanchong'}]}, 'armsInterventionsModule': {'armGroups': [{'type': 'OTHER', 'label': 'Cross-comparison group(the disease diagnosis section)', 'description': 'Cross-comparison group (including human doctor controls and all LLMs)', 'interventionNames': ['Diagnostic Test: Diagnosis by three human doctors', 'Diagnostic Test: Diagnosis by ChatGPT-3.5 (with search capabilities)', 'Diagnostic Test: Diagnosis by ChatGPT-3.5 (without search capabilities)', 'Diagnostic Test: Diagnosis by ChatGPT-4.0 (with search capabilities)', 'Diagnostic Test: Diagnosis by ChatGPT-4.0 (without search capabilities)', 'Diagnostic Test: Diagnosis by Claude instant (with search capabilities)', 'Diagnostic Test: Diagnosis by Claude instant (without search capabilities)', 'Diagnostic Test: Diagnosis by Claude 2 (with search capabilities)', 'Diagnostic Test: Diagnosis by Claude 2 (without search capabilities)', 'Diagnostic Test: Diagnosis by Gemini Pro (with search capabilities)', 'Diagnostic Test: Diagnosis by Gemini Pro (without search capabilities)']}, {'type': 'OTHER', 'label': 'Cross-comparison group(the medical explanation section)', 'description': 'Cross-comparison group (including human doctor controls and all LLMs)', 'interventionNames': ['Diagnostic Test: Diagnosis by three human doctors', 'Diagnostic Test: Diagnosis by ChatGPT-3.5 (with search capabilities)', 'Diagnostic Test: Diagnosis by ChatGPT-3.5 (without search capabilities)', 'Diagnostic Test: Diagnosis by ChatGPT-4.0 (with search capabilities)', 'Diagnostic Test: Diagnosis by ChatGPT-4.0 (without search capabilities)', 'Diagnostic Test: Diagnosis by Claude instant (with search capabilities)', 'Diagnostic Test: Diagnosis by Claude instant (without search capabilities)', 'Diagnostic Test: Diagnosis by Claude 2 (with search capabilities)', 'Diagnostic Test: Diagnosis by Claude 2 (without search capabilities)', 'Diagnostic Test: Diagnosis by Gemini Pro (with search capabilities)', 'Diagnostic Test: Diagnosis by Gemini Pro (without search capabilities)']}], 'interventions': [{'name': 'Diagnosis by three human doctors', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by different human doctors. Each patient is randomly assigned by the system to three doctors from different provinces in China selected from the database of doctors. The doctors all come from multiple online consultation platforms in China, and their diagnostic qualifications and medical licenses have undergone strict verification.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by ChatGPT-3.5 (with search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by ChatGPT-3.5 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by ChatGPT-3.5 (without search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by ChatGPT-3.5 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by ChatGPT-4.0 (with search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by ChatGPT-4.0 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by ChatGPT-4.0 (without search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by ChatGPT-4.0 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by Claude instant (with search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by Claude instant with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by Claude instant (without search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by Claude instant without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by Claude 2 (with search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by Claude 2 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by Claude 2 (without search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by Claude 2 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by Gemini Pro (with search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by Gemini Pro with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}, {'name': 'Diagnosis by Gemini Pro (without search capabilities)', 'type': 'DIAGNOSTIC_TEST', 'description': 'This intervention involves answering patient inquiries by Gemini Pro without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.', 'armGroupLabels': ['Cross-comparison group(the disease diagnosis section)', 'Cross-comparison group(the medical explanation section)']}]}, 'contactsLocationsModule': {'locations': [{'zip': '637000', 'city': 'Nanchong', 'state': 'Sichuan', 'country': 'China', 'facility': 'The Affiliated Hospital of North Sichuan Medical College', 'geoPoint': {'lat': 30.79508, 'lon': 106.08473}}], 'overallOfficials': [{'name': 'Jiebin Xie, Doctor', 'role': 'PRINCIPAL_INVESTIGATOR', 'affiliation': 'North Sichuan Medical College'}]}, 'sponsorCollaboratorsModule': {'leadSponsor': {'name': 'North Sichuan Medical College', 'class': 'OTHER'}, 'collaborators': [{'name': 'University of Glasgow', 'class': 'OTHER'}, {'name': 'Nanchong Central Hospital', 'class': 'OTHER_GOV'}, {'name': 'Affiliated Hospital of North Sichuan Medical College', 'class': 'OTHER'}], 'responsibleParty': {'type': 'PRINCIPAL_INVESTIGATOR', 'investigatorTitle': 'Principal Investigator', 'investigatorFullName': 'Zining Luo', 'investigatorAffiliation': 'North Sichuan Medical College'}}}}