Ignite Creation Date:
2026-03-26 @ 3:16 PM
Ignite Modification Date:
2026-03-30 @ 2:04 AM
Study NCT ID:
NCT07491068
Status:
NOT_YET_RECRUITING
Last Update Posted:
2026-03-24
First Post:
2026-02-25
Is NOT Gene Therapy:
True
Has Adverse Events:
False
Brief Title:
The Accuracy and Efficacy of Large Language Model Written Hospital Course Summaries
Sponsor:
Organization:
Raw JSON
{'hasResults': False, 'derivedSection': {'miscInfoModule': {'versionHolder': '2026-03-25'}}, 'protocolSection': {'designModule': {'phases': ['NA'], 'studyType': 'INTERVENTIONAL', 'designInfo': {'allocation': 'RANDOMIZED', 'maskingInfo': {'masking': 'SINGLE', 'whoMasked': ['OUTCOMES_ASSESSOR']}, 'primaryPurpose': 'HEALTH_SERVICES_RESEARCH', 'interventionModel': 'PARALLEL'}, 'enrollmentInfo': {'type': 'ESTIMATED', 'count': 786}}, 'statusModule': {'overallStatus': 'NOT_YET_RECRUITING', 'startDateStruct': {'date': '2026-06-01', 'type': 'ESTIMATED'}, 'expandedAccessInfo': {'hasExpandedAccess': False}, 'statusVerifiedDate': '2026-03', 'completionDateStruct': {'date': '2027-12-31', 'type': 'ESTIMATED'}, 'lastUpdateSubmitDate': '2026-03-19', 'studyFirstSubmitDate': '2026-02-25', 'studyFirstSubmitQcDate': '2026-03-19', 'lastUpdatePostDateStruct': {'date': '2026-03-24', 'type': 'ACTUAL'}, 'studyFirstPostDateStruct': {'date': '2026-03-24', 'type': 'ACTUAL'}, 'primaryCompletionDateStruct': {'date': '2026-12-31', 'type': 'ESTIMATED'}}, 'outcomesModule': {'primaryOutcomes': [{'measure': 'Safety assessed by outcome evaluator on an ordinal scale (1,2,3)', 'timeFrame': 'Assessment at one time point - hospital discharge (up to 5 days)', 'description': 'Safety will be assessed using an ordinal scale ranging from "acceptable without further changes", through "acceptable with minor revisions", to not "unacceptable in its current form" ( a combination of either absence of important information or presence of incorrect/hallucinated information)'}], 'secondaryOutcomes': [{'measure': 'Safety of the edited and unedited LLM generated course summaries evaluated on an ordinal scale (1,2,3)', 'timeFrame': 'At hospital discharge (up to 5 days)', 'description': 'Safety of LLM-generated, resident-edited and un-edited hospital course summaries compared with standard specialist-written documentation. Safety, will be assessed using an 3 categories ordinal scale ranging from "acceptable without further changes" (best outcome), through "acceptable with minor revisions" (uncertain outcome), to "not unacceptable in its current form" (absence of important information or presence of incorrect/hallucinated information; worst outcome)'}, {'measure': 'Secondary outcome: Content validity across the following predefined domains (5C).', 'timeFrame': 'At hospital discharge (up to 5 days)', 'description': 'Secondary outcomes measures: detailed evaluation of quality across the following predefined domains (5C): completeness, conciseness, cohesiveness, absence of critical errors (including hallucinations), and cultural and linguistic fidelity. Individual domains will be separately assessed using a 5-point Likert scale, (1 - complete absence of agreement, 2 - partial absence of agreement, 3 - neutral agreement, 4- partial agreement, 5- complete agreement)'}, {'measure': 'Time to complete hospital summary (seconds)', 'timeFrame': 'At hospital discharge (up to 5 days)', 'description': 'Workflow-related secondary outcomes include time efficiency of hospital course documentation when using the LLM-generated workflow compared with standard practice. The outcome measure will be the time taken to complete the hospital course summary (seconds), from initiating discharge to "ready to signing".'}, {'measure': 'Generation stability assessed on an ordinal scale (1,2,3)', 'timeFrame': 'From hospital discharge to 30 days after hospital discharge', 'description': 'Generation stability will assess the consistency of LLM generated unedited hospital course summaries when the model is applied repeatedly to identical source clinical documentation. Generation stability will be evaluated by the same adjudicator on the scale of 1 - stable, 2 - acceptable variation, 3 - unacceptable variation'}, {'measure': 'Adoption. Percentage of hospital discharges where the LLM generated course summary was utilized.', 'timeFrame': 'From hospital discharge to 30 days after hospital discharge', 'description': 'After completing the randomized controlled trial, the adoption of the LLM-generated workflow will be evaluated over a one-month period. The outcome measure will be defined as the proportion of eligible discharges for which LLM-derived summary generation was utilized.'}, {'measure': 'Inter-rater reliability. Percentage agreement between two adjudicators of a hospital course summary on safety and content validity ratings.', 'timeFrame': 'From hospital discharge to 30 days after hospital discharge', 'description': 'Reliability-related outcomes will assess the consistency of expert evaluations of hospital course summaries. Percentage will quantify the agreement between evaluators on safety and content validity ratings.'}, {'measure': 'Intra-rater reliability. Concordance of safety and content validity ratings between two evaluations of the same hospital course and same adjudicator.', 'timeFrame': 'At hospital discharge (up to 5 days)', 'description': 'Intra-rater reliability will quantify the consistency of repeated evaluations by the same evaluator based on the concordance of safety and content validity ratings (percentage of agreement)'}, {'measure': 'Temporal stability (test-retest reliability) - concordance between the ratings of the same adjudicator after washout period.', 'timeFrame': '30 days after hospital discharge', 'description': 'Temporal stability of evaluator judgments will be assessed through repeated evaluation of identical hospital course summaries by the same evaluator after a predefined 30 day washout period (test-retest reliability). Outcome measures will be based on the concordance of safety and content validity ratings (percentage)'}]}, 'oversightModule': {'oversightHasDmc': True, 'isFdaRegulatedDrug': False, 'isFdaRegulatedDevice': False}, 'conditionsModule': {'keywords': ['generative large language models', 'automated discharge summary', 'AI', 'documentation', 'Electronic health records'], 'conditions': ['Hospitalizations']}, 'referencesModule': {'references': [{'pmid': '40355506', 'type': 'BACKGROUND', 'citation': 'Ganzinger M, Kunz N, Fuchs P, Lyu CK, Loos M, Dugas M, Pausch TM. Automated generation of discharge summaries: leveraging large language models with clinical data. Sci Rep. 2025 May 12;15(1):16466. doi: 10.1038/s41598-025-01618-7.'}, {'pmid': '17327525', 'type': 'BACKGROUND', 'citation': 'Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA. 2007 Feb 28;297(8):831-41. doi: 10.1001/jama.297.8.831.'}, {'pmid': '38913371', 'type': 'BACKGROUND', 'citation': 'Burden M, Astik G, Auerbach A, Bowling G, Kangelaris KN, Keniston A, Kochar A, Leykum LK, Linker AS, Sakumoto M, Rogers K, Schwatka N, Westergaard S. Identifying and Measuring Administrative Harms Experienced by Hospitalists and Administrative Leaders. JAMA Intern Med. 2024 Sep 1;184(9):1014-1023. doi: 10.1001/jamainternmed.2024.1890.'}]}, 'descriptionModule': {'briefSummary': 'Background: Physicians worldwide face an increasing administrative burden that diverts time from direct patient care. Among inpatient documentation tasks, authoring hospital course summaries is particularly time-consuming and critical for safe care transitions. Large language models (LLMs) have shown promise for clinical text generation; however, robust evidence from randomized, evaluator-blinded trials conducted in routine hospital practice remains limited. Objectives: The CLEAN study aims to evaluate whether LLM-assisted, specialistedited generation of hospital course summaries is non-inferior in safety compared with standard clinician-written documentation in routine inpatient care. Secondary objectives include noninferiority assessments of resident-edited and unedited LLMgenerated summaries. Additional objectives are to evaluate summary quality across predefined domains, quantify physician documentation time, assess LLM generation stability, measure clinician adoption following the randomized phase, and examine inter-, intra-observer, and test-retest reliability of expert assessments. Methods: This is a single-centre, double-campus, exploratory randomized controlled non-inferiority trial conducted at a tertiary university hospital. Consecutive hospital discharges across multiple clinical departments are randomized 1:1 to either an LLM-assisted documentation workflow or standard manual authorship. The intervention integrates an on-premise LLM into a parallel hospital information system, generating draft hospital course summaries from complete, uncurated clinical documentation, which physicians may review and edit prior to finalization. Safety, the primary outcome, defined as presence of all important information and absence of incorrect/hallucinated information, is assessed by an adjudication committee blinded to documentation workflow. Secondary outcomes include content validity, workflow efficiency, generation stability, post-trial clinician adoption, and reliability metrics. A total of 786 discharge episodes are required to assess non-inferiority using a predefined margin of 5 percentage points. Ethics and Dissemination: The study will be conducted in accordance with the Declaration of Helsinki, Good Clinical Practice, and the General Data Protection Regulation. A waiver of informed consent is sought due to minimal risk and exclusive use of routine clinical data. Results will be disseminated through peer-reviewed publication and engagement with healthcare stakeholders.', 'detailedDescription': "The World Health Organization has identified a severe global health workforce crisis, estimating a shortage of approximately 12.7 million physicians worldwide in 2020. Projections indicate that this deficit will continue to worsen in the coming years \\[1\\]. Furthermore, these shortages occur alongside rising healthcare demand driven by population ageing and the increasing prevalence of multimorbidity. Despite that, administrative burden remains a persistent and systemic challenge across healthcare systems worldwide. Initiatives aimed at reducing physicians' non-clinical workload have achieved only limited success, leading to the continued diversion of clinical expertise away from direct patient care \\[2\\]. Among administrative responsibilities, the preparation of discharge letters is one of the most time-consuming tasks for physicians caring for hospitalized patients, yet it is also central to ensuring continuity of care after discharge. Despite major time allocation, discharge summaries frequently fail to meet expected standards, with many physicians reporting shortcomings in both their completeness and quality \\[3\\], \\[4\\]. The adoption of electronic medical records has prompted efforts to alleviate this burden by transitioning from fully manual authorship toward partial automation, enabling selected sections of discharge letters to be generated automatically. However, the hospital course-the most clinically informative component of the discharge letter-remains almost universally dependent on manual narrative documentation by the treating physician. B. Objectives The primary objective of this study is to evaluate whether LLM-assisted, specialist-edited generation of hospital course summaries is safe for use in routine inpatient care, compared with standard clinician-written documentation. Secondary objectives include evaluating the safety of resident-edited and unedited LLM-generated hospital course summaries. Additional objectives are to assess the quality of LLM-assisted summaries across predefined domains, evaluate the stability of generated outputs, quantify the impact of AI-assisted workflows on physician documentation time, examine clinician adoption of LLM-supported documentation following completion of the randomized phase, and assess the reliability of evaluator ratings. Collectively, these objectives aim to determine the safety and feasibility of integrating LLM-based narrative generation into real-world hospital discharge workflows."}, 'eligibilityModule': {'sex': 'ALL', 'stdAges': ['ADULT', 'OLDER_ADULT'], 'minimumAge': '18 Years', 'healthyVolunteers': False, 'eligibilityCriteria': 'Inclusion Criteria:\n\n\\- All consecutively discharged patients, including those who died during hospitalization, will be eligible for inclusion.\n\nExclusion Criteria:\n\n* No exclusion criteria'}, 'identificationModule': {'nctId': 'NCT07491068', 'acronym': 'CLEAN', 'briefTitle': 'The Accuracy and Efficacy of Large Language Model Written Hospital Course Summaries', 'organization': {'class': 'OTHER', 'fullName': 'Pavol Jozef Safarik University'}, 'officialTitle': 'Safety and Workflow Impact of Large Language Model-Assisted Hospital Course Summaries: Protocol for a Randomized, Evaluator-Blinded Non-Inferiority Trial', 'orgStudyIdInfo': {'id': '2IK2026_1'}}, 'armsInterventionsModule': {'armGroups': [{'type': 'EXPERIMENTAL', 'label': 'LLM assisted', 'description': 'The intervention consists of an LLM assisted workflow for generating the hospital course summary at discharge. The treating physician initiates discharge using an application - CorteVision Hospital Suite - connected to the hospital Informix database. The output of the model - generated draft is returned to the application interface, where the treating physician reviews and may edit, correct, expand, or shorten the text before finalization. The finalized hospital course summary is entered into the medical record only after physician review and confirmation.', 'interventionNames': ['Device: LLM assisted workflow for generating the hospital course summary']}, {'type': 'NO_INTERVENTION', 'label': 'Control', 'description': 'Standard, manual generation of hospital course summary manually by responsible physician.'}], 'interventions': [{'name': 'LLM assisted workflow for generating the hospital course summary', 'type': 'DEVICE', 'description': 'The intervention consists of an LLM assisted workflow for generating the hospital course summary at discharge. The treating physician initiates discharge using an application - CorteVision Hospital Suite - connected to the hospital Informix database. The output of the model - generated draft is returned to the application interface, where the treating physician reviews and may edit, correct, expand, or shorten the text before finalization. The finalized hospital course summary is entered into the medical record only after physician review and confirmation.', 'armGroupLabels': ['LLM assisted']}]}, 'contactsLocationsModule': {'locations': [{'zip': '04001', 'city': 'Košice', 'state': 'Košice Region', 'country': 'Slovakia', 'facility': 'Louis Pasteur University Hospital Kosice', 'geoPoint': {'lat': 48.71441, 'lon': 21.25802}}], 'centralContacts': [{'name': 'Jakub Gazda, MD, PhD', 'role': 'CONTACT', 'email': 'jakub.gazda@upjs.sk', 'phone': '+421556403517'}, {'name': 'Martin Janicko, MD, PhD', 'role': 'CONTACT', 'email': 'martin.janicko@upjs.sk', 'phone': '+421556403527'}], 'overallOfficials': [{'name': 'Jakub Gazda, MD, PhD', 'role': 'PRINCIPAL_INVESTIGATOR', 'affiliation': 'Pavol Jozef Safarik University'}]}, 'ipdSharingStatementModule': {'ipdSharing': 'UNDECIDED'}, 'sponsorCollaboratorsModule': {'leadSponsor': {'name': 'Pavol Jozef Safarik University', 'class': 'OTHER'}, 'collaborators': [{'name': 'Louis Pasteur University Hospital, Košice', 'class': 'UNKNOWN'}], 'responsibleParty': {'type': 'PRINCIPAL_INVESTIGATOR', 'investigatorTitle': 'Associate professor', 'investigatorFullName': 'Martin Janičko', 'investigatorAffiliation': 'Pavol Jozef Safarik University'}}}}