Ignite Creation Date:
2025-12-25 @ 4:07 AM
Ignite Modification Date:
2026-02-24 @ 4:04 AM
Study NCT ID:
NCT06434220
Status:
NOT_YET_RECRUITING
Last Update Posted:
2024-08-09
First Post:
2024-05-20
Is NOT Gene Therapy:
True
Has Adverse Events:
False
Brief Title:
Effect of Predictive Model on ED Physician Assessments of Patient Disposition
Sponsor:
Organization:
Raw JSON
{'hasResults': False, 'derivedSection': {'miscInfoModule': {'versionHolder': '2025-12-24'}}, 'protocolSection': {'designModule': {'phases': ['NA'], 'studyType': 'INTERVENTIONAL', 'designInfo': {'allocation': 'RANDOMIZED', 'maskingInfo': {'masking': 'TRIPLE', 'whoMasked': ['PARTICIPANT', 'CARE_PROVIDER', 'INVESTIGATOR'], 'maskingDescription': 'We conduct a randomized, double-blind, controlled before-after (CBA) study of board-certified ED attending physicians not currently caring for patients in the BCH ED over a period of six to twelve weeks. In this experiment, the "treatment" consists of an ML recommendation provided to the ED physicians, who predict admission decisions for individual patients before and after receiving it. The "control" surveys receive the original POPP model recommendation, and "treatment" surveys receive a "fairness-aware" model, determined in prior work to mitigate biases in performance with respect to patient demographics'}, 'primaryPurpose': 'OTHER', 'interventionModel': 'SEQUENTIAL'}, 'enrollmentInfo': {'type': 'ESTIMATED', 'count': 10}}, 'statusModule': {'overallStatus': 'NOT_YET_RECRUITING', 'startDateStruct': {'date': '2026-05-01', 'type': 'ESTIMATED'}, 'expandedAccessInfo': {'hasExpandedAccess': False}, 'statusVerifiedDate': '2024-08', 'completionDateStruct': {'date': '2026-09-01', 'type': 'ESTIMATED'}, 'lastUpdateSubmitDate': '2024-08-08', 'studyFirstSubmitDate': '2024-05-20', 'studyFirstSubmitQcDate': '2024-05-23', 'lastUpdatePostDateStruct': {'date': '2024-08-09', 'type': 'ACTUAL'}, 'studyFirstPostDateStruct': {'date': '2024-05-30', 'type': 'ACTUAL'}, 'primaryCompletionDateStruct': {'date': '2026-07-01', 'type': 'ESTIMATED'}}, 'outcomesModule': {'primaryOutcomes': [{'measure': 'Physician-assessed ED disposition (likelihood of admission)', 'timeFrame': 'Within 24 hours of survey', 'description': 'The primary outcome is physician-assessed ED disposition (categorized as admission or discharge), before and after viewing a model prediction, compared to final disposition of patient'}, {'measure': 'Patient final disposition (admitted/discharged)', 'timeFrame': 'Within 24 hours of survey', 'description': 'The final disposition of the patient, whether admitted to an inpatient service or discharged'}], 'secondaryOutcomes': [{'measure': 'Model-assessed ED disposition', 'timeFrame': 'Within 24 hours of survey', 'description': "The model prediction's assessment of ED disposition compared to final disposition of patient"}]}, 'oversightModule': {'oversightHasDmc': False, 'isFdaRegulatedDrug': False, 'isFdaRegulatedDevice': False}, 'conditionsModule': {'conditions': ['Patient Outcome Assessment']}, 'referencesModule': {'references': [{'pmid': '28188202', 'type': 'BACKGROUND', 'citation': 'Barak-Corren Y, Israelit SH, Reis BY. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg Med J. 2017 May;34(5):308-314. doi: 10.1136/emermed-2014-203819. Epub 2017 Feb 10.'}, {'pmid': '34010406', 'type': 'BACKGROUND', 'citation': 'Barak-Corren Y, Agarwal I, Michelson KA, Lyons TW, Neuman MI, Lipsett SC, Kimia AA, Eisenberg MA, Capraro AJ, Levy JA, Hudgins JD, Reis BY, Fine AM. Prediction of patient disposition: comparison of computer and human approaches and a proposed synthesis. J Am Med Inform Assoc. 2021 Jul 30;28(8):1736-1745. doi: 10.1093/jamia/ocab076.'}, {'pmid': '34912043', 'type': 'BACKGROUND', 'citation': 'Barak-Corren Y, Chaudhari P, Perniciaro J, Waltzman M, Fine AM, Reis BY. Prediction across healthcare settings: a case study in predicting emergency department disposition. NPJ Digit Med. 2021 Dec 15;4(1):169. doi: 10.1038/s41746-021-00537-x.'}, {'pmid': '28557729', 'type': 'BACKGROUND', 'citation': 'Barak-Corren Y, Fine AM, Reis BY. Early Prediction Model of Patient Hospitalization From the Pediatric Emergency Department. Pediatrics. 2017 May;139(5):e20162785. doi: 10.1542/peds.2016-2785.'}, {'pmid': '37576024', 'type': 'BACKGROUND', 'citation': 'La Cava WG, Lett E, Wan G. Fair admission risk prediction with proportional multicalibration. Proc Mach Learn Res. 2023;209:350-378.'}]}, 'descriptionModule': {'briefSummary': 'The goal of this study is to measure the impact of fairness-aware algorithms on physician predictions of ED patient admission. Using an experimentally validated machine learning model tuned for equitable outcomes, the investigators quantify the impact of model recommendations on ED physician assessments of admission risk in a silent, prospective study. The investigators survey ED physicians who are not currently caring for patients using live site data. To quantify the impact of the model on ED physician assessments of admission risk, the investigators collect physician assessments before and after consulting the (original or updated) model prediction.\n\nThe investigators measure ED physician adherence to model suggestions, along with the predictive accuracy and equity of downstream patient outcomes. The outcome of this study is an empirical measure of the extent to which fair ML models may influence admission decisions to mitigate health care disparities.', 'detailedDescription': 'Specific Aims/Objectives:\n\n1. Measure the effect of the sharing of a model prediction of admission on an attending physician\'s assessment of patient disposition within one hour from presentation at a tertiary academic pediatric hospital.\n2. Measure the effect of the sharing of a model prediction from a model tuned for equal subgroup performance on an attending physician\'s assessment of patient disposition within one hour from presentation at a tertiary academic pediatric hospital.\n\nBackground and Significance:\n\nMachine learning (ML) models increasingly provide clinical decision support (CDS) to care teams to help prioritize individuals for specific care based on their predicted health needs and outcomes. AI/ML methods can have a particularly high impact on resource allocation in emergency departments (EDs) across the U.S., which have been described by the Institute for Medicine as "nearing the breaking point" of over-capacity. Unfortunately, models often perform poorly on disinvested subpopulations relative to the population as a whole. As a result, ML models may exacerbate downstream health disparities by under-performing on marginalized patient subpopulations, especially when models are expanded to multiple care centers and or used without subgroup monitoring for long periods of time.\n\nMany prediction models have been developed in recent years to predict patient disposition from the ED, including a prediction tool developed by our group and currently in piloting stages at Boston Children\'s Hospital, South Shore Hospital, and Children\'s Hospital of Los-Angeles. Our prediction tool, the Predictor of Patient Placement (POPP) provides an accurate, real-time likelihood of admission based on data available in the electronic health record at the time of the visit. Advance notice of likely admissions can have an important impact on ED waiting and boarding times with the potential to improve flow and patient satisfaction.\n\nTo this end, the investigation team has submitted a grant proposal to the National Library of Medicine (NLM) \\[1R01LM014300 - 01A1\\] that researches the development and validation of fairness-aware prediction models of patient admission. Aim 2 of this grant studies the effect of these models on ED physician assessments of patient disposition, and corresponds to this protocol. The NLM has indicated its intention to fund this proposal and the investigators are in the process of submitting documents to finalize the award. This component of the study is slated for year 3 of the study.\n\nPreliminary Studies\n\nThe investigators conducted a series of initial retrospective studies that established that patient admission could be predicted with machine learning models ahead of time in the BCH ED, progressively during the visit, as well as across other medical centers with good accuracy (AUROC 0.9-0.93).\n\nNext, the investigators found that the accuracy of POPP in predicting admission likelihood added value to the gestalt assessments of ED attending physicians. The positive predictive value for the prediction of admission was 66% for the clinicians, 73% for POPP, and 86% for a hybrid model combining the two.\n\nFinally, the investigators developed methods for post-processing the ED prediction models to make them well-calibrated across patient demographic groups defined by race, sex, and insurance product.\n\nThe model predictions are currently used to help with bed coordination, but given their high value, may also improve decision making at the bed-side. With this study, our goal is to now test, in a simulated, safe, and realistic setting, how model recommendations affect the assessments of admission likelihood by ED attending physicians.'}, 'eligibilityModule': {'sex': 'ALL', 'stdAges': ['ADULT', 'OLDER_ADULT'], 'maximumAge': '65 Years', 'minimumAge': '18 Years', 'healthyVolunteers': True, 'eligibilityCriteria': "Inclusion Criteria:\n\n* Board certified emergency department attending physicians currently employed by Boston Children's Hospital\n\nExclusion Criteria:\n\n* Physicians are excluded from completely surveys for patients who they are currently caring for"}, 'identificationModule': {'nctId': 'NCT06434220', 'briefTitle': 'Effect of Predictive Model on ED Physician Assessments of Patient Disposition', 'organization': {'class': 'OTHER', 'fullName': "Boston Children's Hospital"}, 'officialTitle': 'Effect of Predictive Model on ED Physician Assessments of Patient Disposition', 'orgStudyIdInfo': {'id': 'IRB-P00048537'}}, 'armsInterventionsModule': {'armGroups': [{'type': 'NO_INTERVENTION', 'label': 'Physician assessment before intervention', 'description': 'No intervention. Physician is surveyed to provide their assessment of patient disposition.'}, {'type': 'ACTIVE_COMPARATOR', 'label': 'Physician assessment after baseline model intervention', 'description': 'Physician is shown a baseline model recommendation for patient disposition including description of factors driving the model prediction.', 'interventionNames': ['Diagnostic Test: Baseline model']}, {'type': 'ACTIVE_COMPARATOR', 'label': 'Physician assessment after fairness-aware model intervention', 'description': 'Physician is shown a model recommendation form a model tuned for subgroup performance for patient disposition including description of factors driving the model prediction.', 'interventionNames': ['Diagnostic Test: Fairness-aware model']}], 'interventions': [{'name': 'Baseline model', 'type': 'DIAGNOSTIC_TEST', 'description': 'Model prediction of patient disposition including feature importance scores driving prediction.', 'armGroupLabels': ['Physician assessment after baseline model intervention']}, {'name': 'Fairness-aware model', 'type': 'DIAGNOSTIC_TEST', 'description': 'Model prediction of patient disposition including feature importance scores driving prediction. This model has been tuned to minimize subgroup calibration errors.', 'armGroupLabels': ['Physician assessment after fairness-aware model intervention']}]}, 'contactsLocationsModule': {'centralContacts': [{'name': 'William La Cava, PhD', 'role': 'CONTACT', 'email': 'william.lacava@childrens.harvard.edu', 'phone': '4133200544'}, {'name': 'Andrew Fine, MD', 'role': 'CONTACT', 'email': 'andrew.fine@childrens.harvard.edu', 'phone': '617-355-9696'}]}, 'ipdSharingStatementModule': {'ipdSharing': 'NO'}, 'sponsorCollaboratorsModule': {'leadSponsor': {'name': "Boston Children's Hospital", 'class': 'OTHER'}, 'responsibleParty': {'type': 'PRINCIPAL_INVESTIGATOR', 'investigatorTitle': 'Assistant Professor', 'investigatorFullName': 'William La Cava', 'investigatorAffiliation': "Boston Children's Hospital"}}}}