Ignite Creation Date:
2025-12-24 @ 11:47 PM
Ignite Modification Date:
2025-12-31 @ 6:24 PM
Study NCT ID:
NCT05009251
Status:
COMPLETED
Last Update Posted:
2025-01-03
First Post:
2021-08-04
Is NOT Gene Therapy:
True
Has Adverse Events:
True
Brief Title:
Using Explainable AI Risk Predictions to Nudge Influenza Vaccine Uptake
Sponsor:
Organization:
Raw JSON
{'hasResults': True, 'derivedSection': {'miscInfoModule': {'versionHolder': '2025-12-24'}, 'conditionBrowseModule': {'meshes': [{'id': 'D007251', 'term': 'Influenza, Human'}, {'id': 'D015438', 'term': 'Health Behavior'}, {'id': 'D040242', 'term': 'Risk Reduction Behavior'}], 'ancestors': [{'id': 'D012141', 'term': 'Respiratory Tract Infections'}, {'id': 'D007239', 'term': 'Infections'}, {'id': 'D009976', 'term': 'Orthomyxoviridae Infections'}, {'id': 'D012327', 'term': 'RNA Virus Infections'}, {'id': 'D014777', 'term': 'Virus Diseases'}, {'id': 'D012140', 'term': 'Respiratory Tract Diseases'}, {'id': 'D001519', 'term': 'Behavior'}]}, 'interventionBrowseModule': {'meshes': [{'id': 'D061366', 'term': 'Numbers Needed To Treat'}], 'ancestors': [{'id': 'D018401', 'term': 'Sample Size'}, {'id': 'D012107', 'term': 'Research Design'}, {'id': 'D008722', 'term': 'Methods'}, {'id': 'D008919', 'term': 'Investigative Techniques'}]}}, 'resultsSection': {'moreInfoModule': {'pointOfContact': {'email': 'grosenbaum@geisinger.edu', 'phone': '800-275-6401', 'title': 'Gail Rosenbaum', 'organization': 'Geisinger'}, 'certainAgreement': {'piSponsorEmployee': False, 'restrictiveAgreement': False}}, 'adverseEventsModule': {'timeFrame': 'Roughly 1 month (9/9/21 - 10/7/21)', 'description': "This is a minimal risk study, and has received expedited IRB review in keeping with 45 CFR ยง 46.110. EHR and claims data will report inputs (i.e., the subject's assigned arm) and outcomes (i.e., flu vaccination, flu diagnosis, flu-like symptoms). We therefore did not monitor the data for AE/SAEs. However, on 10/7/2021, we completed a data pull with an updated death record check, and patients who died during the study as of that data pull are indicated under All-Cause Mortality.", 'eventGroups': [{'id': 'EG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.", 'otherNumAtRisk': 0, 'deathsNumAtRisk': 9017, 'otherNumAffected': 0, 'seriousNumAtRisk': 0, 'deathsNumAffected': 25, 'seriousNumAffected': 0}, {'id': 'EG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message', 'otherNumAtRisk': 0, 'deathsNumAtRisk': 9013, 'otherNumAffected': 0, 'seriousNumAtRisk': 0, 'deathsNumAffected': 22, 'seriousNumAffected': 0}, {'id': 'EG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message', 'otherNumAtRisk': 0, 'deathsNumAtRisk': 9011, 'otherNumAffected': 0, 'seriousNumAtRisk': 0, 'deathsNumAffected': 24, 'seriousNumAffected': 0}, {'id': 'EG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message', 'otherNumAtRisk': 0, 'deathsNumAtRisk': 9003, 'otherNumAffected': 0, 'seriousNumAtRisk': 0, 'deathsNumAffected': 19, 'seriousNumAffected': 0}, {'id': 'EG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message', 'otherNumAtRisk': 0, 'deathsNumAtRisk': 9017, 'otherNumAffected': 0, 'seriousNumAtRisk': 0, 'deathsNumAffected': 17, 'seriousNumAffected': 0}], 'frequencyThreshold': '0'}, 'outcomeMeasuresModule': {'outcomeMeasures': [{'type': 'PRIMARY', 'title': 'Flu Vaccination at 2 Weeks After Final Outreach Date', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8466', 'groupId': 'OG000'}, {'value': '8522', 'groupId': 'OG001'}, {'value': '8498', 'groupId': 'OG002'}, {'value': '8470', 'groupId': 'OG003'}, {'value': '8525', 'groupId': 'OG004'}]}], 'groups': [{'id': 'OG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'OG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'OG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'classes': [{'categories': [{'title': 'Not Vaccinated', 'measurements': [{'value': '5798', 'groupId': 'OG000'}, {'value': '5715', 'groupId': 'OG001'}, {'value': '5560', 'groupId': 'OG002'}, {'value': '5583', 'groupId': 'OG003'}, {'value': '5662', 'groupId': 'OG004'}]}, {'title': 'Vaccinated', 'measurements': [{'value': '2668', 'groupId': 'OG000'}, {'value': '2807', 'groupId': 'OG001'}, {'value': '2938', 'groupId': 'OG002'}, {'value': '2887', 'groupId': 'OG003'}, {'value': '2863', 'groupId': 'OG004'}]}]}], 'analyses': [{'pValue': '.054', 'groupIds': ['OG001', 'OG002', 'OG003', 'OG004'], 'groupDescription': 'This analysis compared groups that were informed they were high risk (High Risk Only, High Risk Based on Medical Records, High Risk Based on Algorithm) to Reminder Control patients who were sent messages that did not disclose their risk status. Null hypothesis: messages that inform patients they are high risk do not increase flu vaccination rate vs. messages that do not disclose risk status; Alternative hypothesis: messages that inform patients they are high risk increase flu vaccination rate.', 'statisticalMethod': 'Regression, Linear', 'nonInferiorityType': 'SUPERIORITY'}, {'pValue': '<.001', 'groupIds': ['OG000', 'OG002', 'OG003', 'OG004'], 'groupDescription': 'This analysis compared groups that were informed they were high risk (High Risk Only, High Risk Based on Medical Records, High Risk Based on Algorithm) to No-Contact Control patients who were not sent messages. Null hypothesis: messages that inform patients they are high risk do not increase flu vaccination rate relative to no messages; Alternative hypothesis: messages that inform patients they are high risk increase flu vaccination rate.', 'statisticalMethod': 'Regression, Linear', 'nonInferiorityType': 'SUPERIORITY'}, {'pValue': '.240', 'groupIds': ['OG002', 'OG003', 'OG004'], 'groupDescription': "This analysis compared groups who received messages including risk reasons (High Risk Based on Medical Records, High Risk Based on Algorithm) to messages that mentioned patients' risk status but do not include reasons (High Risk Only). Null hypothesis: messages that include risk reasons do not increase flu vaccination rate relative to high-risk messages that do not include risk reasons; Alternative hypothesis: messages including risk reasons are more effective at increasing vaccination rates.", 'statisticalMethod': 'Regression, Linear', 'nonInferiorityType': 'SUPERIORITY'}, {'pValue': '.047', 'groupIds': ['OG000', 'OG001'], 'groupDescription': 'Null hypothesis: flu-shot messages that do not mention high-risk status do not increase vaccination relative to no messages; Alternative hypothesis: flu shot messages that do not mention high-risk status increase flu shots relative to no messages.', 'statisticalMethod': 'Regression, Linear', 'nonInferiorityType': 'SUPERIORITY'}, {'pValue': '.781', 'groupIds': ['OG002', 'OG003'], 'pValueComment': "Pairwise comparisons between high-risk messages (Analyses 5, 6 and 7) were analyzed in the same regression. Reported p-values were adjusted using Tukey's HSD test.", 'groupDescription': 'Null hypothesis: High Risk Only and High Risk Based on Medical Records messages are equally effective at promoting flu vaccinations; Alternative hypothesis: there is a difference in effectiveness between High Risk Only and High Risk Based on Medical Records.', 'statisticalMethod': 'Regression, Linear', 'nonInferiorityType': 'SUPERIORITY'}, {'pValue': '.361', 'groupIds': ['OG002', 'OG004'], 'pValueComment': "Pairwise comparisons between high-risk messages (Analyses 5, 6 and 7) were analyzed in the same regression. Reported p-values were adjusted using Tukey's HSD test.", 'groupDescription': 'Null hypothesis: High Risk Only and High Risk Based on Algorithm messages are equally effective at promoting flu vaccinations; Alternative hypothesis: there is a difference in effectiveness between High Risk Only and High Risk Based on Algorithm.', 'statisticalMethod': 'Regression, Linear', 'nonInferiorityType': 'SUPERIORITY'}, {'pValue': '.770', 'groupIds': ['OG003', 'OG004'], 'pValueComment': "Pairwise comparisons between high-risk messages (Analyses 5, 6 and 7) were analyzed in the same regression. Reported p-values were adjusted using Tukey's HSD test.", 'groupDescription': 'Null hypothesis: High Risk Based on Medical Records and High Risk Based on Algorithm messages are equally effective at promoting flu vaccinations; Alternative hypothesis: there is a difference in effectiveness between High Risk Based on Medical Records and High Risk Based on Algorithm.', 'statisticalMethod': 'Regression, Linear', 'nonInferiorityType': 'SUPERIORITY'}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'timeFrame': 'Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start', 'description': 'Received flu vaccination', 'unitOfMeasure': 'Participants', 'reportingStatus': 'POSTED', 'populationDescription': 'Patients were excluded from analyses if they died before or during the study, if they were contraindicated for flu vaccine, if they were sent protective risk reasons, or if they were vaccinated prior to the study beginning. See the Statistical Analysis Plan for more information on exclusions.'}, {'type': 'SECONDARY', 'title': 'Flu Vaccination at 9 Weeks After Final Outreach Date', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8466', 'groupId': 'OG000'}, {'value': '8522', 'groupId': 'OG001'}, {'value': '8498', 'groupId': 'OG002'}, {'value': '8470', 'groupId': 'OG003'}, {'value': '8525', 'groupId': 'OG004'}]}], 'groups': [{'id': 'OG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'OG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'OG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'classes': [{'categories': [{'title': 'Not Vaccinated', 'measurements': [{'value': '5180', 'groupId': 'OG000'}, {'value': '5093', 'groupId': 'OG001'}, {'value': '4973', 'groupId': 'OG002'}, {'value': '4935', 'groupId': 'OG003'}, {'value': '5076', 'groupId': 'OG004'}]}, {'title': 'Vaccinated', 'measurements': [{'value': '3286', 'groupId': 'OG000'}, {'value': '3429', 'groupId': 'OG001'}, {'value': '3525', 'groupId': 'OG002'}, {'value': '3535', 'groupId': 'OG003'}, {'value': '3449', 'groupId': 'OG004'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'timeFrame': 'Within 9 weeks of the final outreach date, 106 days (15.14 weeks) after the study start', 'description': 'Received a flu vaccination', 'unitOfMeasure': 'Participants', 'reportingStatus': 'POSTED'}, {'type': 'SECONDARY', 'title': 'Flu Diagnosis', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8466', 'groupId': 'OG000'}, {'value': '8522', 'groupId': 'OG001'}, {'value': '8498', 'groupId': 'OG002'}, {'value': '8470', 'groupId': 'OG003'}, {'value': '8525', 'groupId': 'OG004'}]}], 'groups': [{'id': 'OG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'OG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'OG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'classes': [{'title': 'No flu diagnosis', 'categories': [{'measurements': [{'value': '8336', 'groupId': 'OG000'}, {'value': '8382', 'groupId': 'OG001'}, {'value': '8365', 'groupId': 'OG002'}, {'value': '8346', 'groupId': 'OG003'}, {'value': '8385', 'groupId': 'OG004'}]}]}, {'title': 'Flu diagnosis', 'categories': [{'measurements': [{'value': '130', 'groupId': 'OG000'}, {'value': '140', 'groupId': 'OG001'}, {'value': '133', 'groupId': 'OG002'}, {'value': '124', 'groupId': 'OG003'}, {'value': '140', 'groupId': 'OG004'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'timeFrame': '8 months (between September 9, 2021 and April 30, 2022)', 'description': 'Received a "high confidence flu" diagnosis (with positive polymerase chain reaction \\[PCR\\]/antigen/molecular test) and/or "likely flu" diagnosis (as assessed via International Classification of Disease \\[ICD\\] codes or Tamiflu administration or positive PCR/antigen/molecular test)\n\nNote that "likely flu" is a superset of the "high confidence flu" diagnoses.', 'unitOfMeasure': 'Participants', 'reportingStatus': 'POSTED'}, {'type': 'SECONDARY', 'title': 'Flu Complications', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8466', 'groupId': 'OG000'}, {'value': '8522', 'groupId': 'OG001'}, {'value': '8498', 'groupId': 'OG002'}, {'value': '8470', 'groupId': 'OG003'}, {'value': '8525', 'groupId': 'OG004'}]}], 'groups': [{'id': 'OG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'OG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'OG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'classes': [{'categories': [{'title': 'No flu complications', 'measurements': [{'value': '5916', 'groupId': 'OG000'}, {'value': '5904', 'groupId': 'OG001'}, {'value': '5910', 'groupId': 'OG002'}, {'value': '5861', 'groupId': 'OG003'}, {'value': '5853', 'groupId': 'OG004'}]}, {'title': 'Flu complications', 'measurements': [{'value': '2550', 'groupId': 'OG000'}, {'value': '2618', 'groupId': 'OG001'}, {'value': '2588', 'groupId': 'OG002'}, {'value': '2609', 'groupId': 'OG003'}, {'value': '2672', 'groupId': 'OG004'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'timeFrame': '11 months (between September 9, 2021 and July 31, 2022)', 'description': 'Diagnosed with flu-related complications', 'unitOfMeasure': 'Participants', 'reportingStatus': 'POSTED'}, {'type': 'SECONDARY', 'title': 'Healthcare Utilization', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8466', 'groupId': 'OG000'}, {'value': '8522', 'groupId': 'OG001'}, {'value': '8498', 'groupId': 'OG002'}, {'value': '8470', 'groupId': 'OG003'}, {'value': '8525', 'groupId': 'OG004'}]}], 'groups': [{'id': 'OG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'OG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'OG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'classes': [{'categories': [{'title': 'No utilization', 'measurements': [{'value': '5203', 'groupId': 'OG000'}, {'value': '5221', 'groupId': 'OG001'}, {'value': '5205', 'groupId': 'OG002'}, {'value': '5160', 'groupId': 'OG003'}, {'value': '5311', 'groupId': 'OG004'}]}, {'title': 'Utilization', 'measurements': [{'value': '3263', 'groupId': 'OG000'}, {'value': '3301', 'groupId': 'OG001'}, {'value': '3293', 'groupId': 'OG002'}, {'value': '3310', 'groupId': 'OG003'}, {'value': '3214', 'groupId': 'OG004'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'timeFrame': '11 months (between September 9, 2021 and July 31, 2022)', 'description': 'Visited ER or was hospitalized\n\nNOTE: Our prespecified outcome was description was "Visited ER, was hospitalized, or had flu-related insurance claims." We did not receive claims data, so this outcomes includes only ER visits and hospitalizations.', 'unitOfMeasure': 'Participants', 'reportingStatus': 'POSTED'}, {'type': 'OTHER_PRE_SPECIFIED', 'title': 'Flu Vaccination at 2 Weeks After Final Outreach Date by Gender', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8466', 'groupId': 'OG000'}, {'value': '8522', 'groupId': 'OG001'}, {'value': '8498', 'groupId': 'OG002'}, {'value': '8470', 'groupId': 'OG003'}, {'value': '8525', 'groupId': 'OG004'}]}], 'groups': [{'id': 'OG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'OG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'OG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'classes': [{'title': 'Female', 'denoms': [{'units': 'Participants', 'counts': [{'value': '5670', 'groupId': 'OG000'}, {'value': '5794', 'groupId': 'OG001'}, {'value': '5746', 'groupId': 'OG002'}, {'value': '5740', 'groupId': 'OG003'}, {'value': '5816', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '1740', 'groupId': 'OG000'}, {'value': '1840', 'groupId': 'OG001'}, {'value': '1933', 'groupId': 'OG002'}, {'value': '1878', 'groupId': 'OG003'}, {'value': '1933', 'groupId': 'OG004'}]}]}, {'title': 'Male', 'denoms': [{'units': 'Participants', 'counts': [{'value': '2796', 'groupId': 'OG000'}, {'value': '2728', 'groupId': 'OG001'}, {'value': '2752', 'groupId': 'OG002'}, {'value': '2730', 'groupId': 'OG003'}, {'value': '2709', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '928', 'groupId': 'OG000'}, {'value': '967', 'groupId': 'OG001'}, {'value': '1005', 'groupId': 'OG002'}, {'value': '1009', 'groupId': 'OG003'}, {'value': '930', 'groupId': 'OG004'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'timeFrame': 'Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start', 'description': 'Received a flu vaccination', 'unitOfMeasure': 'Participants', 'reportingStatus': 'POSTED', 'populationDescription': 'Patients were excluded from analyses if they died before or during the study, if they were contraindicated for flu vaccine, if they were sent protective risk reasons, or if they were vaccinated prior to the study beginning. See the Statistical Analysis Plan for more information on exclusions.'}, {'type': 'OTHER_PRE_SPECIFIED', 'title': 'Flu Vaccination at 2 Weeks After Final Outreach Date by Race', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8466', 'groupId': 'OG000'}, {'value': '8522', 'groupId': 'OG001'}, {'value': '8498', 'groupId': 'OG002'}, {'value': '8470', 'groupId': 'OG003'}, {'value': '8525', 'groupId': 'OG004'}]}], 'groups': [{'id': 'OG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'OG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'OG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'classes': [{'title': 'American Indian or Alaska Native', 'denoms': [{'units': 'Participants', 'counts': [{'value': '11', 'groupId': 'OG000'}, {'value': '16', 'groupId': 'OG001'}, {'value': '10', 'groupId': 'OG002'}, {'value': '16', 'groupId': 'OG003'}, {'value': '15', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '2', 'groupId': 'OG000'}, {'value': '3', 'groupId': 'OG001'}, {'value': '1', 'groupId': 'OG002'}, {'value': '2', 'groupId': 'OG003'}, {'value': '7', 'groupId': 'OG004'}]}]}, {'title': 'Asian', 'denoms': [{'units': 'Participants', 'counts': [{'value': '26', 'groupId': 'OG000'}, {'value': '18', 'groupId': 'OG001'}, {'value': '22', 'groupId': 'OG002'}, {'value': '24', 'groupId': 'OG003'}, {'value': '26', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '8', 'groupId': 'OG000'}, {'value': '5', 'groupId': 'OG001'}, {'value': '7', 'groupId': 'OG002'}, {'value': '12', 'groupId': 'OG003'}, {'value': '7', 'groupId': 'OG004'}]}]}, {'title': 'Black', 'denoms': [{'units': 'Participants', 'counts': [{'value': '337', 'groupId': 'OG000'}, {'value': '277', 'groupId': 'OG001'}, {'value': '284', 'groupId': 'OG002'}, {'value': '316', 'groupId': 'OG003'}, {'value': '290', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '67', 'groupId': 'OG000'}, {'value': '51', 'groupId': 'OG001'}, {'value': '68', 'groupId': 'OG002'}, {'value': '61', 'groupId': 'OG003'}, {'value': '49', 'groupId': 'OG004'}]}]}, {'title': 'Native Hawaiian or Other Pacific Islander', 'denoms': [{'units': 'Participants', 'counts': [{'value': '14', 'groupId': 'OG000'}, {'value': '12', 'groupId': 'OG001'}, {'value': '10', 'groupId': 'OG002'}, {'value': '15', 'groupId': 'OG003'}, {'value': '17', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '4', 'groupId': 'OG000'}, {'value': '1', 'groupId': 'OG001'}, {'value': '1', 'groupId': 'OG002'}, {'value': '6', 'groupId': 'OG003'}, {'value': '3', 'groupId': 'OG004'}]}]}, {'title': 'Unknown', 'denoms': [{'units': 'Participants', 'counts': [{'value': '20', 'groupId': 'OG000'}, {'value': '14', 'groupId': 'OG001'}, {'value': '14', 'groupId': 'OG002'}, {'value': '13', 'groupId': 'OG003'}, {'value': '14', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '1', 'groupId': 'OG000'}, {'value': '5', 'groupId': 'OG001'}, {'value': '4', 'groupId': 'OG002'}, {'value': '5', 'groupId': 'OG003'}, {'value': '1', 'groupId': 'OG004'}]}]}, {'title': 'White', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8058', 'groupId': 'OG000'}, {'value': '8185', 'groupId': 'OG001'}, {'value': '8158', 'groupId': 'OG002'}, {'value': '8086', 'groupId': 'OG003'}, {'value': '8163', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '2586', 'groupId': 'OG000'}, {'value': '2742', 'groupId': 'OG001'}, {'value': '2857', 'groupId': 'OG002'}, {'value': '2801', 'groupId': 'OG003'}, {'value': '2796', 'groupId': 'OG004'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'timeFrame': 'Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start', 'description': 'Received Flu Vaccination', 'unitOfMeasure': 'Participants', 'reportingStatus': 'POSTED', 'populationDescription': 'Patients were excluded from analyses if they died before or during the study, if they were contraindicated for flu vaccine, if they were sent protective risk reasons, or if they were vaccinated prior to the study beginning. See the Statistical Analysis Plan for more information on exclusions.'}, {'type': 'OTHER_PRE_SPECIFIED', 'title': 'Flu Vaccination at 2 Weeks After Final Outreach Date by Ethnicity', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8466', 'groupId': 'OG000'}, {'value': '8522', 'groupId': 'OG001'}, {'value': '8498', 'groupId': 'OG002'}, {'value': '8470', 'groupId': 'OG003'}, {'value': '8525', 'groupId': 'OG004'}]}], 'groups': [{'id': 'OG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'OG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'OG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'OG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'classes': [{'title': 'Hispanic or Latino', 'denoms': [{'units': 'Participants', 'counts': [{'value': '362', 'groupId': 'OG000'}, {'value': '335', 'groupId': 'OG001'}, {'value': '355', 'groupId': 'OG002'}, {'value': '336', 'groupId': 'OG003'}, {'value': '347', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '76', 'groupId': 'OG000'}, {'value': '76', 'groupId': 'OG001'}, {'value': '68', 'groupId': 'OG002'}, {'value': '78', 'groupId': 'OG003'}, {'value': '71', 'groupId': 'OG004'}]}]}, {'title': 'Not Hispanic or Latino', 'denoms': [{'units': 'Participants', 'counts': [{'value': '8076', 'groupId': 'OG000'}, {'value': '8160', 'groupId': 'OG001'}, {'value': '8116', 'groupId': 'OG002'}, {'value': '8113', 'groupId': 'OG003'}, {'value': '8156', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '2578', 'groupId': 'OG000'}, {'value': '2722', 'groupId': 'OG001'}, {'value': '2865', 'groupId': 'OG002'}, {'value': '2804', 'groupId': 'OG003'}, {'value': '2784', 'groupId': 'OG004'}]}]}, {'title': 'Unable to Obtain', 'denoms': [{'units': 'Participants', 'counts': [{'value': '28', 'groupId': 'OG000'}, {'value': '27', 'groupId': 'OG001'}, {'value': '27', 'groupId': 'OG002'}, {'value': '21', 'groupId': 'OG003'}, {'value': '22', 'groupId': 'OG004'}]}], 'categories': [{'measurements': [{'value': '14', 'groupId': 'OG000'}, {'value': '9', 'groupId': 'OG001'}, {'value': '5', 'groupId': 'OG002'}, {'value': '5', 'groupId': 'OG003'}, {'value': '8', 'groupId': 'OG004'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'timeFrame': 'Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start', 'description': 'Received flu vaccination', 'unitOfMeasure': 'Participants', 'reportingStatus': 'POSTED', 'populationDescription': 'Patients were excluded from analyses if they died before or during the study, if they were contraindicated for flu vaccine, if they were sent protective risk reasons, or if they were vaccinated prior to the study beginning. See the Statistical Analysis Plan for more information on exclusions.'}]}, 'participantFlowModule': {'groups': [{'id': 'FG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'FG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'FG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'FG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'FG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}], 'periods': [{'title': 'Overall Study', 'milestones': [{'type': 'STARTED', 'achievements': [{'groupId': 'FG000', 'numSubjects': '9017'}, {'groupId': 'FG001', 'numSubjects': '9013'}, {'groupId': 'FG002', 'numSubjects': '9011'}, {'groupId': 'FG003', 'numSubjects': '9003'}, {'groupId': 'FG004', 'numSubjects': '9017'}]}, {'type': 'COMPLETED', 'achievements': [{'groupId': 'FG000', 'numSubjects': '8892'}, {'groupId': 'FG001', 'numSubjects': '8886'}, {'groupId': 'FG002', 'numSubjects': '8868'}, {'groupId': 'FG003', 'numSubjects': '8864'}, {'groupId': 'FG004', 'numSubjects': '8883'}]}, {'type': 'NOT COMPLETED', 'achievements': [{'groupId': 'FG000', 'numSubjects': '125'}, {'groupId': 'FG001', 'numSubjects': '127'}, {'groupId': 'FG002', 'numSubjects': '143'}, {'groupId': 'FG003', 'numSubjects': '139'}, {'groupId': 'FG004', 'numSubjects': '134'}]}], 'dropWithdraws': [{'type': 'Protocol Violation', 'reasons': [{'groupId': 'FG000', 'numSubjects': '100'}, {'groupId': 'FG001', 'numSubjects': '105'}, {'groupId': 'FG002', 'numSubjects': '119'}, {'groupId': 'FG003', 'numSubjects': '120'}, {'groupId': 'FG004', 'numSubjects': '119'}]}, {'type': 'Death', 'reasons': [{'groupId': 'FG000', 'numSubjects': '25'}, {'groupId': 'FG001', 'numSubjects': '22'}, {'groupId': 'FG002', 'numSubjects': '24'}, {'groupId': 'FG003', 'numSubjects': '19'}, {'groupId': 'FG004', 'numSubjects': '15'}]}]}]}, 'baselineCharacteristicsModule': {'denoms': [{'units': 'Participants', 'counts': [{'value': '9017', 'groupId': 'BG000'}, {'value': '9013', 'groupId': 'BG001'}, {'value': '9011', 'groupId': 'BG002'}, {'value': '9003', 'groupId': 'BG003'}, {'value': '9017', 'groupId': 'BG004'}, {'value': '45061', 'groupId': 'BG005'}]}], 'groups': [{'id': 'BG000', 'title': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'id': 'BG001', 'title': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message'}, {'id': 'BG002', 'title': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message'}, {'id': 'BG003', 'title': 'High Risk With Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'BG004', 'title': 'High Risk With Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.\n\nReminder: Mailed letter, short message service (SMS) text, and/or patient portal message\n\nRisk reduction: Mailed letter, SMS, and/or patient portal message\n\nMedical records-based recommendation: Mailed letter, SMS, and/or patient portal message\n\nAlgorithm-based recommendation: Mailed letter, SMS, and/or patient portal message'}, {'id': 'BG005', 'title': 'Total', 'description': 'Total of all reporting groups'}], 'measures': [{'title': 'Age, Customized', 'classes': [{'categories': [{'title': '>=18 years and <65', 'measurements': [{'value': '6018', 'groupId': 'BG000'}, {'value': '5984', 'groupId': 'BG001'}, {'value': '6049', 'groupId': 'BG002'}, {'value': '6001', 'groupId': 'BG003'}, {'value': '6004', 'groupId': 'BG004'}, {'value': '30056', 'groupId': 'BG005'}]}, {'title': '>=65', 'measurements': [{'value': '2999', 'groupId': 'BG000'}, {'value': '3029', 'groupId': 'BG001'}, {'value': '2962', 'groupId': 'BG002'}, {'value': '3002', 'groupId': 'BG003'}, {'value': '3013', 'groupId': 'BG004'}, {'value': '15005', 'groupId': 'BG005'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'unitOfMeasure': 'Participants'}, {'title': 'Sex: Female, Male', 'classes': [{'categories': [{'title': 'Female', 'measurements': [{'value': '6038', 'groupId': 'BG000'}, {'value': '6121', 'groupId': 'BG001'}, {'value': '6107', 'groupId': 'BG002'}, {'value': '6086', 'groupId': 'BG003'}, {'value': '6140', 'groupId': 'BG004'}, {'value': '30492', 'groupId': 'BG005'}]}, {'title': 'Male', 'measurements': [{'value': '2979', 'groupId': 'BG000'}, {'value': '2892', 'groupId': 'BG001'}, {'value': '2904', 'groupId': 'BG002'}, {'value': '2917', 'groupId': 'BG003'}, {'value': '2877', 'groupId': 'BG004'}, {'value': '14569', 'groupId': 'BG005'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'unitOfMeasure': 'Participants'}, {'title': 'Ethnicity (NIH/OMB)', 'classes': [{'categories': [{'title': 'Hispanic or Latino', 'measurements': [{'value': '381', 'groupId': 'BG000'}, {'value': '351', 'groupId': 'BG001'}, {'value': '374', 'groupId': 'BG002'}, {'value': '353', 'groupId': 'BG003'}, {'value': '365', 'groupId': 'BG004'}, {'value': '1824', 'groupId': 'BG005'}]}, {'title': 'Not Hispanic or Latino', 'measurements': [{'value': '8608', 'groupId': 'BG000'}, {'value': '8635', 'groupId': 'BG001'}, {'value': '8607', 'groupId': 'BG002'}, {'value': '8627', 'groupId': 'BG003'}, {'value': '8630', 'groupId': 'BG004'}, {'value': '43107', 'groupId': 'BG005'}]}, {'title': 'Unknown or Not Reported', 'measurements': [{'value': '28', 'groupId': 'BG000'}, {'value': '27', 'groupId': 'BG001'}, {'value': '30', 'groupId': 'BG002'}, {'value': '23', 'groupId': 'BG003'}, {'value': '22', 'groupId': 'BG004'}, {'value': '130', 'groupId': 'BG005'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'unitOfMeasure': 'Participants'}, {'title': 'Race/Ethnicity, Customized', 'classes': [{'categories': [{'title': 'American Indian or Alaska Native', 'measurements': [{'value': '11', 'groupId': 'BG000'}, {'value': '19', 'groupId': 'BG001'}, {'value': '14', 'groupId': 'BG002'}, {'value': '16', 'groupId': 'BG003'}, {'value': '16', 'groupId': 'BG004'}, {'value': '76', 'groupId': 'BG005'}]}, {'title': 'Asian', 'measurements': [{'value': '27', 'groupId': 'BG000'}, {'value': '21', 'groupId': 'BG001'}, {'value': '23', 'groupId': 'BG002'}, {'value': '26', 'groupId': 'BG003'}, {'value': '27', 'groupId': 'BG004'}, {'value': '124', 'groupId': 'BG005'}]}, {'title': 'Native Hawaiian or Other Pacific Islander', 'measurements': [{'value': '15', 'groupId': 'BG000'}, {'value': '12', 'groupId': 'BG001'}, {'value': '12', 'groupId': 'BG002'}, {'value': '16', 'groupId': 'BG003'}, {'value': '20', 'groupId': 'BG004'}, {'value': '75', 'groupId': 'BG005'}]}, {'title': 'Black or African American', 'measurements': [{'value': '348', 'groupId': 'BG000'}, {'value': '286', 'groupId': 'BG001'}, {'value': '300', 'groupId': 'BG002'}, {'value': '330', 'groupId': 'BG003'}, {'value': '300', 'groupId': 'BG004'}, {'value': '1564', 'groupId': 'BG005'}]}, {'title': 'White', 'measurements': [{'value': '8596', 'groupId': 'BG000'}, {'value': '8660', 'groupId': 'BG001'}, {'value': '8648', 'groupId': 'BG002'}, {'value': '8602', 'groupId': 'BG003'}, {'value': '8636', 'groupId': 'BG004'}, {'value': '43142', 'groupId': 'BG005'}]}, {'title': 'Unknown or Not Reported', 'measurements': [{'value': '20', 'groupId': 'BG000'}, {'value': '15', 'groupId': 'BG001'}, {'value': '14', 'groupId': 'BG002'}, {'value': '13', 'groupId': 'BG003'}, {'value': '18', 'groupId': 'BG004'}, {'value': '80', 'groupId': 'BG005'}]}]}], 'paramType': 'COUNT_OF_PARTICIPANTS', 'unitOfMeasure': 'Participants'}]}}, 'documentSection': {'largeDocumentModule': {'largeDocs': [{'date': '2021-06-28', 'size': 173113, 'label': 'Study Protocol', 'hasIcf': False, 'hasSap': False, 'filename': 'Prot_001.pdf', 'typeAbbrev': 'Prot', 'uploadDate': '2022-11-04T10:09', 'hasProtocol': True}, {'date': '2021-11-10', 'size': 71261, 'label': 'Statistical Analysis Plan', 'hasIcf': False, 'hasSap': True, 'filename': 'SAP_000.pdf', 'typeAbbrev': 'SAP', 'uploadDate': '2021-11-10T13:32', 'hasProtocol': False}]}}, 'protocolSection': {'designModule': {'phases': ['NA'], 'studyType': 'INTERVENTIONAL', 'designInfo': {'allocation': 'RANDOMIZED', 'maskingInfo': {'masking': 'SINGLE', 'whoMasked': ['CARE_PROVIDER'], 'maskingDescription': 'Providers who prescribe vaccination and diagnose conditions will not be randomized to study arms or informed of patient assignment. Although patients will not be explicitly informed which arm they have been randomized to, they will be aware of the messages they receive.'}, 'primaryPurpose': 'PREVENTION', 'interventionModel': 'PARALLEL', 'interventionModelDescription': 'Patients from the high-risk sample will be randomly assigned to one of five arms:\n\n* No-Contact Control Arm\n* Reminder Control Arm\n* High Risk Only Arm\n* High Risk with Explanation Based on Medical Records Arm\n* High Risk with Explanation Based on Algorithm Arm'}, 'enrollmentInfo': {'type': 'ACTUAL', 'count': 45061}}, 'statusModule': {'overallStatus': 'COMPLETED', 'startDateStruct': {'date': '2021-09-09', 'type': 'ACTUAL'}, 'expandedAccessInfo': {'hasExpandedAccess': False}, 'statusVerifiedDate': '2024-12', 'completionDateStruct': {'date': '2022-07-31', 'type': 'ACTUAL'}, 'lastUpdateSubmitDate': '2024-12-16', 'studyFirstSubmitDate': '2021-08-04', 'resultsFirstSubmitDate': '2022-11-04', 'studyFirstSubmitQcDate': '2021-08-14', 'lastUpdatePostDateStruct': {'date': '2025-01-03', 'type': 'ACTUAL'}, 'resultsFirstSubmitQcDate': '2022-12-20', 'studyFirstPostDateStruct': {'date': '2021-08-17', 'type': 'ACTUAL'}, 'resultsFirstPostDateStruct': {'date': '2023-01-12', 'type': 'ACTUAL'}, 'primaryCompletionDateStruct': {'date': '2021-11-05', 'type': 'ACTUAL'}}, 'outcomesModule': {'otherOutcomes': [{'measure': 'Flu Vaccination at 2 Weeks After Final Outreach Date by Gender', 'timeFrame': 'Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start', 'description': 'Received a flu vaccination'}, {'measure': 'Flu Vaccination at 2 Weeks After Final Outreach Date by Race', 'timeFrame': 'Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start', 'description': 'Received Flu Vaccination'}, {'measure': 'Flu Vaccination at 2 Weeks After Final Outreach Date by Ethnicity', 'timeFrame': 'Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start', 'description': 'Received flu vaccination'}], 'primaryOutcomes': [{'measure': 'Flu Vaccination at 2 Weeks After Final Outreach Date', 'timeFrame': 'Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start', 'description': 'Received flu vaccination'}], 'secondaryOutcomes': [{'measure': 'Flu Vaccination at 9 Weeks After Final Outreach Date', 'timeFrame': 'Within 9 weeks of the final outreach date, 106 days (15.14 weeks) after the study start', 'description': 'Received a flu vaccination'}, {'measure': 'Flu Diagnosis', 'timeFrame': '8 months (between September 9, 2021 and April 30, 2022)', 'description': 'Received a "high confidence flu" diagnosis (with positive polymerase chain reaction \\[PCR\\]/antigen/molecular test) and/or "likely flu" diagnosis (as assessed via International Classification of Disease \\[ICD\\] codes or Tamiflu administration or positive PCR/antigen/molecular test)\n\nNote that "likely flu" is a superset of the "high confidence flu" diagnoses.'}, {'measure': 'Flu Complications', 'timeFrame': '11 months (between September 9, 2021 and July 31, 2022)', 'description': 'Diagnosed with flu-related complications'}, {'measure': 'Healthcare Utilization', 'timeFrame': '11 months (between September 9, 2021 and July 31, 2022)', 'description': 'Visited ER or was hospitalized\n\nNOTE: Our prespecified outcome was description was "Visited ER, was hospitalized, or had flu-related insurance claims." We did not receive claims data, so this outcomes includes only ER visits and hospitalizations.'}]}, 'oversightModule': {'oversightHasDmc': True, 'isFdaRegulatedDrug': False, 'isFdaRegulatedDevice': False}, 'conditionsModule': {'keywords': ['Flu Vaccine', 'Choice Architecture', 'Machine Learning', 'Perceived Credibility', 'Personalized Risk Factors', 'Flu Complications'], 'conditions': ['Influenza', 'Vaccination', 'Health Promotion', 'Health Behavior', 'Risk Reduction']}, 'descriptionModule': {'briefSummary': 'The study team previously demonstrated that patients are more likely to receive flu vaccine after learning that they are at high risk for flu complications. Building on this past work, the present study will explore whether providing reasons that patients are considered high risk for flu complications (a) further increases the likelihood they will receive flu vaccine and (b) decreases the likelihood that they receive diagnoses of flu and/or flu-like symptoms in the ensuing flu season. It will also examine whether informing patients that their high-risk status was determined by analyzing their medical records or by an artificial intelligence (AI) / machine-learning (ML) algorithm analyzing their medical records will affect the likelihood of receiving the flu vaccine or diagnoses of flu and/or flu-like symptoms.', 'detailedDescription': 'Geisinger has partnered with Medial EarlySign and developed an ML algorithm to identify patients at risk for serious (moderate to severe) flu-associated complications on the basis of their existing electronic health record (EHR) data. Geisinger will apply this algorithm to current patients during the 2021-22 flu season.\n\nThis study will evaluate the effect of contacting patients identified as high risk with special messages to encourage vaccination. These communications will inform patients they are at high risk with either (a) no additional explanation, (b) an explanation that this determination comes from an analysis of their medical records, along with a short list of the top factors from their medical record that explain their risk, and (c) the additional explanation that an AI or ML algorithm made this determination, along with a short list of the top factors from their medical record that explain their risk.\n\nIncluded in the study will be current Geisinger patients 18+ years of age with no contraindications for flu vaccine and who have been assessed by the Medial algorithm and assigned a risk score. The primary study outcomes will be the rates of flu vaccination and flu diagnosis during the 2020-21 season by targeted patients.'}, 'eligibilityModule': {'sex': 'ALL', 'stdAges': ['ADULT', 'OLDER_ADULT'], 'minimumAge': '18 Years', 'healthyVolunteers': False, 'eligibilityCriteria': 'Inclusion Criteria:\n\n* Aged 18 or older\n* Current Geisinger patient at the time of study\n* Falls in the top 10% of patients at highest risk, as identified by the flu-complication risk scores of machine learning algorithm (which operates on coded EHR data)\n\nExclusion Criteria:\n\n* Has contraindications for flu vaccination\n* Has opted out of receiving communications from Geisinger via all of the modalities being tested'}, 'identificationModule': {'nctId': 'NCT05009251', 'briefTitle': 'Using Explainable AI Risk Predictions to Nudge Influenza Vaccine Uptake', 'organization': {'class': 'OTHER', 'fullName': 'National Bureau of Economic Research, Inc.'}, 'officialTitle': 'Using Explainable AI Risk Predictions to Nudge Influenza Vaccine Uptake', 'orgStudyIdInfo': {'id': '2021-0483'}, 'secondaryIdInfos': [{'id': 'P30AG034532', 'link': 'https://reporter.nih.gov/quickSearch/P30AG034532', 'type': 'NIH'}]}, 'armsInterventionsModule': {'armGroups': [{'type': 'NO_INTERVENTION', 'label': 'No-Contact Control', 'description': "Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted."}, {'type': 'EXPERIMENTAL', 'label': 'Reminder Control', 'description': 'Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.', 'interventionNames': ['Behavioral: Reminder']}, {'type': 'EXPERIMENTAL', 'label': 'High Risk Only', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.', 'interventionNames': ['Behavioral: Reminder', 'Behavioral: Risk reduction']}, {'type': 'EXPERIMENTAL', 'label': 'High Risk with Explanation Based on Medical Records', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.', 'interventionNames': ['Behavioral: Reminder', 'Behavioral: Risk reduction', 'Behavioral: Medical records-based recommendation']}, {'type': 'EXPERIMENTAL', 'label': 'High Risk with Explanation Based on Algorithm', 'description': 'Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.', 'interventionNames': ['Behavioral: Reminder', 'Behavioral: Risk reduction', 'Behavioral: Medical records-based recommendation', 'Behavioral: Algorithm-based recommendation']}], 'interventions': [{'name': 'Reminder', 'type': 'BEHAVIORAL', 'description': 'Mailed letter, short message service (SMS) text, and/or patient portal message', 'armGroupLabels': ['High Risk Only', 'High Risk with Explanation Based on Algorithm', 'High Risk with Explanation Based on Medical Records', 'Reminder Control']}, {'name': 'Risk reduction', 'type': 'BEHAVIORAL', 'description': 'Mailed letter, SMS, and/or patient portal message', 'armGroupLabels': ['High Risk Only', 'High Risk with Explanation Based on Algorithm', 'High Risk with Explanation Based on Medical Records']}, {'name': 'Medical records-based recommendation', 'type': 'BEHAVIORAL', 'otherNames': ['Credibility'], 'description': 'Mailed letter, SMS, and/or patient portal message', 'armGroupLabels': ['High Risk with Explanation Based on Algorithm', 'High Risk with Explanation Based on Medical Records']}, {'name': 'Algorithm-based recommendation', 'type': 'BEHAVIORAL', 'otherNames': ['Credibility'], 'description': 'Mailed letter, SMS, and/or patient portal message', 'armGroupLabels': ['High Risk with Explanation Based on Algorithm']}]}, 'contactsLocationsModule': {'locations': [{'zip': '17822', 'city': 'Danville', 'state': 'Pennsylvania', 'country': 'United States', 'facility': 'Geisinger Clinic', 'geoPoint': {'lat': 40.96342, 'lon': -76.61273}}], 'overallOfficials': [{'name': 'Michelle N Meyer, PhD JD', 'role': 'PRINCIPAL_INVESTIGATOR', 'affiliation': 'Geisinger Clinic'}, {'name': 'Christopher F Chabris, PhD', 'role': 'PRINCIPAL_INVESTIGATOR', 'affiliation': 'Geisinger Clinic'}]}, 'ipdSharingStatementModule': {'url': 'https://osf.io/mqbux/', 'infoTypes': ['STUDY_PROTOCOL', 'SAP', 'CSR', 'ANALYTIC_CODE'], 'timeFrame': "By the paper's online publication date. Data will remain available for as long as the Open Science Framework hosts it.", 'ipdSharing': 'YES', 'description': 'Data with no personally identifiable information will be made available to other researchers on the Open Science Framework for transparency. This will include the essential data and code needed to replicate the analysis that yielded reported findings.', 'accessCriteria': 'The data and code are available in the below repository in the folder ClinicalTrials.gov data/Study 2.'}, 'sponsorCollaboratorsModule': {'leadSponsor': {'name': 'National Bureau of Economic Research, Inc.', 'class': 'OTHER'}, 'collaborators': [{'name': 'Geisinger Clinic', 'class': 'OTHER'}, {'name': 'Massachusetts Institute of Technology', 'class': 'OTHER'}, {'name': 'National Institute on Aging (NIA)', 'class': 'NIH'}], 'responsibleParty': {'type': 'SPONSOR'}}}}