Ignite Creation Date:
2024-05-06 @ 1:56 PM
Last Modification Date:
2024-10-26 @ 1:22 PM
Study NCT ID:
NCT04165213
Status:
COMPLETED
Last Update Posted:
2024-06-05
First Post:
2019-08-26
Brief Title:
Care of Persons With Dementia in Their Environments COPE in Programs of All-Inclusive Care of the Elderly PACE
Sponsor:
University of Pennsylvania
Organization:
University of Pennsylvania
Study Overview
Official Title:
A Training and Fidelity Model to Move and Scale Evidence-based Dementia Care and Caregiver Support Programs Into Practice The Case for COPE in PACE Service Settings
Status:
COMPLETED
Status Verified Date:
2024-10
Last Known Status:
None
Delayed Posting:
No
If Stopped, Why?:
Not Stopped
Has Expanded Access:
False
If Expanded Access, NCT#:
N/A
Has Expanded Access, NCT# Status:
N/A
Acronym:
None
Brief Summary:
The protocol is organized into three Phases - In Phase I an online training program will be developed in Care of Persons with Dementia in their Environments COPE -an evidence-based bio-behavioral dementia program -using state-of-the science simulation and best online learning practices In addition an automated approach to fidelity monitoring using computational linguistics automatic classification programs will be developed In Phase II ten long term care community-based PACE organizations will be randomized into two groups 5 PACE organizations will serve as the control site in which staff training will be provided via the traditional high intensity face-to-face training in the COPE program 5 PACE organizations will serve as the comparison and staff will be trained using the online COPE training program Phase II will evaluate the whether an online training program is the same or better in improving PACE staff competency and fidelity to COPE principles and protocols compared to a high intensity face-to-face traditional form of training In Phase III the efficacy of the COPE program on PACE participant outcomes by type of COPE training will be evaluated Each of the PACE organizations will enroll 5 persons with dementia and their caregivers in the study This will yield 50 family dyads 25 dyads in traditional training sites and 25 dyads in online training sites Dyads will be followed for 4 months Non-inferiority analysis will be used to assess whether dyads will yield the same or better outcomes regardless of how PACE staff were trained
Detailed Description:
PHASE 1a - The Online Training Program - We will develop ten self-paced online learning modules These modules will enable OTs and RNs to participant to have anytimeanywhere access to content and activities to aid their learning The modules will include rich multimedia content and interactive assessments to keep the learner engaged The modules will allow for easy packaging of the content into the latest interoperability standards for such content including the latest Shareable Content Object Reference Model SCORM specifications which will allow for repurposing and sharing with other institutions
To accommodate diversity of learning needs the modules will be designed using a hyperlearning model with four dimensions The general principles will begin with the module learning objectives and follow with a review of core concepts and required andor self-directed learning activities The mini-lecture component of the modules will include information on the major concepts of the module Since the modules will be self-paced the learner can take hisher time going through them and perform in the embedded interactive learning activities The clinical reasoning dimension will provide the learner with an opportunity for problem-solving and clinical decision-making This dimension will contain vignettes and case studies with questions requiring analysis and synthesis The final dimension will be evaluation assessment of learning outcomes This dimension will use teacher-made and standardized pre-and post-tests to assess attainment of specified learning outcomes The self-paced modules will be highly interactive featuring integrated multimedia content assessments and learner evaluations to allow PACE staff to engage with the content at a high level and practice application in simulated scenarios Each module will require approximately 45-60 minutesmodule for the learner to complete Participants can use the modules separately at different times throughout a training curriculum or they can be assigned at the beginning of a training time by having this information front-loaded
To develop the modules we will work intensely in year 01 with an instructional design team at Drexel University with specialists in dementia care the COPE program and experts in simulation use of standardized patients and training of nurses and other health professionals from Penn Trinity Health and Jefferson We anticipate the modules to contain the following content module 1 - introduction to COPE program research evidence and core principles underlying the program module 2 - overview of delivery characteristics role of RN and OT three phases assessment implementation generalizability of the COPE program permissible adaptations module 3 and 4-assessment phase introduction to clinical interview and all assessments and forms module 5 and 6- implementation phase including helping caregiver identify 3 problem areas engaging in problem solving and brainstorming developing and providing an assessment report and offering prescriptions strategies for each identified problem area module 7 and 8 - generalizability phase or helping caregivers use strategies for one problem area to address another and planning for the future module 9- developing rapport and working with family caregivers from different backgrounds cultures living environments and relationships and helping families balance caregiving with other life roles adjusting approach by level of readiness module 10 - challenging cases motivational interviewing how to explain the program how to meet caregivers where they are at and provide validation and support
Scripts for each module will be developed and shared with OTsRNs who are not part of the study but work within Trinity PACE programs This will allow for continuous feedback loops to assure that the scenarios meet the needs of PACE staff We will compare the online program to our traditional 3-day face-to-face training currently used with COPE
The 3-day training program will be conducted by Dr Piersol using a slide deck and case presentations as we have previously done The comparison of the two training programs is described in Phase 2 and 3 below
PHASE 1b -The Fidelity Monitoring Program- We seek to develop a scalable approach to assess fidelity to the COPE Program when it is implemented in a real-world setting such as PACE using computational linguistics techniques eg natural language processing The essence of fidelity to the Core Principles of COPE program will be captured by using automatic classification programs that evaluate both the content that should be included in COPE sessions and the style of delivery While automatic classification programs have been applied to measure quality metrics of transcribed narratives in the field of psychotherapy 21 it has not been used to measure other aspects of quality- namely fidelity to evidence-based practices or dementia care and caregiver supportive programs The development of the automated Fidelity Monitoring Program will occur in three steps and will be carried out by a technical team consisting of an expert in content analysis Dr Ani Nenkova and a consultant expert in speech recognition and prosody Dr Mari Ostendorf Co- I Nenkova has worked extensively on automatic summarization evaluation of automatic summarization and readability and linguistic style The ultimate goal of our efforts is to develop a system that- given a recording of a COPE delivery session eg in real time immediately after interaction between the clinician and the caregiver- produces a three-tiered score indicating if the fidelity was excellent acceptable or problematic Special emphasis will be given to the accuracy of identifying problematic COPE fidelity which is not fateful to training and may not produce the same desired outcomes as intervention delivered with higher fidelity First we will obtain n-best list speech recognition of the COPE interaction This will help mitigate recognition errors in the next stage Until recently audio recording transcription was fraught with challenges particularly in sessions involving two or more speakers Advances in audio signaling and speech recognition have brought technology for automating language analysis within reach Recent research has suggested that text based features may be more effective than using audio features alone when classifying fidelity in behavioral research 47 Automatic speech recognition software will be used to transcribe sessions and the resulting words will be used in a text-based model of fidelity All COPE training practice and implementation sessions will be audio taped with participant consent There are several of automatic speech recognition tools that we can use We will pick the one that best balances accuracy of recognition in our domain and privacyOnce the transcripts are obtained there are two approaches that we will develop and contrast 1 comparison with a reference delivery and 2 a supervised classification approach The first has the advantage of needing only a small number of excellent deliveries and several acceptable deliveries for each of the seven dimensions while the other needs a larger set of labeled data but would potentially lead to higher accuracy of prediction
Comparison or similarity to reference Steps A and B Our approach will leverage techniques widely used in the evaluation of automatically produced content such as machine translation and automatic text summarization In these applications it is not feasible to track system improvement with human judgments of quality Instead most of the progress is measured by computing similarity between a set of sample reference text ie what a good translation or a good summary would be and the system output Such automatic evaluation approaches are widely used for machine translation 48 and summarization 49 While there have been some concerns that the automatic measures are not fine enough to distinguish between levels of very good context these measures show strong ability to distinguish very bad content 5051 or poor fidelity aligns with the needs of our project
In Step A we will compile 10 examples of excellent COPE fidelity deliveries with the regions where desired aspects is expressed will be explicitly marked ie marked parts where person-environment fit is discussed or parts of the interaction where the clinician asks open ended questions or confirms the caregiver understanding of the content Next new interactions will be ranked by their similarity with the reference excellent interactions 52 In Step 2 we will identify problematic fidelity deliveries We will make use of 10 negative examples of acceptable but imperfect deliveries If the new interaction ranks lower than most of these it will be considered unacceptable or problematic Parameters and decision rules will be developed at this step to determine cut-off levels for declaring an interaction problematic
Determination of excellent versus problematic fidelity ratings of the audio recordings will be assessed by Drs Hirshman and Renz who were not the original developers of COPE and thus they offer an independent review using the COPE Adherence Scale developed for the original clinical trial in order to evaluate the extent to which core treatment principles were implemented effectively Ratings from the Adherence Scale have been standardized such that 100 represents perfect fidelity and 000 represent complete non-fidelity The scale was previously modeled off of the NIH REACH I and II fidelity approach Findings from the combined fidelity ratings will be used to refine the automated fidelity program into a best model to be tested in Step C 53
Supervised classification Step C For this approach we will need to examine all recorded sessions 600 We will train a supervised classifier or a regression model to predict the score 1 to 3 corresponding to excellent acceptable and problematic of a given interaction We will experiment with a number of classifiers including deep learning frameworks and more traditional support vector machine and logistic regression modelsThe resulting final validation set of 100 labeled interactions will serve to finalize the best model for fidelity prediction
PHASE 2 - Evaluation of Online Training Program in Interventionist Uptake and Fidelity Phase 2 of this study involves a series of activities designed to evaluate the whether an online training program is the same or better in improving interventionist uptake of- and fidelity to- COPE principles and protocols compared to a high intensity face-to-face traditional form of training
PHASE 3 Aim 3 - Efficacy of COPE on PACE participant outcomes by type of COPE training
This aim will be accomplished by evaluating dyad outcomes of the COPE program under the two different training approaches Following training each of the PACE organizations will enroll 5 persons with dementia and their caregivers in the study This will yield 50 family dyads 25 dyads in traditional training sites and 25 dyads in online training sites
The recruitment plan includes the following goals
25 recruitment complete by August 1 2022 50 recruitment complete by October 31 2022 75 recruitment complete by December 28 2022 100 recruitment complete by February 28 2023 Data analysis completed by April 28 2023
To accommodate diversity of learning needs the modules will be designed using a hyperlearning model with four dimensions The general principles will begin with the module learning objectives and follow with a review of core concepts and required andor self-directed learning activities The mini-lecture component of the modules will include information on the major concepts of the module Since the modules will be self-paced the learner can take hisher time going through them and perform in the embedded interactive learning activities The clinical reasoning dimension will provide the learner with an opportunity for problem-solving and clinical decision-making This dimension will contain vignettes and case studies with questions requiring analysis and synthesis The final dimension will be evaluation assessment of learning outcomes This dimension will use teacher-made and standardized pre-and post-tests to assess attainment of specified learning outcomes The self-paced modules will be highly interactive featuring integrated multimedia content assessments and learner evaluations to allow PACE staff to engage with the content at a high level and practice application in simulated scenarios Each module will require approximately 45-60 minutesmodule for the learner to complete Participants can use the modules separately at different times throughout a training curriculum or they can be assigned at the beginning of a training time by having this information front-loaded
To develop the modules we will work intensely in year 01 with an instructional design team at Drexel University with specialists in dementia care the COPE program and experts in simulation use of standardized patients and training of nurses and other health professionals from Penn Trinity Health and Jefferson We anticipate the modules to contain the following content module 1 - introduction to COPE program research evidence and core principles underlying the program module 2 - overview of delivery characteristics role of RN and OT three phases assessment implementation generalizability of the COPE program permissible adaptations module 3 and 4-assessment phase introduction to clinical interview and all assessments and forms module 5 and 6- implementation phase including helping caregiver identify 3 problem areas engaging in problem solving and brainstorming developing and providing an assessment report and offering prescriptions strategies for each identified problem area module 7 and 8 - generalizability phase or helping caregivers use strategies for one problem area to address another and planning for the future module 9- developing rapport and working with family caregivers from different backgrounds cultures living environments and relationships and helping families balance caregiving with other life roles adjusting approach by level of readiness module 10 - challenging cases motivational interviewing how to explain the program how to meet caregivers where they are at and provide validation and support
Scripts for each module will be developed and shared with OTsRNs who are not part of the study but work within Trinity PACE programs This will allow for continuous feedback loops to assure that the scenarios meet the needs of PACE staff We will compare the online program to our traditional 3-day face-to-face training currently used with COPE
The 3-day training program will be conducted by Dr Piersol using a slide deck and case presentations as we have previously done The comparison of the two training programs is described in Phase 2 and 3 below
PHASE 1b -The Fidelity Monitoring Program- We seek to develop a scalable approach to assess fidelity to the COPE Program when it is implemented in a real-world setting such as PACE using computational linguistics techniques eg natural language processing The essence of fidelity to the Core Principles of COPE program will be captured by using automatic classification programs that evaluate both the content that should be included in COPE sessions and the style of delivery While automatic classification programs have been applied to measure quality metrics of transcribed narratives in the field of psychotherapy 21 it has not been used to measure other aspects of quality- namely fidelity to evidence-based practices or dementia care and caregiver supportive programs The development of the automated Fidelity Monitoring Program will occur in three steps and will be carried out by a technical team consisting of an expert in content analysis Dr Ani Nenkova and a consultant expert in speech recognition and prosody Dr Mari Ostendorf Co- I Nenkova has worked extensively on automatic summarization evaluation of automatic summarization and readability and linguistic style The ultimate goal of our efforts is to develop a system that- given a recording of a COPE delivery session eg in real time immediately after interaction between the clinician and the caregiver- produces a three-tiered score indicating if the fidelity was excellent acceptable or problematic Special emphasis will be given to the accuracy of identifying problematic COPE fidelity which is not fateful to training and may not produce the same desired outcomes as intervention delivered with higher fidelity First we will obtain n-best list speech recognition of the COPE interaction This will help mitigate recognition errors in the next stage Until recently audio recording transcription was fraught with challenges particularly in sessions involving two or more speakers Advances in audio signaling and speech recognition have brought technology for automating language analysis within reach Recent research has suggested that text based features may be more effective than using audio features alone when classifying fidelity in behavioral research 47 Automatic speech recognition software will be used to transcribe sessions and the resulting words will be used in a text-based model of fidelity All COPE training practice and implementation sessions will be audio taped with participant consent There are several of automatic speech recognition tools that we can use We will pick the one that best balances accuracy of recognition in our domain and privacyOnce the transcripts are obtained there are two approaches that we will develop and contrast 1 comparison with a reference delivery and 2 a supervised classification approach The first has the advantage of needing only a small number of excellent deliveries and several acceptable deliveries for each of the seven dimensions while the other needs a larger set of labeled data but would potentially lead to higher accuracy of prediction
Comparison or similarity to reference Steps A and B Our approach will leverage techniques widely used in the evaluation of automatically produced content such as machine translation and automatic text summarization In these applications it is not feasible to track system improvement with human judgments of quality Instead most of the progress is measured by computing similarity between a set of sample reference text ie what a good translation or a good summary would be and the system output Such automatic evaluation approaches are widely used for machine translation 48 and summarization 49 While there have been some concerns that the automatic measures are not fine enough to distinguish between levels of very good context these measures show strong ability to distinguish very bad content 5051 or poor fidelity aligns with the needs of our project
In Step A we will compile 10 examples of excellent COPE fidelity deliveries with the regions where desired aspects is expressed will be explicitly marked ie marked parts where person-environment fit is discussed or parts of the interaction where the clinician asks open ended questions or confirms the caregiver understanding of the content Next new interactions will be ranked by their similarity with the reference excellent interactions 52 In Step 2 we will identify problematic fidelity deliveries We will make use of 10 negative examples of acceptable but imperfect deliveries If the new interaction ranks lower than most of these it will be considered unacceptable or problematic Parameters and decision rules will be developed at this step to determine cut-off levels for declaring an interaction problematic
Determination of excellent versus problematic fidelity ratings of the audio recordings will be assessed by Drs Hirshman and Renz who were not the original developers of COPE and thus they offer an independent review using the COPE Adherence Scale developed for the original clinical trial in order to evaluate the extent to which core treatment principles were implemented effectively Ratings from the Adherence Scale have been standardized such that 100 represents perfect fidelity and 000 represent complete non-fidelity The scale was previously modeled off of the NIH REACH I and II fidelity approach Findings from the combined fidelity ratings will be used to refine the automated fidelity program into a best model to be tested in Step C 53
Supervised classification Step C For this approach we will need to examine all recorded sessions 600 We will train a supervised classifier or a regression model to predict the score 1 to 3 corresponding to excellent acceptable and problematic of a given interaction We will experiment with a number of classifiers including deep learning frameworks and more traditional support vector machine and logistic regression modelsThe resulting final validation set of 100 labeled interactions will serve to finalize the best model for fidelity prediction
PHASE 2 - Evaluation of Online Training Program in Interventionist Uptake and Fidelity Phase 2 of this study involves a series of activities designed to evaluate the whether an online training program is the same or better in improving interventionist uptake of- and fidelity to- COPE principles and protocols compared to a high intensity face-to-face traditional form of training
PHASE 3 Aim 3 - Efficacy of COPE on PACE participant outcomes by type of COPE training
This aim will be accomplished by evaluating dyad outcomes of the COPE program under the two different training approaches Following training each of the PACE organizations will enroll 5 persons with dementia and their caregivers in the study This will yield 50 family dyads 25 dyads in traditional training sites and 25 dyads in online training sites
The recruitment plan includes the following goals
25 recruitment complete by August 1 2022 50 recruitment complete by October 31 2022 75 recruitment complete by December 28 2022 100 recruitment complete by February 28 2023 Data analysis completed by April 28 2023
Study Oversight
Has Oversight DMC:
None
Is a FDA Regulated Drug?:
False
Is a FDA Regulated Device?:
False
Is an Unapproved Device?:
None
Is a PPSD?:
None
Is a US Export?:
None
Is an FDA AA801 Violation?:
None