Ignite Creation Date:
2026-03-26 @ 3:16 PM
Ignite Modification Date:
2026-04-04 @ 5:33 PM
Study NCT ID:
NCT07307157
Status:
ENROLLING_BY_INVITATION
Last Update Posted:
2025-12-29
First Post:
2025-12-13
Is Gene Therapy:
True
Has Adverse Events:
False
Brief Title:
Head-to-Head Evaluation of the Cancer Ontology Supervised Multimodal Orchestration (COSMO) AI System Versus Pathologist-Only Review
Sponsor:
Organization:
Raw JSON
{'hasResults': False, 'derivedSection': {'miscInfoModule': {'versionHolder': '2026-03-25'}, 'conditionBrowseModule': {'meshes': [{'id': 'D001932', 'term': 'Brain Neoplasms'}, {'id': 'D008175', 'term': 'Lung Neoplasms'}, {'id': 'D004194', 'term': 'Disease'}, {'id': 'D007680', 'term': 'Kidney Neoplasms'}], 'ancestors': [{'id': 'D016543', 'term': 'Central Nervous System Neoplasms'}, {'id': 'D009423', 'term': 'Nervous System Neoplasms'}, {'id': 'D009371', 'term': 'Neoplasms by Site'}, {'id': 'D009369', 'term': 'Neoplasms'}, {'id': 'D001927', 'term': 'Brain Diseases'}, {'id': 'D002493', 'term': 'Central Nervous System Diseases'}, {'id': 'D009422', 'term': 'Nervous System Diseases'}, {'id': 'D012142', 'term': 'Respiratory Tract Neoplasms'}, {'id': 'D013899', 'term': 'Thoracic Neoplasms'}, {'id': 'D008171', 'term': 'Lung Diseases'}, {'id': 'D012140', 'term': 'Respiratory Tract Diseases'}, {'id': 'D010335', 'term': 'Pathologic Processes'}, {'id': 'D013568', 'term': 'Pathological Conditions, Signs and Symptoms'}, {'id': 'D014571', 'term': 'Urologic Neoplasms'}, {'id': 'D014565', 'term': 'Urogenital Neoplasms'}, {'id': 'D052776', 'term': 'Female Urogenital Diseases'}, {'id': 'D005261', 'term': 'Female Urogenital Diseases and Pregnancy Complications'}, {'id': 'D000091642', 'term': 'Urogenital Diseases'}, {'id': 'D007674', 'term': 'Kidney Diseases'}, {'id': 'D014570', 'term': 'Urologic Diseases'}, {'id': 'D052801', 'term': 'Male Urogenital Diseases'}]}}, 'protocolSection': {'designModule': {'studyType': 'OBSERVATIONAL', 'designInfo': {'timePerspective': 'PROSPECTIVE', 'observationalModel': 'COHORT'}, 'enrollmentInfo': {'type': 'ESTIMATED', 'count': 30}, 'patientRegistry': False}, 'statusModule': {'overallStatus': 'ENROLLING_BY_INVITATION', 'startDateStruct': {'date': '2025-06-12', 'type': 'ACTUAL'}, 'expandedAccessInfo': {'hasExpandedAccess': False}, 'statusVerifiedDate': '2025-12', 'completionDateStruct': {'date': '2026-01-31', 'type': 'ESTIMATED'}, 'lastUpdateSubmitDate': '2025-12-13', 'studyFirstSubmitDate': '2025-12-13', 'studyFirstSubmitQcDate': '2025-12-13', 'lastUpdatePostDateStruct': {'date': '2025-12-29', 'type': 'ACTUAL'}, 'studyFirstPostDateStruct': {'date': '2025-12-29', 'type': 'ACTUAL'}, 'primaryCompletionDateStruct': {'date': '2026-01-31', 'type': 'ESTIMATED'}}, 'outcomesModule': {'primaryOutcomes': [{'measure': 'Diagnostic performance', 'timeFrame': 'Periprocedural (at the time of slide review)', 'description': 'Diagnostic performance of the COSMO AI system and pathologists in identifying cancer subtypes across brain, lung, and kidney tumors, as assessed by accuracy, balanced accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUROC). We will include both overall comparisons and stratified evaluations by anatomical site and cancer incidence category (common vs. rare or uncommon).'}], 'secondaryOutcomes': [{'measure': 'Inter-Observer Agreement Among Pathologists', 'timeFrame': 'Periprocedural (at the time of slide review)', 'description': "Diagnostic concordance among participating pathologists, measured by Fleiss' kappa, intraclass correlation coefficient (ICC), and pairwise concordance rates."}, {'measure': 'Pathologist-COSMO AI Concordance', 'timeFrame': 'Periprocedural (at the time of slide review)', 'description': 'Agreement patterns between pathologist diagnoses and COSMO AI predictions, including proportion of concordant cases overall and stratified by anatomical site, cancer incidence category, and pathologist experience level.'}, {'measure': 'Diagnostic Confidence', 'timeFrame': 'Periprocedural (at the time of slide review)', 'description': 'Mean confidence scores (5-point scale) reported by pathologists during diagnostic assessment, stratified by anatomical site, cancer incidence category, and diagnostic correctness (correct vs. incorrect).'}, {'measure': 'Time-to-Diagnosis', 'timeFrame': 'Periprocedural (at the time of slide review)', 'description': 'Mean diagnostic time (in seconds) required by pathologists to provide cancer subtype classification, stratified by anatomical site, cancer incidence category, and pathologist experience level.'}, {'measure': 'Diagnostic Performance Stratified by Pathologist Experience', 'timeFrame': 'Periprocedural (at the time of slide review)', 'description': 'Diagnostic accuracy of pathologists stratified by years of clinical experience (3-5 years, 6-10 years, \\>10 years) to assess the relationship between experience level and diagnostic performance in cancer subtype classification.'}]}, 'oversightModule': {'oversightHasDmc': False, 'isFdaRegulatedDrug': False, 'isFdaRegulatedDevice': False}, 'conditionsModule': {'keywords': ['Artificial Intelligence', 'Whole-Slide Images', 'Ontology', 'Multimodal'], 'conditions': ['Brain Cancer', 'Lung Cancer (Diagnosis)', 'Renal Cancer']}, 'descriptionModule': {'briefSummary': 'This study evaluates the diagnostic performance of the Cancer Ontology Supervised Multimodal Orchestration (COSMO) AI system for cancer subtype classification and compares it head-to-head with pathologist-only review. Pathologists will independently review de-identified whole-slide images derived from up to 300 patients across three anatomical sites (brain, lung, kidney) and provide diagnostic assessments. In parallel, COSMO will process the same cases offline to generate independent predictions, enabling direct comparison of diagnostic accuracy between human experts and the AI system.\n\nThe study will characterize the diagnostic accuracy of COSMO and pathologists, inter-observer agreement, and variations in performance across anatomical sites and cancer types with different incidence rates. Results will establish how COSMO compares to pathologists on identical cases and will inform the development of AI-assisted diagnostic systems in clinical practice.', 'detailedDescription': "Study Rationale and Background Diagnostic accuracy in cancer subtype classification varies significantly among pathologists due to differences in expertise, experience, and access to diagnostic resources. The emergence of AI systems in pathology offers the potential to enhance diagnostic performance and consistency in cancer classification. However, direct empirical comparisons of AI-based predictions and pathologists' diagnostic performance on identical cases remain limited in the literature.\n\nStudy Aims This head-to-head comparative study aims to: (1) evaluate the diagnostic performance of the COSMO AI system in cancer subtype classification across multiple anatomical sites; (2) characterize the diagnostic accuracy of experienced pathologists on the same cases; (3) directly compare diagnostic performance metrics between COSMO and pathologists; and (4) examine concordance patterns and performance variation by anatomical site, cancer incidence category, pathologist experience, and case complexity.\n\nStudy Setting and Participants The study will involve up to 25 board-certified pathologists with 3 to 10+ years of diagnostic experience, recruited from institutions across North America, Europe, and the Asia-Pacific region. Participating pathologists will have domain expertise in neuropathology, pulmonary pathology, urologic pathology, or general anatomical pathology.\n\nCases and Stratification The study will employ de-identified archival whole-slide images representing up to 300 patients with confirmed reference diagnoses, including 100 brain cancers, 100 lung cancers, and 100 kidney cancers. Cases will be stratified by cancer type and incidence category (common vs. rare or uncommon), consistent with World Health Organization (WHO) guidelines.\n\nData Collection Pathologists will independently review each case and provide diagnostic classifications along with confidence assessments using a 5-point scale. The digital pathology interface will automatically record time-to-diagnosis metrics. COSMO will process the same cases offline to generate independent diagnostic predictions and confidence scores. Both pathologist and AI predictions will be evaluated against established reference standard diagnoses.\n\nAnalysis Framework The primary analysis will characterize diagnostic performance metrics (including accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUROC)) for both pathologists (at the individual and aggregated levels) and the COSMO system. Secondary analyses will assess performance stratified by anatomical site, cancer incidence category, and pathologist experience level."}, 'eligibilityModule': {'sex': 'ALL', 'stdAges': ['CHILD', 'ADULT', 'OLDER_ADULT'], 'samplingMethod': 'NON_PROBABILITY_SAMPLE', 'studyPopulation': 'We will recruit pathologists from international academic medical centers, hospital systems, and diagnostic pathology practices across North America (United States), Europe (Austria, Hungary), and the Asia-Pacific (Taiwan, Hong Kong, South Korea, China, India) region. Participating sites will include major academic institutions with established pathology departments, with recruitment targeting expertise in neuropathology, pulmonary pathology, and urologic pathology.', 'healthyVolunteers': False, 'eligibilityCriteria': 'Inclusion Criteria:\n\n* Board-certified pathologist with expertise in neuropathology, pulmonary pathology, urologic pathology, or general anatomical pathology\n* Minimum of 3 years of clinical diagnostic experience\n* Active clinical practice involving diagnostic pathology slide review\n* Willingness to independently review and diagnose up to 300 de-identified whole-slide images\n* Ability to access the study platform and complete case reviews within the specified study timeline\n* Provision of informed consent for study participation\n\nExclusion Criteria:\n\n* Prior involvement in the design or validation of the COSMO AI system\n* Inability to commit sufficient time to complete assigned case reviews\n* Presence of significant financial conflicts of interest related to the study outcomes'}, 'identificationModule': {'nctId': 'NCT07307157', 'acronym': 'COSMO', 'briefTitle': 'Head-to-Head Evaluation of the Cancer Ontology Supervised Multimodal Orchestration (COSMO) AI System Versus Pathologist-Only Review', 'organization': {'class': 'OTHER', 'fullName': 'Harvard Medical School (HMS and HSDM)'}, 'officialTitle': 'Head-to-Head Evaluation of the Cancer Ontology Supervised Multimodal Orchestration (COSMO) AI System Versus Pathologist-Only Review', 'orgStudyIdInfo': {'id': 'Yu Lab COSMO Study'}}, 'armsInterventionsModule': {'armGroups': [{'label': 'AI-Based Evaluation using COSMO'}, {'label': 'Pathologist-Based Evaluation', 'interventionNames': ['Diagnostic Test: Digital Pathology Evaluation']}], 'interventions': [{'name': 'Digital Pathology Evaluation', 'type': 'DIAGNOSTIC_TEST', 'description': 'Digital Pathology Evaluation', 'armGroupLabels': ['Pathologist-Based Evaluation']}]}, 'contactsLocationsModule': {'locations': [{'zip': '02115', 'city': 'Boston', 'state': 'Massachusetts', 'country': 'United States', 'facility': 'Harvard Medical School', 'geoPoint': {'lat': 42.35843, 'lon': -71.05977}}], 'overallOfficials': [{'name': 'Kun-Hsing Yu, MD, PhD', 'role': 'PRINCIPAL_INVESTIGATOR', 'affiliation': 'Harvard Medical School (HMS and HSDM)'}]}, 'ipdSharingStatementModule': {'ipdSharing': 'NO', 'description': 'Individual pathologist diagnostic assessments will not be shared to protect evaluator anonymity and privacy. De-identified case data and aggregated performance metrics will be made available through published results and supplementary materials. The protocol document will be uploaded to enable full methodological transparency.'}, 'sponsorCollaboratorsModule': {'leadSponsor': {'name': 'Harvard Medical School (HMS and HSDM)', 'class': 'OTHER'}, 'responsibleParty': {'type': 'PRINCIPAL_INVESTIGATOR', 'investigatorTitle': 'Associate Professor', 'investigatorFullName': 'Kun-Hsing Yu', 'investigatorAffiliation': 'Harvard Medical School (HMS and HSDM)'}}}}