Ignite Creation Date:
2024-06-16 @ 11:48 AM
Last Modification Date:
2024-10-26 @ 3:30 PM
Study NCT ID:
NCT06434220
Status:
NOT_YET_RECRUITING
Last Update Posted:
2024-05-30
First Post:
2024-05-20
Brief Title:
Effect of Predictive Model on ED Physician Assessments of Patient Disposition
Sponsor:
Boston Childrens Hospital
Organization:
Boston Childrens Hospital
Study Overview
Official Title:
Effect of Predictive Model on ED Physician Assessments of Patient Disposition
Status:
NOT_YET_RECRUITING
Status Verified Date:
2024-08
Last Known Status:
None
Delayed Posting:
No
If Stopped, Why?:
Not Stopped
Has Expanded Access:
False
If Expanded Access, NCT#:
N/A
Has Expanded Access, NCT# Status:
N/A
Acronym:
None
Brief Summary:
The goal of this study is to measure the impact of fairness-aware algorithms on physician predictions of ED patient admission Using an experimentally validated machine learning model tuned for equitable outcomes the investigators quantify the impact of model recommendations on ED physician assessments of admission risk in a silent prospective study The investigators survey ED physicians who are not currently caring for patients using live site data To quantify the impact of the model on ED physician assessments of admission risk the investigators collect physician assessments before and after consulting the original or updated model prediction
The investigators measure ED physician adherence to model suggestions along with the predictive accuracy and equity of downstream patient outcomes The outcome of this study is an empirical measure of the extent to which fair ML models may influence admission decisions to mitigate health care disparities
The investigators measure ED physician adherence to model suggestions along with the predictive accuracy and equity of downstream patient outcomes The outcome of this study is an empirical measure of the extent to which fair ML models may influence admission decisions to mitigate health care disparities
Detailed Description:
Specific AimsObjectives
1 Measure the effect of the sharing of a model prediction of admission on an attending physicians assessment of patient disposition within one hour from presentation at a tertiary academic pediatric hospital
2 Measure the effect of the sharing of a model prediction from a model tuned for equal subgroup performance on an attending physicians assessment of patient disposition within one hour from presentation at a tertiary academic pediatric hospital
Background and Significance
Machine learning ML models increasingly provide clinical decision support CDS to care teams to help prioritize individuals for specific care based on their predicted health needs and outcomes AIML methods can have a particularly high impact on resource allocation in emergency departments EDs across the US which have been described by the Institute for Medicine as nearing the breaking point of over-capacity Unfortunately models often perform poorly on disinvested subpopulations relative to the population as a whole As a result ML models may exacerbate downstream health disparities by under-performing on marginalized patient subpopulations especially when models are expanded to multiple care centers and or used without subgroup monitoring for long periods of time
Many prediction models have been developed in recent years to predict patient disposition from the ED including a prediction tool developed by our group and currently in piloting stages at Boston Childrens Hospital South Shore Hospital and Childrens Hospital of Los-Angeles Our prediction tool the Predictor of Patient Placement POPP provides an accurate real-time likelihood of admission based on data available in the electronic health record at the time of the visit Advance notice of likely admissions can have an important impact on ED waiting and boarding times with the potential to improve flow and patient satisfaction
To this end the investigation team has submitted a grant proposal to the National Library of Medicine NLM 1R01LM014300 - 01A1 that researches the development and validation of fairness-aware prediction models of patient admission Aim 2 of this grant studies the effect of these models on ED physician assessments of patient disposition and corresponds to this protocol The NLM has indicated its intention to fund this proposal and the investigators are in the process of submitting documents to finalize the award This component of the study is slated for year 3 of the study
Preliminary Studies
The investigators conducted a series of initial retrospective studies that established that patient admission could be predicted with machine learning models ahead of time in the BCH ED progressively during the visit as well as across other medical centers with good accuracy AUROC 09-093
Next the investigators found that the accuracy of POPP in predicting admission likelihood added value to the gestalt assessments of ED attending physicians The positive predictive value for the prediction of admission was 66 for the clinicians 73 for POPP and 86 for a hybrid model combining the two
Finally the investigators developed methods for post-processing the ED prediction models to make them well-calibrated across patient demographic groups defined by race sex and insurance product
The model predictions are currently used to help with bed coordination but given their high value may also improve decision making at the bed-side With this study our goal is to now test in a simulated safe and realistic setting how model recommendations affect the assessments of admission likelihood by ED attending physicians
1 Measure the effect of the sharing of a model prediction of admission on an attending physicians assessment of patient disposition within one hour from presentation at a tertiary academic pediatric hospital
2 Measure the effect of the sharing of a model prediction from a model tuned for equal subgroup performance on an attending physicians assessment of patient disposition within one hour from presentation at a tertiary academic pediatric hospital
Background and Significance
Machine learning ML models increasingly provide clinical decision support CDS to care teams to help prioritize individuals for specific care based on their predicted health needs and outcomes AIML methods can have a particularly high impact on resource allocation in emergency departments EDs across the US which have been described by the Institute for Medicine as nearing the breaking point of over-capacity Unfortunately models often perform poorly on disinvested subpopulations relative to the population as a whole As a result ML models may exacerbate downstream health disparities by under-performing on marginalized patient subpopulations especially when models are expanded to multiple care centers and or used without subgroup monitoring for long periods of time
Many prediction models have been developed in recent years to predict patient disposition from the ED including a prediction tool developed by our group and currently in piloting stages at Boston Childrens Hospital South Shore Hospital and Childrens Hospital of Los-Angeles Our prediction tool the Predictor of Patient Placement POPP provides an accurate real-time likelihood of admission based on data available in the electronic health record at the time of the visit Advance notice of likely admissions can have an important impact on ED waiting and boarding times with the potential to improve flow and patient satisfaction
To this end the investigation team has submitted a grant proposal to the National Library of Medicine NLM 1R01LM014300 - 01A1 that researches the development and validation of fairness-aware prediction models of patient admission Aim 2 of this grant studies the effect of these models on ED physician assessments of patient disposition and corresponds to this protocol The NLM has indicated its intention to fund this proposal and the investigators are in the process of submitting documents to finalize the award This component of the study is slated for year 3 of the study
Preliminary Studies
The investigators conducted a series of initial retrospective studies that established that patient admission could be predicted with machine learning models ahead of time in the BCH ED progressively during the visit as well as across other medical centers with good accuracy AUROC 09-093
Next the investigators found that the accuracy of POPP in predicting admission likelihood added value to the gestalt assessments of ED attending physicians The positive predictive value for the prediction of admission was 66 for the clinicians 73 for POPP and 86 for a hybrid model combining the two
Finally the investigators developed methods for post-processing the ED prediction models to make them well-calibrated across patient demographic groups defined by race sex and insurance product
The model predictions are currently used to help with bed coordination but given their high value may also improve decision making at the bed-side With this study our goal is to now test in a simulated safe and realistic setting how model recommendations affect the assessments of admission likelihood by ED attending physicians
Study Oversight
Has Oversight DMC:
None
Is a FDA Regulated Drug?:
False
Is a FDA Regulated Device?:
False
Is an Unapproved Device?:
None
Is a PPSD?:
None
Is a US Export?:
None
Is an FDA AA801 Violation?:
None