A Large Language Model Pipeline for Breast Cancer Oncology

This study demonstrates the potential of fine-tuned large language models to assist in breast cancer treatment planning by accurately predicting the need for adjuvant radiation therapy and chemotherapy, suggesting that these models could outperform human oncologists in a significant minority of cases and potentially improve patient outcomes, especially in community healthcare settings.

Abstract

Large language models (LLMs) have demonstrated potential in the innovation of many disciplines. However, how they can best be developed for oncology remains underdeveloped. State-of-the-art OpenAI models were fine-tuned on a clinical dataset and clinical guidelines text corpus for two important cancer treatment factors, adjuvant radiation therapy and chemotherapy, using a novel Langchain prompt engineering pipeline. A high accuracy (0.85+) was achieved in the classification of adjuvant radiation therapy and chemotherapy for breast cancer patients. Furthermore, a confidence interval was formed from observational data on the quality of treatment from human oncologists to estimate the proportion of scenarios in which the model must outperform the original oncologist in its treatment prediction to be a better solution overall as 8.2% to 13.3%. Due to indeterminacy in the outcomes of cancer treatment decisions, future investigation, potentially a clinical trial, would be required to determine if this threshold was met by the models. Nevertheless, with 85% of U.S. cancer patients receiving treatment at local community facilities, these kinds of models could play an important part in expanding access to quality care with outcomes that lie, at minimum, close to a human oncologist.

Method

The authors used a novel Langchain prompt engineering pipeline to fine-tune large language models (LLMs) developed by OpenAI, specifically the GPT-3.5 Turbo, Babbage, and DaVinci models. They employed these models to process and learn from a clinical dataset and a corpus of clinical guidelines. The clinical dataset, the Duke MRI dataset, contained detailed information from 922 breast cancer patients, including demographics, genomics, tumor properties, treatments, and outcomes. The corpus of clinical guidelines included sources from reputable institutions such as ASCO and NCCN. The methodology involved several steps, including data preprocessing, generating and refining question-answer pairs, summarization, and training the models. The fine-tuned models were then evaluated for their accuracy in predicting adjuvant radiation therapy and chemotherapy for breast cancer patients.

Main Finding

The authors discovered that the fine-tuned large language models (LLMs) achieved a high classification accuracy of over 85% in predicting the need for adjuvant radiation therapy and chemotherapy for breast cancer patients. Additionally, they found that the model could potentially outperform human oncologists in 8.2% to 13.3% of scenarios, as estimated by a confidence interval formed from observational data on the quality of treatment decisions. This suggests that LLMs could play a significant role in expanding access to quality cancer care and improving patient outcomes, particularly in community healthcare settings where the majority of U.S. cancer patients receive treatment.

Conclusion

The conclusion of the study is that fine-tuned large language models (LLMs) have the potential to assist oncologists in making more informed and consistent treatment decisions for breast cancer patients, thereby expanding access to quality care and potentially improving patient outcomes. The study found that the LLMs could achieve high accuracy in predicting the need for adjuvant radiation therapy and chemotherapy, and there is a possibility that the models could outperform human oncologists in a significant minority of cases. However, the authors note that future investigations, potentially including clinical trials, would be required to validate the effectiveness of LLMs in real-world oncology settings and to determine if the models meet the threshold for outperforming human decision-making.

Keywords

breast cancer, oncology, large language models, LLMs, OpenAI, Langchain, prompt engineering, clinical dataset, clinical guidelines, adjuvant radiation therapy, adjuvant chemotherapy, treatment planning, machine learning, AI, healthcare, patient outcomes, community healthcare settings, Duke MRI dataset, ASCO, NCCN, GPT-3.5 Turbo, Babbage, DaVinci, model fine-tuning, temperature sensitivity analysis, error analysis, Wilson score interval, confidence interval, treatment decision-making, human oncologists, model performance, clinical decision support systems, natural language processing, NLP, medical guidelines, oncological recommendations, interactive chat interactions, dynamic query handling, context retention, function-calling capabilities, text completion tasks, language understanding, robustness, generalization, unique datasets, clinical settings, patient charts, edge-cases, rare diseases, treatment predictions, human error, data leverage, medical professionals, systems failure, first approximation, treatment predictions, collaboration, unique patients, cancer treatment, irreversibility, optimal treatment, model accuracy