Introduction Although clinically derived information could improve patient care, its full potential remains unrealised because most of it is stored in a format unsuitable for traditional methods of analysis, free-text clinical reports. Various studies have already demonstrated the utility of natural language processing algorithms for medical text analysis. Yet, evidence on their learning efficiency is still lacking. This study aimed to compare the learning curves of various algorithms and develop an open-source framework for text mining in healthcare.
Methods Deep learning and regressions-based models were developed to determine the histopathological diagnosis of patients with brain tumour based on free-text pathology reports. For each model, we characterised the learning curve and the minimal required training examples to reach the area under the curve (AUC) performance thresholds of 0.95 and 0.98.
Results In total, we retrieved 7000 reports on 5242 patients with brain tumour (2316 with glioma, 1412 with meningioma and 1514 with cerebral metastasis). Conventional regression and deep learning-based models required 200–400 and 800–1500 training examples to reach the AUC performance thresholds of 0.95 and 0.98, respectively. The deep learning architecture utilised in the current study required 100 and 200 examples, respectively, corresponding to a learning capacity that is two to eight times more efficient.
Conclusions This open-source framework enables the development of high-performing and fast learning natural language processing models. The steep learning curve can be valuable for contexts with limited training examples (eg, rare diseases and events or institutions with lower patient volumes). The resultant models could accelerate retrospective chart review, assemble clinical registries and facilitate a rapid learning healthcare system.
- assistive technology
Statistics from Altmetric.com
Contributors Conception or design of the work was performed by WBG, TRS, MLDB and OA. Acquisition, analysis or interpretation of data was done by JTS, DJC, AM and RW. JTS, DJC and AM helped in drafting the work. RW, WBG, TRS, MLDB and OA helped in critically revising the work. Final approval of the version to be published was given by JTS, DJC, AM, RW, WBG, TRS, MLDB and OA. Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved by JTS, DJC, AM, RW, WBG, TRS, MLDB and OA.
Funding Funding was received from a National Institutes of Health (NIH) P41EB015898 (AM) and Training Grant T32 CA 009001 (DJC).
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the Declaration of Helsinki 1964 and its later amendments or comparable ethical standards.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. The pathology reports used in the current study contain sensitive patient information. Therefore, the Health Insurance Portability and Accountability Act (HIPAA) prohibits public distribution of the original reports. Anonymised reports are available from the corresponding author upon reasonable request. Code availability: to promote the transparency and reproducibility of our work, we have released the underlying code with an open-source license on a publicly accessible GitHub repository (https://github.com/jtsenders/nlp_learning_curves).
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.