Introduction Although clinically derived information could improve patient care, its full potential remains unrealised because most of it is stored in a format unsuitable for traditional methods of analysis, free-text clinical reports. Various studies have already demonstrated the utility of natural language processing algorithms for medical text analysis. Yet, evidence on their learning efficiency is still lacking. This study aimed to compare the learning curves of various algorithms and develop an open-source framework for text mining in healthcare.

Methods Deep learning and regressions-based models were developed to determine the histopathological diagnosis of patients with brain tumour based on free-text pathology reports. For each model, we characterised the learning curve and the minimal required training examples to reach the area under the curve (AUC) performance thresholds of 0.95 and 0.98.

Results In total, we retrieved 7000 reports on 5242 patients with brain tumour (2316 with glioma, 1412 with meningioma and 1514 with cerebral metastasis). Conventional regression and deep learning-based models required 200–400 and 800–1500 training examples to reach the AUC performance thresholds of 0.95 and 0.98, respectively. The deep learning architecture utilised in the current study required 100 and 200 examples, respectively, corresponding to a learning capacity that is two to eight times more efficient.

Conclusions This open-source framework enables the development of high-performing and fast learning natural language processing models. The steep learning curve can be valuable for contexts with limited training examples (eg, rare diseases and events or institutions with lower patient volumes). The resultant models could accelerate retrospective chart review, assemble clinical registries and facilitate a rapid learning healthcare system.