Machine Learning Prediction of Radiofrequency Thermal Ablation Efficacy: A New Option to Optimize Thyroid Nodule Selection

Introduction

Thyroid nodules are a common finding in endocrine clinical practice. Their prevalence is approximately 5% when detected by palpation but may occur at 70% frequency when the thyroid is examined by ultrasound [1-3]. Most nodules are benign and asymptomatic. In these cases, they may simply be followed without requiring any treatment [4, 5]. Sometimes, nodules grow over time causing symptoms of compression or esthetic concerns. Until a decade ago, surgery was the only available choice to solve the problem of mass effect in such cases. In recent years, thermal treatments such as radiofrequency (RF) and laser (L) therapy have shown to be a reliable alternative to surgery as a result of their ability to induce a significant reduction of benign nodules that, in most cases, is sustained long term [6-8]. In 5–30% of nodules, two unfavorable outcomes have been observed: a volume reduction rate (VRR) <50% after the first thermal ablation and progressive regrowth [9-13]. These two outcomes represent two sides of the same coin, as it has been shown that nodules, which decrease to a lesser extent, are those which are prone to regrowth in 4–5 years [14]. Data analyses with respect to VRR and its durability of shrinkage should consider baseline nodule volume, ultrasound structure, and the number of procedures required to obtain an optimal reduction in volume [15]. If thermal therapies are interpreted as an alternative strategy to surgery, then critical issues in their use are represented by the VRR, and by the durability of nodule reduction over a long time period. Nodules that recur or for which reduction is unsatisfactory may represent failure when symptoms of compression and esthetic concerns persist following thermal ablation. Thus, predicting the efficacy of thermal treatments may be important in identifying the best candidates for a good response in a single session. Nodules predicted to have a poor response would be addressed surgically or through an additional treatment session. For this purpose, we trained a machine learning model and tested its ability to predict the 12-month VRR of nodules treated with RF.

Methods

The machine learning model was trained with a dataset consisting of 402 Italian patients with cytologically benign nodules subjected to RF at six Italian Institutions (benignity was confirmed by two repeated fine-needle aspirations, Class II of Bethesda reporting system). Data from 337 patients were obtained from a prospective study published by Deandrea et al. [16], and additional data from 65 patients were obtained from the Division of Endocrinology at the Mauriziano Hospital (Turin), and recruited with the same modality. All Institutions shared the same training program, and at least 50 RF procedures had been performed using the same unified protocol and similar devices. The dataset contained patient demographics, ultrasound characteristics of nodules, volume at baseline and 12 months after RF, energy delivered by RF for each nodule, and procedure time.

The ultrasound characteristics including echostructure, macrocalcifications, and vascularity were designated as categorical variables (echostructure: #1 mixed, #2 spongiform, #3 solid; macrocalcifications: #0 absent, #1 present; vascularity: #1 intense perinodular, #2 intra- and perinodular, #3 weak perinodular), while nodule volume, energy delivered, time of the procedure and VRR were designated as continuous variables.

We instructed an automatic machine learning model (Operating System: Max OSX Mojave version 10.14.6: Python editor: Spyder version 3.3.1; Python version 3.7; TPOT for automatic machine learning version 0.10.1; scikit-learn version 0.21.2; scipy version 1.2.1) (see online suppl. material; for all online suppl. material, see online Supplementary Materials) by inputting the ultrasound characteristics of 402 selected nodules, with the goal of testing the ability of machine learning to predict whether or not a given nodule will exhibit a VRR of at least 50% 12 months after RF treatment [17-20]. We designed and tested an automatic machine learning model using the following variables: (1) baseline nodule volume, (2) echostructure, (3) macrocalcifications, (4) vascularity, and (5) 12-month VRR. Energy delivered and procedure time were not considered input variables because the energy delivered and the time necessary for the procedure were unknown prior to the procedure. The skill of the operator and intraoperative variables, which contribute to the extent of treatment, were not considered input variables.

The “technique efficacy” was conventionally defined as VRR ≥50% [21]. The “primary efficacy rate” was conventionally defined as the percentage of nodules successfully treated following the initial procedure (VRR ≥50%) [21].

Results

The characteristics of patients and treated nodules are listed in Table 1. At 12 months following treatment, nodules were reduced by 67%. The “primary efficacy rate” was 85.6%. The distribution volume (mL) of thyroid nodules at baseline and 12 months after treatment, divided by VRR ≥ or <50%, is depicted in Figure 1a.

Table 1.

Characteristics of 402 patients with cytologically benign nodules at baseline and 12 months after treatment

View Table

View Full Size

a Distribution of nodule volume at baseline and at 12 months after treatment, divided by reduction ≥ or <50%. b Accuracy of the performances of the trained classifiers on the test set.

Citation: European Thyroid Journal 9, 4; 10.1159/000504882

• Download Figure
• Download figure as PowerPoint slide

The characteristics of patients and treated nodules with 12-month VRR ≥ or <50% are shown in Table 2. Patients whose nodules exhibited VRR <50% (mean reduction 38%) had significantly larger volume at baseline compared to those with VRR ≥50% (mean reduction 72%). There were no differences in the distribution of echostructure, macrocalcifications, and vascularity. Energy delivered per mL was greater in those with VRR <50%, whereas the rate of nodules receiving an amount of energy expressed as “mean energy delivered ± 1 SD” was similar between the two groups.

Table 2.

Characteristics of patients and treated nodules divided by 12-month reduction ≥ or <50%

View Table

The addition of sex, age, and institution did not affect the results obtained using only echostructure, macrocalcifications, and vascularity as input variables. The ranges representing the highest accuracy of predicting a nodule’s reduction were 40–50 and 84–100%. The classifiers in these intervals were extremely accurate in predicting which patients will have VRR <θ, but were not suitable in predicting which patients will have VRR >θ. This was a consequence of the limited number of patients available for study with VRR ≥84%.

For each θ value, we evaluated the performance (accuracy) of the trained classifiers on the test sets (Fig. 1b) and interpreted the graph as follows. For example, for θ = 40, the classifier was able to distinguish between nodules having VRR <40% from those having VRR >40% in 92.5% of cases. For θ = 50, the classifier was able to distinguish between nodules having VRR <50% from those having VRR >50% in 85% of cases. When θ ≥50%, the classification error was 0.15%. There was a 95% likelihood that the confidence interval (0.06, 0.35) covers the true classification error of the model for unseen data. For θ ≥50%, the ROC curve showed an AUC = 0.85; true positive rate = 0.8; false positive rate = 0.2; best cut-off = 1.9.

We also investigated the rate of false positives and false negatives, using the confusion matrix, on the test set containing 201 patients. The classifier correctly classified 171 (true positive) out of 173 nodules having VRR ≥50%, and correctly classified 27 (true negative) out of 28 nodules having VRR <50% (Fig. 2) The performance metrics for the machine learning model, assessed with the validation sets, is shown in Table 3. As an example, we show a simulation resulting from the classifier trained for predicting whether the nodule will have VRR of at least 50% (Fig. 3).

Table 3.

Performance metrics for machine learning assessed on the validation sets

View Table

View Full Size

Confusion matrix. The 2 × 2 contingency table reports the number of true positives, false positives, false negatives, and true negatives; the test set contains 201 patients.

Citation: European Thyroid Journal 9, 4; 10.1159/000504882

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Simulated probability for a given nodule to reduce ≥50% at 12-month follow-up.

Citation: European Thyroid Journal 9, 4; 10.1159/000504882

• Download Figure
• Download figure as PowerPoint slide

We also investigated the relative weight of the input variables; for the classifier corresponding to θ = 50, the most important factor influencing the 12-month volume reduction was by far baseline volume (Fig. 4).

View Full Size

Input variables sorted by their relative weight.

Citation: European Thyroid Journal 9, 4; 10.1159/000504882

• Download Figure
• Download figure as PowerPoint slide

Discussion

The main finding of this study is that the machine learning model is able, based on ultrasound characteristics, to predict with an elevated degree of accuracy, which nodules undergoing RF will have VRR <50% at 1-year follow-up. The results we obtained warrant implementing a new method to identify which nodules are the best candidates for effective treatment in one single RF session.

Both RF and L are able to induce a significant and durable volume reduction of benign thyroid nodules, thus representing a valid alternative option to surgery [22-26]. When the physician and patient have to decide between a thermal treatment or surgery, it would be important to know how much the nodule will be reduced once treated with thermal ablation and the duration of the effect.

Few studies of RF and L have long-term follow-up data. A Korean multicenter study, consisting of 345 nodules, followed patients for 5 years. The authors reported a significant and durable reduction of nodules and a significant improvement in symptoms and cosmetic scores [27]. It should be noted that few patients reached the 4 or 5-year follow-up milestone (57 and 6, respectively), and two RF sessions were necessary for 20% of the patients, with a mean time to additional RF ablation of 8.5 ± 7.5 months after the initial RF ablation. Another Italian single-center study evaluating 215 patients treated with only one session, found a mean 5-year VRR of 67% [10]. For L, studies with a 3- to 6-year follow-up, showed a reduction rate of approximately 50–60%. Symptoms and cosmetic score decreased significantly in 70–80% of the patients [9, 11-13]. Therefore, studies conducted worldwide demonstrate that both RF and L are effective up to 5 years following thermal treatment.

Analysis of the thermal effects on nodule characteristics indicate that VRR and durability of reduction differ from nodule to nodule. These two outcomes may be influenced by several factors, most importantly baseline volume and energy delivered. Nodule volume exceeding 20–30 mL (>3–4 cm of maximum diameter) and less delivered energy resulted in less efficacy. Literature data indicate that larger nodules receive proportionally less energy as compared to smaller nodules [16, 27].

As the aim of the study is to predict the VRR of each nodule before it is subjected to RF, the energy that will be delivered cannot be established a priori. Thus, the training of the machine learning model cannot take into account the energy that will be delivered to each patient. It has been established by published studies that the energy delivered is a critical factor in determining the final outcome. In our dataset, 86% of the nodules received an amount of energy equal to the mean value ± 1 SD of joules per mL, indicating that there was an elevated degree of homogeneity in the amount of energy delivered per mL in each nodule. Moreover, nodules that were reduced to <50% had a significantly greater volume than those that were reduced to ≥50%. The energy delivered/mL was significantly greater in the former, indicating that nodules that were reduced <50% were not treated with an insufficient amount of energy. In fact, the same percentage of nodules in both groups received an amount of energy equal to the mean value ± 1 SD of joules per mL. There was no difference in the distribution of patients who received the mean –1 SD of energy or the mean +1 SD of energy, suggesting that a similar amount of energy per mL was delivered. Table 2 shows the parity of nodule characteristics (echostructure, calcifications, vascularity); baseline volume represents a critical factor: the larger the nodule, the smaller the chance it will shrink. As RF is an operator-dependent technique, another critical issue pertains to the skill of the operator [28]. At such, it is worth noting that in two studies using RF, the performance of different centers was not significantly different, indicating that the use of shared techniques results in similar VRRs [16, 29]. As a matter of fact, if the performance among different centers was different, the machine learning model would not have been correctly trained and as a consequence, would not have displayed a valid response prediction.

In addition to quantitative measures such as nodule volume and delivered energy, structural features are also crucial. Both with RA and L, spongiform appearance of the nodule and intranodular vascularity are associated with a better response, probably because heat spreads more thoroughly than in compact nodules and periphery-only vascularized nodules [9, 16, 30].

As shown in Figure 3, we performed a simulation of nodules with different volume, structure, and vascularity. A 10-mL nodule with spongiform or solid structure and intranodular or perinodular vascularity had a >90% chance of a >50% reduction. However, a 25-mL nodule had an 83% chance of decreasing >50% if it was spongiform with intranodular vascularity. This chance decreases to 20% if it is solid with weak perinodular vascularity. This simulation helps to understand, for example, that ultrasound characteristics are of little relevance when treating small nodules, but mainly important when treating medium-large nodules. These results were confirmed by the analysis of the relative weight of input variables, which confirmed baseline volume as the major determinant of 12-month VRR.

This information may be relevant when offering a patient thermal treatment. Knowing whether one single session will result in a satisfactory effect or whether more than one session will be required to solve the mass effect is important, not only for the clinical outcome, but also for economic reasons. Few studies have evaluated the cost of thermal treatment versus surgery, and though the costs of one single thermal treatment may appear to be smaller, the cost obviously increases when more than one session is needed [31, 32].

The major limitation to our study is the number of nodules used to train the machine learning model. Though it may not be considered very large, it seemed to be enough to obtain reliable results. The second limitation is the criteria used for nodule classification including echostructure, macrocalcification, and vascularity, which we have autonomously established. The systematic registration of nodule characteristics equally for each patient before recruitment resulted in an increased level of accuracy in collecting data and enabling precise training of the machine learning model. This methodology is mandatory when merging data from different centers, to obtain truthful prediction by the machine learning model.

In conclusion, the present study demonstrates that our machine learning model is reliable and able to identify nodules that will have VRR <50% 12 months after one RFA treatment session. Indeed, as thermal therapies confront surgery, predicting which nodules will be poor responders represents valuable data that may help physicians and patients decide whether thermal ablation or surgery is a better option. Moreover, the data may predict whether more than one session is necessary to obtain a significant volume reduction.

Prospective trials should be performed with prespecified collection of ultrasound characteristics and preestablished energy to be delivered to treat larger number of nodules. This might offer the chance to precisely predict the VRR, and it may specify which nodules will not decrease significantly even following a second thermal procedure.

Statement of Ethics

The research presented in this article was ethically conducted in accordance with the World Medical Association Declaration of Helsinki and the appropriate guidelines for human studies as well as according to animal welfare regulations, including the Animal Research. This study was approved by the institutional review board (IRB). Informed consent from patients was waived by the IRB because of the retrospective nature of this study.

Disclosure Statement

The authors have no conflicts of interest to declare.

Funding Sources

None.

Author Contributions

Roberto Negro and Maurilio Deandrea were responsible for study conception, drafting the manuscript and data evaluation; Matteo Rucco was responsible for data evaluation and statistical analysis; Maurilio Deandrea, Alberto Mormile, Paolo Piero Limone, Roberto Garberoglio, Stefano Spiezia, Salvatore Monti, Christian Cugini, Ghassan El Dalati were responsible for radiofrequency procedures, data collection, and for revising the work critically for important intellectual content.