Improvement in neck ultrasound report quality following the implementation of European Thyroid Association guidelines for postoperative cervical ultrasound for thyroid cancer follow-up, a prospective population study

Study subjects

This study was approved by Health Research Ethics Board of Alberta (HREBA) – Cancer Committee (CC) (Ethics ID: HREBA.CC-16-0956). We reviewed 805 patients taken from our prospective REDCap Calgary thyroid cancer database, written consent was obtained from each patient after a full explanation of the purpose and nature of all procedures used. These patients were diagnosed with thyroid cancer and underwent thyroid surgery between April 1, 2017, and March 1, 2023. After excluding 10 patients, whose surgeries were done in other regions, 17 patients with non-invasive follicular thyroid neoplasm with papillary-like features (NIFTP), 8 with medullary thyroid cancer, 2 with anaplastic thyroid cancer, 2 whose tumors were not resected, and 1 struma ovarii, we were left with 775 patients with differentiated thyroid cancer (DTC). We only included patients with at least one POU at their 1-year FU visit. Thus, after further excluding 69 patients with no POU because of short FU, 9 patients without available POU reports, 8 without post-RAI ultrasound, and 7 lost to FU, we included the remaining 672 DTC patients in our analysis and followed them until their last FU (Fig. 1). About 87% of 672 patients underwent thyroid surgery by 1 of 11 high-volume thyroid surgeons.

View Full Size

Patient flow chart. DTC, differentiated thyroid cancer; NPV, negative predictive value; POU, postoperative neck ultrasound; RAI, radioiodine remnant ablation; Tg, thyroglobulin; US, ultrasound.

Citation: European Thyroid Journal 12, 5; 10.1530/ETJ-23-0110

• Download Figure
• Download figure as PowerPoint slide

As per ATA guidelines, the initial FU assessment occurred between 6 and 12 months after surgery and was classified as the 1-year FU, and the 3-year FU visit occurred between 30 and 36 months and was classified as the 3-year FU. During FU assessment at 1-year and 3-year post-treatment, RTT for patients who received TTX plus RAI was classified according to the 2015 ATA guidelines for adult patients with thyroid nodules and DTCs as previously reported (12). In brief, for patients with TTX only or lobectomy, the ATA guidelines do not give specific criteria for RTT. We therefore followed the RTT classification criteria reported by Momesso et al. (12) and the criteria are summarized in Supplementary Table 1 (see section on supplementary materials given at the end of this article). Patients who underwent TTX plus RAI vs TTX alone vs lobectomy were categorized as excellent RTT, indeterminate RTT, biochemical incomplete RTT, or structural incomplete RTT, depending on their biochemical markers and FU ultrasound.

POU evaluation

Starting from January 2018, IRG implemented the ETA guideline-based malignancy risk stratification system for all nodules reported, which categorizes nodules as ETA normal, ETA indeterminate, or ETA suspicious. IRG implemented a predetermined list of terms in a dropdown menu and required fields for each nodule characteristic and the malignancy risk classification. The purpose of these changes was to standardize and accelerate the reporting process in accordance with the ETA guideline-based POU assessment. All non-implementation radiology groups (NIRGs) continued with varied assessment strategies without ETA guideline implementation. Lesions that were not risk stratified according to the ETA guidelines were retrospectively reclassified with an ETA risk classification based on the POU report.

For IRG, the participating radiologists had between 4 and >20 years of experience in assessing postoperative thyroid cancer patients. At the time of implementation, all involved radiologists in IRG had a 2-h refresher course on the specifics of the implemented program, as well as access to written resources and live consultation with an expert for difficult cases. In the highest POU quality non-implementation radiology group (NIRG-HP), the participating radiologists had between 1 and >20 years of experience in assessing postoperative thyroid cancer patients, while the experience of respective radiologists in other non-implementation radiology groups (NIRG-OT) is unknown.

At each FU, ultrasound lesion status was classified as normal (US-N) if all visible lesions were considered normal or no lesion mentioned; indeterminate (US-I) if there were no suspicious lesion but at least one lesion classified as indeterminate; or suspicious (US-S) when at least one lesion was classified as suspicious according to the ETA guideline (7). Serum Tg level status was classified as thyroglobulin levels-negative (Tg-N), thyroglobulin levels-indeterminate (Tg-I), or thyroglobulin levels-suspicious (Tg-S). The definition for each serum Tg level status is provided in Supplementary Table 2.

We followed the utility score (UtS) used by Hu et al. and Symonds et al. for thyroid bed nodules to evaluate the quality of neck US reports in describing thyroid bed nodules (13, 14). For each POU report, the UtS was calculated based on the number of thyroid bed lesion characteristics included in the report that were outlined in the 2013 ETA guidelines. The POU report received 1 point for every characteristic given. The UtS of POU reports ranged from 0 to 7. The characteristics are outlined in Supplementary Table 3. The UtS was evaluated for all abnormal thyroid bed nodules recorded in all POU reports in each FU. Unlike thyroid bed nodules, POU reports that identify abnormal lymph nodes typically only provide information about abnormal features, mostly suspicious features. As a result, even if the UtS is low, the report can still be considered adequate for abnormal lymph nodes. Therefore, we introduced an additional POU quality assessment criterion based on the sufficiency rate, and the quality of POU reports describing lymph nodes was evaluated by the sufficiency rate rather than UtS. A sufficient POU report is defined as including both indeterminate or suspicious lesion characteristics and the lesion classification following the abnormal lesion classification rules summarized in accordance with the ETA guidelines (Supplementary material).

Baseline Tg and POU

We further focused on two specific cohorts within our DTC patient population to evaluate the predictive ability of baseline Tg and POU: cohort 1 included those who had US-N along with Tg-N at 1-year FU and had FU at 3 years, cohort 2 included those who had US-N and Tg-I at 1-year FU and had FU at 3 years (Fig. 1). We then assessed the ultrasound status of these patients at 3-year FU and determined the negative predictive value of their 1-year FU investigations (NPV, defined as the percentage of patients with US-N at 3-year FU, in concordance with previously published definitions) (6).

All statistical analyses were performed with the R statistical software package (15). Continuous variables were expressed as medians and ranges, while nominal variables were expressed as frequency counts and percentages. A P-value of <0.05 was considered statistically significant.

How did the implementation of ETA POU guidelines impact POU report quality in our health-care region?

POU UtS

The assessment of POU quality involved 672 patients with DTC, of whom 299 were treated with TTX and RAI, 253 with TTX only, and 120 with lobectomy. A total of 2212 POU reports were available for the 672 patients, 1600 by IRG and 612 by NIRG, identifying 362 indeterminate lesions (270 identified by IRG and 92 by NIRG, prevalence of 17% and 15% respectively) and 210 suspicious lesions (171 identified by IRG and 39 by NIRG, prevalence of 11% and 6% respectively, Fig. 1).

The UtS of POU reports on indeterminate or suspicious thyroid bed nodules by IRG was significantly higher than NIRG (4.84 vs 3.62 for indeterminate nodules, P < 0.0001; 5.24 vs 3.95 for suspicious nodules, P < 0.001, Table 1). After the implementation of the ETA guidelines by IRG, the mean UtS increased for both IRG and NIRG compared to their own pre-implementation US reports (4.98 vs 3.88 for IRG, P < 0.05; 3.81 vs 2.96 for NIRG, P < 0.05; Table 2). IRG continues to have a significantly higher mean UtS than NIRG after the implementation (4.98 vs 3.81, P < 0.0001, Table 2). The difference is visualized by the UtS distribution demonstrated in Fig. 2. When comparing the effect sizes of the difference between the two groups using Cohen's d values, we observed that the effect size was larger for the post-ETA measurements (0.82) compared to the pre-ETA measurements (0.60). This indicates that the difference in mean UtSs between the two groups is more significant after the ETA implementation than before.

View Full Size

Distribution of indeterminate/suspicious thyroid bed nodules in each UtS category for different radiology groups. IRG, implementation radiology group; NIRG, all non-implementation radiology groups; UtS, utility score.

Citation: European Thyroid Journal 12, 5; 10.1530/ETJ-23-0110

• Download Figure
• Download figure as PowerPoint slide

Table 1

Mean UtS for indeterminate/suspicious thyroid bed nodules.

Mean UtS	IRG (n = 236)	NIRG (n = 83)	P
TTX & RAI	146	30
Indeterminate nodules	4.65	3.1	<0.0001
Suspicious nodules	5.26	3.2	<0.0005
TTX only	40	24
Indeterminate nodules	4.06	3.14	<0.05
Suspicious nodules	4.29	5a
Lobectomy	50	29
Indeterminate nodules	5.97	4.62	<0.0005
Suspicious nodules	5.69	4.63	<0.05
All treatment types	236	83	<0.0001
Indeterminate nodules	4.84	3.62	<0.0001
Suspicious nodules	5.24	3.95	<0.001

^aTwo nodules in this subgroup.

IRG, implementation radiology group; NIRG, all non-implementation radiology groups; RAI, radioiodine remnant ablation; TTX, total thyroidectomy; UtS, utility score.

Table 2

Mean UtS for indeterminate/suspicious thyroid bed nodules: pre-ETA and post-ETA.

Mean UtS	IRG	NIRG	P-value
Pre-ETA	3.88 (95% CI 2.76, 5.00)	2.96 (95% CI 2.60, 3.31)	<0.05
Post-ETA	4.98 (95% CI 4.79, 5.16)	3.81 (95% CI 3.51, 4.10)	<0.0001
P-value	<0.05	<0.05

ETA, European Thyroid Association; IRG, implementation radiology group; NIRG, all non-implementation radiology groups; UtS, utility score.

What is the predictive ability of baseline Tg and POU at 1-year FU in determining the absence of persistent disease or relapse at 3-year FU?

Among 269 patients with Tg-N at 1-year FU, 252 (94%) had US-N at 3-year FU POU, which corresponds to an NPV of 94% for Tg-N at 1-year FU. For 345 patients with US-N at 1-year FU, 330 (96%) had US-N at 3-year FU POU (Table 4).

There were 242 patients in cohort 1 (96 with TTX plus RAI, 106 with TTX, and 40 with lobectomy) and 67 in cohort 2 (36 with TTX plus RAI and 31 with TTX). Details of their tumor characteristics and surgery are outlined in Table 5. About 96% of cohort 1 and 96% of cohort 2 had US-N at 3-year FU (Supplementary Tables 4 and 5). This corresponds to an NPV for US-N and Tg-N at 1-year FU of 96% and an NPV for US-N and Tg-I at 1-year FU of 96% as well. Among cohort 1, the NPV of US-N and Tg-N was 93% (89 of 96) for TTX plus RAI, 99% (105 of 106) for TTX only, and 98% (39 of 40) for lobectomy, there was no statistical difference in NPV for US-N and Tg-N between the TTX+RAI, TTX only, and Lobectomy groups (P > 0.1, Supplementary Table 4). Among cohort 2, the NPV of US-N and Tg-I was 94% (34 of 36) for TTX plus RAI and 97% (30 of 31) for TTX (P > 0.1, Supplementary Table 5). Serum Tg level and ultrasound lesion status of all 672 patients at 1-Year FU are provided in Supplementary Table 6.

The impact of adherence to ETA postoperative ultrasound guidelines on POU report quality

The UtS and sufficiency rate of a POU report directly relate to the reliability and the amount of information available in the report to describe any thyroid bed nodule or lymph node identified. We previously showed that when radiology groups adhered to the 2015 ATA or 2017 Thyroid Imaging Reporting and Data Systems (TIRADS) reporting guidelines for thyroid ultrasound, they significantly improved the UtS and classification reporting rate of their reports compared to other radiology groups who did not implement strict adherence (13). Similarly, our current data show that IRG had significantly higher UtS compared to NIRG when reporting indeterminate or suspicious thyroid bed nodules, both prior to and after their adherence to the ETA guidelines (Table 1). The mean UtS was significantly higher for both IRG and NIRG compared to their POU reports before and after adhering to the 2013 ETA guidelines (Table 2). This could be explained by performance variations within NIRG. Specifically, one radiology group among NIRG exhibited significantly higher mean UtS and sufficiency rates for post-ETA POU reports describing abnormal thyroid bed nodules compared to the remaining groups (Supplementary Tables 7 and 8), which will be referred to as the highest-quality non-implementation radiology group (NIRG-HP). As a result of NIRG-HP’s performance, there was an overall improvement in POU quality for abnormal thyroid bed nodules among the NIRG, comprising NIRG-HP and eight other non-implementation radiology groups (NIRG-OT). However, it is worth noting that the number of POU performed by NIRG-HP was relatively small.

Considering the current reporting practice for lymph nodes, a POU report that describes an abnormal lymph node with a low UtS can still be considered clinically useful in assessing patient RTT if it provides information on abnormal features and corresponding malignancy risk classification. To facilitate the comparison of POU quality related to lymph nodes across different radiology groups, we also incorporated the sufficiency rate criterion to evaluate report quality. However, we still recommend that radiology groups further improve their POU reports describing abnormal lymph nodes by providing lymph node features and malignancy risk classifications according to the ETA guideline recommendations in the future.

POU reports done by IRG describing suspicious thyroid bed nodules have a 94% sufficiency rate (percentage of sufficient POU reports overall POU reports in this category), which is significantly higher than 45% by NIRG. The results are similar for indeterminate thyroid bed nodules (85% vs 27%), suspicious LNs (85% vs 68%), and indeterminate LNs (66% vs 14%). Additional data on NIRG-HP for POU reports describing abnormal lymph nodes showed lower sufficiency rates than IRG, particularly for indeterminate lymph nodes (Supplementary Table 8). Overall, among the four abnormal lesion categories, all radiology groups exhibited the lowest sufficiency rates for describing indeterminate lymph nodes (Supplementary Table 8). This finding highlights the need for improvement in lymph node description following the guideline-based assessment.

Based on our data, IRG had significantly higher quality POU reports compared to NIRG and this difference may be explained by the implementation of the 2013 ETA reporting guidelines, and this difference most likely leads to a reduction in unnecessary repeat POU. POU is crucial in the detection of persistent disease or recurrent disease during the FU of thyroid cancer patients after their initial treatment for their thyroid cancer. A high-quality POU report provides more pertinent information, enabling clinicians to more reliably estimate RTT and future RR.

Given the limited number of FNAs and additional surgeries performed for abnormal lesions in our cohorts, particularly those following POU done by NIRG, we found no significant difference in diagnostic accuracy between IRG and NIRG. Within IRG, FNAs or surgeries conducted for suspicious lesions had higher accuracy compared to those performed for indeterminate lesions. This finding is intuitive, as suspicious lesions are more likely to have suspicious features.

The 2015 ATA guidelines do not differentiate post-treatment ultrasound findings and Tg cutoffs between patients who received RAI vs no RAI, and there are limited data on abnormal findings in patients with lobectomy and Tg cutoffs (1, 16). A recent meta-analysis indicates that following total/near-TTX, using a Tg levels cutoff of 1–2.5 ng/mL might identify patients at low risk for persistent or metastatic disease (17). However, due to the limited availability of qualified patients (n = 10), we were unable to provide any conclusions for this cutoff. The quality of the POU report plays a crucial role in determining the need for subsequent biopsy and the diagnostic accuracy. If only POU reports of low quality are available, clinicians may have to repeat POU and Tg determination or request FNA to gather adequate information to assess the RTT and to make further decisions. POU reports with higher UtS or sufficient POU reports according to ETA guidelines would save many repeat procedures, resulting in lower patient anxiety, fewer complications, and reduced healthcare costs.

Prognostic value of baseline serum Tg and postoperative neck US at 1-year FU

It is well described that baseline Tg values early in FU have strong predictive value for US-N at 3-year FU (5, 6, 16), with Tg-N having an NPV as high as 98.8% and 98.2% for low- and intermediate-risk patients, respectively, and the NPV of Tg-I was as high as 98.2% and 94.5% for low- and intermediate-risk patients, respectively (6). Additionally, Grani et al. found that among low- and intermediate-RR patients with abnormal neck ultrasound at 3-year FU, 75% of 226 lesions were likely false positive and none required treatment; however, no further details were provided.

Similarly, the absence of structural disease early in FU also predicts the ongoing absence of structural disease over 5–10 years. Among 67 ATA high-risk patients with stimulated Tg <1 ng/mL before RAI, the rate of abnormal ultrasound findings is 3% after median FU of 5.6 years (18), while among those with persistently elevated or rising serum Tg, the rate is between 8% and 17% (19).

We observed similar rates in our prospective population data with the implementation of ETA POU guidelines. Patients with US-N at 1-year FU had an NPV of 96% for having US-N at 3-year FU, while Tg-N at 1-year FU had an NPV of 94% for US-N at 3-year FU (Table 4). However, we found that the NPVs for TTX plus RAI group were lower for both US-N and Tg-N compared to TTX only and lobectomy. For US-N, the NPV was 92% for TTX and RAI, 99% for TTX only, and 98% for lobectomy, respectively, and for Tg-N, the NPV was 89%, 99%, and 94%, respectively (Table 4). This may be due to a higher percentage of high-risk patients in the TTX and RAI group. After excluding all high-risk patients, the NPV for US-N at 1-year FU was 97% and for Tg-N at 1-year FU was 91%, which are similar to the NPVs for TTX only and lobectomy groups. For low- and intermediate-risk patients, the treatment modality did not affect the NPV of Tg-N or US-N. Finally, the combination of US-N and Tg-N (cohort 1) at 1-year FU had an NPV of 96%, which was not significantly different from using Tg-N alone (94%, P = 0.1) or US-N alone (96%, P> 0.1).

It is noted in a retrospective study of 619 patients with papillary thyroid cancer (PTC) treated with lobectomy that postoperative serum Tg levels or ultrasound findings do not correlate with disease recurrence (20), but other studies have found that Tg levels >30 ng/mL still indicate a high probability of recurrent structural disease (12). In our study, among 40 lobectomy patients who had US-N and serum Tg <30 ng/mL at 1-year FU, only one developed indeterminate ultrasound findings at 3-year FU (Supplementary Table 4). In this solitary patient, the nodule is stable and less than 1 cm without suspicious features, indicating a 0% recurrence rate at 3-year FU among lobectomy patients with excellent RTT at 1-year FU. However, relapses after lobectomy may occur very late (21).

Overall, our prospective data indicate that for patients treated with TTX, with or without RAI, a normal serum Tg at 1-year FU is highly predictive of no structural disease at 3-year FU and the addition of a neck ultrasound does not significantly increase the predictive value.

Strengths of our data include collection from a large population and in a prospective manner over a FU period of 3 years. We also included a large number of POU reports and many indeterminate and suspicious lesions in the analysis.

Our limitations include the low number of FNAs and subsequent surgeries performed after POU, particularly by NIRG. Our FU was restricted to 3 years, whereas the peak for thyroid cancer RR has been reported at 5 years (22, 23, 24). There are also potential confounders, such as variations in fellowship training, technologist expertise or experience, and potential differences in the referred patient populations for each radiology group. These unmeasured factors could introduce confounding effects and impact the comparison between groups.