An Interobserver Comparison of the Ultrasound Lexicon Classification of Thyroid Nodules: A Single-Center Prospective Validation Study

TL;DR

This study evaluates how consistently different radiologists classify thyroid nodules using a standardized ultrasound lexicon, finding varying levels of agreement across different categories.

Contribution

The study provides a single-center validation of interobserver agreement using the I-TIRADS framework for thyroid nodule classification.

Findings

The 'mixed predominantly cystic' classification showed the highest consensus among assessors (κ = 0.729).

The 'Pure Cyst' subcategory had the lowest agreement (κ = 0.292).

Echogenic foci or calcifications showed moderate agreement across all readers.

Abstract

Background/Objectives: Guidelines for the risk stratification of thyroid nodules are based on certain well-recognized sonographic features of nodules. However, significant variations in reported sensitivity and specificity values are observed due to the overlap of imaging characteristics between benign and malignant nodules. Additionally, differences in ultrasound (US) equipment and the varying experience levels of radiologists performing the imaging procedures contribute to these discrepancies. Inevitably, there are also interobserver differences. The aim of this study was to investigate interobserver agreement on these criteria using the international thyroid imaging reporting and data system (I-TIRADS) thyroid evaluation framework, independently assessed by three residents and one consultant. Methods: We included 393 patients who underwent ultrasound-guided fine needle aspiration biopsy (FNAB) within four months. In each case, longitudinal and transverse video images of the thyroid gland, neck chain, and biopsied nodules were recorded. The evaluations of the parameters defined in the I-TIRADS dictionary were then performed by a radiologist with 15 years of experience and radiology assistants with 3, 3, and 2 years of experience, respectively, blinded to the images, pathology data, and patient demographics. The parameters evaluated included composition, echogenicity, margin, direction of growth, calcification, extension beyond the thyroid, and lymph node. An interobserver comparison between the US lexicon classifications of thyroid nodules was then performed. Results: The results of our study showed that the highest level of consensus was observed in the ‘mixed predominantly cystic’ classification, indicating a solid consistency between the assessors (κ = 0.729). Conversely, the subcategories ‘Solid’, ‘Mixed Predominantly Solid’ and ‘Spongiform’ showed moderate agreement, while the “Pure Cyst” subcategory exhibited the lowest level of agreement among the assessors (κ = 0.292). Agreement among the three radiology assistants was strong concerning the evaluation of nodule composition, growth direction, and lymph node assessment. In contrast, a moderate level of consensus was noted regarding the assessment of extrathyroidal extension, margins, and echogenicity. Notably, the parameter exhibiting moderate agreement across all readers was the presence of echogenic foci or calcifications. Conclusions: the reproducibility observed in the parameters defined within the lexicon supports its potential to enhance consistency and interobserver agreement in thyroid nodule assessment.

Figures

Full text

1. Introduction

In the analysis of the GLOBOCAN 2020 data, the rate per 100,000 population was 10.1 for women and 3.1 for men. This indicates that thyroid cancer is significantly more common in women [1]. However, the mortality rate was found to be relatively low, with female patients demonstrating a rate of 0.5 per 100,000 and male patients exhibiting a rate of 0.3 per 100,000. Thyroid ultrasound is a widely used diagnostic tool for the discrimination of benign nodules from malignant nodules, in addition to its role in guiding fine needle aspiration cytology in cases of suspicious nodules [2,3]. The utilization of fine-needle aspiration biopsy (FNAB) has been demonstrated to yield a diagnostic yield of 9.2–14.8% for malignancies identified via clinical examination [4,5,6]. The ultrasonography (US) features that have been shown to be suggestive of malignancy include the presence of microcalcifications, hypoechogenicity, irregular margins, the absence of halos, a predominantly solid composition, and intranodular vascularity [4,5,6,7].

However, sensitivity and specificity values vary in the literature due to the overlap between benign and malignant nodules, the use of different US equipment, the difference in experience between the radiologists performing the imaging, and inevitably interobserver differences [4,6,8,9]. Furthermore, the increase in incidental thyroid nodules detected using imaging modalities such as US, computed tomography (CT), magnetic resonance imaging (MRI) scans, and 18F-fluorodeoxyglucose (FDG)-positron emission tomography (PET)/CT scans highlights the importance of appropriate clinical management of these patients [10,11,12,13,14,15,16,17]. In clinical management, it is crucial to recognize the different characteristics of thyroid nodules and patient-specific risk profiles. Therefore, over the last two decades, many organizations have developed ultrasound-based risk stratification systems (RSSs) and management guidelines for thyroid nodules [2,18,19,20,21,22,23]. These classification systems aim to identify the risk of malignancy and guide management while establishing a standardized language for communication and reporting. However, estimated cancer risk and management recommendations for a nodule may vary depending on the system used, which can lead to confusion among clinicians and patients [24].

In light of the aforementioned information, Durante et al. tried to develop a lexicon and atlas of US descriptors for thyroid nodules using I-TIRADS [25]. In the present study, observers were tasked with the evaluation of descriptors within seven distinct categories, namely (1) composition, (2) echogenicity, (3) margin, (4) shape, (5) echogenic foci, (6) extrathyroidal extension, and (7) lymph nodes. The objective of this study was to investigate the interobserver agreement of these criteria using the I-TIRADS thyroid evaluation criteria, evaluated separately by three residents and one consultant.

2. Materials and Methods

This study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of Antalya Training and Research Hospital (approval no: No: 2448/dated 25 June 2019). In this prospective study, patients were prospectively followed up from September 2023 to January 2024, and informed consent was obtained. US-guided FNABs were performed in 441 thyroid nodules in 441 patients. The risk stratification system (RSS) for the I-TIRADS lexicon has not yet been officially defined. However, several parameters outlined in the lexicon, such as echogenicity, calcifications, and margin characteristics, have been widely discussed in the literature regarding their sensitivity and specificity in distinguishing benign from malignant thyroid nodules [3,4,6,8]. Some of these features are also utilized in existing RSS classifications [18,19,20]. In our study, biopsy decisions were based on the parameters explicitly defined in the I-TIRADS lexicon [25]. Each nodule was classified according to the I-TIRADS criteria described by Durante et al., with particular attention to lexicon-defined ultrasound characteristics, and no additional sources were considered [25]. All FNABs were performed under ultrasound guidance using an 18G fine needle, ensuring appropriate sterilization protocols, and conducted by experienced interventional radiologists. The demographic data of all patients were obtained from the automation system of our hospital. Image quality was assessed by the expert radiologist of this study before the grading process began. Patients included in this study met the following criteria: they were over 18 years of age, had ultrasound images of sufficient quality for diagnostic evaluation, and had no history of prior thyroid surgery, ablation, or embolization therapy. Patients with inadequate imaging quality or a history of interventional thyroid procedures were excluded to ensure consistency and reliability in the assessment. Finally, 393 patients (86 men and 307 women; mean age, 53 years; age range, 16–80 years) with 393 nodules were enrolled in this study.

US examinations were performed on a Toshiba Aplio 500 (Canon Medical Systems, Tokyo, Japan) equipped with a high-frequency 7.5 MHz linear array transducer. In each case, longitudinal and transverse video images of the whole thyroid gland, the neck chain, and biopsied nodules were recorded. After recording, patient data were removed from videos and images. A radiologist with 15 years of experience and radiology residents with 3, 3, and 2 years of experience, respectively, who were blinded to the pathology data and patient demographics, participated in this study with the following objectives. Prior to the assessment, all assessors were provided with the relevant I-TIRADS lexicon article prepared by Durante et al. and allocated sufficient time for familiarization and review [25]. The video images of the entire thyroid parenchyma, neck chain, and biopsied nodule were sent to each evaluator via cloud storage. The evaluations of the parameters defined in the I-TIRADS lexicon (composition, echogenicity, margin, direction of growth, calcification, extension beyond the thyroid, and lymph node) were recorded independently from each other. All assessments were requested from the observers to be completed within 4 months. The retrospective assessment of the ultrasonographic features of the nodules was followed by their classification according to the I-TIRADS Lexicon. The internal content of a nodule was categorized as solid, mixed predominantly solid, mixed predominantly cystic, spongiform, or pure cystic (Figure 1). The echogenicity of the nodule was categorized as hypoechogenic (marked or mild), isoechogenic, hyperechogenic, and anechoic, with the predominant echogenicity being referenced against that of the normal thyroid gland and anterior neck muscle (Figure 1). The margin of the nodule was categorized as smooth, irregular, or ill defined (Figure 1). The direction of growth was classified as wider-than-tall or taller-than-wide (Figure 2). Calcification was classified as microcalcification, macrocalcification, rim calcification, or echogenic foci with comet-tail artifacts (Figure 2). Extrathyroidal extension was evaluated as gross extrathyroidal extension, suspicious minor extrathyroidal extension, or capsule contact (Figure 3). Lymph nodes were categorized as suspicious, indeterminate, or nonsuspicious lymph nodes (Figure 3).

Statistical analyzes in this study were performed using the SPSS 27.0 (IBM Inc., Chicago, IL, USA) program. The Kolmogrov-Smirnov test, histogram analysis, skewness/kurtosis data, and Q-Q plots were used to evaluate the assumptions of a normal distribution. Qualitative parameters are expressed as frequency and percentage (%). The descriptive statistics of the numerical and categorical American Thyroid Association (ATA) obtained in this study were analyzed and expressed as IQR (median [minimum–maximum]) since the quantitative parameters did not exhibit a normal distribution pattern. Relationships between the two groups were examined with the Mann–Whitney U test. Relationships between nominal parameters were evaluated with either chi-square analysis or Fisher’s exact tests. The agreement between the interobservers and the reliability of decisions was determined using Fleiss kappa analysis. In the entire study, the type-I error rate was taken as 5% (α = 0.05), and p < 0.05 was accepted as the significant limit.

3. Results

3.1. Study Population

The mean patient age was 53 (range, 16–80 years). The mean nodule size was 18 mm (mm) (range, 5–70 mm) (Table 1).

A statistical analysis revealed that female patients exhibited greater age values (p < 0.001), while male patients exhibited higher nodule sizes (p = 0.003), (Table 2). The number of multiple nodules was 79 (91.9%) in male patients and 273 (89.9%) in female patients, while the number of patients with a single nodule was seven (8.1%) and thirty-four (11.1%), respectively (p = 0.431). The number of patients with nodules larger than one centimeter (cm) was 81 (94.2%) in males and 276 (89.9%) in females, while the number of patients with nodules smaller than one cm was five (5.8%) and thirty-one (10.1%), respectively (p = 0.224), (Table 2).

3.2. Agreement for US Characteristic According to the I-TIRADS Lexicon

The US features were evaluated by assessing the interobserver agreements between all four observers, consisting of one radiology expert and three radiology residents. The interpretation and classification of Kappa values were carried out in accordance with Cohen’s (1961) recommendations, following objective and generally accepted literature suggestions for interpretation [26]. The results revealed a statistically substantial consensus across all evaluations. Notably, there was significant agreement on echogenicity, echogenic foci/calcifications, and extrathyroidal extension, with agreement levels at acceptable (fair) values (κ = 0.343, κ = 0.389, and κ = 0.333, respectively). When the evaluations of composition, margin, direction of growth, and lymph nodes were examined, moderate agreement was observed among the observers (κ = 0.538, κ = 0.431, κ = 0.481, and κ = 0.452, respectively). A detailed summary of interobserver agreement percentages and kappa values is provided in Table 3.

Table 4 presents the interobserver agreement among the three readers (residents) for the ultrasound features and the I-TIRADS levels. The agreement among the three radiology residents was good for nodule composition, direction of growth, and lymph node evaluation (κ = 0.728, κ = 0.657, and κ = 0.625, respectively). Conversely, a moderate degree of consensus was observed regarding the evaluation of extrathyroidal extension, margin, and echogenicity (κ = 0.571, κ = 0.585, and κ = 0.565, respectively). The only parameter with fair agreement was the echogenic foci/calcifications among all readers (κ = 0.391).

We conducted a detailed analysis of the decisions made by four observers and evaluated their interobserver agreement across various subgroups of sonographic measurements, as summarized in Table 5. The highest level of consensus was observed in the “mixed predominantly cystic” decision, indicating strong consistency (κ = 0.729). This elevated concordance may be attributed to being well defined in the lexicon, with clear and distinct imaging characteristics, reducing ambiguity during classification. Moderate agreement was found in the “Solid” and “Mixed Predominantly Solid” subgroups (κ = 0.552, κ = 0.498, respectively), while “Spongiform” demonstrated fair agreement (κ = 0.497). The “Pure Cyst” subgroup showed the lowest agreement (κ = 0.292).

For echogenicity, moderate agreement was achieved in “Markedly Hypoechoic” (κ = 0.514), whereas other subgroups, including “Mildly Hypoechoic”, “Isoechoic”, and “Anechoic”, exhibited fair agreement (κ = 0.352, κ = 0.349, and κ = 0.341, respectively). The lowest agreement was noted in the “Hyperechoic” subgroup (κ = 0.151).Regarding margins, the “Smooth” subgroup showed moderate agreement (κ = 0.511), while the “Irregular” and “Defined” subgroups had fair agreement (κ = 0.352, κ = 0.361, respectively). For direction of growth, both the “Taller-than-Wide” and “Wider-than-Tall” subgroups demonstrated moderate agreement (κ = 0.481).

In the evaluation of echogenic foci/calcifications, moderate agreement was observed in “Macrocalcifications” (κ = 0.532), while “Punctate Echogenic Foci” showed fair agreement (κ = 0.448). “Echogenic Foci with Comet-Tail Artifact”, “Peripheral (Rim) Calcifications”, and “Without Foci” had the lowest agreement levels (κ = 0.394, κ = 0.321, and κ = 0.275, respectively). For extrathyroidal extension, all subgroups demonstrated fair agreement. “Capsula Contact” showed the highest consistency (κ = 0.378), followed by “Gross Extrathyroidal Extension” and “Suspicious Minor Extrathyroidal Extension” (κ = 0.298, κ = 0.288, respectively).

Finally, in the evaluation of lymph nodes, moderate agreement was observed in “Nonsuspicious Lymph Nodes” (κ = 0.504) and “Suspicious Lymph Nodes” (κ = 0.483), while “Indeterminate Lymph Nodes” exhibited fair agreement (κ = 0.364).

These findings highlight variability in interobserver reliability across different subgroups, with κ values indicating agreement levels ranging from fair to strong. All results were statistically significant (p < 0.001).

4. Discussion

The results of our study showed that the highest level of consensus was observed in the ‘mixed predominantly cystic’ classification, indicating a solid consistency between the assessors. However, the subcategories ‘Solid’ and ‘Mixed Predominantly Solid’ showed moderate agreement, while the category ‘Spongiform’ also reflected moderate agreement. Higher agreement in these categories might be achieved if the boundaries between them were more clearly defined. On the other hand, the ‘Pure Cyst’ subcategory exhibited the lowest degree of agreement between the assessors. The low interobserver agreement for the “Pure Cyst” category (κ = 0.292) is likely due to the absence of a standardized definition in TIRADS lexicons, which leads to inconsistencies among observers [25]. Additionally, ultrasound artifacts and image quality issues, such as posterior acoustic enhancement and comet-tail artifacts, further complicate accurate classification [27]. Furthermore, some cystic nodules contain small solid components, leading to misclassification and overlap with other subcategories, highlighting the need for improved lexicon standardization [28]. Agreement among the three radiology assistants was strong concerning the evaluation of nodule composition, growth direction, and lymph node assessment. In contrast, a moderate level of consensus was noted regarding the assessment of extrathyroidal extension, margins, and echogenicity. This moderate interobserver agreement likely stems from the inherent subjectivity in interpreting these ultrasonographic features. For extrathyroidal extension, moderate agreement may result from difficulties in distinguishing true capsular invasion from perithyroidal tissue contact or compression artifacts. The absence of precise criteria for minor versus gross extrathyroidal extension in the I-TIRADS lexicon may also contribute to discrepancies. With regard to margins, the challenge lies in differentiating between smooth, irregular, and ill-defined borders, especially in cases with overlapping characteristics or suboptimal image quality. Echogenicity assessments are similarly subjective and are influenced by technical factors such as gain settings and operator experience. Moreover, the parameter exhibiting moderate agreement across all readers was the presence of echogenic foci or calcifications. This variability may be attributed to the small size and subtle appearance of these features, which can be easily overlooked or misclassified, especially in nodules with heterogeneous internal structures.

The I-TIRADS lexicon is emerging as a global consensus ultrasound dictionary that aims to provide consistent, sustainable, and objective communication between radiologists and clinicians [25]. An effective lexicon strives for high diagnostic accuracy and strong interobserver consistency, regardless of experience. Ultrasound is inherently subjective and user-dependent; however, guidelines developed through studies have enabled more objective evaluations and higher diagnostic accuracy for distinguishing benign from malignant thyroid lesions, as is the case for other organs and systems [22,25,29,30].

Numerous studies evaluating the diagnostic performance and interobserver consistency of the European TIRADS (EU-TIRADS), American College of Radiology TIRADS (ACR-TIRADS), Korean TIRADS (K-TIRADS), and Chinese TIRADS guidelines for thyroid ultrasound are available in the literature [18,27,29]. In our study, we aimed to investigate the differences in the evaluation of the I-TIRADS lexicon between observers [22,23,24,25,29,30].

Previous interobserver reliability studies have reported variable levels of agreement for ultrasonographic features defined in existing TIRADS classifications, such as composition, shape, calcifications, echogenicity, and margins [27,30,31,32]. It is known that interobserver agreement also varied greatly depending on the observers’ experience [27]. Moreover, well-defined ultrasonographic features in lexicons have been shown to improve interobserver agreement regardless of experience. On the other hand, insufficient definitions and the use of subjective criteria increase the rate of unnecessary invasive procedures and impose additional costs, thereby increasing the burden on the healthcare system [33,34,35].

A recent multicenter study reported insufficient interobserver agreement in the evaluation of nodule margins, whereas the agreement for microcalcifications was moderate. These findings contrast with earlier studies that reported higher levels of agreement. Additionally, the study revealed that smaller nodules showed better agreement in echogenicity assessments, while larger nodules exhibited greater agreement in identifying the presence of microcalcifications [32].

In our study, high interobserver consistency was observed across all parameters in the evaluation of thyroid nodules. Among the four evaluators, including a radiology expert with over 20 years of experience in thyroid assessments, moderate agreement was achieved for the parameters of composition, margin, direction of growth, extrathyroidal extension, and lymph node assessment. There was fair agreement in the evaluation of extrathyroidal extension, echogenicity, and calcification. Significant agreement was observed in all parameters, with the direction of growth being the most significant. These findings indicate that the I-TIRADS lexicon adequately ensures interobserver agreement, independent of experience. However, the results of our study do not surpass those achieved with previous TIRADS classifications. This raises questions about the I-TIRADS lexicon’s contribution to enhancing objectivity compared to the existing RSSs [27,30,36]. Although our study is single-centered and limited by a relatively small sample size, drawing definitive conclusions at this stage is premature. Larger, multicenter studies are necessary to clarify the contribution of the I-TIRADS lexicon to objective evaluation and interobserver agreement.

The comparison of the three radiology residents showed nominally stronger agreement than the analysis with the expert. There was good agreement on composition, growth direction, and lymph node parameters; moderate agreement on extrathyroidal extension, echogenicity, and margin parameters; and fair agreement on the calcification parameter. The higher agreement among the residents may stem from their shared education and training within the same institution. Additionally, all participating residents had completed their formal ultrasound training and were instructed to study the I-TIRADS lexicon before being included in this study. This preparatory approach may have contributed to the higher consistency observed among the residents. In contrast, some studies in the literature have reported reduced agreement levels between junior and senior observers due to variations in experience [37]. However, in our study, the groups including and excluding the expert yielded similar results, with the junior-only group demonstrating higher agreement across all parameters. This finding highlights the potential impact of standardized training and familiarity with specific lexicons in achieving consistent evaluations [38].

Regarding the subgroups, it was observed that the low level of agreement in hyperechoic parameter decreased the overall level of agreement in echogenicity evaluations. The low agreement in hyperechoic nodules suggests a need for more objective descriptors within the lexicon, as hyperechogenicity is highly dependent on machine settings and operator technique, leading to greater variability among observers [3,20,22]. This emphasizes the necessity for more precise and objective definitions within the I-TIRADS lexicon. In the lexicon, thyroid nodules without echogenic foci or calcifications are excluded from classification under the echogenic focus/calcification parameter [25]. However, it is well documented that nodules lacking echogenic foci or calcifications do exist, raising concerns about the comprehensiveness of this categorization [22,38,39]. The current version of I-TIRADS would benefit from the inclusion of additional subcategories such as “without foci” under the echogenic focus/calcification category. Improvements in categorization criteria may enhance the clarity and objectivity of the lexicon, ultimately aiding in better clinical decision-making.

In our analysis of subgroups, the parameters demonstrating the highest interobserver agreement included mixed predominantly cystic, solid composition, markedly hypoechoic echogenicity, smooth margins, macrocalcifications, and nonsuspicious lymph nodes. Conversely, parameters with the lowest levels of agreement were pure cyst, hyperechoic echogenicity, absence of foci, gross extrathyroidal extension, and suspicious minor extrathyroidal extension. These findings suggest that refining and providing more objective criteria for these parameters could enhance interobserver agreement. To validate the findings of our study, multicenter research with a larger and more diverse group of evaluators is required. Increasing both the number and the variability in the evaluators’ experience would generate higher-quality data, thereby improving the reliability of the lexicon and its application in clinical practice.

As highlighted in a 2024 meta-analysis, ACR-TIRADS, EU-TIRADS, and K-TIRADS categorize thyroid nodules quite differently. K-TIRADS tends to classify nodules as high risk, ACR-TIRADS as intermediate risk, and EU-TIRADS as low risk. This variation in categorization supports the observation that these RSSs exhibit differing behaviors when assessing thyroid nodules, particularly in malignancy risk estimation. The findings are critical for developing a unified international system like I-TIRADS to enhance consistency across clinical practices [40].

Moreover, advancements in artificial intelligence strategies are also being utilized in the evaluation of thyroid nodules. In a study conducted by Tong et al., it was suggested that an optimized artificial intelligence strategy for thyroid nodule management could reduce diagnosis time-related costs without compromising diagnostic accuracy for senior radiologists. However, it is noted that a traditional, fully artificial intelligence strategy may still be more beneficial for less experienced radiologists [41].

Our study has several limitations that need to be addressed. First, the relatively small number of nodules evaluated and the low rate of malignancy among nodules undergoing FNAB may have affected the evaluations. Second, the fact that all observers were from the same institution may have impacted the consistency of the assessments. To overcome these limitations, future studies should aim to adopt a multicenter approach. Increasing the number and diversity of cases can help to improve the effectiveness of the evaluations. Furthermore, the inclusion of additional case series from different centers, with varying levels of observer experience and ultrasound devices, may provide more comprehensive and reliable information.

5. Conclusions

Although different ultrasound parameters were evaluated, our interobserver agreement study yielded results consistent with those of studies on existing RSS classifications. It is noteworthy that various RSSs, such as EU-TIRADS, K-TIRADS, and ACR-TIRADS, assess thyroid nodules, particularly low-risk nodules, in differing ways. However, I-TIRADS offers a universal lexicon. The I-TIRADS lexicon aims to establish a globally standardized RSS for thyroid nodule evaluation, providing a unified framework for classification. The reproducibility observed in the parameters defined within the lexicon supports its potential to enhance consistency and interobserver agreement in thyroid nodule assessment.