Validity and Reproducibility of an Electronic Food Frequency Questionnaire in Argentinian Adults

TL;DR

This study validated an electronic food frequency questionnaire in Argentinian adults, finding it moderately accurate for estimating nutrient intake.

Contribution

The study provides validation data for a new electronic dietary assessment tool tailored to Argentinian adults.

Findings

The eFFQ showed moderate to good reproducibility for most nutrients with ICCs ranging from 0.29 to 0.85.

Agreement within one quintile of exact agreement ranged from 52.2% to 74% for validity and 70.6% to 87.7% for reproducibility.

The eFFQ slightly overestimates nutrient intake but is valid for ranking individuals' dietary habits.

Abstract

This study aimed to validate a semiquantitative electronic food frequency questionnaire (eFFQ) in estimating the intake of a comprehensive list of nutrients and bioactive compounds among adults from six regions of Argentina using multiple 24 h dietary recall (24HR) as a reference. A total of 163 adults completed two administrations of the eFFQ and four 24HRs. The paired t-test/Wilcoxon signed-rank test, Spearman/Pearson correlations, cross-classification, weighted kappa statistics, and Bland–Altman plots were employed to determine relative validity. To determine reproducibility, intraclass correlations (ICC), cross-classification, and weighted kappa statistics were calculated. For relative validity, crude correlations ranged from 0.15 to 0.57; energy adjustment and de-attenuation slightly improved most of these correlations. In cross-classification analysis, agreements within one quintile adjacent to exact agreement (EA ± 1) ranged from 52.2% to ~74%; extreme misclassifications were < 7%. For reproducibility, the crude ICC ranged from 0.29 to 0.85, showing moderate to good correlations for most nutrients. Cross-classification analysis showed agreement levels for the EA ± 1 quintile of 70.6% to 87.7%. Weighted kappa values ranged from 0.21 to 0.62. The results show that this eFFQ is relatively valid in ranking adults according to their nutrient intake and has an acceptable reproducibility, yet it slightly overestimates the intake of most nutrients.

Figures

Full text

1. Introduction

Non-communicable diseases (NCDs), such as cardiovascular diseases, cancers, respiratory diseases, and diabetes, are currently associated with more than two-thirds of all deaths worldwide and 86% of premature deaths that occur in low- and middle-income countries [1]. One of the predominant causes of the growing burden of NCDs is an unhealthy diet as demonstrated by various epidemiological studies that explore these diseases and their relationship with dietary patterns [2,3] and other diet-associated factors [4,5,6]. In Latin America, recent research has examined the relationships between dietary habits and NCDs [7,8,9,10,11,12]. Argentina experiences important sociocultural and economic inequalities that overburden the healthcare system and this scenario demands further studies in dietary lifestyle habits to address the health-risk behaviors and to generate strategies to mitigate NCDs [13,14].

The food frequency questionnaire (FFQ) is a widely used instrument in epidemiological studies to evaluate the typical or usual diet of people and it is one of the most used approaches in nutritional research [15]. In larger studies, they are relatively simple and less expensive to administer than other methods such as the dietary history and the 24 h dietary recall (24HR) [16]. Some FFQs developed in Argentinian cities, such as for residents of Córdoba, Buenos Aires, and Rosario, have been tested for validity and reproducibility [17,18,19]. However, none of these dietary assessment tools are electronic and national in scope. Because the FFQ may be prepared for a specific context, it is essential to document the validity and reproducibility of any new FFQ developed for specific population groups [18,20].

In recent years, the number of studies that collect data using online/electronic tools has increased. Technology has afforded the design of data collection tools including dietary assessment instruments in epidemiological studies. The use of digital instruments has several advantages including reduced time and cost spent on fieldwork, fewer potential interviewer biases, automatic collection and download of large data sets, easy data cleaning, real-time data coding, and automated intake calculations [21,22,23,24]. These advantages greatly optimize the work of researchers [25]. While traditional dietary assessment methods can be supplemented by food photo atlases to aid portion size recognition and estimation [26,27], online dietary assessment methods can also be designed to incorporate photographs of food and utensils for portion sizes. Carlsen et al. (2021) indicate that using images with portion sizes from digital media can facilitate the cognitive task of correctly choosing the portions consumed [28].

Online dietary assessment tools can be adapted to different geographic populations and can be accessed remotely, allowing better access and exploration compared to traditional methods [29]. The use of online dietary assessment methods can be more appreciated in Argentina where different regions of the country are geographically distant and a large proportion of the population are mobile-phone users [30].

We developed a semiquantitative electronic FFQ (eFFQ) designed for use in future epidemiological studies that are meant for tailor-made health interventions. This study aimed to validate the ability of this eFFQ to estimate the intake of nutrients and bioactive compounds among adults from the six main regions of Argentina using multiple 24HR as the standard.

2. Materials and Methods

2.1. Study Population

Adults 21 years and older and residing for at least 3 years in the same region of Argentina were included in this study. Participant recruitment was carried out from the 6 main regions of Argentina: Northeast (the states of Misiones, Chaco, and Formosa); Northwest (the states of Salta and Tucumán); Midwest (the states of San Luis and Mendoza); Central (the states of Entre Ríos, Santa Fe, and Córdoba); the Buenos Aires region; and Southern (the states of Rio Negro, Neuquén, and Chubut). Potential participants were contacted via telephone or in person and invited to the study through Adventist institutions such as schools, churches, hospitals, and healthcare clinics.

Participants who did not fully complete the questionnaires, women with current or recent pregnancies, and those who had changed their usual diet for medical or other reasons during the past year were not included in the study. Also, participants who reported a total energy intake outside the range of 500–4000 kcal were excluded from the analysis [31]. The final sample of participants for the validation and reproducibility study was 163 subjects.

The study was evaluated and approved by the Research and Ethics Committee of the Adventist University of River Plate School of Medicine (resolution #1.7-8/2016), affiliated with the National Registry of Health Research (#237), Ministry of Health, Argentina. Participants were included in the study by invitation and acceptance of the terms of the informed consent. All procedures associated with this study were conducted following the international ethical standards proposed by the Helsinki protocol for human research.

2.2. Study Design

This study follows the guidelines and recommendations for reporting nutritional epidemiology and dietary assessment research described in the statement Strengthening the Reporting of Observational Studies in Epidemiology—Nutritional Epidemiology (STROBE-nut) [32]. The study was cross-sectional with multiple 24HRs and repeated administration of the developed eFFQ, and Figure 1 shows a diagram of the data collection process. Four 24HRs were collected for approximately a year, two during the spring to summer months, and two during the autumn to winter months. The eFFQ was self-administered at two time points which were about 12 months apart. The eFFQ was first administered at around the time the first and second 24HRs were collected and was administered again after participants completed the third and fourth 24HRs. Socio-demographic data (i.e., age, sex, occupation, education, and marital status), and self-reported anthropometric measurements (weight, height, and waist circumference) were also obtained.

2.3. Reference Method: Multiple 24HR

The 24HR was used as a reference method to determine the validity of the eFFQ. The 24HR interviews were carried out when the dietary intake of the previous day reflected usual or habitual intake. If this was not the case, then the interview was postponed until later, considering that the data would reflect an atypical day in the consumption of food and beverages. The collection of 24HR was carried out by researchers, trained dietitians, and advanced dietitian students with the aid of the MAR24, an automated tool for 24HR previously developed by our group [33]. MAR24 uses the United States Department of Agriculture (USDA) 5-step multiple-pass method for dietary recall [34], consisting of a quick list of food items consumed; questions about commonly forgotten foods; detailed reports about the time and place where the food was consumed; food descriptions and amounts consumed estimated by a photo album of kitchen utensils; and a final review.

A total of four 24HRs were collected per person, two representing the food intake over the weekend—a Saturday and a Sunday—and two others on weekdays. For quality control, all the 24HR interviews were recorded using the Express Talk tool (NCH Software, Australia, Version 4.36). A random selection of 24HR audio records was regularly assessed by the investigators (S.L. or R.G.) to ensure data were properly collected and uploaded into the MAR24 database. The MAR24 tool was used to calculate the macronutrient and micronutrient composition of intakes reported in the 24HR as described elsewhere [33].

2.4. Development of Electronic Food Frequency Questionnaire

A semiquantitative FFQ was developed in an electronic format and was designed to assess the habitual intake of foods that reflect regional diversity in Argentina. For the initial selection of the eFFQ food list, the basic structure of food items from a previously validated FFQ for Argentinian adults was considered [19]. Our group carried out a survey to further determine what foods to add to the food list. This survey consisted of 2 interviews with 60 participants from the 6 Argentinian regions, from whom a total of 120 interviewer-administered 24HRs were collected. In considering which foods to add to the eFFQ’s food list, foods that were sporadically mentioned in the 24HR, i.e., accounting for only 5% or less of the total foods mentioned, were excluded [35]. The data from these 24HRs served as the basis for selecting regional foods and in describing portion sizes using familiar or commonly used kitchen/serving utensils. Participants in this survey were not included in the final sample of 163 participants for the eFFQ validation.

The semiquantitative eFFQ consists of a 76-item food list, a frequency of intake section (every day, every week, every month, or no consumption), a portion size section, and the number of servings consumed (0–14 servings) section. Foods are grouped under the following 10 categories: vegetables; fruits; bread, grains, and starches; legumes and derivatives; dairy products and derivatives; eggs, meats, fish, and derivatives; fats, oils, and nuts; sugar and sweets; condiments, dressings, and spices; and beverages and infusions. Also, where appropriate, items include options for foods with 100% plant food ingredients. The eFFQ asks respondents to indicate how often they consume the foods, the usual portion size eaten, and the number of servings every time they ate that food during the past 12 months (see Figure S1). The reported consumption of each food item was converted to grams per day for evaluation.

2.5. Nutrient Intake Determination

Dietary intake reports on both the eFFQ and 24HR were coded using the MAR24 tool [33]. The nutrient profile of food items was derived from the USDA Food Composition Databases [36], except for 5 particular foods (Amargo serrano, Amargo serrano diet, Bizcochos de grasa, Chipá, and pan criollo) for which the composition was obtained from the Argentinian Food Registry System (SARA) [37]. Each food item selected from the USDA database [36] had its nutrient composition (water, energy, fiber, macro, and micronutrients) and ingredient/s (when applicable) compared and checked for similarity with data from the following available Argentinian databases: SARA, ArgenFood Food Composition Table, Nutrinfo database, or the Central American Food Composition Table of the Institute of Nutrition of Central America and Panama (INCAP) [38]. For cooked foods, all cooking methods were searched for each food and those used in Argentina were selected. As there is no information on cooked foods from local databases for comparison, cooked versions of the raw foods from the same USDA data source (Legacy, Survey, or Foundation) were selected [36].

2.6. Statistical Analysis

Descriptive statistics were performed to report the means and standard deviations for energy and nutrient intake variables that were normally distributed, and medians and interquartile ranges for variables with non-normal distributions. These were conducted for the first eFFQ administration (eFFQ1), second eFFQ administration (eFFQ2), and the 24HR. Energy adjustment was performed using the residual method proposed by Willett et al. [39].

2.6.1. Validation of the eFFQ

The ability of our eFFQ to discriminate among individuals and measure true dietary intake was evaluated by comparing individual estimates of nutrient intake based on the questionnaire with those measured by a more accurate method, considered as a gold standard (relative validity). The primary alternative to the use of diet records as a standard for evaluating an FFQ is the collection of the 24HR [17,18,19,20,29]. Although this method relies on memory and the perception of serving sizes, it is considered an option in situations where subjects are illiterate [15] or less highly motivated, such as individuals not experienced in participating in scientific studies. Hence, to determine the relative validity of the eFFQ, the eFFQ2—which was administered after the four 24HRs were collected—was compared with the reference method, the 24HR. The differences in the reported intakes between the methods were calculated using either the ^t^-test or the Wilcoxon signed-rank test, depending on the distribution of the variables. The Spearman’s and Pearson’s correlation coefficients were calculated for crude, energy-adjusted, and de-attenuated data. The de-attenuation process of the correlations to correct for within-person variation in multiple 24HRs was performed with the method using the PC-SIDE software (Version 1.0, software for intake distribution estimation for the Windows operating system) which was developed by researchers in the statistics department of Iowa State University [40]. The ranking ability of the eFFQ was determined with cross-classification [41]. Cross-classification was accomplished by ranking the intake values for both the mean of the four 24HRs and eFFQ into quintiles and then cross-tabulating the quintiles to determine the agreement and gross misclassification in reported intakes between the measures. The proportions of respondents with exact agreement (EA) on the quintile classifications of their reported intake in both methods, those whose intakes deviated from exact agreement by one (EA ± 1) or two (EA ± 2) quintiles, and those whose intakes were on opposite quintiles (grossly misclassified) were computed. We calculated the weighted kappa statistic for the ranking. Finally, Bland–Altman plots [42] were graphed for macronutrients to further determine the agreement between the two methods and evaluate outliers or possible biases.

2.6.2. Reproducibility of the eFFQ

To determine the reproducibility of the eFFQ, we calculated the intraclass correlations (ICC) for unadjusted/crude and energy-adjusted values between eFFQ1 and eFFQ2. Ranking by quintiles and cross-classification were performed to determine the level of agreement and gross misclassification between the two eFFQ administrations. The weighted kappa statistic was also calculated. All statistical values were considered significant when p < 0.05.

3. Results

3.1. Demographic Profile of Participants

Among 186 subjects initially enrolled in the study, 23 either did not fully complete the questionnaires, had changed their usual diet for medical or other reasons during the past year, or reported an energy intake outside the range of 500–4000 kcal. These were excluded from the analysis [31], and the analytical data were based on 163 participants.

Table 1 shows the demographic characteristics of the study population. Approximately 85% of the participants were females, and about 28% belonged to the Buenos Aires region. About 38% had a tertiary level of education and about 88% were employees. The mean age was 41.8 ± 10.4 years and the mean body mass index (BMI) was 26.7 ± 4.9 kg/m^2^.

3.2. Validity of the eFFQ Nutrient and Bioactive Compound Estimates

Data from the eFFQ2 were used to determine the relative validity of the eFFQ since this covers the 24HR data collection period. Compared to the 24HR, both the crude and energy-adjusted nutrient intake estimates of the eFFQ were higher for most of the nutrients, except for total sugar, lycopene, and caffeine (see Table 2). Comparison tests indicated that the intakes of energy and most nutrients, except for cholesterol, β cryptoxanthin, ethanol, and theobromine, were significantly different. Table 2 shows the crude, energy-adjusted, and de-attenuated correlation coefficients between the eFFQ and the 24HR, with crude correlation coefficients ranging from 0.15 for phosphorous and niacin to 0.57 for caffeine; most were within an acceptable range. Energy-adjustment slightly increased the correlations, ranging from 0.15 for vitamin E to 0.58 for caffeine, while the de-attenuated correlations ranged from 0.16 for niacin to 0.57 for animal protein.

The levels of agreement are also shown in Table 2. The proportions of agreement within one quintile adjacent to exact agreement (EA ± 1) ranged from 52.2% and around 74% for linoleic acid (LA) and caffeine, respectively. The levels of agreement for those within two adjacent quintiles of exact agreement (EA ± 2) ranged from 77.9% for ethanol to 93.3% for animal protein and magnesium. Extreme misclassifications into the opposite quintiles were less than 7% for energy and all nutrients. The weighted kappa values described conformity ranging from 0.04 (LA) to 0.38 (caffeine) between eFFQ2 and the mean of 24HR.

The Bland–Altman plots (see Figure 2) show that macronutrient intakes were overestimated by the eFFQ.

3.3. Reproducibility of the eFFQ

Table 3 shows the crude and energy-adjusted intraclass correlation coefficient (ICC) for eFFQ1 and eFFQ2. The crude ICC ranged from 0.29 to 0.85 for lycopene and ethyl alcohol, respectively. However, the correlation coefficients differed slightly after adjusting for energy. The energy-adjusted ICC ranged from 0.23 for lycopene to 0.89 for magnesium. The ICC shows moderate and good correlations for most nutrients except for alpha and beta-carotene, lycopene, lutein zeaxanthin, vitamins A, C, and E for unadjusted values, and vitamin A, alpha-carotene, beta-carotene, and lycopene for energy-adjusted ICC values. In cross-classification analysis, the ranges of the agreement levels for the EA ± 1 quintile were from 70.6% for water, choline, and vitamin E to 87.7% for caffeine. Extreme misclassification into the opposite quintiles was less than 5% for energy and all nutrients. The weighted kappa values described fair and moderate conformity, ranging from 0.21 (vitamin B12) to 0.62 (eicosapentaenoic acid) between the two administrations of the eFFQ.

4. Discussion

This work investigated the validity and reproducibility of an electronic food frequency questionnaire (eFFQ) in estimating the intake of nutrients and bioactive compounds of an adult population from the six main regions of Argentina. The results show that this eFFQ is relatively valid in ranking adults according to their nutrient intake and has an acceptable reproducibility. However, it slightly overestimates the intake of most nutrients.

In Argentina, only a few published studies have evaluated the reproducibility and validity of an FFQ used in the adult population. Navarro et al. conducted a study with 66 individuals 23–80 years of age to validate a specific FFQ for adults with cancer in Córdoba, Argentina [29]. In 2012, Dehghan et al. validated an FFQ with 116 women and 40 men residing in rural and urban areas of Rosario [43]. Elorriaga et al. evaluated the dietary intake of 147 individuals 21–74 years of age from the southern cone of South America (Argentina, Uruguay, and Chile) [18]. Zapata and colleagues validated an FFQ with a sample of 88 adults from Rosario, Argentina [19]. Perovic et al. validated an FFQ to assess the intake of lipids and phytochemicals on a sample of 45 individuals of both sexes, aged between 20 and 72 years [17]. Finally, Olmedo et al. validated a questionnaire to estimate the intake of free sugars and ultra-processed foods of 77 residents of La Plata city and its surroundings [44]. These studies were conducted in specific areas in Argentina, but none considered sampling from the different regions of the country. To the best of our knowledge, this is the first study that validated and tested the reproducibility of an FFQ that was developed in electronic format to estimate and rank the dietary intake of an adult population from the six main regions of Argentina. A total of 163 subjects from different regions of the country participated in this study. Sample size in FFQ validations is variable but, in most studies, it has been less than 200 individuals [18,19,45,46,47,48,49]. Bland–Altman graphs were used to evaluate the absolute concordance between FFQ and 24HR, which recommends using a sample of at least 50 and preferably 100 participants [43].

The number of food items in an FFQ tends to vary widely, ranging from 5 to 350, with an average of 79 questions [50], which is similar to our eFFQ. There is a rapidly diminishing marginal gain in the information obtained from increasingly detailed questionnaires [51,52,53]. Therefore, a long list of food items is not recommended because of the tendency for the overestimation of reported intake in FFQ [54]. The FFQs validated in Argentinian adult populations vary from 66 to 257 food items [17,19]. This present eFFQ has a total of 76 items grouped into 10 food groups, which cover the main foods consumed in Argentina. It is important to highlight that, despite regional variations, Argentine food intake presents a certain monotony; thus, a comprehensive FFQ such as ours may sufficiently cover the foods currently eaten in the different regions of the country.

To assess the reproducibility of the questionnaire, we compared the eFFQ data on two time points. Time intervals between the repeated administration of FFQs range from a few days [55,56,57] to more than one year [58]. For better results, some authors recommend a period of 4 to 6 months between the repeated administration of an FFQ [51,59]. Closer repeated administrations of FFQs allow high correlations, probably because subjects may still remember their previous responses. On the other hand, a long interval may result in weak correlations for variations in responses due to actual changes and disturbances in dietary intake during the time between the two administrations [58,60]. In our study, our repeat administration of the eFFQ was approximately 12 months apart from the first administration which could explain the lower correlation coefficients for reproducibility. According to Willet and Lenart [61], this limitation should not be considered serious from the point of view of measurement error. The mean intakes of most nutrients from our eFFQ1 were higher compared to the data obtained from eFFQ2. This may be due to a training effect, where intake reporting has improved, or due to a real change in dietary intake.

The crude ICCs for the reproducibility of our eFFQ ranged between 0.29 and 0.85, and these results are similar to other studies [60,62]. After adjusting for energy, the ICC for most of the nutrient estimates changed, where most of the coefficients increased while some decreased, with results ranging from 0.23 to 0.89. The association between nutrient intake and energy intake may cause the correlation coefficients to increase after energy adjustment. On the other hand, the decreased correlation coefficients after energy adjustment could be due to systematic overestimation or underestimation [63]. In the cross-classification analysis, the ranges of concordance levels for the same category or in an adjacent category between eFFQ1 and eFFQ2 ranged between 70.6% and 87.7%, similar to another study [60]. The extreme misclassification in the opposite quintiles was <4.9% for energy and all nutrients.

In testing the validity of the eFFQ, crude correlations between eFFQ2 and the mean of multiple 24HR ranged from 0.15 to 0.57, which increased slightly after energy adjustment (0.15–0.58). Our crude correlations were similar to a study among urban participants by Dehghan et al. [43] with coefficients of 0.20–0.47, while the correlations after energy adjustment were similar to those of Ye et al. [64] with energy-adjusted correlations of 0.19 to 0.58. In our cross-classification analyses, the agreements within one quintile between the eFFQ and 24HR ranged from approximately 63% to 72% for macronutrients and approximately 52% to 73% for micronutrients. These results are very similar to another study, where values ranged between 50% and 75% for macronutrients and 48% and 70% for micronutrients [65]. Extreme misclassification was less than 7% for energy and all nutrients; similar findings were observed in a study conducted in Indonesia [66]. On the other hand, the weighted kappa values between eFFQ2 and the mean of 24HR showed a compliance that ranged from 0.04 to 0.38, results similar to the study carried out by Zhuang et al. [60] whose values ranged between 0.12 and 0.40.

Argentina is a country whose diet is largely focused on the consumption of meat, which is the main source of protein. National statistics show that 5 out of 10 individuals consume red meat, poultry, and/or eggs at least once a day [67]. On the other hand, in 2021, Argentina was the country with the highest consumption of beef per inhabitant on the planet, while it ranks ninth in the consumption of chicken meat [68]. Our eFFQ is able to aptly rank individuals on their intake of animal protein as indicated by its moderately strong de-attenuated correlation (r = 0.57) with the reference standard.

The USDA database, which is widely comprehensive in terms of micro- and macronutrient information, was used with the MAR24 tool, an available resource developed for 24HR and nutrient intake estimation in Argentina. The lack of comprehensive nutrient information from Argentine food composition tables is a limitation, but having the USDA database provides security since its data are highly reliable [36]. In many resource-deficient areas of the world where food composition databases are incomplete, outdated, or lacking, the use and sharing of existing databases has become standard practice [69]. Energy intake estimates with our eFFQ could have been validated against doubly labeled water, but this was not feasible in our context. This is often a limitation in many validation studies with limited resources. One limitation of the study is that the study participants were not randomly chosen from the general population but were volunteers who met the study inclusion criteria. However, in general, validation studies are carried out among voluntary participants [70]. Another limitation of the study is the low percentage of male participants (15.3%).

The lack of a more objective reference method, such as the use of biomarkers to triangulate the validation of the eFFQ, is a limitation. However, about 75% of the methods used for the validation of FFQs employ as a reference method 24HR and dietary records [50]. We chose unannounced non-consecutive 24HR as the reference method to validate our eFFQ since this will prevent respondents from meal planning beforehand to report an intake that sounds desirable to the interviewer. Although the 24HR has correlated errors with FFQ due to reliance on memory, recalling the immediate past such as in 24HR employs episodic memory whereas generic memory is what is used in FFQs when recalling the usual intake of the past [16,71]. Multi-day weighed food recording would have eliminated the errors associated with the recall, but participants may (un)consciously alter their food intake to ease the burden associated with the tedious process of food recording [51]. However, it should be noted that food intake cannot be estimated without error, and the nature and magnitude of the error are related not only to the method selected to collect the data but also to the subjects studied [72].

5. Conclusions

This is the first study that validated an instrument developed for the adult population living in different regions of Argentina. As an online tool, the eFFQ allows ease in the administration of the instrument, facilitating its use in population studies. If self-administered, respondents may feel more at ease in filling out the questionnaire in the convenience of their own home or time with a greater assurance of confidentiality. The results of this study show that this eFFQ is relatively valid in ranking adults according to their nutrient intake and has an acceptable reproducibility, yet it slightly overestimates the intake of most nutrients.