Inglês (pdf)
Artigo em XML
Referências do artigo
Tradução automática
Permalink
Sijtsma et al. (2024a) provide a thorough discuspsychometric practice. This paper, and its main argusion to sustain that the sum score on a psychological ments, were critiqued by McNeish (2024) and Mislevy test is, and should continue to be, a central tool in (2024), which were also followed by a rejoinder (Sijtsma et al.; 2024b). In spite of the possible conclusions of this particular interaction, the discussion about the usefulness and overall psychometric robustness of sum scores (also known as Likert scoring, test scores, and others) is not new (e.g., DiStefano et al., 2009; Gorsuch, 1983; Hair et al., 2006). One important aspect to consider is that Likert scoring, factor analysis, Item Response Theory (IRT) and other psychometric models are all implementations of the latent variable theory (LVT; Franco et al., 2022; McDonald 1999). As a consequence, this means that all these models are, in their core, the same, providing only different means to estimate true scores (Wiberg et al., 2019).
Therefore, while Likert scoring provides an efficient monotonic estimate of the true scores (in ideal scenarios; Coenders et al., 1995), methods derived from factor analysis and IRT take the structure of the data generating process into account. In principle, these “refined” methods should then provide better estimates of theta, even when the data generating process reflects ideal conditions for Likert scoring to be an efficient estimate of the true scores. The present study aims at empirically comparing the effectiveness, defined in terms of the correlation with the true score, error and bias measures, of the estimates of Likert, Thurstonian and EAP scores. To achieve this, computational simulation of two-parameter logistic model (2PLM) in a number of different conditions were used to estimate the different scores.
This study is relevant as Likert scores are far less computationally extensive than other methods, but not useful if its estimates are much biased or error prone. Additionally, a simulation study allows to evaluate properties that are not readily understandable from non-formal treatments of the relations between different forms of estimating the true scores. The rest of the paper is structured as follows. First the general psychometrical framework, which include the different scoring methods, and the methods themselves (Likert, Thurstonian, and EAP scores) are briefly described, followed by a method section which describes the setup of the simulation study. Next, the results from the simulations are presented and the paper ends with a discussion with some concluding remarks.
General Psychometrical Framework
There are three major theories in psychometrics aiming at identifying and measuring response patterns in tests (Franco et al., 2022; McDonald, 1999): Classical Test Theory (CTT); Common Factor Theory (CFT); and Item Response Theory (IRT). Despite usually defined as different theories (e.g., Borsboom, 2005), all of them can be understood as different applications of the same general Latent Variable Theory (LVT). Latent variables are not directly observed - sometimes not even observable -, and inferred from other variables that are directly measured or observed. This inferential step from observed data, after applying different functional forms, is what distinguishes them.
Many textbooks and papers discuss different procedures for estimating the magnitude of latent variables (e.g., van der Linden & Hambleton, 2013). Differences in the results after applying these different methods rely, mostly, in the assumptions relating to the formulas and estimation procedures used. For instance, Likert scores stem from CTT (Crocker & Algina, 1986), which aims to describe psychological outcomes given the general characteristics of a test. CTT relies on the assumptions that each respondent gets an observed score which is composed by the true score on the test and an error term:
where Xi stands for the observed test score of respondent i, Ti for the true, latent, score of respondent i, and ɛi for the random error of the measurement of respondent i’s observed score.
For the CFT, on the other hand, items on a test are assumed to be sampled from a population of items that relates to the true score (Borsboom, 2005). Each item then represents, to a different magnitude, the true score. Therefore, the procedures in this theory are mainly aimed at discovering this magnitude, with the general representation of
where Xij stands for the response of respondent i to the jth item, λj for the factor loadings of item j, Ti for the true latent score of respondent i, and ɛij for the random error of the measurement of respondent i’s observed score in the jth item. An important difference between CFT and CTT is that the factor loadings make CFT a more testable model (Bamber & van Santen, 2000), meaning that factor loadings are usually free parameters. On the other hand, scoring procedures stemming from CTT are usually parameter free and, therefore, not testable.
For IRT, it is accepted that the measurement level of the observed score is ordinal or even nominal (McDonald, 1999). Therefore, models based on IRT, usually called IRMs, assume that mixture effects of items and respondents’ latent characteristics generates the probabilities of the response categories of the items. A general representation of this theory is
where Xij stands for the response of respondent i to the jth item, the g() function is some probability mass functions - usually binomial or categorical distributions (van der Linden & Hambleton, 2013) - , Tij is the probability of respondent i giving a certain response to the jth item, the f() function is some link function - usually variations of the logistic function or the cumulative normal function; (van der Linden & Hambleton, 2013) - , θi is the true latent score of respondent i, and bj is the item latent score, usually called the difficulty parameter, of the jth item. The error term ɛij is suppressed in this representation given that it is a natural consequence of the stochastic nature of the g() function.
These differences in assumptions about how the true score is related to the observed scores causes differences on how to estimate the true score depending on what theory is held as true (Borsboom, 2005; Franco et al., 2022; McDonald, 1999). If Equation 1 is held as true, the true score is typically estimated by Likert scoring. The true score in Equation 2 is usually estimated by some combination of algebraic and optimization methods applied to models of confirmatory or exploratory factor analysis. The true score in Equation 3 is usually estimated by conditioning on the response patterns on estimated item parameters of some IRM and by assuming a specific distribution for the true scores.
Classical Test Theory and Likert Scores
Following the representation of CTT in Equation 1, a common operationalization is achieved, for instance, whenever deviations from the true score are assumed to follow a normal distribution with 0 mean (Borsboom, 2005). This is a common assumption for many statistical models (Tabachnick & Fidell, 2007) and simply means that different measures of the same thing can result in different values, due to random noise. This means, considering a CTT model, that observed differences in item scores are deemed to be exclusively due to noise in the responding procedure. It is also a consequence from this assumption, and the formal representation of the theory, that observed scores are expected to be correlated. High average correlation between items in a test means lesser magnitude of random errors, which is measured by reliability.
Reliability is the overall consistency of a measure. One of the most well-known techniques for estimating overall consistency is Cronbach’s alpha (Dunn et al., 2014). This method follows from the assumption that items equally represent the same true score and, therefore, higher average correlation between items means lower error variance. Following the instantiation of a highly reliable test, which thus serves as a test of the overall magnitude of error, respondents true scores can be estimated by Likert scores (Norman, 2010). Likert scoring can be calculated as
where Li represents the Likert score for respondent i, k is the quantity of items in the test and Xij is the observed score given by the ith respondent to the jth item. The calculation of Likert scores is sound from a CTT perspective, but problematic from a statistical perspective. Its assumptions of equally representing the true score and of being measured on an interval level of measurement have long been criticized by psychometricians, whom found several examples when not all items equally represent what is meant to be measured (Borsboom, 2005; Crocker & Algina, 1986). This started the practice of factor analysis (Thompson, 2004) and IRM (van der Linden & Hambleton, 2013).
Common Factor Theory and Thurstonian Factor Scores
Factor analysis (Thompson, 2004) is typically used to reduce the dimensionality in a dataset and describe the variability among the observed variables. Assuming that respondents’ observable scores x are due to a linear combination of their latent trait and a specific skill to each item, one can simply recover the true score f using
where
and Λ is the factor loadings for each kth item, fn is the factor score of the nth respondent, and u is the unique property of each kth item. Thurstone (1935) was the first to propose the use of this model, where Λ is a set of free parameters, to estimate the true scores of respondents.
For the proper estimation of Thurstonian scores, one must first choose an extraction (or factoring) procedure (Tabachnick & Fidell, 2007). The minimum residual procedure, which minimizes the off-diagonal residuals of the correlation matrix, is one of the most effective extraction procedures (Harman & Jones, 1966) and can, therefore, be used as a solver for Equation 5. After estimating factor loadings, the Thurstonian procedure can be used to obtain standardized true score estimates (Grice, 2001):
where W is the matrix of factor score coefficients, Rjj is the matrix of items’ correlations, Sj is the vector of structure coefficients and Zij is the matrix of standardized observed scores. However, the use of traditional factor analysis models in psychology has been criticized as the general model assumes observed variables to be measured on a interval level, when they are, at best, measured on an ordinal level (Wirth & Edwards, 2007). IRT has been proposed to overcome this limita tion (Lord, 1980).
Item Response Theory and EAP scores
IRT (Lord, 1980) is an item centered theory which relies on the assumptions that each observed response is conditioned on a single respondent’s ability θ, that the items have local independence (as in the CFT framework), and that responses can be modelled with an IRM. There are several IRMs, but the Rasch and the family of logistic models are the most famous models (van der Linden & Hambleton, 2013). If we assume 2PLM, then we have that if a respondent i has ability (or true score) θi, then the probability of answering an item correctly can be modelled by
where bj is the item difficulty and aj is the item discrimination of item j. This model is different from the ones used for CTT or CFT given that, when using IRMs, latent score f is thought of a mixture of the θi and bj parameters.
After obtaining item difficulty and item discrimination estimates, Bayesian statistical principles of parameter updating can be used to estimate the respondents true scores using the EAP estimator (Bock & Aitkin, 1981). Let f(θ)dθ be a prior distribution, then we can compute the EAP score estimates by
which simply computes the expected value (θEAP) of the posterior probability, given a prior distribution of the true score, conditioned on respondents’ observed scores, X, and a set of items parameters, π. For the 2PLM this means that aj, bj ∈ π. Usually, this prior distribution is defined as a normal distribution with mean 0 and standard deviation of 1 (Bock & Aitkin, 1981).
What are the Expected Differences Between These Methods?
It should be noted that, in some cases, these procedures will result in equivalent estimates for the true scores. For instance, Likert scoring is a valid procedure for estimating the magnitude of respondents’ true scores when the data generating process is equivalent to a parallel factor analytical model (Coenders et al., 1995). In parallel factor analytical models, all items are considered to be equally influenced by the theorized latent variable (i.e., all factor loadings are equal). This is also equivalent to an IRT model where all the discriminations are equal, such as the Rasch model or the one-parameter logistic model (Alphen et al., 1994).
Therefore, despite of the fact that many researchers defend that respondents’ estimates stemmed from FA (i.e., factor scores; Grice 2001) and IRT (e.g., expected a posteriori, EAP; Bock & Aitkin 1981) are superior to the ones stemmed from CTT (Likert scores; e.g., DiStefano et al., 2009), there are cases where they are quantitatively the same. This endless discussion can be attributed to several reasons, with at least two of them worth to discuss. First, the fact that, in CTT, reliability estimates were, and sometimes still are, improperly used for dimensionality analysis (Flake & Fried, 2020; Schmitt, 1996). Reliability estimates, such as Cronbach’s alpha (Cronbach, 1951), are sometimes used to justify the inference of measurement by means of Likert scores to the particular trait. The second reason is that IRT and FA models have free parameters while CTT is a parameterfree model. Therefore, IRT and FA should be able to provide a better fit to the observed data.
Additionally, some previous studies have found that, overall, both items and respondents’ estimates are highly correlated, despite from which model they were estimated from (Lawson, 1991). For instance, Fan (1998) showed that items and respondents’ estimates derived from IRT and CTT are quite correlated, when assumptions regarding CTT hold. The author also showed that the degree of invariance of item statistics across samples, usually considered as the theoretical superiority IRT models, only differed for some particular cases of discriminations’ estimates. These results are probably because of factor scores indeterminacy (Grice, 1991): an infinite number of sets of latent variables score can be created for the same analysis.
Some other authors (e.g., Franco et al., 2023; Ramsay & Wiberg, 2017; Ramsay et al., 2020; Wallmark et al., 2024) took this matter further and demonstrated that any strictly monotonic transformation can be used as a function for respondents scores’ estimates. Therefore, for unidimensional scales, any estimates for respondents’ true scores which are monotonically related to factor or EAP scores, such as the Likert scores, should provide good approximations of the true scores. However, they also showed that extreme scores are usually more biased for Likert scores than for other estimates of factor scores. Therefore, despite of the fact that previous research has shown that Likert scores are reliable estimates of the true score (Henson et al., 2007; Hinz et al., 2012), there are no study which compares the effectiveness of Likert, Thurstonian, and EAP scoring methods in recovering the true score using simulations as it is done here.
In sum, some psychometricians (e.g., Borsboom, 2005) consider CTT and CFT to be unsatisfying because these theories do not adequately represent the attribute to be measured in the model. This limitation should be surpassed by IRT, which proposes IRMs of the data-generating processes, explicitly stating the relation between observed and latent variables. As a consequence, it is expected that scoring methods esteemed from IRMs will be more effective than other scoring procedure that does not adequately represents the attribute to be measured and the data to be described, such as Likert and Thurstonian scores. This should be true especially when discriminations are very different for each item.
Method
Study Design and Data Simulation
A Monte Carlo simulation study was designed to test the effectiveness of the three discussed scoring methods. To those interested in further examining the results or conducting further simulations, our codes are available from the corresponding author. The simulation used the 2PLM as the data-generating process. The θ parameter was drawn from an evenly spaced vector in the closed interval [-3,3], depending on the sample size, which was drawn from the set {60,100,500}. The difficulty parameter was distributed as a truncated normal distribution with three possible means, {-3,0,3}, two possible standard deviations, {.1,1}, with lower and upper bounds equal to the minimum and maximum of the means’ set, respectively. Similarly, the discrimination parameter was also distributed with three possible means, {.5,1,2}, two possible standard deviations, {.1,1}, with lower and upper bounds equal to the minimum and maximum of the means set, respectively. The size of the drawn for both difficulty and discrimination parameters were dependent on the “test size” (i.e., number of items) parameter, which was drawn from the set {5,10,20}. The last parameter was scale size, fixed in dichotomous or 5-point scale response patterns.
These specifications resulted in 648 crossed conditions, which were iterated 100 times. These conditions were selected to replicate common practices in psychological literature (e.g., Colman et al., 1997; Marszalek et al., 2011; Morgado et al., 2018) and also to test the effects of weighting (i.e., varying discriminations) and extremeness (i.e., overall test difficulty, high or low) on the scoring procedures. Also, the θ parameter was drawn from a linearly spaced vector to avoid the necessity of using any equating procedure given distributional differences between difficulties and aptitudes (González & Wiberg, 2017). The 2PLM was used as the link function for getting the probabilities of respondents answering each item correctly. Finally, simulated dichotomous scores were generated using a binomial distribution and simulated polytomous scores were generated using a multinomial distribution, following recommendations from Agresti (2003). These steps make the 2PLM always true in the sample and, therefore, scores estimated using IRT procedures should be the most effective (Borsboom, 2005).
Data Analysis
Effectiveness of each method was assessed by their capacity to recover the true score - by means of Pearson’s correlation between the observed scores and the known true score - and the use of root mean squared errors (RMSE) and bias estimates. It is expected that more effective methods will correlate more with the true score (i.e. values closer to 1) and also present lower RMSE and bias (i.e., values closer to 0). Using these measures of effectiveness, the levels of the seven conditions - sample size, number of items, number of scale levels, discrimination mean and variance and difficulty mean and variance - were compared so an optimal crossed condition could be used for the following tests. We define the optimal condition as the one that makes the best estimates of the true parameters, minimizing the costs (e.g., smaller sample size) for doing so.
Using only the optimal crossed condition, the methods were compared in their capacity to properly return estimates of mean differences in independent sample t tests, linear regression parameters and correlation between latent variables. Their effectiveness to recover standard mean differences between two independent latent variables was assessed by the average estimates of Cohen’s d (Gignac & Szodorai, 2016). Their effectiveness to recover slopes and intercepts of linear regression parameters were tested by the average of estimated parameters of linear regressions, without using corrections for observed reliabilities (Bacon 2004). Finally, their effectiveness to recover correlation between two latent variables was examined through average Pearson’s correlation on the estimated scores. All the analyses were conducted with R version 4.4.1 (R Core Team, 2024) and packages psych version 2.4.6.26 (Revelle, 2014) and ltm version 1.2-0 (Rizopoulos, 2007).
Results
Table 1 presents the results for each condition. In general, EAP (with a standardized normal as the prior for the true scores) and Likert scores always present higher correlations with the true score than the Thurstonian Scores, but Likert scores always present the highest RMSE. From the overall mean of effectiveness measures, EAP score has the best correlation performance, followed with a difference of .001 by the Likert score and with a difference of .042 by the Thurstonian score. In terms of RMSE, Thurstonian score always performs better, with overall difference of .142 to EAP and of .583 to Likert scores.
Table 1 Mean correlation with the true score and RMSE given the testing conditions for each scoring method
| Correlation with the True Score | RMSE | |||||
|---|---|---|---|---|---|---|
| LS | TS | EAP | LS | TS | EAP | |
| Average performance | .757 | .716 | .758 | 1.460 | .735 | .877 |
| Sample Size 60 |
.751 | .705 | .750 | 1.460 | .823 | .925 |
| 100 | .751 | .713 | .752 | 1.462 | .785 | .912 |
| 500 | .769 | .731 | .772 | 1.457 | .713 | .862 |
| Number of items 5 |
.754 | .719 | .743 | 1.462 | .753 | .788 |
| 10 | .756 | .721 | .758 | 1.462 | .738 | .825 |
| 20 | .762 | .709 | .774 | 1.459 | .720 | .922 |
| Scale Levels 2 |
.685 | .650 | .690 | 1.400 | .778 | 1.026 |
| 5 | .829 | .783 | .826 | 1.394 | .671 | .673 |
| Difficulty M=-3; SD=.1 |
.594 | .569 | .613 | .878 | .660 | .987 |
| M=-3; SD=1 | .784 | .723 | .790 | .826 | .461 | .672 |
| M=0; SD=.1 | .865 | .848 | .845 | .909 | .329 | .519 |
| M=0; SD=1 | .909 | .868 | .884 | .968 | .234 | .380 |
| M=3; SD=.1 | .615 | .570 | .630 | 1.067 | .726 | .979 |
| M=3; SD=1 | .777 | .715 | .788 | .951 | .474 | .738 |
| Discrimination M=.5; SD=.1 |
.713 | .657 | .671 | 1.475 | .516 | .795 |
| M=.5; SD=1 | .769 | .726 | .766 | 1.433 | .705 | .805 |
| M=1; SD=.1 | .768 | .729 | .768 | 1.425 | .671 | .804 |
| M=1; SD=1 | .771 | .731 | .774 | 1.429 | .732 | .823 |
| M=2; SD=.1 | .752 | .724 | .785 | 1.444 | .898 | 1.024 |
| M=2; SD=1 | .771 | .731 | .786 | 1.430 | .775 | .859 |
Note. LS=Likert score; TS=Thurstonian Score; EAP=Estimated a Posteriori. The most effective method in each condition is in bold. The optimal conditions are in italic
These results can be completed with the bias in Figure 1. It is evident that all the methods have similar performances when true scores are close to 0, which can also be numerically verified by the RMSE in Table 1. When the scores are below average, all the methods underestimate the scores. When the scores are above average, all the methods overestimate the scores. Nevertheless, the Likert procedure is clearly the most bi ased, especially for more extreme scores.
The results shown in Table 1 and Figure 1 suggest that the optimal condition is the one with 500 cases, 10 items when using a 5 points scale, with items’ mean difficulty equal to 0, with standard deviation equal to 1, and with items’ mean discriminations equal to 2, with standard deviation equal to 1. From Table 2, it is evident that when comparing the same measure of two independent samples, Likert and Thurstonian scores will give on average the same result, which is close to the real difference for the latent variables. The EAP score will usually underestimate them. The mid part of Table 2 shows the performance of the three methods when examining correlations of two observed variables. It is evident that all methods have similar performance, with Likert scores as the most effective, followed by EAP and then by the Thurstonian scores. The lower part of Table 2 displays when estimating regression’s parameters, scoring methods will display diverging results. For the intercept, none of the methods gave a reliable estimate. For the slope, all the methods underestimated values above 1, with Likert scoring being the least biased one.
Table 2 Comparison of methods performance on standardized mean differences of two means, regression coefficients, and correlation estimates for the optimal crossed condition
| Parameters | Likert | Thurstonian | EAP |
|---|---|---|---|
| d | Mean differences between two independent samples | ||
| .1 | .094 | .091 | .090 |
| .5 | .472 | .462 | .457 |
| 1 | .959 | .950 | .923 |
| 1.5 | 1.451 | 1.443 | 1.392 |
| P | Correlation coefficients | ||
| .100 | .097 | .099 | .106 |
| .300 | .284 | .276 | .286 |
| .500 | .472 | .460 | .469 |
| .800 | .765 | .748 | .754 |
| βo and β1 | Intercepts and slopes | ||
| βo β1 | βo βo1 | βo βo1 | βo βo1 |
| 0 1 | .034 .988 | -.0009 .986 | .009 .983 |
| .5 1.5 | -.066 1.172 | .006 .926 | .067 .973 |
| 1 2 | .067 1.222 | .008 .876 | .190 .958 |
| 1.5 2.5 | .267 1.216 | .009 .823 | .238 .930 |
Note. d is the standard mean difference. ρ is the true correlation. β0 and β1 are the true intercept and slope, respectively
Discussion
Likert scoring is the most commonly used method for scoring respondents. Nevertheless, it is also the most criticized one, given that its assumptions are not deemed as reasonable for many psychometricians. Given that IRMs have a good reputation among psychometric theories, and that logistic models are the most famous IRMs, the present study used 2PLM as the basis for creating simulated data so true scores could be estimated. Overall, the results show that it makes almost no difference if the scoring method is based on CTT, CFT or IRT. The estimates given by the methods tested in the present study - Likert, Thurstonian, and EAP - have similar magnitudes of correlation with the simulated true scores. On the other hand, Thurstonian scores almost always presented the lowest RMSE values and EAP scores almost always presented the highest correlations with the true score. In general, Likert scores presented largest bias than the other methods.
On the other hand, if one was to consider the overall results, despite bias and solely from an efficiency in scoring respondents’ point of view, Likert scoring should always be preferred over Thurstonian or EAP scoring, given two main reasons. First, computing a Likert score is far less intensive and intuitive then computing Thurstonian or EAP scores. Thurstonian and EAP scores usually will be estimated using some optimization method, such as maximum likelihood estimation by Expectation-Maximization algorithm (EM; Bock & Aitkin, 1981), which means sometimes the algorithm may fail to converge. The second reason is that, despite bias, Likert score performs at least as well as the other methods in several conditions, especially in the optimal crossed condition. Nevertheless, scoring respondents in dynamical test sets, such as computerized adaptive and multistage testing (Magis et al., 2017), can benefit by other kind of estimates, such as maximum Fisher infor mation, that can be calculated only by applying IRMs.
An important practical consequence from the results follows for researchers who use Structural Equation Modeling (SEM; Byrne, 2016) to test regression models between latent variables. Most SEM’s softwares, for example AMOS (Arbuckle, 2012) and lavaan, (Rosseel, 2012) use Thurstonian scores (or some variation) to estimate the latent variables. Therefore, SEMs will probably give biased slope estimates, such as the ones found in the present study. Following from the result that all scores are very biased when estimating linear regression parameters, but quite efficient for estimating correlation parameters, non-parametric SEM (Pearl, 2009) or Likert scores’ associations modeled with Markovian Networks (Liu et al., 2017) may be used to reduce this bias.
From a theoretical point of view, the results found in Table 1 also partially refute the idea that IRT provides great practical advantages over CTT in recovering item and respondents estimates, as similarly found by Fan (1998). “Partially” because, as stated before, using a scoring method that is IRT based presented less bias. On the other hand, Likert scores will give at least as good estimates for regression parameters, mean differences and correlation as the other methods, even taking into account the amount of bias for scoring respondents with very high or very low true scores. It should then be remarked that the problem of measurement goes well beyond the simpler problem of making good estimates of a latent variable, given that experimentation, prior theorization and proper validation procedures can be more important than the use of some specific statistical model based on the general psychometrical framework (Sijtsma, 2012).
The results and conclusions from the present study are limited by the number and types of conditions tested. We only used a single process, based on the 2PLM, for the data generation process, as well as only unidimensional data generating processes. Different data generation processes, based on factor analysis, Item Factor Analysis, and other IRMs, may reveal different patterns that could change the overall results found in the present study. Also, there are a number of different IRT methods for estimating the respondents’ true scores, such as maximum a posteriori estimation (MAP; Bock & Aitkin, 1981) and scoring methods that directly use Bayesian modeling (Fox, 2010), which may have different effectiveness when compared to conventional implementations of Likert, Thurstonian, and EAP scores.
We emphasize that, although our results could seem like encouragement for abandoning latent variable methods, we echo Sijtsma et al. (2024b) in saying that the type of evidence and arguments we provided should not be understood like that. Rather, given the limitations of our simulations, in an ideal psychometric framework, the main implication of our study is that sum score should be used only after one has established the fit of an IRT or CFT model to the data. However, we also add to this idea that if the intended use involves selection processes (such as large-scale assessment) or thresholding (such as establishing clinical criteria), maybe Likert scoring should be avoided. The bias identified in the extreme values could add noise to the estimates that could result in injustices and unreliable estimates that could be very harmful in high-stake conditions (Franco et al., 2023).
