13:30 - 15:00
Wed-P14-Poster III-2
Wed-Poster III-2
Room: P14
Allowing for Indecisive Outcomes: A Fair Comparison of Equivalence Tests
Wed-P14-Poster III-201
Presented by: Frieder Göppert
Frieder Göppert, Sascha Meyen, Volker H. Franz
Department of Computer Science, University of Tübingen, Tübingen, Germany
Researchers often need to decide whether two conditions are practically equivalent. Linde et al. (2021, Psychological Methods) compared three methods that allow for such a decision: (a) frequentist two one sided tests (TOST), (b) Bayesian highest density interval combined with a region of practical equivalence (HDI+ROPE), (c) Bayes Factor with an interval null prior (interval BF). Linde et al. concluded that interval BF has superior power to detect equivalence—especially for small sample sizes. However, they did not distinguish between ‘indecisive’ and ‘evidence for inequivalence’ outcomes of HDI+ROPE and interval BF. Instead, they pooled those outcomes as ‘no evidence for equivalence’ to make them comparable to TOST (which has only two outcomes). We redid their simulations, allowed for all three outcomes, and included a frequentist method with three outcomes: (d) the analogue of HDI+ROPE based on confidence intervals (CI+ROPE). Firstly, we again find superior power to detect equivalence of interval BF compared to all other methods. However, this superiority can be compensated by adapting the other methods’ thresholds (cf. Campbell and Gustafson, 2021, arXiv:2104.07834). Secondly, we observe that, unlike HDI+ROPE and CI+ROPE, interval BF’s ‘indecisive’ outcomes are more likely under inequivalence than under equivalence, resulting in an overall bias of the method in favor of equivalence. Notably, this bias cannot be eliminated by adapting symmetrical Bayes factor thresholds alone. These findings suggest that HDI+ROPE and CI+ROPE might actually be better suited to decide between equivalence, indecision, and non-equivalence of two conditions.
Keywords: Hypothesis testing, Equivalence testing, Bayesian, frequentist, Bayes Factor, TOST, HDI+ROPE