Significant strides have been made in the previous half decade in the development of comparative measures of de facto judicial independence based on IRT models using either disparate measures or disparate expert coders, especially the Latent Judicial Independence measure by Linzer & Staton and the measure by the Varieties of Democracy project. The result is an increasing amount of research examining temporal and/or cross-sectional variation in judicial independence. While the validity of previous, simpler measures of de facto independence has been studied, to date no attention has been paid to the validity of these increasingly popular measures. This paper examines the content, construct, and face validity of these measures, examining the degree to which they effectively capture the historical record and track the well-known "global expansion of judicial power" that began in the second half of the 20th century. Quantitative analyses and comparative case studies demonstrate that while each has strengths, significant issues with validity mean careful attention is necessary in their use both cross-nationally and longitudinally in single countries.