Transparency and methodological rigour have been proposed as important contributors to overcoming the replication crisis in psychological science (e.g., Asendorpf et al., 2013). In order to conduct a replication study, and to obtain similar results, we need a complete description of the experimental methods. To compare studies testing identical hypotheses, the studies furthermore need to have the same research methods (Elson, 2019). Here, we focus on studies testing the Temporal Sampling Framework (Goswami, 2011), a causal theory of developmental dyslexia. We first design and evaluate a transparency rating scale, specific for theories of developmental dyslexia and similar research questions. Then, we conduct a meta-methods analysis to examine the variability in the implementation of the experimental paradigms. We found substantial variability in the transparency of the methods. When the methods were described in a detailed way, there was variability in the implementation of the paradigm. Our transparency rating scale can help researchers to include a more complete description when writing a methods section. We conclude with the recommendation of conducting direct replications instead of conceptual replications, so that the methods are kept constant to establish the robustness of a given effect (Simons et al., 2014) and taking methods in addition to results into closer consideration when conducting meta-reviews.