Similarity tasks are widely used in the field of psychology, for example when studying mental representations, object recognition, or semantic processing and categorization. When collecting similarity judgments, one needs to decide which task one should choose, be it pairwise similarity ratings, odd-one-out judgments, distance-based measures, or response time tasks. Despite their regular use, to date little is known about how these tasks relate and the degree to which they measure the same underlying construct. To address this challenge, here we systematically compared eight different similarity tasks: (1) pairwise ratings, (2) pile sorting, (3) single and (4) multiple arrangement, (5) triplet odd-one-out judgments, (6) sequential forced choice similarity judgments, (7) speeded visual search and (8) speeded same-different judgments. We collected data from 100 online participants for each of the tasks using a set of natural and colored object images from the THINGS database. We analyzed the similarity estimates between tasks and their alignment with deep neural network representations and a semantic embedding, respectively. Our findings reveal that these tasks can be grouped into three types: (1) tasks that primarily capture visual features (response time tasks), (2) tasks that primarily capture semantic features (sorting tasks) and (3) tasks that capture both visual and semantic features (choice tasks). Within each group of tasks, we additionally determined their relative reliability and efficiency to support researchers with their choice of paradigm. Together, this work reveals the nature of the representations measured by different similarity tasks and provides suggestions for choosing one task over another.