Sometimes, problems are solved with a sudden leap of understanding, which is often met with an Aha! Experience. Initially, we tend to employ analytical strategies to solve problems (non-insight). Studying insight has proven difficult because of its elusive nature. An often-used procedure to identify whether insight occurred is relying on participants’ insight/non-insight self-reports after each solved problem. However, it remains unclear whether insight and non-insight are consistently identified similarly across participants, and on which features this classification is based (e.g., solution confidence). We argue that if the solution types derived from self-reports are consistent, we should also be able to detect them as naturally occurring clusters in the data, independently from self-reports. To address this, we asked participants (N = 65) to complete word puzzles solvable with insight and non-insight. Participants tracked their closeness-to-solution while problem-solving. After each solved problem, they completed the insight/non-insight self-report, and then indicated their confidence in the solution, and the insight intensity. We applied several clustering algorithms based on the online closeness-to-solution data, solution confidence, insight intensity, solution accuracy, and solution time. Based on these features and clustering algorithms, we observed two clusters closely aligning with self-reported insight and non-insight solutions. Moreover, the cluster identification consisted of a combination of the features, not just one that biases the insight classification (e.g., a preponderant influence of solution time). These results contribute to assess the validity of the insight/non-insight self-report procedure, showing how participants consider their problem-solving experience in a similar and shared way.