We design an incentivized binary line length judgment experiment to better understand stochastic choice and how it is driven by noisy beliefs. We are able to observe whether the objectively optimal---but imperfectly perceived---choice was made. We can also present subjects with materially identical choice sets in different trials, without the repetition being apparent to subjects. In some trials, subjects make incentivized choices between lines and are paid a function of the length of the selected line. In a subset of these trials, subjects can select indifference which directs the computer to make the choice for them based on a known distribution and in other trials the computer's choice would be based on an unknown distribution. We find considerable randomness in the choices, including in trials where the lines are equally long. While we see more indifference choices when the distribution is known to the subjects, we observe considerable choices of indifference even when the distribution is not known. Then subjects are directed to two-stage trials: first they make a binary choice between the lines and then are directed to make pairwise choices between objective lotteries and payments based on whether they selected the longer line in the prior stage. These responses provide a measure of the subject's confidence in their line selection. We find that most subjects have different measures of confidence in materially identical choice sets. We find that subjects are sophisticated in that---controlling for the decision difficulty and the subject-specific decisions---subjects are more confident on trials in which they selected the longer line.