Sequential sampling models are one of the most dominant cognitive accounts in psychological research. These models assume decision makers accumulate evidence as a running sum of the observed evidence strength, until an absorbing decision boundary is reached. We test this summation-based model’s predictions in a value-based binary choice task with skewed (multimodal) and unimodal numerical payoff distributions. While previous studies tend to focus on the latter, we show that in such settings, summation-based models predictions mimic alternative accounts. One such alternative is a retrieval-based accumulation model, integrating insight from the Decisions from Experience and sequential sampling literatures. This simplified model assumes agents integrate evidence from memory, retrieving outcomes previously experienced in the most similar settings. Importantly, the predictions of the summation-based and retrieval-based models diverge in skewed settings, as they predict overreaction and underreaction to extreme observations, respectively. In Study 1, participants faced 60 different sequential sampling tasks, choosing between a unimodal (gaussian) option (e.g., “draw from ~N(8, 10)”) and either another unimodal option (e.g., “draw from ~N(16, 10)”), a left-skewed option (e.g., “draw from ~N(0, 10) with p = .9, ~N(160, 10) otherwise”) or a right-skewed option (e.g., “draw from ~N(16, 10) with p = .9, ~N(-144, 10) otherwise”). Results favor our retrieval-based accumulation model, including the prediction for the distinct impact of rare extreme observations. In Study 2, a reanalysis of a previous dataset clarifies the advantages of our model. We conclude with a discussion of the optimality, ecological validity, and feasibility of the retrieval-based accumulation account.