Sequential priming tasks frequently use word-word pairs as stimuli, but cross-modality stimuli might modulate priming effects and mechanisms underlying these effects. We focus on the word-picture paradigm, because linguistic information is one of top-down factors that influence people's visual perception. The aim of the study is to investigate whether prime words either associated or categorically related with target pictures can lead to faster response times compared to neutral primes. More importantly, hierarchical diffusion model analysis is helpful to disentangle the mechanisms underlying these priming effects. In the experiment, we manipulated the semantic relationships (i.e., association relatedness and category coordinate) between prime words and target pictures in a semantic categorization task. Both associative and categorical priming effects were found in response time. The hierarchical diffusion modeling analysis showed that the associative priming mapped on drift rates, which suggests a spreading activation mechanism like word-word paradigm. The categorical priming was interpreted by the bias at the starting point, which suggests that semantic information generate the prior bias at the decision-making process. The results indicate that there are more complicated processes underlying word-picture priming than word-word priming.