Recent experimental findings suggest that the stimuli representation in the piriform cortex may drift over weeks in animals. Nevertheless, the exact mechanism underlying this phenomenon is still not fully understood. In this work, we use theoretical and computational analyses to study this phenomenon in a simple two-layer network model of the olfactory system. The network receives inputs form the olfactory receptors and yields an output mimicking olfactory percepts. The representation in the expansive hidden layer is expected to model odor representation in the piriform cortex. We show that the stochasticity during learning, arising merely from the experienced stimuli, could lead to a diffusion of neural network parameters, even when the training is complete. As a result, the similarity between the representations of the same odorant decays exponentially over time, as observed in the experiments. We also show that the representation of a stimulus which is presented more often than others drifts at a slower rate, similarly to the experimental observations. We further explore mechanisms underlying the stimulus-dependent drift in the model and relate them to the experimental findings.