What does it mean to define visual sentiment---the emotional resonance conveyed by images---when viewers consistently perceive things differently, especially when their political beliefs are involved? This study introduces a novel approach to visual sentiment analysis that directly addresses these perceptual differences in sentiment classification. In order to achieve this, we developed a dataset reflecting political divisions by curating images on a polarizing topic, annotated by individuals from distinct political affiliations. Using this dataset, we trained a deep learning multi-task, multi-class model to predict visual sentiment from different ideological viewpoints. By incorporating these diverse perspectives into the labeling and model training, our approach improves the accuracy of visual sentiment predictions and better mirrors human judgment. Ultimately, this study advocates for a paradigm shift in visual sentiment decoding, urging a move beyond traditional image-focused approaches to develop classifiers that more accurately capture the complexity of human sentiment.