In acceptor doped perovskite ferroelectrics, the A site or B site ions can be substituted by ions with a lower valence, e.g., Ti ions in BaTiO3 are substituted by Mn. The associates of the acceptors and the compensating oxygen vacancies form non-switchable or hardly switchable defect dipoles, which shift or pinch the dielectric hysteresis, and might affect or even enhance the electrocaloric effect (ECE) significantly.An analytical model based on the entropy change calculation is utilized to investigate the influence of the defect dipoles on the ECE, while the lattice-based Monte-Carlo simulations are applied to reveal the mechanism on the domain structure level. Additionally, for comparison, the Molecular Dynamics with the effective Hamiltonian are performed. All three approaches confirm that there is a competition between the external field and the internal field induced by the defect dipoles, and when the induced internal field is stronger than the external field, the positive ECE transfers to the negative ECE. More complex phenomena, e.g., the coexistence of the positive ECE and the negative ECE, and the double peak behavior of the ECE, are also revealed. Simultaneously, a modified electrocaloric cycle, which makes use of the negative electrocaloric effect in the presence of defect dipoles, is proposed to enhance the cooling effect.