The phase evolution of the (1-x)AgNbO3-xLiTaO3 solid solution is investigated by the neutron diffraction, dielectric and ferroelectric measurements. The symmetry-mode decomposition of the distorted AgNbO3 structure defined on the experimental space group, Pmc21 has been conducted. The four main modes, T4+, H2, Λ3 and Γ4-, exhibit large distorted amplitude to stabilize the overall Pmc21 structure. The mode refinement with referring to the Pmc21 was adopted to this material system. The structural refinement results suggest that with the increasing LiTaO3, the structure will partially transfer from the orthorhombic to rhombohedral phase whose fraction grows up accordingly, along with a sudden drop in the mode amplitudes of the H2 and Λ3. The hidden structural correlation between H2 and Λ3 modes helps understand the improper nature of the antiferroelectricity observed in the AgNbO3. The variation of the main modes helps build the bridge between the Pmc21 and R3c phases, providing the mechanism underlying the phase transition between these two phases. Furthermore, the evolution of the R3c phase fraction and associate mode amplitude in both Pmc21 and R3c phases can well explain the additional peak observed in the temperature-dependent dielectric spectra and composition-dependent polarisation-electric field hysteresis loops.