We report a series of π-conjugated polymers (P1-F, P2-Cl, and P3-OMe) with three different functional groups (fluorine, chlorine, and methoxy) on their conjugated side chains. Although all three polymers showed identical photophysical properties by varying the functional group, they showed a notable difference in their dipole moment difference between the ground and excited state (Δµge) values. Furthermore, photovoltaic properties of fullerene organic solar cells (FOSCs)/non-fullerene organic solar cells (NFOSCs) were significantly affected concerning the functional group in the π-conjugated polymer. Interestingly, halogen-substituted polymers (P1-F and P2-Cl) showed an enhanced PCE than methoxy-substituted polymer (P3-OMe) in both NFOSCs and FOSCs. Also, the FOSCs were much affected upon functional group modulation than did in NFOSCs. The difference in the photovoltaic properties of P1-F, P2-Cl and P3-OMe based OSCs was further analyzed by atomic force microscopy, space charge limited current method, water contact angle and transient photoluminescence measurements. Overall, our work sheds light on the importance of side chain functional group modulation of donor polymers for efficient F and NFOSCs.