A thing that needs to be noted is most TV theses days will remember the settings used for each input. For example if i set the sound at 30 when using the HDMI 3 input of my TV and then switch to another input the sound will revert back to the sound level set the last time that input was used.
There's many settings on most TV influencing image quality. The stretch setting is one of them. And it might be defaulted differently depending on the input. My TV has 4 level on stretch. Dot by dot (no stretch), stretch, wide and s wide or something like that. Stretch is used to remove the small black borders some TV channels might have. It doesnt stretch all that much just a little bit. Wide is used to make 4:3 image stretch to 16:9. S wide is used to make 16:9 image encapsulated in a 4:3 image stretch to full screen 16:9. If i use the s wide setting on a 1080p output it will look terribly blurry as the image is stretched a lot.
My TV has another setting which is noise reduction. It's a nice setting to use for downloaded 480p TV show as it actually does a better job than my gpu i think and i don't have to change my gpu setting for no reason as i rarely watch 480p video from my PC anymore. But if used for 1080p or 720p output then i'll make the image blurry for no reason (there's not enough noise in those video to justify a blurry image).
Your TV might have other settings having an impact on image quality. Those settings might be set differently on the VGA input (or simply not avalaible on this input i.e. turned off by default).