firstly that's a temperature setting, unless your fridge is from like the 50's. secondly it should be set to 4 unless your fridge setting is wrong, put a thermometer in it, and set it to whatever setting gives it 4 degrees Celsius inside. in fact it's likely that when you turn "down" the fridge, you're actually setting it colder and this using more energy. depend on the manufacturer though.
either way you shouldn't adjust the setting based on the amount of food, it should be adjusted to whatever setting gives you 4 degree celsius.
The setting is called cooling power, it's not a thermostat. 1 is low cooling power, 5 is high cooling power.