Again just putting things in vlans does not prevent access.. Unless you don't route between the vlans, which then how do the users access the servers? Kind of pointless to have servers that users can not access.
So do you have a firewall routing traffic between the vlans? Or is the switch just routing intervlan traffic - if so what acls are you putting into place?
You clearly stated "the 20 office computer has are able to read/write to a certain directory" So you must be routing traffic between the vlans, so how are they now secure?
I think you misunderstand what a vlan is, a vlan is just another network segment. What is your security device between them? Firewall, just acls on the switch. What is the exact model of switch, is it layer 4?
If all you have done is create 2 network segments and are now routing traffic between them - all you have done is create 2 different broadcast domains to limit the broadcast traffic the servers will see
If you are not routing traffic between the segments -- how do the uses or guests access your servers services?