You can disable dns caching : cmd> net stop dnscache
Private browsing can not be achieved on a shared/public computer, however. Simply because you wouldn't know what's running on that computer, a simple keylogger (legit software or malware) will be logging every keystroke... other scenarios/threats exist.
If its your own computer, then there's no need for portable FF or a pen-drive. If you're paranoid, full system encryption will mitigate physical access, some VPN solution will mitigate remote access.
Lets say someone wanted to look up porn on their computer. But their family at times also uses their machine. Why couldn't they use portable firefox on a true crypt encrypted thumbstick just to look up their porn, yet still have the benifits of history, cache and favorites and cookies. When they are done looking up porn they dis mount and pull the thumbstick out of the computer. So no traces will ever be found by a family member.
That's just one example.