I don't see why you need an ocx, which is mostly used for components that interface with the user (GUI), I think you mean a normal COM dll
. Anyway, a vector library is very easy to implement yourself, most vector libraries are part of much bigger libraries and including them would probably be overkill. The 4 or 5 most used vector operations are like 10 lines to program. Anyway, if it's speed you are looking for it's better to make your own library in C/C++, compile to dll (normal one) and call it from vb. If that is too advanced for you, just program it in vb, but it will be slower.
If you really want to use a fast library and not afraid of all the overhead that comes with it you could use the math library of directx, which has a COM interface.