Reallocated sectors will definitely slow your drive down
There is no way to fix this. If a sector is unreadable the drive "maps" it elsewhere.
If this number continues to rise it is an indication of a fault and means that your drive is likely to fail.
I would make sure you have backups and think about replacing the drive.
If this number never increases again then the drive may continue to work for years yet, but it will still operate slower than when new.
Watch the "Current Pending Sector" value too. These mean more sectors are awaiting a re-map.