• 0

FLOAT and DOUBLE in Visual Studio is garbage?


Question

I am having some problems with visual studio 2005 professional,

I am declaring a simple double to 0.7 but in runtime it is 0.699999999999996 which is not CLOSE ENOUGH precision for my program...

I was wondering if this is a normal thing, or is it a bug in visual studio or is my hardware broken...?

check the image for a break/debug right after a few doubles have been allocated, you can see my code and you can see whats in mem..

the following info is to try and give a bigger picture of my system:

OS Name Microsoft? Windows Vista? Ultimate

Version 6.0.6000 Build 6000

System Manufacturer FUJITSU SIEMENS

System Model Amilo Mx43y Series

System Type X86-based PC

Processor Intel® Pentium® M processor 1.73GHz, 1733 Mhz, 1 Core(s), 1 Logical Processor(s)

BIOS Version/Date American Megatrends Inc. 1.10C, 05/10/2005

SMBIOS Version 2.3

Hardware Abstraction Layer Version = "6.0.6000.16386"

Total Physical Memory 2,046.81 MB

Microsoft Visual Studio 2005

Version 8.0.50727.867 (vsvista.050727-8600)

Microsoft .NET Framework

Version 2.0.50727

Installed Edition: Professional

Microsoft Visual C++ 2005 77626-009-0000007-41812

Microsoft Visual C++ 2005

Microsoft Visual Studio 2005 Professional Edition - ENU Service Pack 1 (KB926601)

Update for Microsoft Visual Studio 2005 Professional Edition - ENU (KB932232)

post-2527-1183128508_thumb.jpg

14 answers to this question

Recommended Posts

  • 0

Uh. I recommend reading up on the differences between basic data types....If you need absolutely perfect precision, you'll need some kind of BCD-type to store your numbers.

A double is a 64bit floating point number, which is only accurate to like 18ish (Maybe less?) decimal points. After that it does some rounding.

Also, you're programming in c++, Visual Studio is just an IDE, and kinda irrelevent to the question. (Just a tip)

Anyway.

Try using a long double.

Edited by MioTheGreat
  • 0
  noroom said:
That's what Double.Epsilon is for.

That's .NET.

Anyway, I take back what I said above, a long double and a double are the same thing w/ msvc++. They don't have extended precision doubles (Apparently, according to an MSDN blog, it might come in VC9).

You'll have to roll your own or find a sample online with a higher precision float, or a BCD (Which would be slower than the other option, but much more precise)

  • 0

in this case I just needed 0.7 as a nice number, when I saw that it wasnt what I specified I got worried that something might be broken somewhere, I remember reading about how NASA couldnt use the pentium cpu because it calculated numbers wrong... so I got a bit worried.. ;)

thanks for your replies, it really helped me to understand how things work. still I find it weird that I specify 0.7 but it doesnt store it like that...

I tried the same thing in C# and 0.7 is 0.7 in there...

  • 0

Yeah, this one is easy to understand. In fact, it happens very often and very easily if you aren't careful.

What's going on? Consider the value 0.5. That is easily representable in binary because binary uses powers of two. In the case of fractional parts, it works the same way as in decimal:

0.5 = 5 * 10^(-1) = 5 * 1/10 = 5/10 = 1/2

0.1b = 1 * 2^(-1) = 1 * 1/2 = 1/2

Obviously, for negative powers of 2, it is easy to store:

0.25 = 2 * 10^(-1) + 5 * 10^(-2) = 2/10 + 5/100 = 20/100 + 5/100 = 25/100 = 1/4

0.01b = 0 * 2^(-1) + 1 * 2^(-2) = 0 + 1 * 1/4 = 0 + 1/4 = 1/4

However, what about 1/5 (0.2)?

0.2 = 2 * 10^(-1) = 2 * 1/10 = 2/10 = 1/5

0.00110011...b = 1 * 2^(-3) + 1 * 2^(-4) + 1 * 2^(-7) + 1 * 2^(-8) + .... = 1/5

As you can see, unlike in decimal, 1/5 written in binary is a repeating value, just as 1/3 is in decimal.

You are also probably wondering how I got that value, right? The method that I've found easiest to understand is displayed here:

 x = 1 / 5
 y = "."

 x = x * 2 //multiply the value by the number base
 If x >= 1.0 Then // if x > 1 then
	 x = x - 1.0  // subtract 1 from x
	 y = y + "1" // and add a form of "carry" to a string containing the binary
 Else
	 y = y + "0"
 End If
 // repeat the multiplication process until x = 0
 // or until you find a recognizable repetition in y
 // or until the string exceeds the precision of the data type you are using

To illustrate, I'll use 1/5 as an example as if the above pseudocode actually ran and the comment about repeating the multiplication was implemented as some form of loop already:

1/5 * 2 = 2/5

2/5 >= 1 ?? No - ".0"

2/5 * 2 = 4/5

4/5 >= 1 ? No - ".00"

4/5 * 2 = 8/5

8/5 >= 1 ? Yes - ".001" // 8/5 - 1 = 3/5

3/5 * 2 = 6/5

6/5 >= 1 ? Yes - ".0011" // 6/5 - 1 = 1/5

We're back to 1/5, which is obviously where we started. Therefore, we can deduce that 1/5 repeats the 0011 combination.

Note that the above method actually works for any base:

// Hexadecimal

1/5 * 16 = 16/5

16/5 >= 1 ? Yes - ".3" // 16/5 - int(16/5) = 1/5

Since 1/5 ended up being 1/5 again, we have no need for testing again because we know that it will end up being the same thing. But why the .3 this time? Simply put, when you integer divide 16 by 5, you get 3. That value is the value that you need to use for the decimal. Also note the following way to check:

every group of 4 binary digits (bits) is equal to one hexadecimal digit (sometimes known as a hexit or hex bit). In other words, look at the following from the example:

.00110011

notice the 1s? Split them up!

. 0011 0011

Now convert them to hexadecimal:

. 3 3

The result for each group is 3, just like when the hexadecimal conversion was done.

This is probably confusing to you now. However, if you practice the conversion a couple of times, it is not that difficult.

Just remember that you check to see if the fraction is greater than or equal to 1 since the fraction should be less than 1 (if it is more than one, only part of the value is a fraction ;)). If it is greater than or equal to 1, you need to subtract the integer part of the value and you also need to add the integer part of the value to a string containing the values to give you your fraction in whatever base you want. Otherwise, you add a 0 to that string.

  • 0

Try this out:

http://www.alphaworks.ibm.com/tech/decnumb...usplus/download

Or, you can switch over to Managed C++ and use the System.Decimal class which is built in. (You might be using managed c++ for all I know, I can't tell based on what you've told us)

Edited by MioTheGreat
  • 0

You could also try using GMP (the GNU Multi-Precision library). There is a static build and a dynamic build for Visual C++ here as well as the source code to GMP and patches for the source code apparently -

http://cs.nyu.edu/exact/core/gmp/

Using System.Decimal would probably be the best solution since it should be a relatively similar syntax (rather than learning a whole new language by using a 3rd-party library).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Really? Do you want it spelled out? Ok: It was a funsies way to say that we can confidently asume that their update servers were shut down years ago already, and no updates have been developed or delivered to any LG phone in years, since, to begin, LG wasn't too well known for their software support.
    • Good afternoon, I have a customer who purchased a Dell laptop last year. One of the latest Inspirons. they are older so it can be a bit hard to see things on the screen. They are using Windows 11 Home 23H2, I have not upgraded them yet to 24H2. What I did was changed the scaling to 150% and that automatically made all the icons on desktop bigger and using in Chrome webpage elements are bigger etc. The thing is I overlooked the Taskbar and the icons most notably hidden within the arrow pointing up. I though by changing to 150% the scaling even the taskbar would get bigger. Again I don't know how I missed seeing that. Is this something that can be fixed? One other thing is I notice in File Explorer things are also small and not larger after the change. Can this be fixed as well?   Thank You.
    • Apple still has two unannounced features for iOS 26 by Hamid Ganji Apple held its WWDC25 event this month to unveil a slew of AI features and its new design for operating systems, known as Liquid Glass. While iOS 26 currently has no shortage of features, the iPhone maker might still have some features under wraps that didn’t make a debut at this month’s WWDC. Bloomberg’s Mark Gurman writes in his weekly Power On newsletter that Apple didn’t announce two iOS 26 features at the WWDC event. The first feature is a live translation of conversations via AirPods, and the latter is the ability to sync your wireless network login information across devices at a hotel or gym. Even though these features were already tipped to arrive in iOS 26, Apple held them from the event, presumably because they’re still not ready to ship. Apple appears to have learned from its experience with Apple Intelligence and aims to break the habit of unveiling new features before they are ready for release. The Live Translation on iOS 26 is currently integrated with popular apps like Messages, FaceTime, and Phone to help users break language barriers and communicate in different languages. The feature is now under development for AirPods, allowing users to hear real-time speech translations. Moreover, the WiFi syncing feature allows you to sync your sign-in information across your entire Apple ecosystem, enabling you to connect to a public WiFi network with ease. This would eliminate the need to sign into a WiFi network separately on each device. While these features haven’t arrived in iOS 26 yet, they’re more likely to be released later this year, in October or December.
    • Intel Level L4 cache has been around for a long time, so AMD Zen 3D Cache is a copy of Intel L4 cache that was introduced with Broadwell i7-5775c, which had 128mb of Level L4 cache. Secondly, Zen 6 is not faster in single-threaded performance than 13900k/14900k. 285k is something new Intel is trying, basically showing us that chiplet design does not work for gaming, just like AMD design does not work for gaming. Throwing more cores at it and having L4 cache won't fix the issue, and that is frame dip and stuttering caused by a very stupid design decision to keep the memory controller outside the compute tile. Might as well put a memory controller by bringing back a south bridge chip. lol
  • Recent Achievements

    • Community Regular
      Primey_ went up a rank
      Community Regular
    • Reacting Well
      Gromvar earned a badge
      Reacting Well
    • Dedicated
      BreakingBenjamin earned a badge
      Dedicated
    • Week One Done
      Hartej earned a badge
      Week One Done
    • One Year In
      TsunadeMama earned a badge
      One Year In
  • Popular Contributors

    1. 1
      +primortal
      517
    2. 2
      +FloatingFatMan
      183
    3. 3
      ATLien_0
      168
    4. 4
      Skyfrog
      97
    5. 5
      Som
      96
  • Tell a friend

    Love Neowin? Tell a friend!