• 0

FLOAT and DOUBLE in Visual Studio is garbage?


Question

I am having some problems with visual studio 2005 professional,

I am declaring a simple double to 0.7 but in runtime it is 0.699999999999996 which is not CLOSE ENOUGH precision for my program...

I was wondering if this is a normal thing, or is it a bug in visual studio or is my hardware broken...?

check the image for a break/debug right after a few doubles have been allocated, you can see my code and you can see whats in mem..

the following info is to try and give a bigger picture of my system:

OS Name Microsoft? Windows Vista? Ultimate

Version 6.0.6000 Build 6000

System Manufacturer FUJITSU SIEMENS

System Model Amilo Mx43y Series

System Type X86-based PC

Processor Intel® Pentium® M processor 1.73GHz, 1733 Mhz, 1 Core(s), 1 Logical Processor(s)

BIOS Version/Date American Megatrends Inc. 1.10C, 05/10/2005

SMBIOS Version 2.3

Hardware Abstraction Layer Version = "6.0.6000.16386"

Total Physical Memory 2,046.81 MB

Microsoft Visual Studio 2005

Version 8.0.50727.867 (vsvista.050727-8600)

Microsoft .NET Framework

Version 2.0.50727

Installed Edition: Professional

Microsoft Visual C++ 2005 77626-009-0000007-41812

Microsoft Visual C++ 2005

Microsoft Visual Studio 2005 Professional Edition - ENU Service Pack 1 (KB926601)

Update for Microsoft Visual Studio 2005 Professional Edition - ENU (KB932232)

post-2527-1183128508_thumb.jpg

14 answers to this question

Recommended Posts

  • 0

Uh. I recommend reading up on the differences between basic data types....If you need absolutely perfect precision, you'll need some kind of BCD-type to store your numbers.

A double is a 64bit floating point number, which is only accurate to like 18ish (Maybe less?) decimal points. After that it does some rounding.

Also, you're programming in c++, Visual Studio is just an IDE, and kinda irrelevent to the question. (Just a tip)

Anyway.

Try using a long double.

Edited by MioTheGreat
  • 0
  noroom said:
That's what Double.Epsilon is for.

That's .NET.

Anyway, I take back what I said above, a long double and a double are the same thing w/ msvc++. They don't have extended precision doubles (Apparently, according to an MSDN blog, it might come in VC9).

You'll have to roll your own or find a sample online with a higher precision float, or a BCD (Which would be slower than the other option, but much more precise)

  • 0

in this case I just needed 0.7 as a nice number, when I saw that it wasnt what I specified I got worried that something might be broken somewhere, I remember reading about how NASA couldnt use the pentium cpu because it calculated numbers wrong... so I got a bit worried.. ;)

thanks for your replies, it really helped me to understand how things work. still I find it weird that I specify 0.7 but it doesnt store it like that...

I tried the same thing in C# and 0.7 is 0.7 in there...

  • 0

Yeah, this one is easy to understand. In fact, it happens very often and very easily if you aren't careful.

What's going on? Consider the value 0.5. That is easily representable in binary because binary uses powers of two. In the case of fractional parts, it works the same way as in decimal:

0.5 = 5 * 10^(-1) = 5 * 1/10 = 5/10 = 1/2

0.1b = 1 * 2^(-1) = 1 * 1/2 = 1/2

Obviously, for negative powers of 2, it is easy to store:

0.25 = 2 * 10^(-1) + 5 * 10^(-2) = 2/10 + 5/100 = 20/100 + 5/100 = 25/100 = 1/4

0.01b = 0 * 2^(-1) + 1 * 2^(-2) = 0 + 1 * 1/4 = 0 + 1/4 = 1/4

However, what about 1/5 (0.2)?

0.2 = 2 * 10^(-1) = 2 * 1/10 = 2/10 = 1/5

0.00110011...b = 1 * 2^(-3) + 1 * 2^(-4) + 1 * 2^(-7) + 1 * 2^(-8) + .... = 1/5

As you can see, unlike in decimal, 1/5 written in binary is a repeating value, just as 1/3 is in decimal.

You are also probably wondering how I got that value, right? The method that I've found easiest to understand is displayed here:

 x = 1 / 5
 y = "."

 x = x * 2 //multiply the value by the number base
 If x >= 1.0 Then // if x > 1 then
	 x = x - 1.0  // subtract 1 from x
	 y = y + "1" // and add a form of "carry" to a string containing the binary
 Else
	 y = y + "0"
 End If
 // repeat the multiplication process until x = 0
 // or until you find a recognizable repetition in y
 // or until the string exceeds the precision of the data type you are using

To illustrate, I'll use 1/5 as an example as if the above pseudocode actually ran and the comment about repeating the multiplication was implemented as some form of loop already:

1/5 * 2 = 2/5

2/5 >= 1 ?? No - ".0"

2/5 * 2 = 4/5

4/5 >= 1 ? No - ".00"

4/5 * 2 = 8/5

8/5 >= 1 ? Yes - ".001" // 8/5 - 1 = 3/5

3/5 * 2 = 6/5

6/5 >= 1 ? Yes - ".0011" // 6/5 - 1 = 1/5

We're back to 1/5, which is obviously where we started. Therefore, we can deduce that 1/5 repeats the 0011 combination.

Note that the above method actually works for any base:

// Hexadecimal

1/5 * 16 = 16/5

16/5 >= 1 ? Yes - ".3" // 16/5 - int(16/5) = 1/5

Since 1/5 ended up being 1/5 again, we have no need for testing again because we know that it will end up being the same thing. But why the .3 this time? Simply put, when you integer divide 16 by 5, you get 3. That value is the value that you need to use for the decimal. Also note the following way to check:

every group of 4 binary digits (bits) is equal to one hexadecimal digit (sometimes known as a hexit or hex bit). In other words, look at the following from the example:

.00110011

notice the 1s? Split them up!

. 0011 0011

Now convert them to hexadecimal:

. 3 3

The result for each group is 3, just like when the hexadecimal conversion was done.

This is probably confusing to you now. However, if you practice the conversion a couple of times, it is not that difficult.

Just remember that you check to see if the fraction is greater than or equal to 1 since the fraction should be less than 1 (if it is more than one, only part of the value is a fraction ;)). If it is greater than or equal to 1, you need to subtract the integer part of the value and you also need to add the integer part of the value to a string containing the values to give you your fraction in whatever base you want. Otherwise, you add a 0 to that string.

  • 0

Try this out:

http://www.alphaworks.ibm.com/tech/decnumb...usplus/download

Or, you can switch over to Managed C++ and use the System.Decimal class which is built in. (You might be using managed c++ for all I know, I can't tell based on what you've told us)

Edited by MioTheGreat
  • 0

You could also try using GMP (the GNU Multi-Precision library). There is a static build and a dynamic build for Visual C++ here as well as the source code to GMP and patches for the source code apparently -

http://cs.nyu.edu/exact/core/gmp/

Using System.Decimal would probably be the best solution since it should be a relatively similar syntax (rather than learning a whole new language by using a 3rd-party library).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • This is what GPOs are for, assuming they work for this. But for AD/Entra machines this should be opt in imo
    • the yellow rabbits are not trying seaweed
    • LibreWolf 139.0.4-1 by Razvan Serea LibreWolf is an independent “fork” of Firefox, with the primary goals of privacy security and user freedom. It is the community run successor to LibreFox. LibreWolf is designed to increase protection against tracking and fingerprinting techniques, while also including a few security improvements. This is achieved through our privacy and security oriented settings and patches. LibreWolf also aims to remove all the telemetry, data collection and annoyances, as well as disabling anti-freedom features like DRM. LibreWolf features: Latest Firefox — LibreWolf is compiled directly from the latest build of Firefox Stable. You will have the the latest features, and security updates. Independent Build — LibreWolf uses a build independent of Firefox and has its own settings, profile folder and installation path. As a result, it can be installed alongside Firefox or any other browser. No phoning home — Embedded server links and other calling home functions are removed. In other words, minimal background connections by default. User settings updates Extensions firewall: limit internet access for extensions. Multi-platform (Windows/Linux/Mac/and soon Android) Community-Driven Dark theme (classic and advanced) LibreWolf privacy features: Delete cookies and website data on close. Include only privacy respecting search engines like DuckDuckGo and Searx. Include uBlockOrigin with custom default filter lists, and Tracking Protection in strict mode, to block trackers and ads. Strip tracking elements from URLs, both natively and through uBO. Enable dFPI, also known as Total Cookie Protection. Enable RFP which is part of the Tor Uplift project. RFP is considered the best in class anti-fingerprinting solution, and its goal is to make users look the same and cover as many metrics as possible, in an effort to block fingerprinting techniques. Always display user language as en-US to websites, in order to protect the language used in the browser and in the OS. Disable WebGL, as it is a strong fingerprinting vector. Prevent access to the location services of the OS, and use Mozilla's location API instead of Google's API. Limit ICE candidates generation to a single interface when sharing video or audio during a videoconference. Force DNS and WebRTC inside the proxy, when one is being used. Trim cross-origin referrers, so that they don't include the full URI. Disable link prefetching and speculative connections. Disable disk cache and clear temporary files on close. Disable form autofill. Disable search and form history...and more. Notes: To receive automatic updates, you may refer to the unofficial LibreWolf-WinUpdater that employs a scheduled task to accomplish the same. Download: LibreWolf 64-bit | Portable 64-bit |~100.0 MB (Open Source) Download: LibreWolf 32-bit | Portable 32-bit | Other Operating Systems Links: LibreWolf Home Page | Addons | Screenshot | Reddit Get alerted to all of our Software updates on Twitter at @NeowinSoftware
  • Recent Achievements

    • One Month Later
      POR2GAL4EVER earned a badge
      One Month Later
    • One Year In
      Orpheus13 earned a badge
      One Year In
    • One Month Later
      Orpheus13 earned a badge
      One Month Later
    • Week One Done
      Orpheus13 earned a badge
      Week One Done
    • Week One Done
      serfegyed earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      559
    2. 2
      ATLien_0
      255
    3. 3
      +Edouard
      163
    4. 4
      +FloatingFatMan
      157
    5. 5
      Michael Scrip
      109
  • Tell a friend

    Love Neowin? Tell a friend!