• 0

FLOAT and DOUBLE in Visual Studio is garbage?


Question

I am having some problems with visual studio 2005 professional,

I am declaring a simple double to 0.7 but in runtime it is 0.699999999999996 which is not CLOSE ENOUGH precision for my program...

I was wondering if this is a normal thing, or is it a bug in visual studio or is my hardware broken...?

check the image for a break/debug right after a few doubles have been allocated, you can see my code and you can see whats in mem..

the following info is to try and give a bigger picture of my system:

OS Name Microsoft? Windows Vista? Ultimate

Version 6.0.6000 Build 6000

System Manufacturer FUJITSU SIEMENS

System Model Amilo Mx43y Series

System Type X86-based PC

Processor Intel® Pentium® M processor 1.73GHz, 1733 Mhz, 1 Core(s), 1 Logical Processor(s)

BIOS Version/Date American Megatrends Inc. 1.10C, 05/10/2005

SMBIOS Version 2.3

Hardware Abstraction Layer Version = "6.0.6000.16386"

Total Physical Memory 2,046.81 MB

Microsoft Visual Studio 2005

Version 8.0.50727.867 (vsvista.050727-8600)

Microsoft .NET Framework

Version 2.0.50727

Installed Edition: Professional

Microsoft Visual C++ 2005 77626-009-0000007-41812

Microsoft Visual C++ 2005

Microsoft Visual Studio 2005 Professional Edition - ENU Service Pack 1 (KB926601)

Update for Microsoft Visual Studio 2005 Professional Edition - ENU (KB932232)

post-2527-1183128508_thumb.jpg

14 answers to this question

Recommended Posts

  • 0

Uh. I recommend reading up on the differences between basic data types....If you need absolutely perfect precision, you'll need some kind of BCD-type to store your numbers.

A double is a 64bit floating point number, which is only accurate to like 18ish (Maybe less?) decimal points. After that it does some rounding.

Also, you're programming in c++, Visual Studio is just an IDE, and kinda irrelevent to the question. (Just a tip)

Anyway.

Try using a long double.

Edited by MioTheGreat
  • 0
  noroom said:
That's what Double.Epsilon is for.

That's .NET.

Anyway, I take back what I said above, a long double and a double are the same thing w/ msvc++. They don't have extended precision doubles (Apparently, according to an MSDN blog, it might come in VC9).

You'll have to roll your own or find a sample online with a higher precision float, or a BCD (Which would be slower than the other option, but much more precise)

  • 0

in this case I just needed 0.7 as a nice number, when I saw that it wasnt what I specified I got worried that something might be broken somewhere, I remember reading about how NASA couldnt use the pentium cpu because it calculated numbers wrong... so I got a bit worried.. ;)

thanks for your replies, it really helped me to understand how things work. still I find it weird that I specify 0.7 but it doesnt store it like that...

I tried the same thing in C# and 0.7 is 0.7 in there...

  • 0

Yeah, this one is easy to understand. In fact, it happens very often and very easily if you aren't careful.

What's going on? Consider the value 0.5. That is easily representable in binary because binary uses powers of two. In the case of fractional parts, it works the same way as in decimal:

0.5 = 5 * 10^(-1) = 5 * 1/10 = 5/10 = 1/2

0.1b = 1 * 2^(-1) = 1 * 1/2 = 1/2

Obviously, for negative powers of 2, it is easy to store:

0.25 = 2 * 10^(-1) + 5 * 10^(-2) = 2/10 + 5/100 = 20/100 + 5/100 = 25/100 = 1/4

0.01b = 0 * 2^(-1) + 1 * 2^(-2) = 0 + 1 * 1/4 = 0 + 1/4 = 1/4

However, what about 1/5 (0.2)?

0.2 = 2 * 10^(-1) = 2 * 1/10 = 2/10 = 1/5

0.00110011...b = 1 * 2^(-3) + 1 * 2^(-4) + 1 * 2^(-7) + 1 * 2^(-8) + .... = 1/5

As you can see, unlike in decimal, 1/5 written in binary is a repeating value, just as 1/3 is in decimal.

You are also probably wondering how I got that value, right? The method that I've found easiest to understand is displayed here:

 x = 1 / 5
 y = "."

 x = x * 2 //multiply the value by the number base
 If x >= 1.0 Then // if x > 1 then
	 x = x - 1.0  // subtract 1 from x
	 y = y + "1" // and add a form of "carry" to a string containing the binary
 Else
	 y = y + "0"
 End If
 // repeat the multiplication process until x = 0
 // or until you find a recognizable repetition in y
 // or until the string exceeds the precision of the data type you are using

To illustrate, I'll use 1/5 as an example as if the above pseudocode actually ran and the comment about repeating the multiplication was implemented as some form of loop already:

1/5 * 2 = 2/5

2/5 >= 1 ?? No - ".0"

2/5 * 2 = 4/5

4/5 >= 1 ? No - ".00"

4/5 * 2 = 8/5

8/5 >= 1 ? Yes - ".001" // 8/5 - 1 = 3/5

3/5 * 2 = 6/5

6/5 >= 1 ? Yes - ".0011" // 6/5 - 1 = 1/5

We're back to 1/5, which is obviously where we started. Therefore, we can deduce that 1/5 repeats the 0011 combination.

Note that the above method actually works for any base:

// Hexadecimal

1/5 * 16 = 16/5

16/5 >= 1 ? Yes - ".3" // 16/5 - int(16/5) = 1/5

Since 1/5 ended up being 1/5 again, we have no need for testing again because we know that it will end up being the same thing. But why the .3 this time? Simply put, when you integer divide 16 by 5, you get 3. That value is the value that you need to use for the decimal. Also note the following way to check:

every group of 4 binary digits (bits) is equal to one hexadecimal digit (sometimes known as a hexit or hex bit). In other words, look at the following from the example:

.00110011

notice the 1s? Split them up!

. 0011 0011

Now convert them to hexadecimal:

. 3 3

The result for each group is 3, just like when the hexadecimal conversion was done.

This is probably confusing to you now. However, if you practice the conversion a couple of times, it is not that difficult.

Just remember that you check to see if the fraction is greater than or equal to 1 since the fraction should be less than 1 (if it is more than one, only part of the value is a fraction ;)). If it is greater than or equal to 1, you need to subtract the integer part of the value and you also need to add the integer part of the value to a string containing the values to give you your fraction in whatever base you want. Otherwise, you add a 0 to that string.

  • 0

Try this out:

http://www.alphaworks.ibm.com/tech/decnumb...usplus/download

Or, you can switch over to Managed C++ and use the System.Decimal class which is built in. (You might be using managed c++ for all I know, I can't tell based on what you've told us)

Edited by MioTheGreat
  • 0

You could also try using GMP (the GNU Multi-Precision library). There is a static build and a dynamic build for Visual C++ here as well as the source code to GMP and patches for the source code apparently -

http://cs.nyu.edu/exact/core/gmp/

Using System.Decimal would probably be the best solution since it should be a relatively similar syntax (rather than learning a whole new language by using a 3rd-party library).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • I'm sure Denmark would stand to lose a lot if US consumers stopped buying Danish products, whether that's Lurpak butter or hi-fi equipment.
    • JD Vance will be the next President. Who've the Democrats got? Harris again? lol....
    • Microsoft Edge gets new password feature and security fixes by Taras Buria Microsoft has released a new update for the Edge browser in the Stable Channel. Version 137.0.3296.83 introduces a new password feature and fixes security vulnerabilities to make your browsing experience safer. Starting with new features, Microsoft Edge 137 now supports Secure Password Deployment. Microsoft recently announced this for IT admins, allowing them to share encrypted passwords with user groups. This service lets users log into websites without seeing their passwords, thus enhancing the organization's security. You can read more about Microsoft Edge Secure Password Deployment in our recent article here. Security updates in Microsoft Edge 137.0.3296.83 include two fixes for Chromium vulnerabilities: CVE-2025-5958: Use after free in Media in Google Chrome prior to 137.0.7151.103 allowed a remote attacker to potentially exploit heap corruption via a crafted HTML page. (Chromium security severity: High) CVE-2025-5959: Type Confusion in V8 in Google Chrome prior to 137.0.7151.103 allowed a remote attacker to execute arbitrary code inside a sandbox via a crafted HTML page. (Chromium security severity: High) You can update Microsoft Edge to the latest version by heading to edge://settings/help. The browser can also update itself automatically in the background and apply updates between restarts. In case you missed it, Microsoft released Edge 137 by the end of May. The update deprecated quite a lot of existing features, including Wallet, Image Editor, Image Hover, Mini menu, and Video Super Resolution. It also introduced Web Content Filtering and enhancements for the picture-in-picture player and Find on Page in Microsoft Edge for Business. The next feature update for Microsoft Edge, version 138, is expected on the week of June 26, 2025, as part of the standard four-week release cadence.
    • Microsoft commits to upskill 1 million UK workers in AI this year by Paul Hill Microsoft has partnered with the UK government in the latter’s ambitious plan to train 7.5 million workers in AI skills over the next five years. Specifically, Microsoft has committed to upskilling 1 million of those workers by the end of this year. This represents a significant portion of the overall target and within a very short timeframe. The education drive by Microsoft builds on its previous “Get On” program, which has given 1.5 million people basic digital skills. The effort to train up 1 million British workers in AI is part of Microsoft’s broader £2.5 billion investment in UK AI infrastructure. Ensuring workers have the skills to leverage AI tools is important. Microsoft CEO UK Darren Hardman said recently that two-thirds of business people wouldn’t hire someone lacking AI skills, showing just how vital it is to get people’s skills up to date. Microsoft's approach to AI skills development Microsoft has several platforms to offer AI training, including Microsoft Learn, AI Skills Navigator, and through partnerships with non-profit organisations such as Catch22 in the UK. Its educational materials cover everything from the basics of generative AI to helping you prepare for advanced roles like being an AI engineer. With Catch22, Microsoft helps to train people who face various challenges to getting tech skills, including gender and ethnicity barriers, homelessness, mental health issues, school exclusion and disability. Microsoft is also trying to get more women into tech fields through programmes like TechHer, where it has trained thousands of women across UK government departments. Many of the courses that Microsoft offers come complete with certificates that you can show off on your CV when applying for a job to impress potential employers and land a job. Who else is partnering with the UK government? While Microsoft is playing a massive role in the government’s plans, it’s not the only big tech giant helping out. The firms that have partnered with the government are: Accenture, Amazon, Barclays, BT, Google, IBM, Intuit, Microsoft, Sage, SAS, and Salesforce. While all of these firms are helping to train workers, Microsoft’s planned efforts are the most notable. This initiative by the government will help the country brace for the changes AI is expected to bring to the economy. In April, the United Nations said that AI will affect 40% of all jobs, so being ready is a must.
    • Microsoft has an update on Exchange Online Basic Auth removal for Office 365 by Sayan Sen Back in 2022, Microsoft announced the retirement of Basic Authentication as it was moving to modern OAuth 2.0 token-based authentication. The reason was simple, to move away from such simple username-password authentication to more secure sign-ins. While Microsoft had previously planned to "permanently remove support for Basic authentication with Client Submission (SMTP AUTH) in September 2025", the company has now updated this timeline, adding a final delay. Perhaps this was on the cards given that Microsoft recently extended Basic Auth support for High Volume Email to 2028. On the Microsoft 365 Admin Center, a new message has been posted that details the changes regarding SMTP (Simple Mail Transfer Protocol) AUTH Client Submission. The message says: Thus, starting March 1, 2026, Exchange Online will begin phasing out Basic authentication for sending emails via SMTP AUTH. At first, fewer attempts will be blocked, but by April 30, 2026, this older method will be fully disabled. After that, any apps or devices that want to send email this way will need to use OAuth. The message further adds how admins can proceed with the changes in case OAuth is not supported: Users who have access to the M365 Admin Center can view the message under ID MC786329.
  • Recent Achievements

    • First Post
      ThatGuyOnline earned a badge
      First Post
    • One Month Later
      5i3zi1 earned a badge
      One Month Later
    • Week One Done
      5i3zi1 earned a badge
      Week One Done
    • Week One Done
      julien02 earned a badge
      Week One Done
    • One Year In
      Drewidian1 earned a badge
      One Year In
  • Popular Contributors

    1. 1
      +primortal
      540
    2. 2
      ATLien_0
      225
    3. 3
      +FloatingFatMan
      160
    4. 4
      Michael Scrip
      115
    5. 5
      +Edouard
      92
  • Tell a friend

    Love Neowin? Tell a friend!