• 0

FLOAT and DOUBLE in Visual Studio is garbage?


Question

I am having some problems with visual studio 2005 professional,

I am declaring a simple double to 0.7 but in runtime it is 0.699999999999996 which is not CLOSE ENOUGH precision for my program...

I was wondering if this is a normal thing, or is it a bug in visual studio or is my hardware broken...?

check the image for a break/debug right after a few doubles have been allocated, you can see my code and you can see whats in mem..

the following info is to try and give a bigger picture of my system:

OS Name Microsoft? Windows Vista? Ultimate

Version 6.0.6000 Build 6000

System Manufacturer FUJITSU SIEMENS

System Model Amilo Mx43y Series

System Type X86-based PC

Processor Intel® Pentium® M processor 1.73GHz, 1733 Mhz, 1 Core(s), 1 Logical Processor(s)

BIOS Version/Date American Megatrends Inc. 1.10C, 05/10/2005

SMBIOS Version 2.3

Hardware Abstraction Layer Version = "6.0.6000.16386"

Total Physical Memory 2,046.81 MB

Microsoft Visual Studio 2005

Version 8.0.50727.867 (vsvista.050727-8600)

Microsoft .NET Framework

Version 2.0.50727

Installed Edition: Professional

Microsoft Visual C++ 2005 77626-009-0000007-41812

Microsoft Visual C++ 2005

Microsoft Visual Studio 2005 Professional Edition - ENU Service Pack 1 (KB926601)

Update for Microsoft Visual Studio 2005 Professional Edition - ENU (KB932232)

post-2527-1183128508_thumb.jpg

14 answers to this question

Recommended Posts

  • 0

Uh. I recommend reading up on the differences between basic data types....If you need absolutely perfect precision, you'll need some kind of BCD-type to store your numbers.

A double is a 64bit floating point number, which is only accurate to like 18ish (Maybe less?) decimal points. After that it does some rounding.

Also, you're programming in c++, Visual Studio is just an IDE, and kinda irrelevent to the question. (Just a tip)

Anyway.

Try using a long double.

Edited by MioTheGreat
  • 0
  noroom said:
That's what Double.Epsilon is for.

That's .NET.

Anyway, I take back what I said above, a long double and a double are the same thing w/ msvc++. They don't have extended precision doubles (Apparently, according to an MSDN blog, it might come in VC9).

You'll have to roll your own or find a sample online with a higher precision float, or a BCD (Which would be slower than the other option, but much more precise)

  • 0

in this case I just needed 0.7 as a nice number, when I saw that it wasnt what I specified I got worried that something might be broken somewhere, I remember reading about how NASA couldnt use the pentium cpu because it calculated numbers wrong... so I got a bit worried.. ;)

thanks for your replies, it really helped me to understand how things work. still I find it weird that I specify 0.7 but it doesnt store it like that...

I tried the same thing in C# and 0.7 is 0.7 in there...

  • 0

Yeah, this one is easy to understand. In fact, it happens very often and very easily if you aren't careful.

What's going on? Consider the value 0.5. That is easily representable in binary because binary uses powers of two. In the case of fractional parts, it works the same way as in decimal:

0.5 = 5 * 10^(-1) = 5 * 1/10 = 5/10 = 1/2

0.1b = 1 * 2^(-1) = 1 * 1/2 = 1/2

Obviously, for negative powers of 2, it is easy to store:

0.25 = 2 * 10^(-1) + 5 * 10^(-2) = 2/10 + 5/100 = 20/100 + 5/100 = 25/100 = 1/4

0.01b = 0 * 2^(-1) + 1 * 2^(-2) = 0 + 1 * 1/4 = 0 + 1/4 = 1/4

However, what about 1/5 (0.2)?

0.2 = 2 * 10^(-1) = 2 * 1/10 = 2/10 = 1/5

0.00110011...b = 1 * 2^(-3) + 1 * 2^(-4) + 1 * 2^(-7) + 1 * 2^(-8) + .... = 1/5

As you can see, unlike in decimal, 1/5 written in binary is a repeating value, just as 1/3 is in decimal.

You are also probably wondering how I got that value, right? The method that I've found easiest to understand is displayed here:

 x = 1 / 5
 y = "."

 x = x * 2 //multiply the value by the number base
 If x >= 1.0 Then // if x > 1 then
	 x = x - 1.0  // subtract 1 from x
	 y = y + "1" // and add a form of "carry" to a string containing the binary
 Else
	 y = y + "0"
 End If
 // repeat the multiplication process until x = 0
 // or until you find a recognizable repetition in y
 // or until the string exceeds the precision of the data type you are using

To illustrate, I'll use 1/5 as an example as if the above pseudocode actually ran and the comment about repeating the multiplication was implemented as some form of loop already:

1/5 * 2 = 2/5

2/5 >= 1 ?? No - ".0"

2/5 * 2 = 4/5

4/5 >= 1 ? No - ".00"

4/5 * 2 = 8/5

8/5 >= 1 ? Yes - ".001" // 8/5 - 1 = 3/5

3/5 * 2 = 6/5

6/5 >= 1 ? Yes - ".0011" // 6/5 - 1 = 1/5

We're back to 1/5, which is obviously where we started. Therefore, we can deduce that 1/5 repeats the 0011 combination.

Note that the above method actually works for any base:

// Hexadecimal

1/5 * 16 = 16/5

16/5 >= 1 ? Yes - ".3" // 16/5 - int(16/5) = 1/5

Since 1/5 ended up being 1/5 again, we have no need for testing again because we know that it will end up being the same thing. But why the .3 this time? Simply put, when you integer divide 16 by 5, you get 3. That value is the value that you need to use for the decimal. Also note the following way to check:

every group of 4 binary digits (bits) is equal to one hexadecimal digit (sometimes known as a hexit or hex bit). In other words, look at the following from the example:

.00110011

notice the 1s? Split them up!

. 0011 0011

Now convert them to hexadecimal:

. 3 3

The result for each group is 3, just like when the hexadecimal conversion was done.

This is probably confusing to you now. However, if you practice the conversion a couple of times, it is not that difficult.

Just remember that you check to see if the fraction is greater than or equal to 1 since the fraction should be less than 1 (if it is more than one, only part of the value is a fraction ;)). If it is greater than or equal to 1, you need to subtract the integer part of the value and you also need to add the integer part of the value to a string containing the values to give you your fraction in whatever base you want. Otherwise, you add a 0 to that string.

  • 0

Try this out:

http://www.alphaworks.ibm.com/tech/decnumb...usplus/download

Or, you can switch over to Managed C++ and use the System.Decimal class which is built in. (You might be using managed c++ for all I know, I can't tell based on what you've told us)

Edited by MioTheGreat
  • 0

You could also try using GMP (the GNU Multi-Precision library). There is a static build and a dynamic build for Visual C++ here as well as the source code to GMP and patches for the source code apparently -

http://cs.nyu.edu/exact/core/gmp/

Using System.Decimal would probably be the best solution since it should be a relatively similar syntax (rather than learning a whole new language by using a 3rd-party library).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • we also turn off the printer thesee days, because we print something like once a month we were printing several times a week in the past, so printer was kept on, but we don’t print much right now, so we turn it off… btw, it’s a laser printer
    • It's more of snark and rhetorical question, I highly doubt KDE will have this for a while.
    • Interesting for sure, I'd like to see 3rd party benchmarks in due time. I just hope Intel can do more in the dGPU space as well.
    • Not going to connect my PC to MS just for a few months extra support,
    • Microsoft Weekly: Windows 11 version 25H2 is official, free updates for Windows 10, and more by Taras Buria This week's news recap is here, and it is full of interesting and important stories. We have Windows 11 version 25H2 announcement, free extended security updates for Windows 10, redesigned BSOD, non-security updates with new features, and more. Quick links: Windows 10 and 11 Windows Insider Program Updates are available Reviews are in Gaming news Great deals to check Windows 11 and Windows 10 Here, we talk about everything happening around Microsoft's latest operating system in the Stable channel and preview builds: new features, removed features, controversies, bugs, interesting findings, and more. And, of course, you may find a word or two about older versions. The biggest Windows story of this week was undoubtedly the launch of the Extended Security Update program for Windows 10, which will soon be out of support (governments are now issuing warnings to Windows 10 users). What is interesting is that Microsoft is giving away free security updates—all users need to do is back up their PCs with the Windows Backup tool. Other options include paying with 1,000 Microsoft Rewards points or $30, so pick your poison. If you are thinking about staying on Windows 10, perhaps this article will convince you to switch. Microsoft published a few reasons why Windows 11 is a better choice than the outgoing Windows 10. Another post compares the performance of the two operating systems in another attempt to make you ditch Windows 10. There is also a new ESU guide for office PCs that do not support Windows 11. Another major story is about the Blue Screen of Death, which will soon become the Black Screen of Death and lose its iconic smiley face. Microsoft revealed that the redesign is coming later this summer, alongside Quick Machine Recovery. This new tool can fix PCs that cannot boot due to outages, malware, or other software nastiness. We are not done with big news just yet. Microsoft confirmed that Windows 11 version 25H2 is coming later this year. This year's feature update is coming soon, and the first officially marked preview builds are now available for testing. Microsoft also released the June 2025 non-security updates for Windows 10 and 11 users. It all started with Windows 10, which received KB5061087 with build number 19045.6036. Windows 11 version 24H2 received KB5060829, and Windows 11 versions 22H2 and 23H2 received KB5060826. Also, Windows 11 received a new configuration update to resolve the stuck Windows Update (and new setup updates), Windows Server 2025 got a new security baseline, and the Media Creation Tool now downloads the latest Windows 11 images with the June 2025 Patch Tuesday fixes. Another important story is about Secure Boot, one of Windows 11's hardware requirements. Microsoft published a lengthy blog post warning that its first certificates will soon expire, and users should prepare to update them if they want their PCs to be secure and compatible with third-party apps going forward. No week goes by without some Windows issues. This time, Dell acknowledged a problem with Night Light on certain Windows on ARM PCs. The bug breaks Night Light on the secondary display, but Dell says you should blame Qualcomm and its Oryon chipset. Microsoft, on the other hand, confirmed more issues with Chrome on Windows. This week's Windows trivia includes an interesting story from Microsoft veteran Raymond Chen. He published a new blog post where he recalled how PC manufacturers used to trick BIOS copyright strings to get full editions of trial versions of various apps. To finish this week's Windows section, here is a small tip for those who want to make Windows 11 feel a little snappier. A hidden accessibility feature can make the user interface much more responsive and fast, so check it out here. Windows Insider Program Here is what Microsoft released for Windows Insiders this week: Builds Canary Channel Nothing in Canary this week Dev Channel Build 26200.5661 This build introduced a new home page for Recall, a single place where you can access your recent snapshots, recommended documents, and other useful information. The update also lets you change where system indicators appear on the screen. Build 26200.5670 This build introduces 1Password integration for Passkeys, Settings improvements, version 25H2 marking, and more. It also fixes the Windows Vista startup sound after the previously failed attempt. Beta Channel Build 26120.4452 This is the same build as 26120.4452 Build 26120.4520 This build is the same as 26200.5670 from the Dev Channel, minus the version 25H2 part. Release Preview Channel Nothing in Release Preview this week Additionally, Microsoft released new screen-recording capabilities for the Snipping Tool app for more Windows Insiders (Beta and Release Preview). Updates are available This section covers software, firmware, and other notable updates (released and coming soon) delivering new features, security fixes, improvements, patches, and more from Microsoft and third parties. This week's browser updates include some major releases and plenty of Firefox updates. Mozilla released Firefox 140 with custom search engine support, a new ESR release, and more changes. Shortly after, it released version 140.0.1 with fixes for dark theme issues and crashes and version 140.0.2 with fixes for crashes on certain Windows devices. Microsoft released Edge 138 with AI-powered history search and a warning for IT admins, and Google released Chrome 138. Office updates include the new Outlook for Windows coming to Microsoft 365 Education in early next year. Speaking of the new Outlook, Microsoft also published a story that explained why the app is actually great and why haters are wrong. Oh, Microsoft... Teams is getting a new health dashboard feature, PowerPoint can generate presentations from PDFs or text files, and Modern Page Templates are coming to SharePoint. Microsoft also published detailed guides about fixing Office 2024 activation issues. Finally, here is this week's recap of the new features coming soon to the Microsoft 365 productivity suite and a recap of everything new in Excel in June 2025. Here are other updates and releases you may find interesting: Microsoft announced Mu, an on-device small language model for Windows 11. The Comet AI browser is coming to Windows, now in private beta. Discord for Windows on ARM is now in development. Surface Copilot+ PCs are coming to classrooms on July 22. Raycast for Windows is now in closed beta; here is the first look. Visual Studio is now even smarter, thanks to more AI models and billing updates. France's third-largest city is ditching Windows and Office in favor of Linux and FOSS. Here are five things that people want in Microsoft Teams. Here are the latest drivers and firmware updates released this week: Intel 32.0.101.6913 WHQL graphics driver with Mecha BREAK support and more. Reviews are in Here is the hardware and software we reviewed this week Steven Parker reviewed the TerraMaster F4 SSD, an extremely lightweight and quiet all-SSD NAS with some good connectivity, a decent price tag, and good design. It is not flawless, but it still managed to score 8.5 out of 10 on Steven's NAS scale. Robbie Khan reviewed the Keychron Lemokey G2 8K Wireless mouse. It is lightweight, has onboard memory, supports Keychron Launcher, and includes a good cable and an adapter. However, with an 8/10 rating at Robbie's scale, it has some cons that you should consider before buying. On the gaming side Learn about upcoming game releases, Xbox rumors, new hardware, software updates, freebies, deals, discounts, and more. Microsoft finally announced the long-rumored Xbox app launcher for Windows PCs and handhelds. The Xbox app will soon work as a single place for all your games, regardless of their origin, be it Steam, Epic Games Store, Origin, or something else. As of right now, the updated app is being tested in the Xbox Insider Program. Also, Microsoft announced the June 2025 update for Xbox, bringing users unsynced save management, the ability to browse games by publishers, the option to hide system apps on the Dashboard on Xbox consoles, and more. Microsoft finally has its official Xbox-branded VR helmet. However, it is not an entirely Xbox VR per se. It was made in collaboration with Meta, and its stocks are "extremely limited." Sadly, not all Xbox news was positive this week. A new report hit out of the blue, revealing Microsoft's plans to lay off a lot of workers in the Xbox division. Microsoft Flight Simulator received a new city update with upgraded visuals of New York, New Jersey, Massachusetts, and other parts of the state to give you a more realistic experience when flying the sim. City Update 11 is now available on consoles and PCs. Deals and freebies The Epic Games Store is giving away Sable, an interesting-looking exploration game with an open world and a unique art style. If that is not enough, be sure to check out the Steam Summer Sale 2025, which is now in full swing, offering gamers a horde of discounts on various games. More deals are available in this week's edition of Weekend PC Game Deals. Other gaming news includes the following: Senua's Saga: Hellblade II is getting 60 FPS mode, dev commentary, and more. DayZ is getting a desert map with the new Badlands expansion, set to be its biggest yet. Great deals to check Every week, we cover many deals on different hardware and software. The following discounts are still available, so check them out. You might find something you want or need. ASUS NUC 14 Pro+ Core Ultra 9 185H, 32GB RAM, 1TB SSD - $799.99 | 27% off 75" Hisense U7 Series Google Smart TV - $799.99 | 11% off Sony BRAVIA Theater System 6 - $668 | 13% off 6TB WD Blue PC Internal HDD - $99.99 | 17% off 14TB WD Elements Desktop External HDD - $199.99 | 31% off Corsair iCUE 4000D RGB Airflow Mid-Tower Case - $89.99 | 40% off Samsung Galaxy S25+ 512GB - $899 | 20% off This link will take you to other issues of the Microsoft Weekly series. You can also support Neowin by registering a free member account or subscribing for extra member benefits, along with an ad-free tier option. Microsoft Weekly image background by jhenning on Pixabay
  • Recent Achievements

    • Week One Done
      Hartej earned a badge
      Week One Done
    • One Year In
      TsunadeMama earned a badge
      One Year In
    • Week One Done
      shaheen earned a badge
      Week One Done
    • Dedicated
      Cole Multipass earned a badge
      Dedicated
    • Week One Done
      Alexander 001 earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      551
    2. 2
      +FloatingFatMan
      182
    3. 3
      ATLien_0
      169
    4. 4
      Skyfrog
      108
    5. 5
      Som
      106
  • Tell a friend

    Love Neowin? Tell a friend!