• 0

FLOAT and DOUBLE in Visual Studio is garbage?


Question

I am having some problems with visual studio 2005 professional,

I am declaring a simple double to 0.7 but in runtime it is 0.699999999999996 which is not CLOSE ENOUGH precision for my program...

I was wondering if this is a normal thing, or is it a bug in visual studio or is my hardware broken...?

check the image for a break/debug right after a few doubles have been allocated, you can see my code and you can see whats in mem..

the following info is to try and give a bigger picture of my system:

OS Name Microsoft? Windows Vista? Ultimate

Version 6.0.6000 Build 6000

System Manufacturer FUJITSU SIEMENS

System Model Amilo Mx43y Series

System Type X86-based PC

Processor Intel® Pentium® M processor 1.73GHz, 1733 Mhz, 1 Core(s), 1 Logical Processor(s)

BIOS Version/Date American Megatrends Inc. 1.10C, 05/10/2005

SMBIOS Version 2.3

Hardware Abstraction Layer Version = "6.0.6000.16386"

Total Physical Memory 2,046.81 MB

Microsoft Visual Studio 2005

Version 8.0.50727.867 (vsvista.050727-8600)

Microsoft .NET Framework

Version 2.0.50727

Installed Edition: Professional

Microsoft Visual C++ 2005 77626-009-0000007-41812

Microsoft Visual C++ 2005

Microsoft Visual Studio 2005 Professional Edition - ENU Service Pack 1 (KB926601)

Update for Microsoft Visual Studio 2005 Professional Edition - ENU (KB932232)

post-2527-1183128508_thumb.jpg

14 answers to this question

Recommended Posts

  • 0

Uh. I recommend reading up on the differences between basic data types....If you need absolutely perfect precision, you'll need some kind of BCD-type to store your numbers.

A double is a 64bit floating point number, which is only accurate to like 18ish (Maybe less?) decimal points. After that it does some rounding.

Also, you're programming in c++, Visual Studio is just an IDE, and kinda irrelevent to the question. (Just a tip)

Anyway.

Try using a long double.

Edited by MioTheGreat
  • 0
  noroom said:
That's what Double.Epsilon is for.

That's .NET.

Anyway, I take back what I said above, a long double and a double are the same thing w/ msvc++. They don't have extended precision doubles (Apparently, according to an MSDN blog, it might come in VC9).

You'll have to roll your own or find a sample online with a higher precision float, or a BCD (Which would be slower than the other option, but much more precise)

  • 0

in this case I just needed 0.7 as a nice number, when I saw that it wasnt what I specified I got worried that something might be broken somewhere, I remember reading about how NASA couldnt use the pentium cpu because it calculated numbers wrong... so I got a bit worried.. ;)

thanks for your replies, it really helped me to understand how things work. still I find it weird that I specify 0.7 but it doesnt store it like that...

I tried the same thing in C# and 0.7 is 0.7 in there...

  • 0

Yeah, this one is easy to understand. In fact, it happens very often and very easily if you aren't careful.

What's going on? Consider the value 0.5. That is easily representable in binary because binary uses powers of two. In the case of fractional parts, it works the same way as in decimal:

0.5 = 5 * 10^(-1) = 5 * 1/10 = 5/10 = 1/2

0.1b = 1 * 2^(-1) = 1 * 1/2 = 1/2

Obviously, for negative powers of 2, it is easy to store:

0.25 = 2 * 10^(-1) + 5 * 10^(-2) = 2/10 + 5/100 = 20/100 + 5/100 = 25/100 = 1/4

0.01b = 0 * 2^(-1) + 1 * 2^(-2) = 0 + 1 * 1/4 = 0 + 1/4 = 1/4

However, what about 1/5 (0.2)?

0.2 = 2 * 10^(-1) = 2 * 1/10 = 2/10 = 1/5

0.00110011...b = 1 * 2^(-3) + 1 * 2^(-4) + 1 * 2^(-7) + 1 * 2^(-8) + .... = 1/5

As you can see, unlike in decimal, 1/5 written in binary is a repeating value, just as 1/3 is in decimal.

You are also probably wondering how I got that value, right? The method that I've found easiest to understand is displayed here:

 x = 1 / 5
 y = "."

 x = x * 2 //multiply the value by the number base
 If x >= 1.0 Then // if x > 1 then
	 x = x - 1.0  // subtract 1 from x
	 y = y + "1" // and add a form of "carry" to a string containing the binary
 Else
	 y = y + "0"
 End If
 // repeat the multiplication process until x = 0
 // or until you find a recognizable repetition in y
 // or until the string exceeds the precision of the data type you are using

To illustrate, I'll use 1/5 as an example as if the above pseudocode actually ran and the comment about repeating the multiplication was implemented as some form of loop already:

1/5 * 2 = 2/5

2/5 >= 1 ?? No - ".0"

2/5 * 2 = 4/5

4/5 >= 1 ? No - ".00"

4/5 * 2 = 8/5

8/5 >= 1 ? Yes - ".001" // 8/5 - 1 = 3/5

3/5 * 2 = 6/5

6/5 >= 1 ? Yes - ".0011" // 6/5 - 1 = 1/5

We're back to 1/5, which is obviously where we started. Therefore, we can deduce that 1/5 repeats the 0011 combination.

Note that the above method actually works for any base:

// Hexadecimal

1/5 * 16 = 16/5

16/5 >= 1 ? Yes - ".3" // 16/5 - int(16/5) = 1/5

Since 1/5 ended up being 1/5 again, we have no need for testing again because we know that it will end up being the same thing. But why the .3 this time? Simply put, when you integer divide 16 by 5, you get 3. That value is the value that you need to use for the decimal. Also note the following way to check:

every group of 4 binary digits (bits) is equal to one hexadecimal digit (sometimes known as a hexit or hex bit). In other words, look at the following from the example:

.00110011

notice the 1s? Split them up!

. 0011 0011

Now convert them to hexadecimal:

. 3 3

The result for each group is 3, just like when the hexadecimal conversion was done.

This is probably confusing to you now. However, if you practice the conversion a couple of times, it is not that difficult.

Just remember that you check to see if the fraction is greater than or equal to 1 since the fraction should be less than 1 (if it is more than one, only part of the value is a fraction ;)). If it is greater than or equal to 1, you need to subtract the integer part of the value and you also need to add the integer part of the value to a string containing the values to give you your fraction in whatever base you want. Otherwise, you add a 0 to that string.

  • 0

Try this out:

http://www.alphaworks.ibm.com/tech/decnumb...usplus/download

Or, you can switch over to Managed C++ and use the System.Decimal class which is built in. (You might be using managed c++ for all I know, I can't tell based on what you've told us)

Edited by MioTheGreat
  • 0

You could also try using GMP (the GNU Multi-Precision library). There is a static build and a dynamic build for Visual C++ here as well as the source code to GMP and patches for the source code apparently -

http://cs.nyu.edu/exact/core/gmp/

Using System.Decimal would probably be the best solution since it should be a relatively similar syntax (rather than learning a whole new language by using a 3rd-party library).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Hello, The thing about Thunderbolt 2/3/4 PCIe cards is that in additional to the signals carried over the PCIe slot they are plugged into they need to send additional signals to the motherboard for which there is no standardized connection set of connections on the PCIe bus.  To get around this limitation, motherboard manufacturers can include a separate Thunderbolt header. which is sometimes labeled as JTBT1 or TBT1 on the motherboard.  The Thunderbolt PCIe card has a corresponding header on it, and comes with a proprietary cable to connect between the card and motherboard. While the additional signals that need to be sent over the Thunderbolt header are somewhat standardized, motherboard manufacturers are also free to implement whatever custom vendor-specific additions like like, like allowing a the system to recognize an on/off button on a Thunderbolt dock they alone offer to power up the system, or recognize what an external peripheral has been plugged in or unplugged from the Thunderbolt port.  Features like this, plus the lack of requirements for standardized features, or even a standard physical layout for the Thunderbolt header, mean that manufacturers can implement what are Thunderbolt PCIe cards that are essentially proprietary in that all features only work with their motherboard and even then only when their custom cable is used between their motherboard and their Thunderbolt PCIe card.  There may be additional cables required in order to route a video card's signals through the Thunderbolt port as well. Because of this, you need to go with the motherboard manufacturer's Thunderbolt PCIe card, unless you want to get a different brand, build a custom cable to connect it to your motherboard, and potentially give up features like hot-plugging devices to it. I believe @Nik Louch identified a card which will work with your motherboard.  You can always double-check with the motherboard manufacturer just to be sure, or find out if there are any issues or limitations to the combination of your motherboard and the card. If you purchase a used Thunderbolt PCIe card, make sure to check with the seller if it comes with all of the cabling necessary to use it, otherwise you may end up having to purchase that separately as well. Regards, Aryeh Goretsky  
    • ChromeOS M137 goes stable, bringing new Face Control policy and more by David Uzondu Google has brought ChromeOS M137 to the stable channel, and it includes a few focused updates for users and IT admins. A key change is a new policy for managing big groups of Chromebooks. The face control accessibility tool, which Google also updated back in ChromeOS M135, now gets a vital control for managers. A new policy called FaceGazeEnabled lets them switch the feature on or off across a whole school or company. The update also brings a new audio feature called crosstalk cancellation. It aims to create a better sound experience using just the Chromebook's built-in speakers. The software processes audio to make it seem like it is surrounding your head, not just coming from two small points. This tries to copy the feel of a surround sound system or good headphones. Any audio gets a boost, but you will notice it most when watching movies or playing games with directional sound. More accessibility tools have arrived, too. ChromeVox now has a direct keyboard shortcut, Search + O + C, that displays spoken text as braille captions on a connected display. For the poor souls in IT, troubleshooting got a little less painful as well. A new event-based log collection system, when enabled by an admin, will automatically upload relevant logs when something specific fails, like an OS crash or a botched update. Instead of digging through mountains of data, administrators get targeted reports sent straight to them. Here's how to enable it: Turn on the Device system log upload setting. Turn on OS update status reporting—For the Report device OS information setting, select OS update status. Turn on device telemetry reporting on crash information—For the Report device telemetry setting, select Crash information. Google also keeps things sane by limiting these targeted uploads to just twice a day per device. As usual, the update is rolling out slowly. If you do not see ChromeOS M137 for your machine yet, just be patient. This phased release lets Google find and address any issues before the update gets to everyone.
    • LG gram Book 15U50T: Is this lightweight laptop the right upgrade for you? by Paul Hill If you’re in the UK looking for a new mid-range laptop that won’t feel underpowered, check out the LG gram Book 15U50T now because it’s at its all-time lowest price on Amazon UK thanks to a 14% discount from its £699.99 RRP. You can get it now for just £599.99 (Buying link at the end). At this price, the laptop definitely makes this mid-range option much more appealing, it’s also pretty new having only come out in January 2025, so you’re definitely getting more value for your money. The delivery is free and will take a few days to arrive unless you take advantage of a Prime member trial and get it next-day in time for Father’s Day. LG gram Book 15U50T: Key features and who it's for The LG gram Book 15U50T features a 15.6-inch Full-HD (1920x1080) anti-glare IPS display, making it ideal for use in well-lit areas as you won’t see yourself staring back. It’s powered by an Intel Core i5 processor (1334U), 16GB of RAM, and has a very fast 512GB NVMe Gen4 SSD. In my opinion, the storage might be a bit tight for some users; however, the device comes with two M.2 slots if you want to upgrade the storage. The LG gram Book 15U50T is ideal for students or professionals who need a device to carry with them out and about. It has an ultra-lightweight design and weighs just 1.65kg - that’s not too far off a similarly sized MacBook Air, but for a fraction of the cost. In terms of ports, there is an HDMI port, two USB-A ports, and two USB-C ports. There's also a 3.5mm headphone jack if you need to plug in headphones. Other noteworthy details about this laptop include that it's running Windows 11 Home with Copilot integration, it has a HD webcam with a privacy shutter, it uses Dolby Atmos audio for immersive sound, and it has a unique feature called gram Link for multi-device (including Android and iOS) connectivity. Should you buy it? If you are a student or a professional that won’t be doing heavy gaming, or using other super intensive applications, this laptop is a solid pick. It’s lightweight - so easy to carry around, it has an anti-reflective screen - so good in well-lit environments; and it features upgradeable storage slots if 512GB is not enough space. On the downside, this laptop has a mid-range processor that could limit your ability to use high-end professional tools. Another thing I’m not really a fan of here is how opaque LG has been with the battery life. As a portable laptop, you’re obviously going to want to take it on the go where you don’t have a charger handy, but all LG says about the battery is that it has a capacity of 51Wh. According to some online sources, variants of this laptop manage about 7 to 10 hours, so if you need a super long battery life, you might be better off with something like a MacBook Air. So should you buy it? If you’re not going to be doing anything super intensive, but can’t stand underpowered and slow budget laptops then this could be the ideal laptop for you. The £100 discount makes it even more appealing! LG gram Book 15U50T: £599.99 (Amazon UK) / RRP £699.99 This Amazon deal is U.K. specific, and not available in other regions unless specified. If you don't like it or want to look at more options, check out the Amazon UK deals page here. Get Prime, Prime Video, Music Unlimited, Audible or Kindle Unlimited, free for the first 30 days As an Amazon Associate we earn from qualifying purchases.
    • Totally different vehicles. Uber has partnered with Waymo for level 5 autonomous vehicles. Waymo has completed 10 million trips and to date, there have been 696 accidents in 4 years and of those 16 of them appear to have been due to an error by the car. In total airbags have only been deployed 38 times. The technology should always be under review and continued to be improved on, but this is a totally different animal to Tesla FSD PS. no I don't work for them etc. I am an analyst for a market intelligence firm and we have a lot of interest from clients looking at the connected car space for advertising etc. so I have studied them
    • Just seems ridiculous that my 2019 imac is no longer getting the latest macOS. Nowadays a 6 year old PC is still a fairly powerful computer. It could easily run the new OS. I also have a 2015 macbook pro with a 4th gen Intel cpu. Its running sequoia (via OCLP) and can still cope not too bad. My imac is way more powerful. Really puts me off ever buying a mac again, with such short support. Open Core legacy again for me then.
  • Recent Achievements

    • Week One Done
      somar86 earned a badge
      Week One Done
    • One Month Later
      somar86 earned a badge
      One Month Later
    • Apprentice
      Adrian Williams went up a rank
      Apprentice
    • Reacting Well
      BashOrgRu earned a badge
      Reacting Well
    • Collaborator
      CHUNWEI earned a badge
      Collaborator
  • Popular Contributors

    1. 1
      +primortal
      504
    2. 2
      ATLien_0
      260
    3. 3
      +Edouard
      186
    4. 4
      +FloatingFatMan
      174
    5. 5
      snowy owl
      132
  • Tell a friend

    Love Neowin? Tell a friend!