• 0

FLOAT and DOUBLE in Visual Studio is garbage?


Question

I am having some problems with visual studio 2005 professional,

I am declaring a simple double to 0.7 but in runtime it is 0.699999999999996 which is not CLOSE ENOUGH precision for my program...

I was wondering if this is a normal thing, or is it a bug in visual studio or is my hardware broken...?

check the image for a break/debug right after a few doubles have been allocated, you can see my code and you can see whats in mem..

the following info is to try and give a bigger picture of my system:

OS Name Microsoft? Windows Vista? Ultimate

Version 6.0.6000 Build 6000

System Manufacturer FUJITSU SIEMENS

System Model Amilo Mx43y Series

System Type X86-based PC

Processor Intel® Pentium® M processor 1.73GHz, 1733 Mhz, 1 Core(s), 1 Logical Processor(s)

BIOS Version/Date American Megatrends Inc. 1.10C, 05/10/2005

SMBIOS Version 2.3

Hardware Abstraction Layer Version = "6.0.6000.16386"

Total Physical Memory 2,046.81 MB

Microsoft Visual Studio 2005

Version 8.0.50727.867 (vsvista.050727-8600)

Microsoft .NET Framework

Version 2.0.50727

Installed Edition: Professional

Microsoft Visual C++ 2005 77626-009-0000007-41812

Microsoft Visual C++ 2005

Microsoft Visual Studio 2005 Professional Edition - ENU Service Pack 1 (KB926601)

Update for Microsoft Visual Studio 2005 Professional Edition - ENU (KB932232)

post-2527-1183128508_thumb.jpg

14 answers to this question

Recommended Posts

  • 0

Uh. I recommend reading up on the differences between basic data types....If you need absolutely perfect precision, you'll need some kind of BCD-type to store your numbers.

A double is a 64bit floating point number, which is only accurate to like 18ish (Maybe less?) decimal points. After that it does some rounding.

Also, you're programming in c++, Visual Studio is just an IDE, and kinda irrelevent to the question. (Just a tip)

Anyway.

Try using a long double.

Edited by MioTheGreat
  • 0
  noroom said:
That's what Double.Epsilon is for.

That's .NET.

Anyway, I take back what I said above, a long double and a double are the same thing w/ msvc++. They don't have extended precision doubles (Apparently, according to an MSDN blog, it might come in VC9).

You'll have to roll your own or find a sample online with a higher precision float, or a BCD (Which would be slower than the other option, but much more precise)

  • 0

in this case I just needed 0.7 as a nice number, when I saw that it wasnt what I specified I got worried that something might be broken somewhere, I remember reading about how NASA couldnt use the pentium cpu because it calculated numbers wrong... so I got a bit worried.. ;)

thanks for your replies, it really helped me to understand how things work. still I find it weird that I specify 0.7 but it doesnt store it like that...

I tried the same thing in C# and 0.7 is 0.7 in there...

  • 0

Yeah, this one is easy to understand. In fact, it happens very often and very easily if you aren't careful.

What's going on? Consider the value 0.5. That is easily representable in binary because binary uses powers of two. In the case of fractional parts, it works the same way as in decimal:

0.5 = 5 * 10^(-1) = 5 * 1/10 = 5/10 = 1/2

0.1b = 1 * 2^(-1) = 1 * 1/2 = 1/2

Obviously, for negative powers of 2, it is easy to store:

0.25 = 2 * 10^(-1) + 5 * 10^(-2) = 2/10 + 5/100 = 20/100 + 5/100 = 25/100 = 1/4

0.01b = 0 * 2^(-1) + 1 * 2^(-2) = 0 + 1 * 1/4 = 0 + 1/4 = 1/4

However, what about 1/5 (0.2)?

0.2 = 2 * 10^(-1) = 2 * 1/10 = 2/10 = 1/5

0.00110011...b = 1 * 2^(-3) + 1 * 2^(-4) + 1 * 2^(-7) + 1 * 2^(-8) + .... = 1/5

As you can see, unlike in decimal, 1/5 written in binary is a repeating value, just as 1/3 is in decimal.

You are also probably wondering how I got that value, right? The method that I've found easiest to understand is displayed here:

 x = 1 / 5
 y = "."

 x = x * 2 //multiply the value by the number base
 If x >= 1.0 Then // if x > 1 then
	 x = x - 1.0  // subtract 1 from x
	 y = y + "1" // and add a form of "carry" to a string containing the binary
 Else
	 y = y + "0"
 End If
 // repeat the multiplication process until x = 0
 // or until you find a recognizable repetition in y
 // or until the string exceeds the precision of the data type you are using

To illustrate, I'll use 1/5 as an example as if the above pseudocode actually ran and the comment about repeating the multiplication was implemented as some form of loop already:

1/5 * 2 = 2/5

2/5 >= 1 ?? No - ".0"

2/5 * 2 = 4/5

4/5 >= 1 ? No - ".00"

4/5 * 2 = 8/5

8/5 >= 1 ? Yes - ".001" // 8/5 - 1 = 3/5

3/5 * 2 = 6/5

6/5 >= 1 ? Yes - ".0011" // 6/5 - 1 = 1/5

We're back to 1/5, which is obviously where we started. Therefore, we can deduce that 1/5 repeats the 0011 combination.

Note that the above method actually works for any base:

// Hexadecimal

1/5 * 16 = 16/5

16/5 >= 1 ? Yes - ".3" // 16/5 - int(16/5) = 1/5

Since 1/5 ended up being 1/5 again, we have no need for testing again because we know that it will end up being the same thing. But why the .3 this time? Simply put, when you integer divide 16 by 5, you get 3. That value is the value that you need to use for the decimal. Also note the following way to check:

every group of 4 binary digits (bits) is equal to one hexadecimal digit (sometimes known as a hexit or hex bit). In other words, look at the following from the example:

.00110011

notice the 1s? Split them up!

. 0011 0011

Now convert them to hexadecimal:

. 3 3

The result for each group is 3, just like when the hexadecimal conversion was done.

This is probably confusing to you now. However, if you practice the conversion a couple of times, it is not that difficult.

Just remember that you check to see if the fraction is greater than or equal to 1 since the fraction should be less than 1 (if it is more than one, only part of the value is a fraction ;)). If it is greater than or equal to 1, you need to subtract the integer part of the value and you also need to add the integer part of the value to a string containing the values to give you your fraction in whatever base you want. Otherwise, you add a 0 to that string.

  • 0

Try this out:

http://www.alphaworks.ibm.com/tech/decnumb...usplus/download

Or, you can switch over to Managed C++ and use the System.Decimal class which is built in. (You might be using managed c++ for all I know, I can't tell based on what you've told us)

Edited by MioTheGreat
  • 0

You could also try using GMP (the GNU Multi-Precision library). There is a static build and a dynamic build for Visual C++ here as well as the source code to GMP and patches for the source code apparently -

http://cs.nyu.edu/exact/core/gmp/

Using System.Decimal would probably be the best solution since it should be a relatively similar syntax (rather than learning a whole new language by using a 3rd-party library).

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Fresh leak suggests OnePlus Pad Lite in the works, key specs revealed by Sagar Naresh Bhavsar OnePlus recently unveiled the mid-range OnePlus 13s and the OnePlus Pad 3 Android tablet in India (the tablet is also coming to the U.S.). Now, it appears that the company is working on another tablet, this time, a budget model called the OnePlus Pad Lite. Supposedly, it will be the successor to the OnePlus Pad Go, which launched in the U.K., India, and European countries. According to the leaked images, OnePlus Pad Lite follows the same design elements as earlier OnePlus tablets. The tablet is shown to have a circular camera at the center of the rear panel, with the logo sitting in the middle. The USB-C port and speaker grills are located on the right side frame of the tablet. The alleged OnePlus Pad Lite measures 254.9 x 166.5 x 7.4mm and weighs 539 grams, which is slightly taller, wider, and thinner, but weighs less compared to the OnePlus Pad Go. The tablet, courtesy of 91Mobiles and OnLeaks, is displayed in blue, which could be the only color option. One image shows the tablet with a cover that doubles as a kickstand, but may be sold separately. Gallery: OnePlus Pad Lite Based on the leaked specs, the OnePlus Pad Lite could feature an 11-inch LCD 1920x1080 resolution 90Hz display. It could be powered by the MediaTek Helio G100 processor and paired with the Mali G57 GPU. The tablet may come with 6GB of RAM and 128GB of onboard storage, with more storage and RAM options available at launch. Reportedly, the OnePlus Pad Lite could feature a 5MP primary camera and a 5MP selfie camera. Under the hood, it could be juiced by a 9,340 mAh battery. On the software side, OnePlus Pad Lite may come with OxygenOS 15.0.1 based on Android 15 out of the box. There is no clarity on which market OnePlus plans to launch the OnePlus Pad Lite in. However, the report suggests that it could be priced under ₹20,000 (roughly $231). Images by 91mobiles x OnLeaks
    • Zen Browser 1.13b by Razvan Serea Zen Browser is a privacy-focused, open-source web browser built on Mozilla Firefox, offering users a secure and customizable browsing experience. It emphasizes privacy by blocking trackers, ads, and ensuring your data isn't collected. With Zen Mods, users can enhance their browser experience with various customization options, including features like split views and vertical tabs. The browser is designed for efficiency, providing fast browsing speeds and a lightweight interface. Zen Browser prioritizes user control over the browsing experience, offering a minimal yet powerful alternative to traditional web browsers while keeping your online activity private. Zen Browser’s DRM limitation Zen Browser currently lacks support for DRM-protected content, meaning streaming services like Netflix and HBO Max are inaccessible. This is due to the absence of a Widevine license, which requires significant costs and is financially unfeasible for the developer. Additionally, applying for this license would require Zen to be part of a larger company, similar to Mozilla or Brave. Therefore, DRM-protected media won't be supported in Zen Browser for the foreseeable future. Zen Browser offers features that improve user experience, privacy, and customization: Privacy-Focused: Blocks trackers and minimizes data collection. Automatic Updates: Keeps the browser updated with security patches. Zen Mods: Customizable themes and layouts. Workspaces: Organize tabs into different workspaces. Compact Mode: Maximizes screen space by minimizing UI elements. Zen Glance: Quick website previews. Split Views: View multiple tabs in the same window. Sidebar: Access bookmarks and tools quickly. Vertical Tabs: Manage tabs vertically. Container Tabs: Separate browsing sessions. Fast Profile Switcher: Switch between profiles easily. Tab Folders: Organize tabs into folders. Customizable UI: Personalize browser interface. Security Features: Inherits Firefox’s robust security. Fast Performance: Lightweight and optimized for speed. Zen Mods Customization: Deep customization with mods. Quick Access: Easy access to favorite websites. Open Source: Built on Mozilla Firefox with community collaboration. Community-Driven: Active development and feedback from users. GitHub Repository: Contribute and review the source code. Zen Browser 1.13b changes: New Features There's a new way to manage spaces, which brings a more intuitive and user-friendly experience Updated to firefox 139.0.4 Added support for Google safebrowsing for better security Collapsed toolbarr gets a slight UI redesign Fixes Fixed issues related to glance and split view Fixed performance issues and high GPU usage for some users Other small fixes and improvements Breaking Changes Customizable UI buttons at the bottom has been reset to a new default state Download: Zen Browser | 73.6 MB (Open Source) Download: Zen Browser ARM64 | Other Operating Systems View: Zen Browser Home Page | Screenshots 1 | 2 | Reddit Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • TBF, it has had PST support for quite a while now. But I still want them to add the ability to drag & drop between accounts / PSTs.
    • LibreOffice closes in on Microsoft Office, leaves Windows 7/8 behind in 25.8 Beta 1 by David Uzondu The Document Foundation has released LibreOffice 25.8 Beta 1 for public testing on Linux, macOS, and Windows. This is the second pre-release for the 25.8 cycle and the foundation says that the final, stable version of LibreOffice 25.8 is expected to land at the end of August 2025. Starting off with Writer, LibreOffice's Word, the developers have finally addressed some long-standing annoyances, including a new command to easily insert a paragraph break right before a table. This beta also introduces a useful privacy feature in its Auto-Redact tool, letting you strip all images from a document with a single option. To use it, go to Tools and select the Auto-Redact option: The application has improved its ability to handle different languages for punctuation, preventing mix-ups in multilingual documents. Other notable improvements have also been made. A new hyphenation rule lets you choose to prevent a word from splitting at the end of a page, moving the whole line to the next page instead. Microsoft Word has had this feature for years now. The Navigator now displays a handy tooltip with word and character counts for headings and their sub-outlines. Scrolling behavior when selecting text has been improved, making it less erratic. A new command with a keyboard shortcut was added for converting fields into plain text. Calc gets a lot of new functions that bring it closer to its competitors like Excel, including TEXTSPLIT, VSTACK, and WRAPROWS. Impress now properly supports embedded fonts in PPTX files, which should reduce headaches when sharing presentations with PowerPoint users. Alongside these additions, the project is also cleaning house; support for Windows 7, 8, and 8.1 has been completely dropped. There are also smaller UI tweaks across the suite, like allowing a single click to enter rotation mode for objects in Writer and Calc. macOS users get better integration, with proper support for native full screen mode and new window management features from the Sequoia update. In terms of performance, the team has optimized everything from loading huge DOC files and XLSX spreadsheets with tons of conditional formatting to simply switching between sheets in Calc. These improvements should be noticeable, especially when working with complex documents. A new application-wide "Viewer mode" has also been implemented, which opens all files in a read-only state for quick, safe viewing. On a related note, The Document Foundation has joined efforts by the likes of KDE to encourage Windows 10 users to switch to Linux. Also, you might have heard that Denmark, in a bid to lessen its reliance on Microsoft, has decided to make a full switch to LibreOffice, with plans to begin phasing out Office 365 in certain ministries as early as next month. If you're interested in this release, you can read the full release notes and download the binaries for your platform: Windows, macOS (Intel | Apple Silicon), or Linux (DEB | RPM). You can also get the latest stable version from our software stories page.
  • Recent Achievements

    • Week One Done
      julien02 earned a badge
      Week One Done
    • One Year In
      Drewidian1 earned a badge
      One Year In
    • Explorer
      Case_f went up a rank
      Explorer
    • Conversation Starter
      Jamie Smith earned a badge
      Conversation Starter
    • First Post
      NeoToad777 earned a badge
      First Post
  • Popular Contributors

    1. 1
      +primortal
      544
    2. 2
      ATLien_0
      227
    3. 3
      +FloatingFatMan
      160
    4. 4
      Michael Scrip
      113
    5. 5
      +Edouard
      104
  • Tell a friend

    Love Neowin? Tell a friend!