Double type number sum error - VBA/VB

Asked

Viewed 76 times

0

look at the situation:

I have a routine that does a sum of Price * Qty and stores in a variable to show the total.

The sum is composed of 226 items. From 215 the partial result of the sum would be 4750.10 but is shown 4750.099999999999, and all numbers multiplied are integers and decimals with 2 houses.

I did the same simulation in Excel, in VBA, and the same problem occurs.

The only solution I found was to change the type of the variable that stores the total to SINGLE instead of DOUBLE. Thus the error does not appear.

I would like to know if anyone has experienced this and if there is any limitation that causes this type of situation.

Routine in VBA:

Sub Test()

Dim Line As Integer, Total As Double
Dim Qty As Double
Dim Price As Double

Line = 2
Do While Planilha1.Cells(Line, 1) <> ""
    Price = Planilha1.Cells(Line, 2)
    Qty = Planilha1.Cells(Line, 3)
    Total = Total + (Price * Qty)
    'Aqui, na linha 215, o resultado da soma mostra 4750.09999999 quando de fato é  4750.10
    If Line = 215 Then MsgBox Total
    Line = Line + 1
Loop
MsgBox Total

End Sub

If you change the variable declaration Total for the guy Single, the problem does not happen.

In this link is the spreadsheet, with the proper macro that shows the problem:

http://s000.tinyupload.com/index.php?file_id=07636099064547863256

I appreciate any help.

  • "[...] the partial result of the sum would be 4750,10". If the result of the sum has decimal places, the numbers used are not integers. Also, the fact that numbers with decimals only have two houses does not prevent results with more houses. Make your calculator account 1285 * 0,33 * 0,77 (here in my resulted in 326,5185). That is, the result may be essentially certain, only with a too high precision for what you want. You tried to simply round to two decimal places?

  • Function round in Excel: https://support.office.com/en-us/article/ROUND-function-C018C5D8-40FB-4053-90B1-B3E7F61A213C

  • 1

    Hi Luiz, thanks for your help. In this case, the accounts are always of a decimal number with two boxes for another integer value (price * quantity). I can imagine even from the fact that using Double is showing an unnecessary precision, but I couldn’t see where this accuracy is resulting in this value, since in the example, the values would not allow such a result with 3 or more decimal places. If you use the Format function in VBA the problem ceases to exist, and if you declare the variable that stores the total for the Single type.

No answers

Browser other questions tagged

You are not signed in. Login or sign up in order to post.