Avoiding Rounding Errors with QAB - T4000005 - 19 Apr 2000

DP4 Programmers should all be familiar with the use of scale factors on numeric quantities (especially currency amounts) so as to ensure values such as 0.01 can be stored exactly. However, you need to be aware that just using a scale factor will not completely protect you against rounding errors when you calculate fields using formulas.

Consider the following scenario:

• A numeric value is read into a scale factor 0 field in a temporary table using Importer. The value imported represents a price in pence (or cents or whatever the unit of currency is divided into). Then a QAB program copies the data into a "real" table where the value is stored as a scale factor 2 number. The QAB program has to divide the number by 100 using a formula.

QAB programmers have to be careful how they code such a formula as they can easily introduce rounding errors. The formula temp_value/100 will introduce a rounding error even when the result is stored in a scale factor 2 because temp_value is a scale factor 0 number, and the correction to the appropriate scale factor is only done after the formula has been evaluated, by which time there is already a rounding error.

There are three possible ways of avoiding this problem - which you use is entirely a matter of personal preference.

1. temp_value * 0.01

   This formula is very efficient as internally QAB multiplies by 1 and increases the scale factor by 2 which is exactly what needs to be done to the quantity. The only drawback is that this code would be quite wrong in a C program as 0.01 is not an exact binary fraction. This style of code might also be inappropriate if divisors that were not a power of 10 were needed, or if the number being divided might already contain rounding errors.

2. temp_value * 1.00/100

   This formula firsts converts the number to a scale factor 2 number by multiplying by 100, and then divides by 100. It is less efficient than the first formula, but possibly less error prone. Once again it will not remove rounding errors already present.

3. round(temp_value/100,2)

   The QAB round function always scales the number by multiplying by 10ⁿ where n is the number of places rounded to, and then converts the answer to the nearest integer. This is probably the least efficient way of getting an answer without a rounding error, as one has already been introduced in the calculation. However, it works if temp_value already has a rounding error.

You can use the DP4 diagnostic utility ROUNDER to check database fields for rounding errors, and to correct rounding errors that have been introduced by incorrect code.