Arithmetic Conversion: Why Is My Result Rounded?
On a few occasions I have come across developers struggling to understand why a mathematical expression fails to produce the value they were expecting; the latest instance of this was in a StackExchange question today. In this situation in particular the phrasing of the question (which is asking how to specify the precision of a decimal variable) demonstrates just how confusing this issue can be. The behaviour being observed is due to something called arithmetic conversion and specifically the rules which govern its behaviour.
An Example
The clearest way to demonstrate the problem is to work with numeric values directly, as we can work through the expression in the same way that the system does.
Double x = 3 * (5 / 2);
Chances are, if you're not aware of how arithmetic conversion rules typically work, you will expect x to have the value 7.5, so what do we actually get if that is not the case?
// Output: USER_DEBUG [2]|DEBUG|x = 6.0
Oh. What happened? We have been victims of rounding: when the expression (5 / 2) was calculated, the result of it was rounded down to 2, so when it is subsequently multiplied by 3 we get the value 6. All of the values we use in our expression are integers, and we would have obtained the exact same result if we had used integer variables in place of the numeric constants.
Wait, Why?
Put simply, arithmetic conversion is a process whereby a system will convert the types of variables automatically when different types are used together in an expression. The reason for doing so is to ensure precision is maintained, yet in our scenario (and many others) the type of the variable being assigned to in the end is not relevant during the initial calculations.
In the code statement we used the division is the first part of the expression evaluated, and here we only deal with integers and so there is no need for conversion to another type. After the division, we have integer multiplication to perform (3 * 2), and again no conversions are required. Only after this, when the assignment is carried out, do we encounter a type mismatch between operands. On the right hand side we have the value 6, and on the left, a double precision variable to which it must be assigned; predictably the value we end up with is 6.0.
Fixing It
The solution to this is probably simpler than you might expect, or it could be that you have already spotted it. The problem arises because we use integers in a statement where we already know that the result will most likely not be a whole number, so the solution is to use a more precise value to begin with.
Double x = 3 * (5 / 2.0);
// Output: USER_DEBUG [2]|DEBUG|x = 7.5);
Adding .0 indicates to the compiler that our value should be interpreted as a Double or Decimal type (similarly appending L would indicate a Long type), and by doing so the system then knows that it needs to use double precision when using this value in an expression. In the case of (5 / 2.0) it converts the type of the constant 5 to match that of 2.0, and then the result uses double precision also, with the value 2.5.
The documentation explains these suffices and also includes an interesting, but not unexpected, note regarding the use of numeric constants which are too large to be stored in an Integer variable (side note: judging by the 11 digit example the system uses 32 bit integers).
Additional Considerations for Data Types Data Types of Numeric Values Numeric values represent Integer values unless they are appended with L for a Long or with .0 for a Double or Decimal. For example, the expression Long d = 123; declares a Long variable named d and assigns it to an Integer numeric value (123), which is implicitly converted to a Long. The Integer value on the right hand side is within the range for Integers and the assignment succeeds. However, if the numeric value on the right hand side exceeds the maximum value for an Integer, you get a compilation error. In this case, the solution is to append L to the numeric value so that it represents a Long value which has a wider range, as shown in this example: Long d = 2147483648L;.
What if I'm Using Variables, Not Constants?
If you have been caught out by this problem there is a fair chance you're dividing by an integer constant and can use the fix above. If you are using two Integer variables you can probably change the type of one to be more suitable, again making for an easy fix.
Double y = 5; // Integer y = 5;
Double x = 3 * (y / 2);
Last but not least, if you absolutely have to keep the types as they are, you'll be pleased to know that there is yet another easy fix: just use an explicit cast.
Integer y = 5;
Integer z = 2;
Double x = 3 * (y / (Double)z);
Wrap Up
To summarise, if you ever find that the result of a calculation is not as you expect, and you know the expression and input are correct, then check to see what data types are involved. An extra two key presses could be all you need to preserve your sanity.