RSS FEED

Comparing float values

Dealing with floating point values in MAXScript is almost inevitable, as are the accuracy problems connected to that.

Floating point numbers

Most of the floats you will deal with in MAXScript are single precision floats, be it controller values, point3 values of normals and positions etc. When you are free to do the calculations the way you want, it more sense to use for example doubles instead, but not in the cases mentioned above. Meet IEEE-754, by its definition, this format has 6 to 9 significant decimal digits precision, and addition is not necessarily associative which can lead to all sorts of interesting errors (but arguably that would be outside the scope of this topic – if you are interested in learning more, refer to What Every Programmer Should Know About Floating-Point Arithmetic). Suffice it to say that one should always be careful when mixing small and large numbers:

12345.0 + 0.54321 == 12345.0 + 0.543 --> true

Nearly every MAXScript coder (or rather any coder using single precision floats) has been bitten by something like that before, and there are many ways how to cimcurvent it if all you need is testing the values for equality. The easiest way is getting the difference of the two terms, if that is smaller than some deliberately chosen epsilon value, we treat them as equal values:

fn areFloatsEquivalent f1 f2 eps:1e-6 =
    f1 == f2 OR abs (f1 - f2) <= eps

The advantage of this method is that you can manually set the epsilon to whatever value suits you at the moment and it tends to be good enough when you roughly know what order of magnitude are the values you are comparing. With large numbers, however, the epsilon could actually be smaller than the rounding error, and the comparison would always returns false. As this is quite a common issue, there is a built-in function that solves this, namely close_enough – use it whenever it makes sense. If you are curious about the implementation, written in maxscript it would be:

fn close_enough_mxs f1 f2 rounds eps:1.19209e-7 =
    f1 == f2 OR 2.0 * abs(f1 - f2)/(abs f1 + abs f2) < rounds * eps

The value 1.19209e-7 used here is the delta of 1.0 and the next distinguishable float value. If you want to know the value for floats in a different range than around the 1.0, there is an easy way to find out. Under Customize menu item, System Unit Setup pop-up of the Unit Setup dialog shows an interactive slider where you can see the value in a given range:

System Unit Setup :: Float Precision

If you are curious as to how to get these values in a script, here is a function for that:

fn floatAccuracy nr =
    2.^(int(log nr / log 2) - 23)

So if all you really wanted to know was if two values are interchangeable (which is certainly not helpful in cases where the errors add up), a function testing it could make use of it:

fn areFloatsEquivalent f1 f2 =
(
    if f1 < f2 do swap f1 f2
    (f1 - f2) <= floatAccuracy f1
)
Other types based on 32-bit floats

Point2, Point3, Point4

The most straightforward way here is using the distance function:

fn arePtsEquivalent v1 v2 eps:1e-6 =
    distance p1 p2 < eps

Of course, you can also use getHashValue, which can be used this way in many other circumstances (not for comparing floats directly, though, the error margin is too large for that to work on small values):

fn arePtsEquivalent v1 v2 eps:1e-6 =
    getHashValue p1 0 == getHashValue p2 0

This works well with values describing e.g. position, and if the aim is to compare non-unit vectors based on their direction, it would work the same if you normalized both of them prior to passing them to the function. Another possibility is to compare the angle between them with a custom threshold angle:

fn areVectorsEquivalent v1 v2 eps:0.999999 =
    dot (normalize v1) (normalize v2) > eps

To directly compare the angles, acos would be needed, however as we treat the threshold angle as a constant, it makes more sense to instead use cos(thresholdAngle) as our epsilon value to save some computation cycles (and also avoid indeterminate result due to inaccuracies). Note that this also switches the greater-than/less-than operators in the comparison.

Colors

The easiest way is to cast the colors to Point3 format and compare those values, which can be simplified to:

fn areColorsEquivalent c1 c2 eps:1e-6 =
    length ((c1 - c2) as point3) < eps

Matrices

Similar to other values, casting to string works quite good for matrix comparison, the only thing to bear in mind is that MAXScript prints the value of a float to the 6th significant digit, which effectivelly truncates decimals beyond that point.

fn areMatricesEquivalent m1 m2 =
    m1 as string == m2 as string

For increased precison, comparing each row as Point3 values, together with the use of short-circuiting and operator is a better idea:

fn areMatricesEquivalent m1 m2 =
    areVectorsEquivalent m1.row1 m2.row1 AND
    areVectorsEquivalent m1.row2 m2.row2 AND
    areVectorsEquivalent m1.row3 m2.row3 AND
    areVectorsEquivalent m1.row4 m2.row4

Another method, mentioned in MAXScript reference, makes use of the fact that matrix multiplied by its inverse returns the identity matrix:

fn areMatricesEquivalent m1 m2 =
    isIdentity (m1 * inverse m2)

DISCLAIMER: All scripts and snippets are provided as is under Creative Commons Zero (public domain, no restrictions) license. The author and this blog cannot be held liable for any loss caused as a result of inaccuracy or error within these web pages. Use at your own risk.

This Post needs Your Comment!

Anonymous

Could they not use the 16 bit precision that Autocad has?

Swordslayer

Anonymous: Sure, nowadays there'd be very few reasons not to but unless there will be a complete rewrite of the core I doubt it will ever change.

Return to top