next up previous
Next: How accurate is enough Up: Expression evaluation with mixed Previous: Target precision

Treatment of domains

Widening not only affects the notion of precision, but also, in a different sense, of domains.

What should sqrt(-1) be? distinguishing between sqrt(x:real) from sqrt(z : complex) means that sqrt(-1) is an error, but sqrt( -1+0i) is i.

IEEE 754 is not prescriptive with respect to all issues of numerical interest(most esp. wrt complex numbers, but also transcendental functions, etc.). Even within its scope it does not prevent programmers or language designers from setting strange policies.

Example of a questionable default: Common Lisp makes sqrt(-1) into a complex result, thereby invalidating compiler optimizations that deduce the type of sqrt(x) from the type of x.



Richard J. Fateman
Thu Aug 13 21:54:23 PDT 1998