1. Computing & Technology

Discuss in my forum

Definition of Precision

By , About.com Guide

Definition: The precision of a variable is how many digits it can store accurately. A float can hold only 7 whereas a double can hold 15.

Note this is all the digits in a number not just after a decimal point. So 786.5432987 will not be stored accurately in a float, but will be in a double as it contains 10 digits.

Glossary:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
Examples:
In financial calculations, Japanese Yen can be counted in the trilions so need to be stored in doubles.

©2012 About.com. All rights reserved.

A part of The New York Times Company.