How Computers Actually Calculate Square Roots

A clay tablet at Yale University, known as YBC 7289, shows a square with its diagonal marked. Written in Babylonian base-60 numerals is an approximation of √2: 1.41421296… The true value is 1.41421356… That's accurate to about six decimal places, recorded roughly 3,700 years ago.

The method the Babylonians likely used is, in essence, still how software computes square roots today. It's so efficient that each step roughly doubles the number of correct digits.

A Bad Guess Converges Almost Instantly

You'd expect that computing √2 to 15 digits would take many steps, especially starting from a rough guess like 1. It doesn't. From a guess of 1, it takes only 5 steps to reach full double-precision accuracy.

The reason is quadratic convergence: the error at each step is roughly proportional to the square of the previous error. An error of 0.01 becomes about 0.0001, then 0.00000001.

The Babylonian (Heron's) Method

To find √S:

1. Make a guess x
2. Compute S/x
3. Average them: x_new = (x + S/x) / 2
4. Repeat

Why it works: if x is too big, then S/x is too small, and vice versa. The true root always lies between them, so their average is a better guess. The method is often called Heron's method, after Heron of Alexandria, who described it in the 1st century AD.

Example: √2, starting from x = 1

By step 5 it matches √2 to the full precision of a 64-bit double. Compare with the square roots list.

It's Newton's Method in Disguise

Newton's method finds a root of any function f by repeatedly following the tangent line:

x_new = x − f(x) / f′(x)

To find √S, solve f(x) = x² − S = 0. Since f′(x) = 2x:

x_new = x − (x² − S) / 2x = (x + S/x) / 2

That's exactly the Babylonian formula. The ancient averaging trick is a special case of a calculus method published nearly two millennia later. See the derivative rules on the calculus formulas page.

Getting a Good Starting Guess

Computers don't start from 1. Floating-point numbers are stored as significand × 2^exponent, so:

√(m × 2^(2k)) = √m × 2^k

Halving the exponent gives an instant estimate within a small factor. Often a small lookup table refines it further. With a guess accurate to a few bits, just two or three iterations reach full precision.

How Hardware Does It: Digit by Digit

Modern CPUs compute square roots directly in silicon, and the IEEE 754 floating-point standard requires the result to be correctly rounded. Instead of Newton's method, chips usually use digit-by-digit algorithms, similar to long division.

In binary, the idea is simple. Build the root one bit at a time, from the most significant bit down. At each step, tentatively set the next bit to 1, and keep it only if the square of the partial root doesn't exceed the number. Engineers use tricks like SRT algorithms, which share circuitry with division, to produce several bits per step.

Each step produces a fixed number of correct bits, which makes timing predictable and exact rounding easy. You can explore binary digits with the binary to decimal converter.

An Insider Reference: The Fast Inverse Square Root

Games need 1/√x constantly to normalize vectors for lighting. In 1999, id Software's Quake III Arena included an approximation that became legendary when the source code was released in 2005:

float Q_rsqrt(float number) {
    long i;
    float x2, y;
    const float threehalfs = 1.5F;
    x2 = number * 0.5F;
    y  = number;
    i  = * ( long * ) &y;
    i  = 0x5f3759df - ( i >> 1 );
    y  = * ( float * ) &i;
    y  = y * ( threehalfs - ( x2 * y * y ) );
    return y;
}

The bit manipulation produces a surprisingly good first guess by treating the float's bits as an integer, which effectively halves its logarithm. The last line is one step of Newton's method for 1/√x. It was often attributed to John Carmack, but he has said he didn't write it. Its lineage traces back through earlier programmers at Silicon Graphics and elsewhere. Today, CPUs provide dedicated instructions that make the trick unnecessary, but it remains a favorite example of mathematical cleverness.

Two Concepts Worth Knowing

Quadratic Convergence

An iterative method has quadratic convergence if each step squares the error. In practice, the number of correct digits roughly doubles each iteration, which is why Newton's method is so fast near the answer.

Fixed-Point Iteration

A fixed point of a function g satisfies g(x) = x. The Babylonian method repeatedly applies g(x) = (x + S/x)/2, whose fixed point is exactly √S.

Quick Answer: How Do Computers Calculate Square Roots?

Software often uses Newton's method, also known as the Babylonian or Heron's method: repeat x ← (x + S/x)/2, which roughly doubles the correct digits every step. Processors typically compute square roots in hardware using digit-by-digit algorithms similar to long division, which give correctly rounded results.

Try Them Yourself

• Square Roots List: check your iterations against exact values
• Square Numbers List: pick a good starting guess
• Scientific Calculator: run the Babylonian method step by step
• Calculus Formulas: Newton's method from derivatives
• Binary to Decimal Converter: how hardware sees the bits
• The Math Behind 3D Games: where fast square roots matter

Compute √10 with the Babylonian method starting from 3. Count how many steps it takes to get 10 correct digits. (Spoiler: not many.)