A reader asks: “How does YouTube add disk space quickly enough to keep up with the constant stream of data being added?”

Some estimates say YouTube adds something north of a petabyte of new data every day.

That’s impressive in itself, but remember that YouTube is owned by Google, whose servers have to store data for: Google Search; Google Docs/Sheets/Slides; Gmail/Inbox; Google Calendar; Google Drive; Google Translate; Google Maps; Waze; Google Earth; Street View; Google Keep; Google Photos; the Google Chrome web browser’s associated sites; the Chrome OS’s sites; Nexus device sites; Pixel smartphone sites; some Android sites; Google Home smart speaker sites; Google Wifi mesh wireless router sites; and — believe it or not, more. (Google info).

And don’t forget the information and storage needs of Google’s parent company, Alphabet, along with Google’s sibling companies (Calico, Chronicle, GV, CapitalG, Verily, Waymo, X, Loon, Google Fiber….)

It’s a mind-boggling amount of data.

So, how does Alphabet/Google add capacity to keep up?

Alphabet has 16 huge data centers around the world, and many smaller operations for local access and storage.

Google server racks in Oregon; image from this Google corporate photo gallery.

Many of the Google data centers are, um, rather large. One of the larger, in Pryor Creek, Oklahoma, covers a total of almost a million sf (92,000 sm)!

Areal view of Google’s Oklahoma data center campus.

Of course, filling a warehouse-sized building with myriad close-packed servers and drives takes a vast amount of energy (which, to its credit, Google says is 100% renewably-sourced); and produces a huge amount of waste heat.

To give an idea of scale, here’s a photo of just a portion of the cooling-system plumbing needed to keep Google’s myriad servers from melting themselves at its The Dalles, Oregon, data center.

Handling waste heat from myriad servers is a huge task…

Some of its centers have so many drives in operation that humans can’t readily keep up: Alphabet/Google actually uses industrial robots to handle the constant task of pulling dead/decommissioned drives out of server racks, and replacing them with new ones. The discarded drives are then physically shredded, to prevent anyone from recovering whatever they contain. (See: Robots Now Annihilate Hard Drives in Google Data Centers.)

On scales like that, adding storage isn’t done on a drive by drive basis; it’s usually done by adding whole racks of hardware at a time.

As each new, huge rack-full of available servers and storage comes online, it’s then made available, as needed, across the Alphabet/Google internal network.

Only Alphabet/Google knows for sure how much total storage it has and how fast it’s growing. But in 2013, Randall Munroe calculated that Google then had about 15 exabytes of data on tap — which, in true XKCD-style, he calculated could be converted to a stack of 1960s-era computer punchcards about 1.7 miles (4.5km) high. 🙂

And that was in 2013! It’s gotta be way, way more than that, now!

More info:

Permalink: https://langa.com/?p=2068

[seperator]

COMMENT / QUESTION on THIS ITEM? See the Comment box at bottom of this page!

NEW QUESTION?
Ask here!

(Want free notification of new content? Click here!)

1 Reply to “A reader asks: “How does YouTube add disk space quickly enough to keep up with the constant stream of data being added?””

Comment? Question? Reply...?