{"id":2068,"date":"2019-02-18T07:00:33","date_gmt":"2019-02-18T12:00:33","guid":{"rendered":"https:\/\/langa.com\/?p=2068"},"modified":"2019-02-18T07:44:29","modified_gmt":"2019-02-18T12:44:29","slug":"a-reader-asks-how-does-youtube-add-disk-space-quickly-enough-to-keep-up-with-the-constant-stream-of-data-being-added","status":"publish","type":"post","link":"https:\/\/langa.com\/index.php\/2019\/02\/18\/a-reader-asks-how-does-youtube-add-disk-space-quickly-enough-to-keep-up-with-the-constant-stream-of-data-being-added\/","title":{"rendered":"A reader asks: “How does YouTube add disk space quickly enough to keep up with the constant stream of data being added?”"},"content":{"rendered":"\n

Some estimates say YouTube adds something north of a petabyte<\/a> of new data every day.<\/p>\n\n\n\n

That’s impressive in itself, but remember that YouTube is owned by Google, whose servers have to store data for: Google Search; Google Docs\/Sheets\/Slides; Gmail\/Inbox; Google Calendar; Google Drive; Google Translate; Google Maps; Waze; Google Earth; Street View; Google Keep; Google Photos; the Google Chrome web browser’s associated sites; the Chrome OS’s sites; Nexus device sites; Pixel smartphone sites; some Android sites; Google Home smart speaker sites; Google Wifi mesh wireless router sites; and — believe it or not, more. (Google info<\/a>).<\/p>\n\n\n\n

And don’t forget the information and storage needs of Google’s parent company, Alphabet<\/a>, along with Google’s sibling companies (Calico, Chronicle, GV, CapitalG, Verily, Waymo, X, Loon, Google Fiber….)<\/p>\n\n\n\n

It’s a mind-boggling amount of data.<\/p>\n\n\n\n

So, how does Alphabet\/Google add capacity to keep up?<\/p>\n\n\n\n

Alphabet has 16 huge data centers<\/a> around the world, and many smaller operations for local access and storage.<\/p>\n\n\n\n

\"\"
Google server racks in Oregon; image from this Google <\/strong><\/em>corporate photo gallery<\/strong><\/em><\/a>.<\/strong><\/em><\/figcaption><\/figure>\n\n\n\n

Many of the Google data centers are, um, rather large. One of the larger, in Pryor Creek, Oklahoma, covers a total of almost a million sf (92,000 sm)!<\/p>\n\n\n\n

\"\"
Areal view of Google’s Oklahoma data center campus.<\/strong><\/em><\/figcaption><\/figure>\n\n\n\n

Of course, filling a warehouse-sized building with myriad close-packed servers and drives takes a vast amount of energy (which, to its credit, Google says is 100% renewably-sourced<\/a>); and produces a huge amount of waste heat.<\/p>\n\n\n\n

To give an idea of scale, here’s a photo of just a portion of the cooling-system plumbing needed to keep Google’s myriad servers from melting themselves at its The Dalles, Oregon, data center<\/msreadoutspan>.<\/msreadoutspan><\/p>\n\n\n\n

\"\"
Handling waste heat from myriad servers is a huge task…<\/strong><\/em><\/figcaption><\/figure><\/div>\n\n\n\n

Some of its centers have so many drives in operation that humans can’t readily keep up: Alphabet\/Google actually uses industrial robots to handle the constant task of pulling dead\/decommissioned drives out of server racks, and replacing them with new ones. The discarded drives are then physically shredded, to prevent anyone from recovering whatever they contain<\/msreadoutspan>. (See: <\/msreadoutspan>Robots Now Annihilate Hard Drives in Google Data <\/msreadoutspan>Centers<\/a>.)<\/p>\n\n\n\n

On scales like that, adding storage isn\u2019t done on a drive by drive basis; it\u2019s usually done by adding whole racks of hardware at a time.<\/p>\n\n\n\n

As each new, huge rack-full of available servers and storage comes online, it’s then made available, as needed, across the Alphabet\/Google internal network.<\/p>\n\n\n\n

Only Alphabet\/Google knows for sure how much total storage it has and how fast it’s growing. But in 2013, Randall Munroe<\/a> calculated that Google then had about 15 exabytes of data on tap — which, in true XKCD-style<\/a>, he calculated could be converted to a stack of 1960s-era computer punchcards about 1.7 miles (4.5km) high. \ud83d\ude42<\/p>\n\n\n\n

And that was in 2013! It’s gotta be way, way more than that, now!<\/p>\n\n\n\n

More info:<\/p>\n\n\n\n