David Kirkpatrick

June 25, 2008

The petabyte age

Filed under: Science, Technology — Tags: , , , , — David Kirkpatrick @ 12:25 pm

From KurzweilAI.net, Wired’s Chris Anderson has an interesting piece on the “petabyte age,” Google and the business of doing science.

The Petabyte Age: Because More Isn’t Just More — More Is Different
Wired, June 23, 2008

The new availability of huge amounts of data, along with the statistical tools to crunch these numbers, offers a whole new way of understanding the world, suggests Wired editor in chief Chris Anderson.

Science can advance even without coherent models and unified theories, letting statistical algorithms find patterns where science cannot.
Read Original Article>>


From the original article:

The Petabyte Age is different because more is different. Kilobytes were stored on floppy disks. Megabytes were stored on hard disks. Terabytes were stored in disk arrays. Petabytes are stored in the cloud. As we moved along that progression, we went from the folder analogy to the file cabinet analogy to the library analogy to — well, at petabytes we ran out of organizational analogies.

At the petabyte scale, information is not a matter of simple three- and four-dimensional taxonomy and order but of dimensionally agnostic statistics. It calls for an entirely different approach, one that requires us to lose the tether of data as something that can be visualized in its totality. It forces us to view data mathematically first and establish a context for it later. For instance, Google conquered the advertising world with nothing more than applied mathematics. It didn’t pretend to know anything about the culture and conventions of advertising — it just assumed that better data, with better analytical tools, would win the day. And Google was right