What Does One Petabyte Of Storage (And $500K) Look Like?

[cangelini]

We're big fans of big technology, and with more than 1 PB of capacity, Aberdeen's Petarack is sure something to marvel at. We take a look inside and figure out what it takes to deploy 1 000 terabytes of space in an enterprise environment, reliably.

What Does One Petabyte Of Storage (And $500K) Look Like? : Read more

 

[lockhrt999]

I have always wondered about this. Thanks Chris. Keep it coming tom.

 

[Benihana]

It's difficult to imagine 1 PB in an area the size of a deck of playing cards, but I'm going to remember today when it does.

 

[haplo602]

from my point of view, this is a pretty low to mid end array

 

[clownbaby]

I remember marveling at a similar sized cabinet at the Bell Laboratories in Columbus Ohio that held a whole gigabyte. That was about twenty years ago, so I would suspect in another 20 we might be carrying PBs in our pocket.

 

[Deleted member 217926]

"Honey, I downloaded the internet!"

 

[cangelini]

[citation][nom]haplo602[/nom]from my point of view, this is a pretty low to mid end array [/citation]
What do you work on, by chance? =)

 

[razor512]

Seems like a decent setup, but the electric bill will be scary running a system like that.

But then again, the servers I work on barely use 10TB of storage.

 

[LuckyDucky7]

I want to run RAID-0 on it; I'd like to see some benchmarks on THAT.

 

[nikorr]

Can u imagine this 25 years ago? Cool : )

 

[Casper42]

[citation][nom]Razor512[/nom]Seems like a decent setup, but the electric bill will be scary running a system like that.But then again, the servers I work on barely use 10TB of storage.[/citation]

Lets see....

I work for HP and we sell a bunch of these for various uses:
http://www.hp.com/go/mds600
Holds 70 x 2TB (and soon to be 3TB) drives in a 5U foot print.
Can easily hold 6 of these in a single rack (840 TB) and possibly a bit more but you have to actually look at things like floor weight at that point.

I am working on a project right now that involves Video Surveillance and the customer bought 4 fully loaded X9720s which have 912TB Useable (After RAID6 and online spares). The full 3.6PB takes 8 racks (4 of them have 10U free but the factory always builds them a certain way).
The scary part is once all their cameras are online, if they record at the higher bitrate offered by the cameras, this 3.6PB will only hold about 60 days worth of video before it starts eating the old data.
They have the ability and long term plan to double or triple the storage.

Other uses are instead of 2TB drives you can put 70 x 600GB 15K rpm drives.
Thats the basis for the high end VDI Reference Architecture published on our web page.
Each 35 drive drawer is managed by a single blade server and converts the local disk to an iSCSI node. Then you cluster storage volumes across multiple iSCSI nodes (known as Network RAID because you are striping or mirroring across completely different nodes for maximum redundancy)

And all of these are only considered mid level storage.
The truly high end ignores density and goes for raw horsepower like the new 3Par V800.

So Yes, I agree with haplo602. Not very high end when comparing to corporate customers.

 

[Casper42]

Doh, meant to reply to cangelini not Razor.
Sorry

 

[issuemonkey]

Agreeing with Caspe42, from our point of view at NetApp, a few points here a scarier:

1/ the caching; you could have 48GB of ram or more, this is for your server between able to handle to load of your I/Os and nothing more. It requires real caching to proper handle identical request like specific files like word, excel, or even a vmdk.
With proper caching, like the one we have on our gear, you can cache a couple of vmdk used to boot storm a full stack of virtual desktop... access it once on your disks, get it from fast SSD cache

2/ looking at the pictures, the RAID adapter are not battery powered, so does it mean that there is no protection of your data during a controller lost? This is good to have a dual "server" to protect against failure but if you last writing to your DB are lost... basically, you run into trouble.
Enterprise storage is using interconnect card between the controllers with some cache, we call that NVRAM, and if one is going down, this cache is battery powered and will be accessed by the remaining node to discard the set of data o disks.

3/ they do speak about failover mechanism, this is also scary, is this automatically done in a transparent way for the different protocols?

4/ there is no concept of tiering, performance or workload type. This kind of setup will not fit for all...

Well you will understand like Casper42 said, this is quite an expensive not optimal setup!
Check out http://www.netapp.com/us/products/storage-systems/

 

[mbryans]

Is there a portable version of this Petarack?

 

[mbryans]

How about portable 1 PB RAM Disk?

 

[bernardv]

OK boys, now bench this baby against this http://www-03.ibm.com/systems/storage/network/sonas/index.html

 

[issuemonkey]

bench it against everything enterprise like...
this is 360 3TB SATA drives, this is is not for performance but pure capacity like backup or cheap disaster recovery site

 

[haplo602]

[citation][nom]cangelini[/nom]What do you work on, by chance? =)[/citation]

HP Storageworks EVAs and XPs ... also some EMC arrays ...

 

[alidan]

[citation][nom]Benihana[/nom]It's difficult to imagine 1 PB in an area the size of a deck of playing cards, but I'm going to remember today when it does.[/citation]

funny you mention that, i just did math a bit ago on the intel 14nm process.

lets assume that chip size means nothing, because i cant find chip size of ssd boards so just pcb thickness of 1.5mm

playing cards have an area of about 5645mm
and are about 15.87mm thick

in order to even cram 1/4 a pb into that area, you need to be at about a 9nm process. for a full tb, you need to be at a 4.5nm process.

now i don't know about storage, but at least with the cpu, i remember it being said that 6nm would be the limit. well, 6 or 7 but 6 stands out more.

 

[haplo602]

[citation][nom]Casper42[/nom]Lets see....I work for HP and we sell a bunch of these for various uses:http://www.hp.com/go/mds600Holds 70 x 2TB (and soon to be 3TB) drives in a 5U foot print.Can easily hold 6 of these in a single rack (840 TB) and possibly a bit more but you have to actually look at things like floor weight at that point.I am working on a project right now that involves Video Surveillance and the customer bought 4 fully loaded X9720s which have 912TB Useable (After RAID6 and online spares). The full 3.6PB takes 8 racks (4 of them have 10U free but the factory always builds them a certain way).The scary part is once all their cameras are online, if they record at the higher bitrate offered by the cameras, this 3.6PB will only hold about 60 days worth of video before it starts eating the old data.They have the ability and long term plan to double or triple the storage.Other uses are instead of 2TB drives you can put 70 x 600GB 15K rpm drives.Thats the basis for the high end VDI Reference Architecture published on our web page.Each 35 drive drawer is managed by a single blade server and converts the local disk to an iSCSI node. Then you cluster storage volumes across multiple iSCSI nodes (known as Network RAID because you are striping or mirroring across completely different nodes for maximum redundancy)And all of these are only considered mid level storage.The truly high end ignores density and goes for raw horsepower like the new 3Par V800.So Yes, I agree with haplo602. Not very high end when comparing to corporate customers.[/citation]

eh, an MDS is a bit above stupid JBOD )) they are build for capacity and basic redundancy. minimal performance.

 

[freggo]

[citation][nom]anort3[/nom]"Honey, I downloaded the internet!"[/citation]

ROFL... now I have coffee all over my keyboard. You owe me a cleanup, man

 

[willard]

[citation][nom]alidan[/nom]funny you mention that, i just did math a bit ago on the intel 14nm process. lets assume that chip size means nothing, because i cant find chip size of ssd boards so just pcb thickness of 1.5mmplaying cards have an area of about 5645mmand are about 15.87mm thickin order to even cram 1/4 a pb into that area, you need to be at about a 9nm process. for a full tb, you need to be at a 4.5nm process. now i don't know about storage, but at least with the cpu, i remember it being said that 6nm would be the limit. well, 6 or 7 but 6 stands out more.[/citation]
Petabyte SSDs are probably never happening with current technology, for exactly the reasons you state. We'd need to increase the storage amount per cell pretty dramatically, and there are already problems with MLC drives compared to SLC.

Magnetic disks, however, may well reach that capacity. There are some interesting technologies on the horizon, which I can't remember exactly what are called, but promise to greatly increase storage density on magnetic disks. Using lasers to heat up the platter before reading and writing to be able to manipulate smaller portions of the platter, for example.

Assuming Moore's Law holds for magnetic disk storage density (and I have no idea if it's even holding now), we'd need to go from 3TB to 1000TB, ignoring the fact that current drives aren't really 3TB and 1000TB isn't really a PB. That's about eight and a half cycles, for lack of a better word, of Moore's Law. With each one being 18 months long, and rounding up to 9 cycles, we'd be looking at 1PB HDDs in 2Q 2025.

There are a LOT of ifs involved, though. Honestly, I'd be surprised if magnetic disks were still in use in mainstream applications in 2025. We'll probably be eyeing the successor to SSDs by then.

 

[UltimateLord]

Can I add one of these to my Amazon's wish list?!

 

[Guest]

You can build a Petabyte storage array for less than $100000. Checkout Backblaze which is an online back up company, they have come up with their own solution. Its not built for speed but is stores a ton of data:
http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/
For the pods alone it would only cost $56000 but then their is racks, networking and man hours to build and instal them.
Also did anyone notice these are just Supermicro brand servers with a different badge on them?

Its really the backend software side that makes this system impressive but you really have to shop around with stuff like this.

 

[LOL92]

we will use racetrack developed by IBM by then =)