advice needed on large SAS SATA array

[disgust]

I'm looking to build a server to basically be a ton of space for some web projects of mine.

assumptions I'm making but I'm totally not sure of, based on my knowledge of SAS/SATA:

1) I can set up a very large number of disks in single array
2) one card (albeit a high quality card) can accomplish this
3) I can connect the SAS network to multiple computers, correct?

things I'm totally not sure of:

is there a limit to the number of disks that should realistically run into one server via one card before it'd max out?

if I build a RAID 5 array, am I able to add disks to the array later to increase storage? ie, if it's initially 4x500GB drives, can I add a 5th or a 6th without rebuilding the entire array?

is there any problem with "super large" partitions working in operating systems? (ie, an array that's terrabytes large) are there ways to avoid this?

is there any limit to the consistant throughput a computer can handle assuming the disks can push it? I know I won't hit pci-e 4x's limit, but can I really expect one computer to handle sustained throughput of a few hundred megabytes per second, assuming the disks can perform that well? are there any other sorts of limits I should be aware of?

I'm a bit confused when it comes to hardware. http://www.newegg.com/Product/Product.asp?Item=N82E16816116035 for example (just picked a random card) says it supports "4 Multi-lane connectors Support up to 16 Serial ATA II drives"

does that mean there are 4 connections for SATA drives, but I need to use port expanders to actually get 16 disks? or are just the "SFF-8484" 1 -> 4 SATA adapter cables used?

what if hypothetically I wanted more than 16 disks? not neccessarily in a RAID array together, but still in the SAS networt

what hardware is required if I want another computer to be able to have access to the SAS/SATA network exactly?

 

[31computers]

There are some excellent links to RAID sites in this Forum. Just do a search (or Google, even)

 

[sandmanwn]

is there a limit to the number of disks that should realistically run into one server via one card before it'd max out?

This all depends on what card you get and what sort of slot you are planning to put it in (PCIe x4, PCIe x8, PCI-X)

if I build a RAID 5 array, am I able to add disks to the array later to increase storage? ie, if it's initially 4x500GB drives, can I add a 5th or a 6th without rebuilding the entire array?

Look for a drive that supports Online Capacity Expansion (OCE). This is the feature that allows you to expand arrays.

is there any problem with "super large" partitions working in operating systems? (ie, an array that's terrabytes large) are there ways to avoid this?

I believe the windows limit is 2 terabytes but dont hold me to that. You generally have to partition into lesser or equal partitions to 2TB

is there any limit to the consistant throughput a computer can handle assuming the disks can push it? I know I won't hit pci-e 4x's limit, but can I really expect one computer to handle sustained throughput of a few hundred megabytes per second, assuming the disks can perform that well? are there any other sorts of limits I should be aware of?

Depends on where the data is going. May have a bottleneck at the network card if the IO is gets too high. May need to team some copper gigabit network card together or invest in fiber optics.

I'm a bit confused when it comes to hardware. http://www.newegg.com/Product/Product.asp?Item=N82E16816116035 for example (just picked a random card) says it supports "4 Multi-lane connectors Support up to 16 Serial ATA II drives"

does that mean there are 4 connections for SATA drives, but I need to use port expanders to actually get 16 disks? or are just the "SFF-8484" 1 -> 4 SATA adapter cables used?

I believe its one cable that has a specialty end that connects to the card and on the other side where you plug the cable into the hard drive it splits into several regular SATA connectors. Looks like this http://images10.newegg.com/NeweggImage/productimage/16-116-035-06.jpg

what hardware is required if I want another computer to be able to have access to the SAS/SATA network exactly?

Not sure what you mean by this. Would depend on what type of setup you are using. There are a few ways to go about this.

If its a NAS box then it will probably be connected directly to an external port on another server.

If its a standalone server then iSCSI or traditional networking would do the trick.

 

[disgust]

There are some excellent links to RAID sites in this Forum. Just do a search (or Google, even)

I've read a lot about SAS/RAID, but it's a newish technology and most of the talk is theoretical... and not to mention really confusing. wondering what's actually possible with the hardware.

This all depends on what card you get and what sort of slot you are planning to put it in (PCIe x4, PCIe x8, PCI-X)

I was mainly wondering how the consistant throughput, pcie limits aside, would affect a machine, and if there's any sort of max I really shouldn't go over.

Not sure what you mean by this. Would depend on what type of setup you are using. There are a few ways to go about this.

I was under the impression the whole point in the SAS external adapters was to allow one server to hook up to another and natively share its drives. if that's not the case, what is that for, exactly?

other questions:

1) if the box is both a web server (for the files) and storage, should the OS & webserver portion be on a seperate disk array from the storage array? especially since it'd be raid5 and that'd slow writes down, not to mention it'd naturally load balance things. how do people typically do it? one array for the OS, one for storage?

2) what's "64bit" mean in terms of SAS cards?

3) I see the max number of drives supported listed on some products but not others. how do I find out for sure? is it the same as the number of ports? if not, what do the number of ports mean?

 

[qwertycopter]

http://www.infostor.com/sas/sas_home.cfm
http://www.scsita.org/aboutscsi/sas/tutorials/SAS_General_overview_public.pdf
http://www.t10.org/ftp/t10/drafts/sas2/sas2r05a.pdf

 

[SomeJoe7777]

Let's get the terminology correct here first.

You're using the abbreviation SAS, which is Serial Attached SCSI. This is a modernization of the SCSI transport, which connects hard disks to controllers using a standard SATA cable (it does not run the SATA protocol across the cable though, it runs SCSI).

However, the way you're speaking, I believe what you're actually interested in is a SAN, a Storage Area Network. This is a method to consolidate storage (large amounts of storage) in a single enclosure, and have multiple servers able to use that pooled storage.

So, let's look at a couple of example SAN units:

Dell/EMC AX150i
Promise VTrak M500i

Your server communicates with a SAN unit over a network. There are two types of storage network transports available, Fibre Channel and iSCSI. Fibre Channel uses optical interconnects, operates faster, and requires expensive host bus adapters and switches. iSCSI is less expensive, uses standard Gigabit Ethernet switches and cables, and can use either a standard network adapter in the server (slower) or a dedicated iSCSI host bus adapter (faster).

The SAN unit itself usually has space for several drives (SCSI or SATA depending on the SAN model), which can run one or more RAID arrays. On the RAID arrays, one or more logical drives (volumes) can be created. You can then map each server that is attaching to the unit over the storage area network to a particular logical drive/volume. Note, the mapping is one-to-one. i.e. Each logical volume can be seen and mounted by only one server. The SAN unit does not arbitrate access to a single logical volume by more than one server - it is not a sharing device. Each server that attaches to the unit over the SAN must mount and use it's own logical volume that is carved out of the pool of available storage space.

However, you can insert more drives into the SAN, extend the array(s), extend the logical volume(s), and therefore add space to any server that needs it.

I have one of the Promise VTrak M500i units I linked to above. It works OK, but in retrospect I wish I had bought the Dell unit. It can take up to 15 SATA drives. I have 10 750GB Seagate Barracuda ES drives in my unit, one is a hot spare, the remaining 9 are in a RAID 5 config, for a total of 6TB of usable storage. I made a 6TB logical drive on the unit, mapped a server to it, the drive then showed up as a 6TB unpartitioned drive.

To make a partition >2TB on a drive, you must be using Windows Server 2003 and initialize the drive as a GPT disk, not an MBR disk. GPT disks can exceed 2TB. Then make an NTFS partition on the drive.

Throughput is a consideration. Fibre channel SANs can get you 200MB/sec throughput. iSCSI can't get near that fast. iSCSI, using jumbo frames and an iSCSI host adapter can get close to the Gigabit Ethernet maximum (around 100MB/sec or so). Using a standard network card, no jumbo frames, and no TCP offload will only get about 30MB/sec. My Promise VTrak unit doesn't support jumbo frames, but my network adapter has TCP offload, and I'm getting around 45MB/sec, which is fast enough for what I need to do.

I have also tested array expansion with my unit. I originally set it up with 6 drives in a RAID 5, made the logical volume, mapped the server, formatted & partitioned a GPT disk. I then added 4 drives to the unit, did a RAID migration to increase the array space, increased the logical volume space, and then used the DISKPART utility on Windows Server 2003 to expand the partition. All operations were successful, but not fast. It took the unit 3 days to expand the array. 8O

Well, that's a short primer on SANs. Post any additional questions you have.

 

[disgust]

if SAS can't connect to other boxes, why do they have the external SAS adapters? not getting what that's for really

still curious about these, too:

1) if the box is both a web server (for the files) and storage, should the OS & webserver portion be on a seperate disk array from the storage array? especially since it'd be raid5 and that'd slow writes down, not to mention it'd naturally load balance things. how do people typically do it? one array for the OS, one for storage?

2) what's "64bit" mean in terms of SAS cards?

3) I see the max number of drives supported listed on some products but not others. how do I find out for sure? is it the same as the number of ports? if not, what do the number of ports mean?

also, can modern nix distros deal w/ > 2TB partitions?

 

[jt001]

1) if the box is both a web server (for the files) and storage, should the OS & webserver portion be on a seperate disk array from the storage array? especially since it'd be raid5 and that'd slow writes down, not to mention it'd naturally load balance things. how do people typically do it? one array for the OS, one for storage?

I would say a single sata disk should be adequate for the OS

2) what's "64bit" mean in terms of SAS cards?

I think what you're referring to is the interface there, 64bit would use a PCI-X slot. But you're talking about a PCI-e card so that's not really applicable.

3) I see the max number of drives supported listed on some products but not others. how do I find out for sure? is it the same as the number of ports? if not, what do the number of ports mean?

Usually, however some products will use a few proprietary connectors that use cables that breakout into several SATA connectors.


As far as other bottlenecks I don't see any, it has to go over the same bus as the RAM to get to the CPU but we're talking a 6.4GB/s - 12.8GB/s bus compared to a couple hundred MB/s so that's not really an issue.

 

[disgust]

thanks, that got most of my questions.

I'm still confused as to why the externa SAS adapters can't function in a SAN-like way, though?

 

[PCcashCow]

You probably have designed your system, but another take on a your trying to build would to be to set up a base W2k3 (XP would work too) with small SAS drives.
Then I would suggest you build a SATA or SCSI SAN with an open source app. Use an open source os tha enables you to create volumes, then you can use iSCSI to mount volumes that your projects need to live on. Toss a gig switch in the design and your have transparent storage that is unlimited.

 

[disgust]

haven't designed it yet. it'll be a nix system.

I am still utterly confused as to why SAS has external adapters if it's not to connect to other boxes in a SAN-like way