I have setup a satellite data feed at my house. My current receiver system has 1 sata 320gb drive in it. It is not able to keep up. I am looking for the best types of hardware to for this situation.
The data comes in via a network multicast off of a satellite receiver. Average speeds are around 500-700kB/sec with peak speeds exceeding 1.2mB/sec, sometimes for around 10min. This data is broken down into files, averaging between 2000 and 4000 files per minute. These files range in size from less than 1kB to > 400MB depending on the product. The problem comes during the processing of this stream. Each file is written into a spool directory and the file type and name given to various decoding programs. They each read the file, process it, then write it back to the drive in another directory. Each file goes through at least one decoder but can be passed to as many as 4 simultaneously. Then those files are read from across the network by various programs that we use.
I'm not sure what the best way of dealing with this will be. It also doesn't help that don't make much money so I also have to keep this thing cheap considering how much I've already spent on this setup. I've been looking at sata II, scsi and various raid setups but can't really find any good information on which direction to take with this problem.
The data comes in via a network multicast off of a satellite receiver. Average speeds are around 500-700kB/sec with peak speeds exceeding 1.2mB/sec, sometimes for around 10min. This data is broken down into files, averaging between 2000 and 4000 files per minute. These files range in size from less than 1kB to > 400MB depending on the product. The problem comes during the processing of this stream. Each file is written into a spool directory and the file type and name given to various decoding programs. They each read the file, process it, then write it back to the drive in another directory. Each file goes through at least one decoder but can be passed to as many as 4 simultaneously. Then those files are read from across the network by various programs that we use.
I'm not sure what the best way of dealing with this will be. It also doesn't help that don't make much money so I also have to keep this thing cheap considering how much I've already spent on this setup. I've been looking at sata II, scsi and various raid setups but can't really find any good information on which direction to take with this problem.