- Oct 19, 2017
- 2
- 0
- 510
Hi everyone,
I need to find the best storage solution for my PhD experimental set up, I don’t have much experience in storage so I’m seeking help here (question asked on other forums of similar topics).
My set up consists of 8 webcams (normal Logitech webcams) connected to a Linux computer and continuously recording videos. I need to store these videos and eventually move them to another, more powerful machine for analysis. Each webcam has an output of around 300Mb/minute, so a computer connected to 8 webcams fills around 3.5 Tb per day. Experiments are probably going to run for several days in a raw, no stop. This setup is scalable, and I’m aiming to get to 4 setups composed with by a computer and 8 webcams, for a total of 32 cameras recording simultaneously. After the analysis I don’t need to keep all the raw videos, although I will probably keep few just in case. As running the analysis takes longer time than recording the videos, I need a place to store videos as they are continuously recorded while they are waiting to be analysed. How would you create this storage set-up, and with which products? I don’t have unlimited resources and need to keep an eye on my expenses.
First solution coming to my mind is to scale down the project, recording only for a limited amount of hours per day. Although I’m still considering this, having continuous videos would bring to better data. I want to give it a try and record continuously for several days.
Best way to do this would be to use computers with normal storage for the recording and save the data on external drives. As I need to record eight videos per time I need fast-spinning disks, at least 7200 RPM. Best size would be probably 4Tb, each day I can just plug a new drive and physically bring the full one to the analysis. Problem: I would need a lot of drives, one for each day of experiment.
Another possibility would be to increase enormously the storage of the analysis machine, create a network with the recording computers and save videos directly on my analyses computer. PRO: I don’t need drives, and I don’t need to physically move data around (and maybe drop the disks). Videos are already saved there where they are needed. CON: I don’t know how to build or maintain such a big and expensive machine. If I will need extra storage, I can’t easily do it. The computer would fill up and slow down very quickly. Also, I don’t think economically is better than just buy drives.
I could use cloud, but I’m worried about the uploading time. If anything goes wrong, connection falls or is just slow down the system crash and I lose recording data just in the middle of the experiment. Also, I will still need to download my data on a physical storage when I need to analyse them, which again is just probably going to be drives.
My best guess is to buy external drives, 7200 RPM and 4Tb of storage. But I’m totally open to hear different, original solutions I haven’t thought about. Someone also suggested me to buy cheaper internal disks instead, and use them on a dock. I don’t see this huge difference in price, and i wouldn’t feel comfortable working and handling “naked” disks while I could use external drives.
If you have other ideas, I’m open to listen to them! If you don’t and think i should just buy a bunch of drives, which one do you suggest? The one I’m thinking about is this one:
Buffalo HD-LX4.0TU3-EU 4TB DriveStation Velocity USB 3.0 7200RPM Desktop HDD
http://www.buffalo-technology.com/en/products/storage-devices/external-storage/drivestationtm-external-hard-drives/drivestation-velocity/
Which I could get easily on Amazon.uk for less than 150Euro (less than 180 dollars)
What do you think about this product? Do you have other to suggest?
Thank you,
Giacomo
I need to find the best storage solution for my PhD experimental set up, I don’t have much experience in storage so I’m seeking help here (question asked on other forums of similar topics).
My set up consists of 8 webcams (normal Logitech webcams) connected to a Linux computer and continuously recording videos. I need to store these videos and eventually move them to another, more powerful machine for analysis. Each webcam has an output of around 300Mb/minute, so a computer connected to 8 webcams fills around 3.5 Tb per day. Experiments are probably going to run for several days in a raw, no stop. This setup is scalable, and I’m aiming to get to 4 setups composed with by a computer and 8 webcams, for a total of 32 cameras recording simultaneously. After the analysis I don’t need to keep all the raw videos, although I will probably keep few just in case. As running the analysis takes longer time than recording the videos, I need a place to store videos as they are continuously recorded while they are waiting to be analysed. How would you create this storage set-up, and with which products? I don’t have unlimited resources and need to keep an eye on my expenses.
First solution coming to my mind is to scale down the project, recording only for a limited amount of hours per day. Although I’m still considering this, having continuous videos would bring to better data. I want to give it a try and record continuously for several days.
Best way to do this would be to use computers with normal storage for the recording and save the data on external drives. As I need to record eight videos per time I need fast-spinning disks, at least 7200 RPM. Best size would be probably 4Tb, each day I can just plug a new drive and physically bring the full one to the analysis. Problem: I would need a lot of drives, one for each day of experiment.
Another possibility would be to increase enormously the storage of the analysis machine, create a network with the recording computers and save videos directly on my analyses computer. PRO: I don’t need drives, and I don’t need to physically move data around (and maybe drop the disks). Videos are already saved there where they are needed. CON: I don’t know how to build or maintain such a big and expensive machine. If I will need extra storage, I can’t easily do it. The computer would fill up and slow down very quickly. Also, I don’t think economically is better than just buy drives.
I could use cloud, but I’m worried about the uploading time. If anything goes wrong, connection falls or is just slow down the system crash and I lose recording data just in the middle of the experiment. Also, I will still need to download my data on a physical storage when I need to analyse them, which again is just probably going to be drives.
My best guess is to buy external drives, 7200 RPM and 4Tb of storage. But I’m totally open to hear different, original solutions I haven’t thought about. Someone also suggested me to buy cheaper internal disks instead, and use them on a dock. I don’t see this huge difference in price, and i wouldn’t feel comfortable working and handling “naked” disks while I could use external drives.
If you have other ideas, I’m open to listen to them! If you don’t and think i should just buy a bunch of drives, which one do you suggest? The one I’m thinking about is this one:
Buffalo HD-LX4.0TU3-EU 4TB DriveStation Velocity USB 3.0 7200RPM Desktop HDD
http://www.buffalo-technology.com/en/products/storage-devices/external-storage/drivestationtm-external-hard-drives/drivestation-velocity/
Which I could get easily on Amazon.uk for less than 150Euro (less than 180 dollars)
What do you think about this product? Do you have other to suggest?
Thank you,
Giacomo
Facebook
Twitter
Instagram