Question Optimising transfer of data from an SD card

[mnewmana]

I have a need to transfer a lot of data (talking about many terabytes worth of data) from SD cards to a hard disk. I am exploring the way to optimise the PC's that will be used for this task. So thinking of a modern PC (12th Gen i5) with usb3.1+. We can obviously use a SD card reader plugged into USB. Multiway SD card readers don't achieve that much extra as the bottle neck is the USB and/or the disk writes.

Would it make sense to install a further SSD and a PCIE USB card (16x slot) and have one copy go from one USB controller to one disk and another copy t the other USB controller and the other SSD. Would that pretty much double my speed or is there a bottleneck I haven't thought of?

Or does someone have any ideas about custom hardware for this sort of thing?

TIA

 

[USAFRet]

How much "terabytes"?
Is this a one time move, or on a regular basis?

Remember....overall speed depends on the slowest device in the chain.
Here, probably the SD cards. So no matter what uber PC or SSD is in the mix, the cards can only feed it so fast.

 

[mnewmana]

Yeah will be an ongoing requirement until new hardware for the data collection is developed t use IOT technologies. 10's of TB a year concentrated in a 6 month period preferably. Once data is on ssd's it is then sent cloudwards🙂

Now here are some numbers: the PC this was done on is running W11 and no there are no other user apps are running at the time. Ohh, a thought, I wonder what effect turning off AV on these files would have . Do correct my arithmetic if any of this is wrong!
So we can achieve with current kit 25GB upload in around 8 mins (I am talking "very round" numbers here) So thats is 25000 MB in 8 mins = 3125 MB in a minute = 52MB/s =(I know 10 is not 8 but I am handwaving here) 520 Mb/s = .5Gb/s. Which suggests that USB bus is not restricting things. But we have a four way USB3.1 reader and if you put a card in all four slots we take slightly longer than 8 mins per card which suggests that USB IS the bottlneck (or something in the 4 way reader) - or AV having just thought of that.

Currently we have a small M2 system SSD and a SATA SSD. Upload speeds are noticeably but not hugely faster writing to the M2 disk which is interesting as SATA running at 6Gb/s should not be a bottleneck.

I am not a h/w person so I may have misunderstood some of these numbers or be drawing the wrong conclusion.

But I am now thinking a test turning off AV is a good idea.

M

 

[USAFRet]

The difference between the PC storage types is of no consequence.
All types of SSD are 'faster' than what the SD card can feed.

4 cards at once through a single USB connection...they all have to share.

The AV on the PC will also have no consequence.

Where does this data come from?
How is it collected?

My 2x house security cameras gather about 2.1TB every 45 days. They run 24/7, copying to a volume on my NAS via ethernet.
But since a new chunk of data is automatically saved every 15 minutes, there is no real issue in copying.

 

[mnewmana]

I would agree that the SSD type should make no difference but empirically and repeatedly it would seem to make a small but measurable difference. Whats weird also is the faster M2 disk is the system disk.

Well sure the 4 cards have to share the usb link but a 3.2 link should go at a minimum of 5Gb/s whereas each card is empirically shifting at .5Gb/s.

I would expect AV to make a difference if on-access scanning is used which it is on my network.

The data is high frequency ultra sound data collected from custom devices in the field: there is no option for compression, no option for triggerring etc etc

 

[USAFRet]

What is the frequency and amount of new data that needs to be transferred.

I know you mentioned something about "a 6 month period".


But really, there is likely no magical "make it faster". Again, the read rate of the SD cards is likely the limfac.

 

[Paperdoc]

You say, "But we have a four way USB3.1 reader and if you put a card in all four slots we take slightly longer than 8 mins per card...". So the speed PER CARD is almost the same when you use FOUR cards in a multi-port USB 3.2 reader, versus one card. That verifies that the real limit is the speed of the CARD. Most of these items are USB2 units, max data transfer rate typically a bit under 0.5 Gb/s, which is what you observe. The fact that you can do this with FOUR cards simultaneously in ONE reader unit rated as a USB 3.2 system (max data rate nearly 5Gb/s) shows that the USB 3.2 system is NOT a limit there.

I gather your system has only one USB 3 socket available. NOTE that now, all such systems are called versions of USB 3.2, even though there were other numbers used earlier. USB 3.2 Gen1 is the first at 5 Gb/s max data throughput, USB 3.2 Gen2 is newer at 10 Gb/s, USB 3.2 Gen2x2 can reach 20 Gb/s. For what you're doing with memory cards, either Gen1 or Gen2 is quite sufficient.

First, maybe your mobo has an unused USB 3.2 header that is NOT connected to external sockets on your case. If that were the case, a simple unit with two USB 3.2 Gen1 or Gen 2 sockets on a plate to fit into an unused rear PCIe slot and a connecting cable would add the ports you want. If not that, consider buying a simple USB 3.2 Gen1 HUB and a second reader box just like you have. So far the single reader is using less than half the capcity of a USB 3.2 Gen1 port, so using TWO readers sharing even only ONE USB 3.2 Gen1 port still can double what you have already. IF you had that unused mobo header which contains TWO standard USB3.2 Gen1 ports, you would expand to three USB 3.2 Gen 1 ports in total, and you could conceivably connect to EACH of those a Hub and TWO readers.