[SOLVED] Network share slowness - Shared drive vs NAS

[skullsession0]

I come seeking advice on 3 topics:

1) What am I looking for in a proper cable? Because it seems like I've been buying garbage, and when I use a cable longer than 10', there's a huge performance drop.

2) Is there something I can do to make a gigabit network faster for 3-7 users when connecting to the same network share from a laptop hard drive?

3) Is there value in the portable NVME nas units they sell, like the qnap tbs-464? Or is that overkill for this kind of thing?

We're trying to figure out how to speed up our photo sorting machines.

What I'm asking about is all of the little micro transfers we do within the same networked folder. About 1000-2000x a day, 15-45 images each, usually 3-4 machines simultaneously (some of them while the next wave of 1400 photos is imported directly to the drive via USB).

What I want:

• Faster movement of files on a drive being shared and sorted by 4-6 PCs at once
• Something easy to take on a plane, in a car
• Something durable

What we do:

• Share a networked NVME drive with 3-7 PCs (via gigabit switch)
• Swap cards every hour, about 800-1400 images (8-15 gb) per photographer (x 4-7 photogs)
• Import directly from USB to the drive into a folder \Sunday Ingest\PhotographerName TimeofDay\
• Different sorters (on the networked PCs) will take one folder each.
• Using photomechanic (a rapid image culling/browsing software, that does lots of little mini file rename / caching things)
• View the chunk of 15-45 files (5-20 mb each), find the competitor number. Create folder (Sun\101\) and move files in there (stays within same networked drive entire time).

.... this is where the slowness is devastating:

• Original PC may take 1-3 seconds to move 15-45 files.
• Networked PC w/ 8' cable takes 5-8 seconds
• Networked PC w/ 25' cable takes 15-45 seconds

...

• In the background, there is a separate web gallery uploading app watching Sunday for any changes, and anything new it adds to the upload queue
• If the data connection is fast, it will rapidly upload small watermarked thumbnails of all images to the website. On a fast connection, it will process a hundred photos in 5-15 seconds.
• It chunks the upload so it doesn't try to process thousands at once. If working properly, it's about 100-200 total, in bursts of 20 photos

...

• If the hotspot is working well and the uploader is keeping up, it doesn't affect the sorting machines very much
• If it stalled hours ago and no one noticed, and we switch it on and it's now 5,000 images behind, we find that it cripples the network speed of the sorters until it catches up 10-15 min later (and then it's fine)

What our setup is currently (how we started, not married to this system):

card readers plugged into...
1 PC laptop w/ shared network folder (data drives are NVME SSDs of 1-2 TB, usually 970/980 or something not far off)

• Switch - TP Link 8 port Gigabit Switch - Easy Smart Managed (TL-SG108E)
• 3-5 laptops manipulating files on network folder
• 1 laptop uploading to website based on new appearances in Sunday folder

(it ignores the import folder)


Cable problems:

I'm thinking I'm just buying garbage cables (despite review of the inland pro cables). Currently we experience a massive dropoff in speeds when transferring files on a network shared folder (meaning networking in and moving them within that folder on the laptop). I've never tested it but probably 35-60% slower on the machine using the 25-30' cable instead of the closer one using a 6-10'.

Cat 6 network cable (best buy)
Cat 6-pangea E316505 ... awm 2835 60* 30v 26awg tia-eia 568b.2 utp (microcenter)
Cat ? 24 awg (microcenter)
INLANDPRO IL CAT 6 BLACK 25 FT 5-PK (microcenter)

I've read many times that nothing this short should have performance dropoff, but I'm wondering if I'm just buying garbage that can't even hold up over that tiny increase from 10' to 25'.

 

[USAFRet]

A dedicated NAS box would seem to be ideal for your use.
One central box that everyone can connect to.

A QNAP and the QTS OS is its own little server. I have a several year old TS-453A.

Each user can have their own dedicated space, as well as whatever shared space you might need.


For the connection, no reason good quality Cat5e would not work.
Don't buy garbage cables, no matter what number is associated with it.
Cheap/Crap "Cat6" is far worse than good quality Cat5e.

 

[bill001g]

The cable length should make no difference. Ethernet standards specify that the cable will run at say full gigabit speed at 100meters. A cable that is 10meters will transfer the same speed as a 100meter cable.

Now if you have some issue with the cable it might for example be dropping to 100mbps. A cable though will only run at gigabit or 100mbps it will not for example run at 500mbps. Now you might get data errors that reduce the average throughput but that too would indicate a defective cable

 

[skullsession0]

A dedicated NAS box would seem to be ideal for your use.
One central box that everyone can connect to.

A QNAP and the QTS OS is its own little server. I have a several year old TS-453A.

Each user can have their own dedicated space, as well as whatever shared space you might need.

Thank you. I own multiple units like that at home. Trying to envision what I can take on the road or plane, as many of our events require one or the other.

Are the portable units like the TBS-464 a competent option?

I don't need screaming fast speeds. I need more operating room to do multiple things at once.

As said, I download about 40-50 gb once an hour via card readers, but most of the data read/write operations is repeatedly moving a chunk of files totalling 0.15-0.45 gb, or a third party app rapidly scanning the files for changes

(I believe the uploader creates the thumbnails locally on the laptop so that shouldn't impact the network bandwidth)

For the connection, no reason good quality Cat5e would not work.
Don't buy garbage cables, no matter what number is associated with it.
Cheap/Crap "Cat6" is far worse than good quality Cat5e.

Good to know. I will do my research this week and pick up some solid Cat 5e cabling.

 

[USAFRet]

Is there any reason to even glance at the TBS-464?

On a gigabit LAN between PC and NAS, there is no difference between HDD and the fastest NVMe you can find. Except for gb per $$.
The LAN is the 'slow' device in the chain.

I have a 480GB Seagate Ironwolf SSD as the system drive and small shared drive space.
Large HDD's in the other 3 slots. 8TB/8TB/18TB

There is NO difference in copying data from the PCs to either drive type in the NAS. In either direction.

 

[skullsession0]

On a gigabit LAN between PC and NAS, there is no difference between HDD and the fastest NVMe you can find. Except for gb per $$.
The LAN is the 'slow' device in the chain.

I have a 480GB Seagate Ironwolf SSD as the system drive and small shared drive space.
Large HDD's in the other 3 slots. 8TB/8TB/18TB

There is NO difference in copying data from the PCs to either drive type in the NAS. In either direction.

Thanks. I guess what I'm trying to ask is can those PCs more quickly manipulate the files on the NAS after they arrive. And would the drive type have an impact on that.

I realize NVME/SSD will not change the LAN bottle neck to get the files from point A -> NAS, which in this case would be the initial card reader -> NAS.

But would a different model impact how quickly files can move within that NAS, or if 7 users can more consistently sort the same NAS drive.


To clarify, the original setup leaves all files on the same drive.

And this new one would too. Once the files are imported, I'm looking for a more consistent / efficient means of moving files aroud

data drive <--thunderbolt---cardreader
+
pcs use lan to manipulate files within same data drive


Current file structure
D:\Pictures\Import Sunday\PhotoLarry 10am\
files moved to
D:\Pictures\2021-11-21-Sun\101


So while they're not staying in the same subfolder, they're not leaving the drive itself. They aren't being moved back to the sorting PCs at all. They arrive in the import sunday\photog time\ folder, and then using the networked drive, they are moved into their proper folder in the same drive space.

 

[kanewolf]

A cache SSD can speed read access on a NAS. Some SSD implementations will speed writes. Copying dozens of small files could benefit from SSD caching of directory information.
A NAS with 10GE would be the obvious answer to provide gigabit performance for multiple workstations. Ethernet switches with 1 or 2 10GE ports are reasonably priced. You should also ensure that JUMBO frames are enabled in your network. That lowers network overhead by allowing more data bytes per network packet.

 

[USAFRet]

Thanks. I guess what I'm trying to ask is can those PCs more quickly manipulate the files on the NAS after they arrive. And would the drive type have an impact on that.

My QNAP has no issues managing multiple users and connections simultaneously.

Not long after I set it up, a test:

Playing 2x movies out to one PC.
Playing 2x movies out to 2 other systems.
Playing music to another system.
Getting a several hundred GB backup from another system.
Backing up a TB or so data to a USB connected enclosure.

Simultaneously.

Nary a burp in any of those data streams. If there were any hesitation or issue, the music would have been readily apparent.


Setting up a SSD cache space is easy.