Looking for solution - transferring video over gigabit

[JohnWeldt]

I am using FTP server/client, filezilla to transfer content from one server to the next over gigabit networks. The average is 30MB/s on files averaging 36GB. I am running a mixed environment that includes Windows, Linux and Mac. For the Mac I am using cyberduck FTP client because it was recommended.

Is there a better way of transfer content while maintaining the integrity? Perhaps software that could transfer and do a checksum at the end?

Any thoughts?

In addition I have tried Jumbo frames and did not see a speed boost. But it was from a laptop to server. The laptop has a 5400rpm drive which I think may limit the speed. Going to test this futher.

 

[blue68f100]

The 30MB/s is close the the max speed (write) of a single non raid drive. You might try a direct connect with a crossover cable to locate where the bottle neck is.

If you are running Norton, feed it to the shreeder. It will slow any machine down no matter the horsepower. Try turning off any tsr that are not needed. Optimize the drives, will also help.

 

[JohnWeldt]

I thought drive averaged 60MB/s.

I tried a cross over cable and no difference in speeds. Jumbo frames where the same. That is why I am turning to RAID for help.

Nothing running on the servers other then core apps. No AV or anything. I googled tsr and found interesting results:

Dungeons & Dragons Roleplaying Game Official Home Page
The Sims Resource
Tropical Storm Risk (TSR) for long-range forecasts of hurricane ...

What is tsr?

Is there an FTP program that will automatically upload anything that is in a directory, verify the end result, and disconnect/standby?

 

[blue68f100]

tsr are programs that load on boot, like: special mouse drives, AV, FW, Adobe, Printer CP, ... It seams ever body want there it have an edge over the other guide. But 90% of them are un necessary. They preload parts of there program so it will load faster.

WMatch may be able to help you. It is a file/folder sync program with scheduling. Have not used it in auto mode.

 

[Madwand]

I thought drive averaged 60MB/s.

No, drives peak around this rate, and you'll only see the best speed with large unfragmented files at the beginning of drive, when the drive interface, buffer caching, read-ahead, file system, and OS, etc., are working well together.

I tried a cross over cable and no difference in speeds. Jumbo frames where the same. That is why I am turning to RAID for help.

Right, at 30 MB/s, it's unlikely that the network is the bottleneck. Looking at the file systems and drives themselves you can get an idea of underlying bandwidth. If you have two drives in one machine (as you would to have for RAID), try copying from one drive to another.

The problem could be in the write performance. For instance, the Linux journaling systems can take up some of the time in caution. You can eliminate write performance for testing in Linux by using cat ... /dev/null. Beware that this can give you misleading results -- it'll be the maximum read speed, and you have to account for synchronization issues, protocol delays, overhead, error checking, directory management, OS mismatches, etc., before you can arrive at the net best expected file copy time.

The prefix "time" to cat or cp in Linux can measure the time, making the average data rate calculation easy.

 

[blue68f100]

You may want to use a RAID5 system for storage, with gigabit lan connection. This will give you redundency and Speed.

 

[weekendwarrior]

Keep in mind that the FTP protocol itself has significant overhead in file transfers, and is not the best solution for traffic available to you.

Monitor the CPU, paging and drive utilization of your servers to see where the bottleneck lies. If they don't seem to be the problem, try copying the files to two different computers at the same time, and see if they can both get that same speed across them.

 

[blueeyesm]

found interesting results:

Dungeons & Dragons Roleplaying Game Official Home Page
The Sims Resource
Tropical Storm Risk (TSR) for long-range forecasts of hurricane ...

heheh D&D was once owned by a company called TSR.

I'm surprised that came up as one of first hits.

 

[weekendwarrior]

Forgot to add - TSR means Terminate, Stay Resident. It's an old term for DOS programs that could run in the background after execution, and allow you to run other programs.