[SOLVED] Low Latency Uploading & Downloading

[HomeSkillenSlice]

Hi there I'm not sure exactly where to post this. Either here or in the networking section, or somewhere else .. Please let me know if this is not the right place

I don't know under which domain my problem fits under

Basically what I'm trying to do is upload a datastream to a location quite far away. Halfway across the world really .. and what I'm trying to figure out is what I need in order to achieve this. I don't have any network experience but i do know the basics of say building a website for example and hosting it on my local computer

What I'm trying to figure out is that if I build a server and have it establish a direct connection to another server, what can I do to substantially reduce the latency of this datastream? Or in other words what can I do to reduce the time it takes for the receiving server to receive the data? Keep in mind that it is a never-ending datastream I am trying to send and the latency is very important.

Would a CDN be required to solve this problem?

Please keep in mind that I am trying to get the latency as low as theoretically possible. Enough so that it is just slightly behind the time u would measure if u were measuring the travel time of actual light itself reaching that location

I know of the ping measurement tool on wondernetwork and it is quite useful but I don't know how applicable it is given that ISP's tend to send and receive data in both the cheapest and fastest way they can get away with ... Or at least I assume anyways

Are there any network tools available I can use to optimize the connection itself so that the path the server takes to send the data is the lowest latency possible? Or is this not within my control?

Thank you so for much for your help and sorry for being a noob. If the question itself doesn't really have an answer and you know some networking or internet related books I can read in attempting to solve this problem please recommend them!!

 

[SamirD]

I was once faced with a similar problem about a decade back and was looking into cloud providers (new tech at the time) to basically connect at each end with low latency and then at wire speed (no latency) within the cloud to do exactly what you're talking about. It was really tough back then and required a virtual router in the cloud, but today I think it is much easier. The only caveat is that you're going to be charged for every bit of data across that connection, so it might be cost prohibitive in the end when compared to an ipsec tunnel between the sites over some good providers on both ends.

 

[HomeSkillenSlice]

I was once faced with a similar problem about a decade back and was looking into cloud providers (new tech at the time) to basically connect at each end with low latency and then at wire speed (no latency) within the cloud to do exactly what you're talking about. It was really tough back then and required a virtual router in the cloud, but today I think it is much easier. The only caveat is that you're going to be charged for every bit of data across that connection, so it might be cost prohibitive in the end when compared to an ipsec tunnel between the sites over some good providers on both ends.

Thank you for your response good sir and it is even better that u have experience with this

I definitely suspected this. There are many CDN networks that promise low latency and I am not so much concerned about the cost. CDN.net for example advertises 1 cent a gb which is not bad for my specific use case

I will have to try both solutions I guess. Is there an IPsec software solution available and also one that u would recommend? Or is that something an individual would have to build?

 

[bill001g]

There is no generic answer to this because it primarily depends on where the 2 servers are. It gets much more complex if you are talking about more than just a single connection.

What you would need is a direct fiber between the 2 locations. Still even this is not simple. If you look at the times for traffic say between Egypt and India the latency is huge in many cases. This is because most fiber is run in the ocean to avoid one country disrupting another. In this case the fiber can either run via the suez canal or run all the way around south of africa. There are political issues with access to the fiber that runs in the suez canal so many times the data has to go the long way.

Obviously you can't get the fiber yourself. A large amount of the fiber was put in place by a company called global crossing. This company was bought out by level 3 but they were too also bought out. Verizon, att and tata are three other large owners of undersea fiber.

What you need to do is try to get a direct connection to one of these large ISP on both ends. This way you would stay completely on their network and run over their fiber connections. In many cases these companies publish their latency between cities on their network. Many also have what is called looking glass sites that allow you to get into their routers and do ping and trace commands.

Your largest issue is going to be the final connection. I doubt you can afford to buy your own fiber connections to one of these large ISP. In most cases you must place your server in a hosting center that has direct connections to these ISP on both ends. You also must check that they actually use that ISP for the connection when the hosting center has mulitple ISP. Many times they will route the traffic over a longer path because that partner ISP is offering them cheaper data rates.

In the end there is no answer you are going to get on a forum. You are going to have to do your own research to find what your actual options are for where you place the servers and what ISP they can use.

 

[HomeSkillenSlice]

There is no generic answer to this because it primarily depends on where the 2 servers are. It gets much more complex if you are talking about more than just a single connection.

What you would need is a direct fiber between the 2 locations. Still even this is not simple. If you look at the times for traffic say between Egypt and India the latency is huge in many cases. This is because most fiber is run in the ocean to avoid one country disrupting another. In this case the fiber can either run via the suez canal or run all the way around south of africa. There are political issues with access to the fiber that runs in the suez canal so many times the data has to go the long way.

Obviously you can't get the fiber yourself. A large amount of the fiber was put in place by a company called global crossing. This company was bought out by level 3 but they were too also bought out. Verizon, att and tata are three other large owners of undersea fiber.

What you need to do is try to get a direct connection to one of these large ISP on both ends. This way you would stay completely on their network and run over their fiber connections. In many cases these companies publish their latency between cities on their network. Many also have what is called looking glass sites that allow you to get into their routers and do ping and trace commands.

Your largest issue is going to be the final connection. I doubt you can afford to buy your own fiber connections to one of these large ISP. In most cases you must place your server in a hosting center that has direct connections to these ISP on both ends. You also must check that they actually use that ISP for the connection when the hosting center has mulitple ISP. Many times they will route the traffic over a longer path because that partner ISP is offering them cheaper data rates.

In the end there is no answer you are going to get on a forum. You are going to have to do your own research to find what your actual options are for where you place the servers and what ISP they can use.

Thank you and I suspected it would be a complicated endeavor

I will probably check to see if this is a task that can be relegated to a conventional CDN provider but if the latencies are too high then ultimately I will have to directly contact various ISP's to see what arrangements can be made regarding the reduction of latency for the data

I suspect one might also need to pursue arrangements with multiple ISP's in order to get the best result as I doubt At&T has fiber laid out in random patches of India. First I will have to contact one ISP to route it as quickly as possible to this location near the ocean and then have whichever owner of the fiber that goes across the ocean to send it the quickest way possible to N.A . Then if the same owner of the large fiber cable in the ocean doesn't own fiber in the nation it reaches I would have to make yet another arrangement

Your hosting centre arrangement seems better albeit with worse results

Either all this or I relegate the task to a CDN and hope they do a sufficient job

 

[bill001g]

If you REALLY want it you can get a guarantee of both minimum latency and minimum bandwidth. The will put in writing what the latency and bandwidth will be. Most large ISP offer this service. Is kinda a vpn but it is called MPLS. They in effect build a private path between the 2 location. It does not actually travel over the internet but may use the same fiber. The internet traffic would have a different MPLS tag that likely is given less priority when they share a fiber.

This is very commonly used by large corporations to connect their remote offices. It tends to be much more expensive than a VPN over the internet but it because of the service level guarantee. The company I worked for used verizon for most of this and they had agreements with other ISP In countries where they did not have local access. We used to do this because we carried a lot of voice and video conference and it did not work well over the internet especially when the latency was high.

It is not that expensive the hardest part is how do you get to the verizon network. You would need some kind of fiber or private line. Verizon does offer hosting services but I know nothing about it.