Losing Some Data Over WLAN, how to deal with it?

[KiDo3Konvict]

First of all, it's 100% legal, we're authorized to use the contents however we like so we are not breaking any legal terms.

Now here is the things, we have a device which processes some data and displays it to a webpage only accessible by us, the problem we're facing is that the device processes too much data that only if we're on a local network then we can see all of the data being processed before our eyes, when using WLAN, no matter how fast our connection is some data will be lost...

The content of that device is uneditable, we can interact with the data only through the webpage but it's not possible to change the way the device functions.

Now I've ran out of ideas for dealing with this, we want to be able to capture all the data that device is processing!

My first idea was to get another PC, connect it to that device via local network, clone that website and buffer the contents, then we faced another problem... the device requires a username and password, to clone the website we need to login first, but AFAIK and from all the searches I've done it's not possible to make an autologin when the page is using flash to login, not a regular form.

So I'm at a dead end here, and would really appreciate any suggestion.

Also the company which created that device won't allow any editing to the device even by themselves, it's like you have this thing, you deal with it on your own.

Thank you.



EDIT: details regarding mods' points!


1. Our Wifi doesn't suck, it's actually a telecommunication company, so we have a pretty super mega fast connection.
2. The data is various of information regarding the usage of our network, not sure if I can explain the contents in details, but best I can say is various statistics and usage of network.
3. The distance between the device and our PC varies from a 25 miles away to a country size area, we have many of these devices around the country.
4. Regular building, it's not a military thing or something.
5. The device processes huge amount of data per ms, so no matter how fast the connection is we can't get all the data, we even tried other ISPs, highest available plans, nothing could keep up.

 

[USAFRet]

Lacking details, no real suggestions can be given.

1. WiFi sucks
2. What is this 'data'?
3. How far away?
4. Building construction?
5. See #1

 

[KiDo3Konvict]

Lacking details, no real suggestions can be given.

1. WiFi sucks
2. What is this 'data'?
3. How far away?
4. Building construction?
5. See #1

Edited the question with the details.

 

[USAFRet]

3. You're doing WiFi from 25 miles away?
Please describe this in some detail...

 

[KiDo3Konvict]

3. You're doing WiFi from 25 miles away?
Please describe this in some detail...

Oh not a 25 miles Wifi, sorry if I wasn't clear.

The device is something like a server which its content could be access through an IP, so the interface is like a website which can be accessed from everywhere, you don't have to be connected to the same network to access it.

 

[USAFRet]

3. You're doing WiFi from 25 miles away?
Please describe this in some detail...

Oh not a 25 miles Wifi, sorry if I wasn't clear.

The device is something like a server which its content could be access through an IP, so the interface is like a website which can be accessed from everywhere, you don't have to be connected to the same network to access it.

OK.
So, a server local to the capture device, and then through....what?...to your local systems?
Where does the WLAN come in?

Just trying to get a handle on the full chain here.

 

[kanewolf]

How far is the processing "device" from the data source? Is the link from processor to data that works with wired but not with wireless? Is so, I have two possible reasons. The data is being transmitted via UDP rather than TCP. UDP is not a guaranteed delivery so any glitch in the WIFI will cause data loss. OF the processor acknowledges the receipt of data and the half-duplex nature of WIFI causes data loss because the sender can't send while the processor is acknowledging. Wired network is full duplex so the acknowledgements don't interfere.

 

[KiDo3Konvict]

3. You're doing WiFi from 25 miles away?
Please describe this in some detail...

Oh not a 25 miles Wifi, sorry if I wasn't clear.

The device is something like a server which its content could be access through an IP, so the interface is like a website which can be accessed from everywhere, you don't have to be connected to the same network to access it.

OK.
So, a server local to the capture device, and then through....what?...to your local systems?
Where does the WLAN come in?

Just trying to get a handle on the full chain here.

Here's an example which I hope explains it the best;

Say you live in LA, and you have one device in NY, one in Washington, a third one in Alaska, another one in Vegas and so on...
Now you want to access the interface of the server in NY and see the data but that server analyzes way too much data that you in LA can't capture all of them, for example if it processes like 1024 KB of data per second and your connection is say 1000 KBps then 24 KB of data will be lost per second.

So the question is, how can you capture all of the data which is being processed faster than the connection between you and the server?

Just thought of another example, say you want to watch an NBA game, but your connection is not fast enough to live stream the game to your PC!

 

[kanewolf]

There are a couple possible methods. You could add hardware that does real-time compression of the data before putting it on the network so that the effective data rate is lower than your available network bandwidth. That is really what your NBA example is. Hardware that compresses the HD data to a lower effective bandwidth. But that compression is lossy. You would have to use a lossless compression.

The other option I can think of would allow sampling. You locally capture all the data but can only remotely access a time limited sample that gets forwarded at the lower bandwidth.

 

[USAFRet]

If any part of the data chain cannot keep up, you WILL have data loss.
If your connection can't keep up...then you need a faster connection.

Video (NBA game) is a LOT more forgiving, because it can be compressed and lose 'a little bit', without materially impacting the game.

If you need 100% of the data, and it appears you do...then you need to upgrade whichever part is the LIMFAC.
The remote server accepts the data, then pumps it out over the internet, then your local server accepts and does whatever you need it to do.

If the remote server can't accept and transmit fast enough, or there is to much data for your connection speed...$$ to change that.

How 'real time' does this need to be? Are there downtimes in the capture stream?
Can the remote server store, and then transmit in bursts?

 

[KiDo3Konvict]

If any part of the data chain cannot keep up, you WILL have data loss.
If your connection can't keep up...then you need a faster connection.

Video (NBA game) is a LOT more forgiving, because it can be compressed and lose 'a little bit', without materially impacting the game.

If you need 100% of the data, and it appears you do...then you need to upgrade whichever part is the LIMFAC.
The remote server accepts the data, then pumps it out over the internet, then your local server accepts and does whatever you need it to do.

If the remote server can't accept and transmit fast enough, or there is to much data for your connection speed...$$ to change that.

How 'real time' does this need to be? Are there downtimes in the capture stream?
Can the remote server store, and then transmit in bursts?

Trust me, we have tried every plan by every ISP in this country, no one could keep up, so no amount of money can change that.

For us, it doesn't matter whether if we capture the data the moment they're transmitted by the server, a minute later or half an hour, we just want to capture them all with no loss.

And that server does not store anything, it processes some data, transmit it, then replace it within a teeny tiny millisecond, unfortunately I'm still not sure of the size of the data that's being transmitted per ms, but if you connect to the same local network with the server then you can see more data displayed on this PC than the one connected via WLAN.

 

[USAFRet]

If any part of the data chain cannot keep up, you WILL have data loss.
If your connection can't keep up...then you need a faster connection.

Video (NBA game) is a LOT more forgiving, because it can be compressed and lose 'a little bit', without materially impacting the game.

If you need 100% of the data, and it appears you do...then you need to upgrade whichever part is the LIMFAC.
The remote server accepts the data, then pumps it out over the internet, then your local server accepts and does whatever you need it to do.

If the remote server can't accept and transmit fast enough, or there is to much data for your connection speed...$$ to change that.

How 'real time' does this need to be? Are there downtimes in the capture stream?
Can the remote server store, and then transmit in bursts?

Trust me, we have tried every plan by every ISP in this country, no one could keep up, so no amount of money can change that.

For us, it doesn't matter whether if we capture the data the moment they're transmitted by the server, a minute later or half an hour, we just want to capture them all with no loss.

And that server does not store anything, it processes some data, transmit it, then replace it within a teeny tiny millisecond, unfortunately I'm still not sure of the size of the data that's being transmitted per ms, but if you connect to the same local network with the server then you can see more data displayed on this PC than the one connected via WLAN.

Still trying to determine why there needs to be WiFi in this data chain.

 

[KiDo3Konvict]

If any part of the data chain cannot keep up, you WILL have data loss.
If your connection can't keep up...then you need a faster connection.

Video (NBA game) is a LOT more forgiving, because it can be compressed and lose 'a little bit', without materially impacting the game.

If you need 100% of the data, and it appears you do...then you need to upgrade whichever part is the LIMFAC.
The remote server accepts the data, then pumps it out over the internet, then your local server accepts and does whatever you need it to do.

If the remote server can't accept and transmit fast enough, or there is to much data for your connection speed...$$ to change that.

How 'real time' does this need to be? Are there downtimes in the capture stream?
Can the remote server store, and then transmit in bursts?

Trust me, we have tried every plan by every ISP in this country, no one could keep up, so no amount of money can change that.

For us, it doesn't matter whether if we capture the data the moment they're transmitted by the server, a minute later or half an hour, we just want to capture them all with no loss.

And that server does not store anything, it processes some data, transmit it, then replace it within a teeny tiny millisecond, unfortunately I'm still not sure of the size of the data that's being transmitted per ms, but if you connect to the same local network with the server then you can see more data displayed on this PC than the one connected via WLAN.

Still trying to determine why there needs to be WiFi in this data chain.

Oh it's only a step to the final solution, Wifi is slower than the wired connection and faster than any internet we have.
Since we can capture all the data via wired connection then the next step is to capture them all via Wifi, and if it's possible to do then we can proceed to implement the method towards a regular internet connection.

 

[USAFRet]

It may be that the amount of data you seek is just not possible over WiFi, or your even slower internet connection.

Are there any other organizations doing similar work in your area? How do they do it?

 

[kanewolf]

How much control/configuration do you have on the processor? Is it a standard OS with a custom application or is it a black box ?

 

[KiDo3Konvict]

It may be that the amount of data you seek is just not possible over WiFi, or your even slower internet connection.

Are there any other organizations doing similar work in your area? How do they do it?

I don't wanna sound an all-known but as far as I know, it's a new device so probably no other organization is working on similar thing here.

 

[KiDo3Konvict]

How much control/configuration do you have on the processor? Is it a standard OS with a custom application or is it a black box ?

It's a black box, we can't configure anything, we can just interact with it through the available interface and the available options.

 

[USAFRet]

What does the manufacturer recommend?

 

[KiDo3Konvict]

What does the manufacturer recommend?

That's the worst part, nothing... they said here's the device, you're free to do what you want but we can't help you with anything.

[strike]I have absolutely no idea why did the company accept this deal![/strike]