Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Topic Options
#356827 - 16/12/2012 19:07 Optimize huge internet file transfers
jmwking
old hand

Registered: 27/02/2003
Posts: 777
Loc: Washington, DC metro
I have a client who shoots and edits video from a home office in DC for for various tv news programs in Europe. He needs to get files - frequently of several gigabytes - delivered electronically. Currently he ties up his computer for hours at a time on these transfers by uploading to clients' ftp servers.

We're going to set him up with a basic machine in his house so he can dump files there at LAN speeds and let it deal with the time to move data at Internet speeds. Whether it's an ftp client pushing, an ftp server his clients pull, or something else entirely is still up in the air.

It needs to be fairly idiot-proof for the recipient news departments, too, which is why I'm mostly looking at ftp. Is there a faster protocol that has widespread support?

He mostly works on a mac, but has the odd windows machine floating around, too. Am I missing anything else obvious?

thanks,

-jk

Top
#356828 - 16/12/2012 19:22 Re: Optimize huge internet file transfers [Re: jmwking]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5549
Loc: Ajijic, Mexico
Originally Posted By: jmwking
Am I missing anything else obvious?
Maybe something like Dropbox?

tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#356834 - 16/12/2012 21:20 Re: Optimize huge internet file transfers [Re: tanstaafl.]
robricc
carpal tunnel

Registered: 30/10/2000
Posts: 4931
Loc: New Jersey, USA
Dropbox is obvious, but I think Cubby from LogMeIn was built with this situation in mind. I haven't used it very much, but I believe you can share any file/folder on your hard drive and generate a link for it.

The file that's outside of the main Cubby drive won't be sync'd to the cloud first. It just lives on your hard drive. Then, when someone outside your LAN requests the file, that's when it's transferred over the internet.

Don't quote me on that, though.
_________________________
-Rob Riccardelli
80GB 16MB MK2 090000736

Top
#356837 - 16/12/2012 22:29 Re: Optimize huge internet file transfers [Re: tanstaafl.]
jmwking
old hand

Registered: 27/02/2003
Posts: 777
Loc: Washington, DC metro
Originally Posted By: tanstaafl.
Originally Posted By: jmwking
Am I missing anything else obvious?
Maybe something like Dropbox?

tanstaafl.


We've contemplated it. It would require subscribing to their paid service to have the capacity needed. And he'd still have the need for a secondary box.

-jk

Top
#356838 - 16/12/2012 22:33 Re: Optimize huge internet file transfers [Re: robricc]
jmwking
old hand

Registered: 27/02/2003
Posts: 777
Loc: Washington, DC metro
Originally Posted By: robricc
Dropbox is obvious, but I think Cubby from LogMeIn was built with this situation in mind. I haven't used it very much, but I believe you can share any file/folder on your hard drive and generate a link for it.

The file that's outside of the main Cubby drive won't be sync'd to the cloud first. It just lives on your hard drive. Then, when someone outside your LAN requests the file, that's when it's transferred over the internet.

Don't quote me on that, though.


Looks interesting, but they seem to have a 2 GB filesize limit. That would be a problem for him.

thanks,

-jk

Top
#356841 - 16/12/2012 23:25 Re: Optimize huge internet file transfers [Re: jmwking]
drakino
carpal tunnel

Registered: 08/06/1999
Posts: 7868
I beta tested a feature from Egnite (https://www.egnyte.com) coupled with my ReadyNAS. It allowed me to throw data onto the NAS in a specific share, then have that share mirrored to another NAS or computer somewhere out there. Had quite a few options including making it available from "the cloud". Something like this may work out well for your client.

Top
#356842 - 17/12/2012 03:57 Re: Optimize huge internet file transfers [Re: drakino]
Dignan
carpal tunnel

Registered: 08/03/2000
Posts: 12343
Loc: Sterling, VA
Make sure your client is fully aware of his bandwidth limits. If he's in DC there's a good chance he's on Comcast. Uploads count toward your total as much as downloads do, sadly.
_________________________
Matt

Top
#356843 - 17/12/2012 05:58 Re: Optimize huge internet file transfers [Re: jmwking]
andy
carpal tunnel

Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
ftp might have some annoying limitations and features, but on the speed issue you won't actually find anything that is significantly faster when you are talking about just copying a file from one location to another

(the same doesn't apply when you are doing something like syncing a set of changing files, when of course things like rsync that do diffs are far faster)

If the client has something like an ADSL line with low upstream speeds, they might be running into the problem of the upstream bandwidth being saturated for long periods which then breaks everything on the line that is trying to use TCP. When that happens you really need a way of throttling the upload speed to keep it from fully filling the line.
_________________________
Remind me to change my signature to something more interesting someday

Top
#356844 - 17/12/2012 06:02 Re: Optimize huge internet file transfers [Re: jmwking]
andy
carpal tunnel

Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
What do you mean by "he ties up his computer for hours at a time" ?

Uploading some files via ftp should have little effect on a modern computer, it should certainly not "tie it up".
_________________________
Remind me to change my signature to something more interesting someday

Top
#356846 - 17/12/2012 09:57 Re: Optimize huge internet file transfers [Re: andy]
sein
old hand

Registered: 07/01/2005
Posts: 893
Loc: Sector ZZ9pZa
Originally Posted By: andy
What do you mean by "he ties up his computer for hours at a time" ?

I'm guessing it really means 'ties up his connection' since a fully saturated uplink will usually make ADSL unusable.

I think the best solution would be to employ some form of QoS so that he can still use his connection while he is transferring his work. Other solutions may be more convenient, but as said before nothing else is really going to be faster.
_________________________
Hussein

Top
#356847 - 17/12/2012 10:22 Re: Optimize huge internet file transfers [Re: sein]
andy
carpal tunnel

Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
I've tried and failed to get QoS stuff working. In the end I gave up and got a second ADSL line instead.

I run my servers and BitTorrent client on one line and all my other stuff on the second line. wink

(that isn't the only reason I got the second line, but it was a handy side effect)
_________________________
Remind me to change my signature to something more interesting someday

Top
#356848 - 17/12/2012 12:20 Re: Optimize huge internet file transfers [Re: andy]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
QoS works fine when configured correctly -- we use it here with good success. But very few people understand QoS, and therefore it can be very difficult to set up.

Edit: In this specific case, though, it's mainly upstream QoS that is needed, and that part is almost trivial to set up for the specific need in this case.



Edited by mlord (17/12/2012 12:22)

Top
#356849 - 17/12/2012 12:24 Re: Optimize huge internet file transfers [Re: andy]
jmwking
old hand

Registered: 27/02/2003
Posts: 777
Loc: Washington, DC metro
He uses fios and he reports a 2 GB transfer to europe can take a few hours, time he can't move his computer. He can use it for other things, of course, he just can't move it. He spends half his time out and about shooting video, the other half prepping it for broadcast.

A second computer is certainly the first step. I just want to make sure there isn't a significantly more optimized protocol than ftp and that his clients in europe could navigate. Copying via a windows share probably takes a third longer, and some of multi-channel file transfers need special clients. We're trying to optimize and keep it simple.

-jk


Edited by jmwking (17/12/2012 12:24)

Top
#356850 - 17/12/2012 12:52 Re: Optimize huge internet file transfers [Re: mlord]
andy
carpal tunnel

Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
I've good a reasonably good grasp of TCP/UDP/IP (certainly better than 99.99%) of the population planet has). But despite many hours of trying I could not come up with a workable QoS setup.

The best I came up with was with everything throttled back to 25% of the upstream bandwidth, but with you only start with 800k that wasn't exactly workable frown

(and even then VOIP used to regularly get hit with a lack of upsteam bandwidth)

Though I'm not sure QoS on my Tomato router was working quite as it was supposed to, so that probably didn't help.

Instead now I have my CrashPlan backup throttled (within the service itself) to 45% and BitTorrent throttled to 30% (again within the BT app), leaving some space for SMTP/DNS/HTTP even when BT and CP are running at full pelt (which isn't that often). And I have a separate line for all our laptops/desktops/iPads/iPhones/Apple TV/set top box/etc

Works much better than I was able to achieve with QoS.
_________________________
Remind me to change my signature to something more interesting someday

Top
#356851 - 17/12/2012 12:54 Re: Optimize huge internet file transfers [Re: mlord]
andy
carpal tunnel

Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
Originally Posted By: mlord

Edit: In this specific case, though, it's mainly upstream QoS that is needed, and that part is almost trivial to set up for the specific need in this case.


Agreed. And you shouldn't even need to mess with the router to achieve it.

It should be doable in either the ftp client or server (depending on whether he is uploading to a server or sharing the files out). Certainly things like the filezilla client let you set an upstream limit.
_________________________
Remind me to change my signature to something more interesting someday

Top
#356852 - 17/12/2012 13:06 Re: Optimize huge internet file transfers [Re: jmwking]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: jmwking
He uses fios and he reports a 2 GB transfer to europe can take a few hours

Just because it's "fios" doesn't mean "fast speeds".

Originally Posted By: wikipedia
Six different offerings are available:[4]

3/1 – offered as an upgrade to FiOS for DSL customers.
15/5 – available as individual or bundle tier.
25/25 – available as individual or bundle tier.
35/35 – a bundled tier only.
50/25 – available as individual or bundle tier.
75/35 – available as individual or bundle tier.
150/65 – available individually and with bundles in some markets; it requires a GPON optical network terminal, which is only available in GPON deployed central offices.
300/65;[5] Standalone only; requires a GPON optical network terminal as well as a gigabit broadband home router.

Legacy speed tiers include: 5/2, 10/2, 15/2, 15/15, 20/5, 20/20, 25/15, 30/5, 30/15, and 35/20.


What's the uplink speed on his fios plan?

If it's "25" or better, then a 2GB upload should take less than half an hour, unless fios also throttles, or if the other end is the bottleneck.

Top
#356855 - 17/12/2012 16:05 Re: Optimize huge internet file transfers [Re: jmwking]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31602
Loc: Seattle, WA
Quote:
We're going to set him up with a basic machine in his house so he can dump files there at LAN speeds and let it deal with the time to move data at Internet speeds. Whether it's an ftp client pushing, an ftp server his clients pull, or something else entirely is still up in the air.


Empegger and BBS user Aleksander Milewski ("N6mod") used to work for a company that specializied in solving precisely this problem. I'm having a terrible time remembering the name of the company. Anyone else remember it so we can google it?

Basically they were trying to make a cost-effective and time-effective alternative to movie production companies and SFX houses having to ship hard disks full of video files to each other. (Because that was the way they were doing it at the time because it was faster than file transfers over the internet.)

It worked by having PCs at both ends of the link doing a continuous UDP stream that handled the data transfer communications in a round-robin fashion where the client and server didn't have to wait for a response before streaming the next set of packets. It could effectively saturate any connection up to its maximum available bandwidth, something that most standard file transfer protocols can't do due to the latencies involved in their handshaking protocols.
_________________________
Tony Fabris

Top
#356859 - 17/12/2012 17:19 Re: Optimize huge internet file transfers [Re: mlord]
jmwking
old hand

Registered: 27/02/2003
Posts: 777
Loc: Washington, DC metro
Originally Posted By: mlord
Originally Posted By: jmwking
He uses fios and he reports a 2 GB transfer to europe can take a few hours

Just because it's "fios" doesn't mean "fast speeds".

Originally Posted By: wikipedia
Six different offerings are available:[4]

3/1 – offered as an upgrade to FiOS for DSL customers.
15/5 – available as individual or bundle tier.
25/25 – available as individual or bundle tier.
35/35 – a bundled tier only.
50/25 – available as individual or bundle tier.
75/35 – available as individual or bundle tier.
150/65 – available individually and with bundles in some markets; it requires a GPON optical network terminal, which is only available in GPON deployed central offices.
300/65;[5] Standalone only; requires a GPON optical network terminal as well as a gigabit broadband home router.

Legacy speed tiers include: 5/2, 10/2, 15/2, 15/15, 20/5, 20/20, 25/15, 30/5, 30/15, and 35/20.


What's the uplink speed on his fios plan?

If it's "25" or better, then a 2GB upload should take less than half an hour, unless fios also throttles, or if the other end is the bottleneck.


I don't know his plan yet, but I suspect the limit is not within fios, but rather somewhere across the atlantic.

Top
#356860 - 17/12/2012 17:19 Re: Optimize huge internet file transfers [Re: tfabris]
jmwking
old hand

Registered: 27/02/2003
Posts: 777
Loc: Washington, DC metro
Originally Posted By: tfabris
Quote:
We're going to set him up with a basic machine in his house so he can dump files there at LAN speeds and let it deal with the time to move data at Internet speeds. Whether it's an ftp client pushing, an ftp server his clients pull, or something else entirely is still up in the air.


Empegger and BBS user Aleksander Milewski ("N6mod") used to work for a company that specializied in solving precisely this problem. I'm having a terrible time remembering the name of the company. Anyone else remember it so we can google it?

Basically they were trying to make a cost-effective and time-effective alternative to movie production companies and SFX houses having to ship hard disks full of video files to each other. (Because that was the way they were doing it at the time because it was faster than file transfers over the internet.)

It worked by having PCs at both ends of the link doing a continuous UDP stream that handled the data transfer communications in a round-robin fashion where the client and server didn't have to wait for a response before streaming the next set of packets. It could effectively saturate any connection up to its maximum available bandwidth, something that most standard file transfer protocols can't do due to the latencies involved in their handshaking protocols.


this sounds interesting! I'll see what I can find.

-jk

Top
#356863 - 17/12/2012 18:17 Re: Optimize huge internet file transfers [Re: tfabris]
drakino
carpal tunnel

Registered: 08/06/1999
Posts: 7868
Are you thinking of Aspera Tony? http://asperasoft.com/

Used that at a past employer, and it was pretty quick, to the point that it would saturate the uplink when possible.

Top
#356865 - 17/12/2012 21:22 Re: Optimize huge internet file transfers [Re: drakino]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31602
Loc: Seattle, WA
Yep, that's the one!

It requires the whole client/server setup, but, if you control both ends of the link, it will give you the fastest possible transfer rate between those two points.
_________________________
Tony Fabris

Top
#356873 - 18/12/2012 14:27 Re: Optimize huge internet file transfers [Re: tfabris]
jmwking
old hand

Registered: 27/02/2003
Posts: 777
Loc: Washington, DC metro
That does look interesting. It looks like they may have a web based on-demand version that I'll look into.

thanks!

-jk


Edited by jmwking (18/12/2012 14:30)

Top
#356876 - 18/12/2012 18:16 Re: Optimize huge internet file transfers [Re: jmwking]
Dignan
carpal tunnel

Registered: 08/03/2000
Posts: 12343
Loc: Sterling, VA
Just to show that podcast advertising occasionally gets through, I offer Sharefile (a frequent TWiT advertiser).

However, I have no idea if that service would be of any use in this situation...


Edited by Dignan (18/12/2012 18:17)
_________________________
Matt

Top