As production and post workflows become increasingly distributed, the ability to send large source files quickly and reliably is becoming more important.
Unfortunately, upload speeds can be a huge bottleneck for people that want to move source files from one place to another over the Internet. Upload speeds aren’t increasing nearly as fast as file sizes. What may possible with HD source files is not possible when you move to a 4K workflow.
For freelancers and smaller production companies who are using cable and DSL based Internet access, upload speeds aren’t generally fast enough to efficiently send large volumes of source material to remote teams, broadcasters, or other stakeholders.
Even Studios, Broadcasters and VFX facilities that have fast fiber Internet connections run into problems transferring large files. While the physical infrastructure is up to the task, the protocols used to transfer files over the Internet were not designed for efficiently transferring multi-gigabyte files. This means that large files aren’t delivered at anywhere near the maximum speed of the connection.
Fortunately, there are a number of companies with solutions that help media companies accelerate data transfer so that they can securely move files faster and more reliably than they can with FTP, HTTP transfers or consumer file sharing services like Dropbox.
The downside is that these solutions are fairly pricey. It also takes some time to install the software and configure the workflow. As such, the services I’ll discuss below tend to be the domain of larger production & post companies.
For small companies and freelancers that are looking for some of the benefits, without the cost, take a look at this post BitTorrent Sync: Unlimited, Fast, Secure File Sharing for Editors.
A primer on protocols and the upload bottleneck
Upload and download speed are affected by the distance between your computer and the server that is either sending or receiving data.
In general, the greater the distance, the slower the speed of the data transfer. When connecting over larger geographic distances, traffic makes more hops between different connections, and each hop increases the round trip time, or latency.
Your data can take a multitude of different routes between your computer and a remote server. Sometimes transfers simply get caught up in congestion along the way and packets are lost.
The physical networks themselves are another source of lost packets. Files transferred over satellite, mobile, and WiFi networks tend to experience more packet loss than files sent over a fiber connection.
There isn’t much you can do to reduce the distance your data travels, avoid congestion, or eliminate packet loss, but using a different protocol for file transfer can help.
The problem with TCP/IP
The protocol that is used for uploading or downloading a file can have a large impact on the speed of data transfer.
HTTP and FTP use the TCP/IP protocol for sending and receiving data. The protocol is great for reliably sending files. Unfortunately, it’s not the fastest way to transmit large files over long distances when using fast connections.
FileCatalyst, one of the commercial providers I’ll discuss below, does a great job of explaining the problem. I’ll try to paraphrase here, but please check out their site if you want more detail.
Basically, TCP/IP sends packets of data serially (one after the other). A new packet of data won’t be sent until the receipt of the previous one has been confirmed. If delivery isn’t acknowledged after a certain period of time, the packet will be resent. In the meantime, the other packets will essentially be queued up waiting their turn.
Further, the amount of data that can be sent in a packet varies based on the quality of the connection. The more congestion or data loss, the smaller the packets of data the protocol will send. This means a greater percentage of the transfer time is spent waiting when there are poor network connections.
The bottom line is that the greater the distance, the more congested the network, and the more lossy the connection, the lower the effective transfer rate. The cumulative impact of all this waiting can be quite surprising.
According to FileCatalyst:
Assuming packet loss of 0.1% and a line speed of at least 10Mbps:
- Once you hit 50ms of latency, your transfer speed starts to take a major hit
- By 80 ms, your maximum speed is 2500 Kbps—even for links 1Gbps and faster
- By 400ms—not uncommon for satellite connections—you are crippled at 500Kbps
Source: FileCatalyst
How UDP-based solutions can help
Transfer speeds can be accelerated over fast connections (at least 5-10 Mbps) by using the UDP protocol. Where TCP/IP is designed for reliability and file integrity, UDP is designed for speed.
Basically, the UDP protocol works more like a firehose. It will continue blasting packets to the destination without waiting for delivery confirmation. Packets of data can arrive out of order and some may go missing along the way. Essentially, the protocol is designed to saturate the connection and give you transfer rates that are inline with the wire speed of the connection.
The drawback of the UDP protocol by itself is that it doesn’t have a mechanism to acknowledge packet delivery and ask for missing packets to be resent. By itself, this obviously doesn’t work for delivering media files where file integrity is a must.
This is where companies like FileCatalyst, Signiant, and Aspera come in. These companies use the UDP protocol in combination with proprietary mechanisms to ensure that missing packets are resent and that file integrity maintained. In short, these commercial solutions provide a fast, reliable, and secure way to transfer files.
One thing to note, is that in order to make use of these services you need a fast connection. So for uploads, most residential and small business users won’t see much benefit unless they have a consistent uplink speed of >5Mbps.
Commercial Accelerated Transfer Solutions
FileCatalyst
FileCatalyst has a comprehensive suite of products that are geared to organizations that need to transfer large files. One of their main target markets is the media & broadcast industry.
There are four main product categories that make use of its UDP-based accelerated file transfer technology.
FileCatalyst Direct is designed for point to point file transfers. A server is installed in one location, and people can transfer files to and from that server via hot folders, browser based applications, mobile devices, or through the API or a command line interface. The server can also be used to mirror files between multiple locations or distribute them to a number of different points.
FileCatalyst Workflow is a web portal that can manage a wide variety of different workflows. One of the sample workflows that looks quite easy to use is the webmail application. Essentially, you create an email and send the large file to the FileCatalyst server rather than attaching it. When recipients get the message, they can download it from the server. Both the sender and recipient benefit from WAN acceleration. The Workflow tool can also be used for much more complex things like multi-step jobs (ex. sending files to an encoder, waiting while they are processed, and then distributing them to different endpoints).
FileCatalyst Central is a web-based tool that enterprise users can use to manage all of their FileCatalyst applications. FileCatalyst SDK is a suite of APIs and components that can be used to integrate the product into different applications and create custom workflows.
Signiant
Signiant uses the UDP protocol to transfer data while using TCP for rate control, managing congestion on the network, and ensuring that data is sent securely and reliably.
Signiant has three main products in its portfolio.
Media Shuttle provides a very simple way to send large files. Sending a file is a matter of logging into the portal, selecting the files to be transferred, and the entering the names of the recipients. Once you’ve selected the files, they are either uploaded to the cloud (or sent to one of your own servers) and the recipients get a link to download the file. You can also configure the portal to upload / download into designated folders or let people submit files.
All of this sounds similar to things like YouSendIt (Hightail), but it’s the accelerated file transfer that really sets it apart from cheap or free consumer services. Pricing is an annual subscription based on the number of active users.
SkyDrop is Signiant’s service for transferring large files in and out of different cloud storage solutions (AWS, Google Cloud Platform, Windows Azure). For media companies I could see this being used for transferring files for rendering or encoding in the cloud or for backup / archival purposes. To send files to the cloud service of their choice, users must install a Windows or Mac client to setup folders that watch for files that need to be uploaded. There is also an SDK to integrate it with various tools and systems. Pricing is based on the projected amount of data that will be transferred over the course of a year.
Manager and client is a tool for automating the movement of files between geographically dispersed locations. The tool provides IT administrators a high degree of control over things like configuration, job scheduling, reporting and security auditing. Drag and drop workflow templates can be used to automate simple tasks like sending media from point A to point B, or it could involve a multi-step process like moving files into a transcoding queue, waiting for them to be processed and then delivering files to their final destination.
Aspera
Aspera is probably the best know accelerated file transfer solution in North America. IBM acquired the company in late 2013 to bolster it’s cloud solutions and help enterprise clients get data into the cloud.
The company has a broad suite of products that are built on top of its patented FASP protocol. The most relevant ones for video creators include :
Web Applications include Aspera Faspex, Aspera Shares, and a tool to add accelerated file transfer to Microsoft SharePoint. Faspex provides a simple way to email large files to anyone. People can create a package of files and send them via a browser based tool that’s like a webmail program, an outlook plugin, or mobile applications. You simply enter the names of your recipients, attach files, and send them in a familiar manner. Aspera shares is more like a web based portal where authorized users can sign in and upload or download files from multiple different locations.
Transfer Servers are the hub of Aspera deployments and are offered in several variants. Enterprise server enables people to upload files with the FASP protocol using desktop applications, mobile applications or custom applications that are integrated with Aspera. The Connect server adds support for web-based transfer so that anybody with Aspera’s plug-in can upload and download files. Servers are available in cloud-based or on premise deployments.
Transfer Clients connect to Aspera applications and servers. These include desktop clients, mobile clients, a browser plug-in, Aspera Drive (like an accelerated version of Dropbox), an outlook plug-in, an Avid plug-in, and custom clients built on top of Aspera’s SDK.
Management and Automation tools are available for enterprise customers that need to move large amounts of data through multi-step workflows like file encryption, video encoding, and virus scanning.