Update May 11, 2014.
An example app is now available on GitHub.
Update Oct 14, 2013.
In keeping with my practice of only passing primitives to DelayedJob to mitigate serialization errors, I made transfer_and_cleanup a static method that accepts a Document ID, rather than an instance method on Document. The new code has been inserted below.
I’ve been in the dev game for about 15 years, and for whatever reason I always seem to end up needing the ability to handle user-generated content, i.e. uploads. Back in the day, I used attachment_fu, then transitioned to Paperclip and have stuck with it ever since. Other than a few sketchy point releases, Paperclip has progressed remarkably well and is a solid, reliable solution for handling uploads.
The Paperclip S3 workflow generally looks like this:
- Server saves the file to a tmp directory and processes it if necessary
- File is transferred to S3 and deleted out of tmp
Makes sense, right?
Except for the timeouts that occur when your users have crap connections, or if dealing with very large files. Then there are the platform considerations: Heroku has an explicit 30 second http timeout, and while Engine Yard is as configurable as you want it to be, I have personally seen unresolvable issues with this workflow when uploading files in the hundreds of megabytes.
Not to mention you’re making your users wait for two uploads to complete. Again, with larger files, this ends up being pretty gross.
So what’s the answer?
Upload files directly to S3, skipping the middleman and all the associated hassles. Easy right? There are many articles out there that detail how to do just this, but they generally stop short of detailing the full upload lifecycle.
For my most recent project, I was already using Paperclip for user avatars, which are small enough to not have to worry about custom workflows, and I wanted to continue using Paperclip for file uploads in order to leverage its attachment handling syntax and callbacks. I’ve seen many folks asking about this, so I thought I’d document my process for the greater good.
Let’s get started!