Using LFTP in Web Development Workflows

After scouring the interwebs and finding next to no useable examples of LFTP commands (particularly one-liners), these are what I came up with after a good bout of trial-and-error.

Backstory

Over time at the studio we’ve built up several tailored workflow tools to automate and ease our development workflow burdens. One particularly important responsibility we have as developers is managing site backups.

While site backups are thorny enough by themselves, but add to that several dozen different clients, some hosted on VPS’s that we control (with SSH access), others hosted on menial shared hosting with barely FTP access, most with CMSs that they’re adding content to constantly, et cetera. A streamlined, automated backup solution starts to feel out of reach.

We have rsync to backup our remote servers locally, and a workable solution to backup our VPSs to Amazon S3, but nothing for those wayward souls without SSH access. We were getting by with a convoluted reminder system that had one of us performing a “Sync” in Transmit over FTP to our local systems, but that’s just a mess. Enter LFTP: the jig saw powertool to Transmit’s coping saw.

Preamble: Install on OS X

I followed these wonderful instructions to get up and running quickly without macports/homebrew. I believe you can also sudo brew install lftp if it pleases you.

The Goods

The tutorial link on the LFTP site is broken, but here’s one that works. It’s good, but I don’t want to use LFTP interactivly with an persisting FTP connection; I want to run commands out of a script and move on. So initially I got a ton of unknown command errors when following the aforementioned tutorial: specifically, the “nested” commands weren’t working for me at all.

The key is enclosing all your commands in quotes, and chaining them together with “;”

Example 1: Execute multiple remote commands

lftp -c "open -u user,password example.com; ls; pwd; mget public_html/robots.txt"

The formatting is super-important here, because LFTP is super-particular.

• -c tells LFTP to “run these commands, then exit”.
• open is the first command, followed by the credentials and the FTP host
• -u Username and password must be as noted, no quotes (although you can also bolt them onto the FTP host in tradition “@” format)

After that, chain together your commands, and LFTP will exit once they are complete.

Example 2: Download a remote file

lftp -e 'set xfer:clobber true; get -e -O images/ images/logo.jpg; bye' -u username,password example.com

This is another way to write a string of commands for LFTP to execute remotely, the syntax is wholly thanks to Russ Brooks.

• -e tells LFTP to execute the following command (or commands, if separated with a ‘;’)
• set xfer:clobber true this will overwrite without prompting the destination file
• get -e -O images/ images/logo.jpg This runs a GET command, with force overwrite (-e), to download to the base directory ‘images’ (-O images/), and the target remote file to download is at the path ‘images/logo.jpg’.

Example 2: Mirror folder using LFTP

lftp -c "open -u user,password example.com; mirror -n --parallel=4 -v /var/www/wp-content/uploads/ ~/Sites/www/wp-content/uploads/"

• mirror = command to sync remote directory to local. Syntax is mirror <remote location> <local location>.
• -n = mirror only-newer files/folders
• --parallel = download files in parallel if possible (set to 4 at a time)
• -v = verbose, prints the results to stdout
• Paths to mirror from/to. Paths can be full or relative (e.g. wp-content/uploads/ wp-content/uploads/), however, full paths take into account the directories your FTP user has access to (e.g. likely not /var), so YMMV.

I found this approach to be simple and effective, and the transfer is surprisingly fast to boot. If you haven’t looked into it yet, LFTP is definitely one to add to your playbook.