From FileZilla to deploy.sh: Building a Deploy Script

I was always fascinated with what a man can do with just plain text commands and his terminal. I was so glad when I wanted to play CS over a LAN network and needed to go to the Windows CMD and check my IP address with ipconfig. That was my first problem solving using the command line. Fast forward a couple of years and installing Ubuntu for the first time. Couple of full HDD accidental wipes, some programming and loads of other stuff and I'm typing this blog post using Vim. Git this, scp that, SSH over there, a little bit of tail, find some stuff, run an OCR script I wrote by simply executing ./ocr.sh. AMAZING!

As soon as I found out how much automation scripts can help me speed up my work, and help me not to get worried about some stuff I started helping myself by creating shortcuts and scripts. This is the story of one of those scripts and the iterations it took to get from FTP to something I could actually rely on.

Why FTP stopped working

Recently I got pissed off with having to deploy a site I was working on using FTP. Hell, right? Opening a new connection for every single file you have in your project. Using a more complex setup and dependency management tools meant that my projects started having a lot more files than a simple 5-pager with just HTML, CSS, and JS files. With a PHP project written in Laravel, complete with Composer and npm dependencies, the file count shot over 9000. So, if you want a quick upload of a project you can forget uploading the vendor folder every time.

Next problem you face is if you change multiple files, how will you be sure that you transferred every file you changed to the staging server? You won't! You must test it manually.

Sending the project to my server for a client to check was pretty easy by SSH-ing in and doing git pull. I know this is not a great idea but this was how I did it back then. Realising that the client's live server wouldn't have Git installed, I needed to find another solution.

Getting started with scp

Facing this problem I managed to get SSH access to the staging server and that's where all the fun began. I've written my first deploy bash script that looked somehow like this:

deploy.sh

#!/usr/bin/env bash
# Currently ~8:30 minutes for all files

scp -r ./* root@example.com:/var/www/test-deploy/
ssh root@example.com "chown -R www-data:www-data /var/www/test-deploy"

OK, not the best thing you've seen but it worked. And yeah, I felt like some hacker or something. And it was good. I guess you noticed the runtime that was around 8:30 minutes. Mhm, 8 minutes. Bad, but it was still better than the uploading process with FTP that lasted more than half an hour. One of the reasons behind this is pretty bad internet that I have at my place (upload was especially bad). Thank you BT!

scp gets you off FTP, but copying everything every time doesn't scale. You're paying the full transfer cost on every deploy, regardless of what actually changed.

Splitting deploys by area

Realising that this still can't be acceptable I've created two more scripts called deploy-app.sh and deploy-admin.sh. Both of those were handling two separate parts of the website. Let's take a look at one of them:

deploy-app.sh

#!/usr/bin/env bash

# add variable for remote server app path

scp -r ./app/config/ root@example.com:/var/www/test-deploy/app/config
scp -r ./app/controllers/ root@example.com:/var/www/test-deploy/app/controllers
scp -r ./app/models/ root@example.com:/var/www/test-deploy/app/models
scp -r ./app/filters.php ./app/routes.php root@example.com:/var/www/test-deploy/app/
scp -r ./public/site/ ./app/routes.php root@example.com:/var/www/test-deploy/public/site
ssh root@example.com "chown -R www-data:www-data /var/www/test-deploy"

Yeah, loads of repetitive code and not really the nicest thing for the eyes. But, this could still help me speed up the process of deploying, or should I rather call it uploading the website since I don't have anything else than moving files to server in this process. At least I thought this would speed up the process. In the end I realised I am not using helper scripts as I was mostly changing app and admin part of the website at the same commits.

Splitting by directory helps when you're only touching one area but maintaining parallel scripts drifts fast once you change both sides in the same commit.

Adding variables

But, writing those scripts helped me realise I could have some variables that will help remake script more readable. So, next step was changing deploy script to something like this:

#!/usr/bin/env bash
username=epicusername
host=example.com
project=/var/www/test-deploy

scp -r ./app/config/ $username@$host:$project/app/config
scp -r ./app/controllers/ $username@$host:$project/app/controllers
scp -r ./app/models/ $username@$host:$project/app/models
scp -r ./app/filters.php ./app/routes.php $username@$host:$project/app/
scp -r ./public/site/ ./app/routes.php $username@$host:$project/public/site
ssh $username@$host "chown -R www-data:www-data /var/www/test-deploy"

OK. This made the script look a little bit better but adding variables for only one line of code did not make much sense, and also it didn't help improving my 'deploy' process. Next thing that came to my mind was that I could somehow make a one-liner out of this:

#!/usr/bin/env bash
username=epicusername
host=example.com
project=/var/www/test-deploy

scp -r ./app ./bootstrap/ ./public/ ./vendor/ ./readme.md ./server.php $username@$host:$project/

OK, we made this a little bit better. Variables buy readability; collapsing paths buys simplicity. Neither fixes the underlying problem where you're still shipping the whole tree.

Compressing before transfer

Next thing that came to my mind was archiving and compressing those files instead of just sending them to the server. That looked something like this:

#!/usr/bin/env bash

# VARIABLES
username=epicusername
host=example.com
project=/var/www/test-deploy
now=`date +%Y-%m-%d.%H_%M_%S`

gulp styles

zip -r compressed.zip ./app ./bootstrap ./public ./vendor ./readme.md ./server.php -x ".git/*" -x ".DS_Store" -x "app/config/local/*" -x "app/config/test/*" -x "app/config/testing/*" -x "public/images/uploaded/*" -x "app/storage/*"
# ssh $username@$host "zip -r backup_$now.zip $project" #>> deploy.log
scp compressed.zip $username@$host:$project #>> deploy.log
rm -rf ./compressed.zip #>> deploy.log
ssh $username@$host "cd $project && unzip -o compressed.zip && rm -rf compressed.zip" #>> deploy.log

OK, now we have a compressed file that contains all the files necessary for running the website. Pretty easy stuff that improves your deploy time. Compression cuts upload time on slow connections which is worth it when your upload speed is the bottleneck, not the server.

After this I was thinking how can this be improved even more. It crossed my mind that I don't have to send the vendor folder to the server every single time, because those files are not the ones that are constantly getting changed. One idea to solve this is to create a switch in the script which will deploy only part of the website.

The rsync breakthrough

I decided not to go with that because I remembered the command that one of my colleagues told me - rsync.

As the general description from rsync says, "Rsync copies files either to or from a remote host, or locally on the current host (it does not support copying files between two remote hosts)." The biggest advantage of rsync is that it does not just copy all the files you tell it to, it syncs them. This means that we will not send files that are not required (same on the local computer and on the server). Reading this, I've realised that this is exactly what I want.

The initial script looked somehow like this:

#!/usr/bin/env bash

# VARIABLES
username=epicusername
host=example.com
project=/var/www/test-deploy
now=`date +%Y-%m-%d.%H_%M_%S`

rsync -azP --exclude-from='.exclude-files' ./ $username@$host:$project

# .exclude-files
.*
deploy-staging.sh
deploy.sh

package.json
gulpfile.js
phpunit.xml
server.php
composer.lock
composer.json
artisan

node_modules

app/config/local
app/config/test
app/config/remsen

app/storage/cache/*
app/storage/logs/*
app/storage/meta/*
app/storage/sessions/*
app/storage/views/*

public/site/images/uploaded
public/site/assets/scss
public/admin/less
public/site/project-pdfs

This is where the iterations paid off. rsync -azP only transfers what changed, respects an exclude file, and preserves permissions. The .exclude-files list keeps dev tooling, local config, and user uploads out of the sync (the stuff that should never overwrite production).

What I learned

Every version of this script solved a real problem: FTP was unbearable, scp was a stepping stone, zip helped on a slow uplink, and rsync was the tool that matched how deploys actually work. None of it was wasted and each iteration taught me what the next constraint was.

If you're still clicking through FileZilla, start with scp. When that hurts, you'll know exactly why.