A simple zero downtime deploy bash script

I appreciate AWS and Terraform and use it regularly, but find it can be over-kill for small side projects as simple as a single Go binary.

With some low traffic side projects, having a second or so of downtime is acceptable. With others, it is not. In those cases, I now have a bash script for zero downtime deployments that can be used on any server or low-end box.

The success criteria

• I want a simple method to push up Go binaries to a low spec vps and run my service there.
• It has to be a zero downtime deploy.
  • We need two instances running side by side.
  • We need to ensure the new instance is healthy before killing the old instance.
  • We need to clean up after ourselves.
• It needs to be simple and extendable.

The code

First your program has to be able to accept a port to bind to on startup in the form of ./myservice 8082.

Pushing the binary up to the server (Makefile)

We now need a way to push the Go binary up to the server and kick off the whole zero downtime deployment process. For that, as this is a side project, I’m happy using a Makefile on my laptop. This could easily be converted GitHub Actions if you were working in a small team setting.

deploy:
	cd service && GOOS=linux GOARCH=amd64 go build && mv myservice tmp
	cd service; scp boot.sh tmp user@ip:~/myservice
	ssh -t user@ip 'cd myservice; bash boot.sh'
	cd service; rm tmp

The deployment code (boot.sh)

Here the bash script on the server checks if there is already a service bound to 8082, and if not we use that port. However, if there is already a service bound to that port we make use of 8083 and get ready to terminate the process on 8082.

We then make an http request to localhost:$PORT and if that returns as expected, we then update the nginx, reload it, then start killing off and cleaning up the old process.

SERVICENAME="myservice"
NGINXCONFIG="mysite.com"
SERVERSTARTEDHTTPCODE=404 # Go default index is a "404 page not found"

# See which version is running
EXISTING=$(lsof -t -i :8082)

if [ "$EXISTING" != "" ]; then
  NEWPORT=8083
  OLDPORT=8082
else
  NEWPORT=8082
  OLDPORT=8083
fi

NEWNAME=$SERVICENAME$NEWPORT
mv tmp $NEWNAME

# Spin up new instance
nohup ./$NEWNAME $NEWPORT > server.log 2>&1 &

# Allow service time to spin up
sleep 2

# Get the HTTP status code to make sure it's started up correctly
HTTPSTATUS=$(curl -s -o /dev/null -w "%{http_code}" "localhost:$NEWPORT")
if [ "$HTTPSTATUS" -eq "$SERVERSTARTEDHTTPCODE" ]; then

    # Swap out the Nginx config for the new port
    sed -i "s#.*proxy_pass.*#        proxy_pass         \"http://127.0.0.1:$NEWPORT\";#" /etc/nginx/sites-enabled/$NGINXCONFIG

    sleep 1

    # Gracefully restart Nginx
    sudo nginx -s reload

    # Kill the old service
    PID=$(lsof -t -i :$OLDPORT)
    if [ -n "$PID" ]; then
        # Send the SIGTERM signal to gracefully terminate the process
        kill -15 "$PID"
        rm $SERVICENAME$OLDPORT
        echo "Gracefully terminated $SERVICENAME with PID: $PID" >> server.log
    else
        echo "No process found for $SERVICENAME" >> server.log
    fi

else
    echo "Server did not start. Deployment aborted. (HTTP status code: $HTTPSTATUS)" >> server.log
    exit 1
fi

Future improvements

One nice improvement would be to use systemctl so that its managed, easy to control, has a watchdog, and the logging is consistent with other services.

I can also think of ways of cleaning this up and making it more defensive, though as a quick and scrappy script it does the job nicely for now. Maybe if this side project takes off I’ll graduate it to the cloud, but for now a low end box does the trick.