I have been meaning to write about this for a while. Since most of my work revolves around building API's and deploying them, it is pretty important for me (and people like me) to automate as many things as possible in order to get the best out of my time and available resources.

My current tech stack revolves around the following:

Python (that wasn't hard to guess, huhh!?)
FastAPI (For the API)
Celery (For workers to do tasks in the background)
Docker (Why not!?)
GitHub (Version control)
DigitalOcean (Server)

How do the above fit in the bigger picture?

I will try to keep things as simple as possible.

I use Python as my primary language to build API's. FastAPI is a Python framework that makes building production API's way simpler. Celery is a Python tool/library that queues tasks and does them in the background.

Using the above three makes my API fast and production ready.

I use Docker for containerizing my API. The code for the API and Docker is stored on GitHub. Finally, the docker container is deployed on a VPS on DigitalOcean so that my API can be interected with by the people and most importantly, my apps!

When all of them are used together, I have a fast, production ready API that is accessible by my apps.

Typical Scenario

Let's consider two locations:

My PC (where I write code)
The server (where the API is deployed)

In order for my code to go from the first destination (my PC) to the second destination (my server), it has to go through a few steps.

Now, let's say we don't know something called pipeline exists. A typical way for a code change to go from the start point to the destination would be the following steps:

Code pushed to GitHub.
I ssh into my server.
Pull the latest code changes.
Build the latest docker image.
Deploy.

Here's a simple question, out of all the above steps, which ones require me to do them manually or even a part of them manually?

I need to push the code myself, I need to ssh into my server myself. I need to pull the latest code changes myself. I need to build the docker image myself. I need to deploy the docker image myself.

So, as it turns out, all the steps require me to give them some manual input.

Remove redundant steps

Now, out of the above, lets try to filter out the redundant steps. Redundant steps are those that can be executed by my computer with minimal interaction (can be passed through env variables) for me.

Redundant steps are those that can be executed by my computer with minimal interaction (can be passed through env variables) for me.

We can make ssh into server automated since we just need to pass an ssh key.
We can automate pulling the latest code changes.
We can automate the docker image build and push process since it's the same step being repeated.
We can automate the deployment of the latest docker image since that doesn't require any input.

What is GitHub Actions (or anything similar)?

GitHub Actions is a Continuous Integration service. It is very much similar to Travis CI and Jenkins but it is built by GitHub so that obviously comes with perks if we're using GitHub.

We pass GitHub Actions a few steps to follow (remember those redundant steps?). It will gracefully follow all the steps and the end result should be our code being deployed.

Basically, we are defining a path that our code will follow in order to reach their destination (the server) but without any human interaction.

One thing we need to keep in mind is that GitHub Actions will actually run the steps we ask it to run in a server somewhere but not in our server neither in our computer. This is why we say pipelines build on the fly.

We can easily specify which OS GitHub Action uses in order to run the steps. (More on that below).

How do we tell GitHub Actions to do something?

In order to let GitHub know that we want it to follow a few steps and do an action, we can do that by passing a yaml file. This file needs to follow the GitHub workflow syntax.

We can define jobs in the file. Each job can have a few steps that are supposed to be followed. Jobs can be defined in the following way:

jobs:
  # Name of the job
  push-to-docker:
    runs-on: ubuntu-latest # Use latest Ubuntu for the steps
    steps:
      # Do something
      - name: Do something

Removing redundant steps

Now that we know that we can use GitHub Actions to automate our steps, let's see how we can make sure that GitHub Actions follows each step.

NOTE: From now on, we need to keep in mind that the steps are being written for GitHub Actions.

Automate pulling the latest code changes

There is a neat action that is provided by GitHub itself. GitHub Actions Checkout checks out the latest code from the repository we're running the action on.

This means, we can just use this package to pull our latest changes. Now in whatever server GitHub Action is running, we will basically clone the latest changes from our repo to that server.

Following is the code to tell Actions to checkout the latest code from the current repo.

jobs:
  example-job:
    runs-on: ubuntu-latest
    steps:
      # Checkout the code
      - name: Checkout Repo
        uses: actions/checkout@v2

The above code tells action to start a new job named example-job on the latest version of Ubuntu and run the following steps.

The first step pulls the latest changes of the current repo to that OS.

Automate building the docker image and pushing it

Now, since we have the latest code changes avaiable, we can easily build a docker image and push it to Docker Hub or GitHub Container Registry.

In order to do that, we can use the package build-push-action.

However, docker or GitHub doesn't just let anyone push an image to the container registry. This means, we need to login to docker in order to push the latest built image. We can do that by using the login-action package from docker.

Adding the above steps to the yaml file will make it look like following:

jobs:
   example-job:
    runs-on: ubuntu-latest
    steps:
      # Checkout the code
      - name: Checkout Repo
        uses: actions/checkout@v2
      # Login to docker
      # This can be used for both Docker Hub and
      # GitHub container registry.
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          # Remove the following line if you want to
          # login to docker hub.
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          # secrets are GitHub actions that can be added
          # from settings of the repo.
          password: ${{ secrets.CR_PAT }}
      # Build the docker image and push it.
      - name: Build image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          # Remove the `gchr.io/` if you're pushing to
          # Docker Hub
          tags: ghcr.io/owner/packageName:latest

With the above code, we have already removed three steps that we were doing manually. Now we can just ssh into our server, pull the latest image and deploy it. But wait a minute, shouldn't we automate that too?

Automate SSH into server and deploy latest image

Now we have just one more step left. We want our GitHub Action to deploy the latest image in our server as well. Since our action is already running on a machine, can't we just pass a few steps to make the machine ssh into the actual server (the destination) and then deploy the changes?

Yes we can! We will use the ssh action package for that. This package will do what we have been doing manually. SSH into the server and deploy.

Let's add a new job that will wait for our first job to complete and then ssh into our server and deploy the changes:

  # Name of the job is deploy
  deploy:
    needs: job1
    runs-on: ubuntu-latest
    steps:
      # SSH into the server
      - name: SSH into server
        uses: appleboy/ssh-action@master
        with:
          # The server IP of our VPS from DigitalOcean
          # or any other provider. This is again
          # passed in as a secret.
          host: ${{ secrets.SERVER_IP }}
          # Username of the user sshing into the server
          username: ${{ secrets.SERVER_USERNAME }}
          # The private ssh key in order to get access
          key: ${{ secrets.KEY }}
          script: docker pull ghcr.io/owner/packageName:latest && docker run ghcr.io/owner/packageName:latest

So the above job takes care of deploying as well.

Note that you might have a more complex deploy script for Docker in order to consider things like if the container is already running and ports etc.

Final steps

Now that we have our steps file in place, we need to tell GitHub to run it when we push something. We can do that by saving the file in the .github/workflows/ directory of our project. So if the name of the file is build.yml, we will have the following directory structure.

-- project
  |-- .github
    |-- workflows
      |-- build.yml

Once the file is located like above, it will be automatically picked up by GitHub. However, we might not want the action to run when we push to certain branches. The file should just run on a few branches (let's say staging and production), we can do that by the following:

on:
  push:
    branches:
      - "production"
      - "staging"

We can also give the action a name by the name syntax:

name: Build API and Deploy

Adding secrets

Now, you might not want sensitive data to be passed on along with the code. Especially if the repo is public. In a case like this, we can use GitHub's secrets feature to store sensitive data like ssh private key etc.

This question on StackOverflow explains how to add secrets on GitHub

Conclusion

We started with 5 steps, each one of them required manual interaction. Now we are left with one step (git commit) that requires interaction. All the other steps are automatically handled by our CI service (GitHub). CI is a very useful aspect of deployment, especially if you're a web developer.

When we add all the jobs and steps to one file, it turns out as following:

name: Build API and Deploy

on:
  push:
    branches:
      - "production"
      - "staging"

jobs:
  example-job:
    runs-on: ubuntu-latest
    steps:
      # Checkout the code
      - name: Checkout Repo
        uses: actions/checkout@v2
      # Login to docker
      # This can be used for both Docker Hub and
      # GitHub container registry.
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          # Remove the following line if you want to
          # login to docker hub.
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          # secrets are GitHub actions that can be added
          # from settings of the repo.
          password: ${{ secrets.CR_PAT }}
      # Build the docker image and push it.
      - name: Build image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          # Remove the `gchr.io/` if you're pushing to
          # Docker Hub
          tags: ghcr.io/owner/packageName:latest
  # Name of the job is deploy
  deploy:
    needs: example-job
    runs-on: ubuntu-latest
    steps:
      # SSH into the server
      - name: SSH into server
        uses: appleboy/ssh-action@master
        with:
          # The server IP of our VPS from DigitalOcean
          # or any other provider. This is again
          # passed in as a secret.
          host: ${{ secrets.SERVER_IP }}
          # Username of the user sshing into the server
          username: ${{ secrets.SERVER_USERNAME }}
          # The private ssh key in order to get access
          key: ${{ secrets.KEY }}
          script: docker pull ghcr.io/owner/packageName:latest && docker run ghcr.io/owner/packageName:latest

Adding the above workflow requires just a few seconds but it highly removes the manual work for me. I can now just make some changes and merge them to a branch and it will be automatically picked up by my pipeline and deployed in a matter of seconds.

If you'd like to see one of these in action.

CI can also be used for things like testing. The actions can be run on various events like Pull Requests.

Posts for docker

Deploy Celery & RabbitMQ with Compose

Github, Python and Docker: The Sweet Trio

A handy deployment script for your pipeline