I'm a fan of automating my personal infrastructure as much as possible. Therefore, I wrote a little utility called upstream-watch, that is well integrated in my existing git-ops workflow. I wrote this tool to support my personal container based infrastructure, which is completely managed via a single git repository.

What is GitOps?

GitOps upholds the principle that Git is the one and only source of truth. GitOps requires the desired state of the system to be stored in version control such that anyone can view the entire audit trail of changes. All changes to the desired state are fully traceable commits associated with committer information, commit IDs and time stamps. This means that both the application and the infrastructure are now versioned artifacts and can be audited using the gold standards of software development and delivery.

-- ^[1]

So all needed configurations for a service should be managed in a git repository. And if changes are necessary these changes must be commited to the repository and will then be carried out to the actual server. This will ensure two things: First you also have a clear state of your infrastructure and can roll back to a working version (if there were no database migrations executed, but we do have backups for this case, right?) Secondly you can set up the infrastracture on a new machine in minutes, rather than hours or days.

In general, I would also add that GitOps enables CI and CD for your applications/services. And you also can apply software engineering best practices like reviews. So there is no good reason not to use some basic form of GitOps.

Features

Short overview about the features of upstream-watch:

• regulary pull upstream repository
• check if something changed
• execute configured hooks and update routine, if update is necessary
• keep track if an update was already executed, to don't deacrease service downtime by executing multiple updates
• hooks and update process are completely scriptable

Overview

upstream-watch will continously pull an upstream git repository and check if there are changes on the origin and wil pull these to the local repository. If yes, it will execute pre-configured hooks before and after updating the service. This will ensure upstream-watch is compatible with almost all services and a useful tool in an operators toolbelt.

The layout of my infrastructure as code repository looks like this:

    .
    ├── .upstream-watch.yaml
    ├── README.md
    ├── service-1
    │   ├── .update-hooks.yaml
    │   ├── docker-compose.yml
    │   └── README.md
    ├── service-2
    │   ├── .update-hooks.yaml
    │   ├── docker-compose.yml
    │   └── README.md
    └── upstream-watch

Main configuration

The .upstream-watch.yaml is the main configuration file for this instance of upstream-watch. You can set the retry interval (in seconds) and folders that should be ignored. The single_directory_mode: false disables this mode and enables the presented structure with subfolders per service.

    single_directory_mode: false
    retry_interval: 60
    ignore_folders: [".git", ".test"]

Service configuration

In case of an update to any of these files in a subfolder, upstream-watch will execute the pre- and post-hooks defined in the corresponding .update-hooks.yaml, before and after the service is updated. In this case all services are using docker compose.

An example for a .update-hooks.yaml:

    pre_update_commands:  ["docker compose down"]
    update_commands:      ["docker compose pull"]
    post_update_commands: ["docker compose up -d"]

This provides much needed flexibilty to be able to use upstream-watch for almost everything, not only docker compose or any other container mangement tool.

Update process

This results in the following update process for our example service with integration of renovate or dependabot:

1. Dependabot or Renovate opens an MR/PR with an update.
2. Review change and merge to main.
3. upstream-watch will pull the changes.
4. and execute the configured hooks and update routine.

upstream-watch will track if a service was already updated by storing the commit of the submodule and its update status in a local sqlite database.

Some internals

upstream-watch relies on git to determine if a subfolder (with a service in it) was changed between commits. To persist if an update was already executed it relies on in a simple sqlite database, that is stored in the repository.

The used sqlite libary needs CGO to be enabled, therefore there are no macOS or Windows builds right now. I still have to figure out how to build these on different GitHub Action VMs (with goreleaser).

A major thing I plan to add soon is integration of webhooks, to get an alert, if something went wrong. With that, there is theoretically no need to login on the server, after the inital service deployment. Another usefull feature would be to be able to deploy upstream-watch as container aside the exisiting services. For some deployments that could mean, that the upstream-watch container needs access to the docker-socket, but this is solvable (e.g. watchtower does excatly that).

Conclusion

It was fun writing a little utility that I can directly use for my production services. If you need something like this, go and checkout upstream-watch at github.com/andresterba/upstream-watch.

1. https://www.cloudbees.com/gitops/what-is-gitops ↩