I’m a fan of automating my personal infrastructure as much as possible. Therefore, I wrote a little utility called upstream-watch, that is well integrated in my existing git-ops workflow. I wrote this tool to support my personal container based infrastructure, which is completely managed via a single git repository.
What is GitOps?
GitOps upholds the principle that Git is the one and only source of truth. GitOps requires the desired state of the system to be stored in version control such that anyone can view the entire audit trail of changes. All changes to the desired state are fully traceable commits associated with committer information, commit IDs and time stamps. This means that both the application and the infrastructure are now versioned artifacts and can be audited using the gold standards of software development and delivery.
So all needed configurations for a service should be managed in a git repository. And if changes are necessary these changes must be commited to the repository and will then be carried out to the actual server. This will ensure two things: First you also have a clear state of your infrastructure and can roll back to a working version (if there were no database migrations executed, but we do have backups for this case, right?) Secondly you can set up the infrastracture on a new machine in minutes, rather than hours or days.
In general, I would also add that GitOps enables CI and CD for your applications/services. And you also can apply software engineering best practices like reviews. So there is no good reason not to use some basic form of GitOps.
Short overview about the features of
- regulary pull upstream repository
- check if something changed
- execute configured hooks and update routine, if update is necessary
- keep track if an update was already executed, to don’t deacrease service downtime by executing multiple updates
- hooks and update process are completely scriptable
upstream-watch will continously pull an upstream
git repository and check if there are changes on the origin and
wil pull these to the local repository.
If yes, it will execute pre-configured hooks before and after updating the service.
This will ensure
upstream-watch is compatible with almost all services and a useful tool
in an operators toolbelt.
The layout of my infrastructure as code repository looks like this:
. ├── .upstream-watch.yaml ├── README.md ├── service-1 │ ├── .update-hooks.yaml │ ├── docker-compose.yml │ └── README.md ├── service-2 │ ├── .update-hooks.yaml │ ├── docker-compose.yml │ └── README.md └── upstream-watch
.upstream-watch.yaml is the main configuration file for this instance of
You can set the retry interval (in seconds) and folders that should be ignored.
single_directory_mode: false disables this mode and enables the presented structure with subfolders per service.
single_directory_mode: false retry_interval: 60 ignore_folders: [".git", ".test"]
In case of an update to any of these files in a subfolder,
upstream-watch will execute the pre- and post-hooks
defined in the corresponding
.update-hooks.yaml, before and after the service is updated.
In this case all services are using
An example for a
pre_update_commands: ["docker compose down"] update_commands: ["docker compose pull"] post_update_commands: ["docker compose up -d"]
This provides much needed flexibilty to be able to use
upstream-watch for almost everything, not only
or any other container mangement tool.
This results in the following update process for our example service with integration of renovate or dependabot:
- Dependabot or Renovate opens an MR/PR with an update.
- Review change and merge to
upstream-watchwill pull the changes.
- and execute the configured hooks and update routine.
upstream-watch will track if a service was already updated by storing the commit of the submodule and its update
status in a local sqlite database.
upstream-watch relies on
git to determine if a subfolder (with a service in it) was changed between commits.
To persist if an update was already executed it relies on in a simple sqlite database, that is stored in the
The used sqlite libary needs CGO to be enabled, therefore there are no macOS or Windows builds right now.
I still have to figure out how to build these on different GitHub Action VMs (with
A major thing I plan to add soon is integration of webhooks, to get an alert, if something went wrong.
With that, there is theoretically no need to login on the server, after the inital service deployment.
Another usefull feature would be to be able to deploy
upstream-watch as container aside the exisiting services.
For some deployments that could mean, that the
upstream-watch container needs access to the docker-socket, but
this is solvable (e.g.
watchtower does excatly that).
It was fun writing a little utility that I can directly use for my production services.
If you need something like this, go and checkout
upstream-watch at github.com/andresterba/upstream-watch.