Image</th>	Efficiency</th>	Wasted Bytes</th>	User Wasted Percent</th></tr></thead>
buildpacks-test-pack-go</td>	92.1218 %</td>	15691768 bytes (16 MB)</td>	36.1717 %</td></tr>
buildpacks-test-pack-tiny</td>	99.9975 %</td>	1497 bytes (1.5 kB)</td>	0.0116 %</td></tr>
buildpacks-test-docker</td>	100 %</td>	0 bytes (0 B)</td>	NaN %</td></tr> </tbody></table> As expected the docker</code> image build by docker has 0 bytes wasted as it was based on a scratch</code> image, and we just copied the go binary in the image. But the buildpacks-tiny</code> image is also very close with an efficiency of 99.9975 % and only 1.5 kB wasted. As expected the buildpacks-go</code> image, which is based on paketobuildpacks/builder:base</code> (so not</strong> tiny), is the largest image with 16 MB wasted. The largest file in that image seems to be a perl runtime at /usr/bin/perl</code> with 4.2 MB.</p> `I'm not entirely sure how dive calculates these metrics, as we know that our binary has the size of ~6.6 MB, that leaves us with more than 20 MB in the buildpacks-test-pack-tiny</code> image, that is maybe not needed. Parts of it could be the SBOM or vendored modules, but as said earlier I'm not sure about this.</p>` $</span> CI=</span>true</span> dive</span> buildpacks-test-pack-go</span></span> </span> Using</span> default</span> CI</span> config</span></span> Image</span> Source:</span> docker://buildpacks-test-pack-go</span></span> Fetching</span> image...</span> (this</span> can</span> take</span> a</span> while</span> for</span> large</span> images</span>)</span></span> Analyzing</span> image...</span></span> efficiency:</span> 92.1218</span> %</span></span> wastedBytes:</span> 15691768</span> bytes</span> (16</span> MB</span>)</span></span> userWastedPercent:</span> 36.1717</span> %</span></span> Inefficient</span> Files:</span></span> Count</span> Wasted</span> Space</span> File</span> Path</span></span> 2</span> 4.2</span> MB</span> /usr/bin/perl</span></span> 2</span> 1.3</span> MB</span> /var/cache/debconf/templates.dat</span></span> 2</span> 1.1</span> MB</span> /usr/lib/x86_64-linux-gnu/perl-base/auto/re/re.so</span></span></code></pre>$</span> CI=</span>true</span> dive</span> buildpacks-test-pack-tiny</span></span> </span> Using</span> default</span> CI</span> config</span></span> Image</span> Source:</span> docker://buildpacks-test-pack-tiny</span></span> Fetching</span> image...</span> (this</span> can</span> take</span> a</span> while</span> for</span> large</span> images</span>)</span></span> Analyzing</span> image...</span></span> efficiency:</span> 99.9975</span> %</span></span> wastedBytes:</span> 1497</span> bytes</span> (1.5</span> kB</span>)</span></span> userWastedPercent:</span> 0.0116</span> %</span></span> Inefficient</span> Files:</span></span> Count</span> Wasted</span> Space</span> File</span> Path</span></span> 2</span> 1.0</span> kB</span> /etc/os-release</span></span> 2</span> 339</span> B</span> /etc/passwd</span></span> 2</span> 140</span> B</span> /etc/group</span></span></code></pre>$</span> CI=</span>true</span> dive</span> buildpacks-test-docker</span></span> </span> Using</span> default</span> CI</span> config</span></span> Image</span> Source:</span> docker://buildpacks-test-docker</span></span> Fetching</span> image...</span> (this</span> can</span> take</span> a</span> while</span> for</span> large</span> images</span>)</span></span> Analyzing</span> image...</span></span> efficiency:</span> 100.0000</span> %</span></span> wastedBytes:</span> 0</span> bytes</span> (0</span> B</span>)</span></span> userWastedPercent:</span> NaN</span> %</span></span> Inefficient</span> Files:</span></span> Count</span> Wasted</span> Space</span> File</span> Path</span></span> None</span></span></code></pre>SBOM</a></h2> After talking about the SBOM multiple times now, I of course want to see how it looks for my images. To the SBOM you can use pack sbom download <image></code>. This will export the SBOM layer of the image in the local directory in the folder layers</code>.</p> So for the tiny image it looks like this:</p> $</span> tree</span> layers</span></span> </span> layers</span></span> └──</span> sbom</span></span> └──</span> launch</span></span> ├──</span> paketo-buildpacks_ca-certificates</span></span> │ </span> └──</span> helper</span></span> │ </span> └──</span> sbom.syft.json</span></span> ├──</span> paketo-buildpacks_go-build</span></span> │ </span> └──</span> targets</span></span> │ </span> ├──</span> sbom.cdx.json</span></span> │ </span> ├──</span> sbom.spdx.json</span></span> │ </span> └──</span> sbom.syft.json</span></span> └──</span> sbom.legacy.json</span></span> </span> 7</span> directories,</span> 5</span> files</span></span></code></pre> There are multiple SBOMs in 3 formats CycloneDX</a>, Syft</a> and Spdx</a>. All of them look a bit different, but contain more or less the same content, like used go compiler version, build flags and dependencies (that is why I've added zap</code> as logger dependency). 5</a></sup></p> Using Buildpacks on GitLab CI</a></h2> I was thinking about using it in a container, like with a docker in docker build setup, but in the end I reused my existing shell runner.</p> There are ways on how to use it directly within a container, but these are rather cumbersome in comparison to using plain pack</code>. 6</a></sup></p> Therefore, I deduced to just reuse an existing shell runner, and install pack on it. This has the large disadvantage of still needing a privileged runner with root permissions, but in theory you could use Podman (or something else) instead of docker in your runner and with this build images without root permissions needed (which is equal to be in the docker</code> group).</p> With that I used the following GitLab CI configuration:</p> v</span>ariables</span>:</span></span> B</span>UILDPACK_VERSION</span>:</span> 0.29.0</span></span> </span> b</span>ackend-build</span>:</span></span> t</span>ags</span>:</span></span> -</span> d</span>ocker-build</span></span> s</span>tage</span>:</span> b</span>uild</span></span> b</span>efore_script</span>:</span></span> -</span> d</span>ocker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY</span></span> s</span>cript</span>:</span></span> -</span> c</span>d backend</span></span> -</span> (</span>curl -sSL "https://github.com/buildpacks/pack/releases/download/v${BUILDPACK_VERSION}/pack-v${BUILDPACK_VERSION}-linux.tgz" \| tar -C . --no-same-owner -xzv pack)</span></span> -</span> .</span>/pack build gitlab.cloudf.de:4567/andre/shorty/backend:latest --path . --buildpack paketo-buildpacks/go --builder paketobuildpacks/builder:tiny --creation-time now</span></span> -</span> d</span>ocker push gitlab.cloudf.de:4567/andre/shorty/backend:latest</span></span></code></pre> The build times are also very good, once the first build has passed and layers can be reused. For a production project, I've seen build times of < 1 minute with code changes in the code and very limited network upload speeds.</p> Conclusion</a></h2> If you know how to write Dockerfiles, Buildpacks might be a bit weird at the beginning as there is no need to define all dependencies needed by your application in a file. All of this complexity moved in the builders, which can be beneficial as you can provide your own images, for example in a business environment.</p> The downside is that you have limited control of what will be in your image at the end, as the builder has full control over it.</p> In general I like the speed and the earlier mentioned capabilities of Buildpacks. The only thing missing is signing support, but otherwise I'll try Buildpacks for some private projects in the future and may think about using it at work (but with custom builders).</p> Sources</a></h2> https://github.com/golang/go/issues/57120 ↩</a></p> </li> https://github.com/buildpacks/rfcs/pull/195 ↩</a></p> </li> https://buildpacks.io/docs/concepts/#what-is-a-builder ↩</a></p> </li> https://paketo.io/docs/howto/go/ ↩</a></p> </li> https://buildpacks.io/docs/features/bill-of-materials/ ↩</a></p> </li> https://blog.codecentric.de/en/2021/10/gitlab-ci-paketo-buildpacks/ ↩</a></p> </li> </ol> </section> Improve your Go Code with Benchmarks 2023-03-31T00:00:00+00:00 We often write code without thinking about how performant the code will be in the end. Often it is also not worth to sacrifice cleaner code in terms of readability and future maintainability for the sake of a few percents of performance, be it less CPU cycles needed to compute something or a smaller memory footprint.</p> But as always even if you don't need to save a few cycles or bytes, it is good to know what you could do to even get started in optimize for performance. This post will about how to use the tooling provided by the Go programming language.</p> The Scenario</a></h2> Let's do something very easy! We just want to generate a combination 6 random characters out of the alphabet (A-Z and a-z) and integers ranging from 0 to 9. The output could for example look like this bGrS7X</code>. What could that be good for? I've got no idea, but it is a very easy example on how to use the extensive tooling of Go.</p> You can find all code for this post here</a>.</p> What tools do we need?</a></h2> We want multiple metrics for a given algorithm. For example, we want to know how many CPU cycles it needs to perform a specific operation. Additionally, we want to know how much memory it needs to do that. So let's get started with benchmarks.</p> Benchmarks</a></h2> As always Go already provides all we need in the testing</code> package. 1</a></sup> As the documentation states you just have to add the -bench</code> flag to go test</code> and write benchmarks following the presented examples.</p> So let me introduce a very simply algorithm to fulfill to earlier presented requirements of creating a random string.</p> func</span> algStringCasting</span>(</span>)</span> string</span> {</span></span> //</span> Stolen from https://blog.pratimbhosale.com/building-a-url-shortener-using-go-and-sqlite#heading-shortening-the-url</span></span> randomPart</span> :=</span> "</span>"</span></span> for</span> i</span> :=</span> 0</span>;</span> i</span> <</span> 6</span>;</span> i</span>++</span> {</span></span> randomPart</span> +=</span> string</span>(</span>rune</span>(</span>rand</span>.</span>Intn</span>(</span>26</span>)</span> +</span> 97</span>)</span>)</span></span> }</span></span> </span> return</span> randomPart</span></span> }</span></span></code></pre> A simple benchmark for this algorithm would then look like this.</p> func</span> Benchmark_algStringCasting</span>(</span>b</span> </span>testing</span>.</span>B</span>)</span> {</span></span> for</span> n</span> :=</span> 0</span>;</span> n</span> <</span> b</span>.</span>N</span>;</span> n</span>++</span> {</span></span> algStringCasting</span>(</span>)</span></span> }</span></span> }</span></span></code></pre> In this case there is no need to increase any external variable to run it with a different set of inputs, as the algorithm should be static. To get some results we then just execute go test -bench=. ./..</code>, which will give us the following.</p> ➜</span> 03-generate-random-chars</span> git:</span>(</span>main</span>)</span> ✗</span> go</span> test</span> -</span>bench=.</span> ./..</span></span> goos:</span> linux</span></span> goarch:</span> amd64</span></span> pkg:</span> github.com/andresterba/code-for-blog/2023/03</span></span> cpu:</span> Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i5-8265U</span> CPU</span> @</span> 1.60GHz</span></span> Benchmark_algStringCasting-8</span> 1389026</span> 869.3</span> ns/op</span></span> PASS</span></span> ok</span> github.com/andresterba/code-for-blog/2023/03</span> 5.753s</span></span></code></pre> The first number we can see 1389026</code> is the number of runs of this function. The second number is the needed time in average in nanoseconds per operations. The -8</code> suffix shows the number of CPUs used to run the benchmark. [3]</p> As we want more stable results and remove statistical outliers, we always want to run benchmarks multiple times. We can simply do this by adding the -benchtime</code> flag with the special syntax 10x</code> to run the benchmark 10 times.</p> So now we got some metrics for CPU and runtime, but we also want to now how much memory the algorithm is using and how many allocations it needs. To get this we just have to add another flag to the test command, which is -benchmem</code>. With these two additional flags we get the following output.</p> go</span> test</span> -</span>bench=.</span> -</span>benchmem</span> -</span>benchtime=100x</span> ./...</span></span> goos:</span> linux</span></span> goarch:</span> amd64</span></span> pkg:</span> github.com/andresterba/code-for-blog/2023/03</span></span> cpu:</span> Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i5-8265U</span> CPU</span> @</span> 1.60GHz</span></span> Benchmark_algStringCasting-8</span> 100</span> 464.8</span> ns/op</span> 56</span> B/op</span> 11</span> allocs/op</span></span> PASS</span></span> ok</span> github.com/andresterba/code-for-blog/2023/03</span> 0.004s</span></span></code></pre> Each operation allocations 56 bytes of memory in 11 allocations.</p> Add another algorithm</a></h2> Now that we have a working benchmark setup, let's add another algorithm that solves the same problem and use the provided tooling to compare it against our first try.</p> Instead of generating random chars, we will now provide a lookup array and just create random indexes for that array.</p> var</span> charArray</span> [</span>62</span>]</span>string</span> =</span> [</span>62</span>]</span>string</span>{</span></span> "</span>a</span>"</span>,</span> "</span>b</span>"</span>,</span> "</span>c</span>"</span>,</span> "</span>d</span>"</span>,</span> "</span>e</span>"</span>,</span> "</span>f</span>"</span>,</span> "</span>g</span>"</span>,</span></span> "</span>h</span>"</span>,</span> "</span>i</span>"</span>,</span> "</span>j</span>"</span>,</span> "</span>k</span>"</span>,</span> "</span>l</span>"</span>,</span> "</span>m</span>"</span>,</span> "</span>n</span>"</span>,</span></span> "</span>o</span>"</span>,</span> "</span>p</span>"</span>,</span> "</span>q</span>"</span>,</span> "</span>r</span>"</span>,</span> "</span>s</span>"</span>,</span> "</span>t</span>"</span>,</span> "</span>u</span>"</span>,</span></span> "</span>v</span>"</span>,</span> "</span>w</span>"</span>,</span> "</span>x</span>"</span>,</span> "</span>y</span>"</span>,</span> "</span>z</span>"</span>,</span></span> "</span>A</span>"</span>,</span> "</span>B</span>"</span>,</span> "</span>C</span>"</span>,</span> "</span>D</span>"</span>,</span> "</span>E</span>"</span>,</span> "</span>F</span>"</span>,</span> "</span>G</span>"</span>,</span></span> "</span>H</span>"</span>,</span> "</span>I</span>"</span>,</span> "</span>J</span>"</span>,</span> "</span>K</span>"</span>,</span> "</span>L</span>"</span>,</span> "</span>M</span>"</span>,</span> "</span>N</span>"</span>,</span></span> "</span>O</span>"</span>,</span> "</span>P</span>"</span>,</span> "</span>Q</span>"</span>,</span> "</span>R</span>"</span>,</span> "</span>S</span>"</span>,</span> "</span>T</span>"</span>,</span> "</span>U</span>"</span>,</span></span> "</span>V</span>"</span>,</span> "</span>W</span>"</span>,</span> "</span>X</span>"</span>,</span> "</span>Y</span>"</span>,</span> "</span>Z</span>"</span>,</span></span> "</span>0</span>"</span>,</span> "</span>1</span>"</span>,</span> "</span>2</span>"</span>,</span> "</span>3</span>"</span>,</span> "</span>4</span>"</span>,</span> "</span>5</span>"</span>,</span> "</span>6</span>"</span>,</span></span> "</span>7</span>"</span>,</span> "</span>8</span>"</span>,</span> "</span>9</span>"</span>,</span></span> }</span></span> </span> func</span> algArrayLookup</span>(</span>)</span> string</span> {</span></span> //</span> Stolen from https://www.geeksforgeeks.org/program-generate-random-alphabets/</span></span> res</span> :=</span> "</span>"</span></span> </span> for</span> i</span> :=</span> 0</span>;</span> i</span> <</span> 6</span>;</span> i</span>++</span> {</span></span> res</span> =</span> res</span> +</span> charArray</span>[</span>rand</span>.</span>Intn</span>(</span>62</span>)</span>]</span></span> }</span></span> </span> return</span> res</span></span> }</span></span></code></pre> The benchmark looks exactly the same as the one for our first algorithm, we just call algArrayLookup</code> instead of algStringCasting</code>.</p> func</span> Benchmark_algArrayLookup</span>(</span>b</span> </span>testing</span>.</span>B</span>)</span> {</span></span> for</span> n</span> :=</span> 0</span>;</span> n</span> <</span> b</span>.</span>N</span>;</span> n</span>++</span> {</span></span> algArrayLookup</span>(</span>)</span></span> }</span></span> }</span></span></code></pre> And the results? They look like this:</p> go</span> test</span> -</span>bench=.</span> -</span>benchmem</span> -</span>benchtime=100x</span> ./...</span></span> goos:</span> linux</span></span> goarch:</span> amd64</span></span> pkg:</span> github.com/andresterba/code-for-blog/2023/03</span></span> cpu:</span> Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i5-8265U</span> CPU</span> @</span> 1.60GHz</span></span> Benchmark_algStringCasting-8</span> 100</span> 874.9</span> ns/op</span> 56</span> B/op</span> 11</span> allocs/op</span></span> Benchmark_algArrayLookup-8</span> 100</span> 395.0</span> ns/op</span> 24</span> B/op</span> 5</span> allocs/op</span></span> PASS</span></span> ok</span> github.com/andresterba/code-for-blog/2023/03</span> 0.004s</span></span></code></pre> So our second algorithm takes less time per operation, bytes per operation and requires only 5 allocations instead of 11. It might be a good idea to add benchmarks like this to your code base, before you make any major changes in the implementation of a specific functionality. But how do you know if your new implementation is better/faster than the old one?</p> Of course, you can just keep the old benchmarks lying around and check them manually. But there is also a tool for that, called benchstat</a>.</p> Testing with different inputs</a></h2> So let's come up with another example, where we don't want to generate a static element, which is independent of some input. Let's use a well known algorithm that scales with a provided input, for this example we are using the Euclidean algorithm</a>.</p> It is used to find the greatest common divisor of two numbers. An example from the Wikipedia is 21 is the GCD of 252 and 105. Pseudocode for this looks like this. 2</a></sup></p> function gcd(a, b)</span></span> while b ≠ 0</span></span> t := b</span></span> b := a mod b</span></span> a := t</span></span> return a</span></span></code></pre> In Go this will look like this.</p> func</span> Euclid</span>(</span>a</span> int</span>,</span> b</span> int</span>)</span> int</span> {</span></span> for</span> b</span> !=</span> 0</span> {</span></span> t</span> :=</span> b</span></span> b</span> =</span> a</span> %</span> b</span></span> a</span> =</span> t</span></span> }</span></span> </span> return</span> a</span></span> }</span></span></code></pre> And the output for our example 252 and 105 will be as expected:</p> ➜</span> go</span> run</span> main.go</span></span> </span> GCD</span> of</span> 252</span> and</span> 105</span> is</span> 21.</span></span></code></pre> Then let's write a benchmark for this with different inputs. We define a table called inputs</code> to collect a list of inputs we want to benchmark. b.Run()</code> accepts as first argument the name of the benchmark.</p> var</span> inputs</span> =</span> [</span>]</span>struct</span> {</span></span> a</span> int</span></span> b</span> int</span></span> }</span>{</span></span> {</span></span> a</span>:</span> 252</span>,</span></span> b</span>:</span> 105</span>,</span></span> }</span>,</span></span> {</span></span> a</span>:</span> 1337</span>,</span></span> b</span>:</span> 105</span>,</span></span> }</span>,</span></span> }</span></span> </span> func</span> BenchmarkEuclid</span>(</span>b</span> </span>testing</span>.</span>B</span>)</span> {</span></span> for</span> _</span>,</span> v</span> :=</span> range</span> inputs</span> {</span></span> b</span>.</span>Run</span>(</span>fmt</span>.</span>Sprintf</span>(</span>"</span>a:</span>%d</span> b:</span>%d</span>"</span>,</span> v</span>.</span>a</span>,</span> v</span>.</span>b</span>)</span>,</span> func</span>(</span>b</span> </span>testing</span>.</span>B</span>)</span> {</span></span> for</span> i</span> :=</span> 0</span>;</span> i</span> <</span> b</span>.</span>N</span>;</span> i</span>++</span> {</span></span> Euclid</span>(</span>v</span>.</span>a</span>,</span> v</span>.</span>b</span>)</span></span> }</span></span> }</span>)</span></span> }</span></span> }</span></span></code></pre> The result looks like this.</p> go</span> test</span> -</span>benchmem</span> ./..</span></span> </span> goos:</span> linux</span></span> goarch:</span> amd64</span></span> pkg:</span> github.com/andresterba/code-for-blog/2023/03/euclid</span></span> cpu:</span> Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i5-8265U</span> CPU</span> @</span> 1.60GHz</span></span> BenchmarkEuclid/a:252_b:105-8</span> 21771715</span> 50.82</span> ns/op</span> 0</span> B/op</span> 0</span> allocs/op</span></span> BenchmarkEuclid/a:1337_b:105-8</span> 17221270</span> 58.49</span> ns/op</span> 0</span> B/op</span> 0</span> allocs/op</span></span> PASS</span></span> ok</span> github.com/andresterba/code-for-blog/2023/03/euclid</span> 4.139s</span></span></code></pre> Interesting is that the compiler seems to optimize the code to remove the allocation t := b</code>. But profiling is a different topic I will maybe cover in the near future. If you want to know more about how to test with variable inputs, I can recommend this 3</a></sup> blog post.</p> Conclusion</a></h2> Writing benchmarks in Go is very easy, so there should be no reason to not do it for performance critical parts of your application. As always to Go tooling is very handy and easy to use.</p> https://pkg.go.dev/testing#hdr-Benchmarks ↩</a></p> </li> https://en.wikipedia.org/wiki/Euclidean_algorithm ↩</a></p> </li> https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go ↩</a></p> </li> </ol> </section> Why you should update your dependencies automatically 2023-02-28T00:00:00+00:00 When was the last time you decided to update the dependencies of one of your projects? And not because you needed a new feature, just to get the latest security patches for your project which might be running on the public internet? Well yes, I got the same problem: never. I didn't update most dependencies of my own software until sometime earlier last year.</p> My TL;DR today is that the setup was very easy, and I can sleep better now, as I know I will receive a notification when one of my applications is vulnerable. The two systems I have used to achieve this are Dependabot</a> for GitHub projects and renovate</a> for GitLab projects in my private instance of GitLab.</p> These tools do work with a variety of different package managers, e.g. go</code>, npm</code>, pip</code> and even docker</code>.</p> Features</a></h2> The main feature of automated updates is rather obvious: you get new upstream (security) fixes automatically. And you get them via a new PR/MR in your repository, so there is no need to keep track of used dependencies and manually subscribe to new releases. What kind of updates you get depends on your configuration and the locked version in the lock file of your package manager. Normally I tend to enable all updates as most of the time I got at least some tests to cover the main business logic of my app (or even some end-to-end tests), so I can be sure it will work with a new version of a dependency.</p> Requirements</a></h2> One thing that is needed to use these systems are tests. Without tests, you can never be sure if a package update will result in broken features. This is especially the case for frontend applications which tend to rely on a lot of external packages (mostly true for the JS/TS ecosystem in general). In this case unit tests are often not enough, as you also want to update e.g. your ORM or database driver and these are not covered by unit tests.</p> If you got enough trusts in your tests you can even automatically merge MRs/PRs and ensure that everything is still working as expected. In my opinion this is the goal you should aim for.</p> The process</a></h2> Let's go over how this whole setup works with GitHub's Dependabot. At first, you have to enable Dependabot for your repository. This is as easy as adding the following file to your repository as .github/dependabot.yaml</code>:</p> v</span>ersion</span>:</span> 2</span></span> u</span>pdates</span>:</span></span> -</span> p</span>ackage-ecosystem</span>:</span> "</span>gomod</span>"</span></span> d</span>irectory</span>:</span> "</span>/</span>"</span></span> s</span>chedule</span>:</span></span> i</span>nterval</span>:</span> "</span>daily</span>"</span></span></code></pre> Then you have to wait for dependency updates or a vulnerability.</p> Once you got a vulnerable dependency in one of your projects you will be notified via mail and get a Dependabot alert.</p> </p> Dependabot will then create a pull request in the affected project.</p> </p> All you have to do is to merge this PR, once you ensured it doesn't break anything of course.</p> Risks</a></h2> All this sounds just too good to be true, so I won't hide the fact that it also opens up new attack vectors.</p> The first one is dependency confusion, which (in short) can be used to replace your private dependencies by malware infected ones. This is only possible because some dependency management systems (like npm or pip) rely on public services, that host these packages and</strong> they will look there first, even if you provided a private package registry. 1</a></sup></p> Another risk is that you pull malicious code in your repository, because an at the time of creating your project trustworthy author abandoned his project and gave it to someone with bad intentions or the project got attacked and some attacker added the malicious code to the code. 2</a></sup></p> There are multiple more examples and attack vector on software supply chains documented on the internet (for example 3</a></sup> 4</a></sup> 5</a></sup>).</p> Another risk (well the risk you always have with software) is introducing new bugs, as new features getting added. You can sometimes mitigate this by only relying on security updates, but eventually you have to migrate to a newer and support version of a dependency.</p> When to add a new dependency?</a></h2> This is the question you should always ask yourself before adding a new dependency to a project, especially when you want to maintain it for a long time. The answer is coupled to the ecosystem you use as in how rich the standard library of the language is and on how much work you want to do yourself (in re-inventing the wheel). If you are using a language with a rich standard library like Go and</strong> are willing to write some components like a more advanced HTTP server yourself, you can get around most of these problems.</p> On the other hand you might invest your time in re-inventing components other people already did (and maybe better than you did it in terms of security or performance). One should weigh up which decision is the right one here.</p> The future of our software supply chains</a></h2> Your dependencies of a project are your software supply chain. The name suggests that each of the components are interconnected and you must ensure at least some sort of integrity of these packages. I became aware of the topic through some scientific publications. 6</a></sup> 7</a></sup> It is a topic of research and other languages or build systems already provide checksums or other mechanisms to mitigate or minimize risks of being a consumer of other software.</p> Conclusion</a></h2> In my opinion tools like Dependabot or renovate are an easy way to keep your software up to date and secure. They are also only consumers in the whole supply chain and therefore can't mitigate the earlier mentioned risks alone, but they are an essential part when developing software.</p> https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 ↩</a></p> </li> https://thehackernews.com/2021/07/malicious-npm-package-caught-stealing.html ↩</a></p> </li> https://secureteam.co.uk/news/vulnerabilities/javascript-supply-chain-attack-hits-millions-of-users/ ↩</a></p> </li> https://blog.jscrambler.com/how-your-code-dependencies-expose-you-to-web-supply-chain-attacks ↩</a></p> </li> https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/ ↩</a></p> </li> https://arxiv.org/abs/2204.04008 ↩</a></p> </li> https://dl.acm.org/doi/abs/10.1145/3510457.3513044 ↩</a></p> </li> </ol> </section> Use git worktree to work with multiple branches 2023-01-16T00:00:00+00:00 This post is about the awesome git worktree</code>, that simplified my personal workflow with git very much. In the past I had at least 2 versions of repository checked out locally or fiddled around with git stash</code> to find a thing I did weeks ago. But let's talk about how I use git and how git worktree</code> helped me.</p> Use Cases</a></h2> When I'm reviewing a Pull or Merge Request, sometimes I would like to see the code in my own IDE instead of the GitLab or GitHub UI. Normally I have a single clone of a repository on my local machine and</strong> I'm in the process of adding new features or fixing a bug myself. So I don't want to commit things just yet, but just check out another branch. I know I could use git stash</code> or create a WIP</code> commit and then amend by providing a good commit message later, but almost all the time I forget about my existing stash or the earlier WIP</code> commit.</p> Another time I'm starting to work on an improvement or simply experimenting with an implementation detail and therefore also don't want to commit anything just yet.</p> At last, I want to update one of my branches with the current state of the main</code> branch or want to verify that something is working or not working on the current main</code> branch in comparison to my branch.</p> Working with worktree</a></h2> The solution to all of these problems is to use git worktree</code> and have multiple branches checkout out locally at the same time.</p> To be as flexible as possible I tend to use the following structure for all repositories where I think I'll need multiple branches checkout out at the same time:</p> ➜</span> upstream-watch</span> pwd</span></span> /home/aps/Code/github/upstream-watch</span></span> ➜</span> upstream-watch</span> ls</span> -</span>l</span></span> total</span> 8</span></span> drwxr-xr-x</span> 5</span> aps</span> aps</span> 4096</span> Jan</span> 11</span> 08:27</span> my-feature-1</span></span> drwxr-xr-x</span> 6</span> aps</span> aps</span> 4096</span> Jan</span> 11</span> 08:26</span> upstream-watch-main</span></span></code></pre> So I got a <repo-name>--main</code> for the main branch and additional branches was <repo-name>--xxx</code>.</p> Another feature you get with this is you get a clean state and can open your IDE in a new folder. This could be annoying if you have a lot of custom configuration (like local .env</code> files, which are ignored by git), but for most of the changes I don't need these.</p> The main clone of the repository is always repository-name-main</strong>. To not pollute my local file structure I tend to have each project in a dedicated directory under my ~/Code</code> folder.</p> To create a new worktree checkout you simply have to use git worktree add -b my-feature-1 ../my-feature-1</code>. If you don't want to create a new branch then simply skip the -b my-feature-1</code> from the command above and use it like this git worktree add ../my-feature-1</code>.</p> You can also list all your existing worktrees with from any worktree of the cloned repository:</p> ➜</span> my-feature-1</span> git:</span>(</span>my-feature-1</span>)</span> git</span> worktree</span> list</span></span> /home/aps/Code/github/upstream-watch/upstream-watch-main</span> c477b51</span> [main</span>]</span></span> /home/aps/Code/github/upstream-watch/my-feature-1</span> c477b51</span> [my-feature-1</span>]</span></span></code></pre> I'm lazy, so I'm mostly using this very simple wrapper script to create new worktrees. Then I'll simply call cw my-feature</code>, when you store the scrip below as cw</code> (short for create worktree) in your local bin</code> folder.</p> #!</span>/bin/bash</span></span> set</span> -</span>euo</span> pipefail</span></span> </span> name</span>=</span>$</span>1</span></span> </span> git</span> worktree</span> add</span> ../</span>$</span>name</span> -</span>b</span> $</span>name</span></span> </span> printf</span> "</span>Created worktree at: %s \n</span>"</span> "</span>../</span>$</span>name</span>"</span></span></code></pre> Deleting a worktree is as simple as git worktree remove ../my-feature-1</code>.</p> This also enables the usage of git cherry-pick</code> between multiple worktrees, which is very neat when you find a bug while working on another task. Simply commit it on your current working branch and pick it into a fix-bug-xxx</code> branch afterwards and create a dedicated MR/PR for this fix.</p> Conclusion</a></h2> git worktree</code> simplified my personal workflow with git and saves some (not so expensive anymore) space on my local disk, as I don't need multiple versions of a repository on my local machine. I know worktree</code> is a rather old git feature, but nevertheless it is very useful and I know a lot of people that are not using it (yet).</p> Organize your Semester with Kanban inspired Methods 2022-11-30T00:00:00+00:00 I recently finished my studies, so this seems a good time to reflect about how (and why) things worked out. As always I like to share some insights in what I've learned in the last 5 years. This is the second part of my series about how to study. You can find the first part about how to write a thesis here</a></p> Studying is more than about learning new things. It is also about learning how to prioritize your available time and energy to achieve externally set goals. And often these goals are not very specific, but rather vague and open to exploration. In this post I'll present my Kanban inspired workflow to keep track of what I have to do within the semester and hand in all assignments in time.</p> What is Kanban?</a></h2> Kanban is a popular Lean workflow management method for defining, managing, and improving services that deliver knowledge work. It helps you visualize work, maximize efficiency, and improve continuously. -- 1</a></sup></cite></p> </blockquote> Kanban is an agile project management method used in very different projects. Its origins are in car manufacturing (Toyota), but it can be used in almost every context. You can learn more about Kanban here 1</a></sup> and here 2</a></sup>. I used it at work before, especially in projects with small teams as it has a very small management overhead. Additionally, I used it with friends to work on projects together or for university related projects with multiple other students.</p> Another flavor of Kanban is Personal Kanban</code>, which as the name suggests, is tailored to be used for your private projects or day to day tasks 3</a></sup>. The general ideas I am using for my projects (yes I'm still using this setup today, but not as detailed as shown later) are:</p> Visualize Your Work</strong> : We use a board to visualize the tasks that are currently in progress, tasks that need to be finished sooner or later and tasks that are already finished. I enjoy a full lane of finished tasks, and it can provide you some motivation for not so much interesting tasks! Additionally, you get what you have to do today with a simple look on your board.</p> Limit Work-in-Progress</strong> : You only have a limit amount of time available, so you have to get as much done with your available time as possible.</p> There are a few other Kanban rules, which in my opinion are only useful or relevant if you use Kanban in a team and not for your own time management 1</a></sup>.</p> My method</a></h2> The idea is not</strong> to work 60 hours a week, but to use your available time as efficient and effective as possible. My method is a combination of Kanban and other agile methods. I tailored these to my needs as a student with limited time and many focus shifts a day, so I heavily relied on the Pareto principle 4</a></sup>. So let's talk about the things you need to get started.</p> Digital Helpers</a></h3> A digital helper to have your board always a browser tab away or in your pocket can be very beneficial. Especially if you are working from different places most of the time and not on your desk at home.</p> Personally I used the tool Trello</a> for this. There are a lot of other tools, but I started with Trello years ago and never found something that simply worked, so I was stuck with Trello. Just a side-note: I also build some tooling around Trello, to improve my (very opinionated) workflows around Trello. You can find this tool here</a>, but be warned: it is very opinionated and not polished at all.</p> The main task for your digital helpers is to keep your focus on the board and make adding or modifying tasks easy. If the board looks somehow good and inspiring to you, that is even better!</p> Lanes</a></h3> I used 10 lanes to manage the tasks I had to do for one week (and a small peak in the near future). This seems much, but don't worry, we won't need all of these lanes every single day. To be exact you only need a single lane per day, after you have done some initial planning at the start of the week.</p> </p> In the following we will go more in to detail what each of these lanes I used for.</p> Backlog : The backlog is used for all tasks that are not yet scheduled. It can contain long-running tasks like writing a short paper or programming on a larger project. I prefer to keep track of these tasks by creating sub-tasks that can be scheduled in a week and track them via lists within these epic tasks.</p> Next Week : This lane contains items that must be finished next week. In my case we had lab preparations to do every 2 weeks, so I wanted to schedule them right away, while still working on the current task set of the lab.</p> This Week : This lane contains tasks that should be finished this week, but are not yet scheduled.</p> Monday : As the name suggests these are all tasks that should be finished on Monday.</p> Tuesday : As the name suggests these are all tasks that should be finished on Tuesday.</p> Wednesday : As the name suggests these are all tasks that should be finished on Wednesday.</p> Thursday : As the name suggests these are all tasks that should be finished on Thursday.</p> Friday : As the name suggests these are all tasks that should be finished on Friday.</p> Saturday/Sunday : As the name suggests these are all tasks that should be finished on the weekend days. I try not to schedule to many tasks on weekends, as at least at the weekend, you should get some rest and enjoy a not fully organized day.</p> Done this week : This lane will be used to collect all tasks that were finished this week. This is by far my favorite lane!</p> The setup provides the first thing we need for the Kanban methods, a visualization of our work. Besides that this will be your goto point if you are not sure what to do in a free hour between two lectures.</p> Labels</a></h3> Labels are very useful to determine in which category a task belongs to. In my case I simply used to courses of the current semester as labels.</p> </p> Additionally, I added the organization</code> label for tasks that don't belong to a specific course, for example think about the topic of a thesis or organize a meeting with fellow students.</p> The usage of different colors provides a quick overview about what course or topic has a lot of open tasks.</p> Planning</a></h2> In the following we will talk about the major processes needed to keep your board up to date and how to add new tasks. Further, I will present some thoughts about how to get the actual work done and if you should do a retrospective.</p> Adding new task</a></h3> For every task you have to complete you should create a new card. A task should be as small as possible. So if you have for example an assignment that includes reading multiple papers or book chapters, you should create an epic to keep track of the whole process, but individual task for e.g. each paper or chapter. This will ensure that you can finish at least a few tasks a week and stay you motivated (at least that worked for me).</p> To make your planning easier you should estimate how much time you need to finish a task. I know estimating is hard, but for some tasks this is rather simple. 5</a></sup> For example you can easily estimate how much time you need to read a 5 pages paper, as you have done that often in the past.</p> </p> Let's have a look at a simple example: We have to read 'Modern Operating Systems' by Tanenbaum until the end of the semester. At first, you should create an epic to track the whole book and the progress of reading it. Then create a new task for each of the chapters and estimate how long will take to read it (e.g. 1 hour). You can always update your estimates if you think they are wrong in the future!</p> I prefer to add the needed time before the title, but you could also use labels or any other method that works for you, to achieve the same. Important is that it is easily visible how long you estimated the tasks will take. We will see why in a moment.</p> Weekly Planning:</a></h3> Normally I used about 15 minutes on Saturday or Sunday to plan the next week. This is where the estimations for tasks are used to not get overworked in the first few weeks of the semester.</p> So let's say I want to work 40 hours a week on university related tasks. Then count the time you have to use for lectures, let's use 15 hours, and subtract that from your initial work time a week. With that we have 25 hours left to fill with assignments or other things we have to do. Now you can decide if 5 days are enough or if you need some time on the weekends, but let's start with work days, as the weekend work will start sooner or later.</p> You now have different options on how to distribute that time through the week. Personally I prefer to keep it more or less even, so if you have a day with multiple lectures or labs, then don't schedule additional 5 hours of work to that day! Try to work with your fixed hours a day first (e.g. 8 hours), before starting to increase it.</p> Then do a quick prioritization of the tasks in your backlog, to do the things first, that must be finished this week. I recommend to work on the tasks in order of the earliest submission date, so work first on the tasks you have to finish first. Then simply add as many tasks as you have time left on a specific day.</p> If you couldn't finish a task that was scheduled for this day, simply move it to the next day. And don't panic! That's quite normal, and you will get a feeling of what you can accomplish in a day or hour.</p> I know that sounds like a very time intense process, but it is not! You don't have to do planning on a single day, but can also plan while you are adding a new task that came up after a lecture or lab. Therefore, you will only spend 15 minutes or half an hour a week on actual planning and have more time to execute what you have planned.</p> Do the work</a></h3> This is the part where no fancy project management method can help you. You have to do the task you have planned for that day. Sometimes these may be boring, but you have scheduled them on this day, because there won't be much time left elsewhere this week. Keep the Pareto principle in mind, when you are working on something you don't enjoy at all.</p> An additional motivation could be the free evening or weekend, after you have finished your tasks of the day.</p> Day to Day Interactions with your Board</a></h3> Your board will tell you what to do today. Sometimes I also added lectures as tasks of that day, to see all my to-dos of today at a single place.</p> There are just a few rules to keep in mind when working on tasks:</p> If you have finished a task, then congratulations and simply move it to Done this week</code>.</li> If you get a new assignment then add it to your general Backlog</code> , Next week</code> or if it is urgent This week</code>.</li> If you couldn't finish a task on the day it was scheduled on, then think about refining it by splitting it into multiple tasks or/and move it to another day.</li> </ol> Your board is your single point of truth what you have to work on each day. You don't have to think about which assignment is more important, when you have an hour left between two lectures. And if there is nothing left to do for today and you still have some time, while staying at campus, why not take some time to get a coffee/tea and chat with some friends or enjoy a good book?</p> Retrospective</a></h3> To be honest I never did a proper retrospective while the semester was still running. If something went terribly wrong and I didn't manage to get all the tasks done I had scheduled, I shifted priorities between assignments and invested less time in these (again Pareto). Sometimes I did some analysis at the end of the semester to calculate how many hours I worked a week, but this was only the see where all my time went in the last 4 weeks.</p> Conclusion</a></h2> And this is the whole method I used to stay on top of my assignments and getting tasks finished in time. Now in retrospective I think it sounds like much more work than it actually was. I did experiment with other methods (for example the Bullet Journal method), but to be honest nothing felt as easy as the earlier presented process 6</a></sup>. If you have any feedback or experience with time management while being a student or afterwards, don't hesitate to let me know! I'm happy to hear from others on how they manage their very limited time and get the most out of it.</p> Sources</a></h2> https://kanbanize.com/kanban-resources/getting-started/what-is-kanban ↩</a> ↩2</a> ↩3</a></p> </li> https://www.planview.com/resources/guide/introduction-to-kanban/ ↩</a></p> </li> https://www.personalkanban.com/ ↩</a></p> </li> https://en.wikipedia.org/wiki/Pareto_principle ↩</a></p> </li> https://uwaterloo.ca/ist-project-management-office/methodology/project-management/planning/project-schedule/task-estimating ↩</a></p> </li> https://bulletjournal.com/ ↩</a></p> </li> </ol> </section> Speaking at OpenRheinMain 2022 2022-09-30T00:00:00+00:00 I had the chance to speak at a small conference at my home university about the project (goSDN</a>) I did work on the last 6 months. This is also the project I worked on while writing my thesis, therefore I was pretty confident I had enough insides to answer most of the questions.</p> I thought about speaking at a conference before, but most of the time I felt like I couldn't provide much insight in topics or my understanding of the technology I used simply wasn't deep enough. But this conference was the perfect opportunity to change this! The conference was rather small with about ~200 attendees, and it was at my home university, so I had good chances of getting accepted as speaker. Which actually happened, so I had a few weeks to prepare the presentation.</p> The presentation itself went quite well I think. I got some interesting questions afterwards and some companies got interested in what we are building, so hopefully the research group can get more funding.</p> This was my first experience as a speaker at a conference and I really enjoyed it. Hopefully I will be able to speak at a conference again in the future.</p> Build LaTeX with GitLab-CI 2022-08-21T00:00:00+00:00 LaTeX is a software for document preparation. You write your content in plain text .tex</code> files, much like a very basic programming language, and compile it afterwards in the designated format, e.g. pdf</code>. LaTeX is heavily used in academia as it provides an easy way to express mathematical formulae. Besides that most universities provide a template, so you don't have to think about margins, citation style (and more) and can simply focus on your content. And this is exactly the reason why I use LaTeX.</p> Overview</a></h2> As a computer science student I had to write lab reports and finally a final thesis. Most of these were written with LaTeX, as it is easy to add formulas, and it looks quite nice. It is also very convenient if you get a template, so you can focus on your content instead of optics of your document. As a developer it is also easy to use your favorite version management tool, as TeX files are simply text file. But it can be annoying to set up LaTeX on your local machine, therefore I tend to use the same container on my local machine as on a CI server, to ensure I can always build my document with the same version of all needed libraries.</p> Template</a></h2> As I use this setup to have reproducible build of my thesis, I rely on a template provided by my university. If you are interested in the template you can find it here</a>. The template also provides a simple Makefile</code>, which I'll use later. Besides styling the template also provided Dockerfile, but these are based on Ubuntu and I prefer to use the same versions of tools than on my local machine, so I will stick with Arch Linux for the containers as well.</p> The build container</a></h2> To ensure I have the latest release of all used libraries, I use Arch Linux as base layer. It's also handy that Arch provides all needed LaTeX libraries in the distribution repositories.</p> FROM</span> archlinux:latest</span></span> </span> RUN</span> pacman -Syu --noconfirm texlive-most texlive-latexextra texlive-bibtexextra make ghostscript biber</span></span></code></pre> After building the image with docker build -t gitlab.com/andre/masterarbeit/latex-build:ci -f Dockerfile .</code> it must be pushed to the GitLab container registry with docker push gitlab.com/andre/masterarbeit/latex-build:ci</code>. These are quite big, so be ready to wait a few minutes, depending on your internet speed.</p> GitLab CI</a></h2> This setup requires that you have already set up a runner for your GitLab project and can use the internal container registry.</p> With the artifacts</code> setting, GitLab will expose the created PDF directly from the overview page of your run. This is useful if you are working on a machine without TeX installed, but still want to see if you can build your PDF after some changes. The setting expire_in: 4 week</code> ensures that I don't keep all build versions forever on the server. make all</code> in the script</code> section refers to the earlier mentioned Makefile provided by the template, you can find it here</a>.</p> c</span>ompile-thesis</span>:</span></span> i</span>mage</span>:</span> g</span>itlab.com/andre/masterarbeit/latex-build:ci</span></span> s</span>cript</span>:</span></span> -</span> m</span>ake all</span></span> a</span>rtifacts</span>:</span></span> p</span>aths</span>:</span></span> -</span> t</span>hesis.pdf</span></span> e</span>xpire_in</span>:</span> 4</span> week</span></span></code></pre>Local Build</a></h2> To ensure I get the same results locally, I also use the earlier build container for local builds. The following script will simply mount the current directory in the container and execute make all</code>. It also deletes the container afterwards.</p> #!</span>/bin/sh</span></span> </span> docker</span> run</span> -</span>-name</span> paper-build</span> -</span>-user</span> $(</span>id</span> -</span>u</span>)</span>:</span>$(</span>id</span> -</span>g</span>)</span> -</span>-volume</span> $(</span>pwd</span>)</span>:/thesis-template/</span> gitlab.com/andre/masterarbeit/latex-build:ci</span> make</span> -</span>-directory</span> /thesis-template/</span> all</span></span> docker</span> rm</span> paper-build</span></span></code></pre>Conclusion</a></h2> The setup is rather simple, but I don't want to have build problems on my local machine, with the day of delivery in mind. Also, I can give prove readers access to the artifacts, so they always read the latest version.</p> Uptime monitoring with Prometheus and Grafana 2022-07-27T00:00:00+00:00 There are existing services like pingdom</a> or uptimerobot</a> to check if your network services are up and running. But as always I like to be independent of service providers and host as much by myself as possible. So let's build our own uptime monitoring with Prometheus</a>, Grafana</a>, alertmanager</a> and the Prometheus blackbox-exporter</a>.</p> The general idea is to check if a service is up with the blackbox-exporter. Then scrape this metrics with Prometheus and store it for further usage. Build a simple dashboard with Grafana for visualization and finally setup alerting with alertmanager. As this uses a few projects on the Prometheus ecosystem, this is a perfect example on how to integreate these different components.</p> The end result will look like this: </p> What is uptime monitoring?</a></h2> In this example we will monitor http</code> services (in this case my blog) and a simple TCP</code> service (e.g. a teamspeak3 server). So what does uptime mean? In this case it means the service should be accessible via HTTP</code> or TCP</code> and therefore the user should be able to use it.</p> There are more advanced services like the already mentioned pingdom</a> where you can also check for specific parts on a HTTP</code> page, but let's go for a basic setup which will simply check if we get any http 2xx</code> response or can create a TCP connection to the service.</p> Another interesting dashboard or metric is the actual time since the last downtime of the service.</p> Setup Prometheus/Exporter/Alertmanager</a></h2> So let's get started with setting up all needed components. The directories should look like this, to be able to provide all services with the needed configurations:</p> .</span></span> ├── alertmanager</span></span> │ └── config.yml</span></span> ├── blackbox-exporter</span></span> │ └── blackbox.yml</span></span> ├── docker-compose.yml</span></span> ├── grafana</span></span> │ ├── config.monitoring.example</span></span> │ └── provisioning</span></span> │ ├── dashboards</span></span> │ │ └── dashboard.yml</span></span> │ └── datasources</span></span> │ └── datasource.yml</span></span> ├── prometheus</span></span> │ ├── prometheus.yml</span></span> │ └── uptime.rules</span></span></code></pre> The whole setup can be found in my GitHub repository</a>. But let's take a look into the most important configurations.</p> The blackbox-exporter</a> is configured via a simple blackbox.yml</code>. In this case I configured a basic HTTP</code> probe with the http_2xx</code> module and the tcp_connect</code> module. There are more modules</a> for e.g. icmp</code>, dns</code> and grpc</code>, which can be used to monitor services with other protocols.</p> m</span>odules</span>:</span></span> h</span>ttp_2xx</span>:</span></span> p</span>rober</span>:</span> h</span>ttp</span></span> t</span>imeout</span>:</span> 5</span>s</span></span> h</span>ttp</span>:</span></span> t</span>ls_config</span>:</span></span> i</span>nsecure_skip_verify</span>:</span> false</span></span> t</span>cp_connect</span>:</span></span> p</span>rober</span>:</span> t</span>cp</span></span> t</span>imeout</span>:</span> 5</span>s</span></span> t</span>cp</span>:</span></span> p</span>referred_ip_protocol</span>:</span> i</span>p4</span></span></code></pre> The next component to configure is Alertmanager, which will send alerts to a single or multiple recievers. In this case I want to be alerted via mail, but you can choose whatever reciever</a> is the perfect match for you.</p> r</span>oute</span>:</span></span> r</span>eceiver</span>:</span> '</span>default-receiver</span>'</span></span> g</span>roup_wait</span>:</span> 3</span>0s</span></span> g</span>roup_interval</span>:</span> 5</span>m</span></span> r</span>epeat_interval</span>:</span> 1</span>2h</span></span> </span> r</span>eceivers</span>:</span></span> -</span> n</span>ame</span>:</span> '</span>default-receiver</span>'</span></span> e</span>mail_configs</span>:</span></span> -</span> s</span>end_resolved</span>:</span> true</span></span> t</span>o</span>:</span> '</span>'</span></span> f</span>rom</span>:</span> '</span>monitoring@mydomain.com</span>'</span></span> s</span>marthost</span>:</span> '</span>:587</span>'</span></span> a</span>uth_username</span>:</span> '</span>hello</span>'</span></span> a</span>uth_password</span>:</span> '</span>changeme</span>'</span></span></code></pre> Now to the target configuration within the Prometheus config. Most of this config is more or less a plain Prometheus config. There are two additions:</p> Additional uptime.rules</code> with rules for the case of a service downtime.</li> A scrape_config</code> for the blackbox-exporter.</li> </ol> You can add as many targets as you want. The exporter will probe them, and you can query them with probe_success</code> via PromQL</a>.</p> g</span>lobal</span>:</span></span> s</span>crape_interval</span>:</span> 1</span>5s</span></span> e</span>valuation_interval</span>:</span> 1</span>5s</span></span> </span> e</span>xternal_labels</span>:</span></span> m</span>onitor</span>:</span> "</span>my-domain</span>"</span></span> </span> r</span>ule_files</span>:</span></span> -</span> "</span>uptime.rules</span>"</span></span> </span> a</span>lerting</span>:</span></span> a</span>lertmanagers</span>:</span></span> -</span> s</span>cheme</span>:</span> h</span>ttp</span></span> s</span>tatic_configs</span>:</span></span> -</span> t</span>argets</span>:</span></span> -</span> "</span>alertmanager:9093</span>"</span></span> </span> s</span>crape_configs</span>:</span></span> -</span> j</span>ob_name</span>:</span> '</span>blackbox-http</span>'</span></span> m</span>etrics_path</span>:</span> /</span>probe</span></span> p</span>arams</span>:</span></span> m</span>odule</span>:</span> [</span>h</span>ttp_2xx</span>]</span></span> s</span>tatic_configs</span>:</span></span> -</span> t</span>argets</span>:</span></span> -</span> h</span>ttps://sterba.dev</span></span> r</span>elabel_configs</span>:</span></span> -</span> s</span>ource_labels</span>:</span> [</span>_</span>_address__</span>]</span></span> t</span>arget_label</span>:</span> _</span>_param_target</span></span> -</span> s</span>ource_labels</span>:</span> [</span>_</span>_param_target</span>]</span></span> t</span>arget_label</span>:</span> i</span>nstance</span></span> -</span> t</span>arget_label</span>:</span> _</span>_address__</span></span> r</span>eplacement</span>:</span> b</span>lackbox-exporter:9115</span></span></code></pre> This uptime rule will alert you when a probe failed for 5 minutes.</p> g</span>roups</span>:</span></span> -</span> n</span>ame</span>:</span> u</span>ptime-rules</span></span> r</span>ules</span>:</span></span> -</span> a</span>lert</span>:</span> s</span>ite_down</span></span> e</span>xpr</span>:</span> p</span>robe_success == 0</span></span> f</span>or</span>:</span> 5</span>m</span></span> l</span>abels</span>:</span></span> s</span>everity</span>:</span> w</span>arning</span></span> a</span>nnotations</span>:</span></span> d</span>escription</span>:</span> '</span>{{ $labels.instance }} of job {{ $labels.job }} has</span></span> been down for more than five minutes.</span>'</span></span> s</span>ummary</span>:</span> I</span>nstance {{ $labels.instance }} is down</span></span></code></pre>Create the Grafana dashboard</a></h2> At first create a new dashboard and add a new variable called instance</code>. We will use this variable to make the dashboard interactive and refresh all graphs based on the select instance. </p> As displayed earlier the dashboard contains two visualizations of the blackbox-exporter. The first one is a single value that can be 0</code> of the service is down or 1</code> if the service is up. We use a singlestat</code> to display it within Grafana.</p> For the singlestat</code> we use the previously defined variable to dynamically update the statistics.</p> probe_success</span>{</span>i</span>n</span>s</span>t</span>a</span>n</span>c</span>e</span>=</span>"</span>$</span>instance</span>"</span>,</span> job=</span>"</span>blackbox-http</span>"</span>}</span></span></code></pre> </p> The second visualization is a graph of the actual probe duration. You can use it to determine if a service is responding slowly. Besides that it might be a good idea to also create an alert based on the probe duration.</p> probe_duration_seconds</span>{</span>i</span>n</span>s</span>t</span>a</span>n</span>c</span>e</span>=</span>"</span>$</span>instance</span>"</span>,</span> job=</span>"</span>blackbox-http</span>"</span>}</span></span></code></pre>Deploy with docker compose</a></h2> As always this setup will be deployed via containers. The following docker-compose.yaml</code> provides all needed services and their configurations.</p> v</span>ersion</span>:</span> "</span>3</span>"</span></span> </span> v</span>olumes</span>:</span></span> p</span>rometheus_data</span>:</span> {</span>}</span></span> g</span>rafana_data</span>:</span> {</span>}</span></span> </span> s</span>ervices</span>:</span></span> p</span>rometheus</span>:</span></span> i</span>mage</span>:</span> p</span>rom/prometheus:v2.37.0@sha256:56e7f18e05dd567f96c05046519760b356f52450c33f6e0055a110a493a41dc4</span></span> v</span>olumes</span>:</span></span> -</span> .</span>/prometheus/:/etc/prometheus/</span></span> -</span> p</span>rometheus_data:/prometheus</span></span> c</span>ommand</span>:</span></span> -</span> '</span>--config.file=/etc/prometheus/prometheus.yml</span>'</span></span> -</span> '</span>--storage.tsdb.path=/prometheus</span>'</span></span> -</span> '</span>--web.console.libraries=/usr/share/prometheus/console_libraries</span>'</span></span> -</span> '</span>--web.console.templates=/usr/share/prometheus/consoles</span>'</span></span> p</span>orts</span>:</span></span> -</span> 1</span>27.0.0.1:9090:9090</span></span> l</span>inks</span>:</span></span> -</span> a</span>lertmanager:alertmanager</span></span> r</span>estart</span>:</span> a</span>lways</span></span> </span> a</span>lertmanager</span>:</span></span> i</span>mage</span>:</span> p</span>rom/alertmanager:v0.24.0@sha256:088464f949de8065b9da7dfce7302a633d700e9d598e2bebc03310712f083b31</span></span> p</span>orts</span>:</span></span> -</span> 1</span>27.0.0.1:9093:9093</span></span> v</span>olumes</span>:</span></span> -</span> .</span>/alertmanager/:/etc/alertmanager/</span></span> r</span>estart</span>:</span> a</span>lways</span></span> c</span>ommand</span>:</span></span> -</span> '</span>--config.file=/etc/alertmanager/config.yml</span>'</span></span> -</span> '</span>--storage.path=/alertmanager</span>'</span></span> </span> g</span>rafana</span>:</span></span> i</span>mage</span>:</span> g</span>rafana/grafana:9.0.5@sha256:6847695bc353a659a18c6f44bd0b5a94de5d2eeff995a671db6e982370dfe4e8</span></span> d</span>epends_on</span>:</span></span> -</span> p</span>rometheus</span></span> p</span>orts</span>:</span></span> -</span> 1</span>27.0.0.1:3000:3000</span></span> v</span>olumes</span>:</span></span> -</span> g</span>rafana_data:/var/lib/grafana</span></span> -</span> .</span>/grafana/provisioning/:/etc/grafana/provisioning/</span></span> e</span>nv_file</span>:</span></span> -</span> .</span>/grafana/config.monitoring</span></span> r</span>estart</span>:</span> a</span>lways</span></span> </span> b</span>lackbox-exporter</span>:</span></span> p</span>rivileged</span>:</span> true</span></span> i</span>mage</span>:</span> p</span>rom/blackbox-exporter:v0.21.1@sha256:7a255982b24ed3d142f78075a7d8af976355b6f62cec60c573ee115c1beb1d0c</span></span> r</span>estart</span>:</span> a</span>lways</span></span> c</span>ommand</span>:</span></span> '</span>--config.file=/config/blackbox.yml</span>'</span></span> p</span>orts</span>:</span></span> -</span> '</span>127.0.0.1:9115:9115</span>'</span></span> v</span>olumes</span>:</span></span> -</span> '</span>./blackbox-exporter/blackbox.yml:/config/blackbox.yml</span>'</span></span> </span></code></pre>Conclusion</a></h2> And that's it! This is a way to monitor most of my services, as I don't care that much about application metrics and only want to be disturbed if something is really</strong> not working (like the service is down). As always I enjoy hosting services I use myself and reuse services I'm already running. (I also use this stack to get system metrics which I don't actually use for anything but fancy graphs).</p> This way of blackbox monitoring is a good fit if you simply want to check if a service can be reached from the outside. But keep in mind, that you should not host your monitoring stack on the same host as your actual user facing services!</p> If you want to be sure that your services are reachable, you could use multiple sites for uptime checks. Or use different providers like the earlier mentioned pingdom</a> or uptimerobot</a>.</p> You can find all docker/config files at GitHub</a>.</p> GitOps with upstream-watch 2022-06-23T00:00:00+00:00 I'm a fan of automating my personal infrastructure as much as possible. Therefore, I wrote a little utility called upstream-watch</a>, that is well integrated in my existing git-ops workflow. I wrote this tool to support my personal container based infrastructure, which is completely managed via a single git repository.</p> What is GitOps?</a></h2> GitOps upholds the principle that Git is the one and only source of truth. GitOps requires the desired state of the system to be stored in version control such that anyone can view the entire audit trail of changes. All changes to the desired state are fully traceable commits associated with committer information, commit IDs and time stamps. This means that both the application and the infrastructure are now versioned artifacts and can be audited using the gold standards of software development and delivery.</p> -- 1</a></sup></cite></p> </blockquote> So all needed configurations for a service should be managed in a git repository. And if changes are necessary these changes must be commited to the repository and will then be carried out to the actual server. This will ensure two things: First you also have a clear state of your infrastructure and can roll back to a working version (if there were no database migrations executed, but we do have backups for this case, right?) Secondly you can set up the infrastracture on a new machine in minutes, rather than hours or days.</p> In general, I would also add that GitOps enables CI and CD for your applications/services. And you also can apply software engineering best practices like reviews. So there is no good reason not to use some basic form of GitOps.</p> Features</a></h2> Short overview about the features of upstream-watch</code>:</p> regulary pull upstream repository</li> check if something changed</li> execute configured hooks and update routine, if update is necessary</li> keep track if an update was already executed, to don't deacrease service downtime by executing multiple updates</li> hooks and update process are completely scriptable</li> </ul> Overview</a></h2> upstream-watch</code> will continously pull an upstream git</code> repository and check if there are changes on the origin and wil pull these to the local repository. If yes, it will execute pre-configured hooks before and after updating the service. This will ensure upstream-watch</code> is compatible with almost all services and a useful tool in an operators toolbelt.</p> The layout of my infrastructure as code repository looks like this:</p> .</span></span> ├──</span> .upstream-watch.yaml</span></span> ├──</span> README.md</span></span> ├──</span> service-1</span></span> │ </span> ├──</span> .update-hooks.yaml</span></span> │ </span> ├──</span> docker-compose.yml</span></span> │ </span> └──</span> README.md</span></span> ├──</span> service-2</span></span> │ </span> ├──</span> .update-hooks.yaml</span></span> │ </span> ├──</span> docker-compose.yml</span></span> │ </span> └──</span> README.md</span></span> └──</span> upstream-watch</span></span></code></pre>Main configuration</a></h2> The .upstream-watch.yaml</code> is the main configuration file for this instance of upstream-watch</code>. You can set the retry interval (in seconds) and folders that should be ignored. The single_directory_mode: false</code> disables this mode and enables the presented structure with subfolders per service.</p> single_directory_mode:</span> false</span></span> retry_interval:</span> 60</span></span> ignore_folders:</span> [</span>"</span>.git</span>"</span>,</span> "</span>.test</span>"</span>]</span></span></code></pre>Service configuration</a></h2> In case of an update to any of these files in a subfolder, upstream-watch</code> will execute the pre- and post-hooks defined in the corresponding .update-hooks.yaml</code>, before and after the service is updated. In this case all services are using docker compose</code>.</p> An example for a .update-hooks.yaml</code>:</p> pre_update_commands:</span> [</span>"</span>docker compose down</span>"</span>]</span></span> update_commands:</span> [</span>"</span>docker compose pull</span>"</span>]</span></span> post_update_commands:</span> [</span>"</span>docker compose up -d</span>"</span>]</span></span></code></pre> This provides much needed flexibilty to be able to use upstream-watch</code> for almost everything, not only docker compose</code> or any other container mangement tool.</p> Update process</a></h2> This results in the following update process for our example service with integration of renovate or dependabot:</p> Dependabot or Renovate opens an MR/PR with an update.</li> Review change and merge to main</code>.</li> upstream-watch</code> will pull the changes.</li> and execute the configured hooks and update routine.</li> </ol> upstream-watch</code> will track if a service was already updated by storing the commit of the submodule and its update status in a local sqlite database.</p> Some internals</a></h2> upstream-watch</code> relies on git</code> to determine if a subfolder (with a service in it) was changed between commits. To persist if an update was already executed it relies on in a simple sqlite database, that is stored in the repository.</p> The used sqlite libary needs CGO to be enabled, therefore there are no macOS or Windows builds right now. I still have to figure out how to build these on different GitHub Action VMs (with goreleaser</code>).</p> A major thing I plan to add soon is integration of webhooks, to get an alert, if something went wrong. With that, there is theoretically no need to login on the server, after the inital service deployment. Another usefull feature would be to be able to deploy upstream-watch</code> as container aside the exisiting services. For some deployments that could mean, that the upstream-watch</code> container needs access to the docker-socket, but this is solvable (e.g. watchtower</code> does excatly that).</p> Conclusion</a></h2> It was fun writing a little utility that I can directly use for my production services. If you need something like this, go and checkout upstream-watch</code> at github.com/andresterba/upstream-watch</a>.</p> https://www.cloudbees.com/gitops/what-is-gitops ↩</a></p> </li> </ol> </section> Go Generics 2022-05-01T00:00:00+00:00 The latest release of Go (1.18) added generics to the language, besides fuzzing, workspaces and performance improvements. 1</a></sup> So let's use them to implement the repository pattern, which is a way to abstract data handling from your business logic. When working on REST-like APIs you want to split handling of data from and into some kind of data store (e.g. a database, a file, ...) from business logic. This is where the repository pattern comes in handy. Especially with Go 1.18 and the introduction of generics, this is a good example for its usage to don't have to write (more or less) the same code again and again.</p> There is a more formal and longer definition, that you can check out. 2</a></sup></p> Overview</a></h2> In this example we will create a generics repository and store some objects in these. We will use simple user</code> and pet</code> structs, without any complex internals. A user looks like this:</p> type</span> User</span> struct</span> {</span></span> id</span> uuid</span>.</span>UUID</span></span> Name</span> string</span></span> Hobbies</span> [</span>]</span>string</span></span> }</span></span></code></pre> The user must also implement the user interface, as we want to use the ID()</code> method later in our generic repository.</p> type</span> User</span> interface</span> {</span></span> ID</span>(</span>)</span> uuid</span>.</span>UUID</span></span> }</span></span></code></pre>Introduction to generics</a></h2> Generics look more or less like in any other language. We will cover generic structs (and methods), generic functions, their instantiation and usage. I also don't care about error handling in all these examples, I will simply panic if anything went wrong.</p> Generic structs</a></h3> This is a generic struct of a repository:</p> type</span> GenericRepository</span>[</span>T</span> storableConstraint</span>]</span> struct</span> {</span></span> Store</span> map</span>[</span>uuid</span>.</span>UUID</span>]</span>T</span></span> }</span></span></code></pre> The repository will store added objects in a map with a uuid.UUID</code> key and a value T</code>, which will be the actual type of the struct the repository will be created for.</p> A notable thing is the storableConstraint</code>. A constraint can define methods that a struct or interface used to construct this generic repository must provide. In this case the storableConstraint</code> looks like this:</p> type</span> storableConstraint</span> interface</span> {</span></span> ID</span>(</span>)</span> uuid</span>.</span>UUID</span></span> }</span></span></code></pre> This means everything I want to store in a generic repository must implement a ID()</code> method that returns a uuid.UUID</code> attribute.</p> Let's have a look on a method on this repository, e.g. the Add</code> method.</p> func</span> (</span>t</span> </span>GenericRepository</span>[</span>T</span>]</span>)</span> Add</span>(</span>item</span> T</span>)</span> error</span> {</span></span> _</span>,</span> ok</span> :=</span> t</span>.</span>Store</span>[</span>item</span>.</span>ID</span>(</span>)</span>]</span></span> if</span> ok</span> {</span></span> return</span> nil</span></span> }</span></span> </span> t</span>.</span>Store</span>[</span>item</span>.</span>ID</span>(</span>)</span>]</span> =</span> item</span></span> </span> return</span> nil</span></span> }</span></span></code></pre> As our storableConstraint</code> ensures that every object has a ID()</code> method, we can us it to get the ID of this object. The ID will then be used as the unique identifier in our internal store.</p> Generic functions</a></h3> Besides generic structs and methods on these structs, we can also create generic functions. So lets say we want to transform one object into another one, we provide object T</code> as input, but want to get object U</code> as return value.</p> Another new thing with Go 1.18 is the any</code> keyword. any</code> replaces interface{}</code> if you don't have a type for something. In this case we use any</code> as the constraint, which means that every interface or struct is allowed to be transformed. Which is naive to use in real world application, but will suffice for this example.</p> We can create a simple function that will look like this:</p> func</span> transform</span>[</span>T</span>,</span>U</span> any</span>]</span>(</span>incoming</span> T</span>)</span> (</span>U</span>,</span> error</span>)</span> {</span></span> ...</span></span> }</span></span></code></pre> For a single generic value, the type can mostly be inferred and doesn't need to be provided while calling the function. In this case we have to provide it, so the function call looks like this:</p> pet</span>,</span> err</span> :=</span> transform</span>[</span>User</span>,</span> Pet</span>]</span>(</span>user</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> panic</span>(</span>err</span>)</span></span> }</span></span></code></pre>Instantiation</a></h2> So let's instantiate our generic repository, create a user and add it to the repository. In this case we will instantiate the repository with the user interface instead of the user struct. Please mind that the NewUser()</code> function does return a interfaces.User</code> instead of a User</code>, so we ensure that a User</code> actually implements the interface.</p> userRepository</span> :=</span> NewGenericRepository</span>[</span>interfaces</span>.</span>User</span>]</span>(</span>)</span></span> user1</span> :=</span> NewUser</span>(</span>"</span>Peter</span>"</span>,</span> [</span>]</span>string</span>{</span>"</span>1</span>"</span>,</span> "</span>2</span>"</span>}</span>)</span></span> </span> err</span> :=</span> userRepository</span>.</span>Add</span>(</span>user1</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> panic</span>(</span>err</span>)</span></span> }</span></span> </span> userFromRepo</span>,</span> err</span> :=</span> userRepository</span>.</span>Get</span>(</span>user1</span>.</span>ID</span>(</span>)</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> panic</span>(</span>err</span>)</span></span> }</span></span> </span> fmt</span>.</span>Println</span>(</span>userFromRepo</span>)</span></span></code></pre> We then create a new user and add it to the created repository. Then the repository can be queried with the user id, and we get our earlier added user back.</p> As the repository is a generic we can simply add another type, that fulfills our earlier defined constraint.</p> petRepository</span> :=</span> NewGenericRepository</span>[</span>interfaces</span>.</span>Pet</span>]</span>(</span>)</span></span> petOfPeter</span> :=</span> NewPet</span>(</span>"</span>Peter's Cat</span>"</span>,</span> user1</span>.</span>ID</span>(</span>)</span>)</span></span> </span> err</span> =</span> petRepository</span>.</span>Add</span>(</span>petOfPeter</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> panic</span>(</span>err</span>)</span></span> }</span></span> </span> petFromRepo</span>,</span> err</span> :=</span> petRepository</span>.</span>Get</span>(</span>petOfPeter</span>.</span>ID</span>(</span>)</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> panic</span>(</span>err</span>)</span></span> }</span></span> </span> fmt</span>.</span>Println</span>(</span>petFromRepo</span>)</span></span></code></pre> So as you can see, this repository can be used to store any object that implements a ID()</code> method. This is also very handy if you want to mock a repository used in tests.</p> The whole example can be found in my code repository 3</a></sup>.</p> Conclusion</a></h2> To be honest I was a bit skeptic about the addition of generics to go, as I thought with existing and good integrated tooling for code generation go didn't need them. But I was clearly wrong! The generics implementation and usage feels very simple and therefore go-like, and I really enjoy using them. Especially when writing simple APIs or mocks for some tests, generics come in very handy and can save quite some time.</p> If you are more interested in the internals, I can recommend this excellent post from Vicent Marti. It will show how go implements generics, but also how generics can make your code run slower. 4</a></sup></p> https://go.dev/blog/go1.18 ↩</a></p> </li> https://deviq.com/design-patterns/repository-pattern ↩</a></p> </li> https://github.com/andresterba/code-for-blog ↩</a></p> </li> https://planetscale.com/blog/generics-can-make-your-go-code-slower ↩</a></p> </li> </ol> </section> Install Arch Linux 2022-04-01T00:00:00+00:00 This will be a reminder for myself on how to set up a basic Arch Linux and not</strong> a guide with a lot of variety on how to set up arch. I will update this post when I learn something new, while installing arch on one of my machines again. In the past I did some experiments with btrfs</code>, but had some problems while rebalancing, and actually used none of the fancy features like snapshots, btrfs send</code> or other cool things. Therefore, I decided to go back to plain old ext4 with lvm. 1</a></sup> Well, I actually used local snapshots bet never had to recover anything. A lot of this is based on the excellent (german) post from Thomas Leister 2</a></sup>. Everything is (as always) documented in the arch wiki. 3</a></sup></p> Overview</a></h2> Just a quick overview about the features before we get started.</p> ext4 Filesystem with lvm</li> encrypted root partition next to unencrypted /boot</code></li> EFI-Boot with systemd</li> </ul> Steps</a></h2> 1. Setup partitions</a></h3> The disk Layout will look like this:</p> </p> So let's get started with the partitioning. At first we will create the boot partition. I will use /dev/sda</code> as a placeholder, this could be anything on your local machine.</p> 1</span> =</span> boot</span> partition:</span></span> gdisk</span> /dev/sda</span></span> o</span></span> n</span></span> enter</span></span> enter</span></span> +512M</span></span> ef00</span></span></code></pre> After the boot partition is set up, we will use the rest of the space for our data partition.</p> 2</span> =</span> data:</span></span> n</span></span> 2</span></span> enter</span></span> enter</span></span> enter</span></span></code></pre> Write the changes to disk with w</code>. Now we are ready for the next step, the encryption.</p> Setup LUKS encryption</a></h3> Now to the disk encryption, dev/sda2</code> is the earlier created data partition.</p> cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 --use-random luksFormat /dev/sda2</code></p> LVM and format partitions</a></h3> #</span> Open the earlier created encrypted disk partition</span></span> cryptsetup</span> luksOpen</span> /dev/sda2</span> lvm</span></span> </span> pvcreate</span> /dev/mapper/lvm</span></span> </span> vgcreate</span> main</span> /dev/mapper/lvm</span></span> </span> #</span> Adjust to the actual memory size</span></span> lvcreate</span> -</span>L</span> 16G</span> -</span>n</span> swap</span> main</span></span> </span> lvcreate</span> -</span>l</span> 100%FREE</span> -</span>n</span> root</span> main</span></span> </span> mkswap</span> /dev/mapper/main-swap</span></span> </span> #</span> You can use whatever you want instead of ext4</span></span> mkfs.ext4</span> /dev/mapper/main-root</span></span> </span> #</span> The ESP must be formated as vfat</span></span> mkfs.vfat</span> /dev/sda1</span></span></code></pre>Mount disks</a></h3> mount</span> /dev/mapper/main-root</span> /mnt</span></span> mkdir</span> /mnt/boot</span></span> mount</span> /dev/sda1</span> /mnt/boot</span></span> swapon</span> /dev/mapper/main-swap</span></span></code></pre>Get network access</a></h3> If you have an ethernet jack on your device this should work out of the box.</p> Get the fastest local mirror</a></h3> Get the fastest local mirror to speed up the installation of the base system.</p> reflector</span> -</span>-latest</span> 10</span> -</span>-country</span> Germany</span> -</span>-sort</span> rate</span> -</span>-save</span> /etc/pacman.d/mirrorlist</span></span></code></pre>Install base system</a></h3> Don't forget to install dhcpcd</code> as it was removed from base</code>.</p> pacstrap</span> /mnt</span> base</span> linux</span> linux-firmware</span> lvm2</span> neovim</span> dhcpcd</span></span> genfstab</span> -</span>U</span> /mnt</span> >></span> /mnt/etc/fstab</span></span> </span> #</span> Chroot in the installation</span></span> arch-chroot</span> /mnt</span></span> </span> #</span> Set timezone info</span></span> ln</span> -</span>sf</span> /usr/share/zoneinfo/Region/City</span> /etc/localtime</span></span> </span> Edit</span> /etc/locale.gen</span> and</span> uncomment</span> en_US.UTF-8</span> UTF-8</span> and</span> other</span> needed</span> locales.</span></span> Generate</span> the</span> locales</span> by</span> running:</span> locale-gen</span></span> </span> /etc/locale.conf</span></span> LANG</span>=</span>e</span>n</span>_</span>U</span>S</span>.</span>U</span>T</span>F</span>-</span>8</span></span> </span> #</span> Set hostname</span></span> nvim</span> /etc/hostname</span></span> </span> #</span> Change root password</span></span> passwd</span></span></code></pre>Boot</a></h3> Edit /etc/mkinitcpio.conf</code></p> HOOKS</span>=</span>"</span>base udev autodetect modconf block keyboard keymap encrypt lvm2 filesystems fsck</span>"</span></span></code></pre> Run mkinitcpio -p linux</code></p> bootctl</span> -</span>-path=/boot</span> install</span></span> </span> /boot/loader/loader.conf</span></span> default</span> arch</span></span></code></pre> /boot/loader/entries/arch.conf</p> title Arch Linux</span></span> linux /vmlinuz-linux</span></span> initrd /initramfs-linux.img</span></span> options cryptdevice=/dev/sda2:main:allow-discards root=/dev/mapper/main-root resume=/dev/mapper/main-swap</span></span></code></pre> :allow-discards</code> to enable TRIM support.</p> Run systemctl enable fstrim.timer</code> to enable fstrim</code>.</p> Reboot</a></h3> Exit chroot with exit</code></li> </ul> umount</span> /mnt/boot</span></span> umount</span> /mnt</span></span></code></pre>Next steps</a></h3> You can now follow the post-installation recommendation. 4</a></sup></p> Add a user</li> Install software</li> ...</li> </ul> Conclusion</a></h2> I used this guide a few time for my personal machines. Overall the installation of Arch is straight forward, but as always with Arch you have a lot of possibilities (or opportunities) to use more or less equivalent software to reach the same goal. In this guide I decided on a lot of tools (e.g. systemd-boot instead of grub), because I simply like them or have no idea about newer and maybe better tools for the same problem. If you have any recommendations or improvements, don't hesitate to share them with me via mail.</p> https://btrfs.readthedocs.io/en/latest/btrfs-balance.html ↩</a></p> </li> https://legacy.thomas-leister.de/arch-linux-luks-verschluesselt-auf-uefi-system-installieren-2/ ↩</a></p> </li> https://wiki.archlinux.org/title/Installation_guide ↩</a></p> </li> https://wiki.archlinux.org/title/Installation_guide#Post-installation ↩</a></p> </li> </ol> </section> Neovim LSP 2022-03-01T00:00:00+00:00 I was (and still am for some parts) a happy vscode user. On the other hand I was always looking up to all these crazy fast vim/neovim users and their keyboard only drive workflows. After hopping on and off the vim train a few times a year something new came around sometime in 2021 - LSP support on neovim core. LSP stands for Language Server Protocol and defines how your IDE communicates with a specific language server to get features like 'go to definition', 'show references' or 'rename this variable'. This was a huge game changer as you can now use every language server written to work with vscode with neovim. So I took the time to watch a few videos, read the documentation and finally setup neovim and gopls</a> the language server for Go. 1</a></sup> 2</a></sup> 3</a></sup></p> In this post I will present the simplest setup to get an IDE-like feeling within neovim.</p> Overview</a></h2> Just a quick overview about the neovim ecosystem before we get started.</p> Neovim supports Lua to configure plugins and a lot of other things. Actually the possibility to configure it via Lua just got a bit better 4</a></sup>.</p> </li> The old way to configure things is via init.vim</code>, but personally I prefer an actual programming language instead of vimscript.</p> </li> The last things I want to mention are plugins. There are a lot of plugins, for example we are going to use nvim-lspconfig</code>, so you should be able to extend your editor more or less like any other editor/IDE.</p> </li> </ol> Install neovim</a></h2> The configuration I will present needs at least neovim version 0.7</code> since you can not configure keymaps in lua in version 0.6</code>. As this version is currently only available as a development release you have two ways to install it.</p> First you can clone the neovim repository</a> and build it yourself, which is actually pretty straight forward and worked well on macOS. Simply follow the instructions in the README.</p> The second options depend on your OS or Linux distribution. In my case I used the AUR package neovim-nightly-bin</a>.</p> A basic neovim configuration</a></h2> Start with a new init.vim</code> in ~/.config/nvim/</code>. At first, we need the LSP plugin itself and some other helpers. I'm using vim-plugged</a> to download and update plugins automatically. Simply install it from the sources of your distribution or follow the instructions provided in the README.</p> call plug#begin('~/.nvim/plugged')</span></span> </span> Plug 'neovim/nvim-lspconfig'</span></span> </span> Plug 'nvim-lua/plenary.nvim'</span></span> </span> " LSP autocompletion</span></span> Plug 'hrsh7th/nvim-cmp'</span></span> Plug 'hrsh7th/cmp-nvim-lsp'</span></span> Plug 'hrsh7th/cmp-buffer'</span></span> Plug 'hrsh7th/cmp-path'</span></span> </span> " More autocompletion</span></span> Plug 'L3MON4D3/LuaSnip' " There are other snippet engines you could use, but we will be using this one.</span></span> Plug 'saadparwaiz1/cmp_luasnip'</span></span> </span> call plug#end()</span></span></code></pre> Now startup neovim and install all plugins with :PlugInstall</code>. We now have a working neovim, but haven't configured anything related to the LSP.</p> Setup LSP-Plugin</a></h2> Since we don't want to use vimscript for the complex pieces of the lsp configuration, we will create a lua script for that. The configuration must be stored in ~/.config/nvim/lua/aps.lua</code>. You can of course rename the lua file aps.lua</code> to anything you want, but I used my login name on most of my machines.</p> Let's start by requiring the new lua configuration file in the previously created init.vim</code>.</p> lua</span> require</span>(</span>"</span>aps</span>"</span>)</span></span></code></pre> lua require("aps")</code> is the magic keyword to load the lua configuration. This another game changer as I can now configure most things within an actual scripting language, instead of neovim configuration files. So let's have a look at the content of aps.lua</code>.</p> --</span> Hook LSP into completion</span></span> local</span> capabilities</span> =</span> require</span>(</span>'</span>cmp_nvim_lsp</span>'</span>).</span>update_capabilities</span>(</span>vim</span>.</span>lsp</span></span> .</span>protocol</span></span> .</span>make_client_capabilities</span>())</span></span> </span> </span> --</span> Setup go language server gopls</span></span> require</span>'</span>lspconfig</span>'</span>.</span>gopls</span>.</span>setup</span> {</span></span> capabilities</span> =</span> capabilities</span>,</span></span> completeUnimported</span> =</span> true</span>,</span></span> analyses</span> =</span> {</span>unusedparams</span> =</span> true</span>},</span></span> staticcheck</span> =</span> true</span>,</span></span> on_attach</span> =</span> function</span>(</span>)</span></span> --</span> "n" means normal mode</span></span> --</span> {buffer=0} means only for this buffer</span></span> --</span> <cmd> == :</span></span> --</span> <cr> == enter</span></span> </span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>K</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>hover</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gd</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>definition</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gt</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>type_definition</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gi</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>implementation</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gR</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>references</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gr</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>rename</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gf</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>formatting</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>ga</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>code_action</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>en</span>"</span>, </span>vim</span>.</span>diagnostic</span>.</span>goto_next</span>, {</span>buffer</span> =</span> 0</span>})</span></span> vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>ep</span>"</span>, </span>vim</span>.</span>diagnostic</span>.</span>goto_prev</span>, {</span>buffer</span> =</span> 0</span>})</span></span> end</span></span> }</span></span> </span> --</span> LSP autocomplete</span></span> vim</span>.</span>opt</span>.</span>completeopt</span> =</span> {</span>"</span>menu</span>"</span>, </span>"</span>menuone</span>"</span>, </span>"</span>noselect</span>"</span>} </span>--</span> setting vim values</span></span> </span> --</span> Setup nvim-cmp</span></span> local</span> cmp</span> =</span> require</span> '</span>cmp</span>'</span></span> </span> cmp</span>.</span>setup</span>({</span></span> snippet</span> =</span> {</span></span> expand</span> =</span> function</span>(</span>args</span>)</span></span> require</span>(</span>'</span>luasnip</span>'</span>).</span>lsp_expand</span>(</span>args</span>.</span>body</span>) </span>--</span> For `luasnip` users.</span></span> end</span></span> },</span></span> mapping</span> =</span> {</span></span> [</span>'</span><C-b></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>(</span>cmp</span>.</span>mapping</span>.</span>scroll_docs</span>(</span>-</span>4</span>), {</span>'</span>i</span>'</span>, </span>'</span>c</span>'</span>}),</span></span> [</span>'</span><C-f></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>(</span>cmp</span>.</span>mapping</span>.</span>scroll_docs</span>(</span>4</span>), {</span>'</span>i</span>'</span>, </span>'</span>c</span>'</span>}),</span></span> [</span>'</span><C-Space></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>(</span>cmp</span>.</span>mapping</span>.</span>complete</span>(), {</span>'</span>i</span>'</span>, </span>'</span>c</span>'</span>}),</span></span> [</span>'</span><C-y></span>'</span>] </span>=</span> cmp</span>.</span>config</span>.</span>disable</span>, </span>--</span> Specify `cmp.config.disable` if you want to remove the default `<C-y>` mapping.</span></span> [</span>'</span><C-e></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>({</span></span> i</span> =</span> cmp</span>.</span>mapping</span>.</span>abort</span>(),</span></span> c</span> =</span> cmp</span>.</span>mapping</span>.</span>close</span>()</span></span> }),</span></span> --</span> Accept currently selected item. Set `select` to `false` to only confirm explicitly selected items.</span></span> [</span>'</span><CR></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>.</span>confirm</span>({</span>select</span> =</span> true</span>})</span></span> },</span></span> sources</span> =</span> cmp</span>.</span>config</span>.</span>sources</span>({</span></span> {</span>name</span> =</span> '</span>nvim_lsp</span>'</span>},</span></span> {</span>name</span> =</span> '</span>luasnip</span>'</span>} </span>--</span> For luasnip users.</span></span> }, {{</span>name</span> =</span> '</span>buffer</span>'</span>}})</span></span> })</span></span> </span> --</span> Enable inline errors</span></span> vim</span>.</span>lsp</span>.</span>handlers</span>[</span>"</span>textDocument/publishDiagnostics</span>"</span>] </span>=</span></span> vim</span>.</span>lsp</span>.</span>with</span>(</span>vim</span>.</span>lsp</span>.</span>diagnostic</span>.</span>on_publish_diagnostics</span>,</span></span> {</span>update_in_insert</span> =</span> true</span>})</span></span></code></pre> This configuration is more or less based on the excellent video by bashbunni and TJ 1</a></sup>.</p> The exciting part begins with require'lspconfig'.gopls.setup</code>, as this is the place to configure the LSP to your favorite hotkeys. The bindings are set up within the attribute on_attach</code> of the gopls</code> object. I try to keep them in sync with my wording e.g. go to type definition</code> uses the hotkey gt</code> or go to definition</code> uses gd</code>.</p> I especially like the binding gf</code> (or go fmt</code>) to get my Go code formatted. But you configure these to whatever you want. In my case all of these work only in "normal" mode and without usage of the leader key.</p> Most of the other parts are related to autocompletion provided by nvim-cmp</code> and lua-snip</code>, which are covered in these videos by TJ 5</a></sup> 6</a></sup>. I'm still a very basic user of these and especially lua-snip</code> is a magic tool, which I want to explore deeper in the near future. But let's keep it simple for now and simply accept that we need this to get an IDE-like autocompletion.</p> At this point you should be able to open a go file and get basic highlighting and completion.</p> </p> As you can see the completion provided by the LSP works fine, but the highlighting is not as good as I would like it. We will explore how to get there in the next section of this post.</p> Treesitter for better highlighting</a></h2> Treesitter</a> will build a syntax tree for your current file and add highlighting based on this syntax tree. This is a pretty huge dependency, and you can skip this if you are ok with the look of your code in the earlier provided image.</p> I can also recommend this video (again) TJ about LSP vs treesitter 7</a></sup>.</p> To enable treesitter we have to add the treesitter plugin and provide a basic configuration in our lua configuration.</p> Add this to the plugin section if your init.vim</code>.</p> Plug</span> '</span>nvim-treesitter/nvim-treesitter</span>'</span>,</span> {</span>'</span>do</span>'</span>:</span> '</span>:TSUpdate</span>'</span>}</span></span></code></pre> Again open neovim and run :PlugInstall</code> to get the plugin. The initial run of TSUpdate</code> will take a long time (long in 30 seconds to 1 minute) to install all the treesitter plugin for the most common languages.</p> Now add the following at the bottom of your lua configuration. This is basically the simplest configuration provided in the treesitter README 8</a></sup>.</p> require</span>'</span>nvim-treesitter.configs</span>'</span>.</span>setup</span> {</span></span> --</span> One of "all", "maintained" (parsers with maintainers), or a list of languages</span></span> ensure_installed</span> =</span> "</span>maintained</span>"</span>,</span></span> </span> --</span> Install languages synchronously (only applied to `ensure_installed`)</span></span> sync_install</span> =</span> false</span>,</span></span> </span> --</span> List of parsers to ignore installing</span></span> ignore_install</span> =</span> { </span>"</span>javascript</span>" </span>},</span></span> </span> highlight</span> =</span> {</span></span> --</span> `false` will disable the whole extension</span></span> enable</span> =</span> true</span>,</span></span> </span> --</span> list of language that will be disabled</span></span> disable</span> =</span> { </span>"</span>c</span>"</span>, </span>"</span>rust</span>" </span>},</span></span> </span> --</span> Setting this to true will run `:h syntax` and tree-sitter at the same time.</span></span> --</span> Set this to `true` if you depend on 'syntax' being enabled (like for indentation).</span></span> --</span> Using this option may slow down your editor, and you may see some duplicate highlights.</span></span> --</span> Instead of true it can also be a list of languages</span></span> additional_vim_regex_highlighting</span> =</span> false</span>,</span></span> },</span></span> }</span></span></code></pre> And that's it! Let's open up the file from before again and have a look at the differences.</p> </p> As you can see for example functions are now highlighted! In case you wonder, I'm using the Dracula</code> theme.</p> Conclusion</a></h2> To be honest it was a lot of fun getting this to work, do some research and write it up in a post. Maybe I will write about other cool tools in the neovim ecosystem like telescope</code> or all the magic you can do with luasnip</code>. But for now I'm very happy with the setup and the speed you get, especially compared to vscode. As a fan of keyboard driven workflows this is more or less the best you can get, and I really enjoy it!</p> https://www.youtube.com/watch?v=puWgHa7k3SY ↩</a> ↩2</a></p> </li> https://www.youtube.com/watch?v=2WPC8rZQvQU ↩</a></p> </li> https://www.youtube.com/watch?v=0fOIp5PH648 ↩</a></p> </li> https://github.com/neovim/neovim/pull/14661 ↩</a></p> </li> https://www.youtube.com/watch?v=_DnmphIwnjo ↩</a></p> </li> https://www.youtube.com/watch?v=Dn800rlPIho ↩</a></p> </li> https://www.youtube.com/watch?v=c17j09vY5sw ↩</a></p> </li> https://github.com/nvim-treesitter/nvim-treesitter ↩</a></p> </li> </ol> </section> Build a simple VPN with WireGuard 2022-02-06T00:00:00+00:00 As you already know I host most of the services I use on a daily basis myself. Some of these services are hosted on the public Internet, while others are running on my home server. In this post I will cover the basic on how to get a peers-to-site VPN connection with WireGuard</a>.</p> WireGuard uses a simple public/private keypair to set up an encrypted VPN tunnel, therefore there is no need for a complicated certificate authority setup. In my opinion this is a major step when upgrading from e.g. OpenVPN. Also, WireGuard is superfast and included in all newer versions of the Linux kernel.</p> In my case both the server and the client are running Arch Linux and the only additional package you have to install is wireguard-tools</code>.</p> Set up a server</a></h2> The server must be a machine with a public IPv4 and/or v6 address. In my case I use the cheapest Hetzner cloud machine CX11</code> which costs about 4 Euro a month.</p> At first generate keys with wg genkey \| (umask 0077 && tee server.key) \| wg pubkey > server.pub</code>. Then use the following file as template, replace the needed variables and save it at /etc/wireguard/wg-server.conf</code>. The preshared key can simply be generated with wg genpsk > peer1-peer2.psk</code>. Each connection or peer needs its own preshared key!</p> #</span> wg-server.conf</span></span> [</span>Interface</span>]</span></span> Address</span> =</span> 10.0.0.1/24,fd42:42:42::1/64</span></span> ListenPort</span> =</span> 1194</span></span> PrivateKey</span> =</span> <</span>generated-private-ke</span>y</span>></span></span> PostUp</span> =</span> iptables</span> -</span>A</span> FORWARD</span> -</span>i</span> %i</span> -</span>j</span> ACCEPT</span>;</span> iptables</span> -</span>t</span> nat</span> -</span>A</span> POSTROUTING</span> -</span>o</span> eth0</span> -</span>j</span> MASQUERADE</span></span> PostDown</span> =</span> iptables</span> -</span>D</span> FORWARD</span> -</span>i</span> %i</span> -</span>j</span> ACCEPT</span>;</span> iptables</span> -</span>t</span> nat</span> -</span>D</span> POSTROUTING</span> -</span>o</span> eth0</span> -</span>j</span> MASQUERADE</span></span> </span> #</span> Home-VPN Gateway</span></span> [</span>Peer</span>]</span></span> PublicKey</span> =</span> <</span>generated-client-public-ke</span>y</span>></span></span> AllowedIPs</span> =</span> 10.0.0.2/32,</span> fd42:42:42::2/128,</span> 192.168.0.0/24</span></span> PresharedKey</span> =</span> <</span>preshared-ke</span>y</span>></span></span> </span> #</span> Example for a roadwarrior client</span></span> [</span>Peer</span>]</span></span> PublicKey</span> =</span> <</span>generated-public-ke</span>y</span>></span></span> AllowedIPs</span> =</span> 10.0.0.3/32,fd42:42:42::3/128</span></span> PresharedKey</span> =</span> <</span>preshared-ke</span>y</span>></span></span></code></pre> The important part is the network 192.168.0.0/24</code> in the AllowedIPs</code> part of the gateways peer config. This will add a route on the server side, so the server knows that your home network can be reached through this specific WireGuard interface.</p> Enable IP forwarding with the following commands. -w</code> will make these settings persistent across reboots.</p> $</span> sysctl</span> -</span>w</span> net.ipv4.ip_forward=</span>1</span></span> $</span> sysctl</span> -</span>w</span> net.ipv6.conf.all.forwarding=</span>1</span></span></code></pre> Start the client with wg-quick up wg-server</code> or systemctl enable --now wg-quick@wg0</code>.</p> Configure the client</a></h2> Again generate keys with wg genkey \| (umask 0077 && tee client.key) \| wg pubkey > client.pub</code>. Then use the following file as template, replace the needed variables and save it at /etc/wireguard/wg-client.conf</code>.</p> #</span> wg-client.conf</span></span> </span> [</span>Interface</span>]</span></span> PrivateKey</span> =</span> <</span>generated-private-ke</span>y</span>></span></span> Address</span> =</span> 10.0.0.2/24,fd42:42:42::3/64</span></span> </span> [</span>Peer</span>]</span></span> PublicKey</span> =</span> <</span>generated-server-public-ke</span>y</span>></span></span> Endpoint</span> =</span> my.server.com:1194</span></span> AllowedIPs</span> =</span> 10.0.0.0/24,fd42:42:42::0/64,192.168.0.1/24</span></span> PresharedKey</span> =</span> <</span>preshared-ke</span>y</span>></span></span> PersistentKeepalive</span> =</span> 25</span></span></code></pre> PersistentKeepalive</code> will ensure that the connection will be open</code>, even if there is no need for it. As WireGuard is a silent protocol there will be no communication without a client sending or recieving messages. This is especially critical with peers behind NAT, which will likely be the case with almost every home setup.</p> Again enable IP forwarding with the following commands, as we want to forward traffic from local clients to VPN clients.</p> $</span> sysctl</span> -</span>w</span> net.ipv4.ip_forward=</span>1</span></span> $</span> sysctl</span> -</span>w</span> net.ipv6.conf.all.forwarding=</span>1</span></span></code></pre> Start the client with wg-quick up wg-client</code> or systemctl enable --now wg@wg-client</code>.</p> To route all traffic through the tunnel you can use 0.0.0.0/0</code> in the AllowedIPs</code> section.</p> Static route</a></h2> The last thing to do is to set a static route to your VPN network. You can do this on each home device or simply add a static IPv4/v6 route on your Router. The route should look like this 10.0.0.0/24 via <ip of your client on the local network></code>.</p> At my home network the Pi I'm using as the client has the IP 192.168.0.8</code>, therefore the static route looks like this 10.0.0.0/24 via 192.168.0.8</code>.</p> Debugging</a></h2> As WireGuard is a very silent protocol it can be hard the debug in case of any connectivity problems. Try to ping from each side of the tunnel and use wg show</code> to check if the initial handshake was successfull. Check the logs with dmesg -wT</code> or journalctl -u wg@wg-client/server</code>.</p> $</span> modprobe</span> wireguard</span></span> $</span> echo</span> module</span> wireguard</span> +p</span> ></span> /sys/kernel/debug/dynamic_debug/control</span></span></code></pre>Performance</a></h2> I simply used iperf3</code> to run an upload and download test from my home to the server. As I have an 1000 Mbit download and 50 Mbit upload connection we should see some useful results.</p> Let's start with the upload test. This is more or less my max. possible upload, therefore this a very good result.</p> ➜</span> ~</span> iperf3</span> -</span>c</span> 10.66.66.1</span></span> Connecting</span> to</span> host</span> 10.66.66.1,</span> port</span> 5201</span></span> [</span> 5</span>]</span> local</span> 192</span>.</span>168</span>.</span>0</span>.</span>28</span> port</span> 34676</span> connected</span> to</span> 10</span>.</span>66</span>.</span>66</span>.</span>1</span> port</span> 5201</span></span> [</span> ID</span>]</span> Interval Transfer Bitrate Retr Cwnd</span></span> [</span> 5</span>]</span> 0.00-1.00 sec 6.20 MBytes 52.0 Mbits/sec 1 179 KBytes</span></span> [</span> 5</span>]</span> 1.00-2.00 sec 5.76 MBytes 48.3 Mbits/sec 1 151 KBytes</span></span> [</span> 5</span>]</span> 2.00-3.00 sec 5.70 MBytes 47.8 Mbits/sec 0 178 KBytes</span></span> [</span> 5</span>]</span> 3.00-4.00 sec 5.82 MBytes 48.8 Mbits/sec 1 152 KBytes</span></span> [</span> 5</span>]</span> 4.00-5.00 sec 6.00 MBytes 50.3 Mbits/sec 0 179 KBytes</span></span> [</span> 5</span>]</span> 5.00-6.00 sec 6.00 MBytes 50.3 Mbits/sec 2 110 KBytes</span></span> [</span> 5</span>]</span> 6.00-7.00 sec 5.46 MBytes 45.8 Mbits/sec 0 143 KBytes</span></span> [</span> 5</span>]</span> 7.00-8.00 sec 6.00 MBytes 50.3 Mbits/sec 1 122 KBytes</span></span> [</span> 5</span>]</span> 8.00-9.00 sec 6.00 MBytes 50.3 Mbits/sec 0 152 KBytes</span></span> [</span> 5</span>]</span> 9.00-10.00 sec 5.76 MBytes 48.3 Mbits/sec 1 128 KBytes</span></span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span></span> [</span> ID</span>]</span> Interval Transfer Bitrate Retr</span></span> [</span> 5</span>]</span> 0.00-10.00 sec 58.7 MBytes 49.2 Mbits/sec 7 sender</span></span> [</span> 5</span>]</span> 0.00-10.02 sec 58.1 MBytes 48.7 Mbits/sec receiver</span></span> </span> iperf</span> Done.</span></span></code></pre> Now to the download test which is not as good as expected. This is about 1/4 of the possible download. The server has about 35-40 % CPU load while running the test, so it looks like the cheap VM might be the limit here.</p> ➜</span> ~</span> iperf3</span> -</span>c</span> 192.168.0.28</span></span> Connecting</span> to</span> host</span> 192.168.0.28,</span> port</span> 5201</span></span> [</span> 5</span>]</span> local</span> 10</span>.</span>66</span>.</span>66</span>.</span>1</span> port</span> 52106</span> connected</span> to</span> 192</span>.</span>168</span>.</span>0</span>.</span>28</span> port</span> 5201</span></span> [</span> ID</span>]</span> Interval Transfer Bitrate Retr Cwnd</span></span> [</span> 5</span>]</span> 0.00-1.00 sec 26.2 MBytes 219 Mbits/sec 0 1.31 MBytes</span></span> [</span> 5</span>]</span> 1.00-2.00 sec 31.2 MBytes 262 Mbits/sec 379 1.41 MBytes</span></span> [</span> 5</span>]</span> 2.00-3.00 sec 31.2 MBytes 262 Mbits/sec 0 1.55 MBytes</span></span> [</span> 5</span>]</span> 3.00-4.00 sec 31.2 MBytes 262 Mbits/sec 146 1.24 MBytes</span></span> [</span> 5</span>]</span> 4.00-5.00 sec 26.2 MBytes 220 Mbits/sec 0 1.24 MBytes</span></span> [</span> 5</span>]</span> 5.00-6.00 sec 26.2 MBytes 220 Mbits/sec 0 1.30 MBytes</span></span> [</span> 5</span>]</span> 6.00-7.00 sec 26.2 MBytes 220 Mbits/sec 0 1.35 MBytes</span></span> [</span> 5</span>]</span> 7.00-8.00 sec 27.5 MBytes 231 Mbits/sec 0 1.38 MBytes</span></span> [</span> 5</span>]</span> 8.00-9.00 sec 27.5 MBytes 231 Mbits/sec 0 1.39 MBytes</span></span> [</span> 5</span>]</span> 9.00-10.00 sec 33.8 MBytes 283 Mbits/sec 0 1.40 MBytes</span></span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span></span> [</span> ID</span>]</span> Interval Transfer Bitrate Retr</span></span> [</span> 5</span>]</span> 0.00-10.00 sec 287 MBytes 241 Mbits/sec 525 sender</span></span> [</span> 5</span>]</span> 0.00-10.05 sec 286 MBytes 238 Mbits/sec receiver</span></span> </span> iperf</span> Done.</span></span></code></pre> But this is more than enough for my day to day computing and even streaming a movie from my jellyfin instance at home.</p> Conclusion</a></h2> I am a huge fan of WireGuard as it is simple to set up and fast. The tooling is easy to use (especially when compared against OpenVPN), the documentation is straight forward, and it simply works. And if you are running a recent Linux kernel most of the things you need are already there.</p> Using macOS as a Linux user 2021-12-17T00:00:00+00:00 I never expected to use any other OS than my beloved Linux (and especially Arch Linux) for developing and day-to-day computing. Well, I actually use Windows to play some games with friends occasionally, but the time has come to use a Mac at work. It was simply the easiest way to get a much more powerful machine than my ThinkPad X390. So here I'm trying to write up what I like about macOS and what I miss from my highly customized Linux machine. This isn't a fair comparison as I know much more about Linux than about macOS, but maybe you (as in the reader) can correct me, and I'll learn something new!</p> Things I like</a></h2> So let's start with the things I like about macOS.</p> 1. No UI glitches</a></h3> As a Wayland user I'm used to UI glitches with not Wayland native applications. For example getting IntelliJ IDEs running took quite some time, to set all needed environment flags. And most of the apps and system UIs do look great by default. I think it might be hard to do some customization if you want to, but as I don't care that much I'm fine with the default.</p> 2. Hardware accelerated apps (especially browsers)</a></h3> Yes this is still a thing on Linux. You can get it to work with chromium and Vaapi or Firefox</a>, but it's often buggy, your browser decides to freeze, or it simply won't work at all. At least this was my experience on my X250 and my X390. On my desktop computer I do have enough performance to simply not need hardware acceleration, but it is really handy on a mobile device.</p> 3. Time Machine Backups</a></h3> In my opinion a modern OS must provide a simple tool to back up your system. Time Machine may not be the fastest backup tool, but it does the job and restoring files is nicely integrated in the UI. I also wrote a post about how to transform a Raspberry Pi into a Time Capsule</a>.</p> 4. It's a UNIX</a></h3> And therefore most of my used applications and workflows simply work. I mostly rely on a terminal with zsh, git, neovim, tmux and some other tools and within the terminal there absolutely no difference between my Linux machine and the Mac. Getting all my settings was also very easy, clone my dotfiles repository, add all need symlinks via stow</code>, and I was ready to go.</p> 5. The magic touchpad and the gestures to navigate the UI</a></h3> On my ThinkPad I was a happy trackpoint user as it provides access to a mouse without moving your finger too far away from the keyboard. I was surprised to enjoy the magic trackpad, but the experience of a touchpad instead of a mouse is somewhat nice. My only problem is wrist pain after using the touchpad for a whole workday, so I'm currently switching between mouse and trackpad. A annoying thing about that is that you can't disable/enable natural scrolling per device. I enjoy it on the trackpad, but don't like it on the mouse.</p> 6. APFS</a></h3> On my Linux machine I'm using brtfs because it supports snapshots, compression and is a copy-on-write (CoW) file system. But to be honest the initial learning curve and setup took quite some time. On macOS, I get all of this for free with APFS.</p> 7. Application Launcher</a></h3> I am used to rofi</code> or wofi</code> to start my applications via keyboard only. On macOS, I simply use Command + Space</code> to get spotlight search to start applications and search for other things. This is more or less the same workflow as on my Linux machine.</p> 8. Application Scaling</a></h3> Scaling on a 4K display works like a charm.</p> Things I don't like:</a></h2> Now to the things I don't like, or I'm still searching for a workaround.</p> 1. No package manager</a></h3> As Linux and especially Arch Linux user not using a package manager to install my software feels like using Windows 98. (I don't use Windows that often, but I think they also have a package manager now?). There are three options to install most of the software on macOS. At first there is the 'download it from the web and run the installer', second the official App Store and last using a package manager like homebrew</code>.</p> 2. Can't disable dock</a></h3> I simply don't need it. When I want to start applications I do it via my terminal or via spotlight, but this seems like another problem of a keyboard centric user.</p> 3. Can't integrate into spotlight search (gopass)</a></h3> I let gopass</a> handle all my password/secret related things. On my Linux machines I've integrated it perfectly within rofi</a> and it looks like Alfred</a> can do the same on macOS. But this is one of the things that's still on my to-do list.</p> 4. Touchbar and no physical escape key</a></h3> I have no idea who could possible like the Touchbar. Especially as because if it there is no</strong> physical escape key. As I like to use Neovim</a> for editing config files and more ore less anything besides code, this is just annoying.</p> 5. Different keyboard layout</a></h3> I think this just takes some time to adjust to it.</p> 6. Floating window management</a></h3> I'm a happy user of sway</a> and really enjoy using tiling window managers, so using a floating window manager feels like a step back. There are things like virtual desktops that work like workspaces in sway, but you can't move apps to a different workspace with a shortcut (please correct me if I'm wrong here!). You can use the magic touchpad/mouse, but I prefer to use the mouse/touchpad as rarely as possible.</p> 7. launchd uses xml configuration files</a></h3> As a Linux user I'm used to systemd units, that are sometimes long, but not as unreadable as XML. But maybe that's because I'm actually never used XML in the past.</p> 8. Click in window to actually focus it</a></h3> At first, I thought the system was somehow unresponsive, or the mouse is not in the window I want to type in, but I later noticed that you have to actually focus a window with a click. This is so annoying, especially with a floating window manager, where it is normal to have multiple windows overlapping each other. Why shouldn't I be able to type in a window that I didn't focus with a click?</p> 9. Updates take a long time</a></h3> I have no idea what the mac is doing in the background, but I never experienced updates taking that much time. For example the update from 12.0 to 12.1 took about 30 minutes. And this was only the operating system update, after that there were also some brew updates, docker and more.</p> 10. Bad heat distribution and loud fans</a></h3> The hardware seems overwhelmed by the heat produced by the Intel i9. If you open a few apps, start debugging or a few containers it will start to get hot very fast. And the fan is very noisy, but I think all if this will be fixed with the new M1 Macs. Besides that the device is pretty well build.</p> Conclusion</a></h2> I think I will update this post with new learnings from time to time. For now, I'm quite happy with the macOS setup, as I could simply copy most of my dotfiles. To be honest the things I use most of the time are a terminal, a browser and an editor and these things are working quite well. The thing I miss the most is a good package manager. Pacman does an excellent job at this, because it is fast and make it so much easier to keep your system up to date.</p> Transform your Raspberry Pi into a Apple Time Capsule 2021-11-23T00:00:00+00:00 This is a short guide (and reminder for myself) how to add a Samba share as target for macOS Time Machine. I found some blog posts about how to do this, but most of the time the author created a new disk on the share 1</a></sup> 2</a></sup>. This seems like a rather unusual complicated setup to do backups and there are products like the Apple Time Capsule</a>, that works via network discovery. I actually found a blog post (which I can't anymore), how to set up Avahi to announce a share as Time Capsule.</p> This actually works with every Linux host with samba</code> and avahi</code> installed, but the title with Raspberry Pi in it is much more catchy. And I actually use a Raspberry Pi 4 to provide backup space for my machines and servers.</p> Setup Samba</a></h3> To get SMB up and running you have to:</p> Install samba via the package manager of your distribution.</li> Create a Unix user.</li> Create a samba user with smbpasswd -a</code>.</li> Create a new folder on your backup volume (in my case this is an external 8 TB Disk on a Raspberry Pi).</li> Add the following entry in the smb.conf</code>. You should adjust the path</code> and user</code> to match your environment.</li> </ol> # /etc/samba/smb.conf</span></span> </span> [backups]</span></span> comment = Time Machine Backups</span></span> path = /mnt/backups/andre/time-machine</span></span> guest ok = no</span></span> read only = no</span></span> browseable = yes</span></span> valid users = andre</span></span> write list = andre</span></span> vfs objects = catia fruit streams_xattr</span></span> fruit:time machine = yes</span></span></code></pre> Restart smbd</code>.</li> </ol> You should now be able to access the share. If not, check that the user you created earlier actually has the rights to access the folder! You can also add quotas to this share, but I have plenty of free space left, so there is really no need to do this right now.</p> Configure Avahi</a></h3> This is the part that will announce the backup server and the SMB service as an Apple Time Capsule. Avahi</a> is a system which provides service discovery on a local network via the mDNS/DNS-SD protocol suite.</p> Install Avahi via the package manager of your distribution.</li> Add the following configuration for SMB.</li> Restart avahi</code>.</li> </ol> # /etc/avahi/services/samba.service</span></span> </span> <?</span>xml</span> version</span>=</span>"</span>1.0</span>"</span> standalone</span>=</span>'</span>no</span>'</span>?></span><!--</span>-nxml-</span>--></span></span> <!</span>DOCTYPE</span> service-group</span> SYSTEM "avahi-service.dtd"</span>></span></span> <</span>service-group</span>></span></span> <</span>name</span> replace-wildcards</span>=</span>"</span>yes</span>"</span>></span>%h</span></</span>name</span>></span></span> <</span>service</span>></span></span> <</span>type</span>></span>_smb._tcp</span></</span>type</span>></span></span> <</span>port</span>></span>445</span></</span>port</span>></span></span> </</span>service</span>></span></span> <</span>service</span>></span></span> <</span>type</span>></span>_device-info._tcp</span></</span>type</span>></span></span> <</span>port</span>></span>9</span></</span>port</span>></span></span> <</span>txt-record</span>></span>model=TimeCapsule8,119</span></</span>txt-record</span>></span></span> </</span>service</span>></span></span> <</span>service</span>></span></span> <</span>type</span>></span>_adisk._tcp</span></</span>type</span>></span></span> <</span>port</span>></span>9</span></</span>port</span>></span></span> <</span>txt-record</span>></span>dk0=adVN=backups,adVF=0x82</span></</span>txt-record</span>></span></span> <</span>txt-record</span>></span>sys=adVF=0x100</span></</span>txt-record</span>></span></span> </</span>service</span>></span></span> </</span>service-group</span>></span></span></code></pre> I found this configuration a few months ago but couldn't find it while researching for this post to provide the original source.</p> </p> But with this magic piece of XML we have done it! You should now see your Raspberry Pi as an Apple Time Capsule in the network tab of Finder.</p> Setup Time Machine</a></h3> You can now create a new Time Machine backup. Please don't forget to enable encryption, even if the disk you are using on the Pi is already encrypted! While selecting the target disk you should be able to select our newly created backup location.</p> </p> Conclusion</a></h2> Well that was more complicated than I think it should be. Why shouldn't Time Machine be able to use a regular SMB share as backup endpoint? Maybe some APFS magic is involved here, and Time Machine does simply send snapshots to the backup location? I have no idea, but I'm pretty happy everything is working now, and I can let backups be as they should be: simple, boring and working.</p> https://www.imore.com/how-use-time-machine-backup-your-mac-windows-shared-folder ↩</a></p> </li> https://kirb.me/2018/03/24/using-samba-as-a-time-machine-network-server.html ↩</a></p> </li> </ol> </section> How to find durable technology? 2021-11-14T00:00:00+00:00 This post is not focused on a specific technology, but more about how to find hardware with good support (e.g. spare parts, software updates, ...) so you can use it for multiple years. In this case I use headphones as an example, but the general concept can be applied to just every technology you want to buy.</p> A headphone example</a></h2> So let's dive into my example. I currently own two different headphones. The first are my Sony WH-1000MX4</a> with active noise-cancelling (ANC), that I mostly use when I'm commuting, listening to podcasts while cooking, working in the office or at some other noisy place. The ANC works very good and is just perfect if you need an hour to do some deep work or get some reading done in a noisy environment. You can learn more about the Sony headphones in an older post</a>.</p> The second headphones are my trusty Beyerdynamic DT-770 Pro</a> that I bought at least 9 years ago. I mostly use the DT-770 when I'm working/studying at my desk and they are connected via the Focusrite Scarlett 2i2</a> interface to my computer.</p> What I really like about both headphones is that you can easily get spare parts to repair them. So this week it was time to change the ear pads of the Beyerdynamics headphones as they tend to become dirty after a few years of usage. In this case I only bought new ear pads, but you can get about every part of the headphones directly</a> from Beyerdynamics. The change itself takes about 10 minutes and is very straight forward.</p> The point I want to make is that these headphones are about 10 years old, and I still can get new ear pads (or any other part) without any effort! That is something I really like about pro</strong> (or on this case studio</strong>) hardware.</p> So in this case the investment of ~130 Euro was more than worth it. I actually hope the same holds true for the Sony headphones, but I'm not so sure of that. The Sony's are more of a consumer product than the Beyerdynamic's and therefore will get updated more often. The predecessor of the MX4, the MX3 were released in August 2018 and the MX4 in August 2020. You can get new ear pads for the MX3/MX4, but I couldn't find any official other parts than that. Hopefully I will never need more than ear pads from Sony.</p> If it is possible I would prefer to get parts from the original equipment manufacturer (OEM) as they should know the specific specs of their product. The problem there is that they often call for absurd high prices, so you have to use some other Chinese company to get them for a reasonable price. Fortunately, that's not the case here!</p> My conclusion is that headphones like the DT-770 are a good investment as you can use them for many years and if something is broken you can simply repair it yourself. Besides, they sound excellent!</p> A generic view on technology</a></h2> The trick is to know if a product will be supported that good for a time span like 10 years. In my opinion an indicator for that could be that a company exists at least for a few years. If you get a product via Kickstarter I am not so sure the (maybe new founded) company behind it can provide any support. You can't be sure if the company will exist in a few years.</p> Also some people in the scene (like audio in this case) recommend their products. But with people in the scene I explicitly do not mean some random tech youtuber or influencer that uses a product for a few weeks and the declares it as 'the best product I've ever used'. I mean real feedback from people that use a product or a predecessor of a product for a long time. Most of the time you can't directly infer the experiences from a predecessor to current product, but it can give you an idea how the product could be supported (especially on the software/firmware side).</p> These people can of course be found on the internet, just do some more research than simply trust your favorite youtube/random video service star. This is of course associated with the risk that the same circumstances do not apply to the new product, but this is the risk we have to take with every new product we buy.</p> Conclusion</a></h2> Finally I think the headphone example is a very easy one as musicians/studios all around the world use Beyerdynamics headphones and you can see them often in studio videos. And in my case this was more or less a coincidence as I simply followed a recommendation in a forum. Besides that the Beyerdynamics do not contain any software like the Sony's, so it's much easier for them to simply work. But EOL your hardware because of missing software (security) updates is a whole other topic for another post.</p> If you have any thoughts on this please let me know via mail!</p> Automate Terraform with Atlantis and GitLab 2021-05-26T00:00:00+00:00 Atlantis</a> is an infrastructure as code (IaC) tool to automate Terraform interactions within your GitLab/GitHub MR/PR workflow. This post uses the setup explained in an older post</a>, where we used GitLab as Terraform backend.</p> Instead of running terraform init/apply/plan</code> on your local dev machine, you can now run it automagically when opening a merge request. This works by interacting via comments with Atlantis.</p> </p> Setup</a></h1> 1. Preparations</a></h2> First generate a personal access token for the gitlab user with scope api</code>. As this user will comment on MRs you should create a new atlantisbot</code> user or reuse an already existing bot user.</p> Then generate a secret which will be shared between GitLab and Atlantis. For example with pwgen -s 64 1 </code>. This is not necessary but helps against random request against your Atlantis instance, as it will only accept requests with this secret in the header.</p> 2. Setup Atlantis deployment.</a></h2> Most of my services are running as containers and as Atlantis provides an upstream image there is no good reason to not use it. All the TF_</code> variables must match the configured Terraform backend. HCLOUD_TOKEN</code> and HETZNER_DNS_API_TOKEN</code> are specific token needed by the Hetzner Cloud</a> and Hetzner Cloud DNS</a> Terraform provider.</p> v</span>ersion</span>:</span> "</span>3</span>"</span></span> s</span>ervices</span>:</span></span> a</span>tlantis</span>:</span></span> i</span>mage</span>:</span> r</span>unatlantis/atlantis</span></span> c</span>ontainer_name</span>:</span> a</span>tlantis</span></span> r</span>estart</span>:</span> a</span>lways</span></span> p</span>orts</span>:</span></span> -</span> 1</span>27.0.0.1:4141:4141</span></span> e</span>nvironment</span>:</span></span> A</span>TLANTIS_GITLAB_TOKEN</span>:</span> "</span>wasd</span>"</span></span> A</span>TLANTIS_GITLAB_WEBHOOK_SECRET</span>:</span> "</span>wasd</span>"</span></span> H</span>CLOUD_TOKEN</span>:</span> "</span>wasd</span>"</span></span> H</span>ETZNER_DNS_API_TOKEN</span>:</span> "</span>wasd</span>"</span></span> T</span>F_HTTP_ADDRESS</span>:</span> h</span>ttps://gitlab.mydomain.com/api/v4/projects/<ID>/terraform/state/project</span></span> T</span>F_HTTP_LOCK_ADDRESS</span>:</span> h</span>ttps://gitlab.mydomain.com/api/v4/projects/<ID>/terraform/state/project/lock</span></span> T</span>F_HTTP_UNLOCK_ADDRESS</span>:</span> h</span>ttps://gitlab.mydomain.com/api/v4/projects/<ID>/terraform/state/project/lock</span></span> T</span>F_HTTP_USERNAME</span>:</span> m</span>e</span></span> T</span>F_HTTP_PASSWORD</span>:</span> p</span>assword</span></span> T</span>F_HTTP_LOCK_METHOD</span>:</span> P</span>OST</span></span> T</span>F_HTTP_UNLOCK_METHOD</span>:</span> D</span>ELETE</span></span> c</span>ommand</span>:</span></span> -</span> s</span>erver</span></span> -</span> --</span>atlantis-url=https://atlantis.mydomain.com</span></span> -</span> --</span>gitlab-user=my-bot</span></span> -</span> --</span>repo-allowlist=gitlab.mydomain.com/my-project/iac</span></span> -</span> --</span>gitlab-hostname=gitlab.mydomain.com</span></span></code></pre> Add repositories in the repo-allowlist</code> parameters to enable Atlantis for these specific repositories.</p> 3. Register a GitLab webhook in your project.</a></h2> Add a webhook to all projects previously added in the repo-allowlist</code> option. Select the trigger push events</code>, comments</code> and merge request events</code>. Also check that the URL ends with /events</code>.</p> </p> 4. Setup Caddy</a> as reverse proxy to expose Atlantis.</a></h2> Setup caddy as a revsere proxy to handle TLS certificates and disable all log outputs.</p> a</span>tlantis.mydomain.com {</span></span> r</span>everse_proxy http://localhost:4141 {</span></span> h</span>eader_up Host {http.reverse_proxy.upstream.hostport}</span></span> }</span></span> l</span>og {</span></span> o</span>utput discard</span></span> }</span></span> }</span></span></code></pre>Secure access to the dashboard</a></h1> Atlantis provides a dashboard that displays current locks on running deployments. The lock locks the Terraform state to mitigate multiple apply</code> operations. This is a neat feature as it prevents an inconsistent Terraform state. But it also leaks the name of your repos/projects if the dashboard is available on the public internet (and anyone with access to it can remove any state lock). Also the /events</code> endpoint must be reachable from the public internet as it will be called by the earlier registered webhook(s).</p> </p> This is an task for the basicauth Caddy module</a>. The password must</strong> be hashed as Caddy will not accept plain text passwords. You can generate a password with caddy hash-password</a>.</p> root@server:~#</span> caddy</span> hash-password</span> -</span>-plaintext</span> my-secure-password</span></span> JDJhJDE0JGVsMXhGeDI0UTZ4SkhhSi9yWlZ5Uk93Q08uU0twa05vOXNQVDY0MFpkVGdZMjZUT09Ha2hH</span></span></code></pre>a</span>tlantis.mydomain.com {</span></span> r</span>everse_proxy http://localhost:4141 {</span></span> h</span>eader_up Host {http.reverse_proxy.upstream.hostport}</span></span> }</span></span> b</span>asicauth / {</span></span> m</span>y-user JDJhJDE0JGVsMXhGeDI0UTZ4SkhhSi9yWlZ5Uk93Q08uU0twa05vOXNQVDY0MFpkVGdZMjZUT09Ha2hH</span></span> }</span></span> l</span>og {</span></span> o</span>utput discard</span></span> }</span></span> }</span></span></code></pre>Conclusion</a></h1> Atlantis provides an easy way to let teams work together without problems like locking the state because two members are running Terraform on their local machine at the same time. The interaction within MRs/PRs is simple and it also makes changes visible within an MR/PR. With that a reviewer can directly see what Terraform will do and doesn't have to check out the branch locally and run a terraform plan</code>.</p> Backups with borg/borgmatic 2021-04-21T00:00:00+00:00 I am a big fan of the backup tool borgbackup</a>. Borg</code> is space efficient as it supports compression and deduplication and is also secure, so you don't have to trust the storage provider where your backup will be stored.</p> In this short guide we will setup automatic backups for a server. But most of it can also be used to backup your clients. An addition to borg</code> I use the wrapper borgmatic</a> which will remove any custom scripts needed around borg</code>. Everything related to backups will be stored in a single configuration file that configures borgmatic</code> which will then perform borg</code> operations.</p> As my homeserver is an Arch Linux machine this guide is specific for Arch, but as most distributions nowadays are also systemd based there should not be much of a difference than the used package manager or the versions of some packages.</p> Install needed tools</a></h2> As both borg</code> and borgmatic</code> are in the official repositories they are easily installed with:</p> pacman</span> -</span>S</span> borg</span> borgmatic</span></span></code></pre>Configure borgmatic</a></h2> Borgmatic is configured in a single configuration file that will be stored in /etc/borgmatic/config.yaml</code>. You can also use any other location and provide the path via -c path/to/my/config</code>. A minimal configuration could be like this:</p> l</span>ocation</span>:</span></span> s</span>ource_directories</span>:</span></span> -</span> /</span>home</span></span> -</span> /</span>mnt/storage</span></span> </span> r</span>epositories</span>:</span></span> -</span> u</span>ser@my-backup-host:/mnt/backups/server</span></span> </span> r</span>etention</span>:</span></span> k</span>eep_daily</span>:</span> 7</span></span> k</span>eep_weekly</span>:</span> 4</span></span> k</span>eep_monthly</span>:</span> 6</span></span> </span> s</span>torage</span>:</span></span> e</span>ncryption_passphrase</span>:</span> "</span>my-secret-1337</span>"</span></span> c</span>ompression</span>:</span> z</span>std,22</span></span> </span> c</span>onsistency</span>:</span></span> c</span>hecks</span>:</span></span> -</span> r</span>epository</span></span> -</span> a</span>rchives</span></span></code></pre> This means backup everything in the directories home</code> and mnt/storage</code> to /mnt/backups/server</code> on the host my-backup-host</code>. We also want to keep 7 daily backups, 4 weekly backups and 6 monthly backups. The password to access backups is my-secret-1337</code> and the used compression is zstd</code> with the highest compression (ranges from 0 (almost no compression, but fast) to 22 (the highest compression, but slow)). Read more about the compression options borg provides here</a>.</p> Setup remote server and ssh access</a></h2> In this case we will backup to a remote server via ssh. Therefore generate an ssh key pair on the server you want to backup with ssh-keygen</code>. And copy the public key to the remote server with ssh-copy-id user@my-backup-host</code>. After that you should be able to login to the remote backup server.</p> You also need borg</code> installed on the remote machine (in my case also Arch Linux). As the backups created by borg are stored encrypted you can use the same backup server for multiple clients. But please mind that the backups are only encrypted if specified at the initial creation of the repository, which we will handle now.</p> Setup borg repository</a></h2> Borg provides different encryption models which are documented here</a>. In my case I want encrypted and</strong> authenticated backups these options are also the fastest on most modern AMD/Intel CPUs (as the documentation states). As we use borgmatic</code> as a borg</code> wrapper we use borgmatic init</code> instead borg borg init</code>.</p> borgmatic</span> init</span> -</span>e</span> repokey-blake2</span></span></code></pre>Start the initial backup</a></h2> Backups can now be started with borgmatic</code>. If you want further information like which file is currently backed up or some statistics at the end use borgmatic --files --stats</code>.</p> Automate backups with systemd</a></h2> These systemd units are based on the offical</a> ones from borgmatic. I prefer systemd-timers over a simple cron daemon for two reasons:</p> Systemd will ensure that the job will be executed even if the system is down at the specific time the job was scheduled.</li> Systemd will ensure all logs are written into the journal</code>. With that they easy to inspect via journalctl -u borgmatic</code>.</li> </ol> Store this unit as borgmatic.service</code> in /etc/systemd/system</code>.</p> [Unit]</span></span> Description=borgmatic backup</span></span> Wants=network-online.target</span></span> After=network-online.target</span></span> ConditionACPower=true</span></span> </span> [Service]</span></span> Type=oneshot</span></span> </span> LockPersonality=true</span></span> MemoryDenyWriteExecute=no</span></span> NoNewPrivileges=yes</span></span> PrivateDevices=yes</span></span> PrivateTmp=yes</span></span> ProtectClock=yes</span></span> ProtectControlGroups=yes</span></span> ProtectHostname=yes</span></span> ProtectKernelLogs=yes</span></span> ProtectKernelModules=yes</span></span> ProtectKernelTunables=yes</span></span> RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK</span></span> RestrictNamespaces=yes</span></span> RestrictRealtime=yes</span></span> RestrictSUIDSGID=yes</span></span> SystemCallArchitectures=native</span></span> SystemCallFilter=@system-service</span></span> SystemCallErrorNumber=EPERM</span></span> ProtectSystem=full</span></span> </span> CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW</span></span> </span> Nice=19</span></span> CPUSchedulingPolicy=batch</span></span> IOSchedulingClass=best-effort</span></span> IOSchedulingPriority=7</span></span> IOWeight=100</span></span> </span> Restart=no</span></span> LogRateLimitIntervalSec=0</span></span> </span> ExecStartPre=sleep 1m</span></span> ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" borgmatic --syslog-verbosity 1</span></span></code></pre> Reload all units with systemctl daemon-reload</code> and test the unit with systemctl start borgmatic</code>. You should see that a backup is running via journalctl -u borgmatic -f</code>.</p> If everything is running as expected create a borgmatic.timer</code> with the following content. It must be stored in the same directory as the service file (/etc/systemd/system</code>).</p> [Unit]</span></span> Description=Run borgmatic backup</span></span> </span> [Timer]</span></span> OnCalendar=--* 4:00:00</span></span> Persistent=true</span></span> </span> [Install]</span></span> WantedBy=timers.target</span></span></code></pre> OnCalendar</code> uses the format WeekDay Year-Month-Day Hour:Minute:Second</code> so if you want a backup running every day at 2 am: --* 2:00:00</code>. If you want to run multiple backups simply add addition OnCalendar</code> entries to the timer.</p> Enable the timer with systemctl enable --now borgmatic.timer</code> and check its status:</p> systemctl</span> list-timers</span></span> NEXT</span> LEFT</span> LAST</span> PASSED</span> UNIT</span> ACTIVATES</span></span> Mon</span> 2021-04-19</span> 00:00:00</span> CEST</span> 8h</span> left</span> Sun</span> 2021-04-18</span> 00:00:10</span> CEST</span> 15h</span> ago</span> shadow.timer</span> shadow.service</span></span> Mon</span> 2021-04-19</span> 00:40:11</span> CEST</span> 8h</span> left</span> Mon</span> 2021-04-12</span> 00:32:56</span> CEST</span> 6</span> days</span> ago</span> fstrim.timer</span> fstrim.service</span></span> Mon</span> 2021-04-19</span> 04:00:00</span> CEST</span> 12h</span> left</span> n/a</span> n/a</span> borgmatic.timer</span> borgmatic.service</span></span> Mon</span> 2021-04-19</span> 12:59:22</span> CEST</span> 21h</span> left</span> Sun</span> 2021-04-18</span> 12:59:22</span> CEST</span> 2h</span> 46min</span> ago</span> systemd-tmpfiles-clean.timer</span> systemd-tmpfiles-clean.service</span></span> </span> 4</span> timers</span> listed.</span></span></code></pre> Your server will now create a backup every night at 4:00 am.</p> Check existing backups</a></h2> To list all existing backups use borgmatic list</code>.</p> Restore a backup</a></h2> In my opinion the easiest way to restore a backup is to use the borg mount</code> feature. borg mount</code> allows you to mount a created backup to a specified directory. On Arch you have to install python-llfuse</code> with pacman -S python-llfuse</code>. For example:</p> #</span> borgmatic list</span></span> my-backup-server:my-repo:</span> Listing</span> archives</span></span> Server-2021-04-16T16:36:35.431066</span> Fri,</span> 2021-04-16</span> 16:36:36</span> [0a1b2933a2f1cab81ab164c7ef01370793c5d688710b1c29e94d8d1a8f22c414</span>]</span></span> Server-2021-04-18T12:52:59.997607</span> Sun,</span> 2021-04-18</span> 12:53:01</span> [9e2b0d1b635c36ede5653135821cdcc9c5181615c5cc0b4b8b06ef1e81ff29b8</span>]</span></span> </span> #</span> mount last backup to ~/mount</span></span> borg</span> mount</span> my-backup-server:my-repo::Server-2021-04-18T12:52:59.997607</span> ~/mount</span></span></code></pre> You can now access the backup in ~/mount</code>. Simply copy the files wherever you want.</p> Conclusion</a></h1> borg</code> and borgmatic</code> are perfect tools to automate backups. There are advanced features like append only backups to make it impossible to delete backups if a server has been compromised. Overall I am very happy with by backup setup as it requires no maintenance and simply does its job. And that's exactly how backups should be - just working and boring.</p> Release tools with goreleaser 2021-03-31T00:00:00+00:00 I'm writing most of my own tools in go</a>. Sometime ago I found a tool called goreleaser</a> which automates the release process of software written in go.</p> goreleaser</code> is pretty well documented, therefore this is just a quick overview how I use it. It will do the following things for you:</p> Generate a changelog with the latest commit since the last release.</li> Build artifacts for different architectures (Linux/Mac/Windows).</li> Create a GitHub/GitLab release and attach changelog and artifacts.</li> </ul> Initial setup</a></h2> Install goreleaser</code> on your machine. (Check out the official docs here</a>) I went with the go install github.com/goreleaser/goreleaser@latest</code> method.</p> </li> Run goreleaser init</code> to get a default .goreleaser.yml</code> which is used to configure goreleaser.</p> </li> Adjust the configuration.</p> </li> </ol> b</span>efore</span>:</span></span> h</span>ooks</span>:</span></span> -</span> g</span>o mod tidy</span></span> b</span>uilds</span>:</span></span> -</span> e</span>nv</span>:</span></span> -</span> C</span>GO_ENABLED=0</span></span> g</span>oos</span>:</span></span> -</span> l</span>inux</span></span> -</span> w</span>indows</span></span> -</span> d</span>arwin</span></span> a</span>rchives</span>:</span></span> -</span> r</span>eplacements</span>:</span></span> d</span>arwin</span>:</span> D</span>arwin</span></span> l</span>inux</span>:</span> L</span>inux</span></span> w</span>indows</span>:</span> W</span>indows</span></span> a</span>md64</span>:</span> x</span>86_64</span></span> c</span>hecksum</span>:</span></span> n</span>ame_template</span>:</span> '</span>checksums.txt</span>'</span></span> s</span>napshot</span>:</span></span> n</span>ame_template</span>:</span> "</span>{{ .Tag }}-next</span>"</span></span> c</span>hangelog</span>:</span></span> s</span>ort</span>:</span> a</span>sc</span></span> f</span>ilters</span>:</span></span> e</span>xclude</span>:</span></span> -</span> '</span>^docs:</span>'</span></span> -</span> '</span>^test:</span>'</span></span></code></pre> I used this configuration to release go-jump</a>.</p> Release via CI</a></h2> To simplify the release process even more you can run gorelease</code> on GitHub Actions or GitLab CI. This is an example for GitHub Actions. Store it as .github/workflows/release.yaml</code> in your repository root.</p> n</span>ame</span>:</span> g</span>oreleaser</span></span> </span> on</span>:</span></span> p</span>ush</span>:</span></span> t</span>ags</span>:</span></span> -</span> '</span></span>'</span></span> </span> j</span>obs</span>:</span></span> g</span>oreleaser</span>:</span></span> r</span>uns-on</span>:</span> u</span>buntu-latest</span></span> s</span>teps</span>:</span></span> -</span> n</span>ame</span>:</span> C</span>heckout</span></span> u</span>ses</span>:</span> a</span>ctions/checkout@v2</span></span> w</span>ith</span>:</span></span> f</span>etch-depth</span>:</span> 0</span></span> -</span> n</span>ame</span>:</span> S</span>et up Go</span></span> u</span>ses</span>:</span> a</span>ctions/setup-go@v2</span></span> w</span>ith</span>:</span></span> g</span>o-version</span>:</span> 1.16</span></span> -</span> n</span>ame</span>:</span> R</span>un GoReleaser</span></span> u</span>ses</span>:</span> g</span>oreleaser/goreleaser-action@v2</span></span> w</span>ith</span>:</span></span> v</span>ersion</span>:</span> l</span>atest</span></span> a</span>rgs</span>:</span> r</span>elease --rm-dist</span></span> e</span>nv</span>:</span></span> G</span>ITHUB_TOKEN</span>:</span> $</span>{{ secrets.GITHUB_TOKEN }}</span></span></code></pre> Then create a new git tag and push it:</p> git</span> tag</span> -</span>a</span> v0.1.0</span> -</span>m</span> "</span>My release</span>"</span></span> git</span> push</span> origin</span> v0.1.0</span></span></code></pre>Conclusion</a></h2> I really like how easy gorelease</code> is to use and it removes a repetitive task from my release process. The documentation is perfect and the defaults are sane (the example above are more or less exactly the defaults!). So I'm going to add goreleaser</code> to all of my projects!</p> Arch Linux and Sony WH-1000MX4 2021-03-28T00:00:00+00:00 I recently got active noise-cancelling headphones (Sony WH-1000MX4</a>). They work just fine with my Android phone but on my Linux notebook I had a bit of a fight to get them working. I am running Arch Linux</a> on most of my machines, so this post is written with Arch specific commands, but most of them (and the general idea) should also work on Ubuntu or Debian based distros. Keep in mind that older versions of PulseAudio and/or PipeWire may not have the features described here! So lets get started with the things I tried (if you are not interested in these just jump to [PipeWire]({{< relref "linux-and-bluetooth.md#pipewire" >}} "link to PipeWire")).</p> Failed experiments</a></h2> At first we have to install the bluez</code> and bluez-utils</code> packages. bluez-utils</code> will provide bluetoothctl</code> to interact via CLI with bluez</code>. As I want to avoid unnecessary sources of error I'm not going for a graphical helper for now.</p> All of this is (as always) documented in the arch wiki</a>. The following is more or less exactly what can be found there.</p> Start the Bluetooth service with systemctl start bluetooth.service</code></li> Open bluetoothctl</code></li> Power on the Bluetooth module with power on</code></li> Scan for devices with scan on</code></li> Wait for the headphones to show up. In my case they did not, so I got the Bluetooth MAC from my phone.</li> Pair the headphones with pair <MAC></code></li> Connect the headphones with connect <MAC></code></li> </ol> Until now everything was working as expected, but now the struggling started. As I played some music the sound was just bad. Therefore I checked the configuration in pavucontrol</code> and saw that the select profile was HSP/HDF</code>. This is also documented in the Arch wiki</a>. After a bit of research I now know that I want to use the A2DP</code> profile for crispy audio output, but the internal microphone will only work with HSP/HDF</code>. As my notebook got a decent internal microphone and I also got an external one this is no problem for my use case. But keep this in mind if you need the internal mic of the Sony WH-1000MX4 or any other Bluetooth headphones.</p> I tried a few more things mentioned in the troubleshooting guide</a> like:</p> #</span> /etc/pulse/default.pa</span></span> #</span> Let me use l2dp via bluetooth</span></span> load-module</span> module-bluetooth-policy</span> auto_switch=</span>false</span></span></code></pre>#</span> /etc/bluetooth/main.conf</span></span> [</span>General</span>]</span></span> Disable</span>=</span>H</span>e</span>a</span>d</span>s</span>e</span>t</span></span> MultiProfile</span>=</span>m</span>u</span>l</span>t</span>i</span>p</span>l</span>e</span></span></code></pre> But none of them worked, and the sound was still just bad. So I did some research about how all of these Bluetooth stuff actually works and discovered that the Sony WH-1000MX4 are using the LDAC</code> codec which is not</strong> supported by vanilla pulseaudio. An alternative is APTx</code> which may be supported by most other headphones, as LDAC</code> is a proprietary Sony codec. Also, you want to use a A2DP</code> sink instead of HSP/HDF</code> (mind the earlier described disadvantage of not working microphones!). Therefore, it's time to revisit the Arch wiki and search for this specific codec.</p> </p> Migration to PipeWire</a></h2> At this point I discovered</a> a pulseaudio module</a> that adds support for APTx</code> and LDAC</code> codecs. But it was deprecated</a> in January 2021 with a recommendation to use vanilla PulseAudio or PipeWire. I already tried PulseAudio earlier (and was annoyed) so I decided to give PipeWire a chance. And PipeWire supports LDAC since release 0.3.19</code> 🎉!</p> Many bluetooth improvements in LDAC, AptX-HD. AAC was also added. Headsets should work better now. -- https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/NEWS/</li> </ul> </blockquote> There are 3 packages needed to get started: libldac</code>, pipewire</code> and pipewire-pulse</code>. Install them with pacman -S libldac pipewire pipewire-pulse</code>.</p> After that reboot your machine and connect the headphones with bluetoothctl</code> as described earlier. I had to remove the headphones with remove <MAC></code> and pair them again. Now there should be the A2DP</code> option in pavucontrol</code> and the LDAC</code> codec.</p> </p> If you want to learn more about PipeWire (and its history) there is a recently published article on LWN.net</a>.</p> Getting the touch gestures to work</a></h2> Thanks to Marcelo for pointing this out!</p> As always this was already documented in the arch wiki</a>. Simply create a new systemd user unit:</p> ~</span>/.config/systemd/user/mpris-proxy.service</span></span> </span> [</span>Unit</span>]</span></span> Description</span>=</span>F</span>o</span>r</span>w</span>a</span>r</span>d</span> bluetooth</span> media</span> controls</span> to</span> MPRIS</span></span> </span> [</span>Service</span>]</span></span> Type</span>=</span>s</span>i</span>m</span>p</span>l</span>e</span></span> ExecStart</span>=</span>/</span>u</span>s</span>r</span>/</span>b</span>i</span>n</span>/</span>m</span>p</span>r</span>i</span>s</span>-</span>p</span>r</span>o</span>x</span>y</span></span> </span> [</span>Install</span>]</span></span> WantedBy</span>=</span>d</span>e</span>f</span>a</span>u</span>l</span>t</span>.</span>t</span>a</span>r</span>g</span>e</span>t</span></span></code></pre> Reload all units with systemd --user daemon-reload</code> and enable the unit with systemctl --user enable --now mpris-proxy</code>. Now all touch gestures and the wearing detection</code> should work!</p> Conclusion</a></h2> As I thought about getting Bluetooth headphones I have not given a thought that Bluetooth (with modern codecs) could be problematic on Linux. Especially as Sony decided to use their own LDAC</code> codec instead of something open (but I'm not even sure if there is an open Bluetooth codec).</p> So far I had no problems with PipeWire, but I am only running it for a few weeks now, so we will see how it holds up. Keep in mind that PipeWire is not stable yet! There might be updates that require configuration changes (like this one</a>).</p> Setup Wallabag 2021-02-28T00:00:00+00:00 I am subscribed to a lot of blog feeds, but (sadly) I don't have enough time to read all of them. This is especially the case for rather long (but often interesting) posts. Therefore it would be perfect to save these posts and read them in the near future.</p> There are already existing solutions for this like Mozilla Pocket</a>. But as always I want to host this service on my own server (I have to justify the server somehow). A self-hosted solution is wallabag</a>.</p> wallabag is a self-hostable PHP application allowing you to not miss any content anymore. Click, save and read it when you can. It extracts content so that you can read it when you have time. -- https://github.com/wallabag/wallabag#what-is-wallabag</cite></p> </blockquote> Wallabag needs 2 things to work properly:</p> A wallabag server to actually store the articles.</p> </li> The wallabag browser plugin Wallabagger</a> to add new articles to your backlog.</p> </li> </ul> Additionally wallabag can be accessed via browser or one of the official apps (Android</a> and iOS</a>).</p> I am going to run the wallabag server as a container because it is easy to manage the configuration with git and to migrate it to a new host (or share it with you).</p> Installation with Docker</a></h2> The following files are stored in this layout:</p> .</span></span> ├──</span> data</span></span> ├──</span> docker</span></span> │ </span> ├──</span> Dockerfile</span></span> │ </span> └──</span> nginx</span></span> │ </span> ├──</span> nginx.conf</span></span> │ </span> └──</span> wallabag</span></span> ├──</span> docker-compose.yml</span></span> └──</span> images</span></span></code></pre> The data</code> and images</code> directories will be used to store persistent data like images and the scraped articles.</p> The docker-compose.yml</code> is based on an example provided</a> by the wallabag project.</p> As I host the server at home this configuration is not hardened at all! I can't recommend using</strong> this configuration to host this service on the public internet!</p> So let's have a look at the main docker-compose</code> file. We declare two containers, wallabag</code> itself and a mariadb</code> database. There is also a nginx container to handle missing icons, but more about this later.</p> v</span>ersion</span>:</span> '</span>3</span>'</span></span> s</span>ervices</span>:</span></span> n</span>ginx</span>:</span></span> b</span>uild</span>:</span></span> c</span>ontext</span>:</span> .</span>/docker</span></span> p</span>orts</span>:</span></span> -</span> 0</span>.0.0.0:1337:1337</span></span> l</span>inks</span>:</span></span> -</span> w</span>allabag</span></span> w</span>allabag</span>:</span></span> i</span>mage</span>:</span> w</span>allabag/wallabag:2.4.1</span></span> e</span>nvironment</span>:</span></span> -</span> M</span>YSQL_ROOT_PASSWORD=wallaroot</span></span> -</span> S</span>YMFONY__ENV__DATABASE_DRIVER=pdo_mysql</span></span> -</span> S</span>YMFONY__ENV__DATABASE_HOST=db</span></span> -</span> S</span>YMFONY__ENV__DATABASE_PORT=3306</span></span> -</span> S</span>YMFONY__ENV__DATABASE_NAME=wallabag</span></span> -</span> S</span>YMFONY__ENV__DATABASE_USER=wallabag</span></span> -</span> S</span>YMFONY__ENV__DATABASE_PASSWORD=wallapass</span></span> -</span> S</span>YMFONY__ENV__DATABASE_CHARSET=utf8mb4</span></span> -</span> S</span>YMFONY__ENV__MAILER_HOST=127.0.0.1</span></span> -</span> S</span>YMFONY__ENV__MAILER_USER=~</span></span> -</span> S</span>YMFONY__ENV__MAILER_PASSWORD=~</span></span> -</span> S</span>YMFONY__ENV__FROM_EMAIL=wallabag@example.com</span></span> -</span> S</span>YMFONY__ENV__DOMAIN_NAME=http://DOMAIN</span></span> v</span>olumes</span>:</span></span> -</span> .</span>/images:/var/www/wallabag/web/assets/images</span></span> d</span>b</span>:</span></span> i</span>mage</span>:</span> m</span>ariadb:10</span></span> e</span>nvironment</span>:</span></span> -</span> M</span>YSQL_ROOT_PASSWORD=wallaroot</span></span> v</span>olumes</span>:</span></span> -</span> .</span>/data:/var/lib/mysql</span></span> r</span>edis</span>:</span></span> i</span>mage</span>:</span> r</span>edis:alpine</span></span></code></pre> The nginx is needed because of CORS</a> errors. A short look on the GitHub issue tracker showed that this is a known problem</a>. A fix</a> is to use a nginx as reverse proxy and add the following header add_header 'Access-Control-Allow-Origin' 'https://example.com'</code>. And this is exactly what I did.</p> The nginx</code> is based on the upstream image, but we will run a quick apt update && apt upgrade</code> to ensure we have the latest packages installed. Then we copy the base config nginx.conf</code> and our site config wallabag</code> into the container.</p> FROM</span> nginx:1.19 </span>AS</span> nginx</span></span> ENV</span> TZ=Etc/UTC</span></span> </span> RUN</span> export DEBIAN_FRONTEND=noninteractive \</span></span> && apt-get update \</span></span> && apt-get dist-upgrade -y --no-install-recommends</span></span> </span> COPY</span> nginx/wallabag /etc/nginx/sites-available/wallabag</span></span> COPY</span> nginx/nginx.conf /etc/nginx/nginx.conf</span></span> </span> RUN</span> mkdir /etc/nginx/sites-enabled/ \</span></span> && ln -s /etc/nginx/sites-available/wallabag /etc/nginx/sites-enabled/wallabag \</span></span> && rm /etc/nginx/conf.d/default.conf</span></span> </span> EXPOSE</span> 1337</span></span></code></pre> The main configuration of our nginx is basic, as I do not intent to expose this service to the public internet. I also do not care about error or access logs.</p> events {}</span></span> </span> http {</span></span> server_tokens off;</span></span> sendfile on;</span></span> keepalive_timeout 65;</span></span> tcp_nodelay on;</span></span> include /etc/nginx/mime.types;</span></span> </span> access_log off;</span></span> error_log /dev/null;</span></span> </span> include /etc/nginx/conf.d/.conf;</span></span> include /etc/nginx/sites-enabled/;</span></span> }</span></span> </span></code></pre> The site configuration exposes the server on port 1337</code> and adds the missing CORS</code> header described earlier.</p> server</span> {</span></span> listen</span> 1337</span>;</span></span> #</span>server_name wallabag.foo.bar;</span></span> </span> location</span> /</span> {</span></span> proxy_pass</span> http://wallabag</span>;</span></span> proxy_set_header</span> X-Forwarded-Host</span> $</span>server_name</span>;</span></span> proxy_set_header</span> X-Forwarded-Proto</span> https</span>;</span></span> proxy_set_header</span> X-Forwarded-For</span> $</span>remote_addr</span>;</span></span> }</span></span> </span> add_header</span> '</span>Access-Control-Allow-Origin</span>'</span> '</span>http://DOMAIN</span>'</span></span> </span> }</span></span></code></pre>Deployment</a></h2> Now to the easy part. Run docker-compose up</code> to start the wallabag stack. Wallabag should now be online, and you can pair the browser plugin with the instance.</p> Blog deployment with GitLab CI 2021-02-13T00:00:00+00:00 As stated earlier</a> I use a combination of technologies to run this blog.</p> This blog is created with hugo</a> (a static site generator) and uses the papermod</a> theme. Additional technologies that are used:</p> GitLab as git remote</li> GitLab CI to build container images (development and production)</li> Docker to run the images (development on my home server and production on a machine on the public internet)</li> Watchtower</a> to check for new images every ~5 minutes</li> Caddy</a> to handle TLS certificates (only for production)</li> A VM on Hetzner Cloud</a> used as a host for multiple containers</li> </ul> GitLab</a></h2> At first the development and production hosts need access to the container registry. Therefore:</p> Create a new deploy_token</code> official docs</a></li> </ol> </p> Login on both the development/production docker host using the created user and password.</li> </ol> docker</span> login</span> <</span>your.gitlab.instance:456</span>7></span></span></code></pre>CI</a></h2> Here are a few things to mention.</p> The variable GIT_SUBMODULE_STRATEGY: recursive</code> is needed as hugo themes are added as git submodules</a>. Therefore all themes must be cloned before</strong> the static page can be build.</p> I use two stages to build containers for production and development with a single CI run. The build-containers-production</code> stage will only be executed if the $CI_COMMIT_BRANCH</code> equals main</code> as only code/articles on this branch are ready to be published.</p> The second stage build-container-dev</code> is used for a development deployment on my home server. This stage will be executed on every branch, so I can always check if the article I am currently working on looks good. All of this could be achieved by using hugo server -D</code> on my local machine, but this is not as much fun as playing around with GitLab CI, Docker and my home server (which will be covered in another post in the near future).</p> The script</code> part in both stages are more or less the same. For production</code> the created image will be tagged with production</code> and for development</code> with dev</code>.</p> An important change is cd myblog && sed -i 's\|https://sterba.dev\|http://192.168.0.2:9337\|g' config.yaml && cd ..</code>. Without changing the baseURL</code> all links in the dev image will direct to https://sterba.dev/...</code>.</p> Please mind that this will not</strong> work if you are using hugo drafts</code>. Drafts will not be rendered with this build script. This is no problem for my workflow as only finished posts are on the main</code> branch.</p> v</span>ariables</span>:</span></span> G</span>IT_SUBMODULE_STRATEGY</span>:</span> r</span>ecursive</span></span> </span> s</span>tages</span>:</span></span> -</span> b</span>uild-containers-production</span></span> -</span> b</span>uild-container-dev</span></span> </span> b</span>efore_script</span>:</span></span> -</span> d</span>ocker info</span></span> </span> b</span>uild_image_production</span>:</span></span> s</span>tage</span>:</span> b</span>uild-containers-production</span></span> r</span>ules</span>:</span></span> -</span> i</span>f</span>:</span> '</span>$CI_COMMIT_BRANCH == "main"</span>'</span></span> w</span>hen</span>:</span> a</span>lways</span></span> -</span> w</span>hen</span>:</span> n</span>ever</span></span> </span> s</span>cript</span>:</span></span> -</span> d</span>ocker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY</span></span> -</span> d</span>ocker build -t your.gitlab.com:4567/andre/blog:production -f docker/Dockerfile .</span></span> -</span> d</span>ocker push your.gitlab.com:4567/andre/blog:production</span></span> </span> b</span>uild_image_test</span>:</span></span> s</span>tage</span>:</span> b</span>uild-container-dev</span></span> </span> s</span>cript</span>:</span></span> -</span> s</span>ed -i 's\|https://sterba.dev\|http://192.168.0.2:9337\|g' myblog/config.yaml</span></span> -</span> d</span>ocker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY</span></span> -</span> d</span>ocker build -t your.gitlab.com:4567/andre/blog:dev -f docker/Dockerfile .</span></span> -</span> d</span>ocker push your.gitlab.com:4567/andre/blog:dev</span></span></code></pre>Docker</a></h2> The blog container is a multistage build because I only want the generated static content in my webserver container (nginx</code>). Therefore the first container (based on debian:buster-slim</code>) installs the current version of hugo</code>, copies the blog in an app</code> directory and generates the static content.</p> The second container is a basic nginx</code> with a simple configuration which will be displayed later. With COPY --from=build /app/public/ /opt/public/</code> the previously generated content is copied into the configured webserver root directory of nginx</code>.</p> Always</strong> update the packages in your base images (in this case debian</code>)! Because the base images are not rebuild every few days they do not contain the newest version of a specific package. So you could go to production with vulnerable or old packages.</p> FROM</span> debian:buster-slim </span>AS</span> build</span></span> </span> ENV</span> HUGO_VERSION=0.80.0</span></span> ADD</span> https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz /tmp/</span></span> RUN</span> tar -xf /tmp/hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz -C /usr/bin/</span></span> </span> COPY</span> ./myblog /app</span></span> </span> RUN</span> hugo --source=/app/</span></span> </span> FROM</span> nginx:stable </span>AS</span> nginx</span></span> ENV</span> TZ=Etc/UTC</span></span> </span> RUN</span> export DEBIAN_FRONTEND=noninteractive \</span></span> && apt-get update \</span></span> && apt-get dist-upgrade -y --no-install-recommends</span></span> </span> COPY</span> docker/nginx/blog /etc/nginx/sites-available/blog</span></span> COPY</span> docker/nginx/nginx.conf /etc/nginx/nginx.conf</span></span> </span> RUN</span> mkdir /etc/nginx/sites-enabled/ \</span></span> && ln -s /etc/nginx/sites-available/blog /etc/nginx/sites-enabled/blog \</span></span> && rm /etc/nginx/conf.d/default.conf</span></span> </span> COPY</span> --from=build \</span></span> /app/public/ /opt/public/</span></span> </span> EXPOSE</span> 9337</span></span></code></pre> The nginx</code> site configuration is pretty simple. The root</code> is the /opt/public</code> folder we used earlier to copy our build artifacts to. As hugo</code> and papermod</code> generate a custom 404</code> page all 404</code>'s should be directed to the generated page instead of the default nginx page.</p> server {</span></span> listen 9337;</span></span> root /opt/public/;</span></span> index index.html;</span></span> error_page 404 /404.html;</span></span> location / {</span></span> try_files $uri $uri/ =404;</span></span> }</span></span> }</span></span></code></pre> The main configuration of nginx</code> is also pretty straight forward. It simply loads the previously created site configuration. I disabled any logging as I don't care about logs because I don't want any information about my readers (yes, I actually care about your privacy!).</p> events {}</span></span> </span> http {</span></span> server_tokens off;</span></span> sendfile on;</span></span> keepalive_timeout 65;</span></span> tcp_nodelay on;</span></span> include /etc/nginx/mime.types;</span></span> </span> access_log off;</span></span> error_log /dev/null;</span></span> </span> include /etc/nginx/conf.d/.conf;</span></span> include /etc/nginx/sites-enabled/*;</span></span> }</span></span></code></pre>Deployment</a></h2> The deployment process contains two different components. First there is docker-compose</code> which handles the container build previously. Additionally, there is a caddy</code> server which acts as reverse proxy (only used for production).</p> Docker Compose</a></h3> Although there are two deployment environments their configuration is the same. Only the used tag</code> is different. There is not only the image of the blog we build earlier, but also a watchtower</a> container. Watchtower</a> can be used to monitor the upstream registry for a new image. So after every CI run for development</code> watchtower will pull the new image, shutdown the running container and restart it with the new image.</p> But there are other containers running on the host and by default watchtower</code> will try to update all containers currently running on a host. A fix is to provide the --label-enable</code> command flag. With this flag only containers with the label com.centurylinklabs.watchtower.enable</code> set to true</code> will be updated by watchtower</code>.</p> Replace the port configuration 10.13.37.1</code> with the ip of your machine or use 0.0.0.0</code> to bind to any address.</p> v</span>ersion</span>:</span> "</span>3</span>"</span></span> s</span>ervices</span>:</span></span> a</span>ndre-blog</span>:</span></span> i</span>mage</span>:</span> y</span>our.gitlab.com/andre/blog:<production or dev></span></span> c</span>ontainer_name</span>:</span> a</span>ndre-blog</span></span> r</span>estart</span>:</span> a</span>lways</span></span> l</span>abels</span>:</span></span> -</span> "</span>com.centurylinklabs.watchtower.enable=true</span>"</span></span> p</span>orts</span>:</span></span> -</span> 1</span>92.168.0.2:9337:9337</span></span> w</span>atchtower</span>:</span></span> i</span>mage</span>:</span> c</span>ontainrrr/watchtower</span></span> c</span>ommand</span>:</span> --</span>label-enable --interval 30</span></span> v</span>olumes</span>:</span></span> -</span> /</span>var/run/docker.sock:/var/run/docker.sock</span></span> -</span> ~</span>/.docker/config.json:/config.json</span></span></code></pre>Caddy</a></h3> I mostly rely on Caddy</a> as web server because it has an easy configuration and provides TLS certificates with Let's Encrypt</a> out of the box.</p> There are two sub-settings in the main config:</p> The configuration for the reverse-proxy</a> module which simply forwards all traffic to the container created above.</p> </li> Some additional headers.</p> </li> </ol> sterba.dev {</span></span> reverse_proxy http://localhost:9337 {</span></span> }</span></span> header {</span></span> Strict-Transport-Security max-age=31536000;</span></span> Referrer-Policy no-referrer-when-downgrade</span></span> }</span></span> }</span></span></code></pre>Conclusion</a></h2> As you can see the setup is pretty straight forward and requires not that much configuration. With this setup I have a complete CI/CD stack to build, test (in a development environment) and deploy the blog. The release process of a finished post is no work at all. I simply need to push a post to the main</code> branch and everything else will be handled by the components presented in this post.</p> GitLab as Terraform Backend 2021-01-23T00:00:00+00:00 I manage a few servers with some friends. We mostly rely on Terraform to update/change DNS records or create new VMs.</p> Terraform uses a state</a> to store the last applied configuration of the managed infrastructure. The default location for this state is your local filesystem, but if you manage your infrastructure with multiple persons it must be shared. This can be problematic as it also contains secrets, like api tokens, so it should not be accessible for everyone.</p> Terraform uses backends</a> to provide a solution for storing the state somewhere remote instead of your local disk. If you have an account on a GitLab</a> instance you already have access to a remote storage for your state which can be used as http backend</a>. The backend also handles locking, so you can't execute terraform apply</code> from different machines at the same time. You can find it also the official documentation, but I will provide you with a quick start.</p> Create a personal API token for your GitLab account with the scope api</code>.</p> </li> Add the following settings to your main.tf</code>.</p> </li> </ol> terraform {</span></span> backend "http" {</span></span> }</span></span> }</span></span></code></pre> Run terraform init</code> with the following arguments and replace USERNAME</code></li> API-TOKEN</code></li> PROJECT-ID</code></li> STATE-NAME</code></li> </ul> </li> </ol> terraform</span> init</span> \</span></span> -</span>backend-config=</span>"</span>address=https://gitlab.com/api/v4/projects/<PROJECT-ID>/terraform/state/<STATE-NAME></span>"</span> \</span></span> -</span>backend-config=</span>"</span>lock_address=https://gitlab.com/api/v4/projects/<PROJECT-ID>/terraform/state/<STATE-NAME>/lock</span>"</span> \</span></span> -</span>backend-config=</span>"</span>unlock_address=https://gitlab.com/api/v4/projects/<PROJECT-ID>/terraform/state/<STATE-NAME>/lock</span>"</span> \</span></span> -</span>backend-config=</span>"</span>username=USERNAME</span>"</span> \</span></span> -</span>backend-config=</span>"</span>password=API-TOKEN</span>"</span> \</span></span> -</span>backend-config=</span>"</span>lock_method=POST</span>"</span> \</span></span> -</span>backend-config=</span>"</span>unlock_method=DELETE</span>"</span> \</span></span> -</span>backend-config=</span>"</span>retry_wait_min=5</span>"</span></span></code></pre> Run terraform plan</code> or terraform apply</code> and the remote state should be used.</li> </ol> There are other ways to use a remote state with Terraform like a Postgres database</a> or S3</a>, but if you have access to a GitLab instance or an account on gitlab.com</a> this is a simple way to get the advantages of a remote Terraform state.</p> Terraform and env variables 2021-01-08T00:00:00+00:00 If you work with Terraform</a> you might know the struggle setting the right environment variables. Especially if you work with different Terraform configurations on a daily basis.</p> I found 2 tools (or a tool and a script) to make this much easier. The first is direnv</a> which automagically sources env variables stored in a .envrc</code> file if you change in a managed directory.</p> $</span> mkdir</span> direnv-test</span></span> $</span> echo</span> "</span>export mykey=wasd</span>"</span> >></span> .envrc</span></span> $</span> cd</span> ..</span> &&</span> cd</span> direnv-test</span></span> </span> #</span> direnv will show this error</span></span> direnv:</span> error</span> /home/andre/tmp/direnv-test/.envrc</span> is</span> blocked.</span></span> Run</span> `</span>direnv</span> allow</span>`</span> to</span> approve</span> its</span> content</span></span> </span> $</span> direnv</span> allow</span></span> </span> $</span> cd</span> ..</span> &&</span> cd</span> direnv-test</span></span> direnv:</span> loading</span> ~/tmp/direnv-test/.envrc</span></span> direnv:</span> export</span> +mykey</span></span> </span> $</span> echo</span> $</span>mykey</span></span> wasd</span></span></code></pre> And that's it! Simply install direnv</code>, create a .envrc</code>, run direnv allow</code> and all the needed env variables will be set when you change in the directory.</p> I mentioned 2 tools at the beginning and the second tool is a little script I use to get API tokens out of my password store. My password store is powered by gopass</a> therefore it is pretty easy to script around it.</p> The setup is based on this blog post of Shibumi</a>. Thanks for the idea!</p> Store this script in the same directory as your Terraform code.</p> #!</span>/bin/sh</span></span> </span> cloud_api_key</span>=</span>$(</span>gopass</span> show</span> -</span>-password</span> tokens/cloud-project-token</span>)</span></span> </span> echo</span> "</span>{ </span>\"</span>cloud_api_key</span>\"</span>: </span>\"</span>$</span>{</span>cloud_api_key</span>}</span>\"</span> }</span>"</span></span></code></pre> It will be called by the Terraform module external</code>.</p> data "external" "cloud_api_key" {</span></span> program = ["${path.module}/fetch-key.sh"]</span></span> }</span></span> </span> provider "mycloud" {</span></span> token = data.external.cloud_api_key.result.cloud_api_key</span></span> }</span></span></code></pre> Terraform will use this script to get the needed token and that's it.</p> This is so much better than going back in your shell history or use reverse search (and sometimes apply the wrong config to the wrong infrastructure).</p> SSH Agents 2021-01-03T00:00:00+00:00 With every installation of a new system I struggle setting up an ssh-agent. Although it only requires 4 simple steps I somehow manage to always miss at least one of them.</p> I use a systemd user unit to start the ssh-agent as I don't want to manage services with shell scripts or in my sway</a> config, if avoidable.</p> This is more or less exactly what can be found in the Arch wiki</a>.</p> Step 1</a></h3> Create systemd user unit in ~/.config/systemd/user/ssh-agent.service</code>.</p> [Unit]</span></span> Description=SSH key agent</span></span> </span> [Service]</span></span> Type=simple</span></span> Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket</span></span> # DISPLAY required for ssh-askpass to work</span></span> Environment=DISPLAY=:0</span></span> ExecStart=/usr/bin/ssh-agent -D -a $SSH_AUTH_SOCK</span></span> </span> [Install]</span></span> WantedBy=default.target</span></span></code></pre>Step 2</a></h3> Add SSH_AUTH_SOCK DEFAULT="${XDG_RUNTIME_DIR}/ssh-agent.socket"</code> to .pam_environment</code>.</p> Step 3</a></h3> Add AddKeysToAgent yes</code> to your .ssh/config</code> to actually use the ssh-agent you set up previously.</p> Step 4</a></h3> Let systemd pick up the new unit with systemctl --user daemon-reload</code> and start the ssh-agent with systemctl start --user ssh-agent</code>. You should also enable the service with systemctl enable --user ssh-agent</code> (Thank you, Neil!).</p> That's it! If something is not working properly now restart your user session (logout and login again). Now you should only be prompted once for your ssh-key passphrase at the start of your session.</p> Init 2020-12-31T00:00:00+00:00 Welcome to my blog!</p> I've wanted to start a blog for a long time, but I've never done it because of unclear reasons. So here it is!</p> technical details</a></h2> The blog is created with hugo</a> and the papermod</a> theme. I use GitLab-CI to build a container that is pushed into the GitLab container registry. Caddy</a> is used to serve the blog (as reverse proxy in front of the container) and to handle TLS.</p> more content</a></h2> I plan to post some of the more interesting problems in my daily life as a user of technology. So be ready to read about my struggles of getting things to run or program things that I absolutely need!</p> about me</a></h2> I'm a computer science student and an enthusiast for open source software. I'm a Linux user an my favorite desktop distribution is Arch Linux</a>. For servers I mostly rely on Debian</a>.</p> I'm interested in distributed systems, (cloud) infrastructure, privacy and independent hosting of services. Therefore I host most of the services at home or on several machines on the internet.</p>

André Sterba

Nix Dev Shells

2026-04-17T00:00:00+00:00

I recently stumbled across Nix again — specifically its declarative dev shells</a>, which turned out to be a good fit for a problem I kept running into.

My use case is pretty straightforward. I have to work with ops tools like Ansible, Terraform, and a handful of supporting utilities, often in different versions across different projects. Anyone who worked with Ansible has likely felt the pain with not matching Python versions or dependencies and I really wanted to get rid of this problem. Luckily, there are two ways to approach this with Nix shells.

The first is storing a dedicated configuration in each repository and activating it manually — or automatically with something like direnv</code> — when entering the project directory.

The second is maintaining a general Nix shell registered in the local Nix registry, which can then be sourced from anywhere.

I'm currently using the second approach, though the per-repository option is interesting, especially when other engineers on the same project are also using Nix, but I have yet to play around with it.

The layout of a custom shell looks like this:

➜</span>  infra-ops</span> git:</span>(</span>main</span>)</span> tree</span></span>
.</span></span>
├──</span> flake.lock</span></span>
├──</span> flake.nix</span></span>
└──</span> README.md</span></span>
</span>
1</span> directory,</span> 3</span> files</span></span></code></pre>
flake.lock</code> is the lockfile for pinned versions — comparable to package-lock.json</code> in the JavaScript ecosystem. The interesting part is flake.nix</code>, which declares all packages available in the dev shell.
I won't go into details here, because I also don't understand all of it yet (well I just know how to use it).
I'm also using flake-utils</code> to abstract over the target architecture (x86_64 or aarch64).</p>
{</span></span>
  description</span> =</span> "</span>Reproducible Infrastructure Environment</span>"</span>;</span></span>
</span>
  #</span> 1. Inputs: Where do we get the packages from?</span></span>
  inputs</span> =</span> {</span></span>
    nixpkgs</span>.</span>url</span> =</span> "</span>github:NixOS/nixpkgs/nixos-25.11</span>"</span>;</span></span>
    flake-utils</span>.</span>url</span> =</span> "</span>github:numtide/flake-utils</span>"</span>;</span></span>
  }</span>;</span></span>
</span>
  #</span> 2. Outputs: What does this flake produce?</span></span>
  outputs</span> =</span></span>
    {</span></span>
      self</span>,</span></span>
      nixpkgs</span>,</span></span>
      flake-utils</span>,</span></span>
    }</span>:</span></span>
    flake-utils</span>.</span>lib</span>.</span>eachDefaultSystem</span> (</span></span>
      system</span>:</span></span>
      let</span></span>
        pkgs</span> =</span> import</span> nixpkgs</span> {</span></span>
          inherit</span> system</span>;</span></span>
          config</span> =</span> {</span></span>
            allowUnfree</span> =</span> true</span>;</span></span>
          }</span>;</span></span>
        }</span>;</span></span>
</span>
        #</span> 3. Custom Python Environment</span></span>
        #</span> If your Ansible modules need specific Python libraries,</span></span>
        #</span> add them to this list.</span></span>
        pythonWithPackages</span> =</span> pkgs</span>.</span>python3</span>.</span>withPackages</span> (</span></span>
          p</span>:</span> with</span> p</span>;</span> [</span></span>
            ansible-core</span> #</span> The core Ansible engine</span></span>
            requests</span>     #</span> Example: HTTP library</span></span>
            netaddr</span>      #</span> Example: IP address manipulation</span></span>
            google-auth</span></span>
            hvac</span></span>
          ]</span></span>
        )</span>;</span></span>
</span>
      in</span></span>
      {</span></span>
        formatter</span> =</span> pkgs</span>.</span>nixfmt-tree</span>;</span></span>
</span>
        #</span> 4. The Development Shell</span></span>
        devShells</span>.</span>default</span> =</span> pkgs</span>.</span>mkShell</span> {</span></span>
          buildInputs</span> =</span> [</span></span>
            #</span> The Python environment defined above</span></span>
            pythonWithPackages</span></span>
</span>
            #</span> Other non-Python tools</span></span>
            pkgs</span>.</span>ansible-lint</span></span>
            pkgs</span>.</span>sshpass</span></span>
            pkgs</span>.</span>google-cloud-sdk</span></span>
</span>
            pkgs</span>.</span>vault</span></span>
            pkgs</span>.</span>packer</span></span>
            pkgs</span>.</span>tenv</span></span>
            pkgs</span>.</span>git</span></span>
          ]</span>;</span></span>
</span>
          #</span> Ansible flags</span></span>
          OBJC_DISABLE_INITIALIZE_FORK_SAFETY</span> =</span> "</span>YES</span>"</span>;</span></span>
          OS_ACTIVITY_MODE</span> =</span> "</span>disable</span>"</span>;</span></span>
</span>
          #</span> 5. Shell Hook: Runs when entering the shell</span></span>
          shellHook</span> =</span> ''</span></span>
            export VAULT_ADDR=https://vault.example.com</span></span>
            export TENV_AUTO_INSTALL=true</span></span>
</span>
            echo "🚀 Welcome to the Infra Ops Shell!"</span></span>
            echo "Ansible version: $(ansible --version | head -n1)"</span></span>
            echo "Python location: $(which python)"</span></span>
          ''</span>;</span></span>
        }</span>;</span></span>
      }</span></span>
    )</span>;</span></span>
}</span></span></code></pre>
To use this from anywhere, register the flake in the local Nix registry:</p>
nix</span> registry</span> add</span> infra</span> /path/to/nix-templates/infra-ops</span></span></code></pre>
Running nix develop infra</code> from any directory opens the shell with all configured tools available:</p>
$</span> nix</span> develop</span> infra</span></span>
🚀</span> Welcome</span> to</span> the</span> Infra</span> Ops</span> Shell!</span></span>
Ansible</span> version:</span> ansible</span> [core</span> 2.19.4]</span></span>
Python</span> location:</span> /nix/store/zbcd1vkil51sfl628pbh2b5z59fibv1r-python3-3.13.12/bin/python</span></span>
$</span> which</span> python</span></span>
/nix/store/zbcd1vkil51sfl628pbh2b5z59fibv1r-python3-3.13.12/bin/python</span></span>
$</span> which</span> ansible</span></span>
/nix/store/q2whihir9x88dgrlz80fh3my44vmsxav-python3-3.13.12-env/bin/ansible</span></span></code></pre>
This way I avoid polluting my system with multiple conflicting versions of Ansible, Python, or any other tool.
Any other dependency can be added to flake.nix</code> and will be available immediately on next entry.</p>
So far I'm happy with the setup, though build times are longer than I'd like — something I need to investigate further as I get a better understanding of Nix.
Expect more posts about Nix in the near future!</p>


Hash Tables
2025-05-29T00:00:00+00:00
Go 1.24 switched to Swiss Tables as the underlying
implementation of maps in Go 1</a></sup>.
Despite having used maps in Go since I started using it, I never really thought
about different implementations of maps and the performance implications of
using one or the other.
To get an idea on the history of this core data structure in computer science,
I can recommend reading the earlier linked blog post from the Go team.</p>
In this post, I want to take a step back and implement a simple hash table in Go,
so we can understand the core concepts of hash tables.
This will enable us to dive a bit deeper into the topic in a future post.</p>
What is a Hash Table?</a></h2>
A hash table is a data structure that allows for fast retrieval of values based on keys.
From a users' perspective, it is a key-value store that allows for storing and retrieving
values.
To my knowledge, there is no difference between a hash table and a hash map,
it depends on the language which term is used.
In some languages only one of these data structures is available, or it provides
additional features like thread-safety or ordered keys.
For our tiny implementation there will be no difference between these two terms.
Additionally, it allows for checking if a key exists in the map and deleting keys.
The following is a simple example of a map in Go:</p>
m</span> :=</span> make</span>(</span>map</span>[</span>string</span>]</span>int</span>)</span></span>
</span>
m</span>[</span>"</span>my-key</span>"</span>]</span> =</span> 42</span></span>
m</span>[</span>"</span>your-key</span>"</span>]</span> =</span> 1337</span></span></code></pre>
In our case the interface of our implementation will be a bit more verbose, like
show in the following example.
But it will also have a benefit over the built-in map in Go, as it will
allow for storing any type of value without the need to define a type when
initializing the map.</p>
myMap</span> :=</span> NewHashMap</span>(</span>)</span></span>
</span>
myMap</span>.</span>Insert</span>(</span>"</span>key1</span>"</span>,</span> "</span>value1</span>"</span>)</span></span>
myMap</span>.</span>Insert</span>(</span>"</span>key2</span>"</span>,</span> 1337</span>)</span></span>
</span>
value</span>,</span> exists</span> :=</span> myMap</span>.</span>Get</span>(</span>"</span>key1</span>"</span>)</span></span>
if</span> exists</span> {</span></span>
	println</span>(</span>"</span>key1:</span>"</span>,</span> value</span>.</span>(</span>string</span>)</span>)</span></span>
}</span> else</span> {</span></span>
	println</span>(</span>"</span>key1 does not exist</span>"</span>)</span></span>
}</span></span></code></pre>
We need the following core components:</p>

Buckets</strong> are the primary storage unit of our implementation.</li>
Hash Function</strong> transforms a key into an index value that we can use to access the correct bucket.</li>
Collision Resolution Mechanism</strong> makes sure that we can handle cases where two different keys hash to the same index.</li>
Dynamic Resize</strong> allows us to increase the capacity of the hash map when it is full.</li>
</ol>
1.Buckets</a></h2>
A Bucket is the core storage unit of a hash map.
There is no formal definition of what a bucket is as it can be implemented in various ways.
The most common implementations are arrays or linked lists.
In our case, we will use a linked list to store the entries in each bucket.
Therefore, we don't need a structure Bucket</code>, but rather our chained list of Entry</code> structs
represents the bucket.
Additionally, we will use a HashMap</code> struct to hold the buckets and other metadata
as the current size and the overall capacity of the hash map.
The capacity is a critical part of our implement as it defines how many buckets we have
and how many entries we can store in the hash map before we need to resize it.
But we will talk about resizing later.</p>
const</span> (</span></span>
    baseCapacity</span> =</span> 8</span></span>
)</span></span>
</span>
type</span> Entry</span> struct</span> {</span></span>
    key</span>   string</span></span>
    value</span> any</span></span>
    next</span>  *</span>Entry</span></span>
}</span></span>
</span>
type</span> HashMap</span> struct</span> {</span></span>
    buckets</span>  [</span>]</span>*</span>Entry</span></span>
    size</span>     int</span></span>
    capacity</span> int</span></span>
}</span></span></code></pre>2.Hash Function</a></h2>
To be able to store and retrieve values in our hash map, we need a way to transform
the key into an index value that we can use to access the correct bucket where the value is stored.
Luckily there is a type of function that does exactly that: a hash function.
A hash function takes an arbitrary input (in our case a string key) and maps it
to a fixed-size output.
There are different hash functions available, and depending on the use case,
the choice of the hash function can have a significant impact on the performance
or security of a system.
But in our case we can use a simple hash function that is fast, as we will
call it for essentially any operation on the map implementation.</p>
First we can rule out all hash functions that must compute hash values in
the context of a cryptographic system, as we don't need any level of security.
What is of interest to us is at first the speed of the hash function, and
secondly the distribution of the hash values.
The distribution is important to ensure that we don't have too many collisions.
I can recommend this excellent post on stack overflow about hash functions
in the context of hash tables2</a></sup>.
Another even more detailed source is, which compares different hash functions
and their performance3</a></sup>.
Based on this we can consider two hash functions for our implementation:</p>

FNV-1a</li>
MurmurHash3</li>
</ol>
I decided to go with the FNV-1a as it is available in the
Go standard library</a> and therefore no external dependency
is required.
It implements the Hash.Hash</a> interface and
is pretty straightforward to use:</p>
func</span> hashFunction</span>(</span>key</span> string</span>)</span> uint32</span> {</span></span>
	h</span> :=</span> fnv</span>.</span>New32a</span>(</span>)</span></span>
	h</span>.</span>Write</span>(</span>[</span>]</span>byte</span>(</span>key</span>)</span>)</span></span>
	return</span> h</span>.</span>Sum32</span>(</span>)</span></span>
}</span></span>
</span>
Result</span> for</span> "</span>my-key</span>"</span> is</span>:</span> 3018227921</span></span></code></pre>
If we feed the string "my-key" into the hash function, we get the value 3018227921</code>.
With this covered we now need to map this value to an index in our buckets.
A simple way to do this is to use the modulo operator with the current capacity of the hash map.
We will come back to this later when we think about resizing the hash map, but for
now this is sufficient.</p>
func</span> (</span>hm </span>*</span>HashMap</span>)</span> getBucketIndex</span>(</span>key</span> string</span>)</span> int</span> {</span></span>
	hashValue</span> :=</span> hashFunction</span>(</span>key</span>)</span></span>
	return</span> int</span>(</span>hashValue</span> %</span> uint32</span>(</span>hm</span>.</span>capacity</span>)</span>)</span></span>
}</span></span></code></pre>
We then can use this function for inserting and retrieving values from the hash map.</p>
func</span> (</span>hm </span>*</span>HashMap</span>)</span> Insert</span>(</span>key</span> string</span>,</span> value</span> any</span>)</span> {</span></span>
	hm</span>.</span>mutex</span>.</span>Lock</span>(</span>)</span></span>
	defer</span> hm</span>.</span>mutex</span>.</span>Unlock</span>(</span>)</span></span>
</span>
	hashResult</span> :=</span> hashFunction</span>(</span>key</span>)</span></span>
	log</span>.</span>Println</span>(</span>"</span>Hash result for key:</span>"</span>,</span> key</span>,</span> "</span>is</span>"</span>,</span> hashResult</span>)</span></span>
	index</span> :=</span> int</span>(</span>hashFunction</span>(</span>key</span>)</span>)</span> %</span> hm</span>.</span>capacity</span></span>
	log</span>.</span>Println</span>(</span>"</span>Index for key:</span>"</span>,</span> key</span>,</span> "</span>is</span>"</span>,</span> index</span>)</span></span>
</span>
	log</span>.</span>Println</span>(</span>"</span>Putting key:</span>"</span>,</span> key</span>,</span> "</span>with value:</span>"</span>,</span> value</span>,</span> "</span>at index:</span>"</span>,</span> index</span>)</span></span>
	head</span> :=</span> hm</span>.</span>buckets</span>[</span>index</span>]</span></span>
</span>
	//</span> Insert new entry</span></span>
	newEntry</span> :=</span> &</span>Entry</span>{</span></span>
		key</span>:</span>   key</span>,</span></span>
		value</span>:</span> value</span>,</span></span>
		next</span>:</span>  head</span>,</span></span>
	}</span></span>
</span>
	hm</span>.</span>buckets</span>[</span>index</span>]</span> =</span> newEntry</span></span>
	hm</span>.</span>size</span>++</span></span>
}</span></span></code></pre>3.Collision Resolution</a></h2>
A collision occurs when two different keys hash to the same index</strong> in the hash map.
There are multiple ways to handle collisions, but the most common ones are:</p>

Chaining: Store multiple entries in the same bucket by using a linked list in each entry.</li>
Open Addressing: Find another empty bucket for the new entry.</li>
</ol>
In our case, we will use the chaining method, as it is simple to implement.
It has the downside that the performance can degrade if many entries hash to the same bucket,
and lookups will take longer.</p>
This means two changes in our current implementation on the Insert</code> and Get</code> methods.
In the Insert</code> method we need to check if the key already exists in the bucket
and update the value if it does.
The magic part happens when we insert a new entry into the bucket.
We will set its next pointer to the current head of the bucket.
By this the new entry will be the first entry in the linked list,
and the previous entries will be accessible via the next pointers.</p>
func</span> (</span>hm </span>*</span>HashMap</span>)</span> Insert</span>(</span>key</span> string</span>,</span> value</span> any</span>)</span> {</span></span>
	...</span></span>
</span>
	for</span> e</span> :=</span> head</span>;</span> e</span> !=</span> nil</span>;</span> e</span> =</span> e</span>.</span>next</span> {</span></span>
		log</span>.</span>Println</span>(</span>"</span>Checking entry:</span>"</span>,</span> e</span>.</span>key</span>)</span></span>
		if</span> e</span>.</span>key</span> ==</span> key</span> {</span></span>
			e</span>.</span>value</span> =</span> value</span></span>
			return</span></span>
		}</span></span>
	}</span></span>
</span>
	newEntry</span> :=</span> &</span>Entry</span>{</span></span>
		key</span>:</span>   key</span>,</span></span>
		value</span>:</span> value</span>,</span></span>
		next</span>:</span>  head</span>,</span></span>
	}</span></span>
</span>
	hm</span>.</span>buckets</span>[</span>index</span>]</span> =</span> newEntry</span></span>
	hm</span>.</span>size</span>++</span></span>
}</span></span></code></pre>
In the Get</code> method we need to iterate over the linked list in the bucket
to find the entry with the given key.</p>
func</span> (</span>hm </span>*</span>HashMap</span>)</span> Get</span>(</span>key</span> string</span>)</span> (</span>any</span>,</span> bool</span>)</span> {</span></span>
	hm</span>.</span>mutex</span>.</span>RLock</span>(</span>)</span></span>
	defer</span> hm</span>.</span>mutex</span>.</span>RUnlock</span>(</span>)</span></span>
</span>
	hashResult</span> :=</span> hashFunction</span>(</span>key</span>)</span></span>
	log</span>.</span>Println</span>(</span>"</span>Hash result for key:</span>"</span>,</span> key</span>,</span> "</span>is</span>"</span>,</span> hashResult</span>)</span></span>
	index</span> :=</span> int</span>(</span>hashFunction</span>(</span>key</span>)</span>)</span> %</span> hm</span>.</span>capacity</span></span>
	log</span>.</span>Println</span>(</span>"</span>Getting key:</span>"</span>,</span> key</span>,</span> "</span>at index:</span>"</span>,</span> index</span>)</span></span>
	head</span> :=</span> hm</span>.</span>buckets</span>[</span>index</span>]</span></span>
</span>
	for</span> e</span> :=</span> head</span>;</span> e</span> !=</span> nil</span>;</span> e</span> =</span> e</span>.</span>next</span> {</span></span>
		log</span>.</span>Println</span>(</span>"</span>Checking entry:</span>"</span>,</span> e</span>.</span>key</span>)</span></span>
		if</span> e</span>.</span>key</span> ==</span> key</span> {</span></span>
			return</span> e</span>.</span>value</span>,</span> true</span></span>
		}</span></span>
	}</span></span>
</span>
	return</span> nil</span>,</span> false</span></span>
}</span></span></code></pre>4.Resizing the Map</a></h2>
If the current capacity of our map implementation is reached, and we want to insert a new entry,
we need to resize the hash map.
Based on our earlier decision I'll opt for a simple resizing strategy:</p>

Double the current capacity of the map.</li>
Create a new array of buckets with the new capacity.</li>
Rehash all existing entries and insert them into the new buckets.</li>
Delete the old map.</li>
</ol>
There are more sophisticated resizing strategies that don't require rehashing all entries,
but let's keep it simple for now and let some performance improvements for another post.</p>
func</span> (</span>hm </span>*</span>HashMap</span>)</span> resize</span>(</span>)</span> {</span></span>
	oldBuckets</span> :=</span> hm</span>.</span>buckets</span></span>
	//</span> double the capacity</span></span>
	hm</span>.</span>capacity</span> *=</span> 2</span></span>
	hm</span>.</span>buckets</span> =</span> make</span>(</span>[</span>]</span>*</span>Entry</span>,</span> hm</span>.</span>capacity</span>)</span></span>
	hm</span>.</span>size</span> =</span> 0</span></span>
</span>
	for</span> _</span>,</span> entry</span> :=</span> range</span> oldBuckets</span> {</span></span>
		for</span> e</span> :=</span> entry</span>;</span> e</span> !=</span> nil</span>;</span> e</span> =</span> e</span>.</span>next</span> {</span></span>
			log</span>.</span>Println</span>(</span>"</span>Rehashing key:</span>"</span>,</span> e</span>.</span>key</span>,</span> "</span>with value:</span>"</span>,</span> e</span>.</span>value</span>)</span></span>
			hm</span>.</span>Insert</span>(</span>e</span>.</span>key</span>,</span> e</span>.</span>value</span>)</span></span>
		}</span></span>
	}</span></span>
}</span></span></code></pre>
Determining when to resize the hash map is pretty straightforward.
During insert operations we can check if the current size of the hash map
has reached the current capacity and if so, we can call the resize</code> method.</p>
func</span> (</span>hm </span>*</span>HashMap</span>)</span> Insert</span>(</span>key</span> string</span>,</span> value</span> any</span>)</span> {</span></span>
	if</span> hm</span>.</span>size</span> ==</span> hm</span>.</span>capacity</span> {</span></span>
		log</span>.</span>Println</span>(</span>"</span>HashMap is full, resizing...</span>"</span>)</span></span>
		hm</span>.</span>resize</span>(</span>)</span></span>
	}</span></span>
</span>
	...</span></span>
}</span></span></code></pre>Thread Safety</a></h2>
As of now our implementation is not thread-safe, meaning that if multiple go routines
access the hash map concurrently, it can lead to undefined behavior.
But we can easily make it thread-safe by using a mutex to lock the hash map
during read and write operations.
For this we are going to adjust the HashMap</code> struct to include a mutex:</p>
type</span> HashMap</span> struct</span> {</span></span>
	buckets</span>  [</span>]</span>*</span>Entry</span></span>
	size</span>     int</span></span>
	capacity</span> int</span></span>
	mutex</span>    sync</span>.</span>RWMutex</span></span>
}</span></span></code></pre>
Then we have to lock and unlock the mutex in all methods that modify or read the
map values.
For read only operations we can use a read lock, which allows multiple
routines to read the map concurrently.
For write operations we use a write lock, which ensures that only one routine
can modify the map at a time.</p>
//</span> read operation</span></span>
hm</span>.</span>mutex</span>.</span>RLock</span>(</span>)</span></span>
defer</span> hm</span>.</span>mutex</span>.</span>RUnlock</span>(</span>)</span></span>
</span>
//</span> write operation</span></span>
hm</span>.</span>mutex</span>.</span>Lock</span>(</span>)</span></span>
defer</span> hm</span>.</span>mutex</span>.</span>Unlock</span>(</span>)</span></span></code></pre>Conclusion</a></h2>
In comparison to the new Go implementation of hash tables, our implementation
is for sure not as performant, but it is a good example to understand the
core concepts of hash tables.
Additionally, we covered different hash functions and how to handle
collisions.
Something we haven't covered or handled at all is hash collisions on keys we want
to insert.
But this will be a topic of a future post.
The examples are inspired by a blog post I read a while ago4</a></sup>, and as always
the full code is available on GitHub</a>.</p>



https://go.dev/blog/swisstable ↩</a></p>
</li>

https://softwareengineering.stackexchange.com/questions/49550/which-hashing-algorithm-is-best-for-uniqueness-and-speed ↩</a></p>
</li>

https://github.com/rurban/smhasher ↩</a></p>
</li>

https://medium.com/@alameerashraf/from-amazon-interview-jitters-to-mastering-hash-maps-building-a-hashmap-in-go-from-scratch-022e53d47636 ↩</a></p>
</li>
</ol>
</section>


Brewfile
2025-04-07T00:00:00+00:00
If you are switching from Linux to a Mac you might be confused that there is no default
package manager on macOS.
The de facto default is homebrew</a>, at least that is what I
and most people I know are using (there still exists some alternatives
like macports</a> to name one).
You can use brew</code> more or less like any package manager on Linux like apt</code> or
pacman</code>.
It supports installing programs via brew install jj</code> and removing them again with
brew uninstall jj</code>.</p>
But I stumbled upon the interesting concept of managing all external packages
in a single file called Brewfile</code>, by reading the excellent post from
Matthias</a>.
Please head over to his blog, because he documented the approach perfectly.
To me this sounds like a perfect way to have most external programs and dependencies
stored at a single place and to actually know which external dependencies
I got on my system.</p>
All the commands to manage your single file with all external programs and
dependencies in it are available via brew bundle</code>.</p>
I see the following advantages:</p>

I can install and update all my programs with a single command.</li>
I can try out programs without adding them permanently to my system.
Even if I forget to remove them, the next update/install will remove them again.</li>
I can easily distribute my favorite apps via my dotfiles between machines.</li>
</ol>
My setup is mostly matching what Matthias wrote on his blog post.
I have a single command called buu</code>, which is my legacy command to call
brew update && brew upgrade</code>, which now calls brew bundle</code>.
This is my fish</a> alias for it:</p>
alias</span> buu=</span>"</span>brew update &&</span>\</span></span>
    brew bundle install --cleanup --file=~/.config/Brewfile &&</span>\</span></span>
    brew upgrade</span>"</span></span></code></pre>
The --cleanup</code> flag will remove all other packages that were installed via brew,
which are not in the Brewfile.</p>
An example for a very simple Brewfile:</p>
brew "jq"</span></span>
</span>
# basic unix tools</span></span>
brew "fish"</span></span>
brew "stow"</span></span>
</span>
# devtools</span></span>
cask "visual-studio-code"</span></span>
brew "neovim"</span></span>
cask "ghostty"</span></span>
cask "eurkey"</span></span>
brew "fzf"</span></span>
brew "go"</span></span>
brew "jj"</span></span>
</span>
# macos tools</span></span>
cask "stats"</span></span>
cask "alt-tab"</span></span>
</span>
# apps</span></span>
cask "obsidian"</span></span>
cask "firefox"</span></span>
cask "thunderbird"</span></span>
brew "syncthing"</span></span></code></pre>
As you can see it is still possible to install both, casks</code> and normal non GUI
tools with the Brewfile.
Furthermore, it is also possible to use taps</code> by just declaring the tap</code> in the
Brewfile.</p>
So far this setup is very convenient to use, and I like the more declarative
approach to manage the software on my Macs.</p>


Replacing tmux with Ghostty
2025-01-18T00:00:00+00:00
Tmux</a> is a tool that I have been using on a
daily basis for the last 7+ years.
It is a terminal multiplexer that allows me to have more than one terminal
session in a single terminal window.
This is very much in alignment with my workflow, as I like to have a terminal
window open for each project that I'm working on.
It also allows me to disconnect from a session and reconnect later.
Since its discovery, I have never used a shell without tmux.
This is what a terminal split looks like in Ghostty:</p>
</p>
But let's start at the beginning.
Late last year, I discovered Ghostty</a>.
It was publicly released in December 2024, but I somehow managed to get access
to the closed beta in November.
I wasn't really searching for a new terminal emulator, as I was happy with
Kitty</a>, but nevertheless I gave it a try anyway.
The first thing that I noticed was the snappy performance and the very clean
configuration file.
I was aware of Kitty's support for tabs and multiplexing, but I had never used them.
While I was changing my terminal emulator, I thought about trying to replace tmux with it as well.</p>
In the following you can see my whole configuration file:</p>
shell-integration = zsh</span></span>
font-family = JetBrains Mono</span></span>
theme = Dracula</span></span>
mouse-hide-while-typing = true</span></span>
scrollback-limit = 1000000</span></span>
</span>
keybind = ctrl+n=new_window</span></span>
</span>
keybind = ctrl+h=goto_split:left</span></span>
keybind = ctrl+j=goto_split:bottom</span></span>
keybind = ctrl+k=goto_split:top</span></span>
keybind = ctrl+l=goto_split:right</span></span>
</span>
keybind = ctrl+a>h=new_split:left</span></span>
keybind = ctrl+a>j=new_split:down</span></span>
keybind = ctrl+a>k=new_split:up</span></span>
keybind = ctrl+a>l=new_split:right</span></span>
keybind = ctrl+a>f=toggle_split_zoom</span></span>
</span>
keybind = ctrl+a>n=next_tab</span></span>
keybind = ctrl+a>p=previous_tab</span></span>
</span>
keybind = super+r=reload_config</span></span>
</span>
window-save-state = always</span></span></code></pre>
I tried to fully replicate all the shortcuts that I'm used to from using tmux.
So I have the same keybindings for creating new splits and navigating between them.
My modifier key is ctrl+a</code> which was my default in tmux.
I also like Ghostty's window state saving feature, something I never tried in tmux.</p>
Overall, I'm very happy with my new setup and look forward to seeing what Ghostty will bring in the future.
I'm also thinking about updating my most used tools and seeing if there are any modern alternatives I could use instead.</p>


Simple Container Deployments on Virtual Machines
2024-12-01T00:00:00+00:00
There were a few posts about how people host their own services on hackernews and
lobsters recently.
Some of them used Kubernetes, some plain Docker and others are into Nix (which
I will probably check out too soon).
This post is about how I manage my services on virtual machines using Docker
and a few other tools.
More specifically, I run most of my services on virtual machines I rent from a provider
and some at my home on a rather old server.
Despite that I have to manage the virtual machines myself, I still use
containers to deploy the actual services on them.
This post will cover how I deploy and manage these containers on virtual machines.</p>
You might ask yourself: "Why not use Kubernetes?".
Well the answer is simple: I don't need any of its feature and I most importantly,
don't want to handle to operational and financial overhead of managing
a Kubernetes cluster.</p>
Overview of Components on a Single Machine</a></h2>
In the following I will describe the overall idea of my setup.
The VM obviously runs a Linux distribution, on my case Debian on the cloud
or Arch at my home.</p>
</p>
As can be seen in the image above, it consists of four major parts.</p>

A docker compose file for each service.</li>
Caddy as reverse proxy to handle TLS termination.</li>
upstream-watch to check for updates in the service definitions.</li>
watchtower to update the images of individual services.</li>
</ol>
Service Definitions</a></h2>
All services are containerized and therefore run in containers.
I use a single git repository to manage all services of a single VM.
Therefore, each service has its own folder in the repository.
The following is an example directory structure:</p>
    .</span></span>
    ├──</span> service-1</span></span>
    │</span>   ├──</span> docker-compose.yml</span></span>
    │</span>   └──</span> README.md</span></span>
    ├──</span> service-2</span></span>
    │</span>   ├──</span> docker-compose.yml</span></span>
    │</span>   └──</span> README.md</span></span>
    └──</span> README.md</span></span></code></pre>
There are two services, each in its own subfolder.
Each of these services holds a README.md</code> (which is not interesting) and
a docker-compose.yml</code> that defines and configures the containers.</p>
An example service could look like this:</p>
s</span>ervices</span>:</span></span>
    s</span>ervice-1</span>:</span></span>
    i</span>mage</span>:</span> m</span>y.registry.dev/service-1:production</span></span>
        c</span>ontainer_name</span>:</span> a</span>ndre-blog</span></span>
        r</span>estart</span>:</span> a</span>lways</span></span>
        l</span>abels</span>:</span></span>
          -</span> "</span>com.centurylinklabs.watchtower.enable=true</span>"</span></span>
        p</span>orts</span>:</span></span>
            -</span> 1</span>27.0.0.1:1337:1337</span></span></code></pre>
Note that the com.centurylinklabs.watchtower.enable</code> label is used to enable
automated updates of the service via watchtower, which will be covered later.
Additionally, the service is only accessible from the local machine on port 1337.
Again, this will be covered later.</p>
Reverse Proxy</a></h2>
As can be seen in the overview image, I use Caddy as a reverse proxy.
I prefer Caddy to nginx or Apache because it very easy to configure and
handles automatic HTTPS via Let's Encrypt out of the box.
All services are therefore only accessible via HTTPS.</p>
Caddy is configured via a Caddyfile</code> that looks like this:</p>
    service1.my-domain.dev</span> {</span></span>
        reverse_proxy</span> 127.0.0.1:1337</span></span>
    }</span></span>
    service2.my-domain.dev</span> {</span></span>
        reverse_proxy</span> 127.0.0.1:1338</span></span>
    }</span></span></code></pre>
In my setup, Caddy is directly installed on the host system and not running in a
container.
It is managed via a systemd service and the configuration file is located at
/etc/caddy/Caddyfile</code>.
Both the systemd service and the Caddyfile are managed via Ansible.
It should be mentioned that Caddy could also run as a container, but that
requires to use a shared network between the Caddy container and the services,
which I wanted to avoid keeping the setup a bit simpler.</p>
Upstream Watch</a></h2>
Upstream watch is a small tool I wrote to check for changes in an upstream
git repository and pull them if there are any.1</a></sup>
With this tool I can update the service definitions in the git repository and
upstream-watch</code> will automatically pull the changes and restart the services
if necessary.
To use it you must provide two main configuration files:</p>
    .</span></span>
    ├──</span> .upstream-watch.yaml</span></span>
    ├──</span> README.md</span></span>
    ├──</span> service-1</span></span>
    │</span>   ├──</span> .update-hooks.yaml</span></span>
    │</span>   ├──</span> docker-compose.yml</span></span>
    │</span>   └──</span> README.md</span></span>
    ├──</span> service-2</span></span>
    │</span>   ├──</span> .update-hooks.yaml</span></span>
    │</span>   ├──</span> docker-compose.yml</span></span>
    │</span>   └──</span> README.md</span></span>
    └──</span> upstream-watch</span></span></code></pre>
The .upstream-watch.yaml</code> is the main configuration file.
You can set the retry interval (in seconds) and folders that should be ignored and
not trigger the later configured hooks.</p>
    single_directory_mode:</span> false</span></span>
    retry_interval:</span> 10</span></span>
    ignore_folders:</span> [</span>"</span>.git</span>"</span>,</span> "</span>.test</span>"</span>]</span></span></code></pre>
We need to adjust the earlier presented directory structure a bit.
The main configuration file per service is .update-hooks.yaml</code>, which is the
configuration file of upstream-watch</code> for this specific service.</p>
In case of an update to any of these files in a subfolder, upstream-watch</code>
will execute the pre- and post-hooks defined in the corresponding .update-hooks.yaml</code>.</p>
An example for a .update-hooks.yaml</code>:</p>
	pre_update_commands:</span> [</span>"</span>docker compose down</span>"</span>]</span></span>
	update_commands:</span> [</span>"</span>docker compose pull</span>"</span>]</span></span>
	post_update_commands:</span> [</span>"</span>docker compose up -d</span>"</span>]</span></span></code></pre>
upstream-watch</code> will stop all containers, pull updates from the registry and
start them afterwards.
Of course, you can do almost anything in these hooks, depending on the needs of
your service.</p>
Or the other way around to reduce the downtime:</p>
    pre_update_commands:</span> [</span>"</span>docker compose pull</span>"</span>]</span></span>
    update_commands:</span> [</span>"</span>docker compose down</span>"</span>]</span></span>
    post_update_commands:</span> [</span>"</span>docker compose up -d</span>"</span>]</span></span></code></pre>
Currently, upstream watch must be run manually, but I plan to containerize it and
remove the need to run it manually.</p>
Watchtower</a></h2>
Watchtower handles updating the running containers, if there is a new version with
the same tag available.2</a></sup>
I decided to configure the services that should be updated via the label
"com.centurylinklabs.watchtower.enable=true"</code> in the
docker-compose.yml</code> files, allowing me to enable or disable automatic updates
per service.</p>
The full configuration of watchtower looks like this:</p>
s</span>ervices</span>:</span></span>
    w</span>atchtower</span>:</span></span>
      i</span>mage</span>:</span> c</span>ontainrrr/watchtower</span></span>
      c</span>ommand</span>:</span> --</span>label-enable --interval 30</span></span>
      v</span>olumes</span>:</span></span>
        -</span> /</span>var/run/docker.sock:/var/run/docker.sock</span></span>
        -</span> ~</span>/.docker/config.json:/config.json</span></span></code></pre>
In this case it will check for updates every 30 seconds and update the services
with the label com.centurylinklabs.watchtower.enable=true</code>.
Additionally, it mounts the docker socket and the docker configuration file to
be able to interact with the container runtime.</p>
Backups</a></h2>
With this setup you automatically get backups of your service definition
and configuration.
But you still need to backup the data of each of the services.
In my case that is done via a separate backup script that is run via a systemd
timer.
I've written a post about the backup setup here</a>.</p>
Conclusion</a></h2>
In my opinion this setup is a good compromise between the flexibility and
isolation of using containers and the ease of use of a single virtual machine
running Linux.
The setup is independent of any cloud provider or it's APIs and provides
close to bare metal performance.
On the other hand it provides no automatic scaling or failover, and you need to
learn multiple tools like Ansible and Docker (compose) to manage the setup.
So if you are in need for a simple setup that is easy to manage, and you don't
need any of the features of Kubernetes, this setup might be for you.</p>



https://github.com/andresterba/upstream-watch ↩</a></p>
</li>

https://containrrr.dev/watchtower/ ↩</a></p>
</li>
</ol>
</section>


Container Fundamentals
2024-11-06T00:00:00+00:00
Containers allow you to package applications and all their dependencies in a common
format and ship it to your runtime environment.
But what are the fundamental technologies containers are build on?
And why do we need a Linux Kernel for that?</p>
The following post is a written summary of a lecture I gave as an introduction
to Kubernetes.
The code is based on the excellent talk and code of Liz Rice</a>
some years ago.</p>
What is a container?</a></h2>
If you think about a container you might think about a tiny operating system
that runs your application.
But a container is no virtual machine or micro virtual machine.
A container is an isolated process that runs on a Linux system.
It is isolated in the sense that it has its own file system, network, and process space,
depending on the configuration.
The question is why does this work only on Linux (Kernel)?
And yes I know there are other container technologies like FreeBSD jails, but
let's focus on the main driver for adoption, the Linux Kernel.</p>
Containers provide us the following benefits:</p>

Stable runtime environment for applications</li>
Pack all dependencies together with the application</li>
Isolation from other applications. With proper configuration we don't run into
the noisy neighbor problem.1</a></sup></li>
</ol>
If we start an Ubuntu container in interactive mode (docker run -it ubuntu /bin/bash</code>)
and run ps</code> in it, we should only see two processes.
The first one will be the shell we started the container with and the second one
will be our ps</code> command.</p>
</p>
In the meantime we also see our running container as a process on the host system!</p>
</p>
And with that let's have a look on the Kernel features that enables us to isolate
a process to a degree where it thinks it is the one and only process of a system.</p>
Kernel Features</a></h2>
The Linux Kernel has a few features that make containers possible.
Namely these are:</p>

Namespaces 2</a></sup></li>
Chroot 3</a></sup></li>
Control Groups (cgroups) 4</a></sup></li>
</ol>
They will be explained a bit more in detail in the following section.
All of them are documented in their corresponding man pages, so you can read more
about them there.</p>
Namespaces</a></h3>
Namespaces are a feature of the Linux Kernel that allows you to isolate resources
like network, process, and file system.
They limit what a container (or process) can see and access.
For example, a container can have its own network namespace, so it can have its
own IP address and network devices and its own file system with a different Linux
distribution than the host. 2</a></sup></p>
Chroot</a></h3>
Change root (chroot) is a system call that changes the root directory for the
current process and its children.
This might sound like a not very important feature, but it is crucial for containers.
It enables us to tell a process that its root directory is a different directory
than the host's root directory.
And with that we can create a process that is running with the default file system
of a different Linux distribution. 3</a></sup></p>
Control Groups (cgroups)</a></h3>
Control Groups (cgroups) are a feature of the Linux Kernel that allows you to limit
the resources a process can use.
This is especially useful to tackle the earlier mentioned noisy neighbor problem.
With cgroups you can limit the CPU, memory, and I/O usage of a process, so it doesn't
starve other processes on the same system. 4</a></sup></p>
What is Docker then?</a></h2>
Docker is a tool (and there exist multiple!) that makes it easy to build,
run, and manage containers.
It uses all the above-mentioned capabilities of the Linux Kernel to isolate
processes and wraps them in a nice and easy to use interface.
In the following we will write some code to imitate the docker run</code> command.
We want to start a "container" that thinks it is the only process on the system
and despite running it on an Ubuntu machine, it should think it is running
the Debian distribution.</p>
Writing our own Container CLI Tool</a></h2>
With the basic knowledge about the capabilities the Kernel provides let's try
to implement some of it.</p>
Attention:</strong> The following code only runs a Linux system as we rely on the earlier
mentioned Kernel features for process isolation.</p>
func</span> run</span>(</span>)</span> {</span></span>
	fmt</span>.</span>Printf</span>(</span>"</span>Running </span>%v</span> as process </span>%d</span>\n</span>"</span>,</span> os</span>.</span>Args</span>[</span>2</span>:</span>]</span>,</span> os</span>.</span>Getpid</span>(</span>)</span>)</span></span>
</span>
	cmd</span> :=</span> exec</span>.</span>Command</span>(</span>os</span>.</span>Args</span>[</span>2</span>]</span>,</span> os</span>.</span>Args</span>[</span>3</span>:</span>]</span>...</span>)</span></span>
	cmd</span>.</span>SysProcAttr</span> =</span> &</span>syscall</span>.</span>SysProcAttr</span>{</span></span>
		Cloneflags</span>:</span>   syscall</span>.</span>CLONE_NEWUTS</span> |</span> syscall</span>.</span>CLONE_NEWPID</span> |</span> syscall</span>.</span>CLONE_NEWNS</span>,</span></span>
		Unshareflags</span>:</span> syscall</span>.</span>CLONE_NEWNS</span>,</span></span>
	}</span></span>
</span>
	err</span> :=</span> cmd</span>.</span>Run</span>(</span>)</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        panic</span>(</span>err</span>)</span></span>
    }</span></span>
}</span></span></code></pre>
Before running the command we have to add some attributes.
These attributes are exactly the features we talked about earlier.
They tell the Kernel how to spawn this process and how to isolate it.</p>
NEWUTS</code> means Unix Timesharing System</code> we need it to get an own hostname
for our isolated process.
After settings this attribute we are able to change the hostname inside the process
without interfering with the host system.
NewPID</code> gives the process a new PID starting from 0.
If the process forks a new child the child would get the PID 1 and so on.</p>
So with this we have a somewhat isolated process.</p>
</p>
But wait!
This process still sees all the other processes on the system and isn't running as
PID 0 or 1.
For this we need to fork our own process to get our process running with PID 1.
Luckily the Kernel or more specific the virtual file system /proc</code> offers a
convenient way to do this.
We can just call /proc/self/exec</code> to fork ourselves and apply all the discussed
isolation features.</p>
func</span> run</span>(</span>)</span> {</span></span>
	fmt</span>.</span>Printf</span>(</span>"</span>Running </span>%v</span> as process </span>%d</span>\n</span>"</span>,</span> os</span>.</span>Args</span>[</span>2</span>:</span>]</span>,</span> os</span>.</span>Getpid</span>(</span>)</span>)</span></span>
</span>
	cmd</span> :=</span> exec</span>.</span>Command</span>(</span>"</span>/proc/self/exe</span>"</span>,</span> append</span>(</span>[</span>]</span>string</span>{</span>"</span>child</span>"</span>}</span>,</span> os</span>.</span>Args</span>[</span>2</span>:</span>]</span>...</span>)</span>...</span>)</span></span>
	cmd</span>.</span>SysProcAttr</span> =</span> &</span>syscall</span>.</span>SysProcAttr</span>{</span></span>
		Cloneflags</span>:</span>   syscall</span>.</span>CLONE_NEWUTS</span> |</span> syscall</span>.</span>CLONE_NEWPID</span> |</span> syscall</span>.</span>CLONE_NEWNS</span>,</span></span>
		Unshareflags</span>:</span> syscall</span>.</span>CLONE_NEWNS</span>,</span></span>
	}</span></span>
</span>
	err</span> :=</span> cmd</span>.</span>Run</span>(</span>)</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        panic</span>(</span>err</span>)</span></span>
    }</span></span>
}</span></span>
</span>
func</span> child</span>(</span>)</span> {</span></span>
	fmt</span>.</span>Printf</span>(</span>"</span>Running </span>%v</span> as process </span>%d</span>\n</span>"</span>,</span> os</span>.</span>Args</span>[</span>2</span>:</span>]</span>,</span> os</span>.</span>Getpid</span>(</span>)</span>)</span></span>
</span>
	syscall</span>.</span>Sethostname</span>(</span>[</span>]</span>byte</span>(</span>"</span>container</span>"</span>)</span>)</span></span>
</span>
	cmd</span> :=</span> exec</span>.</span>Command</span>(</span>os</span>.</span>Args</span>[</span>2</span>]</span>,</span> os</span>.</span>Args</span>[</span>3</span>:</span>]</span>...</span>)</span></span>
	cmd</span>.</span>Stdin</span> =</span> os</span>.</span>Stdin</span></span>
	cmd</span>.</span>Stdout</span> =</span> os</span>.</span>Stdout</span></span>
	cmd</span>.</span>Stderr</span> =</span> os</span>.</span>Stderr</span></span>
</span>
	err</span> :=</span> cmd</span>.</span>Run</span>(</span>)</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        panic</span>(</span>err</span>)</span></span>
    }</span></span>
}</span></span></code></pre>
We fork ourselves and additionally use the Sethostname</code> syscall to directly
change the hostname of our process.
The result looks like this:</p>
</p>
Please note that we now see the Running [/bin/bash] as ...</code> two times.
The reason for this is the fork of the parent process to be able to call
syscall.Sethostname([]byte("container"))</code> for our child process.</p>
Unexpectedly, this process still sees all the other processes running on the system.
The reason for this is the /proc</code> virtual file system.
It exposes all the runtime metrics the Kernel knows about the system, and
we just mounted the host file system into our isolated process.
But we already have a way to fix this.
Instead of only setting the hostname of our container we can additionally get
our own /proc</code> file system.</p>
func</span> child</span>(</span>)</span> {</span></span>
	fmt</span>.</span>Printf</span>(</span>"</span>Running </span>%v</span> as process </span>%d</span>\n</span>"</span>,</span> os</span>.</span>Args</span>[</span>2</span>:</span>]</span>,</span> os</span>.</span>Getpid</span>(</span>)</span>)</span></span>
</span>
	syscall</span>.</span>Sethostname</span>(</span>[</span>]</span>byte</span>(</span>"</span>container</span>"</span>)</span>)</span></span>
	syscall</span>.</span>Mount</span>(</span>"</span>proc</span>"</span>,</span> "</span>proc</span>"</span>,</span> "</span>proc</span>"</span>,</span> 0</span>,</span> "</span>"</span>)</span></span>
</span>
	cmd</span> :=</span> exec</span>.</span>Command</span>(</span>os</span>.</span>Args</span>[</span>2</span>]</span>,</span> os</span>.</span>Args</span>[</span>3</span>:</span>]</span>...</span>)</span></span>
	cmd</span>.</span>Stdin</span> =</span> os</span>.</span>Stdin</span></span>
	cmd</span>.</span>Stdout</span> =</span> os</span>.</span>Stdout</span></span>
	cmd</span>.</span>Stderr</span> =</span> os</span>.</span>Stderr</span></span>
</span>
	err</span> :=</span> cmd</span>.</span>Run</span>(</span>)</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        panic</span>(</span>err</span>)</span></span>
    }</span></span>
}</span></span></code></pre>
Finally, we want to provide our process its own file system.
And once we are already on it let's use the official docker image of Debian
to transform our process into a process running on a Debian Linux.
The image of a running container can easily extracted like this:</p>
    docker</span> run</span> -</span>it</span> debian</span> sh</span></span>
</span>
    #</span> install ps and other tools</span></span>
    apt</span> update</span> &&</span> apt</span> install</span> procps</span></span>
</span>
    docker</span> export</span> container_name</span> ></span> output.tar</span></span>
    mkdir</span> debianfs</span> &&</span> cd</span> debianfs</span></span>
    mv</span> debian.tar</span> debianfs</span></span>
    tar</span> xvf</span> debian.tar</span></span></code></pre>
And with that we have the full file system with all default configurations
of a Debian Linux.
Now we just have to attach it to our process using chroot</code>.</p>
func</span> child</span>(</span>)</span> {</span></span>
	fmt</span>.</span>Printf</span>(</span>"</span>Running </span>%v</span> as process </span>%d</span>\n</span>"</span>,</span> os</span>.</span>Args</span>[</span>2</span>:</span>]</span>,</span> os</span>.</span>Getpid</span>(</span>)</span>)</span></span>
</span>
	syscall</span>.</span>Sethostname</span>(</span>[</span>]</span>byte</span>(</span>"</span>container</span>"</span>)</span>)</span></span>
	syscall</span>.</span>Chroot</span>(</span>"</span>/home/aps/share/gocker/debianfs</span>"</span>)</span></span>
	syscall</span>.</span>Chdir</span>(</span>"</span>/</span>"</span>)</span></span>
	syscall</span>.</span>Mount</span>(</span>"</span>proc</span>"</span>,</span> "</span>proc</span>"</span>,</span> "</span>proc</span>"</span>,</span> 0</span>,</span> "</span>"</span>)</span></span>
</span>
	cmd</span> :=</span> exec</span>.</span>Command</span>(</span>os</span>.</span>Args</span>[</span>2</span>]</span>,</span> os</span>.</span>Args</span>[</span>3</span>:</span>]</span>...</span>)</span></span>
	cmd</span>.</span>Stdin</span> =</span> os</span>.</span>Stdin</span></span>
	cmd</span>.</span>Stdout</span> =</span> os</span>.</span>Stdout</span></span>
	cmd</span>.</span>Stderr</span> =</span> os</span>.</span>Stderr</span></span>
</span>
	err</span> :=</span> cmd</span>.</span>Run</span>(</span>)</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        panic</span>(</span>err</span>)</span></span>
    }</span></span>
}</span></span></code></pre>
This is the output of ps</code> and the os-release</code> of our process after applying all
the changes:</p>
</p>
We know have a process that is isolated from the host system.
The process doesn't see any processes from the host and even has its own
file system based on Debian.
Obviously there are many more features of namespaces we have not used, like
isolating the network for our process.
But I hope these are enough insights to get the core idea of containers on
Linux.</p>
The missing part are control groups (cgroups), but I will maybe cover them
in a future post.
But if you know Kubernetes: cgroups enable the limits on a pod.</p>
Conclusion</a></h2>
Containers are no black magic.
They use resources provided by the Linux Kernel to isolate process and manipulate
their root directory.
We needed about 75 lines of code (with comments) to implement the basic features
to emulate the docker run</code> command.
The code is based on the excellent talk and
code of Liz Rice</a>
some years ago, so if you want to get a more interactive version of this post
check the talk!</p>
As always the full code can be found on GitHub</a>.</p>
Sources</a></h2>



https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor ↩</a></p>
</li>

https://man7.org/linux/man-pages/man7/namespaces.7.html ↩</a> ↩2</a></p>
</li>

https://linux.die.net/man/1/chroot ↩</a> ↩2</a></p>
</li>

https://man7.org/linux/man-pages/man7/cgroups.7.html ↩</a> ↩2</a></p>
</li>
</ol>
</section>


Basic In-Memory Cache in Go
2024-09-19T00:00:00+00:00
I often tend to stumble upon a basic problem and then try to solve it with some
external library.
In this case I was thinking about caching some data in memory to spare some time
and resources on an external API.
This time I've decided to write something myself instead of using an external
library.
And with that you can even learn something from it.</p>
So let's write a very basic in-memory cache in Go.</p>
Requirements</a></h2>
The cache should be able to store key-value pairs.
We should be able to set a key-value pair, get a value by key, and delete a
key-value pair.</p>
Each key-value pair should have a time-to-live (TTL) value and should be
automatically deleted after the TTL has expired.
There are different ways to achieve that, but we will come back to this later.</p>
The interface for the cache is very simple and only contains the following 3 methods:</p>
type</span> Cache</span> interface</span> {</span></span>
	Set</span>(</span>key</span> string</span>,</span> value</span> any</span>)</span></span>
	Get</span>(</span>key</span> string</span>)</span> (</span>any</span>,</span> bool</span>)</span></span>
    Delete</span>(</span>key</span> string</span>)</span></span>
}</span></span></code></pre>Implementation</a></h2>
We can represent the cache as a map of keys to a struct (CacheItem</code>) that holds
the value and the time when it was added or should be expiring.</p>
type</span> CacheItem</span> struct</span> {</span></span>
    Value</span> any</span></span>
    Expiration</span> time</span>.</span>Time</span></span>
}</span></span></code></pre>
There are two design decisions to make here.</p>

We can decide to only store specific values in the cache, e.g. only strings.</li>
For the expiration we can decide to use a fixed time or a relative time.
Additionally, we can decide between the time when the item was added or the
time when the item should expire.</li>
</ol>
As the cache is very basic, we will use the any</code> type for the value and the
time when the item should expire.
We have to keep in mind to add the expiration time when adding an item to the cache.</p>
expiration</span> :=</span> time</span>.</span>Now</span>(</span>)</span>.</span>Add</span>(</span>c</span>.</span>expiration</span>)</span>.</span>UnixNano</span>(</span>)</span></span></code></pre>
As we want to have somewhat good time precision, we use the UnixNano()</code> function
to get the current time in nanoseconds.
This enables us to store the time as a simple int64</code> value in our CacheItem</code>.</p>
type</span> CacheItem</span> struct</span> {</span></span>
    Value</span> any</span></span>
    Expiration</span> int64</span></span>
}</span></span></code></pre>
To enable concurrent access to the cache we have to use a mutex.
Go provides a sync</code> package that contains a Mutex</code> and a RWMutex</code>.
With that we can easily protect our cache from concurrent access.</p>
The Set</code>, Get</code> and Delete</code> methods are straight forward:</p>
Set</code> locks the map and inserts the value by wrapping it in a CacheItem</code>.
We will cover the Expiration</code> value in a second.</p>
Get</code> read locks the map and is checking if a value for the given key exists,
if yes we return it, if not then nothing will be returned.
Additionally, there is a boolean value to indicate if there was an item found.
This could also return an error, but let's keep it simple for now.</p>
Delete</code> locks the map and deletes the key-value pair from the map.</p>
const</span> defaultExpirationTime</span> =</span> 10</span> *</span> time</span>.</span>Second</span></span>
</span>
type</span> CacheItem</span> struct</span> {</span></span>
	Value</span>      any</span></span>
	Expiration</span> int64</span></span>
}</span></span>
</span>
type</span> Cache</span> struct</span> {</span></span>
	mu</span>    sync</span>.</span>RWMutex</span></span>
	items</span> map</span>[</span>string</span>]</span>CacheItem</span></span>
}</span></span>
</span>
func</span> (</span>c </span>*</span>Cache</span>)</span> Set</span>(</span>key</span> string</span>,</span> value</span> any</span>)</span> {</span></span>
	c</span>.</span>mu</span>.</span>Lock</span>(</span>)</span></span>
	defer</span> c</span>.</span>mu</span>.</span>Unlock</span>(</span>)</span></span>
</span>
	c</span>.</span>items</span>[</span>key</span>]</span> =</span> CacheItem</span>{</span></span>
		Value</span>:</span>      value</span>,</span></span>
		Expiration</span>:</span> time</span>.</span>Now</span>(</span>)</span>.</span>Add</span>(</span>defaultExpirationTime</span>)</span>.</span>UnixNano</span>(</span>)</span>,</span></span>
	}</span></span>
}</span></span>
</span>
func</span> (</span>c </span>*</span>Cache</span>)</span> Get</span>(</span>key</span> string</span>)</span> (</span>any</span>,</span> bool</span>)</span> {</span></span>
	c</span>.</span>mu</span>.</span>RLock</span>(</span>)</span></span>
	defer</span> c</span>.</span>mu</span>.</span>RUnlock</span>(</span>)</span></span>
</span>
	item</span>,</span> ok</span> :=</span> c</span>.</span>items</span>[</span>key</span>]</span></span>
	if</span> !</span>ok</span> {</span></span>
		return</span> nil</span>,</span> false</span></span>
	}</span></span>
</span>
	return</span> item</span>.</span>Value</span>,</span> true</span></span>
}</span></span>
</span>
func</span> (</span>c </span>*</span>Cache</span>)</span> Delete</span>(</span>key</span> string</span>)</span> {</span></span>
	c</span>.</span>mu</span>.</span>Lock</span>(</span>)</span></span>
	defer</span> c</span>.</span>mu</span>.</span>Unlock</span>(</span>)</span></span>
	delete</span>(</span>c</span>.</span>items</span>,</span> key</span>)</span></span>
}</span></span></code></pre>
With that we have an in-memory cache that can be read or written to by multiple goroutines.
But our values will be kept forever, if we don't delete them manually.
So let's talk about the eviction behavior we defined earlier.</p>
Eviction</a></h2>
There are multiple ways to evict items from the cache.
Well, there are also multiple different algorithms to evict, but for now
let's keep it simple and stick with the original requirements.</p>

We can evict expired items on every access.</li>
We can evict expired items in the background.</li>
</ol>
For the first option we would have to check if the item is expired every time
we access the cache, which would be very inefficient.
Therefor we will use the second option.</p>
To delete expired items from the cache we can use a simple goroutine that
runs in the background and checks for expired items.
We can use a time.Ticker</code> to check for expired items every second.</p>
func</span> (</span>c </span>*</span>Cache</span>)</span> StartEvict</span>(</span>)</span> {</span></span>
	ticker</span> :=</span> time</span>.</span>NewTicker</span>(</span>time</span>.</span>Second</span>)</span></span>
</span>
	for</span> {</span></span>
		select</span> {</span></span>
		case</span> <-</span>ticker</span>.</span>C</span>:</span></span>
			c</span>.</span>evict</span>(</span>)</span></span>
		}</span></span>
	}</span></span>
}</span></span></code></pre>
The evict</code> methods checks for expired items and deletes them from the cache.
As we are just using a simple map, we can iterate over all items and check
if the expiration time is in the past and call the delete</code> function.</p>
func</span> (</span>c </span>*</span>Cache</span>)</span> evict</span>(</span>)</span> {</span></span>
	c</span>.</span>mu</span>.</span>Lock</span>(</span>)</span></span>
	defer</span> c</span>.</span>mu</span>.</span>Unlock</span>(</span>)</span></span>
</span>
	for</span> key</span>,</span> item</span> :=</span> range</span> c</span>.</span>items</span> {</span></span>
		if</span> time</span>.</span>Now</span>(</span>)</span>.</span>UnixNano</span>(</span>)</span> ></span> item</span>.</span>Expiration</span> {</span></span>
			fmt</span>.</span>Println</span>(</span>"</span>\t</span>Evicting: </span>"</span>,</span> key</span>)</span></span>
			delete</span>(</span>c</span>.</span>items</span>,</span> key</span>)</span></span>
		}</span></span>
	}</span></span>
}</span></span></code></pre>
A downside of this approach is that we have to lock the cache while we are
evicting items, and we are doing this every second.</p>
It might not be the most elegant or efficient way, but it fulfills our
requirements and is only around 50 lines of code.</p>
Improvements</a></h2>
There are a few improvements that can be made to this simple cache.</p>

Support for different expiration times for each item.</li>
Support for different eviction strategies like Least Recently Used (LRU)</em> or First In First Out (FIFO)</em> instead of just time based eviction.</li>
Dynamic eviction times based on the size of the cache.
Currently, we just check every second for expired items, but we could vary the time based on the size of the cache.</li>
Persist the cache to disk.
This would enable us to restore the cache after a restart of the application.</li>
Properly handle shutdown.</li>
There are more improvements that can be made, but these are the most obvious ones.</li>
</ol>
Conclusion</a></h2>
Of course this is not really production ready as it is nether very efficient nor
fast.
But for a hobby project that is doing a few API calls and has to spare some
requests, it might be enough.
And there was no need to fetch any external dependencies in our project!</p>


Sign your Git commits
2023-12-28T00:00:00+00:00
I recently learned that you can simply sign your Git commits with your SSH key.
With that there is no need for a (rather complex) PGP/GPG setup!
Instead, just re-use the SSH key you are probably already using to interact with your git remote.</p>
This is already supported by GitHub 1</a></sup> and GitLab 2</a></sup>, and you will see a green check mark or verified</code> next to your commit message.</p>
</p>
You only need the following in your .gitconfig</code> (I named the example file .gitconfig-sign</code>, but you can just add it
to your regular .gitconfig</code>).
Ensure that you reference the ssh key you want to use for signing and that's it.</p>
</p>
If you want to explicitly sign a commit, you can use the -S</code> flag while committing a change.
But as shown above, you can simply enable it for all commits with setting gpgsign = true</code>.</p>
git</span> commit</span> -</span>S</span> -</span>m</span> "</span>I'm signed</span>"</span></span></code></pre>
Now might be a good time to add a password to your ssh key.
If you already got a key you can easily add a passphrase with the following command:</p>
ssh-keygen</span> -</span>p</span> -</span>f</span> ~/.ssh/id_ed25519</span></span></code></pre>
In the case you got multiple keys just use the -f</code> flag to target the private key you want to add a passphrase to.</p>
If you don't want to type your passphrase for every commit, just add this setting to your ~/.ssh/config</code> to ensure
the key will be picked up by your key agent.
And if your ssh agent is not working, head over to this</a> post and enable it.</p>
AddKeysToAgent</span> yes</span></span></code></pre>Sources</a></h2>



https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits ↩</a></p>
</li>

https://docs.gitlab.com/ee/user/project/repository/signed_commits/ssh.html ↩</a></p>
</li>
</ol>
</section>


Buildpacks
2023-06-02T00:00:00+00:00
At KubeCon 2023 Amsterdam I attended a talk about Buildpacks</a>.
This is not the first time I heard about Buildpacks, but the talk in the maintainer track was very entertaining and
the features like advanced caching and reproducibility sounded very convincing.
So in this post I'll try to replace an existing build pipeline based on pure docker with Buildpacks.
I'll also go a bit in detail on how to use buildpacks and show differences in the resulting images.</p>
Why use Buildpacks instead of plain docker build?</a></h2>
You can find all features on this overview page</a>, but in my opinion the most relevant
features are:</p>

Bill-of-Materials</li>
</ol>
I talked about the risks of adding dependencies in an earlier post</a>.
Adding a Software Bill of Materials (SBOM) is the first step to care about your dependencies, as you will at least provide
a list of dependencies with the released artifact of your application.</p>

Advanced Caching</li>
</ol>
This will speed up your builds, especially if you just change a single line of code.
In my short experience with Buildpacks, this works very well and after an initial build to fill up caches, it is very fast.</p>

Minimal app image</li>
</ol>
We will come later to this, but in short images are relatively small, but larger than a multi-stage build image using
a scratch</code> image as second base.</p>

Reproducibility</li>
</ol>
One downside if reproducibility is that the creation timestamp of the image will be ~40 years in the past.
But you can easily adjust that with supplying --creation-time now</code> as additional argument to pack</code>.
Otherwise, this is a very much needed feature (as for example go already produces reproducible images in almost all cases). 1</a></sup></p>
One major thing that is missing as of now is support to sign images.
There is an RFC, but it seems rather inactive. 2</a></sup></p>
Buildpacks Concepts</a></h2>
All of this is based on this upstream documentation</a>.</p>
Buildpacks uses builders to produce OCI images.
These builders can themselves consist of multiple buildpacks and a single buildpack is normally used to target a single
language to build e.g. go</code> or Python</code>.
That combination of buildpacks and some lifecycle layer from buildpacks itself is then used as a builder image.
This builder image will at build time combined with your app code and will then build the actual OCI app image. 3</a></sup></p>
If a builder can't detect any relevant files (e.g. a go.mod</code>) and therefore can't build, it will simply fail.
The interesting part is, that you can have multiple buildpacks in a builder, so for example a go</code> builder to build your
backend and a Node</code> builder to build your frontend.</p>
Instead of choosing a base image with Docker, you now need to decide on which builder to use.
There are multiple providers available for builders, even from public cloud providers like Google, but a good source
seems to be the Paketo Builders</a>.</p>
Comparison</a></h2>
Let's say we want to containerize a simple Go web service and instead of using a Dockerfile</code> and docker/podman to build it,
we want to use Buildpacks.
As always you can find all related files in this GitHub repository</a>.</p>
The Dockerfile looks like this:</p>
FROM</span> golang:1.20-alpine </span>AS</span> builder</span></span>
</span>
COPY</span> . /app</span></span>
WORKDIR</span> /app</span></span>
</span>
RUN</span> go get ./...</span></span>
</span>
RUN</span> go build -o test ./...</span></span>
</span>
</span>
</span>
FROM</span> scratch</span></span>
</span>
COPY</span> --from=builder /app/test /test</span></span>
</span>
ENTRYPOINT</span> [</span>"</span>/test</span>"</span>]</span></span></code></pre>
As stated earlier for Buildpacks we don't need a Dockerfile</code>, but a builder we want to use.
In my case I decided to go with the paketo-buildpacks/go</code> buildpack and the paketobuildpacks/builder:tiny</code> builder. 4</a></sup>
You can reproduce this in the earlier mentioned repository by running make size</code>.</p>
The size of the resulting images is depending on the used builder (tiny</code> or base</code>):</p>
    Name</span>                    Size</span></span>
buildpacks-test-pack-go</span>    107MB</span></span>
buildpacks-test-pack-tiny</span>  30.5MB</span></span>
buildpacks-test-docker</span>     6.62MB</span></span></code></pre>
The resulting Docker image size is 6.62MB</code>.
The resulting Buildpacks image size is 30.5MB</code> with the tiny</code> builder and 107MB</code> with the base</code> builder.</p>
Let's take a dive</a> and figure out what is in these images.
Dive provides an easy way to see how much waste there is in your image by enabling the CI=true</code> mode.
To be clear, this is in no way perfect to compare the results here, but a good indicator of what parts of the images
are maybe not needed at all.</p>
Image</th> Efficiency</th> Wasted Bytes</th> User Wasted Percent</th></tr></thead>

buildpacks-test-pack-go</td> 92.1218 %</td> 15691768 bytes (16 MB)</td> 36.1717 %</td></tr>
buildpacks-test-pack-tiny</td> 99.9975 %</td> 1497 bytes (1.5 kB)</td> 0.0116 %</td></tr>
buildpacks-test-docker</td> 100 %</td> 0 bytes (0 B)</td> NaN %</td></tr>
</tbody></table>
As expected the docker</code> image build by docker has 0 bytes wasted as it was based on a scratch</code> image, and we just
copied the go binary in the image.
But the buildpacks-tiny</code> image is also very close with an efficiency of 99.9975 % and only 1.5 kB wasted.
As expected the buildpacks-go</code> image, which is based on paketobuildpacks/builder:base</code> (so not</strong> tiny), is the largest
image with 16 MB wasted.
The largest file in that image seems to be a perl runtime at /usr/bin/perl</code> with 4.2 MB.</p>
I'm not entirely sure how dive calculates these metrics, as we know that our binary has the size of ~6.6 MB, that leaves
us with more than 20 MB in the buildpacks-test-pack-tiny</code> image, that is maybe not needed.
Parts of it could be the SBOM or vendored modules, but as said earlier I'm not sure about this.</p>
$</span> CI=</span>true</span> dive</span> buildpacks-test-pack-go</span></span>
</span>
Using</span> default</span> CI</span> config</span></span>
Image</span> Source:</span> docker://buildpacks-test-pack-go</span></span>
Fetching</span> image...</span> (this</span> can</span> take</span> a</span> while</span> for</span> large</span> images</span>)</span></span>
Analyzing</span> image...</span></span>
  efficiency:</span> 92.1218</span> %</span></span>
  wastedBytes:</span> 15691768</span> bytes</span> (16</span> MB</span>)</span></span>
  userWastedPercent:</span> 36.1717</span> %</span></span>
Inefficient</span> Files:</span></span>
Count</span>  Wasted</span> Space</span>  File</span> Path</span></span>
    2</span>        4.2</span> MB</span>  /usr/bin/perl</span></span>
    2</span>        1.3</span> MB</span>  /var/cache/debconf/templates.dat</span></span>
    2</span>        1.1</span> MB</span>  /usr/lib/x86_64-linux-gnu/perl-base/auto/re/re.so</span></span></code></pre>$</span> CI=</span>true</span> dive</span> buildpacks-test-pack-tiny</span></span>
</span>
Using</span> default</span> CI</span> config</span></span>
Image</span> Source:</span> docker://buildpacks-test-pack-tiny</span></span>
Fetching</span> image...</span> (this</span> can</span> take</span> a</span> while</span> for</span> large</span> images</span>)</span></span>
Analyzing</span> image...</span></span>
  efficiency:</span> 99.9975</span> %</span></span>
  wastedBytes:</span> 1497</span> bytes</span> (1.5</span> kB</span>)</span></span>
  userWastedPercent:</span> 0.0116</span> %</span></span>
Inefficient</span> Files:</span></span>
Count</span>  Wasted</span> Space</span>  File</span> Path</span></span>
    2</span>        1.0</span> kB</span>  /etc/os-release</span></span>
    2</span>         339</span> B</span>  /etc/passwd</span></span>
    2</span>         140</span> B</span>  /etc/group</span></span></code></pre>$</span> CI=</span>true</span> dive</span> buildpacks-test-docker</span></span>
</span>
Using</span> default</span> CI</span> config</span></span>
Image</span> Source:</span> docker://buildpacks-test-docker</span></span>
Fetching</span> image...</span> (this</span> can</span> take</span> a</span> while</span> for</span> large</span> images</span>)</span></span>
Analyzing</span> image...</span></span>
  efficiency:</span> 100.0000</span> %</span></span>
  wastedBytes:</span> 0</span> bytes</span> (0</span> B</span>)</span></span>
  userWastedPercent:</span> NaN</span> %</span></span>
Inefficient</span> Files:</span></span>
Count</span>  Wasted</span> Space</span>  File</span> Path</span></span>
None</span></span></code></pre>SBOM</a></h2>
After talking about the SBOM multiple times now, I of course want to see how it looks for my images.
To the SBOM you can use pack sbom download <image></code>.
This will export the SBOM layer of the image in the local directory in the folder layers</code>.</p>
So for the tiny image it looks like this:</p>
$</span> tree</span> layers</span></span>
</span>
layers</span></span>
└──</span> sbom</span></span>
    └──</span> launch</span></span>
        ├──</span> paketo-buildpacks_ca-certificates</span></span>
        │  </span> └──</span> helper</span></span>
        │  </span>     └──</span> sbom.syft.json</span></span>
        ├──</span> paketo-buildpacks_go-build</span></span>
        │  </span> └──</span> targets</span></span>
        │  </span>     ├──</span> sbom.cdx.json</span></span>
        │  </span>     ├──</span> sbom.spdx.json</span></span>
        │  </span>     └──</span> sbom.syft.json</span></span>
        └──</span> sbom.legacy.json</span></span>
</span>
7</span> directories,</span> 5</span> files</span></span></code></pre>
There are multiple SBOMs in 3 formats CycloneDX</a>, Syft</a>
and Spdx</a>.
All of them look a bit different, but contain more or less the same content, like used go compiler version, build flags
and dependencies (that is why I've added zap</code> as logger dependency). 5</a></sup></p>
Using Buildpacks on GitLab CI</a></h2>
I was thinking about using it in a container, like with a docker in docker build setup, but in the end I reused my
existing shell runner.</p>
There are ways on how to use it directly within a container, but these are rather cumbersome in comparison to using
plain pack</code>. 6</a></sup></p>
Therefore, I deduced to just reuse an existing shell runner, and install pack on it.
This has the large disadvantage of still needing a privileged runner with root permissions, but in theory you could
use Podman (or something else) instead of docker in your runner and with this build images without root permissions needed
(which is equal to be in the docker</code> group).</p>
With that I used the following GitLab CI configuration:</p>
v</span>ariables</span>:</span></span>
    B</span>UILDPACK_VERSION</span>:</span> 0.29.0</span></span>
</span>
b</span>ackend-build</span>:</span></span>
    t</span>ags</span>:</span></span>
        -</span> d</span>ocker-build</span></span>
    s</span>tage</span>:</span> b</span>uild</span></span>
    b</span>efore_script</span>:</span></span>
        -</span> d</span>ocker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY</span></span>
    s</span>cript</span>:</span></span>
        -</span> c</span>d backend</span></span>
        -</span> (</span>curl -sSL "https://github.com/buildpacks/pack/releases/download/v${BUILDPACK_VERSION}/pack-v${BUILDPACK_VERSION}-linux.tgz" | tar -C . --no-same-owner -xzv pack)</span></span>
        -</span> .</span>/pack build gitlab.cloudf.de:4567/andre/shorty/backend:latest --path . --buildpack paketo-buildpacks/go --builder paketobuildpacks/builder:tiny --creation-time now</span></span>
        -</span> d</span>ocker push gitlab.cloudf.de:4567/andre/shorty/backend:latest</span></span></code></pre>
The build times are also very good, once the first build has passed and layers can be reused.
For a production project, I've seen build times of < 1 minute with code changes in the code and very limited network
upload speeds.</p>
Conclusion</a></h2>
If you know how to write Dockerfiles, Buildpacks might be a bit weird at the beginning as there is no need to define
all dependencies needed by your application in a file.
All of this complexity moved in the builders, which can be beneficial as you can provide your own images, for example
in a business environment.</p>
The downside is that you have limited control of what will be in your image at the end, as the builder has full control
over it.</p>
In general I like the speed and the earlier mentioned capabilities of Buildpacks.
The only thing missing is signing support, but otherwise I'll try Buildpacks for some private projects in the future
and may think about using it at work (but with custom builders).</p>
Sources</a></h2>



https://github.com/golang/go/issues/57120 ↩</a></p>
</li>

https://github.com/buildpacks/rfcs/pull/195 ↩</a></p>
</li>

https://buildpacks.io/docs/concepts/#what-is-a-builder ↩</a></p>
</li>

https://paketo.io/docs/howto/go/ ↩</a></p>
</li>

https://buildpacks.io/docs/features/bill-of-materials/ ↩</a></p>
</li>

https://blog.codecentric.de/en/2021/10/gitlab-ci-paketo-buildpacks/ ↩</a></p>
</li>
</ol>
</section>


Improve your Go Code with Benchmarks
2023-03-31T00:00:00+00:00
We often write code without thinking about how performant the code will be in the end.
Often it is also not worth to sacrifice cleaner code in terms of readability and future
maintainability for the sake of a few percents of performance, be it less CPU cycles needed
to compute something or a smaller memory footprint.</p>
But as always even if you don't need to save a few cycles or bytes, it is good to know what
you could do to even get started in optimize for performance.
This post will about how to use the tooling provided by the Go programming language.</p>
The Scenario</a></h2>
Let's do something very easy!
We just want to generate a combination 6 random characters out of the alphabet (A-Z and a-z) and
integers ranging from 0 to 9.
The output could for example look like this bGrS7X</code>.
What could that be good for?
I've got no idea, but it is a very easy example on how to use the extensive tooling of Go.</p>
You can find all code for this post here</a>.</p>
What tools do we need?</a></h2>
We want multiple metrics for a given algorithm.
For example, we want to know how many CPU cycles it needs to perform a specific operation.
Additionally, we want to know how much memory it needs to do that.
So let's get started with benchmarks.</p>
Benchmarks</a></h2>
As always Go already provides all we need in the testing</code> package. 1</a></sup>
As the documentation states you just have to add the -bench</code> flag to go test</code> and write benchmarks
following the presented examples.</p>
So let me introduce a very simply algorithm to fulfill to earlier presented requirements of creating
a random string.</p>
func</span> algStringCasting</span>(</span>)</span> string</span> {</span></span>
	//</span> Stolen from https://blog.pratimbhosale.com/building-a-url-shortener-using-go-and-sqlite#heading-shortening-the-url</span></span>
	randomPart</span> :=</span> "</span>"</span></span>
	for</span> i</span> :=</span> 0</span>;</span> i</span> <</span> 6</span>;</span> i</span>++</span> {</span></span>
		randomPart</span> +=</span> string</span>(</span>rune</span>(</span>rand</span>.</span>Intn</span>(</span>26</span>)</span> +</span> 97</span>)</span>)</span></span>
	}</span></span>
</span>
	return</span> randomPart</span></span>
}</span></span></code></pre>
A simple benchmark for this algorithm would then look like this.</p>
func</span> Benchmark_algStringCasting</span>(</span>b</span> *</span>testing</span>.</span>B</span>)</span> {</span></span>
	for</span> n</span> :=</span> 0</span>;</span> n</span> <</span> b</span>.</span>N</span>;</span> n</span>++</span> {</span></span>
		algStringCasting</span>(</span>)</span></span>
	}</span></span>
}</span></span></code></pre>
In this case there is no need to increase any external variable to run it with a different set of inputs,
as the algorithm should be static.
To get some results we then just execute go test -bench=. ./..</code>, which will give us the following.</p>
➜</span>  03-generate-random-chars</span> git:</span>(</span>main</span>)</span> ✗</span> go</span> test</span> -</span>bench=.</span> ./..</span></span>
goos:</span> linux</span></span>
goarch:</span> amd64</span></span>
pkg:</span> github.com/andresterba/code-for-blog/2023/03</span></span>
cpu:</span> Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i5-8265U</span> CPU</span> @</span> 1.60GHz</span></span>
Benchmark_algStringCasting-8</span>    1389026</span>        869.3</span> ns/op</span></span>
PASS</span></span>
ok</span>      github.com/andresterba/code-for-blog/2023/03</span>    5.753s</span></span></code></pre>
The first number we can see 1389026</code> is the number of runs of this function.
The second number is the needed time in average in nanoseconds per operations.
The -8</code> suffix shows the number of CPUs used to run the benchmark.
[3]</p>
As we want more stable results and remove statistical outliers, we always want to run benchmarks
multiple times.
We can simply do this by adding the -benchtime</code> flag with the special syntax 10x</code> to run the benchmark
10 times.</p>
So now we got some metrics for CPU and runtime, but we also want to now how much memory the algorithm is using
and how many allocations it needs.
To get this we just have to add another flag to the test command, which is -benchmem</code>.
With these two additional flags we get the following output.</p>
go</span> test</span> -</span>bench=.</span> -</span>benchmem</span> -</span>benchtime=100x</span> ./...</span></span>
goos:</span> linux</span></span>
goarch:</span> amd64</span></span>
pkg:</span> github.com/andresterba/code-for-blog/2023/03</span></span>
cpu:</span> Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i5-8265U</span> CPU</span> @</span> 1.60GHz</span></span>
Benchmark_algStringCasting-8</span>         100</span>               464.8</span> ns/op</span>            56</span> B/op</span>         11</span> allocs/op</span></span>
PASS</span></span>
ok</span>      github.com/andresterba/code-for-blog/2023/03</span>    0.004s</span></span></code></pre>
Each operation allocations 56 bytes of memory in 11 allocations.</p>
Add another algorithm</a></h2>
Now that we have a working benchmark setup, let's add another algorithm that solves the same problem and use
the provided tooling to compare it against our first try.</p>
Instead of generating random chars, we will now provide a lookup array and just create random indexes for
that array.</p>
var</span> charArray</span> [</span>62</span>]</span>string</span> =</span> [</span>62</span>]</span>string</span>{</span></span>
	"</span>a</span>"</span>,</span> "</span>b</span>"</span>,</span> "</span>c</span>"</span>,</span> "</span>d</span>"</span>,</span> "</span>e</span>"</span>,</span> "</span>f</span>"</span>,</span> "</span>g</span>"</span>,</span></span>
	"</span>h</span>"</span>,</span> "</span>i</span>"</span>,</span> "</span>j</span>"</span>,</span> "</span>k</span>"</span>,</span> "</span>l</span>"</span>,</span> "</span>m</span>"</span>,</span> "</span>n</span>"</span>,</span></span>
	"</span>o</span>"</span>,</span> "</span>p</span>"</span>,</span> "</span>q</span>"</span>,</span> "</span>r</span>"</span>,</span> "</span>s</span>"</span>,</span> "</span>t</span>"</span>,</span> "</span>u</span>"</span>,</span></span>
	"</span>v</span>"</span>,</span> "</span>w</span>"</span>,</span> "</span>x</span>"</span>,</span> "</span>y</span>"</span>,</span> "</span>z</span>"</span>,</span></span>
	"</span>A</span>"</span>,</span> "</span>B</span>"</span>,</span> "</span>C</span>"</span>,</span> "</span>D</span>"</span>,</span> "</span>E</span>"</span>,</span> "</span>F</span>"</span>,</span> "</span>G</span>"</span>,</span></span>
	"</span>H</span>"</span>,</span> "</span>I</span>"</span>,</span> "</span>J</span>"</span>,</span> "</span>K</span>"</span>,</span> "</span>L</span>"</span>,</span> "</span>M</span>"</span>,</span> "</span>N</span>"</span>,</span></span>
	"</span>O</span>"</span>,</span> "</span>P</span>"</span>,</span> "</span>Q</span>"</span>,</span> "</span>R</span>"</span>,</span> "</span>S</span>"</span>,</span> "</span>T</span>"</span>,</span> "</span>U</span>"</span>,</span></span>
	"</span>V</span>"</span>,</span> "</span>W</span>"</span>,</span> "</span>X</span>"</span>,</span> "</span>Y</span>"</span>,</span> "</span>Z</span>"</span>,</span></span>
	"</span>0</span>"</span>,</span> "</span>1</span>"</span>,</span> "</span>2</span>"</span>,</span> "</span>3</span>"</span>,</span> "</span>4</span>"</span>,</span> "</span>5</span>"</span>,</span> "</span>6</span>"</span>,</span></span>
	"</span>7</span>"</span>,</span> "</span>8</span>"</span>,</span> "</span>9</span>"</span>,</span></span>
}</span></span>
</span>
func</span> algArrayLookup</span>(</span>)</span> string</span> {</span></span>
	//</span> Stolen from https://www.geeksforgeeks.org/program-generate-random-alphabets/</span></span>
	res</span> :=</span> "</span>"</span></span>
</span>
	for</span> i</span> :=</span> 0</span>;</span> i</span> <</span> 6</span>;</span> i</span>++</span> {</span></span>
		res</span> =</span> res</span> +</span> charArray</span>[</span>rand</span>.</span>Intn</span>(</span>62</span>)</span>]</span></span>
	}</span></span>
</span>
	return</span> res</span></span>
}</span></span></code></pre>
The benchmark looks exactly the same as the one for our first algorithm, we just call algArrayLookup</code> instead of algStringCasting</code>.</p>
func</span> Benchmark_algArrayLookup</span>(</span>b</span> *</span>testing</span>.</span>B</span>)</span> {</span></span>
	for</span> n</span> :=</span> 0</span>;</span> n</span> <</span> b</span>.</span>N</span>;</span> n</span>++</span> {</span></span>
		algArrayLookup</span>(</span>)</span></span>
	}</span></span>
}</span></span></code></pre>
And the results?
They look like this:</p>
go</span> test</span> -</span>bench=.</span> -</span>benchmem</span> -</span>benchtime=100x</span> ./...</span></span>
goos:</span> linux</span></span>
goarch:</span> amd64</span></span>
pkg:</span> github.com/andresterba/code-for-blog/2023/03</span></span>
cpu:</span> Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i5-8265U</span> CPU</span> @</span> 1.60GHz</span></span>
Benchmark_algStringCasting-8</span>         100</span>               874.9</span> ns/op</span>            56</span> B/op</span>         11</span> allocs/op</span></span>
Benchmark_algArrayLookup-8</span>           100</span>               395.0</span> ns/op</span>            24</span> B/op</span>          5</span> allocs/op</span></span>
PASS</span></span>
ok</span>      github.com/andresterba/code-for-blog/2023/03</span>    0.004s</span></span></code></pre>
So our second algorithm takes less time per operation, bytes per operation and requires only 5 allocations instead of 11.
It might be a good idea to add benchmarks like this to your code base, before you make
any major changes in the implementation of a specific functionality.
But how do you know if your new implementation is better/faster than the old one?</p>
Of course, you can just keep the old benchmarks lying around and check them manually.
But there is also a tool for that, called benchstat</a>.</p>
Testing with different inputs</a></h2>
So let's come up with another example, where we don't want to generate a static element, which is independent of some
input.
Let's use a well known algorithm that scales with a provided input, for this example we are using the
Euclidean algorithm</a>.</p>
It is used to find the greatest common divisor of two numbers.
An example from the Wikipedia is 21 is the GCD of 252 and 105.
Pseudocode for this looks like this. 2</a></sup></p>
function gcd(a, b)</span></span>
    while b ≠ 0</span></span>
        t := b</span></span>
        b := a mod b</span></span>
        a := t</span></span>
    return a</span></span></code></pre>
In Go this will look like this.</p>
func</span> Euclid</span>(</span>a</span> int</span>,</span> b</span> int</span>)</span> int</span> {</span></span>
	for</span> b</span> !=</span> 0</span> {</span></span>
		t</span> :=</span> b</span></span>
		b</span> =</span> a</span> %</span> b</span></span>
		a</span> =</span> t</span></span>
	}</span></span>
</span>
	return</span> a</span></span>
}</span></span></code></pre>
And the output for our example 252 and 105 will be as expected:</p>
➜</span> go</span> run</span> main.go</span></span>
</span>
GCD</span> of</span> 252</span> and</span> 105</span> is</span> 21.</span></span></code></pre>
Then let's write a benchmark for this with different inputs.
We define a table called inputs</code> to collect a list of inputs we want to benchmark.
b.Run()</code> accepts as first argument the name of the benchmark.</p>
var</span> inputs</span> =</span> [</span>]</span>struct</span> {</span></span>
	a</span> int</span></span>
	b</span> int</span></span>
}</span>{</span></span>
	{</span></span>
		a</span>:</span> 252</span>,</span></span>
		b</span>:</span> 105</span>,</span></span>
	}</span>,</span></span>
	{</span></span>
		a</span>:</span> 1337</span>,</span></span>
		b</span>:</span> 105</span>,</span></span>
	}</span>,</span></span>
}</span></span>
</span>
func</span> BenchmarkEuclid</span>(</span>b</span> *</span>testing</span>.</span>B</span>)</span> {</span></span>
	for</span> _</span>,</span> v</span> :=</span> range</span> inputs</span> {</span></span>
		b</span>.</span>Run</span>(</span>fmt</span>.</span>Sprintf</span>(</span>"</span>a:</span>%d</span> b:</span>%d</span>"</span>,</span> v</span>.</span>a</span>,</span> v</span>.</span>b</span>)</span>,</span> func</span>(</span>b</span> *</span>testing</span>.</span>B</span>)</span> {</span></span>
			for</span> i</span> :=</span> 0</span>;</span> i</span> <</span> b</span>.</span>N</span>;</span> i</span>++</span> {</span></span>
				Euclid</span>(</span>v</span>.</span>a</span>,</span> v</span>.</span>b</span>)</span></span>
			}</span></span>
		}</span>)</span></span>
	}</span></span>
}</span></span></code></pre>
The result looks like this.</p>
go</span> test</span> -</span>benchmem</span> ./..</span></span>
</span>
goos:</span> linux</span></span>
goarch:</span> amd64</span></span>
pkg:</span> github.com/andresterba/code-for-blog/2023/03/euclid</span></span>
cpu:</span> Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i5-8265U</span> CPU</span> @</span> 1.60GHz</span></span>
BenchmarkEuclid/a:252_b:105-8</span>         	21771715</span>	        50.82</span> ns/op</span>	       0</span> B/op</span>	       0</span> allocs/op</span></span>
BenchmarkEuclid/a:1337_b:105-8</span>        	17221270</span>	        58.49</span> ns/op</span>	       0</span> B/op</span>	       0</span> allocs/op</span></span>
PASS</span></span>
ok</span>  	github.com/andresterba/code-for-blog/2023/03/euclid</span>	4.139s</span></span></code></pre>
Interesting is that the compiler seems to optimize the code to remove the allocation t := b</code>.
But profiling is a different topic I will maybe cover in the near future.
If you want to know more about how to test with variable inputs, I can recommend this 3</a></sup> blog post.</p>
Conclusion</a></h2>
Writing benchmarks in Go is very easy, so there should be no reason to not do it for performance critical
parts of your application.
As always to Go tooling is very handy and easy to use.</p>



https://pkg.go.dev/testing#hdr-Benchmarks ↩</a></p>
</li>

https://en.wikipedia.org/wiki/Euclidean_algorithm ↩</a></p>
</li>

https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go ↩</a></p>
</li>
</ol>
</section>


Why you should update your dependencies automatically
2023-02-28T00:00:00+00:00
When was the last time you decided to update the dependencies of one of your projects?
And not because you needed a new feature, just to get the latest security patches for your project
which might be running on the public internet?
Well yes, I got the same problem: never.
I didn't update most dependencies of my own software until sometime earlier last year.</p>
My TL;DR today is that the setup was very easy, and I can sleep better now, as I know I will receive a notification
when one of my applications is vulnerable.
The two systems I have used to achieve this are Dependabot</a> for GitHub projects and
renovate</a> for GitLab projects in my private instance of GitLab.</p>
These tools do work with a variety of different package managers, e.g. go</code>, npm</code>, pip</code> and even docker</code>.</p>
Features</a></h2>
The main feature of automated updates is rather obvious: you get new upstream (security) fixes automatically.
And you get them via a new PR/MR in your repository, so there is no need to keep track of used dependencies and
manually subscribe to new releases.
What kind of updates you get depends on your configuration and the locked version in the lock file
of your package manager.
Normally I tend to enable all updates as most of the time I got at least some tests to cover the main
business logic of my app (or even some end-to-end tests), so I can be sure it will work with a new version of a dependency.</p>
Requirements</a></h2>
One thing that is needed to use these systems are tests.
Without tests, you can never be sure if a package update will result in broken features.
This is especially the case for frontend applications which tend to rely on a lot of external packages (mostly true for the JS/TS ecosystem in general).
In this case unit tests are often not enough, as you also want to update e.g. your ORM or database driver and
these are not covered by unit tests.</p>
If you got enough trusts in your tests you can even automatically merge MRs/PRs and ensure that everything is
still working as expected.
In my opinion this is the goal you should aim for.</p>
The process</a></h2>
Let's go over how this whole setup works with GitHub's Dependabot.
At first, you have to enable Dependabot for your repository.
This is as easy as adding the following file to your repository as .github/dependabot.yaml</code>:</p>
v</span>ersion</span>:</span> 2</span></span>
u</span>pdates</span>:</span></span>
  -</span> p</span>ackage-ecosystem</span>:</span> "</span>gomod</span>"</span></span>
    d</span>irectory</span>:</span> "</span>/</span>"</span></span>
    s</span>chedule</span>:</span></span>
      i</span>nterval</span>:</span> "</span>daily</span>"</span></span></code></pre>
Then you have to wait for dependency updates or a vulnerability.</p>
Once you got a vulnerable dependency in one of your projects you will be notified via mail
and get a Dependabot alert.</p>
</p>
Dependabot will then create a pull request in the affected project.</p>
</p>
All you have to do is to merge this PR, once you ensured it doesn't break anything of course.</p>
Risks</a></h2>
All this sounds just too good to be true, so I won't hide the fact that it also opens up new attack vectors.</p>
The first one is dependency confusion, which (in short) can be used to replace your private dependencies
by malware infected ones.
This is only possible because some dependency management systems (like npm or pip) rely on public services,
that host these packages and</strong> they will look there first, even if you provided a private package registry. 1</a></sup></p>
Another risk is that you pull malicious code in your repository, because an at the time of creating
your project trustworthy author abandoned his project and gave it to someone with bad intentions
or the project got attacked and some attacker added the malicious code to the code. 2</a></sup></p>
There are multiple more examples and attack vector on software supply chains documented on the internet (for example 3</a></sup> 4</a></sup> 5</a></sup>).</p>
Another risk (well the risk you always have with software) is introducing new bugs, as new features getting added.
You can sometimes mitigate this by only relying on security updates, but eventually you have to migrate
to a newer and support version of a dependency.</p>
When to add a new dependency?</a></h2>
This is the question you should always ask yourself before adding a new dependency to a project,
especially when you want to maintain it for a long time.
The answer is coupled to the ecosystem you use as in how rich the standard library of the language is and on how much work you want to do yourself (in re-inventing the wheel).
If you are using a language with a rich standard library like Go and</strong> are willing to write some
components like a more advanced HTTP server yourself, you can get around most of these problems.</p>
On the other hand you might invest your time in re-inventing components other people already did
(and maybe better than you did it in terms of security or performance).
One should weigh up which decision is the right one here.</p>
The future of our software supply chains</a></h2>
Your dependencies of a project are your software supply chain.
The name suggests that each of the components are interconnected and you must ensure at least
some sort of integrity of these packages.
I became aware of the topic through some scientific publications. 6</a></sup> 7</a></sup>
It is a topic of research and other languages or build systems already provide checksums or other mechanisms to mitigate or minimize risks of being a consumer of other software.</p>
Conclusion</a></h2>
In my opinion tools like Dependabot or renovate are an easy way to keep your software up to date
and secure.
They are also only consumers in the whole supply chain and therefore can't mitigate the earlier
mentioned risks alone, but they are an essential part when developing software.</p>



https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 ↩</a></p>
</li>

https://thehackernews.com/2021/07/malicious-npm-package-caught-stealing.html ↩</a></p>
</li>

https://secureteam.co.uk/news/vulnerabilities/javascript-supply-chain-attack-hits-millions-of-users/ ↩</a></p>
</li>

https://blog.jscrambler.com/how-your-code-dependencies-expose-you-to-web-supply-chain-attacks ↩</a></p>
</li>

https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/ ↩</a></p>
</li>

https://arxiv.org/abs/2204.04008 ↩</a></p>
</li>

https://dl.acm.org/doi/abs/10.1145/3510457.3513044 ↩</a></p>
</li>
</ol>
</section>


Use git worktree to work with multiple branches
2023-01-16T00:00:00+00:00
This post is about the awesome git worktree</code>, that simplified my personal workflow with git very much.
In the past I had at least 2 versions of repository checked out locally or fiddled around with git stash</code> to find
a thing I did weeks ago.
But let's talk about how I use git and how git worktree</code> helped me.</p>
Use Cases</a></h2>
When I'm reviewing a Pull or Merge Request, sometimes I would like to see the code in my own IDE instead of the
GitLab or GitHub UI.
Normally I have a single clone of a repository on my local machine and</strong> I'm in the process of adding new features or
fixing a bug myself.
So I don't want to commit things just yet, but just check out another branch.
I know I could use git stash</code> or create a WIP</code> commit and then amend by providing a good
commit message later, but almost all the time I forget about my existing stash or the earlier WIP</code>
commit.</p>
Another time I'm starting to work on an improvement or simply experimenting with an implementation detail and
therefore also don't want to commit anything just yet.</p>
At last, I want to update one of my branches with the current state of the main</code> branch or want to verify that something
is working or not working on the current main</code> branch in comparison to my branch.</p>
Working with worktree</a></h2>
The solution to all of these problems is to use git worktree</code> and have multiple branches checkout out locally at the
same time.</p>
To be as flexible as possible I tend to use the following structure for all repositories where I think I'll need
multiple branches checkout out at the same time:</p>
➜</span>  upstream-watch</span> pwd</span></span>
/home/aps/Code/github/upstream-watch</span></span>
➜</span>  upstream-watch</span> ls</span> -</span>l</span></span>
total</span> 8</span></span>
drwxr-xr-x</span> 5</span> aps</span> aps</span> 4096</span> Jan</span> 11</span> 08:27</span> my-feature-1</span></span>
drwxr-xr-x</span> 6</span> aps</span> aps</span> 4096</span> Jan</span> 11</span> 08:26</span> upstream-watch-main</span></span></code></pre>
So I got a <repo-name>--main</code> for the main branch and additional branches was <repo-name>--xxx</code>.</p>
Another feature you get with this is you get a clean state and can open your IDE in a new folder.
This could be annoying if you have a lot of custom configuration (like local .env</code> files, which are ignored by
git), but for most of the changes I don't need these.</p>
The main clone of the repository is always repository-name-main</strong>.
To not pollute my local file structure I tend to have each project in a dedicated directory under my ~/Code</code> folder.</p>
To create a new worktree checkout you simply have to use git worktree add -b my-feature-1 ../my-feature-1</code>.
If you don't want to create a new branch then simply skip the -b my-feature-1</code> from the command above and use it like this
git worktree add ../my-feature-1</code>.</p>
You can also list all your existing worktrees with from any worktree of the cloned repository:</p>
➜</span>  my-feature-1</span> git:</span>(</span>my-feature-1</span>)</span> git</span> worktree</span> list</span></span>
/home/aps/Code/github/upstream-watch/upstream-watch-main</span>  c477b51</span> [main</span>]</span></span>
/home/aps/Code/github/upstream-watch/my-feature-1</span>         c477b51</span> [my-feature-1</span>]</span></span></code></pre>
I'm lazy, so I'm mostly using this very simple wrapper script to create new worktrees.
Then I'll simply call cw my-feature</code>, when you store the scrip below as cw</code> (short for create worktree) in your local bin</code> folder.</p>
#!</span>/bin/bash</span></span>
set</span> -</span>euo</span> pipefail</span></span>
</span>
name</span>=</span>$</span>1</span></span>
</span>
git</span> worktree</span> add</span> ../</span>$</span>name</span> -</span>b</span> $</span>name</span></span>
</span>
printf</span> "</span>Created worktree at:  %s \n</span>"</span> "</span>../</span>$</span>name</span>"</span></span></code></pre>
Deleting a worktree is as simple as git worktree remove ../my-feature-1</code>.</p>
This also enables the usage of git cherry-pick</code> between multiple worktrees, which is very neat when you find a bug while
working on another task.
Simply commit it on your current working branch and pick it into a fix-bug-xxx</code> branch afterwards and create
a dedicated MR/PR for this fix.</p>
Conclusion</a></h2>
git worktree</code> simplified my personal workflow with git and saves some (not so expensive anymore) space on my local disk,
as I don't need multiple versions of a repository on my local machine.
I know worktree</code> is a rather old git feature, but nevertheless it is very useful and I know a lot of people that are not using it (yet).</p>


Organize your Semester with Kanban inspired Methods
2022-11-30T00:00:00+00:00
I recently finished my studies, so this seems a good time to reflect about how (and why) things worked out.
As always I like to share some insights in what I've learned in the last 5 years.
This is the second part of my series about how to study.
You can find the first part about how to write a thesis here</a></p>
Studying is more than about learning new things.
It is also about learning how to prioritize your available time and energy to achieve externally set goals.
And often these goals are not very specific, but rather vague and open to exploration.
In this post I'll present my Kanban inspired workflow to keep track of what I have to do within the semester and
hand in all assignments in time.</p>
What is Kanban?</a></h2>

Kanban is a popular Lean workflow management method for defining, managing, and improving services that deliver knowledge work.
It helps you visualize work, maximize efficiency, and improve continuously.
-- 1</a></sup></cite></p>
</blockquote>
Kanban is an agile project management method used in very different projects.
Its origins are in car manufacturing (Toyota), but it can be used in almost every context.
You can learn more about Kanban here 1</a></sup> and here 2</a></sup>.
I used it at work before, especially in projects with small teams as it has a very small management overhead.
Additionally, I used it with friends to work on projects together or for university related projects with multiple
other students.</p>
Another flavor of Kanban is Personal Kanban</code>, which as the name suggests, is tailored to be used for your private
projects or day to day tasks 3</a></sup>.
The general ideas I am using for my projects (yes I'm still using this setup today, but not as detailed as shown later)
are:</p>
Visualize Your Work</strong>
:
We use a board to visualize the tasks that are currently in progress, tasks that need to be finished sooner or later
and tasks that are already finished.
I enjoy a full lane of finished tasks, and it can provide you some motivation for not so much interesting tasks!
Additionally, you get what you have to do today with a simple look on your board.</p>
Limit Work-in-Progress</strong>
:
You only have a limit amount of time available, so you have to get as much done with your available time as possible.</p>
There are a few other Kanban rules, which in my opinion are only useful or relevant if you use Kanban in a team and not for
your own time management 1</a></sup>.</p>
My method</a></h2>
The idea is not</strong> to work 60 hours a week, but to use your available time as efficient and effective as possible.
My method is a combination of Kanban and other agile methods.
I tailored these to my needs as a student with limited time and many focus shifts a day, so I heavily
relied on the Pareto principle 4</a></sup>.
So let's talk about the things you need to get started.</p>
Digital Helpers</a></h3>
A digital helper to have your board always a browser tab away or in your pocket can be very beneficial.
Especially if you are working from different places most of the time and not on your desk at home.</p>
Personally I used the tool Trello</a> for this.
There are a lot of other tools, but I started with Trello years ago and never found something that simply worked,
so I was stuck with Trello.
Just a side-note: I also build some tooling around Trello, to improve my (very opinionated) workflows around Trello.
You can find this tool here</a>, but be warned: it is very opinionated
and not polished at all.</p>
The main task for your digital helpers is to keep your focus on the board and make adding or modifying tasks easy.
If the board looks somehow good and inspiring to you, that is even better!</p>
Lanes</a></h3>
I used 10 lanes to manage the tasks I had to do for one week (and a small peak in the near future).
This seems much, but don't worry, we won't need all of these lanes every single day.
To be exact you only need a single lane per day, after you have done some initial planning at the start of the week.</p>
</p>
In the following we will go more in to detail what each of these lanes I used for.</p>
Backlog
: The backlog is used for all tasks that are not yet scheduled.
It can contain long-running tasks like writing a short paper or programming on a larger project.
I prefer to keep track of these tasks by creating sub-tasks that can be scheduled in a week and track
them via lists within these epic tasks.</p>
Next Week
: This lane contains items that must be finished next week.
In my case we had lab preparations to do every 2 weeks, so I wanted to schedule them right away, while still working
on the current task set of the lab.</p>
This Week
: This lane contains tasks that should be finished this week, but are not yet scheduled.</p>
Monday
: As the name suggests these are all tasks that should be finished on Monday.</p>
Tuesday
: As the name suggests these are all tasks that should be finished on Tuesday.</p>
Wednesday
: As the name suggests these are all tasks that should be finished on Wednesday.</p>
Thursday
: As the name suggests these are all tasks that should be finished on Thursday.</p>
Friday
: As the name suggests these are all tasks that should be finished on Friday.</p>
Saturday/Sunday
: As the name suggests these are all tasks that should be finished on the weekend days.
I try not to schedule to many tasks on weekends, as at least at the weekend, you should get some rest and enjoy a not
fully organized day.</p>
Done this week
: This lane will be used to collect all tasks that were finished this week.
This is by far my favorite lane!</p>
The setup provides the first thing we need for the Kanban methods, a visualization of our work.
Besides that this will be your goto point if you are not sure what to do in a free hour between two lectures.</p>
Labels</a></h3>
Labels are very useful to determine in which category a task belongs to.
In my case I simply used to courses of the current semester as labels.</p>
</p>
Additionally, I added the organization</code> label for tasks that don't belong to a specific course,
for example think about the topic of a thesis or organize a meeting with fellow students.</p>
The usage of different colors provides a quick overview about what course or topic has a lot of open tasks.</p>
Planning</a></h2>
In the following we will talk about the major processes needed to keep your board up to date and how to
add new tasks.
Further, I will present some thoughts about how to get the actual work done and if you should do a
retrospective.</p>
Adding new task</a></h3>
For every task you have to complete you should create a new card.
A task should be as small as possible.
So if you have for example an assignment that includes reading multiple papers or book chapters, you should create
an epic to keep track of the whole process, but individual task for e.g. each paper or chapter.
This will ensure that you can finish at least a few tasks a week and stay you motivated (at least that worked for me).</p>
To make your planning easier you should estimate how much time you need to finish a task.
I know estimating is hard, but for some tasks this is rather simple. 5</a></sup>
For example you can easily estimate how much time you need to read a 5 pages paper, as you have done that often in the past.</p>
</p>
Let's have a look at a simple example: We have to read 'Modern Operating Systems' by Tanenbaum until the end of
the semester.
At first, you should create an epic to track the whole book and the progress of reading it.
Then create a new task for each of the chapters and estimate how long will take to read it (e.g. 1 hour).
You can always update your estimates if you think they are wrong in the future!</p>
I prefer to add the needed time before the title, but you could also use labels or any other method that works for you,
to achieve the same.
Important is that it is easily visible how long you estimated the tasks will take.
We will see why in a moment.</p>
Weekly Planning:</a></h3>
Normally I used about 15 minutes on Saturday or Sunday to plan the next week.
This is where the estimations for tasks are used to not get overworked in the first few weeks of the semester.</p>
So let's say I want to work 40 hours a week on university related tasks.
Then count the time you have to use for lectures, let's use 15 hours, and subtract that from your initial work time
a week.
With that we have 25 hours left to fill with assignments or other things we have to do.
Now you can decide if 5 days are enough or if you need some time on the weekends, but let's start with work
days, as the weekend work will start sooner or later.</p>
You now have different options on how to distribute that time through the week.
Personally I prefer to keep it more or less even, so if you have a day with multiple lectures or labs, then don't schedule
additional 5 hours of work to that day!
Try to work with your fixed hours a day first (e.g. 8 hours), before starting to increase it.</p>
Then do a quick prioritization of the tasks in your backlog, to do the things first, that must be finished this week.
I recommend to work on the tasks in order of the earliest submission date, so work first on the tasks you
have to finish first.
Then simply add as many tasks as you have time left on a specific day.</p>
If you couldn't finish a task that was scheduled for this day, simply move it to the next day.
And don't panic! That's quite normal, and you will get a feeling of what you can accomplish in a day or hour.</p>
I know that sounds like a very time intense process, but it is not!
You don't have to do planning on a single day, but can also plan while you are adding a new task that came up after
a lecture or lab.
Therefore, you will only spend 15 minutes or half an hour a week on actual planning and have more time
to execute what you have planned.</p>
Do the work</a></h3>
This is the part where no fancy project management method can help you.
You have to do the task you have planned for that day.
Sometimes these may be boring, but you have scheduled them on this day, because there won't be much time left elsewhere
this week.
Keep the Pareto principle in mind, when you are working on something you don't enjoy at all.</p>
An additional motivation could be the free evening or weekend, after you have finished your tasks of the day.</p>
Day to Day Interactions with your Board</a></h3>
Your board will tell you what to do today.
Sometimes I also added lectures as tasks of that day, to see all my to-dos of today at a single place.</p>
There are just a few rules to keep in mind when working on tasks:</p>

If you have finished a task, then congratulations and simply move it to Done this week</code>.</li>
If you get a new assignment then add it to your general Backlog</code> , Next week</code> or if it is urgent This week</code>.</li>
If you couldn't finish a task on the day it was scheduled on, then think about refining it by splitting it into multiple
tasks or/and move it to another day.</li>
</ol>
Your board is your single point of truth what you have to work on each day.
You don't have to think about which assignment is more important, when you have an hour left between two lectures.
And if there is nothing left to do for today and you still have some time, while staying at campus, why not take some
time to get a coffee/tea and chat with some friends or enjoy a good book?</p>
Retrospective</a></h3>
To be honest I never did a proper retrospective while the semester was still running.
If something went terribly wrong and I didn't manage to get all the tasks done I had scheduled, I shifted priorities
between assignments and invested less time in these (again Pareto).
Sometimes I did some analysis at the end of the semester to calculate how many hours I worked a week, but this was only
the see where all my time went in the last 4 weeks.</p>
Conclusion</a></h2>
And this is the whole method I used to stay on top of my assignments and getting tasks finished in time.
Now in retrospective I think it sounds like much more work than it actually was.
I did experiment with other methods (for example the Bullet Journal method), but to be honest nothing felt as easy
as the earlier presented process 6</a></sup>.
If you have any feedback or experience with time management while being a student or afterwards, don't hesitate to let
me know!
I'm happy to hear from others on how they manage their very limited time and get the most out of it.</p>
Sources</a></h2>



https://kanbanize.com/kanban-resources/getting-started/what-is-kanban ↩</a> ↩2</a> ↩3</a></p>
</li>

https://www.planview.com/resources/guide/introduction-to-kanban/ ↩</a></p>
</li>

https://www.personalkanban.com/ ↩</a></p>
</li>

https://en.wikipedia.org/wiki/Pareto_principle ↩</a></p>
</li>

https://uwaterloo.ca/ist-project-management-office/methodology/project-management/planning/project-schedule/task-estimating ↩</a></p>
</li>

https://bulletjournal.com/ ↩</a></p>
</li>
</ol>
</section>


Speaking at OpenRheinMain 2022
2022-09-30T00:00:00+00:00
I had the chance to speak at a small conference at my home university about the project
(goSDN</a>) I did work on the last 6 months.
This is also the project I worked on while writing my thesis, therefore I was pretty confident I had enough insides to
answer most of the questions.</p>
I thought about speaking at a conference before, but most of the time I felt like I couldn't provide much insight
in topics or my understanding of the technology I used simply wasn't deep enough.
But this conference was the perfect opportunity to change this!
The conference was rather small with about ~200 attendees, and it was at my home university, so I had good chances
of getting accepted as speaker.
Which actually happened, so I had a few weeks to prepare the presentation.</p>
The presentation itself went quite well I think.
I got some interesting questions afterwards and some companies got interested in
what we are building, so hopefully the research group can get more funding.</p>
This was my first experience as a speaker at a conference and I really enjoyed it.
Hopefully I will be able to speak at a conference again in the future.</p>


Build LaTeX with GitLab-CI
2022-08-21T00:00:00+00:00
LaTeX is a software for document preparation.
You write your content in plain text .tex</code> files, much like a very basic programming language, and compile it
afterwards in the designated format, e.g. pdf</code>.
LaTeX is heavily used in academia as it provides an easy way to express mathematical formulae.
Besides that most universities provide a template, so you don't have to think about margins, citation style (and more) and
can simply focus on your content.
And this is exactly the reason why I use LaTeX.</p>
Overview</a></h2>
As a computer science student I had to write lab reports and finally a final thesis.
Most of these were written with LaTeX, as it is easy to add formulas, and it looks quite nice.
It is also very convenient if you get a template, so you can focus on your content instead of
optics of your document.
As a developer it is also easy to use your favorite version management tool, as TeX files are simply text file.
But it can be annoying to set up LaTeX on your local machine, therefore I tend to use the same container
on my local machine as on a CI server, to ensure I can always build my document with the same version of all
needed libraries.</p>
Template</a></h2>
As I use this setup to have reproducible build of my thesis, I rely on a template provided by my university.
If you are interested in the template you can find it here</a>.
The template also provides a simple Makefile</code>, which I'll use later.
Besides styling the template also provided Dockerfile, but these are based on Ubuntu and I prefer to use
the same versions of tools than on my local machine, so I will stick with Arch Linux for the containers as well.</p>
The build container</a></h2>
To ensure I have the latest release of all used libraries, I use Arch Linux as base layer.
It's also handy that Arch provides all needed LaTeX libraries in the distribution repositories.</p>
FROM</span> archlinux:latest</span></span>
</span>
RUN</span> pacman -Syu --noconfirm texlive-most texlive-latexextra texlive-bibtexextra make ghostscript biber</span></span></code></pre>
After building the image with
docker build -t gitlab.com/andre/masterarbeit/latex-build:ci -f Dockerfile .</code>
it must be pushed to the GitLab container registry with
docker push gitlab.com/andre/masterarbeit/latex-build:ci</code>.
These are quite big, so be ready to wait a few minutes, depending on your internet speed.</p>
GitLab CI</a></h2>
This setup requires that you have already set up a runner for your GitLab project and can use the
internal container registry.</p>
With the artifacts</code> setting, GitLab will expose the created PDF directly from the overview page of your run.
This is useful if you are working on a machine without TeX installed, but still want to see if you can build
your PDF after some changes.
The setting expire_in: 4 week</code> ensures that I don't keep all build versions forever on the server.
make all</code> in the script</code> section refers to the earlier mentioned Makefile provided by the template, you can find it
here</a>.</p>
c</span>ompile-thesis</span>:</span></span>
  i</span>mage</span>:</span> g</span>itlab.com/andre/masterarbeit/latex-build:ci</span></span>
  s</span>cript</span>:</span></span>
    -</span> m</span>ake all</span></span>
  a</span>rtifacts</span>:</span></span>
    p</span>aths</span>:</span></span>
      -</span> t</span>hesis.pdf</span></span>
    e</span>xpire_in</span>:</span> 4</span> week</span></span></code></pre>Local Build</a></h2>
To ensure I get the same results locally, I also use the earlier build container for local builds.
The following script will simply mount the current directory in the container and execute make all</code>.
It also deletes the container afterwards.</p>
#!</span>/bin/sh</span></span>
</span>
docker</span> run</span> -</span>-name</span> paper-build</span> -</span>-user</span> $(</span>id</span> -</span>u</span>)</span>:</span>$(</span>id</span> -</span>g</span>)</span> -</span>-volume</span> $(</span>pwd</span>)</span>:/thesis-template/</span> gitlab.com/andre/masterarbeit/latex-build:ci</span> make</span> -</span>-directory</span> /thesis-template/</span> all</span></span>
docker</span> rm</span> paper-build</span></span></code></pre>Conclusion</a></h2>
The setup is rather simple, but I don't want to have build problems on my local machine, with the day of delivery in mind.
Also, I can give prove readers access to the artifacts, so they always read the latest version.</p>


Uptime monitoring with Prometheus and Grafana
2022-07-27T00:00:00+00:00
There are existing services like pingdom</a> or uptimerobot</a> to
check if your network services are up and running.
But as always I like to be independent of service providers and host as much by myself as possible.
So let's build our own uptime monitoring with Prometheus</a>, Grafana</a>,
alertmanager</a> and the
Prometheus blackbox-exporter</a>.</p>
The general idea is to check if a service is up with the blackbox-exporter.
Then scrape this metrics with Prometheus and store it for further usage.
Build a simple dashboard with Grafana for visualization and finally setup alerting with alertmanager.
As this uses a few projects on the Prometheus ecosystem, this is a perfect example on how to integreate these
different components.</p>
The end result will look like this:
</p>
What is uptime monitoring?</a></h2>
In this example we will monitor http</code> services (in this case my blog) and a simple TCP</code> service (e.g. a teamspeak3 server).
So what does uptime mean?
In this case it means the service should be accessible via HTTP</code> or TCP</code> and therefore the user should be able to
use it.</p>
There are more advanced services like the already mentioned pingdom</a> where you can also check
for specific parts on a HTTP</code> page, but let's go for a basic setup which will simply check if we get any http 2xx</code> response
or can create a TCP connection to the service.</p>
Another interesting dashboard or metric is the actual time since the last downtime of the service.</p>
Setup Prometheus/Exporter/Alertmanager</a></h2>
So let's get started with setting up all needed components.
The directories should look like this, to be able to provide all services with the needed configurations:</p>
.</span></span>
├── alertmanager</span></span>
│   └── config.yml</span></span>
├── blackbox-exporter</span></span>
│   └── blackbox.yml</span></span>
├── docker-compose.yml</span></span>
├── grafana</span></span>
│   ├── config.monitoring.example</span></span>
│   └── provisioning</span></span>
│       ├── dashboards</span></span>
│       │   └── dashboard.yml</span></span>
│       └── datasources</span></span>
│           └── datasource.yml</span></span>
├── prometheus</span></span>
│   ├── prometheus.yml</span></span>
│   └── uptime.rules</span></span></code></pre>
The whole setup can be found in my GitHub repository</a>.
But let's take a look into the most important configurations.</p>
The blackbox-exporter</a> is configured via a simple blackbox.yml</code>.
In this case I configured a basic HTTP</code> probe with the http_2xx</code> module and the tcp_connect</code> module.
There are more modules</a>
for e.g. icmp</code>, dns</code> and grpc</code>, which can be used to monitor services with other protocols.</p>
m</span>odules</span>:</span></span>
  h</span>ttp_2xx</span>:</span></span>
    p</span>rober</span>:</span> h</span>ttp</span></span>
    t</span>imeout</span>:</span> 5</span>s</span></span>
    h</span>ttp</span>:</span></span>
      t</span>ls_config</span>:</span></span>
        i</span>nsecure_skip_verify</span>:</span> false</span></span>
  t</span>cp_connect</span>:</span></span>
    p</span>rober</span>:</span> t</span>cp</span></span>
    t</span>imeout</span>:</span> 5</span>s</span></span>
    t</span>cp</span>:</span></span>
      p</span>referred_ip_protocol</span>:</span> i</span>p4</span></span></code></pre>
The next component to configure is Alertmanager, which will send alerts to a single or multiple recievers.
In this case I want to be alerted via mail, but you can choose whatever reciever</a>
is the perfect match for you.</p>
r</span>oute</span>:</span></span>
  r</span>eceiver</span>:</span> '</span>default-receiver</span>'</span></span>
  g</span>roup_wait</span>:</span> 3</span>0s</span></span>
  g</span>roup_interval</span>:</span> 5</span>m</span></span>
  r</span>epeat_interval</span>:</span> 1</span>2h</span></span>
</span>
r</span>eceivers</span>:</span></span>
  -</span> n</span>ame</span>:</span> '</span>default-receiver</span>'</span></span>
    e</span>mail_configs</span>:</span></span>
        -</span> s</span>end_resolved</span>:</span> true</span></span>
          t</span>o</span>:</span> '</span>'</span></span>
          f</span>rom</span>:</span> '</span>monitoring@mydomain.com</span>'</span></span>
          s</span>marthost</span>:</span> '</span>:587</span>'</span></span>
          a</span>uth_username</span>:</span> '</span>hello</span>'</span></span>
          a</span>uth_password</span>:</span> '</span>changeme</span>'</span></span></code></pre>
Now to the target configuration within the Prometheus config.
Most of this config is more or less a plain Prometheus config.
There are two additions:</p>

Additional uptime.rules</code> with rules for the case of a service downtime.</li>
A scrape_config</code> for the blackbox-exporter.</li>
</ol>
You can add as many targets as you want.
The exporter will probe them, and you can query them with probe_success</code> via PromQL</a>.</p>
g</span>lobal</span>:</span></span>
    s</span>crape_interval</span>:</span> 1</span>5s</span></span>
    e</span>valuation_interval</span>:</span> 1</span>5s</span></span>
</span>
    e</span>xternal_labels</span>:</span></span>
        m</span>onitor</span>:</span> "</span>my-domain</span>"</span></span>
</span>
r</span>ule_files</span>:</span></span>
    -</span> "</span>uptime.rules</span>"</span></span>
</span>
a</span>lerting</span>:</span></span>
    a</span>lertmanagers</span>:</span></span>
        -</span> s</span>cheme</span>:</span> h</span>ttp</span></span>
          s</span>tatic_configs</span>:</span></span>
              -</span> t</span>argets</span>:</span></span>
                    -</span> "</span>alertmanager:9093</span>"</span></span>
</span>
s</span>crape_configs</span>:</span></span>
    -</span> j</span>ob_name</span>:</span> '</span>blackbox-http</span>'</span></span>
      m</span>etrics_path</span>:</span> /</span>probe</span></span>
      p</span>arams</span>:</span></span>
        m</span>odule</span>:</span> [</span>h</span>ttp_2xx</span>]</span></span>
      s</span>tatic_configs</span>:</span></span>
        -</span> t</span>argets</span>:</span></span>
          -</span> h</span>ttps://sterba.dev</span></span>
      r</span>elabel_configs</span>:</span></span>
        -</span> s</span>ource_labels</span>:</span> [</span>_</span>_address__</span>]</span></span>
          t</span>arget_label</span>:</span> _</span>_param_target</span></span>
        -</span> s</span>ource_labels</span>:</span> [</span>_</span>_param_target</span>]</span></span>
          t</span>arget_label</span>:</span> i</span>nstance</span></span>
        -</span> t</span>arget_label</span>:</span> _</span>_address__</span></span>
          r</span>eplacement</span>:</span> b</span>lackbox-exporter:9115</span></span></code></pre>
This uptime rule will alert you when a probe failed for 5 minutes.</p>
g</span>roups</span>:</span></span>
-</span> n</span>ame</span>:</span> u</span>ptime-rules</span></span>
  r</span>ules</span>:</span></span>
      -</span> a</span>lert</span>:</span> s</span>ite_down</span></span>
        e</span>xpr</span>:</span> p</span>robe_success == 0</span></span>
        f</span>or</span>:</span> 5</span>m</span></span>
        l</span>abels</span>:</span></span>
          s</span>everity</span>:</span> w</span>arning</span></span>
        a</span>nnotations</span>:</span></span>
          d</span>escription</span>:</span> '</span>{{ $labels.instance }} of job {{ $labels.job }} has</span></span>
                        been down for more than five minutes.</span>'</span></span>
          s</span>ummary</span>:</span> I</span>nstance {{ $labels.instance }} is down</span></span></code></pre>Create the Grafana dashboard</a></h2>
At first create a new dashboard and add a new variable called instance</code>.
We will use this variable to make the dashboard interactive and refresh all graphs based on the select instance.
</p>
As displayed earlier the dashboard contains two visualizations of the blackbox-exporter.
The first one is a single value that can be 0</code> of the service is down or 1</code> if the service is up.
We use a singlestat</code> to display it within Grafana.</p>

For the singlestat</code> we use the previously defined variable to dynamically update the statistics.</p>
probe_success</span>{</span>i</span>n</span>s</span>t</span>a</span>n</span>c</span>e</span>=</span>"</span>$</span>instance</span>"</span>,</span> job=</span>"</span>blackbox-http</span>"</span>}</span></span></code></pre>
</p>
The second visualization is a graph of the actual probe duration.
You can use it to determine if a service is responding slowly.
Besides that it might be a good idea to also create an alert based on the probe duration.</p>
probe_duration_seconds</span>{</span>i</span>n</span>s</span>t</span>a</span>n</span>c</span>e</span>=</span>"</span>$</span>instance</span>"</span>,</span> job=</span>"</span>blackbox-http</span>"</span>}</span></span></code></pre>Deploy with docker compose</a></h2>
As always this setup will be deployed via containers.
The following docker-compose.yaml</code> provides all needed services and their configurations.</p>
v</span>ersion</span>:</span> "</span>3</span>"</span></span>
</span>
v</span>olumes</span>:</span></span>
    p</span>rometheus_data</span>:</span> {</span>}</span></span>
    g</span>rafana_data</span>:</span> {</span>}</span></span>
</span>
s</span>ervices</span>:</span></span>
  p</span>rometheus</span>:</span></span>
    i</span>mage</span>:</span> p</span>rom/prometheus:v2.37.0@sha256:56e7f18e05dd567f96c05046519760b356f52450c33f6e0055a110a493a41dc4</span></span>
    v</span>olumes</span>:</span></span>
      -</span> .</span>/prometheus/:/etc/prometheus/</span></span>
      -</span> p</span>rometheus_data:/prometheus</span></span>
    c</span>ommand</span>:</span></span>
      -</span> '</span>--config.file=/etc/prometheus/prometheus.yml</span>'</span></span>
      -</span> '</span>--storage.tsdb.path=/prometheus</span>'</span></span>
      -</span> '</span>--web.console.libraries=/usr/share/prometheus/console_libraries</span>'</span></span>
      -</span> '</span>--web.console.templates=/usr/share/prometheus/consoles</span>'</span></span>
    p</span>orts</span>:</span></span>
      -</span> 1</span>27.0.0.1:9090:9090</span></span>
    l</span>inks</span>:</span></span>
      -</span> a</span>lertmanager:alertmanager</span></span>
    r</span>estart</span>:</span> a</span>lways</span></span>
</span>
  a</span>lertmanager</span>:</span></span>
    i</span>mage</span>:</span> p</span>rom/alertmanager:v0.24.0@sha256:088464f949de8065b9da7dfce7302a633d700e9d598e2bebc03310712f083b31</span></span>
    p</span>orts</span>:</span></span>
      -</span> 1</span>27.0.0.1:9093:9093</span></span>
    v</span>olumes</span>:</span></span>
      -</span> .</span>/alertmanager/:/etc/alertmanager/</span></span>
    r</span>estart</span>:</span> a</span>lways</span></span>
    c</span>ommand</span>:</span></span>
      -</span> '</span>--config.file=/etc/alertmanager/config.yml</span>'</span></span>
      -</span> '</span>--storage.path=/alertmanager</span>'</span></span>
</span>
  g</span>rafana</span>:</span></span>
    i</span>mage</span>:</span> g</span>rafana/grafana:9.0.5@sha256:6847695bc353a659a18c6f44bd0b5a94de5d2eeff995a671db6e982370dfe4e8</span></span>
    d</span>epends_on</span>:</span></span>
      -</span> p</span>rometheus</span></span>
    p</span>orts</span>:</span></span>
      -</span> 1</span>27.0.0.1:3000:3000</span></span>
    v</span>olumes</span>:</span></span>
      -</span> g</span>rafana_data:/var/lib/grafana</span></span>
      -</span> .</span>/grafana/provisioning/:/etc/grafana/provisioning/</span></span>
    e</span>nv_file</span>:</span></span>
      -</span> .</span>/grafana/config.monitoring</span></span>
    r</span>estart</span>:</span> a</span>lways</span></span>
</span>
  b</span>lackbox-exporter</span>:</span></span>
    p</span>rivileged</span>:</span> true</span></span>
    i</span>mage</span>:</span> p</span>rom/blackbox-exporter:v0.21.1@sha256:7a255982b24ed3d142f78075a7d8af976355b6f62cec60c573ee115c1beb1d0c</span></span>
    r</span>estart</span>:</span> a</span>lways</span></span>
    c</span>ommand</span>:</span></span>
        '</span>--config.file=/config/blackbox.yml</span>'</span></span>
    p</span>orts</span>:</span></span>
        -</span> '</span>127.0.0.1:9115:9115</span>'</span></span>
    v</span>olumes</span>:</span></span>
        -</span> '</span>./blackbox-exporter/blackbox.yml:/config/blackbox.yml</span>'</span></span>
</span></code></pre>Conclusion</a></h2>
And that's it!
This is a way to monitor most of my services, as I don't care that much about application metrics
and only want to be disturbed if something is really</strong> not working (like the service is down).
As always I enjoy hosting services I use myself and reuse services I'm already running.
(I also use this stack to get system metrics which I don't actually use for anything but fancy graphs).</p>
This way of blackbox monitoring is a good fit if you simply want to check if a service can be reached from the outside.
But keep in mind, that you should not host your monitoring stack on the same host as your actual user facing services!</p>
If you want to be sure that your services are reachable, you could use multiple sites for uptime checks.
Or use different providers like the earlier mentioned pingdom</a> or uptimerobot</a>.</p>
You can find all docker/config files at GitHub</a>.</p>


GitOps with upstream-watch
2022-06-23T00:00:00+00:00
I'm a fan of automating my personal infrastructure as much as possible.
Therefore, I wrote a little utility called upstream-watch</a>, that is well
integrated in my existing git-ops workflow.
I wrote this tool to support my personal container based infrastructure, which is completely managed via a single git
repository.</p>
What is GitOps?</a></h2>

GitOps upholds the principle that Git is the one and only source of truth.
GitOps requires the desired state of the system to be stored in version control such that anyone can view the entire audit trail of changes.
All changes to the desired state are fully traceable commits associated with committer information, commit IDs and time stamps.
This means that both the application and the infrastructure are now versioned artifacts and can be audited using the gold standards of software development and delivery.</p>
-- 1</a></sup></cite></p>
</blockquote>
So all needed configurations for a service should be managed in a git repository.
And if changes are necessary these changes must be commited to the repository and will then be carried out
to the actual server.
This will ensure two things:
First you also have a clear state of your infrastructure and can roll back to a working version (if there were no database
migrations executed, but we do have backups for this case, right?)
Secondly you can set up the infrastracture on a new machine in minutes, rather than hours or days.</p>
In general, I would also add that GitOps enables CI and CD for your applications/services.
And you also can apply software engineering best practices like reviews.
So there is no good reason not to use some basic form of GitOps.</p>
Features</a></h2>
Short overview about the features of upstream-watch</code>:</p>

regulary pull upstream repository</li>
check if something changed</li>
execute configured hooks and update routine, if update is necessary</li>
keep track if an update was already executed, to don't deacrease service downtime by executing multiple updates</li>
hooks and update process are completely scriptable</li>
</ul>
Overview</a></h2>
upstream-watch</code> will continously pull an upstream git</code> repository and check if there are changes on the origin and
wil pull these to the local repository.
If yes, it will execute pre-configured hooks before and after updating the service.
This will ensure upstream-watch</code> is compatible with almost all services and a useful tool
in an operators toolbelt.</p>
The layout of my infrastructure as code repository looks like this:</p>
    .</span></span>
    ├──</span> .upstream-watch.yaml</span></span>
    ├──</span> README.md</span></span>
    ├──</span> service-1</span></span>
    │  </span> ├──</span> .update-hooks.yaml</span></span>
    │  </span> ├──</span> docker-compose.yml</span></span>
    │  </span> └──</span> README.md</span></span>
    ├──</span> service-2</span></span>
    │  </span> ├──</span> .update-hooks.yaml</span></span>
    │  </span> ├──</span> docker-compose.yml</span></span>
    │  </span> └──</span> README.md</span></span>
    └──</span> upstream-watch</span></span></code></pre>Main configuration</a></h2>
The .upstream-watch.yaml</code> is the main configuration file for this instance of upstream-watch</code>.
You can set the retry interval (in seconds) and folders that should be ignored.
The single_directory_mode: false</code> disables this mode and enables the presented structure with subfolders per service.</p>
    single_directory_mode:</span> false</span></span>
    retry_interval:</span> 60</span></span>
    ignore_folders:</span> [</span>"</span>.git</span>"</span>,</span> "</span>.test</span>"</span>]</span></span></code></pre>Service configuration</a></h2>
In case of an update to any of these files in a subfolder, upstream-watch</code> will execute the pre- and post-hooks
defined in the corresponding .update-hooks.yaml</code>, before and after the service is updated.
In this case all services are using docker compose</code>.</p>
An example for a .update-hooks.yaml</code>:</p>
    pre_update_commands:</span>  [</span>"</span>docker compose down</span>"</span>]</span></span>
    update_commands:</span>      [</span>"</span>docker compose pull</span>"</span>]</span></span>
    post_update_commands:</span> [</span>"</span>docker compose up -d</span>"</span>]</span></span></code></pre>
This provides much needed flexibilty to be able to use upstream-watch</code> for almost everything, not only docker compose</code>
or any other container mangement tool.</p>
Update process</a></h2>
This results in the following update process for our example service with integration of renovate or dependabot:</p>

Dependabot or Renovate opens an MR/PR with an update.</li>
Review change and merge to main</code>.</li>
upstream-watch</code> will pull the changes.</li>
and execute the configured hooks and update routine.</li>
</ol>
upstream-watch</code> will track if a service was already updated by storing the commit of the submodule and its update
status in a local sqlite database.</p>
Some internals</a></h2>
upstream-watch</code> relies on git</code> to determine if a subfolder (with a service in it) was changed between commits.
To persist if an update was already executed it relies on in a simple sqlite database, that is stored in the
repository.</p>
The used sqlite libary needs CGO to be enabled, therefore there are no macOS or Windows builds right now.
I still have to figure out how to build these on different GitHub Action VMs (with goreleaser</code>).</p>
A major thing I plan to add soon is integration of webhooks, to get an alert, if something went wrong.
With that, there is theoretically no need to login on the server, after the inital service deployment.
Another usefull feature would be to be able to deploy upstream-watch</code> as container aside the exisiting services.
For some deployments that could mean, that the upstream-watch</code> container needs access to the docker-socket, but
this is solvable (e.g. watchtower</code> does excatly that).</p>
Conclusion</a></h2>
It was fun writing a little utility that I can directly use for my production services.
If you need something like this, go and checkout upstream-watch</code> at github.com/andresterba/upstream-watch</a>.</p>



https://www.cloudbees.com/gitops/what-is-gitops ↩</a></p>
</li>
</ol>
</section>


Go Generics
2022-05-01T00:00:00+00:00
The latest release of Go (1.18) added generics to the language, besides fuzzing, workspaces and performance
improvements. 1</a></sup>
So let's use them to implement the repository pattern, which is a way to abstract data handling from your business logic.
When working on REST-like APIs you want to split handling of data from and into some kind of
data store (e.g. a database, a file, ...) from business logic.
This is where the repository pattern comes in handy.
Especially with Go 1.18 and the introduction of generics, this is a good example for its usage
to don't have to write (more or less) the same code again and again.</p>
There is a more formal and longer definition, that you can check out. 2</a></sup></p>
Overview</a></h2>
In this example we will create a generics repository and store some objects in these.
We will use simple user</code> and pet</code> structs, without any complex internals.
A user looks like this:</p>
    type</span> User</span> struct</span> {</span></span>
    	id</span>      uuid</span>.</span>UUID</span></span>
    	Name</span>    string</span></span>
    	Hobbies</span> [</span>]</span>string</span></span>
    }</span></span></code></pre>
The user must also implement the user interface, as we want to use the ID()</code> method later in our generic
repository.</p>
    type</span> User</span> interface</span> {</span></span>
    	ID</span>(</span>)</span> uuid</span>.</span>UUID</span></span>
    }</span></span></code></pre>Introduction to generics</a></h2>
Generics look more or less like in any other language.
We will cover generic structs (and methods), generic functions, their instantiation and usage.
I also don't care about error handling in all these examples, I will simply panic if anything went wrong.</p>
Generic structs</a></h3>
This is a generic struct of a repository:</p>
    type</span> GenericRepository</span>[</span>T</span> storableConstraint</span>]</span> struct</span> {</span></span>
    	Store</span> map</span>[</span>uuid</span>.</span>UUID</span>]</span>T</span></span>
    }</span></span></code></pre>
The repository will store added objects in a map with a uuid.UUID</code> key and a value T</code>, which will
be the actual type of the struct the repository will be created for.</p>
A notable thing is the storableConstraint</code>.
A constraint can define methods that a struct or interface used to construct this generic repository
must provide.
In this case the storableConstraint</code> looks like this:</p>
    type</span> storableConstraint</span> interface</span> {</span></span>
    	ID</span>(</span>)</span> uuid</span>.</span>UUID</span></span>
    }</span></span></code></pre>
This means everything I want to store in a generic repository must implement a ID()</code> method that returns
a uuid.UUID</code> attribute.</p>
Let's have a look on a method on this repository, e.g. the Add</code> method.</p>
    func</span> (</span>t</span> *</span>GenericRepository</span>[</span>T</span>]</span>)</span> Add</span>(</span>item</span> T</span>)</span> error</span> {</span></span>
    	_</span>,</span> ok</span> :=</span> t</span>.</span>Store</span>[</span>item</span>.</span>ID</span>(</span>)</span>]</span></span>
    	if</span> ok</span> {</span></span>
    		return</span> nil</span></span>
    	}</span></span>
</span>
    	t</span>.</span>Store</span>[</span>item</span>.</span>ID</span>(</span>)</span>]</span> =</span> item</span></span>
</span>
    	return</span> nil</span></span>
    }</span></span></code></pre>
As our storableConstraint</code> ensures that every object has a ID()</code> method, we can us it to
get the ID of this object.
The ID will then be used as the unique identifier in our internal store.</p>
Generic functions</a></h3>
Besides generic structs and methods on these structs, we can also create generic functions.
So lets say we want to transform one object into another one, we provide object T</code> as input, but
want to get object U</code> as return value.</p>
Another new thing with Go 1.18 is the any</code> keyword.
any</code> replaces interface{}</code> if you don't have a type for something.
In this case we use any</code> as the constraint, which means that every interface or struct is allowed to be transformed.
Which is naive to use in real world application, but will suffice for this example.</p>
We can create a simple function that will look like this:</p>
    func</span> transform</span>[</span>T</span>,</span>U</span> any</span>]</span>(</span>incoming</span> T</span>)</span> (</span>U</span>,</span> error</span>)</span> {</span></span>
        ...</span></span>
    }</span></span></code></pre>
For a single generic value, the type can mostly be inferred and doesn't need to be provided while calling the function.
In this case we have to provide it, so the function call looks like this:</p>
    pet</span>,</span> err</span> :=</span> transform</span>[</span>User</span>,</span> Pet</span>]</span>(</span>user</span>)</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        panic</span>(</span>err</span>)</span></span>
    }</span></span></code></pre>Instantiation</a></h2>
So let's instantiate our generic repository, create a user and add it to the repository.
In this case we will instantiate the repository with the user interface instead of the user struct.
Please mind that the NewUser()</code> function does return a interfaces.User</code> instead of a User</code>, so we ensure that a User</code>
actually implements the interface.</p>
	userRepository</span> :=</span> NewGenericRepository</span>[</span>interfaces</span>.</span>User</span>]</span>(</span>)</span></span>
	user1</span> :=</span> NewUser</span>(</span>"</span>Peter</span>"</span>,</span> [</span>]</span>string</span>{</span>"</span>1</span>"</span>,</span> "</span>2</span>"</span>}</span>)</span></span>
</span>
	err</span> :=</span> userRepository</span>.</span>Add</span>(</span>user1</span>)</span></span>
	if</span> err</span> !=</span> nil</span> {</span></span>
		panic</span>(</span>err</span>)</span></span>
	}</span></span>
</span>
	userFromRepo</span>,</span> err</span> :=</span> userRepository</span>.</span>Get</span>(</span>user1</span>.</span>ID</span>(</span>)</span>)</span></span>
	if</span> err</span> !=</span> nil</span> {</span></span>
		panic</span>(</span>err</span>)</span></span>
	}</span></span>
</span>
	fmt</span>.</span>Println</span>(</span>userFromRepo</span>)</span></span></code></pre>
We then create a new user and add it to the created repository.
Then the repository can be queried with the user id, and we get our earlier added user back.</p>
As the repository is a generic we can simply add another type, that fulfills our earlier defined constraint.</p>
	petRepository</span> :=</span> NewGenericRepository</span>[</span>interfaces</span>.</span>Pet</span>]</span>(</span>)</span></span>
    petOfPeter</span> :=</span> NewPet</span>(</span>"</span>Peter's Cat</span>"</span>,</span> user1</span>.</span>ID</span>(</span>)</span>)</span></span>
</span>
	err</span> =</span> petRepository</span>.</span>Add</span>(</span>petOfPeter</span>)</span></span>
	if</span> err</span> !=</span> nil</span> {</span></span>
		panic</span>(</span>err</span>)</span></span>
	}</span></span>
</span>
	petFromRepo</span>,</span> err</span> :=</span> petRepository</span>.</span>Get</span>(</span>petOfPeter</span>.</span>ID</span>(</span>)</span>)</span></span>
	if</span> err</span> !=</span> nil</span> {</span></span>
		panic</span>(</span>err</span>)</span></span>
	}</span></span>
</span>
	fmt</span>.</span>Println</span>(</span>petFromRepo</span>)</span></span></code></pre>
So as you can see, this repository can be used to store any object that implements a ID()</code> method.
This is also very handy if you want to mock a repository used in tests.</p>
The whole example can be found in my code repository 3</a></sup>.</p>
Conclusion</a></h2>
To be honest I was a bit skeptic about the addition of generics to go, as I thought with existing and good
integrated tooling for code generation go didn't need them.
But I was clearly wrong!
The generics implementation and usage feels very simple and therefore go-like, and I really enjoy using them.
Especially when writing simple APIs or mocks for some tests, generics come in very handy and can save quite some time.</p>
If you are more interested in the internals, I can recommend this excellent post from Vicent Marti.
It will show how go implements generics, but also how generics can make your code run slower. 4</a></sup></p>



https://go.dev/blog/go1.18 ↩</a></p>
</li>

https://deviq.com/design-patterns/repository-pattern ↩</a></p>
</li>

https://github.com/andresterba/code-for-blog ↩</a></p>
</li>

https://planetscale.com/blog/generics-can-make-your-go-code-slower ↩</a></p>
</li>
</ol>
</section>


Install Arch Linux
2022-04-01T00:00:00+00:00
This will be a reminder for myself on how to set up a basic Arch Linux and not</strong> a guide with a lot of
variety on how to set up arch.
I will update this post when I learn something new, while installing arch on one of my machines again.
In the past I did some experiments with btrfs</code>, but had some problems while rebalancing, and actually used
none of the fancy features like snapshots, btrfs send</code> or other cool things.
Therefore, I decided to go back to plain old ext4 with lvm. 1</a></sup>
Well, I actually used local snapshots bet never had to recover anything.
A lot of this is based on the excellent (german) post from Thomas Leister 2</a></sup>.
Everything is (as always) documented in the arch wiki. 3</a></sup></p>
Overview</a></h2>
Just a quick overview about the features before we get started.</p>

ext4 Filesystem with lvm</li>
encrypted root partition next to unencrypted /boot</code></li>
EFI-Boot with systemd</li>
</ul>
Steps</a></h2>
1. Setup partitions</a></h3>
The disk Layout will look like this:</p>
</p>
So let's get started with the partitioning.
At first we will create the boot partition.
I will use /dev/sda</code> as a placeholder, this could be anything on your local machine.</p>
    1</span> =</span> boot</span> partition:</span></span>
    gdisk</span> /dev/sda</span></span>
    o</span></span>
    n</span></span>
    enter</span></span>
    enter</span></span>
    +512M</span></span>
    ef00</span></span></code></pre>
After the boot partition is set up, we will use the rest of the space for our data partition.</p>
    2</span> =</span> data:</span></span>
    n</span></span>
    2</span></span>
    enter</span></span>
    enter</span></span>
    enter</span></span></code></pre>
Write the changes to disk with w</code>.
Now we are ready for the next step, the encryption.</p>
Setup LUKS encryption</a></h3>
Now to the disk encryption, dev/sda2</code> is the earlier created data partition.</p>
cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 --use-random luksFormat /dev/sda2</code></p>
LVM and format partitions</a></h3>
    #</span> Open the earlier created encrypted disk partition</span></span>
    cryptsetup</span> luksOpen</span> /dev/sda2</span> lvm</span></span>
</span>
    pvcreate</span> /dev/mapper/lvm</span></span>
</span>
    vgcreate</span> main</span> /dev/mapper/lvm</span></span>
</span>
    #</span> Adjust to the actual memory size</span></span>
    lvcreate</span> -</span>L</span> 16G</span> -</span>n</span> swap</span> main</span></span>
</span>
    lvcreate</span> -</span>l</span> 100%FREE</span> -</span>n</span> root</span> main</span></span>
</span>
    mkswap</span> /dev/mapper/main-swap</span></span>
</span>
    #</span> You can use whatever you want instead of ext4</span></span>
    mkfs.ext4</span> /dev/mapper/main-root</span></span>
</span>
    #</span> The ESP must be formated as vfat</span></span>
    mkfs.vfat</span> /dev/sda1</span></span></code></pre>Mount disks</a></h3>
    mount</span> /dev/mapper/main-root</span> /mnt</span></span>
    mkdir</span> /mnt/boot</span></span>
    mount</span> /dev/sda1</span> /mnt/boot</span></span>
    swapon</span> /dev/mapper/main-swap</span></span></code></pre>Get network access</a></h3>
If you have an ethernet jack on your device this should work out of the box.</p>
Get the fastest local mirror</a></h3>
Get the fastest local mirror to speed up the installation of the base system.</p>
    reflector</span> -</span>-latest</span> 10</span> -</span>-country</span> Germany</span> -</span>-sort</span> rate</span> -</span>-save</span> /etc/pacman.d/mirrorlist</span></span></code></pre>Install base system</a></h3>
Don't forget to install dhcpcd</code> as it was removed from base</code>.</p>
    pacstrap</span> /mnt</span> base</span> linux</span> linux-firmware</span> lvm2</span> neovim</span> dhcpcd</span></span>
    genfstab</span> -</span>U</span> /mnt</span> >></span> /mnt/etc/fstab</span></span>
</span>
    #</span> Chroot in the installation</span></span>
    arch-chroot</span> /mnt</span></span>
</span>
    #</span> Set timezone info</span></span>
    ln</span> -</span>sf</span> /usr/share/zoneinfo/Region/City</span> /etc/localtime</span></span>
</span>
    Edit</span> /etc/locale.gen</span> and</span> uncomment</span> en_US.UTF-8</span> UTF-8</span> and</span> other</span> needed</span> locales.</span></span>
    Generate</span> the</span> locales</span> by</span> running:</span> locale-gen</span></span>
</span>
    /etc/locale.conf</span></span>
    LANG</span>=</span>e</span>n</span>_</span>U</span>S</span>.</span>U</span>T</span>F</span>-</span>8</span></span>
</span>
    #</span> Set hostname</span></span>
    nvim</span> /etc/hostname</span></span>
</span>
    #</span> Change root password</span></span>
    passwd</span></span></code></pre>Boot</a></h3>
Edit /etc/mkinitcpio.conf</code></p>
    HOOKS</span>=</span>"</span>base udev autodetect modconf block keyboard keymap encrypt lvm2 filesystems fsck</span>"</span></span></code></pre>
Run mkinitcpio -p linux</code></p>
    bootctl</span> -</span>-path=/boot</span> install</span></span>
</span>
    /boot/loader/loader.conf</span></span>
    default</span> arch</span></span></code></pre>
/boot/loader/entries/arch.conf</p>
    title    Arch Linux</span></span>
    linux    /vmlinuz-linux</span></span>
    initrd   /initramfs-linux.img</span></span>
    options  cryptdevice=/dev/sda2:main:allow-discards root=/dev/mapper/main-root resume=/dev/mapper/main-swap</span></span></code></pre>
:allow-discards</code> to enable TRIM support.</p>
Run systemctl enable fstrim.timer</code> to enable fstrim</code>.</p>
Reboot</a></h3>

Exit chroot with exit</code></li>
</ul>
    umount</span> /mnt/boot</span></span>
    umount</span> /mnt</span></span></code></pre>Next steps</a></h3>
You can now follow the post-installation recommendation. 4</a></sup></p>

Add a user</li>
Install software</li>
...</li>
</ul>
Conclusion</a></h2>
I used this guide a few time for my personal machines.
Overall the installation of Arch is straight forward, but as always with Arch you have a lot
of possibilities (or opportunities) to use more or less equivalent software to reach the same goal.
In this guide I decided on a lot of tools (e.g. systemd-boot instead of grub), because I simply like them
or have no idea about newer and maybe better tools for the same problem.
If you have any recommendations or improvements, don't hesitate to share them with me via mail.</p>



https://btrfs.readthedocs.io/en/latest/btrfs-balance.html ↩</a></p>
</li>

https://legacy.thomas-leister.de/arch-linux-luks-verschluesselt-auf-uefi-system-installieren-2/ ↩</a></p>
</li>

https://wiki.archlinux.org/title/Installation_guide ↩</a></p>
</li>

https://wiki.archlinux.org/title/Installation_guide#Post-installation ↩</a></p>
</li>
</ol>
</section>


Neovim LSP
2022-03-01T00:00:00+00:00
I was (and still am for some parts) a happy vscode user.
On the other hand I was always looking up to all these crazy fast vim/neovim users and their keyboard only
drive workflows.
After hopping on and off the vim train a few times a year something new came around sometime in 2021 -
LSP support on neovim core.
LSP stands for Language Server Protocol and defines how your IDE communicates with a specific language server
to get features like 'go to definition', 'show references' or 'rename this variable'.
This was a huge game changer as you can now use every language server written to work with vscode with neovim.
So I took the time to watch a few videos, read the documentation and finally setup neovim and
gopls</a> the language server for Go. 1</a></sup> 2</a></sup> 3</a></sup></p>
In this post I will present the simplest setup to get an IDE-like feeling within neovim.</p>
Overview</a></h2>
Just a quick overview about the neovim ecosystem before we get started.</p>


Neovim supports Lua to configure plugins and a lot of other things.
Actually the possibility to configure it via Lua just got a bit better 4</a></sup>.</p>
</li>

The old way to configure things is via init.vim</code>, but personally I prefer an actual programming language instead
of vimscript.</p>
</li>

The last things I want to mention are plugins.
There are a lot of plugins, for example we are going to use nvim-lspconfig</code>, so you should be able to extend
your editor more or less like any other editor/IDE.</p>
</li>
</ol>
Install neovim</a></h2>
The configuration I will present needs at least neovim version 0.7</code> since you can not configure keymaps in lua in version 0.6</code>.
As this version is currently only available as a development release you have two ways to install it.</p>
First you can clone the neovim repository</a> and build it yourself,
which is actually pretty straight forward and worked well on macOS.
Simply follow the instructions in the README.</p>
The second options depend on your OS or Linux distribution.
In my case I used the AUR package neovim-nightly-bin</a>.</p>
A basic neovim configuration</a></h2>
Start with a new init.vim</code> in ~/.config/nvim/</code>.
At first, we need the LSP plugin itself and some other helpers.
I'm using vim-plugged</a> to download and update plugins automatically.
Simply install it from the sources of your distribution or follow the instructions provided in the README.</p>
call plug#begin('~/.nvim/plugged')</span></span>
</span>
    Plug 'neovim/nvim-lspconfig'</span></span>
</span>
    Plug 'nvim-lua/plenary.nvim'</span></span>
</span>
    " LSP autocompletion</span></span>
    Plug 'hrsh7th/nvim-cmp'</span></span>
    Plug 'hrsh7th/cmp-nvim-lsp'</span></span>
    Plug 'hrsh7th/cmp-buffer'</span></span>
    Plug 'hrsh7th/cmp-path'</span></span>
</span>
    " More autocompletion</span></span>
    Plug 'L3MON4D3/LuaSnip' " There are other snippet engines you could use, but we will be using this one.</span></span>
    Plug 'saadparwaiz1/cmp_luasnip'</span></span>
</span>
call plug#end()</span></span></code></pre>
Now startup neovim and install all plugins with :PlugInstall</code>.
We now have a working neovim, but haven't configured anything related to the LSP.</p>
Setup LSP-Plugin</a></h2>
Since we don't want to use vimscript for the complex pieces of the lsp configuration, we will create a lua script for that.
The configuration must be stored in ~/.config/nvim/lua/aps.lua</code>.
You can of course rename the lua file aps.lua</code> to anything you want, but I used my login name on most of my machines.</p>
Let's start by requiring the new lua configuration file in the previously created init.vim</code>.</p>
lua</span> require</span>(</span>"</span>aps</span>"</span>)</span></span></code></pre>
lua require("aps")</code> is the magic keyword to load the lua configuration.
This another game changer as I can now configure most things within an actual scripting language, instead of
neovim configuration files.
So let's have a look at the content of aps.lua</code>.</p>
--</span> Hook LSP into completion</span></span>
local</span> capabilities</span> =</span> require</span>(</span>'</span>cmp_nvim_lsp</span>'</span>).</span>update_capabilities</span>(</span>vim</span>.</span>lsp</span></span>
  .</span>protocol</span></span>
  .</span>make_client_capabilities</span>())</span></span>
</span>
</span>
--</span> Setup go language server gopls</span></span>
require</span>'</span>lspconfig</span>'</span>.</span>gopls</span>.</span>setup</span> {</span></span>
    capabilities</span> =</span> capabilities</span>,</span></span>
    completeUnimported</span> =</span> true</span>,</span></span>
    analyses</span> =</span> {</span>unusedparams</span> =</span> true</span>},</span></span>
    staticcheck</span> =</span> true</span>,</span></span>
    on_attach</span> =</span> function</span>(</span>)</span></span>
        --</span> "n" means normal mode</span></span>
        --</span> {buffer=0} means only for this buffer</span></span>
        --</span> <cmd> == :</span></span>
        --</span> <cr> == enter</span></span>
</span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>K</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>hover</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gd</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>definition</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gt</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>type_definition</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gi</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>implementation</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gR</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>references</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gr</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>rename</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>gf</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>formatting</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>ga</span>"</span>, </span>vim</span>.</span>lsp</span>.</span>buf</span>.</span>code_action</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>en</span>"</span>, </span>vim</span>.</span>diagnostic</span>.</span>goto_next</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
        vim</span>.</span>keymap</span>.</span>set</span>(</span>"</span>n</span>"</span>, </span>"</span>ep</span>"</span>, </span>vim</span>.</span>diagnostic</span>.</span>goto_prev</span>, {</span>buffer</span> =</span> 0</span>})</span></span>
    end</span></span>
}</span></span>
</span>
--</span> LSP autocomplete</span></span>
vim</span>.</span>opt</span>.</span>completeopt</span> =</span> {</span>"</span>menu</span>"</span>, </span>"</span>menuone</span>"</span>, </span>"</span>noselect</span>"</span>} </span>--</span> setting vim values</span></span>
</span>
--</span> Setup nvim-cmp</span></span>
local</span> cmp</span> =</span> require</span> '</span>cmp</span>'</span></span>
</span>
cmp</span>.</span>setup</span>({</span></span>
    snippet</span> =</span> {</span></span>
        expand</span> =</span> function</span>(</span>args</span>)</span></span>
            require</span>(</span>'</span>luasnip</span>'</span>).</span>lsp_expand</span>(</span>args</span>.</span>body</span>) </span>--</span> For `luasnip` users.</span></span>
        end</span></span>
    },</span></span>
    mapping</span> =</span> {</span></span>
        [</span>'</span><C-b></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>(</span>cmp</span>.</span>mapping</span>.</span>scroll_docs</span>(</span>-</span>4</span>), {</span>'</span>i</span>'</span>, </span>'</span>c</span>'</span>}),</span></span>
        [</span>'</span><C-f></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>(</span>cmp</span>.</span>mapping</span>.</span>scroll_docs</span>(</span>4</span>), {</span>'</span>i</span>'</span>, </span>'</span>c</span>'</span>}),</span></span>
        [</span>'</span><C-Space></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>(</span>cmp</span>.</span>mapping</span>.</span>complete</span>(), {</span>'</span>i</span>'</span>, </span>'</span>c</span>'</span>}),</span></span>
        [</span>'</span><C-y></span>'</span>] </span>=</span> cmp</span>.</span>config</span>.</span>disable</span>, </span>--</span> Specify `cmp.config.disable` if you want to remove the default `<C-y>` mapping.</span></span>
        [</span>'</span><C-e></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>({</span></span>
            i</span> =</span> cmp</span>.</span>mapping</span>.</span>abort</span>(),</span></span>
            c</span> =</span> cmp</span>.</span>mapping</span>.</span>close</span>()</span></span>
        }),</span></span>
        --</span> Accept currently selected item. Set `select` to `false` to only confirm explicitly selected items.</span></span>
        [</span>'</span><CR></span>'</span>] </span>=</span> cmp</span>.</span>mapping</span>.</span>confirm</span>({</span>select</span> =</span> true</span>})</span></span>
    },</span></span>
    sources</span> =</span> cmp</span>.</span>config</span>.</span>sources</span>({</span></span>
        {</span>name</span> =</span> '</span>nvim_lsp</span>'</span>},</span></span>
		{</span>name</span> =</span> '</span>luasnip</span>'</span>} </span>--</span> For luasnip users.</span></span>
    }, {{</span>name</span> =</span> '</span>buffer</span>'</span>}})</span></span>
})</span></span>
</span>
--</span> Enable inline errors</span></span>
vim</span>.</span>lsp</span>.</span>handlers</span>[</span>"</span>textDocument/publishDiagnostics</span>"</span>] </span>=</span></span>
    vim</span>.</span>lsp</span>.</span>with</span>(</span>vim</span>.</span>lsp</span>.</span>diagnostic</span>.</span>on_publish_diagnostics</span>,</span></span>
      {</span>update_in_insert</span> =</span> true</span>})</span></span></code></pre>
This configuration is more or less based on the excellent video by bashbunni and TJ 1</a></sup>.</p>
The exciting part begins with require'lspconfig'.gopls.setup</code>, as this is the place to configure the LSP
to your favorite hotkeys.
The bindings are set up within the attribute on_attach</code> of the gopls</code> object.
I try to keep them in sync with my wording e.g. go to type definition</code> uses the hotkey gt</code>
or go to definition</code> uses gd</code>.</p>
I especially like the binding gf</code> (or go fmt</code>) to get my Go code formatted.
But you configure these to whatever you want.
In my case all of these work only in "normal" mode and without usage of the leader key.</p>
Most of the other parts are related to autocompletion provided by nvim-cmp</code> and lua-snip</code>, which are
covered in these videos by TJ 5</a></sup> 6</a></sup>.
I'm still a very basic user of these and especially lua-snip</code> is a magic tool, which I want to explore
deeper in the near future.
But let's keep it simple for now and simply accept that we need this to get an IDE-like autocompletion.</p>
At this point you should be able to open a go file and get basic highlighting and completion.</p>
</p>
As you can see the completion provided by the LSP works fine, but the highlighting is not as good as I would
like it.
We will explore how to get there in the next section of this post.</p>
Treesitter for better highlighting</a></h2>
Treesitter</a> will build a syntax tree for your current file and add
highlighting based on this syntax tree.
This is a pretty huge dependency, and you can skip this if you are ok with the look of your code in the
earlier provided image.</p>
I can also recommend this video (again) TJ about LSP vs treesitter 7</a></sup>.</p>
To enable treesitter we have to add the treesitter plugin and provide a basic configuration in our lua
configuration.</p>
Add this to the plugin section if your init.vim</code>.</p>
Plug</span> '</span>nvim-treesitter/nvim-treesitter</span>'</span>,</span> {</span>'</span>do</span>'</span>:</span> '</span>:TSUpdate</span>'</span>}</span></span></code></pre>
Again open neovim and run :PlugInstall</code> to get the plugin.
The initial run of TSUpdate</code> will take a long time (long in 30 seconds to 1 minute) to install all the
treesitter plugin for the most common languages.</p>
Now add the following at the bottom of your lua configuration.
This is basically the simplest configuration provided in the treesitter README 8</a></sup>.</p>
require</span>'</span>nvim-treesitter.configs</span>'</span>.</span>setup</span> {</span></span>
  --</span> One of "all", "maintained" (parsers with maintainers), or a list of languages</span></span>
  ensure_installed</span> =</span> "</span>maintained</span>"</span>,</span></span>
</span>
  --</span> Install languages synchronously (only applied to `ensure_installed`)</span></span>
  sync_install</span> =</span> false</span>,</span></span>
</span>
  --</span> List of parsers to ignore installing</span></span>
  ignore_install</span> =</span> { </span>"</span>javascript</span>" </span>},</span></span>
</span>
  highlight</span> =</span> {</span></span>
    --</span> `false` will disable the whole extension</span></span>
    enable</span> =</span> true</span>,</span></span>
</span>
    --</span> list of language that will be disabled</span></span>
    disable</span> =</span> { </span>"</span>c</span>"</span>, </span>"</span>rust</span>" </span>},</span></span>
</span>
    --</span> Setting this to true will run `:h syntax` and tree-sitter at the same time.</span></span>
    --</span> Set this to `true` if you depend on 'syntax' being enabled (like for indentation).</span></span>
    --</span> Using this option may slow down your editor, and you may see some duplicate highlights.</span></span>
    --</span> Instead of true it can also be a list of languages</span></span>
    additional_vim_regex_highlighting</span> =</span> false</span>,</span></span>
  },</span></span>
}</span></span></code></pre>
And that's it!
Let's open up the file from before again and have a look at the differences.</p>
</p>
As you can see for example functions are now highlighted!
In case you wonder, I'm using the Dracula</code> theme.</p>
Conclusion</a></h2>
To be honest it was a lot of fun getting this to work, do some research and write it up in a post.
Maybe I will write about other cool tools in the neovim ecosystem like telescope</code> or all the magic
you can do with luasnip</code>.
But for now I'm very happy with the setup and the speed you get, especially compared to vscode.
As a fan of keyboard driven workflows this is more or less the best you can get, and I really enjoy it!</p>



https://www.youtube.com/watch?v=puWgHa7k3SY ↩</a> ↩2</a></p>
</li>

https://www.youtube.com/watch?v=2WPC8rZQvQU ↩</a></p>
</li>

https://www.youtube.com/watch?v=0fOIp5PH648 ↩</a></p>
</li>

https://github.com/neovim/neovim/pull/14661 ↩</a></p>
</li>

https://www.youtube.com/watch?v=_DnmphIwnjo ↩</a></p>
</li>

https://www.youtube.com/watch?v=Dn800rlPIho ↩</a></p>
</li>

https://www.youtube.com/watch?v=c17j09vY5sw ↩</a></p>
</li>

https://github.com/nvim-treesitter/nvim-treesitter ↩</a></p>
</li>
</ol>
</section>


Build a simple VPN with WireGuard
2022-02-06T00:00:00+00:00
As you already know I host most of the services I use on a daily basis myself.
Some of these services are hosted on the public Internet, while others are running on my home server.
In this post I will cover the basic on how to get a peers-to-site VPN connection with
WireGuard</a>.</p>
WireGuard uses a simple public/private keypair to set up an encrypted VPN tunnel, therefore there is no
need for a complicated certificate authority setup.
In my opinion this is a major step when upgrading from e.g. OpenVPN.
Also, WireGuard is superfast and included in all newer versions of the Linux kernel.</p>
In my case both the server and the client are running Arch Linux and the only additional package you have
to install is wireguard-tools</code>.</p>
Set up a server</a></h2>
The server must be a machine with a public IPv4 and/or v6 address.
In my case I use the cheapest Hetzner cloud machine CX11</code> which costs about 4 Euro a month.</p>
At first generate keys with wg genkey | (umask 0077 && tee server.key) | wg pubkey > server.pub</code>.
Then use the following file as template, replace the needed variables and save it at /etc/wireguard/wg-server.conf</code>.
The preshared key can simply be generated with wg genpsk > peer1-peer2.psk</code>. Each connection or peer needs its own preshared key!</p>
    #</span> wg-server.conf</span></span>
    [</span>Interface</span>]</span></span>
    Address</span> =</span> 10.0.0.1/24,fd42:42:42::1/64</span></span>
    ListenPort</span> =</span> 1194</span></span>
    PrivateKey</span> =</span> <</span>generated-private-ke</span>y</span>></span></span>
    PostUp</span> =</span> iptables</span> -</span>A</span> FORWARD</span> -</span>i</span> %i</span> -</span>j</span> ACCEPT</span>;</span> iptables</span> -</span>t</span> nat</span> -</span>A</span> POSTROUTING</span> -</span>o</span> eth0</span> -</span>j</span> MASQUERADE</span></span>
    PostDown</span> =</span> iptables</span> -</span>D</span> FORWARD</span> -</span>i</span> %i</span> -</span>j</span> ACCEPT</span>;</span> iptables</span> -</span>t</span> nat</span> -</span>D</span> POSTROUTING</span> -</span>o</span> eth0</span> -</span>j</span> MASQUERADE</span></span>
</span>
    #</span> Home-VPN Gateway</span></span>
    [</span>Peer</span>]</span></span>
    PublicKey</span> =</span> <</span>generated-client-public-ke</span>y</span>></span></span>
    AllowedIPs</span> =</span>  10.0.0.2/32,</span> fd42:42:42::2/128,</span> 192.168.0.0/24</span></span>
    PresharedKey</span> =</span> <</span>preshared-ke</span>y</span>></span></span>
</span>
    #</span> Example for a roadwarrior client</span></span>
    [</span>Peer</span>]</span></span>
    PublicKey</span> =</span> <</span>generated-public-ke</span>y</span>></span></span>
    AllowedIPs</span> =</span> 10.0.0.3/32,fd42:42:42::3/128</span></span>
    PresharedKey</span> =</span> <</span>preshared-ke</span>y</span>></span></span></code></pre>
The important part is the network 192.168.0.0/24</code> in the AllowedIPs</code> part of the gateways peer config.
This will add a route on the server side, so the server knows that your home network can be reached through
this specific WireGuard interface.</p>
Enable IP forwarding with the following commands.
-w</code> will make these settings persistent across reboots.</p>
    $</span> sysctl</span> -</span>w</span> net.ipv4.ip_forward=</span>1</span></span>
    $</span> sysctl</span> -</span>w</span> net.ipv6.conf.all.forwarding=</span>1</span></span></code></pre>
Start the client with wg-quick up wg-server</code> or systemctl enable --now wg-quick@wg0</code>.</p>
Configure the client</a></h2>
Again generate keys with wg genkey | (umask 0077 && tee client.key) | wg pubkey > client.pub</code>.
Then use the following file as template, replace the needed variables and save it at /etc/wireguard/wg-client.conf</code>.</p>
#</span> wg-client.conf</span></span>
</span>
[</span>Interface</span>]</span></span>
PrivateKey</span> =</span> <</span>generated-private-ke</span>y</span>></span></span>
Address</span> =</span> 10.0.0.2/24,fd42:42:42::3/64</span></span>
</span>
[</span>Peer</span>]</span></span>
PublicKey</span> =</span> <</span>generated-server-public-ke</span>y</span>></span></span>
Endpoint</span> =</span> my.server.com:1194</span></span>
AllowedIPs</span> =</span> 10.0.0.0/24,fd42:42:42::0/64,192.168.0.1/24</span></span>
PresharedKey</span> =</span>  <</span>preshared-ke</span>y</span>></span></span>
PersistentKeepalive</span> =</span> 25</span></span></code></pre>
PersistentKeepalive</code> will ensure that the connection will be open</code>, even if there is no need for it.
As WireGuard is a silent protocol there will be no communication without a client sending or recieving
messages. This is especially critical with peers behind NAT, which will likely be the case with almost every home setup.</p>
Again enable IP forwarding with the following commands, as we want to forward traffic from local clients to VPN clients.</p>
    $</span> sysctl</span> -</span>w</span> net.ipv4.ip_forward=</span>1</span></span>
    $</span> sysctl</span> -</span>w</span> net.ipv6.conf.all.forwarding=</span>1</span></span></code></pre>
Start the client with wg-quick up wg-client</code> or systemctl enable --now wg@wg-client</code>.</p>
To route all traffic through the tunnel you can use 0.0.0.0/0</code> in the AllowedIPs</code> section.</p>
Static route</a></h2>
The last thing to do is to set a static route to your VPN network.
You can do this on each home device or simply add a static IPv4/v6 route on your Router.
The route should look like this 10.0.0.0/24 via <ip of your client on the local network></code>.</p>
At my home network the Pi I'm using as the client has the IP 192.168.0.8</code>, therefore the static route looks like
this 10.0.0.0/24 via 192.168.0.8</code>.</p>
Debugging</a></h2>
As WireGuard is a very silent protocol it can be hard the debug in case of any connectivity problems.
Try to ping from each side of the tunnel and use wg show</code> to check if the initial handshake was successfull.
Check the logs with dmesg -wT</code> or journalctl -u wg@wg-client/server</code>.</p>
    $</span> modprobe</span> wireguard</span></span>
    $</span> echo</span> module</span> wireguard</span> +p</span> ></span> /sys/kernel/debug/dynamic_debug/control</span></span></code></pre>Performance</a></h2>
I simply used iperf3</code> to run an upload and download test from my home to the server.
As I have an 1000 Mbit download and 50 Mbit upload connection we should see some useful results.</p>
Let's start with the upload test.
This is more or less my max. possible upload, therefore this a very good result.</p>
➜</span>  ~</span> iperf3</span> -</span>c</span> 10.66.66.1</span></span>
Connecting</span> to</span> host</span> 10.66.66.1,</span> port</span> 5201</span></span>
[</span>  5</span>]</span> local</span> 192</span>.</span>168</span>.</span>0</span>.</span>28</span> port</span> 34676</span> connected</span> to</span> 10</span>.</span>66</span>.</span>66</span>.</span>1</span> port</span> 5201</span></span>
[</span> ID</span>]</span> Interval           Transfer     Bitrate         Retr  Cwnd</span></span>
[</span>  5</span>]</span>   0.00-1.00   sec  6.20 MBytes  52.0 Mbits/sec    1    179 KBytes</span></span>
[</span>  5</span>]</span>   1.00-2.00   sec  5.76 MBytes  48.3 Mbits/sec    1    151 KBytes</span></span>
[</span>  5</span>]</span>   2.00-3.00   sec  5.70 MBytes  47.8 Mbits/sec    0    178 KBytes</span></span>
[</span>  5</span>]</span>   3.00-4.00   sec  5.82 MBytes  48.8 Mbits/sec    1    152 KBytes</span></span>
[</span>  5</span>]</span>   4.00-5.00   sec  6.00 MBytes  50.3 Mbits/sec    0    179 KBytes</span></span>
[</span>  5</span>]</span>   5.00-6.00   sec  6.00 MBytes  50.3 Mbits/sec    2    110 KBytes</span></span>
[</span>  5</span>]</span>   6.00-7.00   sec  5.46 MBytes  45.8 Mbits/sec    0    143 KBytes</span></span>
[</span>  5</span>]</span>   7.00-8.00   sec  6.00 MBytes  50.3 Mbits/sec    1    122 KBytes</span></span>
[</span>  5</span>]</span>   8.00-9.00   sec  6.00 MBytes  50.3 Mbits/sec    0    152 KBytes</span></span>
[</span>  5</span>]</span>   9.00-10.00  sec  5.76 MBytes  48.3 Mbits/sec    1    128 KBytes</span></span>
-</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span></span>
[</span> ID</span>]</span> Interval           Transfer     Bitrate         Retr</span></span>
[</span>  5</span>]</span>   0.00-10.00  sec  58.7 MBytes  49.2 Mbits/sec    7             sender</span></span>
[</span>  5</span>]</span>   0.00-10.02  sec  58.1 MBytes  48.7 Mbits/sec                  receiver</span></span>
</span>
iperf</span> Done.</span></span></code></pre>
Now to the download test which is not as good as expected.
This is about 1/4 of the possible download.
The server has about 35-40 % CPU load while running the test, so it looks like the cheap VM might be
the limit here.</p>
➜</span>  ~</span> iperf3</span> -</span>c</span> 192.168.0.28</span></span>
Connecting</span> to</span> host</span> 192.168.0.28,</span> port</span> 5201</span></span>
[</span>  5</span>]</span> local</span> 10</span>.</span>66</span>.</span>66</span>.</span>1</span> port</span> 52106</span> connected</span> to</span> 192</span>.</span>168</span>.</span>0</span>.</span>28</span> port</span> 5201</span></span>
[</span> ID</span>]</span> Interval           Transfer     Bitrate         Retr  Cwnd</span></span>
[</span>  5</span>]</span>   0.00-1.00   sec  26.2 MBytes   219 Mbits/sec    0   1.31 MBytes</span></span>
[</span>  5</span>]</span>   1.00-2.00   sec  31.2 MBytes   262 Mbits/sec  379   1.41 MBytes</span></span>
[</span>  5</span>]</span>   2.00-3.00   sec  31.2 MBytes   262 Mbits/sec    0   1.55 MBytes</span></span>
[</span>  5</span>]</span>   3.00-4.00   sec  31.2 MBytes   262 Mbits/sec  146   1.24 MBytes</span></span>
[</span>  5</span>]</span>   4.00-5.00   sec  26.2 MBytes   220 Mbits/sec    0   1.24 MBytes</span></span>
[</span>  5</span>]</span>   5.00-6.00   sec  26.2 MBytes   220 Mbits/sec    0   1.30 MBytes</span></span>
[</span>  5</span>]</span>   6.00-7.00   sec  26.2 MBytes   220 Mbits/sec    0   1.35 MBytes</span></span>
[</span>  5</span>]</span>   7.00-8.00   sec  27.5 MBytes   231 Mbits/sec    0   1.38 MBytes</span></span>
[</span>  5</span>]</span>   8.00-9.00   sec  27.5 MBytes   231 Mbits/sec    0   1.39 MBytes</span></span>
[</span>  5</span>]</span>   9.00-10.00  sec  33.8 MBytes   283 Mbits/sec    0   1.40 MBytes</span></span>
-</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span> -</span></span>
[</span> ID</span>]</span> Interval           Transfer     Bitrate         Retr</span></span>
[</span>  5</span>]</span>   0.00-10.00  sec   287 MBytes   241 Mbits/sec  525             sender</span></span>
[</span>  5</span>]</span>   0.00-10.05  sec   286 MBytes   238 Mbits/sec                  receiver</span></span>
</span>
iperf</span> Done.</span></span></code></pre>
But this is more than enough for my day to day computing and even streaming a movie from my jellyfin
instance at home.</p>
Conclusion</a></h2>
I am a huge fan of WireGuard as it is simple to set up and fast.
The tooling is easy to use (especially when compared against OpenVPN),
the documentation is straight forward, and it simply works.
And if you are running a recent Linux kernel most of the things you need are already there.</p>


Using macOS as a Linux user
2021-12-17T00:00:00+00:00
I never expected to use any other OS than my beloved Linux (and especially Arch Linux) for developing and day-to-day
computing.
Well, I actually use Windows to play some games with friends occasionally, but the time has come to use a Mac at
work.
It was simply the easiest way to get a much more powerful machine than my ThinkPad X390.
So here I'm trying to write up what I like about macOS and what I miss from my highly customized Linux machine.
This isn't a fair comparison as I know much more about Linux than about macOS, but maybe you (as in the reader)
can correct me, and I'll learn something new!</p>
Things I like</a></h2>
So let's start with the things I like about macOS.</p>
1. No UI glitches</a></h3>
As a Wayland user I'm used to UI glitches with not Wayland native applications.
For example getting IntelliJ IDEs running took quite some time, to set all needed environment flags.
And most of the apps and system UIs do look great by default.
I think it might be hard to do some customization if you want to, but as I don't care that much
I'm fine with the default.</p>
2. Hardware accelerated apps (especially browsers)</a></h3>
Yes this is still a thing on Linux.
You can get it to work with chromium and Vaapi or Firefox</a>,
but it's often buggy, your browser decides to freeze, or it simply won't work at all.
At least this was my experience on my X250 and my X390.
On my desktop computer I do have enough performance to simply not need hardware acceleration, but it is
really handy on a mobile device.</p>
3. Time Machine Backups</a></h3>
In my opinion a modern OS must provide a simple tool to back up your system.
Time Machine may not be the fastest backup tool, but it does the job and restoring files is nicely integrated
in the UI.
I also wrote a post about how to transform a Raspberry Pi into a Time Capsule</a>.</p>
4. It's a UNIX</a></h3>
And therefore most of my used applications and workflows simply work.
I mostly rely on a terminal with zsh, git, neovim, tmux and some other tools and within the
terminal there absolutely no difference between my Linux machine and the Mac.
Getting all my settings was also very easy, clone my dotfiles repository, add all need symlinks via stow</code>, and
I was ready to go.</p>
5. The magic touchpad and the gestures to navigate the UI</a></h3>
On my ThinkPad I was a happy trackpoint user as it provides access to a mouse without moving your finger
too far away from the keyboard.
I was surprised to enjoy the magic trackpad, but the experience of a touchpad instead of a mouse is somewhat nice.
My only problem is wrist pain after using the touchpad for a whole workday, so I'm currently switching between
mouse and trackpad.
A annoying thing about that is that you can't disable/enable natural scrolling per device.
I enjoy it on the trackpad, but don't like it on the mouse.</p>
6. APFS</a></h3>
On my Linux machine I'm using brtfs because it supports snapshots, compression and is a copy-on-write (CoW) file system.
But to be honest the initial learning curve and setup took quite some time.
On macOS, I get all of this for free with APFS.</p>
7. Application Launcher</a></h3>
I am used to rofi</code> or wofi</code> to start my applications via keyboard only.
On macOS, I simply use Command + Space</code> to get spotlight search to start applications and search for other things.
This is more or less the same workflow as on my Linux machine.</p>
8. Application Scaling</a></h3>
Scaling on a 4K display works like a charm.</p>
Things I don't like:</a></h2>
Now to the things I don't like, or I'm still searching for a workaround.</p>
1. No package manager</a></h3>
As Linux and especially Arch Linux user not using a package manager to install my software feels like using Windows 98.
(I don't use Windows that often, but I think they also have a package manager now?).
There are three options to install most of the software on macOS.
At first there is the 'download it from the web and run the installer', second the official App Store and last using a
package manager like homebrew</code>.</p>
2. Can't disable dock</a></h3>
I simply don't need it.
When I want to start applications I do it via my terminal or via spotlight, but this seems like another problem
of a keyboard centric user.</p>
3. Can't integrate into spotlight search (gopass)</a></h3>
I let gopass</a> handle all my password/secret related things.
On my Linux machines I've integrated it perfectly within rofi</a> and it looks like
Alfred</a> can do the same on macOS.
But this is one of the things that's still on my to-do list.</p>
4. Touchbar and no physical escape key</a></h3>
I have no idea who could possible like the Touchbar.
Especially as because if it there is no</strong> physical escape key.
As I like to use Neovim</a> for editing config files and more ore less anything besides code,
this is just annoying.</p>
5. Different keyboard layout</a></h3>
I think this just takes some time to adjust to it.</p>
6. Floating window management</a></h3>
I'm a happy user of sway</a> and really enjoy using tiling window managers, so using a floating
window manager feels like a step back.
There are things like virtual desktops that work like workspaces in sway, but you can't move apps to a different workspace
with a shortcut (please correct me if I'm wrong here!).
You can use the magic touchpad/mouse, but I prefer to use the mouse/touchpad as rarely as possible.</p>
7. launchd uses xml configuration files</a></h3>
As a Linux user I'm used to systemd units, that are sometimes long, but not as unreadable as XML.
But maybe that's because I'm actually never used XML in the past.</p>
8. Click in window to actually focus it</a></h3>
At first, I thought the system was somehow unresponsive, or the mouse is not in the window I want to type in,
but I later noticed that you have to actually focus a window with a click.
This is so annoying, especially with a floating window manager, where it is normal to have multiple windows
overlapping each other.
Why shouldn't I be able to type in a window that I didn't focus with a click?</p>
9. Updates take a long time</a></h3>
I have no idea what the mac is doing in the background, but I never experienced updates taking that much time.
For example the update from 12.0 to 12.1 took about 30 minutes.
And this was only the operating system update, after that there were also some brew updates, docker and more.</p>
10. Bad heat distribution and loud fans</a></h3>
The hardware seems overwhelmed by the heat produced by the Intel i9.
If you open a few apps, start debugging or a few containers it will start to get hot very fast.
And the fan is very noisy, but I think all if this will be fixed with the new M1 Macs.
Besides that the device is pretty well build.</p>
Conclusion</a></h2>
I think I will update this post with new learnings from time to time.
For now, I'm quite happy with the macOS setup, as I could simply copy most of my dotfiles.
To be honest the things I use most of the time are a terminal, a browser and an editor and
these things are working quite well.
The thing I miss the most is a good package manager.
Pacman does an excellent job at this, because it is fast and make it so much easier to keep your system up to date.</p>


Transform your Raspberry Pi into a Apple Time Capsule
2021-11-23T00:00:00+00:00
This is a short guide (and reminder for myself) how to add a Samba share as target for macOS Time Machine.
I found some blog posts about how to do this, but most of the time the author created a new disk on the share 1</a></sup> 2</a></sup>.
This seems like a rather unusual complicated setup to do backups and
there are products like the Apple Time Capsule</a>, that
works via network discovery.
I actually found a blog post (which I can't anymore), how to set up Avahi to announce a share as Time Capsule.</p>
This actually works with every Linux host with samba</code> and avahi</code> installed, but the title with Raspberry Pi in it is much more catchy.
And I actually use a Raspberry Pi 4 to provide backup space for my machines and servers.</p>
Setup Samba</a></h3>
To get SMB up and running you have to:</p>

Install samba via the package manager of your distribution.</li>
Create a Unix user.</li>
Create a samba user with smbpasswd -a</code>.</li>
Create a new folder on your backup volume (in my case this is an external 8 TB Disk on a Raspberry Pi).</li>
Add the following entry in the smb.conf</code>. You should adjust the path</code> and user</code> to match your environment.</li>
</ol>
# /etc/samba/smb.conf</span></span>
</span>
[backups]</span></span>
    comment = Time Machine Backups</span></span>
    path = /mnt/backups/andre/time-machine</span></span>
    guest ok = no</span></span>
    read only = no</span></span>
    browseable = yes</span></span>
    valid users = andre</span></span>
    write list = andre</span></span>
    vfs objects = catia fruit streams_xattr</span></span>
    fruit:time machine = yes</span></span></code></pre>

Restart smbd</code>.</li>
</ol>
You should now be able to access the share.
If not, check that the user you created earlier actually has the rights to access the folder!
You can also add quotas to this share, but I have plenty of free space left, so there is really no need to do this right now.</p>
Configure Avahi</a></h3>
This is the part that will announce the backup server and the SMB service as an Apple Time Capsule.
Avahi</a> is a system which provides service discovery on a local network via the mDNS/DNS-SD protocol suite.</p>

Install Avahi via the package manager of your distribution.</li>
Add the following configuration for SMB.</li>
Restart avahi</code>.</li>
</ol>
# /etc/avahi/services/samba.service</span></span>
</span>
<?</span>xml</span> version</span>=</span>"</span>1.0</span>"</span> standalone</span>=</span>'</span>no</span>'</span>?></span><!--</span>*-nxml-*</span>--></span></span>
<!</span>DOCTYPE</span> service-group</span> SYSTEM "avahi-service.dtd"</span>></span></span>
<</span>service-group</span>></span></span>
  <</span>name</span> replace-wildcards</span>=</span>"</span>yes</span>"</span>></span>%h</span></</span>name</span>></span></span>
  <</span>service</span>></span></span>
    <</span>type</span>></span>_smb._tcp</span></</span>type</span>></span></span>
    <</span>port</span>></span>445</span></</span>port</span>></span></span>
  </</span>service</span>></span></span>
  <</span>service</span>></span></span>
    <</span>type</span>></span>_device-info._tcp</span></</span>type</span>></span></span>
    <</span>port</span>></span>9</span></</span>port</span>></span></span>
    <</span>txt-record</span>></span>model=TimeCapsule8,119</span></</span>txt-record</span>></span></span>
  </</span>service</span>></span></span>
  <</span>service</span>></span></span>
    <</span>type</span>></span>_adisk._tcp</span></</span>type</span>></span></span>
    <</span>port</span>></span>9</span></</span>port</span>></span></span>
    <</span>txt-record</span>></span>dk0=adVN=backups,adVF=0x82</span></</span>txt-record</span>></span></span>
    <</span>txt-record</span>></span>sys=adVF=0x100</span></</span>txt-record</span>></span></span>
  </</span>service</span>></span></span>
</</span>service-group</span>></span></span></code></pre>
I found this configuration a few months ago but couldn't find it while researching for this post to provide the original source.</p>
</p>
But with this magic piece of XML we have done it!
You should now see your Raspberry Pi as an Apple Time Capsule in the network tab of Finder.</p>
Setup Time Machine</a></h3>
You can now create a new Time Machine backup.
Please don't forget to enable encryption, even if the disk you are using on the Pi is already encrypted!
While selecting the target disk you should be able to select our newly created backup location.</p>
</p>
Conclusion</a></h2>
Well that was more complicated than I think it should be.
Why shouldn't Time Machine be able to use a regular SMB share as backup endpoint?
Maybe some APFS magic is involved here, and Time Machine does simply send snapshots to the backup location?
I have no idea, but I'm pretty happy everything is working now, and I can let backups be as they should be: simple, boring and working.</p>



https://www.imore.com/how-use-time-machine-backup-your-mac-windows-shared-folder ↩</a></p>
</li>

https://kirb.me/2018/03/24/using-samba-as-a-time-machine-network-server.html ↩</a></p>
</li>
</ol>
</section>


How to find durable technology?
2021-11-14T00:00:00+00:00
This post is not focused on a specific technology, but more about how to find hardware with good support
(e.g. spare parts, software updates, ...) so you can use it for multiple years.
In this case I use headphones as an example, but the general concept can be applied to just every technology you want to buy.</p>
A headphone example</a></h2>
So let's dive into my example.
I currently own two different headphones.
The first are my Sony WH-1000MX4</a> with active noise-cancelling (ANC),
that I mostly use when I'm commuting, listening to podcasts while cooking, working in the office or at some other noisy place.
The ANC works very good and is just perfect if you need an hour to do some deep work or get some reading done in a noisy environment.
You can learn more about the Sony headphones in an older post</a>.</p>
The second headphones are my trusty Beyerdynamic DT-770 Pro</a> that I bought at least 9 years ago.
I mostly use the DT-770 when I'm working/studying at my desk and they are connected via the Focusrite Scarlett 2i2</a> interface to my computer.</p>
What I really like about both headphones is that you can easily get spare parts to repair them.
So this week it was time to change the ear pads of the Beyerdynamics headphones as they tend to become dirty after a few years of usage.
In this case I only bought new ear pads, but you can get about every part of the headphones directly</a> from Beyerdynamics.
The change itself takes about 10 minutes and is very straight forward.</p>
The point I want to make is that these headphones are about 10 years old, and I still can get new ear pads (or any other part) without any effort!
That is something I really like about pro</strong> (or on this case studio</strong>) hardware.</p>
So in this case the investment of ~130 Euro was more than worth it.
I actually hope the same holds true for the Sony headphones, but I'm not so sure of that.
The Sony's are more of a consumer product than the Beyerdynamic's and therefore will get updated more often.
The predecessor of the MX4, the MX3 were released in August 2018 and the MX4 in August 2020.
You can get new ear pads for the MX3/MX4, but I couldn't find any official other parts than that.
Hopefully I will never need more than ear pads from Sony.</p>
If it is possible I would prefer to get parts from the original equipment manufacturer (OEM) as they should know the specific specs of their product.
The problem there is that they often call for absurd high prices, so you have to use some other Chinese company to get them for a reasonable price.
Fortunately, that's not the case here!</p>
My conclusion is that headphones like the DT-770 are a good investment as you can use them for many years and if something is broken you can simply repair it yourself.
Besides, they sound excellent!</p>
A generic view on technology</a></h2>
The trick is to know if a product will be supported that good for a time span like 10 years.
In my opinion an indicator for that could be that a company exists at least for a few years.
If you get a product via Kickstarter I am not so sure the (maybe new founded) company behind it can provide any support.
You can't be sure if the company will exist in a few years.</p>
Also some people in the scene (like audio in this case) recommend their products.
But with people in the scene I explicitly do not mean some random tech youtuber or influencer that uses a product for a few weeks and the declares it as 'the best product I've ever used'.
I mean real feedback from people that use a product or a predecessor of a product for a long time.
Most of the time you can't directly infer the experiences from a predecessor to current product, but it can give you
an idea how the product could be supported (especially on the software/firmware side).</p>
These people can of course be found on the internet, just do some more research than simply trust your favorite youtube/random video service star.
This is of course associated with the risk that the same circumstances do not apply to the new product, but this is the risk we have to take with every new product we buy.</p>
Conclusion</a></h2>
Finally I think the headphone example is a very easy one as musicians/studios all around the world use Beyerdynamics headphones and you can see
them often in studio videos.
And in my case this was more or less a coincidence as I simply followed a recommendation in a forum.
Besides that the Beyerdynamics do not contain any software like the Sony's, so it's much easier for them to simply work.
But EOL your hardware because of missing software (security) updates is a whole other topic for another post.</p>
If you have any thoughts on this please let me know via mail!</p>


Automate Terraform with Atlantis and GitLab
2021-05-26T00:00:00+00:00
Atlantis</a> is an infrastructure as code (IaC) tool to automate Terraform interactions
within your GitLab/GitHub MR/PR workflow.
This post uses the setup explained in an older post</a>, where we used GitLab as
Terraform backend.</p>
Instead of running terraform init/apply/plan</code> on your local dev machine, you can now run it automagically when opening
a merge request.
This works by interacting via comments with Atlantis.</p>
</p>
Setup</a></h1>
1. Preparations</a></h2>
First generate a personal access token for the gitlab user with scope api</code>.
As this user will comment on MRs you should create a new atlantisbot</code> user or reuse an already existing bot user.</p>
Then generate a secret which will be shared between GitLab and Atlantis.
For example with pwgen -s 64 1 </code>.
This is not necessary but helps against random request against your Atlantis instance, as it will only accept requests
with this secret in the header.</p>
2. Setup Atlantis deployment.</a></h2>
Most of my services are running as containers and as Atlantis provides an upstream image there is no good reason
to not use it.
All the TF_*</code> variables must match the configured Terraform backend.
HCLOUD_TOKEN</code> and HETZNER_DNS_API_TOKEN</code> are specific token needed by the Hetzner Cloud</a>
and Hetzner Cloud DNS</a> Terraform provider.</p>
v</span>ersion</span>:</span> "</span>3</span>"</span></span>
s</span>ervices</span>:</span></span>
  a</span>tlantis</span>:</span></span>
    i</span>mage</span>:</span> r</span>unatlantis/atlantis</span></span>
    c</span>ontainer_name</span>:</span> a</span>tlantis</span></span>
    r</span>estart</span>:</span> a</span>lways</span></span>
    p</span>orts</span>:</span></span>
      -</span> 1</span>27.0.0.1:4141:4141</span></span>
    e</span>nvironment</span>:</span></span>
      A</span>TLANTIS_GITLAB_TOKEN</span>:</span> "</span>wasd</span>"</span></span>
      A</span>TLANTIS_GITLAB_WEBHOOK_SECRET</span>:</span> "</span>wasd</span>"</span></span>
      H</span>CLOUD_TOKEN</span>:</span> "</span>wasd</span>"</span></span>
      H</span>ETZNER_DNS_API_TOKEN</span>:</span> "</span>wasd</span>"</span></span>
      T</span>F_HTTP_ADDRESS</span>:</span> h</span>ttps://gitlab.mydomain.com/api/v4/projects/<ID>/terraform/state/project</span></span>
      T</span>F_HTTP_LOCK_ADDRESS</span>:</span> h</span>ttps://gitlab.mydomain.com/api/v4/projects/<ID>/terraform/state/project/lock</span></span>
      T</span>F_HTTP_UNLOCK_ADDRESS</span>:</span> h</span>ttps://gitlab.mydomain.com/api/v4/projects/<ID>/terraform/state/project/lock</span></span>
      T</span>F_HTTP_USERNAME</span>:</span> m</span>e</span></span>
      T</span>F_HTTP_PASSWORD</span>:</span> p</span>assword</span></span>
      T</span>F_HTTP_LOCK_METHOD</span>:</span> P</span>OST</span></span>
      T</span>F_HTTP_UNLOCK_METHOD</span>:</span> D</span>ELETE</span></span>
    c</span>ommand</span>:</span></span>
      -</span> s</span>erver</span></span>
      -</span> --</span>atlantis-url=https://atlantis.mydomain.com</span></span>
      -</span> --</span>gitlab-user=my-bot</span></span>
      -</span> --</span>repo-allowlist=gitlab.mydomain.com/my-project/iac</span></span>
      -</span> --</span>gitlab-hostname=gitlab.mydomain.com</span></span></code></pre>
Add repositories in the repo-allowlist</code> parameters to enable Atlantis for these specific repositories.</p>
3. Register a GitLab webhook in your project.</a></h2>
Add a webhook to all projects previously added in the repo-allowlist</code> option.
Select the trigger push events</code>, comments</code> and merge request events</code>.
Also check that the URL ends with /events</code>.</p>
</p>
4. Setup Caddy</a> as reverse proxy to expose Atlantis.</a></h2>
Setup caddy as a revsere proxy to handle TLS certificates and disable all log outputs.</p>
a</span>tlantis.mydomain.com {</span></span>
  r</span>everse_proxy http://localhost:4141 {</span></span>
    h</span>eader_up Host {http.reverse_proxy.upstream.hostport}</span></span>
  }</span></span>
  l</span>og {</span></span>
    o</span>utput discard</span></span>
  }</span></span>
}</span></span></code></pre>Secure access to the dashboard</a></h1>
Atlantis provides a dashboard that displays current locks on running deployments.
The lock locks the Terraform state to mitigate multiple apply</code> operations.
This is a neat feature as it prevents an inconsistent Terraform state.
But it also leaks the name of your repos/projects if the dashboard is available on the public internet (and anyone
with access to it can remove any state lock).
Also the /events</code> endpoint must be reachable from the public internet as it will be called by the earlier
registered webhook(s).</p>
</p>
This is an task for the basicauth Caddy module</a>.
The password must</strong> be hashed as Caddy will not accept plain text passwords.
You can generate a password with caddy hash-password</a>.</p>
root@server:~#</span> caddy</span> hash-password</span> -</span>-plaintext</span> my-secure-password</span></span>
JDJhJDE0JGVsMXhGeDI0UTZ4SkhhSi9yWlZ5Uk93Q08uU0twa05vOXNQVDY0MFpkVGdZMjZUT09Ha2hH</span></span></code></pre>a</span>tlantis.mydomain.com {</span></span>
  r</span>everse_proxy http://localhost:4141 {</span></span>
    h</span>eader_up Host {http.reverse_proxy.upstream.hostport}</span></span>
  }</span></span>
  b</span>asicauth / {</span></span>
    m</span>y-user JDJhJDE0JGVsMXhGeDI0UTZ4SkhhSi9yWlZ5Uk93Q08uU0twa05vOXNQVDY0MFpkVGdZMjZUT09Ha2hH</span></span>
  }</span></span>
  l</span>og {</span></span>
    o</span>utput discard</span></span>
  }</span></span>
}</span></span></code></pre>Conclusion</a></h1>
Atlantis provides an easy way to let teams work together without problems like locking the state because
two members are running Terraform on their local machine at the same time.
The interaction within MRs/PRs is simple and it also makes changes visible within an MR/PR.
With that a reviewer can directly see what Terraform will do and doesn't have to check out the branch locally and run a
terraform plan</code>.</p>


Backups with borg/borgmatic
2021-04-21T00:00:00+00:00
I am a big fan of the backup tool borgbackup</a>.
Borg</code> is space efficient as it supports compression and deduplication and is also
secure, so you don't have to trust the storage provider where your backup will be
stored.</p>
In this short guide we will setup automatic backups for a server.
But most of it can also be used to backup your clients.
An addition to borg</code> I use the wrapper borgmatic</a> which will remove any custom scripts
needed around borg</code>.
Everything related to backups will be stored in a single configuration file that configures borgmatic</code> which will
then perform borg</code> operations.</p>
As my homeserver is an Arch Linux machine this guide is specific for Arch, but as most distributions nowadays are
also systemd based there should not be much of a difference than the used package manager or the versions of some packages.</p>
Install needed tools</a></h2>
As both borg</code> and borgmatic</code> are in the official repositories they are easily installed with:</p>
pacman</span> -</span>S</span> borg</span> borgmatic</span></span></code></pre>Configure borgmatic</a></h2>
Borgmatic is configured in a single configuration file that will be stored in /etc/borgmatic/config.yaml</code>.
You can also use any other location and provide the path via -c path/to/my/config</code>.
A minimal configuration could be like this:</p>
l</span>ocation</span>:</span></span>
    s</span>ource_directories</span>:</span></span>
        -</span> /</span>home</span></span>
        -</span> /</span>mnt/storage</span></span>
</span>
    r</span>epositories</span>:</span></span>
        -</span> u</span>ser@my-backup-host:/mnt/backups/server</span></span>
</span>
r</span>etention</span>:</span></span>
    k</span>eep_daily</span>:</span> 7</span></span>
    k</span>eep_weekly</span>:</span> 4</span></span>
    k</span>eep_monthly</span>:</span> 6</span></span>
</span>
s</span>torage</span>:</span></span>
    e</span>ncryption_passphrase</span>:</span> "</span>my-secret-1337</span>"</span></span>
    c</span>ompression</span>:</span> z</span>std,22</span></span>
</span>
c</span>onsistency</span>:</span></span>
    c</span>hecks</span>:</span></span>
        -</span> r</span>epository</span></span>
        -</span> a</span>rchives</span></span></code></pre>
This means backup everything in the directories home</code> and mnt/storage</code> to /mnt/backups/server</code> on the host
my-backup-host</code>.
We also want to keep 7 daily backups, 4 weekly backups and 6 monthly backups.
The password to access backups is my-secret-1337</code> and the used compression is zstd</code> with the highest compression
(ranges from 0 (almost no compression, but fast) to 22 (the highest compression, but slow)).
Read more about the compression options borg provides here</a>.</p>
Setup remote server and ssh access</a></h2>
In this case we will backup to a remote server via ssh.
Therefore generate an ssh key pair on the server you want to backup with ssh-keygen</code>.
And copy the public key to the remote server with ssh-copy-id user@my-backup-host</code>.
After that you should be able to login to the remote backup server.</p>
You also need borg</code> installed on the remote machine (in my case also Arch Linux).
As the backups created by borg are stored encrypted you can use the same backup server for multiple clients.
But please mind that the backups are only encrypted if specified at the initial creation of the repository, which we will
handle now.</p>
Setup borg repository</a></h2>
Borg provides different encryption models which are documented here</a>.
In my case I want encrypted and</strong> authenticated backups these options are also the fastest on most
modern AMD/Intel CPUs (as the documentation states).
As we use borgmatic</code> as a borg</code> wrapper we use borgmatic init</code> instead borg borg init</code>.</p>
borgmatic</span> init</span> -</span>e</span> repokey-blake2</span></span></code></pre>Start the initial backup</a></h2>
Backups can now be started with borgmatic</code>.
If you want further information like which file is currently backed up or some statistics at the end use
borgmatic --files --stats</code>.</p>
Automate backups with systemd</a></h2>
These systemd units are based on the offical</a> ones from borgmatic.
I prefer systemd-timers over a simple cron daemon for two reasons:</p>

Systemd will ensure that the job will be executed even if the system is down at the specific time the job was scheduled.</li>
Systemd will ensure all logs are written into the journal</code>. With that they easy to inspect via journalctl -u borgmatic</code>.</li>
</ol>
Store this unit as borgmatic.service</code> in /etc/systemd/system</code>.</p>
[Unit]</span></span>
Description=borgmatic backup</span></span>
Wants=network-online.target</span></span>
After=network-online.target</span></span>
ConditionACPower=true</span></span>
</span>
[Service]</span></span>
Type=oneshot</span></span>
</span>
LockPersonality=true</span></span>
MemoryDenyWriteExecute=no</span></span>
NoNewPrivileges=yes</span></span>
PrivateDevices=yes</span></span>
PrivateTmp=yes</span></span>
ProtectClock=yes</span></span>
ProtectControlGroups=yes</span></span>
ProtectHostname=yes</span></span>
ProtectKernelLogs=yes</span></span>
ProtectKernelModules=yes</span></span>
ProtectKernelTunables=yes</span></span>
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK</span></span>
RestrictNamespaces=yes</span></span>
RestrictRealtime=yes</span></span>
RestrictSUIDSGID=yes</span></span>
SystemCallArchitectures=native</span></span>
SystemCallFilter=@system-service</span></span>
SystemCallErrorNumber=EPERM</span></span>
ProtectSystem=full</span></span>
</span>
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW</span></span>
</span>
Nice=19</span></span>
CPUSchedulingPolicy=batch</span></span>
IOSchedulingClass=best-effort</span></span>
IOSchedulingPriority=7</span></span>
IOWeight=100</span></span>
</span>
Restart=no</span></span>
LogRateLimitIntervalSec=0</span></span>
</span>
ExecStartPre=sleep 1m</span></span>
ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" borgmatic --syslog-verbosity 1</span></span></code></pre>
Reload all units with systemctl daemon-reload</code> and test the unit with systemctl start borgmatic</code>.
You should see that a backup is running via journalctl -u borgmatic -f</code>.</p>
If everything is running as expected create a borgmatic.timer</code> with the following content.
It must be stored in the same directory as the service file (/etc/systemd/system</code>).</p>
[Unit]</span></span>
Description=Run borgmatic backup</span></span>
</span>
[Timer]</span></span>
OnCalendar=*-*-* 4:00:00</span></span>
Persistent=true</span></span>
</span>
[Install]</span></span>
WantedBy=timers.target</span></span></code></pre>
OnCalendar</code> uses the format WeekDay Year-Month-Day Hour:Minute:Second</code> so if you want a backup running every day at
2 am: *-*-* 2:00:00</code>.
If you want to run multiple backups simply add addition OnCalendar</code> entries to the timer.</p>
Enable the timer with systemctl enable --now borgmatic.timer</code> and check its status:</p>
systemctl</span> list-timers</span></span>
NEXT</span>                         LEFT</span>     LAST</span>                         PASSED</span>       UNIT</span>                         ACTIVATES</span></span>
Mon</span> 2021-04-19</span> 00:00:00</span> CEST</span> 8h</span> left</span>  Sun</span> 2021-04-18</span> 00:00:10</span> CEST</span> 15h</span> ago</span>      shadow.timer</span>                 shadow.service</span></span>
Mon</span> 2021-04-19</span> 00:40:11</span> CEST</span> 8h</span> left</span>  Mon</span> 2021-04-12</span> 00:32:56</span> CEST</span> 6</span> days</span> ago</span>   fstrim.timer</span>                 fstrim.service</span></span>
Mon</span> 2021-04-19</span> 04:00:00</span> CEST</span> 12h</span> left</span> n/a</span>                          n/a</span>          borgmatic.timer</span>              borgmatic.service</span></span>
Mon</span> 2021-04-19</span> 12:59:22</span> CEST</span> 21h</span> left</span> Sun</span> 2021-04-18</span> 12:59:22</span> CEST</span> 2h</span> 46min</span> ago</span> systemd-tmpfiles-clean.timer</span> systemd-tmpfiles-clean.service</span></span>
</span>
4</span> timers</span> listed.</span></span></code></pre>
Your server will now create a backup every night at 4:00 am.</p>
Check existing backups</a></h2>
To list all existing backups use borgmatic list</code>.</p>
Restore a backup</a></h2>
In my opinion the easiest way to restore a backup is to use the borg mount</code> feature.
borg mount</code> allows you to mount a created backup to a specified directory.
On Arch you have to install python-llfuse</code> with pacman -S python-llfuse</code>.
For example:</p>
#</span> borgmatic list</span></span>
my-backup-server:my-repo:</span> Listing</span> archives</span></span>
Server-2021-04-16T16:36:35.431066</span>       Fri,</span> 2021-04-16</span> 16:36:36</span> [0a1b2933a2f1cab81ab164c7ef01370793c5d688710b1c29e94d8d1a8f22c414</span>]</span></span>
Server-2021-04-18T12:52:59.997607</span>       Sun,</span> 2021-04-18</span> 12:53:01</span> [9e2b0d1b635c36ede5653135821cdcc9c5181615c5cc0b4b8b06ef1e81ff29b8</span>]</span></span>
</span>
#</span> mount last backup to ~/mount</span></span>
borg</span> mount</span> my-backup-server:my-repo::Server-2021-04-18T12:52:59.997607</span>  ~/mount</span></span></code></pre>
You can now access the backup in ~/mount</code>.
Simply copy the files wherever you want.</p>
Conclusion</a></h1>
borg</code> and borgmatic</code> are perfect tools to automate backups.
There are advanced features like append only backups to make it impossible to delete backups if a server has been
compromised.
Overall I am very happy with by backup setup as it requires no maintenance and simply does its job.
And that's exactly how backups should be - just working and boring.</p>


Release tools with goreleaser
2021-03-31T00:00:00+00:00
I'm writing most of my own tools in go</a>.
Sometime ago I found a tool called goreleaser</a> which automates the release process of
software written in go.</p>
goreleaser</code> is pretty well documented, therefore this is just a quick overview how I use it.
It will do the following things for you:</p>

Generate a changelog with the latest commit since the last release.</li>
Build artifacts for different architectures (Linux/Mac/Windows).</li>
Create a GitHub/GitLab release and attach changelog and artifacts.</li>
</ul>
Initial setup</a></h2>


Install goreleaser</code> on your machine. (Check out the official docs here</a>)
I went with the go install github.com/goreleaser/goreleaser@latest</code> method.</p>
</li>

Run goreleaser init</code> to get a default .goreleaser.yml</code> which is used to configure goreleaser.</p>
</li>

Adjust the configuration.</p>
</li>
</ol>
b</span>efore</span>:</span></span>
  h</span>ooks</span>:</span></span>
    -</span> g</span>o mod tidy</span></span>
b</span>uilds</span>:</span></span>
  -</span> e</span>nv</span>:</span></span>
      -</span> C</span>GO_ENABLED=0</span></span>
    g</span>oos</span>:</span></span>
      -</span> l</span>inux</span></span>
      -</span> w</span>indows</span></span>
      -</span> d</span>arwin</span></span>
a</span>rchives</span>:</span></span>
  -</span> r</span>eplacements</span>:</span></span>
      d</span>arwin</span>:</span> D</span>arwin</span></span>
      l</span>inux</span>:</span> L</span>inux</span></span>
      w</span>indows</span>:</span> W</span>indows</span></span>
      a</span>md64</span>:</span> x</span>86_64</span></span>
c</span>hecksum</span>:</span></span>
  n</span>ame_template</span>:</span> '</span>checksums.txt</span>'</span></span>
s</span>napshot</span>:</span></span>
  n</span>ame_template</span>:</span> "</span>{{ .Tag }}-next</span>"</span></span>
c</span>hangelog</span>:</span></span>
  s</span>ort</span>:</span> a</span>sc</span></span>
  f</span>ilters</span>:</span></span>
    e</span>xclude</span>:</span></span>
      -</span> '</span>^docs:</span>'</span></span>
      -</span> '</span>^test:</span>'</span></span></code></pre>
I used this configuration to release go-jump</a>.</p>
Release via CI</a></h2>
To simplify the release process even more you can run gorelease</code> on GitHub Actions or GitLab CI.
This is an example for GitHub Actions.
Store it as .github/workflows/release.yaml</code> in your repository root.</p>
n</span>ame</span>:</span> g</span>oreleaser</span></span>
</span>
on</span>:</span></span>
  p</span>ush</span>:</span></span>
    t</span>ags</span>:</span></span>
      -</span> '</span>*</span>'</span></span>
</span>
j</span>obs</span>:</span></span>
  g</span>oreleaser</span>:</span></span>
    r</span>uns-on</span>:</span> u</span>buntu-latest</span></span>
    s</span>teps</span>:</span></span>
      -</span> n</span>ame</span>:</span> C</span>heckout</span></span>
        u</span>ses</span>:</span> a</span>ctions/checkout@v2</span></span>
        w</span>ith</span>:</span></span>
          f</span>etch-depth</span>:</span> 0</span></span>
      -</span> n</span>ame</span>:</span> S</span>et up Go</span></span>
        u</span>ses</span>:</span> a</span>ctions/setup-go@v2</span></span>
        w</span>ith</span>:</span></span>
          g</span>o-version</span>:</span> 1.16</span></span>
      -</span> n</span>ame</span>:</span> R</span>un GoReleaser</span></span>
        u</span>ses</span>:</span> g</span>oreleaser/goreleaser-action@v2</span></span>
        w</span>ith</span>:</span></span>
          v</span>ersion</span>:</span> l</span>atest</span></span>
          a</span>rgs</span>:</span> r</span>elease --rm-dist</span></span>
        e</span>nv</span>:</span></span>
          G</span>ITHUB_TOKEN</span>:</span> $</span>{{ secrets.GITHUB_TOKEN }}</span></span></code></pre>
Then create a new git tag and push it:</p>
git</span> tag</span> -</span>a</span> v0.1.0</span> -</span>m</span> "</span>My release</span>"</span></span>
git</span> push</span> origin</span> v0.1.0</span></span></code></pre>Conclusion</a></h2>
I really like how easy gorelease</code> is to use and it removes a repetitive task from my release process.
The documentation is perfect and the defaults are sane (the example above are more or less exactly the defaults!).
So I'm going to add goreleaser</code> to all of my projects!</p>


Arch Linux and Sony WH-1000MX4
2021-03-28T00:00:00+00:00
I recently got active noise-cancelling headphones (Sony WH-1000MX4</a>).
They work just fine with my Android phone but on my Linux notebook I had a bit of a fight to get them working.
I am running Arch Linux</a> on most of my machines, so this post is written with Arch specific
commands, but most of them (and the general idea) should also work on Ubuntu or Debian based distros.
Keep in mind that older versions of PulseAudio and/or PipeWire may not have the features described here!
So lets get started with the things I tried (if you are not interested in these just jump to
[PipeWire]({{< relref "linux-and-bluetooth.md#pipewire" >}} "link to PipeWire")).</p>
Failed experiments</a></h2>
At first we have to install the bluez</code> and bluez-utils</code> packages.
bluez-utils</code> will provide bluetoothctl</code> to interact via CLI with bluez</code>.
As I want to avoid unnecessary sources of error I'm not going for a graphical helper for now.</p>
All of this is (as always) documented in the arch wiki</a>.
The following is more or less exactly what can be found there.</p>

Start the Bluetooth service with systemctl start bluetooth.service</code></li>
Open bluetoothctl</code></li>
Power on the Bluetooth module with power on</code></li>
Scan for devices with scan on</code></li>
Wait for the headphones to show up. In my case they did not, so I got the Bluetooth MAC from my phone.</li>
Pair the headphones with pair <MAC></code></li>
Connect the headphones with connect <MAC></code></li>
</ol>
Until now everything was working as expected, but now the struggling started.
As I played some music the sound was just bad.
Therefore I checked the configuration in pavucontrol</code> and saw that the select profile was HSP/HDF</code>.
This is also documented in the Arch wiki</a>.
After a bit of research I now know that I want to use the A2DP</code> profile for crispy audio output, but the
internal microphone will only work with HSP/HDF</code>.
As my notebook got a decent internal microphone and I also got an external one this is no problem for my use case.
But keep this in mind if you need the internal mic of the Sony WH-1000MX4 or any other Bluetooth headphones.</p>
I tried a few more things mentioned in the troubleshooting guide</a> like:</p>
#</span> /etc/pulse/default.pa</span></span>
#</span> Let me use l2dp via bluetooth</span></span>
load-module</span> module-bluetooth-policy</span> auto_switch=</span>false</span></span></code></pre>#</span> /etc/bluetooth/main.conf</span></span>
[</span>General</span>]</span></span>
Disable</span>=</span>H</span>e</span>a</span>d</span>s</span>e</span>t</span></span>
MultiProfile</span>=</span>m</span>u</span>l</span>t</span>i</span>p</span>l</span>e</span></span></code></pre>
But none of them worked, and the sound was still just bad.
So I did some research about how all of these Bluetooth stuff actually works and discovered that the Sony WH-1000MX4
are using the LDAC</code> codec which is not</strong> supported by vanilla pulseaudio.
An alternative is APTx</code> which may be supported by most other headphones, as LDAC</code> is a proprietary Sony codec.
Also, you want to use a A2DP</code> sink instead of HSP/HDF</code> (mind the earlier described disadvantage of not working microphones!).
Therefore, it's time to revisit the Arch wiki and search for this specific codec.</p>
</p>
Migration to PipeWire</a></h2>
At this point I discovered</a> a
pulseaudio module</a> that adds support for APTx</code> and LDAC</code> codecs.
But it was deprecated</a> in January 2021 with a recommendation
to use vanilla PulseAudio or PipeWire.
I already tried PulseAudio earlier (and was annoyed) so I decided to give PipeWire a chance.
And PipeWire supports LDAC since release 0.3.19</code> 🎉!</p>


Many bluetooth improvements in LDAC, AptX-HD. AAC was
also added. Headsets should work better now.
-- https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/NEWS/</li>
</ul>
</blockquote>
There are 3 packages needed to get started: libldac</code>, pipewire</code> and pipewire-pulse</code>.
Install them with pacman -S libldac pipewire pipewire-pulse</code>.</p>
After that reboot your machine and connect the headphones with bluetoothctl</code> as described earlier.
I had to remove the headphones  with remove <MAC></code> and pair them again.
Now there should be the A2DP</code> option in pavucontrol</code> and the LDAC</code> codec.</p>
</p>
If you want to learn more about PipeWire (and its history) there is a recently published article on
LWN.net</a>.</p>
Getting the touch gestures to work</a></h2>
Thanks to Marcelo for pointing this out!</p>
As always this was already documented in the arch wiki</a>.
Simply create a new systemd user unit:</p>
~</span>/.config/systemd/user/mpris-proxy.service</span></span>
</span>
[</span>Unit</span>]</span></span>
Description</span>=</span>F</span>o</span>r</span>w</span>a</span>r</span>d</span> bluetooth</span> media</span> controls</span> to</span> MPRIS</span></span>
</span>
[</span>Service</span>]</span></span>
Type</span>=</span>s</span>i</span>m</span>p</span>l</span>e</span></span>
ExecStart</span>=</span>/</span>u</span>s</span>r</span>/</span>b</span>i</span>n</span>/</span>m</span>p</span>r</span>i</span>s</span>-</span>p</span>r</span>o</span>x</span>y</span></span>
</span>
[</span>Install</span>]</span></span>
WantedBy</span>=</span>d</span>e</span>f</span>a</span>u</span>l</span>t</span>.</span>t</span>a</span>r</span>g</span>e</span>t</span></span></code></pre>
Reload all units with systemd --user daemon-reload</code> and enable the unit with
systemctl --user enable --now mpris-proxy</code>.
Now all touch gestures and the wearing detection</code> should work!</p>
Conclusion</a></h2>
As I thought about getting Bluetooth headphones I have not given a thought that Bluetooth (with modern codecs) could be
problematic on Linux.
Especially as Sony decided to use their own LDAC</code> codec instead of something open (but I'm not even sure if there is
an open Bluetooth codec).</p>
So far I had no problems with PipeWire, but I am only running it for a few weeks now, so we will see how it holds up.
Keep in mind that PipeWire is not stable yet!
There might be updates that require configuration changes
(like this one</a>).</p>


Setup Wallabag
2021-02-28T00:00:00+00:00
I am subscribed to a lot of blog feeds, but (sadly) I don't have enough time to read all of them.
This is especially the case for rather long (but often interesting) posts.
Therefore it would be perfect to save these posts and read them in the near future.</p>
There are already existing solutions for this like Mozilla Pocket</a>.
But as always I want to host this service on my own server (I have to justify the server somehow).
A self-hosted solution is wallabag</a>.</p>

wallabag is a self-hostable PHP application allowing
you to not miss any content anymore. Click, save and read it when you can. It extracts content so that you can read it
when you have time.
-- https://github.com/wallabag/wallabag#what-is-wallabag</cite></p>
</blockquote>
Wallabag needs 2 things to work properly:</p>


A wallabag server to actually store the articles.</p>
</li>

The wallabag browser plugin Wallabagger</a> to add new articles to your backlog.</p>
</li>
</ul>
Additionally wallabag can be accessed via browser or one of the official apps (Android</a>
and iOS</a>).</p>
I am going to run the wallabag server as a container because it is easy to manage the configuration with git and
to migrate it to a new host (or share it with you).</p>
Installation with Docker</a></h2>
The following files are stored in this layout:</p>
.</span></span>
├──</span> data</span></span>
├──</span> docker</span></span>
│  </span> ├──</span> Dockerfile</span></span>
│  </span> └──</span> nginx</span></span>
│  </span>     ├──</span> nginx.conf</span></span>
│  </span>     └──</span> wallabag</span></span>
├──</span> docker-compose.yml</span></span>
└──</span> images</span></span></code></pre>
The data</code> and images</code> directories will be used to store persistent data like images and the scraped articles.</p>
The docker-compose.yml</code> is based on an example provided</a>
by the wallabag project.</p>
As I host the server at home this configuration is not hardened at all!
I can't recommend using</strong> this configuration to host this service on the public internet!</p>
So let's have a look at the main docker-compose</code> file.
We declare two containers, wallabag</code> itself and a mariadb</code> database.
There is also a nginx container to handle missing icons, but more about this later.</p>
v</span>ersion</span>:</span> '</span>3</span>'</span></span>
s</span>ervices</span>:</span></span>
  n</span>ginx</span>:</span></span>
      b</span>uild</span>:</span></span>
          c</span>ontext</span>:</span> .</span>/docker</span></span>
      p</span>orts</span>:</span></span>
          -</span> 0</span>.0.0.0:1337:1337</span></span>
      l</span>inks</span>:</span></span>
          -</span> w</span>allabag</span></span>
  w</span>allabag</span>:</span></span>
    i</span>mage</span>:</span> w</span>allabag/wallabag:2.4.1</span></span>
    e</span>nvironment</span>:</span></span>
      -</span> M</span>YSQL_ROOT_PASSWORD=wallaroot</span></span>
      -</span> S</span>YMFONY__ENV__DATABASE_DRIVER=pdo_mysql</span></span>
      -</span> S</span>YMFONY__ENV__DATABASE_HOST=db</span></span>
      -</span> S</span>YMFONY__ENV__DATABASE_PORT=3306</span></span>
      -</span> S</span>YMFONY__ENV__DATABASE_NAME=wallabag</span></span>
      -</span> S</span>YMFONY__ENV__DATABASE_USER=wallabag</span></span>
      -</span> S</span>YMFONY__ENV__DATABASE_PASSWORD=wallapass</span></span>
      -</span> S</span>YMFONY__ENV__DATABASE_CHARSET=utf8mb4</span></span>
      -</span> S</span>YMFONY__ENV__MAILER_HOST=127.0.0.1</span></span>
      -</span> S</span>YMFONY__ENV__MAILER_USER=~</span></span>
      -</span> S</span>YMFONY__ENV__MAILER_PASSWORD=~</span></span>
      -</span> S</span>YMFONY__ENV__FROM_EMAIL=wallabag@example.com</span></span>
      -</span> S</span>YMFONY__ENV__DOMAIN_NAME=http://DOMAIN</span></span>
    v</span>olumes</span>:</span></span>
      -</span> .</span>/images:/var/www/wallabag/web/assets/images</span></span>
  d</span>b</span>:</span></span>
    i</span>mage</span>:</span> m</span>ariadb:10</span></span>
    e</span>nvironment</span>:</span></span>
      -</span> M</span>YSQL_ROOT_PASSWORD=wallaroot</span></span>
    v</span>olumes</span>:</span></span>
      -</span> .</span>/data:/var/lib/mysql</span></span>
  r</span>edis</span>:</span></span>
    i</span>mage</span>:</span> r</span>edis:alpine</span></span></code></pre>
The nginx is needed because of CORS</a> errors.
A short look on the GitHub issue tracker showed that this is a known problem</a>.
A fix</a> is to use a nginx as reverse proxy
and add the following header add_header 'Access-Control-Allow-Origin' 'https://example.com'</code>.
And this is exactly what I did.</p>
The nginx</code> is based on the upstream image, but we will run a quick apt update && apt upgrade</code> to ensure
we have the latest packages installed.
Then we copy the base config nginx.conf</code> and our site config wallabag</code> into the container.</p>
FROM</span> nginx:1.19 </span>AS</span> nginx</span></span>
ENV</span> TZ=Etc/UTC</span></span>
</span>
RUN</span>  export DEBIAN_FRONTEND=noninteractive \</span></span>
    && apt-get update \</span></span>
    && apt-get dist-upgrade -y --no-install-recommends</span></span>
</span>
COPY</span> nginx/wallabag /etc/nginx/sites-available/wallabag</span></span>
COPY</span> nginx/nginx.conf /etc/nginx/nginx.conf</span></span>
</span>
RUN</span> mkdir /etc/nginx/sites-enabled/ \</span></span>
    && ln -s /etc/nginx/sites-available/wallabag /etc/nginx/sites-enabled/wallabag \</span></span>
    && rm /etc/nginx/conf.d/default.conf</span></span>
</span>
EXPOSE</span> 1337</span></span></code></pre>
The main configuration of our nginx is basic, as I do not intent to expose this service
to the public internet.
I also do not care about error or access logs.</p>
events {}</span></span>
</span>
http {</span></span>
    server_tokens off;</span></span>
    sendfile on;</span></span>
    keepalive_timeout 65;</span></span>
    tcp_nodelay on;</span></span>
    include /etc/nginx/mime.types;</span></span>
</span>
    access_log off;</span></span>
    error_log /dev/null;</span></span>
</span>
    include /etc/nginx/conf.d/*.conf;</span></span>
    include /etc/nginx/sites-enabled/*;</span></span>
}</span></span>
</span></code></pre>
The site configuration exposes the server on port 1337</code> and adds the missing CORS</code> header described earlier.</p>
server</span> {</span></span>
        listen</span> 1337</span>;</span></span>
        #</span>server_name wallabag.foo.bar;</span></span>
</span>
	location</span> /</span> {</span></span>
		proxy_pass</span> http://wallabag</span>;</span></span>
		proxy_set_header</span> X-Forwarded-Host</span> $</span>server_name</span>;</span></span>
                proxy_set_header</span> X-Forwarded-Proto</span> https</span>;</span></span>
                proxy_set_header</span> X-Forwarded-For</span> $</span>remote_addr</span>;</span></span>
	}</span></span>
</span>
        add_header</span> '</span>Access-Control-Allow-Origin</span>'</span> '</span>http://DOMAIN</span>'</span></span>
</span>
}</span></span></code></pre>Deployment</a></h2>
Now to the easy part.
Run docker-compose up</code> to start the wallabag stack.
Wallabag should now be online, and you can pair the browser plugin with the instance.</p>


Blog deployment with GitLab CI
2021-02-13T00:00:00+00:00
As stated earlier</a> I use a combination of technologies to
run this blog.</p>
This blog is created with hugo</a> (a static site generator) and uses the
papermod</a> theme.
Additional technologies that are used:</p>

GitLab as git remote</li>
GitLab CI to build container images (development and production)</li>
Docker to run the images (development on my home server and production on a machine on the public internet)</li>
Watchtower</a> to check for new images every ~5 minutes</li>
Caddy</a> to handle TLS certificates (only for production)</li>
A VM on Hetzner Cloud</a> used as a host for multiple containers</li>
</ul>
GitLab</a></h2>
At first the development and production hosts need access to the container registry.
Therefore:</p>

Create a new deploy_token</code> official docs</a></li>
</ol>
</p>

Login on both the development/production docker host using the created user and password.</li>
</ol>
docker</span> login</span> <</span>your.gitlab.instance:456</span>7></span></span></code></pre>CI</a></h2>
Here are a few things to mention.</p>
The variable GIT_SUBMODULE_STRATEGY: recursive</code> is needed as hugo themes are added as git
submodules</a>.
Therefore all themes must be cloned before</strong> the static page can be build.</p>
I use two stages to build containers for production and development with a single CI run.
The build-containers-production</code> stage will only be executed if the $CI_COMMIT_BRANCH</code> equals main</code> as
only code/articles on this branch are ready to be published.</p>
The second stage build-container-dev</code> is used for a development deployment on my home server.
This stage will be executed on every branch, so I can always check if the article I am currently working on looks good.
All of this could be achieved by using hugo server -D</code> on my local machine, but this is not as much fun
as playing around with GitLab CI, Docker and my home server (which will be covered in another post in the near future).</p>
The script</code> part in both stages are more or less the same.
For production</code> the created image will be tagged with production</code> and for development</code> with dev</code>.</p>
An important change is cd myblog && sed -i 's|https://sterba.dev|http://192.168.0.2:9337|g' config.yaml && cd ..</code>.
Without changing the baseURL</code> all links in the dev image will direct to https://sterba.dev/...</code>.</p>
Please mind that this will not</strong> work if you are using hugo drafts</code>.
Drafts will not be rendered with this build script.
This is no problem for my workflow as only finished posts are on the main</code> branch.</p>
v</span>ariables</span>:</span></span>
  G</span>IT_SUBMODULE_STRATEGY</span>:</span> r</span>ecursive</span></span>
</span>
s</span>tages</span>:</span></span>
  -</span> b</span>uild-containers-production</span></span>
  -</span> b</span>uild-container-dev</span></span>
</span>
b</span>efore_script</span>:</span></span>
  -</span> d</span>ocker info</span></span>
</span>
b</span>uild_image_production</span>:</span></span>
  s</span>tage</span>:</span> b</span>uild-containers-production</span></span>
  r</span>ules</span>:</span></span>
    -</span> i</span>f</span>:</span> '</span>$CI_COMMIT_BRANCH == "main"</span>'</span></span>
      w</span>hen</span>:</span> a</span>lways</span></span>
    -</span> w</span>hen</span>:</span> n</span>ever</span></span>
</span>
  s</span>cript</span>:</span></span>
    -</span> d</span>ocker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY</span></span>
    -</span> d</span>ocker build -t your.gitlab.com:4567/andre/blog:production -f docker/Dockerfile .</span></span>
    -</span> d</span>ocker push your.gitlab.com:4567/andre/blog:production</span></span>
</span>
b</span>uild_image_test</span>:</span></span>
  s</span>tage</span>:</span> b</span>uild-container-dev</span></span>
</span>
  s</span>cript</span>:</span></span>
    -</span> s</span>ed -i 's|https://sterba.dev|http://192.168.0.2:9337|g' myblog/config.yaml</span></span>
    -</span> d</span>ocker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY</span></span>
    -</span> d</span>ocker build -t your.gitlab.com:4567/andre/blog:dev -f docker/Dockerfile .</span></span>
    -</span> d</span>ocker push your.gitlab.com:4567/andre/blog:dev</span></span></code></pre>Docker</a></h2>
The blog container is a multistage build because I only want the generated static content in my webserver
container (nginx</code>).
Therefore the first container (based on debian:buster-slim</code>) installs the current version of hugo</code>, copies
the blog in an app</code> directory and generates the static content.</p>
The second container is a basic nginx</code> with a simple configuration which will be displayed later.
With COPY --from=build /app/public/ /opt/public/</code> the previously generated content is copied into the
configured webserver root directory of nginx</code>.</p>
Always</strong> update the packages in your base images (in this case debian</code>)! 

Because the base images are not rebuild every few days they do not contain the newest version of a specific package.
So you could go to production with vulnerable or old packages.</p>
FROM</span> debian:buster-slim </span>AS</span> build</span></span>
</span>
ENV</span> HUGO_VERSION=0.80.0</span></span>
ADD</span> https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz /tmp/</span></span>
RUN</span> tar -xf /tmp/hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz -C /usr/bin/</span></span>
</span>
COPY</span> ./myblog /app</span></span>
</span>
RUN</span> hugo --source=/app/</span></span>
</span>
FROM</span> nginx:stable </span>AS</span> nginx</span></span>
ENV</span> TZ=Etc/UTC</span></span>
</span>
RUN</span>  export DEBIAN_FRONTEND=noninteractive \</span></span>
    && apt-get update \</span></span>
    && apt-get dist-upgrade -y --no-install-recommends</span></span>
</span>
COPY</span> docker/nginx/blog /etc/nginx/sites-available/blog</span></span>
COPY</span> docker/nginx/nginx.conf /etc/nginx/nginx.conf</span></span>
</span>
RUN</span> mkdir /etc/nginx/sites-enabled/ \</span></span>
    && ln -s /etc/nginx/sites-available/blog /etc/nginx/sites-enabled/blog \</span></span>
    && rm /etc/nginx/conf.d/default.conf</span></span>
</span>
COPY</span> --from=build \</span></span>
    /app/public/ /opt/public/</span></span>
</span>
EXPOSE</span> 9337</span></span></code></pre>
The nginx</code> site configuration is pretty simple.
The root</code> is the /opt/public</code> folder we used earlier to copy our build artifacts to.
As hugo</code> and papermod</code> generate a custom 404</code> page all 404</code>'s should be directed to the generated page instead
of the default nginx page.</p>
server {</span></span>
        listen 9337;</span></span>
        root /opt/public/;</span></span>
        index index.html;</span></span>
        error_page    404    /404.html;</span></span>
        location / {</span></span>
                try_files $uri $uri/ =404;</span></span>
        }</span></span>
}</span></span></code></pre>
The main configuration of nginx</code> is also pretty straight forward.
It simply loads the previously created site configuration.
I disabled any logging as I don't care about logs because I don't want any information about my readers
(yes, I actually care about your privacy!).</p>
events {}</span></span>
</span>
http {</span></span>
    server_tokens off;</span></span>
    sendfile on;</span></span>
    keepalive_timeout 65;</span></span>
    tcp_nodelay on;</span></span>
    include /etc/nginx/mime.types;</span></span>
</span>
    access_log off;</span></span>
    error_log /dev/null;</span></span>
</span>
    include /etc/nginx/conf.d/*.conf;</span></span>
    include /etc/nginx/sites-enabled/*;</span></span>
}</span></span></code></pre>Deployment</a></h2>
The deployment process contains two different components.
First there is docker-compose</code> which handles the container build previously.
Additionally, there is a caddy</code> server which acts as reverse proxy (only used for production).</p>
Docker Compose</a></h3>
Although there are two deployment environments their configuration is the same. 

Only the used tag</code> is different.
There is not only the image of the blog we build earlier, but also a
watchtower</a> container.
Watchtower</a> can be used to monitor the upstream registry for a new image.
So after every CI run for development</code> watchtower will pull the new image, shutdown the running container and restart
it with the new image.</p>
But there are other containers running on the host and by default watchtower</code> will try to update all containers
currently running on a host.
A fix is to provide the --label-enable</code> command flag.
With this flag only containers with the label com.centurylinklabs.watchtower.enable</code> set to true</code> will be updated
by watchtower</code>.</p>
Replace the port configuration 10.13.37.1</code> with the ip of your machine or use 0.0.0.0</code> to bind to any address.</p>
v</span>ersion</span>:</span> "</span>3</span>"</span></span>
s</span>ervices</span>:</span></span>
    a</span>ndre-blog</span>:</span></span>
        i</span>mage</span>:</span> y</span>our.gitlab.com/andre/blog:<production or dev></span></span>
        c</span>ontainer_name</span>:</span> a</span>ndre-blog</span></span>
        r</span>estart</span>:</span> a</span>lways</span></span>
        l</span>abels</span>:</span></span>
          -</span> "</span>com.centurylinklabs.watchtower.enable=true</span>"</span></span>
        p</span>orts</span>:</span></span>
            -</span> 1</span>92.168.0.2:9337:9337</span></span>
    w</span>atchtower</span>:</span></span>
      i</span>mage</span>:</span> c</span>ontainrrr/watchtower</span></span>
      c</span>ommand</span>:</span> --</span>label-enable --interval 30</span></span>
      v</span>olumes</span>:</span></span>
        -</span> /</span>var/run/docker.sock:/var/run/docker.sock</span></span>
        -</span> ~</span>/.docker/config.json:/config.json</span></span></code></pre>Caddy</a></h3>
I mostly rely on Caddy</a> as web server because it has an easy configuration and provides
TLS certificates with Let's Encrypt</a> out of the box.</p>
There are two sub-settings in the main config:</p>


The configuration for the reverse-proxy</a>
module which simply forwards all traffic to the container created above.</p>
</li>

Some additional headers.</p>
</li>
</ol>
sterba.dev {</span></span>
  reverse_proxy http://localhost:9337 {</span></span>
  }</span></span>
  header {</span></span>
    Strict-Transport-Security max-age=31536000;</span></span>
    Referrer-Policy no-referrer-when-downgrade</span></span>
  }</span></span>
}</span></span></code></pre>Conclusion</a></h2>
As you can see the setup is pretty straight forward and requires not that much configuration.
With this setup I have a complete CI/CD stack to build, test (in a development environment) and deploy the blog.
The release process of a finished post is no work at all.
I simply need to push a post to the main</code> branch and everything else will be handled by the components presented
in this post.</p>


GitLab as Terraform Backend
2021-01-23T00:00:00+00:00
I manage a few servers with some friends.
We mostly rely on Terraform to update/change DNS records or create new VMs.</p>
Terraform uses a state</a> to store the last applied configuration of
the managed infrastructure.
The default location for this state is your local filesystem, but if you manage your infrastructure with multiple
persons it must be shared.
This can be problematic as it also contains secrets, like api tokens, so it should not be accessible for everyone.</p>
Terraform uses backends</a> to provide a solution for storing the
state somewhere remote instead of your local disk.
If you have an account on a GitLab</a> instance you already have access to a remote storage
for your state which can be used as http backend</a>.
The backend also handles locking, so you can't execute terraform apply</code> from different machines at the same time.
You can find it also the official documentation, but I will provide you with a quick start.</p>


Create a personal API token for your GitLab account with the scope api</code>.</p>
</li>

Add the following settings to your main.tf</code>.</p>
</li>
</ol>
terraform {</span></span>
  backend "http" {</span></span>
  }</span></span>
}</span></span></code></pre>

Run terraform init</code> with the following arguments and replace

USERNAME</code></li>
API-TOKEN</code></li>
PROJECT-ID</code></li>
STATE-NAME</code></li>
</ul>
</li>
</ol>
terraform</span> init</span> \</span></span>
    -</span>backend-config=</span>"</span>address=https://gitlab.com/api/v4/projects/<PROJECT-ID>/terraform/state/<STATE-NAME></span>"</span> \</span></span>
    -</span>backend-config=</span>"</span>lock_address=https://gitlab.com/api/v4/projects/<PROJECT-ID>/terraform/state/<STATE-NAME>/lock</span>"</span> \</span></span>
    -</span>backend-config=</span>"</span>unlock_address=https://gitlab.com/api/v4/projects/<PROJECT-ID>/terraform/state/<STATE-NAME>/lock</span>"</span> \</span></span>
    -</span>backend-config=</span>"</span>username=USERNAME</span>"</span> \</span></span>
    -</span>backend-config=</span>"</span>password=API-TOKEN</span>"</span> \</span></span>
    -</span>backend-config=</span>"</span>lock_method=POST</span>"</span> \</span></span>
    -</span>backend-config=</span>"</span>unlock_method=DELETE</span>"</span> \</span></span>
    -</span>backend-config=</span>"</span>retry_wait_min=5</span>"</span></span></code></pre>

Run terraform plan</code> or terraform apply</code> and the remote state should be used.</li>
</ol>
There are other ways to use a remote state with Terraform like
a Postgres database</a> or
S3</a>, but
if you have access to a GitLab instance or an account on gitlab.com</a>
this is a simple way to get the advantages of a remote Terraform state.</p>


Terraform and env variables
2021-01-08T00:00:00+00:00
If you work with Terraform</a> you might know the struggle setting the right environment
variables.
Especially if you work with different Terraform configurations on a daily basis.</p>
I found 2 tools (or a tool and a script) to make this much easier.
The first is direnv</a> which automagically sources env variables stored in a .envrc</code> file
if you change in a managed directory.</p>
$</span> mkdir</span> direnv-test</span></span>
$</span> echo</span> "</span>export mykey=wasd</span>"</span> >></span> .envrc</span></span>
$</span> cd</span> ..</span> &&</span> cd</span> direnv-test</span></span>
</span>
#</span> direnv will show this error</span></span>
direnv:</span> error</span> /home/andre/tmp/direnv-test/.envrc</span> is</span> blocked.</span></span>
Run</span> `</span>direnv</span> allow</span>`</span> to</span> approve</span> its</span> content</span></span>
</span>
$</span> direnv</span> allow</span></span>
</span>
$</span> cd</span> ..</span> &&</span> cd</span> direnv-test</span></span>
direnv:</span> loading</span> ~/tmp/direnv-test/.envrc</span></span>
direnv:</span> export</span> +mykey</span></span>
</span>
$</span> echo</span> $</span>mykey</span></span>
wasd</span></span></code></pre>
And that's it!
Simply install direnv</code>, create a .envrc</code>, run direnv allow</code> and all the needed env variables will be set when you
change in the directory.</p>
I mentioned 2 tools at the beginning and the second tool is a little script I use to get API tokens
out of my password store.
My password store is powered by gopass</a> therefore it is pretty easy to script around it.</p>
The setup is based on this blog post of Shibumi</a>.
Thanks for the idea!</p>
Store this script in the same directory as your Terraform code.</p>
#!</span>/bin/sh</span></span>
</span>
cloud_api_key</span>=</span>$(</span>gopass</span> show</span> -</span>-password</span> tokens/cloud-project-token</span>)</span></span>
</span>
echo</span> "</span>{ </span>\"</span>cloud_api_key</span>\"</span>: </span>\"</span>$</span>{</span>cloud_api_key</span>}</span>\"</span> }</span>"</span></span></code></pre>
It will be called by the Terraform module external</code>.</p>
data "external" "cloud_api_key" {</span></span>
  program = ["${path.module}/fetch-key.sh"]</span></span>
}</span></span>
</span>
provider "mycloud" {</span></span>
  token = data.external.cloud_api_key.result.cloud_api_key</span></span>
}</span></span></code></pre>
Terraform will use this script to get the needed token and that's it.</p>
This is so much better than going back in your shell history or use reverse search (and sometimes apply the wrong
config to the wrong infrastructure).</p>


SSH Agents
2021-01-03T00:00:00+00:00
With every installation of a new system I struggle setting up an ssh-agent.
Although it only requires 4 simple steps I somehow manage to always miss at least one of them.</p>
I use a systemd user unit to start the ssh-agent as I don't want to manage services with
shell scripts or in my sway</a> config, if avoidable.</p>
This is more or less exactly what can be found in the Arch wiki</a>.</p>
Step 1</a></h3>
Create systemd user unit in ~/.config/systemd/user/ssh-agent.service</code>.</p>
[Unit]</span></span>
Description=SSH key agent</span></span>
</span>
[Service]</span></span>
Type=simple</span></span>
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket</span></span>
# DISPLAY required for ssh-askpass to work</span></span>
Environment=DISPLAY=:0</span></span>
ExecStart=/usr/bin/ssh-agent -D -a $SSH_AUTH_SOCK</span></span>
</span>
[Install]</span></span>
WantedBy=default.target</span></span></code></pre>Step 2</a></h3>
Add SSH_AUTH_SOCK DEFAULT="${XDG_RUNTIME_DIR}/ssh-agent.socket"</code> to .pam_environment</code>.</p>
Step 3</a></h3>
Add AddKeysToAgent yes</code> to your .ssh/config</code> to actually use the ssh-agent you set up previously.</p>
Step 4</a></h3>
Let systemd pick up the new unit with systemctl --user daemon-reload</code> and
start the ssh-agent with systemctl start --user ssh-agent</code>.
You should also enable the service with systemctl enable --user ssh-agent</code> (Thank you, Neil!).</p>
That's it!
If something is not working properly now restart your user session (logout and login again).
Now you should only be prompted once for your ssh-key passphrase at the start of your session.</p>


Init
2020-12-31T00:00:00+00:00
Welcome to my blog!</p>
I've wanted to start a blog for a long time, but I've never done it because of unclear reasons.
So here it is!</p>
technical details</a></h2>
The blog is created with hugo</a> and the papermod</a> theme.
I use GitLab-CI to build a container that is pushed into the GitLab container registry.
Caddy</a> is used to serve the blog (as reverse proxy in front of the container) and
to handle TLS.</p>
more content</a></h2>
I plan to post some of the more interesting problems in my daily life as a user of
technology.
So be ready to read about my struggles of getting things to run or program things that
I absolutely need!</p>
about me</a></h2>
I'm a computer science student and an enthusiast for open source software.
I'm a Linux user an my favorite desktop distribution is Arch Linux</a>.
For servers I mostly rely on Debian</a>.</p>
I'm interested in distributed systems, (cloud) infrastructure, privacy and independent hosting of services.
Therefore I host most of the services at home or on several machines on the internet.</p>

André Sterba

Nix Dev Shells

Hash Tables

Brewfile

Replacing tmux with Ghostty

Simple Container Deployments on Virtual Machines

Container Fundamentals

Basic In-Memory Cache in Go

Eviction</a></h2> There are multiple ways to evict items from the cache. Well, there are also multiple different algorithms to evict, but for now let's keep it simple and stick with the original requirements.</p> We can evict expired items on every access.</li>

Sign your Git commits

Buildpacks

Improve your Go Code with Benchmarks

Why you should update your dependencies automatically

Use git worktree to work with multiple branches

Organize your Semester with Kanban inspired Methods

Speaking at OpenRheinMain 2022

Build LaTeX with GitLab-CI

Uptime monitoring with Prometheus and Grafana

GitOps with upstream-watch

Update process</a></h2> This results in the following update process for our example service with integration of renovate or dependabot:</p> Dependabot or Renovate opens an MR/PR with an update.</li> Review change and merge to main</code>.</li> upstream-watch</code> will pull the changes.</li>

Go Generics

Install Arch Linux

Overview</a></h2> Just a quick overview about the features before we get started.</p> ext4 Filesystem with lvm</li> encrypted root partition next to unencrypted /boot</code></li>

Neovim LSP

Build a simple VPN with WireGuard

Using macOS as a Linux user