Jekyll2024-03-28T18:02:21+00:00https://paperless.blog/feed.xmlPaperlessVictor EngmarkAWS Azure login interactive profile selection function2023-11-26T00:00:00+00:002023-11-26T00:00:00+00:00https://paperless.blog/aws-azure-login-interactive-profile-selection-function<p>I’ve created a wrapper for <code class="language-plaintext highlighter-rouge">aws-azure-login</code> to let you interactively select the
profile you want before logging in. I have a bunch of profiles, I can never
remember what they are called, and some of them have short session lifetimes, so
this has been a lifesaver.</p>
<h2 id="features">Features</h2>
<ul>
<li>Uses local variables to avoid polluting the namespace.</li>
<li>Forwards any extra arguments (such as <code class="language-plaintext highlighter-rouge">--no-prompt</code>) to the underlying
<code class="language-plaintext highlighter-rouge">aws-ozure-login</code> seamlessly.</li>
<li>Lets you select from profiles without a <code class="language-plaintext highlighter-rouge">source_profile</code>.</li>
<li>Logs in by running the original <code class="language-plaintext highlighter-rouge">aws-azure-login</code> command under the hood.</li>
<li>Sets <code class="language-plaintext highlighter-rouge">AWS_PROFILE</code>.</li>
<li>If there are any sub-profiles of the selected profile, it lets you select from
them to set <code class="language-plaintext highlighter-rouge">AWS_PROFILE</code> after logging in. This is necessary at work, where
we use sub-profiles to assume a different role from the login step.</li>
</ul>
<h2 id="dependencies">Dependencies</h2>
<ul>
<li><a href="https://www.npmjs.com/package/aws-azure-login">aws-azure-login</a></li>
<li><a href="https://www.gnu.org/software/bash/">Bash</a></li>
<li><a href="https://github.com/BurntSushi/ripgrep">RipGrep</a> (but feel free to convert it
to use GNU Grep)</li>
</ul>
<h2 id="the-function">The function</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws-azure-login<span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"Select profile, or press Ctrl-c to cancel."</span>
<span class="k">select </span>AWS_PROFILE <span class="k">in</span> <span class="si">$(</span>
rg <span class="nt">--multiline</span> <span class="nt">--only-matching</span> <span class="nt">--pcre2</span> <span class="s1">'(?x)
(?<=^\[profile\ ) # Profile line prefix
([^\]]+) # Capture profile name
(?=\]$) # Profile line suffix
(?!\]\nsource_profile=) # Profile does *not* have a parent
'</span> ~/.aws/config
<span class="si">)</span><span class="p">;</span> <span class="k">do
</span><span class="nb">export </span>AWS_PROFILE
<span class="nb">break
</span><span class="k">done
</span><span class="nb">command </span>aws-azure-login <span class="nt">--profile</span><span class="o">=</span><span class="s2">"</span><span class="nv">$AWS_PROFILE</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="nb">local </span>sub_profiles
<span class="nb">mapfile</span> <span class="nt">-t</span> sub_profiles < <<span class="o">(</span>
rg <span class="nt">--multiline</span> <span class="nt">--only-matching</span> <span class="nt">--pcre2</span> <span class="s1">'(?x)
(?<=^\[profile\ ) # Profile line prefix
([^\]]+) # Capture profile name
(?=\]$) # Profile line suffix
(?=\]\nsource_profile= # Profile *does* have a parent …
'</span><span class="s2">"</span><span class="nv">$AWS_PROFILE</span><span class="s2">"</span><span class="s1">') # … with the given name
'</span> ~/.aws/config
<span class="o">)</span>
<span class="k">if</span> <span class="o">((</span><span class="s2">"</span><span class="k">${#</span><span class="nv">sub_profiles</span><span class="p">[@]</span><span class="k">}</span><span class="s2">"</span> <span class="o">!=</span> 0<span class="o">))</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Select sub-profile, or press Ctrl-c to keep </span><span class="k">${</span><span class="nv">AWS_PROFILE</span><span class="k">}</span><span class="s2">."</span>
<span class="k">select </span>AWS_PROFILE <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">sub_profiles</span><span class="p">[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
</span><span class="nb">break
</span><span class="k">done
fi</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="setup">Setup</h2>
<ol>
<li>Copy the function into your <code class="language-plaintext highlighter-rouge">~/.bashrc</code>.</li>
<li>Restart your shell.</li>
</ol>
<h2 id="usage">Usage</h2>
<p>Run <code class="language-plaintext highlighter-rouge">aws-azure-login</code> and follow the instructions.</p>Victor EngmarkI’ve created a wrapper for aws-azure-login to let you interactively select the profile you want before logging in. I have a bunch of profiles, I can never remember what they are called, and some of them have short session lifetimes, so this has been a lifesaver.Being open about openness2023-09-24T00:00:00+00:002023-09-24T00:00:00+00:00https://paperless.blog/being-open-about-openness<p>tl;dr: Being open about what you’re open about is helpful to everyone involved,
including maintainers.</p>
<h2 id="background">Background</h2>
<p>Technically, very little is required to call something “free/open source
software”. The developer community consensus<sup id="fnref:open-source" role="doc-noteref"><a href="#fn:open-source" class="footnote" rel="footnote">1</a></sup> seems to require
exactly two things: the source code and a license permitting sharing and
modification, together somewhere online. Back in the days of source tarballs,
FTP, and <code class="language-plaintext highlighter-rouge">./configure && make && make install</code>, that was pretty much the norm.</p>
<p>However, there are a lot of other ways a project can be open than just providing
source code and a license. Let’s look at a few.</p>
<h2 id="open-features">Open features</h2>
<p>Before you shoot me an angry email, I’m not saying every project should have all
the features below. Some of them are overkill for small projects. Some of them
involve a lot of work to set up, maintain, and interact with the users. In other
cases it might be policy not to spend time interacting with the community. And
having a badly implemented or zombie feature, like all those abandoned user
forums out there, is worse than not having it in the first place. Make sure you
have the necessary resources and buy-in for at least the medium term if you’re
going to go beyond the basics.</p>
<h3 id="ground-level">Ground level</h3>
<p>Let’s start with the absolutely base criteria: <strong>the source code and license can
be downloaded from anywhere and extracted with extremely basic tools.</strong>
Actually, hold on. A lot of projects fail even these entry-level criteria:</p>
<ul>
<li>Some projects have <em>no explicit license.</em> In some jurisdictions this could
make life difficult for both the original author and contributors.</li>
<li>The source is <em>hard to get to.</em> Maybe it’s on an unreliable host or only
available via an unreliable connection. Maybe the provider requires you to
register before giving access. Such a registration process probably requires
an email address, which needs to be verified, which means you have to provide
a valid email address, their server has to accept it<sup id="fnref:throwaway-email" role="doc-noteref"><a href="#fn:throwaway-email" class="footnote" rel="footnote">2</a></sup>, they
need to actually send you a verification email, your email provider has to
accept the email, you have to react to the email within the time limit, and
the system has to actually work when you click the link or respond to verify.
Finally, some providers require you to explicitly accept the license before
letting you download.</li>
<li>The download is <em>corrupted</em> in some way. This is not something that only
happens in oppressive regimes. SourceForge did it<sup id="fnref:sourceforge" role="doc-noteref"><a href="#fn:sourceforge" class="footnote" rel="footnote">3</a></sup>, and at that
point they were still a big provider of open source software. Scratch that, a
lot of projects are <em>still</em> on SourceForge<sup id="fnref:sourceforge-in-nixpkgs" role="doc-noteref"><a href="#fn:sourceforge-in-nixpkgs" class="footnote" rel="footnote">4</a></sup>, despite
this appalling behaviour. Other package providers still support plain HTTP or
even FTP, and
<a href="https://mywiki.wooledge.org/FtpMustDie?action=recall&rev=40">FTP must die</a>!
Anyone between you and the server could corrupt the download, for fun, profit,
or worse.</li>
<li>Some projects use <em>uncommon or proprietary tools</em> to wrap their source code
releases. Ever heard of WinRAR? It uses a
<a href="https://www.7-zip.org/license.txt">proprietary compression algorithm</a>. The
decompression source code is open, but you’re not allowed to build a tool to
produce RAR files.</li>
</ul>
<p>I’m not saying the problems above are common. Thankfully not, but they do exist.
The fact that they are dying out means the world has moved on. But we can do so
much better than this.</p>
<h3 id="commit-log">Commit log</h3>
<p>An obvious step up from a tarball is a complete, public, and
<a href="https://xkcd.com/1296/">detailed</a> commit log. Potential and actual users can
learn a lot from looking through the logs:</p>
<ul>
<li>How old is the project?</li>
<li>Who was involved at what time?</li>
<li>Is the project is still active?</li>
<li>What were the big pivot points?</li>
<li>How much do the authors care about various aspects of the code, like speed,
security, UX, maintainability, etc?</li>
</ul>
<p>Can your users figure this out from the publicly available commit log?</p>
<h3 id="issue-tracker">Issue tracker</h3>
<p>Having publicly visible issues is probably the most obvious and ubiquitous open
feature of modern repositories. The days when you’d need to scour a barely
searchable email archive for duplicates are almost gone, and good riddance. It’s
difficult enough to search even a good issue tracker, without having to
simultaneously search through support requests.</p>
<p>(But wait, a lot of projects <em>do</em> use issues for support requests. Welp. That’s
another one for the list: having somewhere <em>separate from your work log</em> where
people can ask questions (and expect answers within a few days) about your
software is really useful. Tags can help, but since every project implements and
uses tags in different ways, it is far from obvious how to search through the
right subset. In many cases it might be enough to simply tell people about Stack
Overflow (or the relevant Stack Exchange site), and, if you’re keen, to monitor
the relevant <a href="https://stackoverflow.com/feeds/tag/python">tag feed</a>. Zero
maintenance, and the community might even do a lot of the work.</p>
<p>How do you know when you’ve got this right? The vast majority of the entries in
your issue tracker are at least actionable - they describe a specific problem
which is solvable.)</p>
<p>Back to issues, it’s important for community building that they actually include
<em>all</em> your (past, current, and future) work. If you hide away important tickets
in some private system you’re sabotaging your chances of having healthy
community interactions. That’s not to say every single thought or utterance
about the project should be public. A good candidate for a publicly visible
ticket might be something you intend to do in the near future, as opposed to
just a weakly held opinion or far future vision of ultimate possibility.</p>
<h3 id="documentation">Documentation</h3>
<p>On a similar note, any <em>documentation</em> which would be useful to new developers
should probably be public. Architecture diagrams, engineering decisions, code
review guidelines, release processes, development setup instructions, you name
it. The gold standard here is that a new developer who is familiar with the
technology but <em>not this particular project</em> should be able to start
contributing within hours, not weeks.</p>
<h3 id="pipeline">Pipeline</h3>
<p>It’s still not all that common to let end users see your full pipeline
configuration and run logs. In some cases there might be legitimate security
reasons for this, but it’s easy to see why it could be the opposite. If you
don’t trust developers to keep secrets out of public configuration and logs, why
should users expect that they understand security?</p>
<p>Some CI/CD systems make it almost impossible to be open in a useful fashion.
Jenkins and other GUI-configured systems are the worst offenders. To
independently reproduce a Jenkins pipeline, you basically have to install the
same version as the project is using, then manually configure it the same way.
GitHub is better: at least you can fork a repo on GitHub and run the same jobs.
But that’s the rub: you can’t run a GitHub pipeline on any other platform,
including locally. There are third party solutions, but like any third party
solution they’ll at best be playing catch-up with the closed source GitHub
implementation. GitLab is better than GitHub in this respect: a lot of the
configuration can be copied and pasted locally or into another CI/CD system
configuration with minimal work. Probably the worst part, which won’t apply to a
lot of open source projects, is having to chase down and set up missing secrets,
which can be time-consuming whack-a-mole unless the commands explicitly call out
any missing variables.</p>
<h3 id="level-n1">Level N+1</h3>
<p>Which other openness features would you want to see more of?</p>
<h2 id="readme">Readme</h2>
<p>Most projects do a lot of the above, but it’s not obvious from looking at the
front page. Why not mention what you do to be open front-and-center, in your
project readme? Or if you have reasons not to be open in some way, why not just
say so explicitly? If you don’t intend to fix issues, why not disable that
feature? Let anyone interested know what they can expect, and don’t be surprised
if it helps build a healthy community.</p>
<p><a href="https://gitlab.com/engmark/vcard#how-open-is-this-project">Example readme section</a>.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:open-source" role="doc-endnote">
<p>As opposed to the Open Source Initiative or GNU definitions. I’m not a
lawyer, so let’s just leave it at that. <a href="#fnref:open-source" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:throwaway-email" role="doc-endnote">
<p>Throwaway email addresses are nowhere near as ubiquitous or simple to set up
as they should be, and the website might refuse such addresses. <a href="#fnref:throwaway-email" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:sourceforge" role="doc-endnote">
<p>I don’t know whether the SourceForge debacle only involved binary release
files, but once they’ve seen fit to wrap installers in adware, what reason
do you have to trust that they won’t modify source downloads? <a href="#fnref:sourceforge" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:sourceforge-in-nixpkgs" role="doc-endnote">
<p><code class="language-plaintext highlighter-rouge">git grep --name-only sourceforge | wc -l</code> in
<a href="https://github.com/NixOS/nixpkgs/tree/312aa96fe67ad945543611cec05172b67bf61a35">nixpkgs</a>
returned matches in 1088 out of 37278 files, so probably <em>roughly 3% of
official Nix packages are from SourceForge.</em> <a href="#fnref:sourceforge-in-nixpkgs" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Victor Engmarktl;dr: Being open about what you’re open about is helpful to everyone involved, including maintainers.Easy PostgreSQL testing with pgTAP and Nix2023-08-03T00:00:00+00:002023-08-03T00:00:00+00:00https://paperless.blog/easy-postgresql-testing-with-pgtap-and-nix<p>This article explains one way to get from zero to a <code class="language-plaintext highlighter-rouge">psql</code> shell inside a local
PostgreSQL instance with pgTAP enabled. It’s based on the <a href="/nix-shell-template">Nix shell
template</a> article.</p>
<h2 id="intro">Intro</h2>
<p>To recap, all you need is to install Nix
<a href="https://nixos.org/download.html">install Nix</a>, declare your environment in a
<code class="language-plaintext highlighter-rouge">shell.nix</code> file, and run <code class="language-plaintext highlighter-rouge">nix-shell</code> to enter the environment. We’ll start from
the template <code class="language-plaintext highlighter-rouge">shell.nix</code> in the original article:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span>
<span class="nv">pkgs</span> <span class="o">=</span>
<span class="kr">import</span> <span class="p">(</span>
<span class="kr">fetchTarball</span> <span class="p">{</span>
<span class="nv">name</span> <span class="o">=</span> <span class="s2">"nixos-23.05_2023-06-30"</span><span class="p">;</span>
<span class="nv">url</span> <span class="o">=</span> <span class="s2">"https://github.com/NixOS/nixpkgs/archive/b72aa95f7f096382bff3aea5f8fde645bca07422.tar.gz"</span><span class="p">;</span>
<span class="nv">sha256</span> <span class="o">=</span> <span class="s2">"1ndnsfzff0jdxvjnjnrdm74x8xq2c221hfr7swdnxm7pkmi5w9q5"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">{};</span>
<span class="kn">in</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">mkShell</span> <span class="p">{</span>
<span class="nv">packages</span> <span class="o">=</span> <span class="p">[</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">bashInteractive</span>
<span class="p">];</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="building-it">Building it</h2>
<p>(Feel free to <a href="#the-result">skip this part</a> if you just want the thrilling
conclusion.)</p>
<p>How do we need to change the template to achieve this? We can split it into a
few easily manageable tasks. First, we’ll need to install PostgreSQL itself.
That means changing the <code class="language-plaintext highlighter-rouge">packages</code> list to include <code class="language-plaintext highlighter-rouge">pkgs.postgresql</code>. Then we
need a small Bash script to set up and connect to the database:</p>
<ol>
<li>Set up some directives to exit early in case of failure in any of the
subsequent commands.</li>
<li>Configure an “exit trap” to stop the database and remove the
files<sup id="fnref:keep-files" role="doc-noteref"><a href="#fn:keep-files" class="footnote" rel="footnote">1</a></sup> when exiting the interactive database session below.</li>
<li>Create the database cluster using the <code class="language-plaintext highlighter-rouge">initdb</code> command.</li>
<li>Start PostgreSQL itself using the <code class="language-plaintext highlighter-rouge">pg_ctl</code> command.</li>
<li>Create a test database using the <code class="language-plaintext highlighter-rouge">createdb</code> command. This lets us work on a
completely blank database, rather than an internal database.</li>
<li>Open an interactive shell within the test database using the <code class="language-plaintext highlighter-rouge">psql</code> command.</li>
</ol>
<p>The result so far:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span>
<span class="nv">pkgs</span> <span class="o">=</span>
<span class="kr">import</span> <span class="p">(</span>
<span class="kr">fetchTarball</span> <span class="p">{</span>
<span class="nv">name</span> <span class="o">=</span> <span class="s2">"nixos-23.05_2023-06-30"</span><span class="p">;</span>
<span class="nv">url</span> <span class="o">=</span> <span class="s2">"https://github.com/NixOS/nixpkgs/archive/b72aa95f7f096382bff3aea5f8fde645bca07422.tar.gz"</span><span class="p">;</span>
<span class="nv">sha256</span> <span class="o">=</span> <span class="s2">"1ndnsfzff0jdxvjnjnrdm74x8xq2c221hfr7swdnxm7pkmi5w9q5"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">{};</span>
<span class="kn">in</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">mkShell</span> <span class="p">{</span>
<span class="nv">packages</span> <span class="o">=</span> <span class="p">[</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">bashInteractive</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">postgresql</span>
<span class="p">];</span>
<span class="nv">env</span><span class="o">.</span><span class="nv">PGDATA</span> <span class="o">=</span> <span class="s2">"</span><span class="si">${</span><span class="kr">toString</span> <span class="sx">./.pgdata</span><span class="si">}</span><span class="s2">"</span><span class="p">;</span>
<span class="nv">shellHook</span> <span class="o">=</span> <span class="s2">''</span><span class="err">
</span><span class="s2"> set -o errexit -o noclobber -o nounset</span><span class="err">
</span><span class="s2"> # Remove traces of running server when exiting this shell hook</span><span class="err">
</span><span class="s2"> cleanup() {</span><span class="err">
</span><span class="s2"> pg_ctl stop</span><span class="err">
</span><span class="s2"> rm --force --recursive "$PGDATA"</span><span class="err">
</span><span class="s2"> }</span><span class="err">
</span><span class="s2"> trap cleanup EXIT</span><span class="err">
</span><span class="s2"> # Create database cluster</span><span class="err">
</span><span class="s2"> initdb --auth-local=trust --auth-host=trust</span><span class="err">
</span><span class="s2"> # Start server</span><span class="err">
</span><span class="s2"> pg_ctl --log="$PGDATA/db.log" --options="-c unix_socket_directories='$PGDATA'" start</span><span class="err">
</span><span class="s2"> # Create test database</span><span class="err">
</span><span class="s2"> db_name=test</span><span class="err">
</span><span class="s2"> createdb "$db_name" --host="$PGDATA"</span><span class="err">
</span><span class="s2"> # Connect to database</span><span class="err">
</span><span class="s2"> psql --dbname="$db_name" --host="$PGDATA"</span><span class="err">
</span><span class="s2"> # Return from Bash after exiting psql</span><span class="err">
</span><span class="s2"> exit</span><span class="err">
</span><span class="s2"> ''</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now we just need one more ingredient, pgTAP itself. We need to install it in
such a way that the PostgreSQL package is aware of it, and then we need to
enable it on our test database.</p>
<h2 id="the-result">The result</h2>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span>
<span class="nv">pkgs</span> <span class="o">=</span>
<span class="kr">import</span> <span class="p">(</span>
<span class="kr">fetchTarball</span> <span class="p">{</span>
<span class="nv">name</span> <span class="o">=</span> <span class="s2">"nixos-23.05_2023-06-30"</span><span class="p">;</span>
<span class="nv">url</span> <span class="o">=</span> <span class="s2">"https://github.com/NixOS/nixpkgs/archive/b72aa95f7f096382bff3aea5f8fde645bca07422.tar.gz"</span><span class="p">;</span>
<span class="nv">sha256</span> <span class="o">=</span> <span class="s2">"1ndnsfzff0jdxvjnjnrdm74x8xq2c221hfr7swdnxm7pkmi5w9q5"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">{};</span>
<span class="kn">in</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">mkShell</span> <span class="p">{</span>
<span class="nv">packages</span> <span class="o">=</span> <span class="p">[</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">bashInteractive</span>
<span class="p">(</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">postgresql</span><span class="o">.</span><span class="nv">withPackages</span> <span class="p">(</span>
<span class="nv">postgresqlPackages</span><span class="p">:</span> <span class="p">[</span>
<span class="nv">postgresqlPackages</span><span class="o">.</span><span class="nv">pgtap</span>
<span class="p">]</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="p">];</span>
<span class="nv">env</span><span class="o">.</span><span class="nv">PGDATA</span> <span class="o">=</span> <span class="s2">"</span><span class="si">${</span><span class="kr">toString</span> <span class="sx">./.pgdata</span><span class="si">}</span><span class="s2">"</span><span class="p">;</span>
<span class="nv">shellHook</span> <span class="o">=</span> <span class="s2">''</span><span class="err">
</span><span class="s2"> set -o errexit -o noclobber -o nounset</span><span class="err">
</span><span class="s2"> # Remove traces of running server when exiting this shell hook</span><span class="err">
</span><span class="s2"> cleanup() {</span><span class="err">
</span><span class="s2"> pg_ctl stop</span><span class="err">
</span><span class="s2"> rm --force --recursive "$PGDATA"</span><span class="err">
</span><span class="s2"> }</span><span class="err">
</span><span class="s2"> trap cleanup EXIT</span><span class="err">
</span><span class="s2"> # Create database cluster</span><span class="err">
</span><span class="s2"> initdb --auth-local=trust --auth-host=trust</span><span class="err">
</span><span class="s2"> # Start server</span><span class="err">
</span><span class="s2"> pg_ctl --log="$PGDATA/db.log" --options="-c unix_socket_directories='$PGDATA'" start</span><span class="err">
</span><span class="s2"> # Create test database</span><span class="err">
</span><span class="s2"> db_name=test</span><span class="err">
</span><span class="s2"> createdb "$db_name" --host="$PGDATA"</span><span class="err">
</span><span class="s2"> # Enable pgTAP</span><span class="err">
</span><span class="s2"> psql --command="CREATE EXTENSION pgtap" --dbname="$db_name" --host="$PGDATA"</span><span class="err">
</span><span class="s2"> # Connect to database</span><span class="err">
</span><span class="s2"> psql --dbname="$db_name" --host="$PGDATA"</span><span class="err">
</span><span class="s2"> # Return from Bash after exiting psql</span><span class="err">
</span><span class="s2"> exit</span><span class="err">
</span><span class="s2"> ''</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>pgTAP is now available by simply running <code class="language-plaintext highlighter-rouge">nix-shell</code>! To verify, try running a
trivial test suite:</p>
<pre><code class="language-postgresql">SELECT * FROM no_plan();
SELECT is(2, 2);
SELECT * FROM finish();
</code></pre>
<h2 id="variants">Variants</h2>
<p>As you can see, enabling other PostgreSQL extensions from
<a href="https://search.nixos.org/packages">nixpkgs</a> is just a matter of adding two
lines: <code class="language-plaintext highlighter-rouge">postgresqlPackages.[…]</code> and the
<code class="language-plaintext highlighter-rouge">psql --command="CREATE EXTENSION […]" --dbname="$db_name" --host="$PGDATA"</code>
command.</p>
<p>It’s also possible to run ancient versions of PostgreSQL by
<a href="https://lazamar.co.uk/nix-versions/?channel=nixpkgs-unstable&package=postgresql">installing it from an older version of nixpkgs</a>.
Click on any of the versions above to get detailed instructions.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:keep-files" role="doc-endnote">
<p>If you want to keep the database files around after stopping PostgreSQL,
simply delete the <code class="language-plaintext highlighter-rouge">rm […]</code> line in the <code class="language-plaintext highlighter-rouge">cleanup</code> function. <a href="#fnref:keep-files" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Victor EngmarkThis article explains one way to get from zero to a psql shell inside a local PostgreSQL instance with pgTAP enabled. It’s based on the Nix shell template article.Nix shell template2023-07-02T00:00:00+00:002023-07-02T00:00:00+00:00https://paperless.blog/nix-shell-template<p>Nix shells are the best tool for creating software development environments
right now. This article provides a template to get you started with Nix shells
from scratch, and explains how to add common features.</p>
<p><a href="#the-template">The template</a> has the following features:</p>
<ul>
<li>Everybody using the template gets the same package versions, because it locks
in the version of the entire package collection in a single expression. This
also means there’s no need for a separate lock file.</li>
<li>Seamlessly integrates with software already installed on your platform, with a
<a href="#pure-nix-shells">single flag</a> to make sure only the Nix shell packages are
available for improved reproducibility.</li>
<li>Does not interfere with software already installed on your platform. All the
Nix packages are installed in their own directory.</li>
<li><a href="#adding-packages">Add a package</a> with a single line change.</li>
<li>Uses a subset of the packaging-specific, declarative, lazily evaluated,
dynamically typed, purely functional
<a href="https://nixos.org/manual/nix/stable/language/">Nix language</a>.</li>
</ul>
<h2 id="prerequisites">Prerequisites</h2>
<p><a href="https://nixos.org/download.html">Install Nix</a><sup id="fnref:nix-install-note" role="doc-noteref"><a href="#fn:nix-install-note" class="footnote" rel="footnote">1</a></sup>.</p>
<h2 id="the-template">The template</h2>
<p>This is basically the simplest <em>reproducible</em> Nix shell declaration you can get:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span>
<span class="nv">pkgs</span> <span class="o">=</span>
<span class="kr">import</span> <span class="p">(</span>
<span class="kr">fetchTarball</span> <span class="p">{</span>
<span class="nv">name</span> <span class="o">=</span> <span class="s2">"nixos-23.05_2023-06-30"</span><span class="p">;</span>
<span class="nv">url</span> <span class="o">=</span> <span class="s2">"https://github.com/NixOS/nixpkgs/archive/b72aa95f7f096382bff3aea5f8fde645bca07422.tar.gz"</span><span class="p">;</span>
<span class="nv">sha256</span> <span class="o">=</span> <span class="s2">"1ndnsfzff0jdxvjnjnrdm74x8xq2c221hfr7swdnxm7pkmi5w9q5"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">{};</span>
<span class="kn">in</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">mkShell</span> <span class="p">{</span>
<span class="nv">packages</span> <span class="o">=</span> <span class="p">[</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">bashInteractive</span>
<span class="p">];</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This should be stored in a <code class="language-plaintext highlighter-rouge">shell.nix</code> file in your project root.</p>
<h2 id="use">Use</h2>
<p>Enter the shell by running <code class="language-plaintext highlighter-rouge">nix-shell</code> in the same directory as the <code class="language-plaintext highlighter-rouge">shell.nix</code>
file above. This will download the nixpkgs tarball and the Bash package, and
cache both of them for (much faster) future runs.</p>
<p>Nix shells do not automatically reload when you change <code class="language-plaintext highlighter-rouge">shell.nix</code>. To load an
updated <code class="language-plaintext highlighter-rouge">shell.nix</code>, exit the current Nix shell (if you’re already in it), then
re-run <code class="language-plaintext highlighter-rouge">nix-shell</code>.</p>
<h2 id="adding-packages">Adding packages</h2>
<p>To install another package in your Nix shell, add it to the <code class="language-plaintext highlighter-rouge">packages</code> list.
(List entries are whitespace separated, so there’s no need for any commas.) For
example, to install the default Python 3 interpreter, add a line below
<code class="language-plaintext highlighter-rouge">pkgs.bashInteractive</code> with <code class="language-plaintext highlighter-rouge">pkgs.python3</code>.</p>
<p>You can search through <a href="https://search.nixos.org/packages">supported packages</a>
or <a href="https://lazamar.co.uk/nix-versions/">older packages</a>. (Mixing packages from
different nixpkgs versions in the same Nix shell is beyond the scope of this
article.)</p>
<h3 id="pure-nix-shells">Pure Nix shells</h3>
<p>This might come as a surprise<sup id="fnref:version-caveat" role="doc-noteref"><a href="#fn:version-caveat" class="footnote" rel="footnote">2</a></sup>:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>nix-shell <span class="nt">--run</span> <span class="s1">'git --version'</span>
<span class="go">git version 2.40.1
</span></code></pre></div></div>
<p>Git is not installed in the Nix shell, but it still runs successfully. It turns
out that this is the Git version installed in the underlying OS. This is a
feature during normal development: you probably don’t want your IDE and other
auxiliary tools to bloat your Nix shell, since they shouldn’t affect the project
outputs. But this decreases reproducibility. In the best case, another developer
or your automated build complains that the command is not available. Worse, they
may get subtly different results, resulting in lost time working out why. At
worst you don’t find out until customers complain about a broken product.</p>
<p>The fix: run <code class="language-plaintext highlighter-rouge">nix-shell --pure</code> to avoid inheriting variables such as <code class="language-plaintext highlighter-rouge">PATH</code>
from the underlying shell:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>nix-shell <span class="nt">--pure</span> <span class="nt">--run</span> <span class="s1">'git --version'</span>
<span class="go">[omitted] git: command not found
</span></code></pre></div></div>
<p>Basically, make sure to use only pure Nix shells in your automated builds (and
run automated builds when pushing to unmerged branches). This way you get the
best of both worlds: builds are reproducible, but everyone can use whichever
auxiliary tools they want during development.</p>
<h3 id="nixpkgs-version-updates">nixpkgs version updates</h3>
<p>The <code class="language-plaintext highlighter-rouge">fetchTarball</code> function call in the template specifies the exact version of
nixpkgs to use for all the packages in the Nix shell. So to update packages, you
will need to update the function call:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">name</code> is basically an arbitrary identifier, and I chose to use the format
<code class="language-plaintext highlighter-rouge">[nixpkgs branch name]_[date of commit in URL]</code>. You can of course choose
whatever you like, but including these two makes it pretty easy to check
whether the packages are recent or not.</li>
<li><code class="language-plaintext highlighter-rouge">url</code> is where you can download the tarball for a specific version of nixpkgs.
You could point it to any URL, but you probably want to use one of the commit
IDs listed on the <a href="https://status.nixos.org/">nixpkgs status page</a>. I chose to
use the commit at the head of the most recent release branch, “nixos-23.05”,
which is effectively the “latest stable” nixpkgs right now.</li>
<li><code class="language-plaintext highlighter-rouge">sha256</code> is the SHA-256 checksum of the contents of the <code class="language-plaintext highlighter-rouge">url</code> above, after
unpacking the gzipped tarball. Don’t worry if any or all of that was
gobbledygook. You can run <code class="language-plaintext highlighter-rouge">nix-prefetch-url --unpack URL</code>, replacing <code class="language-plaintext highlighter-rouge">URL</code>
with the <code class="language-plaintext highlighter-rouge">url</code> you specified, and replace the value with the last line that
command prints.</li>
</ul>
<p>Of course, the above is a bit clunky, especially if you want to update often. In
that case I would recommend using <a href="https://github.com/nmattia/niv/">Niv</a>. You
can see how that works in the
<a href="https://gitlab.com/engmark/engmark.gitlab.io/-/blob/1db544d0af9619355bce3db8fc66a22fe25e8a41/shell.nix#L2-3">blog repository</a> -
the <code class="language-plaintext highlighter-rouge">shell.nix</code> imports a separate static Nix file, which reads configuration
from a dynamic JSON file. Both files are maintained by the <code class="language-plaintext highlighter-rouge">niv</code> command and
versioned like any other file in your repository.</p>
<h2 id="garbage-collection">Garbage collection</h2>
<p>You will probably end up changing the nixpkgs URL several times over the life of
a project. As versions change, more packages are saved locally. To avoid wasting
space, it is a good idea to run <code class="language-plaintext highlighter-rouge">nix-collect-garbage</code> once in a while to delete
unreferenced Nix store paths.</p>
<h2 id="summary">Summary</h2>
<p>At this point you’re right to suspect that I’ve glossed over many details. Why
<code class="language-plaintext highlighter-rouge">bashInteractive</code>? How can I avoid ever having to wait for packages to build?
What about cross-platform compatibility? What about supporting my colleagues who
can’t/won’t use Nix? How do I <a href="/easy-postgresql-testing-with-pgtap-and-nix">spin up a PostgreSQL server with pgTAP in a
single
command?</a>
How do I combine Nix shells for crazy internet points? Are there still cases
where this setup can be non-reproducible? I only know the answer to some of
those, and if you’re interested in more I’ll try to answer them.</p>
<p>This is all you need to know to use Nix shells productively for many projects.
And once you need something more advanced, rest assured it can be done without
losing any of the advantages listed above.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>Thanks to <a href="https://github.com/adisbladis">Adam Höse</a> and
<a href="https://imincik.github.io/">Ivan Minčík</a> for reviewing a draft of this post!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:nix-install-note" role="doc-endnote">
<p>Really, that’s it. <a href="#fnref:nix-install-note" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:version-caveat" role="doc-endnote">
<p>The output might be different for you. This just happens to be what the
version installed on my system prints. <a href="#fnref:version-caveat" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Victor EngmarkNix shells are the best tool for creating software development environments right now. This article provides a template to get you started with Nix shells from scratch, and explains how to add common features.Convert date/time from machine-readable to human-readable2023-01-31T00:00:00+00:002023-01-31T00:00:00+00:00https://paperless.blog/machine-readable-to-human-readable-date-time<p>Enter a date/time string below to get the human-readable local time instantly.</p>
<p><strong>Update:</strong> The
<a href="https://linz.github.io/emergency-management-tools/machine-readable-to-human-readable-date-time/">web page</a>,
<a href="https://linz.github.io/emergency-management-tools/machine-readable-to-human-readable-date-time/human-readable-local-date-time.mjs">code</a>
and
<a href="https://github.com/linz/emergency-management-tools/blob/master/machine-readable-to-human-readable-date-time/test/human-readable-local-date-time.spec.mjs">tests</a>
now live in a work repository.</p>
<form id="human-readable-date-time-form">
<div>
<label for="machine-readable-date-time">Machine readable</label>
<input id="machine-readable-date-time" name="machine-readable-date-time" type="text" required="" /> in language code <span id="language"></span> and time zone <span id="time-zone"></span>:
<p><strong><time id="human-readable-date-time"></time></strong> <button id="copy-human-readable-date-time">Copy</button></p>
</div>
</form>
<script type="module">
import {humanReadableLocalDateTime} from "/assets/scripts/human-readable-local-date-time.mjs";
const formElement = document.getElementById("human-readable-date-time-form");
const inputElement = document.getElementById("machine-readable-date-time");
const languageElement = document.getElementById("language");
const timeZoneElement = document.getElementById("time-zone");
const outputElement = document.getElementById("human-readable-date-time");
const copyButtonElement = document.getElementById("copy-human-readable-date-time");
const nowAsISOString = new Date().toISOString();
inputElement.value = nowAsISOString;
languageElement.innerText = navigator.language;
timeZoneElement.innerText = Intl.DateTimeFormat().resolvedOptions().timeZone;
outputElement.setAttribute("datetime", nowAsISOString);
outputElement.innerText = humanReadableLocalDateTime(nowAsISOString);
inputElement.addEventListener("input", (event) => {
event.preventDefault();
new FormData(formElement);
});
inputElement.addEventListener("focus", (event) => {
inputElement.value = "";
});
formElement.addEventListener("formdata", (event) => {
outputElement.innerText = humanReadableLocalDateTime(event.formData.get("machine-readable-date-time"));
});
copyButtonElement.addEventListener("click", (event) => {
navigator.clipboard.writeText(outputElement.innerText);
});
formElement.addEventListener("submit", (event) => {
/* Avoid changing the page URL */
event.preventDefault();
});
</script>
<hr />
<h2 id="features">Features</h2>
<ul>
<li>Localised - the output is in the main language of your browser. Defaults to US
English if no language is configured.</li>
<li>
<p>Time zone conversion to match your local configuration. Defaults to UTC if no
language is configured, or if
<a href="https://mzl.la/3dlIvK2">Firefox’s protection against fingerprinting</a> is
enabled:</p>
<blockquote>
<p>Your timezone is reported to be UTC</p>
</blockquote>
</li>
<li>Updates whenever you change the input field.</li>
<li>Clears the input field when you click it, for faster pasting.</li>
<li>Forces 24-hour clock.</li>
</ul>
<h2 id="technical-stuff">Technical stuff</h2>
<p>Implemented for a colleague when I realised there doesn’t seem to be anything
like this online, and <code class="language-plaintext highlighter-rouge">date --date=INPUT +'%A, %-d %B %+4Y at %-I:%M:%S %P %Z'</code>
wasn’t an option.</p>
<p>If you’re interested in how this works, the <a href="https://gitlab.com/engmark/engmark.gitlab.io/-/commits/master/assets/scripts/human-readable-local-date-time.mjs">code</a>
and <a href="https://gitlab.com/engmark/engmark.gitlab.io/-/commits/master/test/human-readable-local-date-time.spec.mjs">tests</a>
are both available.</p>
<p>Basically, the input must be a valid
<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/Date"><code class="language-plaintext highlighter-rouge">Date</code> constructor</a>
argument, and will be converted to the locale and time zone of your browser.</p>Victor EngmarkEnter a date/time string below to get the human-readable local time instantly.On brevity2022-12-26T00:00:00+00:002022-12-26T00:00:00+00:00https://paperless.blog/on-brevity<p>There was a young joker, a git,<br />Who thought himself quite the wit,<br />But
he wasn’t funny,<br />His jokes were runny,<br />Like “Conciseness, rather than
prolixity, is the very being, the very body, the very soul, of wit!”</p>Victor EngmarkThere was a young joker, a git,Who thought himself quite the wit,But he wasn’t funny,His jokes were runny,Like “Conciseness, rather than prolixity, is the very being, the very body, the very soul, of wit!”YouTube channel web feed bookmarklet2022-11-11T00:00:00+00:002022-11-11T00:00:00+00:00https://paperless.blog/youtube-channel-web-feed-bookmarklet<p>tl;dr Drag →→→ <a href="javascript:(function%20()%20%7B%0A%20%20if%20(%0A%20%20%20%20(channelId%20=%0A%20%20%20%20%20%20window?.ytInitialPlayerResponse?.videoDetails?.channelId%20??%0A%20%20%20%20%20%20document%0A%20%20%20%20%20%20%20%20.querySelector(%0A%20%20%20%20%20%20%20%20%20%20%22link[rel='canonical'][href%5E='https://www.youtube.com/channel/']%22,%0A%20%20%20%20%20%20%20%20)%0A%20%20%20%20%20%20%20%20?.getAttribute(%22href%22)%0A%20%20%20%20%20%20%20%20?.substring(32))%0A%20%20)%20%7B%0A%20%20%20%20console.debug(%22Going%20to%20feed%20URL%22);%0A%20%20%20%20location.href%20=%0A%20%20%20%20%20%20%22https://www.youtube.com/feeds/videos.xml?channel_id=%22%20+%20channelId;%0A%20%20%7D%20else%20%7B%0A%20%20%20%20console.warn(%22Could%20not%20find%20a%20channel%20ID%20feed%20at%20%22%20+%20location.href);%0A%20%20%7D%0A%7D)();%0A">YouTube channel feed</a> ←←← to your bookmarks to get to the web
feed of the current channel.</p>
<!--more-->
<p>Did you know YouTube has Atom web feeds for channels? That means you can follow
your favourite content creators <em>individually</em> in your favourite feed reader,
rather than relying on the jumbled-up subscriptions page. Unfortunately, YouTube
doesn’t mention this feed in the relevant pages, hence this bookmarklet. Just go
to a user, channel, or video page, and click the bookmarklet to go to the web
feed.</p>
<hr />
<h2 id="technical-stuff">Technical stuff</h2>
<p>If you’re interested in how this works, here’s the code:</p>
<!-- prettier-ignore -->
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span>
<span class="p">(</span><span class="nx">channelId</span> <span class="o">=</span>
<span class="nb">window</span><span class="p">?.</span><span class="nx">ytInitialPlayerResponse</span><span class="p">?.</span><span class="nx">videoDetails</span><span class="p">?.</span><span class="nx">channelId</span> <span class="p">??</span>
<span class="nb">document</span>
<span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span>
<span class="dl">"</span><span class="s2">link[rel='canonical'][href^='https://www.youtube.com/channel/']</span><span class="dl">"</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">?.</span><span class="nx">getAttribute</span><span class="p">(</span><span class="dl">"</span><span class="s2">href</span><span class="dl">"</span><span class="p">)</span>
<span class="p">?.</span><span class="nx">substring</span><span class="p">(</span><span class="mi">32</span><span class="p">))</span>
<span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">debug</span><span class="p">(</span><span class="dl">"</span><span class="s2">Going to feed URL</span><span class="dl">"</span><span class="p">);</span>
<span class="nx">location</span><span class="p">.</span><span class="nx">href</span> <span class="o">=</span>
<span class="dl">"</span><span class="s2">https://www.youtube.com/feeds/videos.xml?channel_id=</span><span class="dl">"</span> <span class="o">+</span> <span class="nx">channelId</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">warn</span><span class="p">(</span><span class="dl">"</span><span class="s2">Could not find a channel ID feed at </span><span class="dl">"</span> <span class="o">+</span> <span class="nx">location</span><span class="p">.</span><span class="nx">href</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">})();</span>
</code></pre></div></div>Victor Engmarktl;dr Drag →→→ YouTube channel feed ←←← to your bookmarks to get to the web feed of the current channel.Reproducible Jupyter Notebook with Nix2022-09-20T00:00:00+00:002022-09-20T00:00:00+00:00https://paperless.blog/reproducible-jupyter-notebook-with-nix<p>So you have a Jupyter Notebook with <code class="language-plaintext highlighter-rouge">pyproject.toml</code> and <code class="language-plaintext highlighter-rouge">poetry.lock</code> files,
and you want to productionise it? You’ll just need a Linux or macOS machine.
This is the magic sauce:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
<span class="nv">pkgs</span> <span class="o">?</span>
<span class="kr">import</span>
<span class="p">(</span>
<span class="kr">fetchTarball</span> <span class="p">{</span>
<span class="nv">name</span> <span class="o">=</span> <span class="s2">"22.05"</span><span class="p">;</span>
<span class="nv">url</span> <span class="o">=</span> <span class="s2">"https://github.com/NixOS/nixpkgs/archive/ce6aa13369b667ac2542593170993504932eb836.tar.gz"</span><span class="p">;</span>
<span class="nv">sha256</span> <span class="o">=</span> <span class="s2">"0d643wp3l77hv2pmg2fi7vyxn4rwy0iyr8djcw1h5x72315ck9ik"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">{},</span>
<span class="p">}:</span> <span class="kd">let</span>
<span class="nv">poetryEnvironment</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">poetry2nix</span><span class="o">.</span><span class="nv">mkPoetryEnv</span> <span class="p">{</span>
<span class="nv">projectDir</span> <span class="o">=</span> <span class="kr">builtins</span><span class="o">.</span><span class="nv">path</span> <span class="p">{</span>
<span class="nv">path</span> <span class="o">=</span> <span class="sx">./.</span><span class="p">;</span>
<span class="nv">name</span> <span class="o">=</span> <span class="s2">"my_project_name"</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="kn">in</span>
<span class="nv">poetryEnvironment</span><span class="o">.</span><span class="nv">env</span><span class="o">.</span><span class="nv">overrideAttrs</span> <span class="p">(</span>
<span class="nv">oldAttrs</span><span class="p">:</span> <span class="p">{</span>
<span class="nv">buildInputs</span> <span class="o">=</span> <span class="p">[</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">cacert</span>
<span class="p">];</span>
<span class="p">}</span>
<span class="p">)</span>
</code></pre></div></div>
<p>The top half of that
<a href="https://nix.dev/tutorials/towards-reproducibility-pinning-nixpkgs">pins</a> the
most recent <a href="https://github.com/NixOS/nixpkgs/tags">nixpkgs release</a>, to make
sure we always build from the same sources. <code class="language-plaintext highlighter-rouge">mkPoetryEnv</code> creates a Nix
derivation based on your <code class="language-plaintext highlighter-rouge">poetry.lock</code> file. And the certificate authority certs
package is needed to download Python packages using HTTPS.</p>
<p>And this is the magic spoon:
<code class="language-plaintext highlighter-rouge">nix-shell --pure --run 'jupyter nbconvert --debug --execute --inplace --to=notebook my.ipynb'</code>.
It runs the notebook from start to finish within the Nix shell defined above. It
is very verbose to make it easier to detect any issues, so you might want to
remove <code class="language-plaintext highlighter-rouge">--debug</code>.</p>
<p>Based on a real-life
<a href="https://github.com/linz/emergency-management-tools/">project</a> which includes
<a href="https://github.com/linz/emergency-management-tools/blob/a9117c9128c4570d9d940ee2b95e9a39401872b8/.github/workflows/test.yml">automated tests</a>
for both Nix and Poetry-based workflows.</p>Victor EngmarkSo you have a Jupyter Notebook with pyproject.toml and poetry.lock files, and you want to productionise it? You’ll just need a Linux or macOS machine. This is the magic sauce:systemd services & timers in NixOS2022-04-23T00:00:00+00:002022-04-23T00:00:00+00:00https://paperless.blog/systemd-services-and-timers-in-nixos<p>systemd services and timers are a game changer if you’re used to cron jobs. This
looks at a couple of simple real-life examples to show off the advantages.</p>
<p>From my <code class="language-plaintext highlighter-rouge">/etc/nixos/configuration.nix</code>:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span> <span class="nv">config</span><span class="p">,</span> <span class="nv">lib</span><span class="p">,</span> <span class="nv">modulesPath</span><span class="p">,</span> <span class="nv">pkgs</span><span class="p">,</span> <span class="nv">specialArgs</span><span class="p">,</span> <span class="nv">options</span> <span class="p">}:</span> <span class="p">{</span>
<span class="nv">systemd</span><span class="o">.</span><span class="nv">services</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">dynamic-dns-updater</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">path</span> <span class="o">=</span> <span class="p">[</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">curl</span>
<span class="p">];</span>
<span class="nv">script</span> <span class="o">=</span> <span class="s2">"curl https://example.org/?token"</span><span class="p">;</span>
<span class="nv">serviceConfig</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">User</span> <span class="o">=</span> <span class="nv">config</span><span class="o">.</span><span class="nv">users</span><span class="o">.</span><span class="nv">users</span><span class="o">.</span><span class="nv">default</span><span class="o">.</span><span class="nv">name</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">startAt</span> <span class="o">=</span> <span class="s2">"hourly"</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">sync-images</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">environment</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">DARKTABLE_PATH</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">darktable</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">path</span> <span class="o">=</span> <span class="p">[</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">procps</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">rclone</span>
<span class="nv">pkgs</span><span class="o">.</span><span class="nv">sqlite</span>
<span class="p">];</span>
<span class="nv">script</span> <span class="o">=</span> <span class="kr">builtins</span><span class="o">.</span><span class="nv">readFile</span> <span class="sx">./sync-images.bash</span><span class="p">;</span>
<span class="nv">serviceConfig</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">User</span> <span class="o">=</span> <span class="nv">config</span><span class="o">.</span><span class="nv">users</span><span class="o">.</span><span class="nv">users</span><span class="o">.</span><span class="nv">default</span><span class="o">.</span><span class="nv">name</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">startAt</span> <span class="o">=</span> <span class="s2">"daily"</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="c"># Other stuff…</span>
<span class="p">}</span>
</code></pre></div></div>
<p>But how is this better than two lines in a crontab?</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">environment</code> specifies variables as key/value pairs, so your command won’t be
cluttered by them.</li>
<li><code class="language-plaintext highlighter-rouge">path</code> defines the packages available on the <code class="language-plaintext highlighter-rouge">$PATH</code>. As far as I know only
Bash is available by default. This avoids any kind of dependency on the rest
of your system, so you can trivially do things like running a different
version of Python. (If you <em>are</em> using the same package in
<code class="language-plaintext highlighter-rouge">environment.systemPackages</code> or other services Nix only stores one copy of the
package.)</li>
<li><code class="language-plaintext highlighter-rouge">serviceConfig.User</code> links the service inextricably to a user declaration, so
it’s easy to isolate jobs to a specific user.</li>
<li><code class="language-plaintext highlighter-rouge">startAt</code> supports the more flexible and intuitive
<a href="https://www.freedesktop.org/software/systemd/man/systemd.time.html">systemd time and date specification</a>.</li>
<li>systemd services and timers log everything, so it’s easier to check whether
your service is working: <code class="language-plaintext highlighter-rouge">systemctl --user start dynamic-dns-updater</code> and
<code class="language-plaintext highlighter-rouge">journalctl --unit=dynamic-dns-updater --user</code>. Compare to having to change
the cron job itself to set to a time shortly in the future, waiting for it to
trigger, figuring out where your specific cron runner logs to, manually
changing the cron job to the “production” value after checking that it works,
and hoping you didn’t mess up the new timing.</li>
<li>systemd timers can tell you when they will run next:
<code class="language-plaintext highlighter-rouge">systemctl --user status dynamic-dns-updater.timer</code>.</li>
<li>The script is part of the service derivation. So if the original script goes
away your service won’t break.</li>
</ul>Victor Engmarksystemd services and timers are a game changer if you’re used to cron jobs. This looks at a couple of simple real-life examples to show off the advantages.Elden Ring Linux performance settings2022-04-05T00:00:00+00:002022-04-05T00:00:00+00:00https://paperless.blog/elden-ring-tweaks<p>tl;dr Just set everything to max performance in
<a href="https://gitlab.com/corectrl/corectrl">CoreCtrl</a> and you’re good to go.</p>
<p>Internet advice is usually of the form “X worked for me.” It’s extremely rare to
get an explanation <em>why</em> it should work, a link to documentation, or actual
measurements. After seeing a bunch of Internet advice to improve the performance
of Elden Ring on Linux I decided to just try some of them and <em>actually
<a href="https://danluu.com/why-benchmark/">benchmark</a>.</em> The result was surprising, but
comes with the usual caveats: my hardware, firmware, and software are probably
different from yours, things change fast, and any of this might be obsolete
before it’s published.</p>
<h2 id="specs">Specs</h2>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>inxi <span class="nt">--cpu</span> <span class="nt">--graphics</span>
<span class="go">CPU: Info: 6-Core model: Intel Core i7-8700 bits: 64 type: MT MCP cache: L2: 12 MiB
Speed: 4508 MHz min/max: 800/4600 MHz Core speeds (MHz): 1: 4508 2: 4564 3: 4604 4: 4362 5: 4510 6: 4568 7: 4572
8: 4415 9: 4338 10: 4401 11: 4402 12: 4401
Graphics: Device-1: Advanced Micro Devices [AMD/ATI] Navi 14 [Radeon RX 5500/5500M / Pro 5500M] driver: amdgpu v: kernel
Device-2: N/A type: USB driver: snd-usb-audio,uvcvideo
Display: x11 server: X.Org 1.20.13 driver: loaded: amdgpu unloaded: fbdev,modesetting,vesa resolution:
1: 3840x2160~60Hz 2: 1920x1080~60Hz
OpenGL: renderer: AMD Radeon RX 5500 XT (NAVI14 DRM 3.42.0 5.15.32 LLVM 13.0.1) v: 4.6 Mesa 21.3.7
</span></code></pre></div></div>
<p>So an Intel i7-8700, AMD Radeon RX 5500, an extra “screen” which is really just
an audio receiver, and the latest stable drivers.</p>
<h2 id="software">Software</h2>
<ul>
<li><a href="https://nixos.org/">NixOS</a> unstable</li>
<li>Steam stable</li>
<li>GNOME on Wayland</li>
<li>Various other software running in the background, like a browser, email client
and IDE</li>
</ul>
<h2 id="steam-game-settings">Steam game settings</h2>
<ul>
<li>Compatibility set to Proton Experimental</li>
<li>Launch options set as mentioned below, always enabling
<a href="https://github.com/flightlessmango/MangoHud">Mango HUD</a> for an FPS counter</li>
</ul>
<h2 id="game-graphics-settings">Game graphics settings</h2>
<ul>
<li>Fullscreen</li>
<li>1920x1080 resolution</li>
<li>Auto-detect best rendering settings off</li>
<li>All graphics settings at “high”</li>
</ul>
<h2 id="methodology">Methodology</h2>
<ol>
<li>Load latest save</li>
<li>While standing in the same spot, turn the camera around in all sorts of
directions</li>
<li>Note the <em>lowest FPS</em> which shows up in the Mango HUD in about 10 seconds</li>
</ol>
<h2 id="results">Results</h2>
<ol>
<li>No extra options, just <code class="language-plaintext highlighter-rouge">MANGOHUD=1 %command%</code>: 35 FPS</li>
<li><a href="https://github.com/FeralInteractive/gamemode"><code class="language-plaintext highlighter-rouge">gamemoderun MANGOHUD=1 %command%</code></a>:
33 FPS</li>
<li><a href="https://www.amd.com/en/technologies/fidelityfx-super-resolution"><code class="language-plaintext highlighter-rouge">WINE_FULLSCREEN_FSR=1 MANGOHUD=1 %command%</code></a>:
34 FPS</li>
<li><code class="language-plaintext highlighter-rouge">AMD_VULKAN_ICD=RADV MANGOHUD=1 %command%</code>: 33 FPS (no reference, sorry)</li>
<li><code class="language-plaintext highlighter-rouge">ENABLE_VKBASALT=1 MANGOHUD=1 %command%</code>: 34 FPS (no reference, sorry)</li>
<li><code class="language-plaintext highlighter-rouge">VKD3D_CONFIG=no_upload_hvv,single_queue MANGOHUD=1 %command%</code>: 31 FPS (no
reference, sorry)</li>
</ol>
<p>Doesn’t get much clearer than that: none of these make a significant difference
with my setup.</p>
<h2 id="things-i-didnt-test">Things I <em>didn’t</em> test:</h2>
<ul>
<li>After tweaking the CoreCtrl settings I saw no appreciable difference between
X11 and Wayland performance, so I’ve only included Wayland measurements here.
If you know of any reason why either of them should perform <em>significantly</em>
better than the other for any specific settings, please let me know.</li>
<li>With a different FPS counter, since I assume its overhead is unnoticeable.</li>
<li>Disconnecting the receiver, since it’s necessary to get sound.</li>
<li><a href="https://www.reddit.com/r/linux_gaming/comments/t2o67v/did_anyone_manage_to_get_the_fps_counter_to_work/"><code class="language-plaintext highlighter-rouge">DXVK</code> options won’t do anything since this isn’t a DX12 game</a>.</li>
<li>Other desktop environments, since I don’t expect them to make significant
difference.</li>
<li><a href="https://github.com/ValveSoftware/Proton"><code class="language-plaintext highlighter-rouge">PROTON_USE_SECCOMP</code></a> is long
obsolete.</li>
</ul>Victor Engmarktl;dr Just set everything to max performance in CoreCtrl and you’re good to go.