Jekyll2021-11-02T14:31:57+01:00https://blog.jan-ahrens.eu/feed.xmlSelected Thoughtsmy blog Jan AhrensExploring APIs and data structures with Jupyter notebooks2020-03-02T00:00:00+01:002020-03-02T00:00:00+01:00https://blog.jan-ahrens.eu/2020/03/02/explore-apis<p>Recently a colleague shared a useful technique for exploring Web APIs with me: <a href="https://jupyter.org/">Jupyter notebooks</a>.</p>
<p>Previously I used to use Bash scripts and <a href="https://curl.haxx.se/">curl</a> for tasks like this. Other colleagues preferred GUI tools like <a href="https://www.postman.com/">Postman</a>.</p>
<p>Jupyter brings both worlds together:</p>
<ul>
<li>You can write code and have access to Python libraries
<ul>
<li><a href="https://requests.readthedocs.io/en/master/">Requests</a> library for HTTP requests</li>
<li><a href="https://pandas.pydata.org/">Pandas</a> library for data analysis</li>
</ul>
</li>
<li>You get documentation to share with your colleagues (and your future self)
<ul>
<li>GitHub will <a href="https://help.github.com/en/github/managing-files-in-a-repository/working-with-jupyter-notebook-files-on-github">render Jupyter notebooks</a> as static HTML</li>
<li>You can include images, tables, and even interactive elements like maps</li>
</ul>
</li>
</ul>
<p>By the way: This post was written <a href="https://github.com/JanAhrens/JanAhrens.github.io/blob/master/files/explore-apis.ipynb">in a Jupyter notebook</a> itself.</p>
<h2 id="interested-lets-get-started-by-setting-everything-up">Interested? Let’s get started by setting everything up.</h2>
<p>The first step is (of course) to install the Jupyter package:</p>
<blockquote>
<p>pip install jupyterlab</p>
</blockquote>
<p><strong>Note</strong>: Depending on when you read this (it was written in early 2020), you might have to check if <code class="language-plaintext highlighter-rouge">pip</code> is the Python 3.x version of Python or still the <a href="https://pythonclock.org/">legacy Python 2.7</a> version. On my machine I had to use the <code class="language-plaintext highlighter-rouge">pip3</code> command that <a href="https://brew.sh/">Homebrew</a> created. If that’s the case, the Python executable is most likely also named <code class="language-plaintext highlighter-rouge">python3</code>. To make it less confusing, I’ll be using the regular <code class="language-plaintext highlighter-rouge">pip</code> and <code class="language-plaintext highlighter-rouge">python</code> commands in this post.</p>
<p>Next you can start Jupyter:</p>
<blockquote>
<p>python -m jupyterlab</p>
</blockquote>
<p>You’ll be greeted with a Web UI like this:</p>
<p><img src="/files/jupyter.png" alt="" /></p>
<p>In this post I’ll be using some Python libraries. Here are it’s version number so that you can recognize if your version differ:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">pkg_resources</span>
<span class="p">[</span><span class="n">pkg_resources</span><span class="p">.</span><span class="n">get_distribution</span><span class="p">(</span><span class="n">lib</span><span class="p">)</span> <span class="k">for</span> <span class="n">lib</span> <span class="ow">in</span> <span class="p">[</span><span class="s">'jupyterlab'</span><span class="p">,</span> <span class="s">'requests'</span><span class="p">,</span> <span class="s">'curlify'</span><span class="p">,</span> <span class="s">'pandas'</span><span class="p">,</span> <span class="s">'nbconvert'</span><span class="p">]]</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[jupyterlab 1.2.6 (/usr/local/lib/python3.7/site-packages),
requests 2.21.0 (/usr/local/lib/python3.7/site-packages),
curlify 2.2.1 (/usr/local/lib/python3.7/site-packages),
pandas 1.0.0 (/usr/local/lib/python3.7/site-packages),
nbconvert 5.6.1 (/usr/local/lib/python3.7/site-packages)]
</code></pre></div></div>
<h2 id="getting-started-with-requests">Getting started with Requests</h2>
<p>The first library I want to introduce is <a href="https://requests.readthedocs.io/en/master/">Requests</a>, the de facto standard HTTP library for Python.</p>
<p>If you haven’t done so, you should install it using:</p>
<blockquote>
<p>pip install requests</p>
</blockquote>
<p>Then you are able to load it:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">requests</span>
</code></pre></div></div>
<p>Let request something simple to try out requests (no pun intended):</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">request</span><span class="p">(</span><span class="s">'GET'</span><span class="p">,</span> <span class="s">'http://httpbin.org/json'</span><span class="p">)</span>
<span class="n">response</span><span class="p">.</span><span class="n">status_code</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>200
</code></pre></div></div>
<p>To get a pretty output from the JSON data, a quick helper function comes in handy:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span>
<span class="k">def</span> <span class="nf">pp</span><span class="p">(</span><span class="n">item</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">item</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">))</span>
</code></pre></div></div>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">pp</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">())</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"slideshow": {
"author": "Yours Truly",
"date": "date of publication",
"slides": [
{
"title": "Wake up to WonderWidgets!",
"type": "all"
},
{
"items": [
"Why <em>WonderWidgets</em> are great",
"Who <em>buys</em> WonderWidgets"
],
"title": "Overview",
"type": "all"
}
],
"title": "Sample Slide Show"
}
}
</code></pre></div></div>
<p>If you want to print the response headers, you need to remember that <a href="https://github.com/psf/requests/blob/fd13816d015c4c90ee65297fa996caea6a094ed1/requests/models.py#L445">Headers</a> is a <code class="language-plaintext highlighter-rouge">CaseInsensitiveDict</code> structure. Wrappingg it in a <code class="language-plaintext highlighter-rouge">dict()</code> function enables you to print it using the <code class="language-plaintext highlighter-rouge">json.dumps</code> function.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">pp</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">headers</span><span class="p">))</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"Date": "Wed, 04 Mar 2020 07:05:33 GMT",
"Content-Type": "application/json",
"Content-Length": "429",
"Connection": "keep-alive",
"Server": "gunicorn/19.9.0",
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Credentials": "true"
}
</code></pre></div></div>
<p>You can get a curl version of your request by using the <a href="https://github.com/ofw/curlify">curlify</a> package:</p>
<blockquote>
<p>pip install curlify</p>
</blockquote>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">curlify</span>
<span class="k">print</span><span class="p">(</span><span class="n">curlify</span><span class="p">.</span><span class="n">to_curl</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">request</span><span class="p">))</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl -X GET -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate' -H 'Connection: keep-alive' -H 'User-Agent: python-requests/2.21.0' http://httpbin.org/json
</code></pre></div></div>
<h2 id="using-pandas-to-explore-json-documents">Using Pandas to explore JSON documents</h2>
<p><a href="https://pandas.pydata.org/">Pandas</a> is a data analysis and manipulation library that’s popular in the <a href="https://github.com/jakevdp/PythonDataScienceHandbook">Data Science</a> community. I find it very useful to explore JSON documents.</p>
<p>Let’s first install the package (you might need to use <code class="language-plaintext highlighter-rouge">pip3</code>):</p>
<blockquote>
<p>pip install pandas</p>
</blockquote>
<p>Now let’s take a look how it would work without pandas:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">request</span><span class="p">(</span><span class="s">'GET'</span><span class="p">,</span> <span class="s">'https://api.github.com/users/janahrens/repos'</span><span class="p">)</span>
<span class="n">json</span> <span class="o">=</span> <span class="n">r</span><span class="p">.</span><span class="n">json</span><span class="p">()</span>
<span class="n">json</span><span class="p">.</span><span class="n">__class__</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>list
</code></pre></div></div>
<p>We now know that the call returns a JSON list. Let’s examine what items this list has by looking at the first one.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">json</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">keys</span><span class="p">()</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dict_keys(['id', 'node_id', 'name', 'full_name', 'private', 'owner', 'html_url', 'description', 'fork', 'url', 'forks_url', 'keys_url', 'collaborators_url', 'teams_url', 'hooks_url', 'issue_events_url', 'events_url', 'assignees_url', 'branches_url', 'tags_url', 'blobs_url', 'git_tags_url', 'git_refs_url', 'trees_url', 'statuses_url', 'languages_url', 'stargazers_url', 'contributors_url', 'subscribers_url', 'subscription_url', 'commits_url', 'git_commits_url', 'comments_url', 'issue_comment_url', 'contents_url', 'compare_url', 'merges_url', 'archive_url', 'downloads_url', 'issues_url', 'pulls_url', 'milestones_url', 'notifications_url', 'labels_url', 'releases_url', 'deployments_url', 'created_at', 'updated_at', 'pushed_at', 'git_url', 'ssh_url', 'clone_url', 'svn_url', 'homepage', 'size', 'stargazers_count', 'watchers_count', 'language', 'has_issues', 'has_projects', 'has_downloads', 'has_wiki', 'has_pages', 'forks_count', 'mirror_url', 'archived', 'disabled', 'open_issues_count', 'license', 'forks', 'open_issues', 'watchers', 'default_branch', 'permissions'])
</code></pre></div></div>
<p>With the knowledge of available fields, we could now use standard Python methods to further explore the data.</p>
<p>This process gets a lot easier with <a href="https://pandas.pydata.org/">Pandas</a> and it’s <code class="language-plaintext highlighter-rouge">json_normalize</code> function.
With <code class="language-plaintext highlighter-rouge">json_normalize</code> the data gets parsed into a <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html">DataFrame</a>, which is a core data structure for “Two-dimensional, size-mutable, potentially heterogeneous tabular data”. In other words: It represents the data as a table.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">pandas</span> <span class="kn">import</span> <span class="n">json_normalize</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">json_normalize</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">json</span><span class="p">())</span>
<span class="n">df</span><span class="p">.</span><span class="n">shape</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(30, 98)
</code></pre></div></div>
<p>Calling the <code class="language-plaintext highlighter-rouge">.shape</code> method is a good first step to explore the data. It shows that our DataFrame/table has 30 rows and 98 columns.</p>
<p>Let’s see what those columns are:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">df</span><span class="p">.</span><span class="n">columns</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Index(['id', 'node_id', 'name', 'full_name', 'private', 'html_url',
'description', 'fork', 'url', 'forks_url', 'keys_url',
'collaborators_url', 'teams_url', 'hooks_url', 'issue_events_url',
'events_url', 'assignees_url', 'branches_url', 'tags_url', 'blobs_url',
'git_tags_url', 'git_refs_url', 'trees_url', 'statuses_url',
'languages_url', 'stargazers_url', 'contributors_url',
'subscribers_url', 'subscription_url', 'commits_url', 'git_commits_url',
'comments_url', 'issue_comment_url', 'contents_url', 'compare_url',
'merges_url', 'archive_url', 'downloads_url', 'issues_url', 'pulls_url',
'milestones_url', 'notifications_url', 'labels_url', 'releases_url',
'deployments_url', 'created_at', 'updated_at', 'pushed_at', 'git_url',
'ssh_url', 'clone_url', 'svn_url', 'homepage', 'size',
'stargazers_count', 'watchers_count', 'language', 'has_issues',
'has_projects', 'has_downloads', 'has_wiki', 'has_pages', 'forks_count',
'mirror_url', 'archived', 'disabled', 'open_issues_count', 'license',
'forks', 'open_issues', 'watchers', 'default_branch', 'owner.login',
'owner.id', 'owner.node_id', 'owner.avatar_url', 'owner.gravatar_id',
'owner.url', 'owner.html_url', 'owner.followers_url',
'owner.following_url', 'owner.gists_url', 'owner.starred_url',
'owner.subscriptions_url', 'owner.organizations_url', 'owner.repos_url',
'owner.events_url', 'owner.received_events_url', 'owner.type',
'owner.site_admin', 'permissions.admin', 'permissions.push',
'permissions.pull', 'license.key', 'license.name', 'license.spdx_id',
'license.url', 'license.node_id'],
dtype='object')
</code></pre></div></div>
<p>The list of columns itself isn’t a very good demonstration of Pandas analysis capabilities. It gets more useful if we use it’s sorting and filtering capabilities.</p>
<p>Let’s find out what GitHub repositories have the most stars and only select some of the columns:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">df</span><span class="p">.</span><span class="n">sort_values</span><span class="p">(</span><span class="n">by</span><span class="o">=</span><span class="s">'stargazers_count'</span><span class="p">,</span> <span class="n">ascending</span><span class="o">=</span><span class="bp">False</span><span class="p">).</span><span class="n">head</span><span class="p">()[[</span><span class="s">'name'</span><span class="p">,</span> <span class="s">'created_at'</span><span class="p">,</span> <span class="s">'size'</span><span class="p">,</span> <span class="s">'language'</span><span class="p">,</span> <span class="s">'stargazers_count'</span><span class="p">]]</span>
</code></pre></div></div>
<div>
<style scoped="">
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>name</th>
<th>created_at</th>
<th>size</th>
<th>language</th>
<th>stargazers_count</th>
</tr>
</thead>
<tbody>
<tr>
<th>24</th>
<td>threema-protocol-analysis</td>
<td>2014-03-16T14:38:56Z</td>
<td>311</td>
<td>TeX</td>
<td>17</td>
</tr>
<tr>
<th>11</th>
<td>ipconfig-http-server</td>
<td>2014-05-12T06:15:38Z</td>
<td>152</td>
<td>C</td>
<td>6</td>
</tr>
<tr>
<th>29</th>
<td>yesod-oauth-demo</td>
<td>2012-05-15T21:02:29Z</td>
<td>216</td>
<td>Haskell</td>
<td>5</td>
</tr>
<tr>
<th>27</th>
<td>xing-api-haskell</td>
<td>2013-01-28T07:28:41Z</td>
<td>508</td>
<td>Haskell</td>
<td>5</td>
</tr>
<tr>
<th>4</th>
<td>dotfiles</td>
<td>2011-09-05T09:39:29Z</td>
<td>2337</td>
<td>Shell</td>
<td>5</td>
</tr>
</tbody>
</table>
</div>
<p>We can also request entries in the table using the <code class="language-plaintext highlighter-rouge">.iloc</code> method. The table can be transformed (change rows and columns) with the <code class="language-plaintext highlighter-rouge">.T</code> method:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">df</span><span class="p">.</span><span class="n">iloc</span><span class="p">[[</span><span class="mi">0</span><span class="p">]].</span><span class="n">T</span>
</code></pre></div></div>
<div>
<style scoped="">
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>0</th>
</tr>
</thead>
<tbody>
<tr>
<th>id</th>
<td>207344689</td>
</tr>
<tr>
<th>node_id</th>
<td>MDEwOlJlcG9zaXRvcnkyMDczNDQ2ODk=</td>
</tr>
<tr>
<th>name</th>
<td>alb-fargate-demo</td>
</tr>
<tr>
<th>full_name</th>
<td>JanAhrens/alb-fargate-demo</td>
</tr>
<tr>
<th>private</th>
<td>False</td>
</tr>
<tr>
<th>...</th>
<td>...</td>
</tr>
<tr>
<th>license.key</th>
<td>NaN</td>
</tr>
<tr>
<th>license.name</th>
<td>NaN</td>
</tr>
<tr>
<th>license.spdx_id</th>
<td>NaN</td>
</tr>
<tr>
<th>license.url</th>
<td>NaN</td>
</tr>
<tr>
<th>license.node_id</th>
<td>NaN</td>
</tr>
</tbody>
</table>
<p>98 rows × 1 columns</p>
</div>
<p>Pandas can do a lot more and it’s definetely worth to take a look at the <a href="https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html">10 minutes to pandas</a> guide.</p>
<h2 id="bonus-generating-a-blog-post-from-a-jupyter-notebook">Bonus: Generating a blog post from a Jupyter notebook</h2>
<p>My blog gets generated by feeding Markdown files <a href="https://help.github.com/en/github/working-with-github-pages/setting-up-a-github-pages-site-with-jekyll">into Jekyll</a>. Using <a href="https://nbconvert.readthedocs.io/en/latest/">nbconvert</a> I was able to convert this notebook into a Markdown file. The only thing I had to add manually was the header for Jekyll. The rest of this post is directly from the notebook.</p>
<p>First install the nbconvert package</p>
<blockquote>
<p>pip install nbconvert</p>
</blockquote>
<p>Then you can invoke nbconvert on this file:</p>
<blockquote>
<p>python -m nbconvert files/explore-apis.ipynb –to markdown –stdout > _posts/2020-03-02-explore-apis.markdown</p>
</blockquote>Jan AhrensRecently a colleague shared a useful technique for exploring Web APIs with me: Jupyter notebooks.Videos as podcasts2019-11-03T00:00:00+01:002019-11-03T00:00:00+01:00https://blog.jan-ahrens.eu/2019/11/03/videos-as-podcasts<p>Today I discovered that you can easily convert YouTube (and a <a href="http://ytdl-org.github.io/youtube-dl/supportedsites.html">few more</a>) videos to audio files using <a href="https://youtube-dl.org/">youtube-dl</a>.</p>
<p>I’m found that these parameters work best for me:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>youtube-dl <span class="nt">-x</span> <span class="nt">--audio-format</span> mp3 <span class="s2">"https://www.youtube.com/watch?v=2u0sNRO-QKQ"</span>
</code></pre></div></div>
<p>If you’re an <a href="https://overcast.fm/">Overcast</a> premium subscriber you can easily <a href="https://marco.org/2016/03/14/overcast25">upload</a> the extracted files and listen on the go.</p>
<p>I hope you find this useful, too.</p>Jan AhrensToday I discovered that you can easily convert YouTube (and a few more) videos to audio files using youtube-dl.Small Docker images with embedded go binaries2019-08-10T00:00:00+02:002019-08-10T00:00:00+02:00https://blog.jan-ahrens.eu/2019/08/10/multi-stage-go<p>I recently wanted to include <a href="https://golang.org">Go</a> binaries in a Docker image and tried to make the image as small as possible. My goal was to speed up its execution and download time.</p>
<p>The solution is actually pretty simple:</p>
<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> golang:1.12.7-alpine3.10 AS builder</span>
<span class="k">RUN </span>apk <span class="nt">--no-cache</span> add git
<span class="k">RUN </span>go get github.com/wakeful/yaml2json
<span class="k">RUN </span>go get github.com/santhosh-tekuri/jsonschema/cmd/jv
<span class="k">FROM</span><span class="s"> alpine:3.10</span>
<span class="k">WORKDIR</span><span class="s"> /root/</span>
<span class="k">COPY</span><span class="s"> --from=builder /go/bin/yaml2json /usr/local/bin</span>
<span class="k">COPY</span><span class="s"> --from=builder /go/bin/jv /usr/local/bin</span>
</code></pre></div></div>
<p>The trick is to use Dockers’ <a href="https://docs.docker.com/develop/develop-images/multistage-build/">multi-stage build</a> feature. While Docker is building the image, it actually creates another image called “builder” first. This image gets discarded after the binaries got copied over to the final image and thus the final image doesn’t need the entire go build system.</p>
<p>One caveat is that the builder image and the final image have to be binary compatible. In my first attempt I used the <code class="language-plaintext highlighter-rouge">golang-1.12.7</code> image for the builder image. The binaries got build successfully but wouldn’t run in the final image. The reason for that was that this image was using <a href="https://www.debian.org/News/2019/20190706">Debian buster</a> under the hood and not <a href="https://alpinelinux.org">Alpine Linux</a>. Fortunately there’s also the <code class="language-plaintext highlighter-rouge">golang-1.12.7-alpine3.10</code> image available on Dockerhub that is based on the same Alpine version that the final image uses.</p>
<p>If you want to experiment with this, you can find the complete code in my <a href="https://github.com/JanAhrens/multi-stage-go">multi-stage-go</a> repository.</p>Jan AhrensI recently wanted to include Go binaries in a Docker image and tried to make the image as small as possible. My goal was to speed up its execution and download time.Hello Signal, Goodbye PGP2019-08-04T00:00:00+02:002019-08-04T00:00:00+02:00https://blog.jan-ahrens.eu/2019/08/04/good-bye-pgp<p>After years of maintaining my key I decided to drop PGP and move on to <a href="https://signal.org">Signal</a>. In this post I’ll explain why I made this decision.</p>
<p>For a long time I thought that I needed to maintain a PGP key. It was just something that you had to do if you wanted to be serious about your online life.</p>
<p>I went to great lengths to <a href="https://alexcabal.com/creating-the-perfect-gpg-keypair">create the perfect gpg keypair</a>. My master key got stored offline in a secure location and I used a <a href="https://www.yubico.com/products/yubikey-hardware/">YubiKey</a> to access my private subkey. To establish trust I went to a keysigning party at the <a href="https://www.ccc.de/en/updates/2013/30c3">30C3</a> and managed to collect 43 signatures. Every year I went through the process of renewing my key to ensure that if I’d loose access some day, it won’t be valid forever. The next renewal is due later this year and I’m gonna give it a pass. It’s time to say goodbye to my old friend <a href="http://pgp.mit.edu/pks/lookup?op=vindex&search=0xB911E6A22B4F3B5F">B911 E6A2 2B4F 3B5F</a>.</p>
<p>Let’s face it: Almost nobody uses PGP. Its adoption rate is very low despite being around since 1991. Maybe it’s because it’s too complicated, even though tools like <a href="https://gpgtools.org/">GPG Suite</a> make it very easy. The problem remains that not enough people use it 28 years after its initial release to make a difference.</p>
<p>Encrypting email isn’t a good idea in general. Put all complications with PGP and its adopting rate aside, you can only encrypt email bodies. Still, all metadata is unencrypted. Your provider and other parties can see who is communicating with whom and which email subject is used.</p>
<p>Another problem is the lack of forward secrecy. Once your PGP key gets compromised, all your past communication is also (potentially) compromised. Moxie Marlinspike explains the problem <a href="https://vimeo.com/124887048">in a talk</a> where he introduces the Signal protocol.</p>
<p>If you want to read more about the many problems of PGP, I can recommend <a href="https://latacora.micro.blog/2019/07/16/the-pgp-problem.html">The PGP Problem</a>.</p>
<p>That’s why I decided to use <a href="https://signal.org">Signal</a> instead. It’s an <a href="https://github.com/signalapp">open-source project</a> that uses modern cryptography (elliptic curve) and provides <a href="https://signal.org/docs/specifications/doubleratchet/">forward secrecy</a>. Signal is available as a mobile app for <a href="https://play.google.com/store/apps/details?id=org.thoughtcrime.securesms">Android</a> and <a href="https://apps.apple.com/us/app/signal-private-messenger/id874139669">iOS</a>. You can also use Signal on <a href="https://signal.org/download/">Linux, Windows and macOS</a> - so basically on every platform. You don’t need to do anything special to get started. Just download the app, register an account and you’re all set. No key generation, no key exchange. Signal trusts the keys of your contact on first use (<a href="https://en.wikipedia.org/wiki/Trust_on_first_use">TOFU</a>) and alerts you if it changes.</p>
<p>When I migrated to Signal, I wanted to provide everyone with the opportunity to contact me. As it requires a phone number to register, I would have had to publish my private number (I wasn’t willing to do that). Luckily, I found a well-written guide that explains how to <a href="https://theintercept.com/2017/09/28/signal-tutorial-second-phone-number/">use Signal without giving out your phone number</a>. In that guide there are a couple of ways listed on how you can obtain an alternative phone number. I ended up using <a href="https://www.satellite.me/">satellite</a> which is a VoIP app that is only available in Germany. Even though satellite doesn’t support text messages, I could still register the account because Signal has a phone call fallback for verification.</p>
<p>I recommend that you give Signal a try and stop worrying about PGP. You can send me your feedback at <a href="https://signal.org/download">+49 156 7856 2789</a>.</p>Jan AhrensAfter years of maintaining my key I decided to drop PGP and move on to Signal. In this post I’ll explain why I made this decision.cron is dead, long live launchd!2017-01-13T00:00:00+01:002017-01-13T00:00:00+01:00https://blog.jan-ahrens.eu/2017/01/13/cron-is-dead-long-live-launchd<p>Now that I finally created my <a href="https://tarsnap.com">tarsnap</a> backup script, how
do I execute it regularly? Oh, I know: My Mac is just an Unix system, I’ll use
cron!</p>
<p>At least that’s what I thought I’ll do. After a few attempts to get cron to do
the job, I learned that there’s a better way on macOS:
<a href="https://en.wikipedia.org/wiki/Launchd">launchd</a>.</p>
<p>launchd does a lot more than executing scripts cron-style. Like
<a href="https://en.wikipedia.org/wiki/Systemd">systemd</a> on Linux, launchd is a
replacement for a lot of old school Unix tools, like cron, inetd, init,
<a href="https://en.wikipedia.org/wiki/Launchd#History">etc</a>.</p>
<p>At it’s core, launchd distincts daemons and agents. Dameons are processes that
always run in the background, while agents describe regular jobs that are to be
executed on certain events. There are a lot of different events to choose from.
For example you can trigger an agent, when a device gets mounted, when a file
gets created, or when a certain time arrives.</p>
<p>What really helped me in learning how to write my first launchd agent was
<a href="http://www.launchd.info/">launchd.info</a>. Unlike the <a href="https://developer.apple.com/library/content/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html">Apple
documentation</a>,
it contains useful snippets and concise explanations. I highly recommend that
you also have a look at the launchd agents that some of your applications put into
<code class="language-plaintext highlighter-rouge">~/Library/LaunchAgents</code>.</p>
<p>Below you can see the agent that I ended up creating. You can learn how to
load/unload agents and about the meaning of the different options at
<a href="http://www.launchd.info/">launchd.info</a>.</p>
<p>If you’re testing your script and you don’t want to wait for the next hour to
arrive, you can start it immediately with <code class="language-plaintext highlighter-rouge">launchctl start
eu.jan-ahrens.tarsnap</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>eu.jan-ahrens.tarsnap</string>
<key>EnvironmentVariables</key>
<dict>
<key>PATH</key>
<string>/bin:/usr/bin:/usr/local/bin</string>
</dict>
<key>ProgramArguments</key>
<array>
<string>/bin/bash</string>
<string>/Users/jan/bin/run-tarsnap-backup</string>
</array>
<key>StartInterval</key>
<integer>3600</integer>
<key>StandardOutPath</key>
<string>/Users/jan/.tarsnap.log</string>
<key>StandardErrorPath</key>
<string>/Users/jan/.tarsnap.log</string>
<key>KeepAlive</key>
<dict>
<key>NetworkState</key>
<true/>
</dict>
<key>ExitTimeout</key>
<integer>900</integer>
<key>Nice</key>
<integer>10</integer>
</dict>
</plist>
</code></pre></div></div>
<p>P.S.: cron itself is implemented as a launchd daemon. You can find it at <code class="language-plaintext highlighter-rouge">/System/Library/LaunchDaemons/com.vix.cron.plist</code>.</p>Jan AhrensNow that I finally created my tarsnap backup script, how do I execute it regularly? Oh, I know: My Mac is just an Unix system, I’ll use cron!The German income tax algorithm2015-12-13T00:00:00+01:002015-12-13T00:00:00+01:00https://blog.jan-ahrens.eu/2015/12/13/the-german-tax-algorithm<p>You might have heard it before: The German tax system is
complicated. There are a multitude of rules that can be applied. But
how complicated is it exactly? And how the heck is my income tax
calculated? I recently started to find answers to these questions.</p>
<p>The “Lohnsteuer”, as it’s called in German, used to be calculated with
the help of a simple table - the “Lohnsteuer-Tabelle”. You could look up your
annual salary to find out how much taxes you had to pay. In 2004 this
changed (The German Wikipedia has
<a href="https://de.wikipedia.org/wiki/Lohnsteuertabelle">some details</a> on
this - unfortunately only in German).</p>
<p>The interesting thing about it is that the German government decided
to replace the table with an algorithm. I think, their intention was
quite obvious: Algorithms don’t leave much room for interpretation.</p>
<p>Publishing an algorithm is a good idea, but how exactly do you do
this? To start with, you could publish an implementation in a well
known language - for example in Java. Java developers would like
this. They could simply use this implementation.</p>
<p>However, luckily not everyone uses Java. Having only one
implementation in a popular language is problematic for developers
that need to use other languages (think about iOS apps). The
government agency that published the algorithm could of course release
implementations in different languages. This leads to a scalability
problem. To name a few: Java, C, Ruby, Erlang, Lua, Go, Scheme,
Haskell, JavaScript, VisualBasic, C#, SmallTalk and probably much
more. To complicate things, implementing the algorithm isn’t a onetime
task. Laws change and with them the algorithm. The solution that was
chosen by the German tax office is both, fascinating and shocking:
They’re publishing
<a href="https://en.wikipedia.org/wiki/Flowchart">flowcharts</a>.</p>
<p class="text-center">
<a href="/assets/lohnsteuer-flowchart.png"><img src="/assets/lohnsteuer-flowchart-small.png" /></a>
</p>
<p>Yup! You heared right. They’re publishing a
<a href="https://www.bundesfinanzministerium.de/Content/DE/Downloads/Steuern/Steuerarten/Lohnsteuer/Programmablaufplan/2015-11-16-PAP-2016-anlage-2.pdf?__blob=publicationFile&v=3">thirty-something page document</a>
that includes a list of input parameters, a list of output parameters
and a flowchart.</p>
<p>To be honest: This was the first time I ever saw a flowchart outside
of an education context. “Are they really expecting people to implement
their algorithm by reading the flowchart?”, I thought. It seems like
an error-prone and tedious task: translate thirty pages of flowchart
blocks into an implemenation for the language of your choice.</p>
<p>And it’s true: The task is error-prone and tedious. That’s why there’s
a table at the end of the flowchart document. Ironically, that table
is quite similar to the “Lohnsteuer-Tabelle” that was used before.</p>
<p>Then it got me thinking. Isn’t there a better way to communicate an
algorithm in a way that it can be implemented in any programming
language? Something like a meta language?</p>
<p>Along with their flowchart document, they
<a href="https://www.bmf-steuerrechner.de/javax.faces.resource/daten/xmls/Lohnsteuer2020.xml.xhtml">published XML files</a>
that should achieve exactly that (they call it “XML pseudocode”). In
my opinion, they didn’t quite succeed. Although, the file contains
elements for abstract concepts, like branches, it still includes
expressions that hide in regular text. Those expressions aren’t
abstracted from the programming language. It’s quite clear that they
were written with Java in mind.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><IF</span> <span class="na">expr=</span><span class="s">"STKL == 4"</span><span class="nt">></span>
<span class="nt"><THEN></span>
<span class="nt"><EVAL</span> <span class="na">exec=</span><span class="s">"SAP= BigDecimal.valueOf (36)"</span><span class="nt">/></span>
<span class="nt"><EVAL</span> <span class="na">exec=</span><span class="s">"KFB= (ZKF.multiply (BigDecimal.valueOf (3624))).setScale (0, BigDecimal.ROUND_DOWN)"</span><span class="nt">/></span>
<span class="nt"></THEN></span>
<span class="nt"><ELSE></span> <span class="c"><!-- ... --></span> <span class="nt"></ELSE></span>
<span class="nt"></IF></span>
</code></pre></div></div>
<p>In the end, I got curious and started to implement the algorithm
straight from the flowchart. I chose Ruby, because there’s no Ruby
implementation, yet. Also I’m quite fluent in it. It took me a whole
evening to parse and implement the algorithm by following the
flowchart boxes. After that I spent an additional evening to find the
mistakes I made while implementing it. Maybe it would have been wiser
to write a compiler for the XML format.</p>
<p>Nevertheless, I published the results as a Ruby gem:
<a href="https://rubygems.org/gems/lohnsteuer">lohnsteuer</a>. I hope this will
be helpful for you.</p>Jan AhrensEver wondered how complicated the German tax system is? I wrote a Ruby implementation of the tax algorithm. In this post I describe what I learned along the way.Analyse app traffic with mitmproxy2015-09-22T00:00:00+02:002015-09-22T00:00:00+02:00https://blog.jan-ahrens.eu/2015/09/22/traffic-analysis-with-mitmproxy<p><em>This post is part of a series on reverse engineering mobile apps.</em></p>
<ol>
<li><a href="/2015/09/07/why-reverse-engineer.html">Why reverse engineer mobile apps?</a></li>
<li><strong>Analyse app traffic with mitmproxy</strong></li>
</ol>
<hr />
<p>When you want to inspect an app’s behavior, the first step is to look
at the traffic it produces. In this post I’ll show you how to do this
with “mitmproxy”.</p>
<p>Fortunately, most apps encrypt their traffic nowadays. This means that
you will have a hard time using a regular sniffer like
<a href="http://www.tcpdump.org/">tcpdump</a> or
<a href="https://www.wireshark.org/">Wireshark</a>. Instead you have to do a
<a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">man-in-the-middle attack</a>
to see what traffic the app produces. I’m using
<a href="https://mitmproxy.org/doc/mitmproxy.html">mitmproxy</a> for this.</p>
<p>After you <a href="https://mitmproxy.org/doc/install.html">installed</a> and started mitmproxy on your computer, you need
to configure your phone to send its traffic through your computer.</p>
<p>On Android, this can be done by connecting both devices with the same
Wi-Fi. Your computer needs to serve as a proxy server for your phone.
This can be configured under
“Settings → Wi-Fi”. Press long on the network you’re
connected to, select “Advanced options” and put your computer’s IP
address as “Proxy hostname” and the mitmproxy port as “Proxy port”
(the default setting is 8080).</p>
<p><a href="/assets/android-proxy.png"><img src="/assets/android-proxy-small.png" alt="" /></a></p>
<p>To setup mitmproxy for other platforms, have a look at <a href="http://docs.mitmproxy.org/en/latest/modes.html">the
documentation</a>.</p>
<p>Once your app sends its traffic through your computer, you need to get
your phone to trust the mitmproxy certificate authority. This process
has to be repeated every time you change the CA in mitmproxy. For
Android, iOS and Windows phone, this is very easy. Just open the
browser on your phone, visit <a href="http://mitm.it">mitm.it</a>, select your
platform and that’s it.</p>
<p>Now you can open the app you want to inspect and see the unencrypted
traffic in mitmproxy on your computer.</p>
<p>If you don’t see traffic popping up or the app is not getting any
data, this might have various reasons. Before assuming that the MITM
attack failed, you should keep in mind that the data might be still
cached on the device. For example, try “pull-to-refresh” or closing and
reopening the app to force it to get fresh data.</p>
<p>It’s possible that the app doesn’t respect the proxy settings on the
phone and thus escapes the MITM attack. This would be the case if the
app still gets data and you don’t see activity in mitmproxy. You can
use
<a href="http://docs.mitmproxy.org/en/latest/modes.html">one of the other techniques</a>
to intercept the traffic, when you suspect that the setting
isn’t respected.</p>
<p>Sometimes app developers configure their app not to trust the certificate
authorities provided by the mobile phone. They bundle
the correct certificate or certificate authority with their app. This
technique is called
<a href="https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning">Certificate Pinning</a>.
In those cases it’s not possible to inspect the traffic even if the
traffic gets routed through your computer. The app simply rejects the
fake certificate authority and refuses to connect to its servers. You can
spot those cases when neither the app nor mitmproxy get any data. If we want to continue our analysis, we have to go one step further and replace the pinned
certificate or certificate authority inside the app.</p>
<p>It’s also possible that the app isn’t using HTTP(S) to get its
data. I came across such a situation when I tried to inspect the traffic of the
<a href="/2014/03/22/threema-protocol-analysis.html">Threema app</a>.</p>
<p>Threema opens a regular socket and implements its own
protocol. Prebuild MITM tools can’t handle those situations because
they know nothing about the protocol. Even though the apps traffic
reaches the app, it can’t MITM attack it. Once again, simple traffic
analysis doesn’t help here.</p>
<p>Although, traffic analysis has it’s limitations I still recommend it
as a first step. It’s simple to setup and a lot of apps can be
analysed in a short amount of time.</p>
<hr />
<p><em>In the next post I will show you how to learn more about an app by
looking at it’s binary. This post will focus purely on Android apps.</em></p>Jan AhrensIn the second post of my "reverse engineering mobile apps" series, I give an introduction on how to analyse the traffic of apps.Why reverse engineer mobile apps?2015-09-07T00:00:00+02:002015-09-07T00:00:00+02:00https://blog.jan-ahrens.eu/2015/09/07/why-reverse-engineer<p><em>This post is part of a series on reverse engineering mobile apps.</em></p>
<ol>
<li><strong>Why reverse engineer mobile apps?</strong></li>
<li><a href="/2015/09/22/traffic-analysis-with-mitmproxy.html">Analyse app traffic with mitmproxy</a></li>
</ol>
<hr />
<p>Mobile apps are in demand - for several years now. Every business
needs an app, even tiny ones. Sometimes I’m under the impression that
building mobile apps became, what was in the early 2000s, building
websites. “Hey, your business doesn’t have a website, therefore it isn’t
future proof. Here, let me build (and sell) you one.” Replace the word
“website” with “mobile app” and you have today’s situation.</p>
<p>Nevertheless this post isn’t to moan about the status quo. It’s about
knowing what’s going on with your data. If you think about it, “mobile
app” is just a fancy word for “software”. Software, that runs on your
mobile phone. Software, that messes with your data. Data like your
current location, your contacts, your photos, and so on. Most of the
time you’ll have a pretty good understanding of what your data is
being used for, but what if there are doubts? In the past, there were
apps, that
<a href="https://venturebeat.com/2012/02/14/iphone-address-book/">stole your address book</a>,
in order to provide you with “valuable insights and opportunities”. Do
you want this?</p>
<p>The different mobile platforms offer different security models to
manage the access to your data. On iOS, for example, the user is asked
if she wants to give an app access to her address book. When a user
installs an app on Android, he has to confirm, whether it gets access
to his address book. There is no choice left. Either you give access,
or you can’t install the app at all. No matter how an app got access
to the data, how will a user know what happens with it next?</p>
<p>Image a local transport providers’ app, that gives you the directions
to the homes of your friends. In order to simplify the process, the
app uses the address book data to now where your friends live. It’s
not necessary to upload the whole address book to build this
feature. The user selects a friend’s address and the app sends this
particular information to the server, in order to give directions. On
the other hand, with the next update, the local transport provider
could change their implementation and upload every address to their
servers. Maybe they want to do this, to suggest that you visit your
friends more often - automatically: “Hey, it’s time to visit Jane
again. It’s only 15 minutes if you take the next subway”. Wouldn’t
this be an innovative feature?</p>
<p>The point I want to make is that we don’t know when and how apps use
our data. If the app is Open Source Software (or even
<a href="https://en.wikipedia.org/wiki/Free_and_open-source_software">Free Software</a>),
we can have a look at its source code. Unfortunately, it’s not that
easy. How do you know that the source code was used to build the
binary, you downloaded from the app store?
<a href="https://wiki.debian.org/ReproducibleBuilds/About">Reproducible builds</a>
could help, but providing them isn’t a trivial task. Debian is working
hard to make
<a href="https://reproducible.debian.net/reproducible.html">every binary package reproducible</a>,
but they’re still not there, yet. I’m not aware that there’s a similar
project for mobile apps.</p>
<p>The source code of most commercial apps isn’t available anyway. In
those cases it’s good to have a closer look at the app’s behavior from
time to time. I feel it’s important to do so, to raise the companies
awareness, make them fear bad press, and keep them from messing around
with your data. The more people know about how to do this, the more
pressure will be put on the app owners.</p>
<hr />
<p><em>In the <a href="/2015/09/22/traffic-analysis-with-mitmproxy.html">next post</a> of this series, I’ll be looking at your options to
find out what an app is doing.</em></p>Jan AhrensIn the first post of my "reverse engineering mobile apps" series, I'm arguing why you should learn how to reverse engineer apps.tmux - my terminal window manager2015-02-20T00:00:00+01:002015-02-20T00:00:00+01:00https://blog.jan-ahrens.eu/2015/02/20/tmux<p>A dark full-screen terminal window. That is what you see mostly on my
screen. My fascination with the terminal started when I found out how
to run <a href="https://en.wikipedia.org/wiki/COMMAND.COM">COMMAND.COM</a> on my
parents Windows computer. What a joy! I could edit files with “edit”,
start “fdisk” and explore the file system with “DIR”. I felt like one of
those cool hackers from the movies! A few years later I was lucky
enough to discover that Windows and COMMAND.COM isn’t the most
effective way to work and I started to learn about Linux and Bash.</p>
<p>Nowadays I’m still using terminal windows to get stuff done. Until
recently it really where windows. Many windows. My favorite shortcut
on my Linux machine was <code class="language-plaintext highlighter-rouge">Ctrl-Alt-t</code> - start a new terminal. At work
I’m using a Mac. My muscle memory contains key combinations to create
terminal windows and switch between them on both systems. It was
okay. It never really bothered me - until a colleague introduced me to
“tmux” - the <strong>t</strong>erminal<strong> mu</strong>ltiple<strong>x</strong>er.</p>
<p class="text-center">
<img src="/assets/tmux.png" alt="tmux" />
</p>
<p>The idea of tmux is really simple. Instead of having multiple windows
with one shell prompt each, why not have a single terminal window with
many shell prompts. tmux saves me from opening a lot of terminal
windows. But wait, there’s more!</p>
<p>You can split windows (horizontally and vertically), scroll and search
through the terminal output, copy-and-paste from inside tmux, manage
groups of windows and use it for
<a href="http://pivotallabs.com/how-we-use-tmux-for-remote-pair-programming/">remote pair programming</a>.
Interested? Then let’s get started with the vocabulary.</p>
<h3 id="vocabulary">Vocabulary</h3>
<p>Windows, panes and sessions. Those are the building blocks. When I
started to learn about tmux it took me some time to get the
meaning. What helped me was to map the tmux concepts to those of my
beloved terminal programs iTerm and gnome-terminal.</p>
<p>You can think of tmux as the equivalent of your terminal program (
iTerm, gnome-terminal). A tab in your terminal program is called
“window” in tmux. When you would open another window inside your
terminal program, you start a new “session” in tmux. Splitting one tab
of your terminal window tabs into different parts is what tmux calls
splitting a window into “panes”. Unfortunately you can’t do that with
gnome-terminal, so it’s not included on the screenshot.</p>
<p class="text-center">
<img src="/assets/tmux-terminal-analogy.png" alt="tmux terminal window analogy" />
</p>
<h3 id="the-prefix-key">The prefix key</h3>
<p>The first thing that my colleague recommended me to configure was the
“prefix key”. It’s the key combination that tmux intercepts to
recognize its commands. By default the prefix key is
<a href="http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/tmux.1#x4b45592042494e44494e4753"><code class="language-plaintext highlighter-rouge">Ctrl-b</code></a>.
This combination is also used in Emacs to
<a href="https://www.gnu.org/software/emacs/manual/html_node/emacs/Moving-Point.html">move one character backward</a>
so I ended up
<a href="https://github.com/JanAhrens/dotfiles/blob/6cc0a7ad7e1292f2dfb77580cb4f72fb42fd8d52/.tmux.conf#L5-L7">changing the prefix key</a>
to <code class="language-plaintext highlighter-rouge">Ctrl-]</code>. It’s easy to type on an English keyboard and not bound
in any of the programs I use. In most posts about tmux you’ll see that
people reference the prefix key as <code class="language-plaintext highlighter-rouge">prefix</code> instead of assuming that
it’s <code class="language-plaintext highlighter-rouge">Ctrl-b</code>. I’ll do the same and write for example <code class="language-plaintext highlighter-rouge">prefix c</code>
instead of <code class="language-plaintext highlighter-rouge">Ctrl-b c</code> (default prefix key) or <code class="language-plaintext highlighter-rouge">Ctrl-] c</code> (my custom
prefix key).</p>
<p>If you haven’t done already it’s now time to start tmux and follow
along. Reading about tmux is not as much fun as reading about tmux and
immediately trying it. Just open your regular terminal program,
install tmux and start a new session with <code class="language-plaintext highlighter-rouge">tmux new</code>.</p>
<h3 id="windows">Windows</h3>
<p>Type <code class="language-plaintext highlighter-rouge">prefix c</code> to create a new window. At the bottom of the screen
you see the “status line”. An asterisks next to the window name shows
the active window. It should currently be the second one, because you
just created a new window.</p>
<p>tmux automatically uses the name of the current running program as the
name of the window. If you’re running a Bash shell, your window will
be named “bash”. When you run a program inside that Bash (like “man
tmux”), the name of the window will change to “man”. If you don’t want
your window to automatically change its name, press <code class="language-plaintext highlighter-rouge">prefix ,</code> to
assign a name. Try it by naming your newly created window “demo”.</p>
<p>tmux starts to count its windows from “0” on. To switch back to the
first window, type <code class="language-plaintext highlighter-rouge">prefix 0</code>. You can also used <code class="language-plaintext highlighter-rouge">prefix n</code> and
<code class="language-plaintext highlighter-rouge">prefix p</code> to jump to the next and previous window. That enough with
windows for now, let’s focus on panes. Close your “demo” window
(<code class="language-plaintext highlighter-rouge">prefix 1</code>) by ending the shell process (<code class="language-plaintext highlighter-rouge">exit</code> or <code class="language-plaintext highlighter-rouge">Ctrl-d</code>).</p>
<h3 id="panes">Panes</h3>
<p>To split your current window in two horizontal panes type <code class="language-plaintext highlighter-rouge">prefix "</code>.
You can move to the next pane with <code class="language-plaintext highlighter-rouge">prefix o</code>. A very cool feature is
to directly jump to a pane by typing <code class="language-plaintext highlighter-rouge">prefix q</code> (you’ll see numbers
appearing in the panes) and then press the number of the pane you want
to jump to (<code class="language-plaintext highlighter-rouge">prefix q 0</code> for example). That feature blew my mind when
I discovered it.</p>
<p>If you want to focus on one pane you can zoom it with <code class="language-plaintext highlighter-rouge">prefix z</code>. In
the status bar you’ll see a “Z” appearing next to the window
name. Press the zoom key again to shrink the pane back to its original
size.</p>
<p>Vertically panes can be created with <code class="language-plaintext highlighter-rouge">prefix %</code>. Closing a pane
requires you to end its process (for example the shell). If for any
reason that process gets stuck (for example a dead ssh connection) use
<code class="language-plaintext highlighter-rouge">prefix x</code> to kill that pane. Don’t worry about accidentally killing a
pane - you’ll be asked to confirm before a pane gets killed. If you
only have one pane, killing the pane means also killing the window.</p>
<h3 id="sessions">Sessions</h3>
<p>Remember: Opening a new session is like opening a new terminal
window. Unfortunately there is no default key combination to create a
new session. You can get around it by going to the tmux command prompt
(<code class="language-plaintext highlighter-rouge">prefix :</code>) and executing the <code class="language-plaintext highlighter-rouge">new-session</code> command.</p>
<p>The first thing that you want to do with your new session is to give
it a name. Having a name for your session will make it much easier to
differentiate between different ones. Press <code class="language-plaintext highlighter-rouge">prefix $</code> to rename
your current session.</p>
<p>Switching between sessions is easy: <code class="language-plaintext highlighter-rouge">prefix s</code> shows a list of
session. You can use the arrow keys, <code class="language-plaintext highlighter-rouge">Ctrl-n</code> and <code class="language-plaintext highlighter-rouge">Ctrl-p</code> (Emacs
keys) or the number of the session to jump to it. When you have more
than two session in that list you’ll be glad that you assigned names.</p>
<p>To close a session you have to close all of its windows. You’ll exit
tmux when you close your current session. Don’t worry though, the
other session are still running in tmux. With <code class="language-plaintext highlighter-rouge">tmux attach</code> you can
return to them. Speaking of attaching: Whenever you feel the need to
exit tmux, you can press <code class="language-plaintext highlighter-rouge">prefix d</code> to detach tmux from your terminal
window.</p>
<h3 id="how-im-using-tmux">How I’m using tmux</h3>
<p>The session where I’m currently writing this post is named “blog”. I
started two windows in that session: “edit” and “man”. The edit window
contains two horizontally split panes. One pane running Emacs and the
other pane running <a href="http://jekyllrb.com/">Jekyll</a>. In the “man” window
I’ve opened the tmux manpage.</p>
<p>I’m using sessions to arrange the different kind of things that I do.
When I write code I’m created a new session for that project I’m
working on. I use windows and panes for the compile output, manpages
etc. The first window runs an emacsclient in most of my sessions.</p>
<h3 id="protips">Protips</h3>
<ul>
<li>
<p>Change the prefix key to something that fits your needs. If you’re
using vim, keeping <code class="language-plaintext highlighter-rouge">Ctrl-b</code> should be fine.</p>
</li>
<li>
<p>Force yourself to learn the tmux key combinations. Stop using your
terminal window shortcuts. Press <code class="language-plaintext highlighter-rouge">prefix ?</code> to get all available key
combinations.</p>
</li>
<li>
<p>Change the
<a href="https://github.com/JanAhrens/dotfiles/blob/724838f03acefb0ca46897c46646de8ec8d9949b/.tmux.conf#L9-L11">base index</a>
for panes and windows to “1”. You can then press <code class="language-plaintext highlighter-rouge">prefix 1</code> to get
to the first window and <code class="language-plaintext highlighter-rouge">prefix q 1</code> to get to the first pane. This
is more naturally to me given that the numbers on my keyboard start
from “1” on.</p>
</li>
<li>
<p>Use a tmux theme. I started by forking the
<a href="https://github.com/thiderman/dotfiles/blob/master/tmux/colors/neverland.conf">“neverland” colorscheme</a>
by Lowe Thiderman.</p>
</li>
<li>
<p>Familiarize yourself with the
<a href="http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/tmux.1#x57494e444f575320414e442050414e4553">copy-mode</a>.
Being able to copy and paste everything that appears in tmux is a
killer feature.</p>
</li>
<li>
<p>Use the copy-mode also to scroll and search your current pane.</p>
</li>
</ul>Jan AhrensA dark full-screen terminal window. That is what you see mostly on my screen. My fascination with the terminal started when I found out how to run COMMAND.COM on my parents Windows computer. What a joy! I could edit files with “edit”, start “fdisk” and explore the file system with “DIR”. I felt like one of those cool hackers from the movies! A few years later I was lucky enough to discover that Windows and COMMAND.COM isn’t the most effective way to work and I started to learn about Linux and Bash.JSON Schema talk2014-08-27T00:00:00+02:002014-08-27T00:00:00+02:00https://blog.jan-ahrens.eu/2014/08/27/json-schema<p>Some time ago <a href="https://twitter.com/DASKaJA">Jens</a> and I started the
<a href="http://www.meetup.com/webapi-hamburg/">WebAPI Hamburg Meetup</a>. At the
last meetup I did a lightning talk on JSON Schema.</p>
<p>It’s a rather basic talk with the intention of motivating people to
look into JSON Schema for their APIs. You can have a look at the
slides below:</p>
<div class="auto-resizable-iframe">
<div>
<iframe src="http://blog.jan-ahrens.eu/json-schema-talk"></iframe>
</div>
</div>
<p>If you are on a small screen, you might want to open the
<a href="http://blog.jan-ahrens.eu/json-schema-talk">slides</a> in a new tab.</p>
<p>I created the slides using <a href="https://github.com/gnab/remark/">remark</a>,
which I can highly recommend. You can write your slides using Markdown
and that’s really great.</p>Jan AhrensSome time ago Jens and I started the WebAPI Hamburg Meetup. At the last meetup I did a lightning talk on JSON Schema.