<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="http://jnwllm.be/feed.xml" rel="self" type="application/atom+xml" /><link href="http://jnwllm.be/" rel="alternate" type="text/html" /><updated>2026-06-06T21:35:39+00:00</updated><id>http://jnwllm.be/feed.xml</id><title type="html">jnwllm.be</title><author><name>Janwillem</name></author><entry><title type="html">Survey results: use of generative AI tools by university students</title><link href="http://jnwllm.be/blog/genai-survey" rel="alternate" type="text/html" title="Survey results: use of generative AI tools by university students" /><published>2026-01-30T16:00:00+00:00</published><updated>2026-01-30T16:00:00+00:00</updated><id>http://jnwllm.be/blog/genai-survey</id><content type="html" xml:base="http://jnwllm.be/blog/genai-survey"><![CDATA[<p>After I taught the Multicore Programming course at Vrije Universiteit Brussel last year, I asked my students to fill in a survey, including questions about their use of (generative) AI tools. Below are the results.</p>

<h2 id="q1-which-ai-tools">Q1: Which AI tools?</h2>

<figure style="max-width: 35rem;">
  <img src="/assets/images/genai-survey-tools-question.png" alt="Survey question about which AI tools students use" />
</figure>

<p>Out of 16 respondents, <strong>14 students (88%) indicated using AI</strong> tools during the course; only 2 did not. Most (13 students, 81%) used <strong>chatbots</strong>. Additionally, 3 used IDE extensions such as GitHub Copilot and 1 used an AI-powered IDE like Cursor. (They could select multiple options.)</p>

<p>No one reported using a coding agent. This is probably due to their recency: Claude Code was released only in February 2025 (and is paid) and Gemini CLI in June (for free); while students worked on their projects between March and June 2025.</p>

<p>We can conclude AI is commonplace, especially chatbots. Newer tools like coding agents are still unpopular. It’ll be interesting to see how this evolves in the next few years.</p>

<h2 id="q2-for-which-tasks">Q2: For which tasks?</h2>

<figure style="max-width: 35rem;">
  <img src="/assets/images/genai-survey-use-question.png" alt="Survey question about tasks for which students use AI tools" />
</figure>

<p>Students reported using AI tools for a variety of tasks:</p>

<ul>
  <li>9 students (56%) asked ChatGPT <strong>questions during exercise</strong> sessions. I also observed this in class. This is honestly a bizarre experience as a teacher: students use ChatGPT even though I’m right there. Some students seem to prefer asking their questions to a chatbot over a human being (fear of being judged?). Overall, my experience is that AI chatbots are remarkably effective at spotting and explaining mistakes, and can be very conducive to the learning process.</li>
  <li>6 students (38%) used AI to <strong>code during the exercises</strong>. I saw several students with GitHub Copilot enabled in their IDE. This is something we’ll have to figure out: if the goal of an exercise is to practice, should you use a tool that gives you the answer immediately? Students may simply have the Copilot extension enabled by default in their VSCode, so we’ll need to explicitly ask them to disable it.</li>
  <li>8 students (50%) used AI for questions during their work on <strong>programming projects</strong>, 3 (19%) used it for coding projects, and 4 (25%) for writing reports. My view on this is nuanced: using AI as a companion for brainstorming, debugging, or gaining a deeper understanding can be very educational. However, delegating core parts of the projects to AI means you are no longer learning. Similarly, using AI to improve or spell-check reports is fine, but of course students should remain in control of and responsible for what they write.</li>
  <li>Finally, a surprising 10 students (62%) asked chatbots to further <strong>explain concepts from theory</strong> lectures. This seems like an excellent use of AI, although I wonder how high the risk of hallucinations is. (And, it makes me wonder what I should improve!)</li>
</ul>

<p>In an open question on their experience with AI, many students said it gave them a productivity boost, e.g. for debugging, generating test code and data, writing repetitive code. They were also well aware of its limitations: hallucinations, inconsistencies, and a need to guide the AI. Some mentioned that it had replaced Google or Stack Overflow for them. They indicated it helped them to understand some topics better, but they also emphasized that it remains important to understand and master the material yourself.</p>

<h2 id="q3-payment">Q3: Payment</h2>

<figure style="max-width: 25rem;">
  <img src="/assets/images/genai-survey-payment-question.png" alt="Survey question about payment for AI tools" />
</figure>

<p>4 students (25%) said they paid for AI tools. This is concerning: we don’t want a situation where students who can afford (better) AI tools end up getting a better education or higher grades. This is certainly something to pay attention to in the future.</p>

<h2 id="policy-on-ai-use">Policy on AI use</h2>
<p>Last year, we didn’t yet have a specific policy on the use of AI tools; only our standard policy on plagiarism and an oral defense that checks students’ work and understanding. AI tools were in a gray zone. Next year, we’ll introduce a clear policy, but this will be a tricky balancing act. Student opinions varied widely (paraphrased):<br />
“It should be prohibited, otherwise we don’t learn.”<br />
“It should be allowed only for reports, not for code.”<br />
“It should be allowed, because in the workplace it will also be there.”<br />
“It’s impossible to detect, so there’s no point in prohibiting it anyway.”</p>

<h2 id="remarks-and-disclaimers">Remarks and disclaimers</h2>

<p>Out of 43 students registered for the course, only 16 completed the survey. In my experience, it’s usually the more ‘motivated’ students who fill in surveys, which skews results. In the open questions, many provided long and thoughtful answers, suggesting that this topic is on their minds as well. The survey was labelled as anonymous to encorage honest answers. Finally, this is an elective Master’s course, with students who already know how to program well. For a first-year Bachelor course, my policy would be different than for an advanced course in the Master.</p>

<p><br /></p>

<p><a href="https://www.linkedin.com/feed/update/urn:li:activity:7390762230159462400/">💬 Comment on this post on LinkedIn</a></p>]]></content><author><name>Janwillem</name></author><category term="genai" /><category term="ai" /><category term="education" /><summary type="html"><![CDATA[After I taught the Multicore Programming course at Vrije Universiteit Brussel last year, I asked my students to fill in a survey, including questions about their use of (generative) AI tools. Below are the results.]]></summary></entry><entry><title type="html">10 Clojure gotchas</title><link href="http://jnwllm.be/blog/clojure-gotchas" rel="alternate" type="text/html" title="10 Clojure gotchas" /><published>2026-01-30T15:00:00+00:00</published><updated>2026-01-30T15:00:00+00:00</updated><id>http://jnwllm.be/blog/clojure-gotchas</id><content type="html" xml:base="http://jnwllm.be/blog/clojure-gotchas"><![CDATA[<p>After teaching the <a href="https://soft.vub.ac.be/teaching/multicore/">Multicore Programming course</a> at Vrije Universiteit Brussel for quite a few years, I’ve compiled a list of some common gotchas students encounter when they first program in Clojure:</p>

<ul id="markdown-toc">
  <li><a href="#unexpected-laziness-of-for" id="markdown-toc-unexpected-laziness-of-for">Unexpected laziness of <code class="language-plaintext highlighter-rouge">for</code></a></li>
  <li><a href="#unexpected-laziness-combined-with-multithreading" id="markdown-toc-unexpected-laziness-combined-with-multithreading">Unexpected laziness combined with multithreading</a></li>
  <li><a href="#unexpected-behavior-of-pmap-with-lazy-sequences" id="markdown-toc-unexpected-behavior-of-pmap-with-lazy-sequences">Unexpected behavior of <code class="language-plaintext highlighter-rouge">pmap</code> with lazy sequences</a></li>
  <li><a href="#propagation-of-nil" id="markdown-toc-propagation-of-nil">Propagation of <code class="language-plaintext highlighter-rouge">nil</code></a></li>
  <li><a href="#nil-and--sometimes-act-the-same-sometimes-differently" id="markdown-toc-nil-and--sometimes-act-the-same-sometimes-differently"><code class="language-plaintext highlighter-rouge">nil</code> and <code class="language-plaintext highlighter-rouge">'()</code> sometimes act the same, sometimes differently</a></li>
  <li><a href="#contains-vs-contains" id="markdown-toc-contains-vs-contains"><code class="language-plaintext highlighter-rouge">contains?</code> vs. <code class="language-plaintext highlighter-rouge">.contains</code></a></li>
  <li><a href="#firstrest-lastbutlast-peekpop-on-list-vs-vector" id="markdown-toc-firstrest-lastbutlast-peekpop-on-list-vs-vector"><code class="language-plaintext highlighter-rouge">first</code>/<code class="language-plaintext highlighter-rouge">rest</code>, <code class="language-plaintext highlighter-rouge">last</code>/<code class="language-plaintext highlighter-rouge">butlast</code>, <code class="language-plaintext highlighter-rouge">peek</code>/<code class="language-plaintext highlighter-rouge">pop</code> on list vs. vector</a></li>
  <li><a href="#unexpected-conversion-to-other-sequence-type" id="markdown-toc-unexpected-conversion-to-other-sequence-type">Unexpected conversion to other sequence type</a></li>
  <li><a href="#60-second-wait-before-program-ends-shutdown-agents" id="markdown-toc-60-second-wait-before-program-ends-shutdown-agents">60 second wait before program ends (<code class="language-plaintext highlighter-rouge">shutdown-agents</code>)</a></li>
  <li><a href="#def-is-global-for-ex-schemers" id="markdown-toc-def-is-global-for-ex-schemers"><code class="language-plaintext highlighter-rouge">def</code> is global (for ex-Schemers)</a></li>
  <li><a href="#general-conclusions" id="markdown-toc-general-conclusions">General conclusions</a></li>
</ul>

<p>Note that although some of these are mentioned pretty clearly in the documentation, that doesn’t make them less of a gotcha for newcomers.</p>

<h2 id="unexpected-laziness-of-for">Unexpected laziness of <code class="language-plaintext highlighter-rouge">for</code></h2>

<p>Clojure’s laziness often results in unexpected code for beginners. For instance, what does the following code do?</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="nb">second</span><span class="w"> </span><span class="p">[</span><span class="n">lst</span><span class="p">]</span><span class="w">
  </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"Elements:"</span><span class="p">)</span><span class="w">
  </span><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="p">[</span><span class="n">element</span><span class="w"> </span><span class="n">lst</span><span class="p">]</span><span class="w">
    </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"*"</span><span class="w"> </span><span class="n">element</span><span class="p">))</span><span class="w">
  </span><span class="p">(</span><span class="nb">nth</span><span class="w"> </span><span class="n">lst</span><span class="w"> </span><span class="mi">1</span><span class="p">))</span><span class="w">

</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nb">second</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]))</span><span class="w">
</span></code></pre></div></div>

<p>We define a function <code class="language-plaintext highlighter-rouge">second</code>, which takes a list and returns the second element (position 1). While debugging, we use <code class="language-plaintext highlighter-rouge">for</code> to loop through the list and print its elements. If you’ve never programmed in Clojure before, you probably think this program prints the following:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Elements:
* 1
* 2
* 3
2
</code></pre></div></div>

<p>That’s incorrect! This program actually prints:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Elements:
2
</code></pre></div></div>

<p>This code snippet is confusing Clojure’s <code class="language-plaintext highlighter-rouge">for</code> with a <code class="language-plaintext highlighter-rouge">for</code> loop as you can find it in a traditional language. In Clojure, <code class="language-plaintext highlighter-rouge">for</code> returns a list with the results of its body, and is evaluated lazily. You can consider the <code class="language-plaintext highlighter-rouge">for</code> loop above syntactic sugar for <code class="language-plaintext highlighter-rouge">(map (fn [element] (prinln "*" element)) lst)</code>, which is also evaluated lazily. To get the desired behavior, you should use <code class="language-plaintext highlighter-rouge">doseq</code>:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="nb">second</span><span class="w"> </span><span class="p">[</span><span class="n">lst</span><span class="p">]</span><span class="w">
  </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"Elements:"</span><span class="p">)</span><span class="w">
  </span><span class="p">(</span><span class="nb">doseq</span><span class="w"> </span><span class="p">[</span><span class="n">element</span><span class="w"> </span><span class="n">lst</span><span class="p">]</span><span class="w">
    </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"*"</span><span class="w"> </span><span class="n">element</span><span class="p">))</span><span class="w">
  </span><span class="p">(</span><span class="nb">nth</span><span class="w"> </span><span class="n">lst</span><span class="w"> </span><span class="mi">1</span><span class="p">))</span><span class="w">

</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nb">second</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]))</span><span class="w">
</span></code></pre></div></div>

<p>This code snippet will print the expected output. <code class="language-plaintext highlighter-rouge">doseq</code> is similar to <code class="language-plaintext highlighter-rouge">for</code>, but is evaluated strictly (= not lazily).</p>

<p>However, another important difference between <code class="language-plaintext highlighter-rouge">doseq</code> and <code class="language-plaintext highlighter-rouge">for</code> is that <code class="language-plaintext highlighter-rouge">doseq</code> returns <code class="language-plaintext highlighter-rouge">nil</code>. If you want to force evaluation of <code class="language-plaintext highlighter-rouge">for</code>, <code class="language-plaintext highlighter-rouge">map</code>, or another construct that generates a lazy sequence, you can surround it with <code class="language-plaintext highlighter-rouge">doall</code>.</p>

<p>Some examples:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">print-and-increment</span><span class="w"> </span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="w">
  </span><span class="p">(</span><span class="nb">print</span><span class="w"> </span><span class="n">i</span><span class="p">)</span><span class="w">
  </span><span class="p">(</span><span class="nb">+</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="mi">1</span><span class="p">))</span><span class="w">

</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="p">(</span><span class="nb">map</span><span class="w"> </span><span class="n">print-and-increment</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]))</span><span class="w">
</span><span class="c1">; prints nothing - map is lazy</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nb">last</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w">
</span><span class="c1">; prints 1234 ⚠️</span><span class="w">
</span><span class="c1">; the first 123 is the 'delayed' lazy code from inside the function,</span><span class="w">
</span><span class="c1">; 4 is the value of (last a) (= 3 + 1).</span><span class="w">

</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">b</span><span class="w">
  </span><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="p">[</span><span class="n">i</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]]</span><span class="w">
    </span><span class="p">(</span><span class="nf">print-and-increment</span><span class="w"> </span><span class="n">i</span><span class="p">)))</span><span class="w">
</span><span class="c1">; prints nothing - for is lazy</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nb">last</span><span class="w"> </span><span class="n">b</span><span class="p">))</span><span class="w">
</span><span class="c1">; prints 1234 - same as 'map' above</span><span class="w">

</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">c</span><span class="w">
  </span><span class="p">(</span><span class="nb">doseq</span><span class="w"> </span><span class="p">[</span><span class="n">i</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]]</span><span class="w">
    </span><span class="p">(</span><span class="nf">print-and-increment</span><span class="w"> </span><span class="n">i</span><span class="p">)))</span><span class="w">
</span><span class="c1">; prints 123 - doseq is evaluated immediately</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="n">c</span><span class="p">)</span><span class="w">
</span><span class="c1">; prints nil - doseq returns nil</span><span class="w">

</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">d</span><span class="w">
  </span><span class="p">(</span><span class="nb">doall</span><span class="w"> </span><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="p">[</span><span class="n">i</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]]</span><span class="w">
    </span><span class="p">(</span><span class="nf">print-and-increment</span><span class="w"> </span><span class="n">i</span><span class="p">))))</span><span class="w">
</span><span class="c1">; prints 123 - doall forces evaluation of lazy sequence returned by for</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nb">last</span><span class="w"> </span><span class="n">d</span><span class="p">))</span><span class="w">
</span><span class="c1">; prints 4 - the value of (last d).</span><span class="w">
</span></code></pre></div></div>

<p><strong>Take-away</strong>: remember that <code class="language-plaintext highlighter-rouge">for</code> is lazy. Use <code class="language-plaintext highlighter-rouge">doseq</code> or <code class="language-plaintext highlighter-rouge">doall</code> to trigger strict evaluation.</p>

<p>Relevant documentation: <a href="http://clojuredocs.org/clojure.core/for">for</a>, <a href="http://clojuredocs.org/clojure.core/doseq">doseq</a>, <a href="http://clojuredocs.org/clojure.core/doall">doall</a></p>

<h2 id="unexpected-laziness-combined-with-multithreading">Unexpected laziness combined with multithreading</h2>

<p>This is an extension of the problem above: laziness when combined with multithreading. For example, imagine we want to sum a large list in parallel. We might write:</p>

<figure class="highlight"><pre><code class="language-clj" data-lang="clj"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="code"><pre><span class="c1">; Sum list, e.g. (sum [1 2 3]) = 1+2+3 = 6</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">sum</span><span class="w"> </span><span class="p">[</span><span class="n">lst</span><span class="p">]</span><span class="w">
  </span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="nb">+</span><span class="w"> </span><span class="n">lst</span><span class="p">))</span><span class="w">

</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">large-list</span><span class="w"> </span><span class="p">(</span><span class="nb">range</span><span class="w"> </span><span class="mi">10000</span><span class="p">))</span><span class="w">
</span><span class="c1">; Divide large-list into 4 partitions of 2500 items:</span><span class="w">
</span><span class="c1">; '((0 .. 2499) (2500 .. 4999) (5000 .. 7499) (7500 .. 9999))</span><span class="w">
</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">partitions</span><span class="w"> </span><span class="p">(</span><span class="nf">partition-all</span><span class="w"> </span><span class="mi">2500</span><span class="w"> </span><span class="n">large-list</span><span class="p">))</span><span class="w">
</span><span class="c1">; Create 4 futures, in each future sum the partition: (⚠️ this is incorrect)</span><span class="w">
</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">futures</span><span class="w"> </span><span class="p">(</span><span class="nb">map</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">partition</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">sum</span><span class="w"> </span><span class="n">partition</span><span class="p">)))</span><span class="w"> </span><span class="n">partitions</span><span class="p">))</span><span class="w">
</span><span class="c1">; Wait until 4 futures have completed and get their results:</span><span class="w">
</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">results</span><span class="w"> </span><span class="p">(</span><span class="nb">map</span><span class="w"> </span><span class="nb">deref</span><span class="w"> </span><span class="n">futures</span><span class="p">))</span><span class="w">
</span><span class="c1">; Sum 4 partial sums into one big sum:</span><span class="w">
</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="p">(</span><span class="nf">sum</span><span class="w"> </span><span class="n">results</span><span class="p">))</span>
</pre></td></tr></tbody></table></code></pre></figure>

<p>This code is incorrect! It will not actually execute the futures in parallel. The problem is again that <code class="language-plaintext highlighter-rouge">map</code> is lazy, and so on line 10, the futures are <em>not</em> actually created when that line is executed. The <code class="language-plaintext highlighter-rouge">map</code>s on both lines 10 and 12 are lazy. It is only on line 14, when reducing the final list, that each element of the the <code class="language-plaintext highlighter-rouge">results</code> (lazy) list is computed, needing the corresponding element in the <code class="language-plaintext highlighter-rouge">futures</code> list. Because <code class="language-plaintext highlighter-rouge">reduce</code> uses the elements in sequence, they are computed sequentially…</p>

<p>Here, the solution is to wrap the <code class="language-plaintext highlighter-rouge">map</code> on line 10 in <code class="language-plaintext highlighter-rouge">doall</code>:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">futures</span><span class="w"> </span><span class="p">(</span><span class="nb">doall</span><span class="w"> </span><span class="p">(</span><span class="nb">map</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">partition</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">sum</span><span class="w"> </span><span class="n">partiton</span><span class="p">)))</span><span class="w"> </span><span class="n">partitions</span><span class="p">)))</span><span class="w">
</span></code></pre></div></div>

<p>This forces the computation of the futures to start at that point.</p>

<p><strong>Take-away</strong>: when doing parallel computations on lists, check whether you aren’t using lazy operations.</p>

<h2 id="unexpected-behavior-of-pmap-with-lazy-sequences">Unexpected behavior of <code class="language-plaintext highlighter-rouge">pmap</code> with lazy sequences</h2>

<p><code class="language-plaintext highlighter-rouge">pmap</code> implements a parallel map. According to the documentation, pmap is “semi-lazy”: it evaluates lazily, but “tries to stay ahead of the consumption”. However, depending on whether its input is a lazy sequence or not, and whether it is “chunked”, its behavior will be different. Moreover, some types you might expect to be lazy are not (or only “partially”), in particular <code class="language-plaintext highlighter-rouge">range</code>. This is all quite complicated and confusing. Take a look at the following examples:</p>

<figure class="highlight"><pre><code class="language-clj" data-lang="clj"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
</pre></td><td class="code"><pre><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nb">range</span><span class="w"> </span><span class="mi">100</span><span class="p">))</span><span class="w">                </span><span class="c1">; =&gt; (0 1 .. 99)</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nb">take</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">(</span><span class="nb">iterate</span><span class="w"> </span><span class="nb">inc</span><span class="w"> </span><span class="mi">0</span><span class="p">)))</span><span class="w"> </span><span class="c1">; =&gt; (0 1 .. 99)</span><span class="w">
</span><span class="c1">; Both look the same</span><span class="w">

</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">one-second</span><span class="w"> </span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="w">
  </span><span class="p">(</span><span class="nf">Thread/sleep</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w">
  </span><span class="n">i</span><span class="p">)</span><span class="w">

</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">pmap-range</span><span class="w"> </span><span class="p">[]</span><span class="w">
  </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">x</span><span class="w"> </span><span class="p">(</span><span class="nf">pmap</span><span class="w"> </span><span class="n">one-second</span><span class="w"> </span><span class="p">(</span><span class="nb">range</span><span class="w"> </span><span class="mi">100</span><span class="p">))]</span><span class="w">
    </span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="nb">+</span><span class="w"> </span><span class="n">x</span><span class="p">)))</span><span class="w">

</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">pmap-lazy</span><span class="w"> </span><span class="p">[]</span><span class="w">
  </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">x</span><span class="w"> </span><span class="p">(</span><span class="nf">pmap</span><span class="w"> </span><span class="n">one-second</span><span class="w"> </span><span class="p">(</span><span class="nb">take</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">(</span><span class="nb">iterate</span><span class="w"> </span><span class="nb">inc</span><span class="w"> </span><span class="mi">0</span><span class="p">)))]</span><span class="w">
    </span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="nb">+</span><span class="w"> </span><span class="n">x</span><span class="p">)))</span><span class="w">

</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">pmap-forced-lazy</span><span class="w"> </span><span class="p">[]</span><span class="w">
  </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">x</span><span class="w"> </span><span class="p">(</span><span class="nf">pmap</span><span class="w"> </span><span class="n">one-second</span><span class="w"> </span><span class="p">(</span><span class="nb">doall</span><span class="w"> </span><span class="p">(</span><span class="nb">take</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">(</span><span class="nb">iterate</span><span class="w"> </span><span class="nb">inc</span><span class="w"> </span><span class="mi">0</span><span class="p">))))]</span><span class="w">
    </span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="nb">+</span><span class="w"> </span><span class="n">x</span><span class="p">)))</span><span class="w">

</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">pmap-vec-lazy</span><span class="w"> </span><span class="p">[]</span><span class="w">
  </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">x</span><span class="w"> </span><span class="p">(</span><span class="nf">pmap</span><span class="w"> </span><span class="n">one-second</span><span class="w"> </span><span class="p">(</span><span class="nf">vec</span><span class="w"> </span><span class="p">(</span><span class="nb">take</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">(</span><span class="nb">iterate</span><span class="w"> </span><span class="nb">inc</span><span class="w"> </span><span class="mi">0</span><span class="p">))))]</span><span class="w">
    </span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="nb">+</span><span class="w"> </span><span class="n">x</span><span class="p">)))</span><span class="w">

</span><span class="c1">; On a 4-core machine:</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nf">pmap-range</span><span class="p">))</span><span class="w">       </span><span class="c1">; Elapsed time: 4036.040718 msecs</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nf">pmap-lazy</span><span class="p">))</span><span class="w">        </span><span class="c1">; Elapsed time: 15108.031248 msecs ⚠️</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nf">pmap-forced-lazy</span><span class="p">))</span><span class="w"> </span><span class="c1">; Elapsed time: 15063.460062 msecs ⚠️</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nf">pmap-vec-lazy</span><span class="p">))</span><span class="w">    </span><span class="c1">; Elapsed time: 4017.902369 msecs</span><span class="w">

</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nf">type</span><span class="w"> </span><span class="p">(</span><span class="nb">range</span><span class="w"> </span><span class="mi">100</span><span class="p">)))</span><span class="w">                      </span><span class="c1">; =&gt; clojure.lang.LongRange</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nf">type</span><span class="w"> </span><span class="p">(</span><span class="nb">take</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">(</span><span class="nb">iterate</span><span class="w"> </span><span class="nb">inc</span><span class="w"> </span><span class="mi">0</span><span class="p">))))</span><span class="w">       </span><span class="c1">; =&gt; clojure.lang.LazySeq</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nf">type</span><span class="w"> </span><span class="p">(</span><span class="nf">vec</span><span class="w"> </span><span class="p">(</span><span class="nb">take</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">(</span><span class="nb">iterate</span><span class="w"> </span><span class="nb">inc</span><span class="w"> </span><span class="mi">0</span><span class="p">)))))</span><span class="w"> </span><span class="c1">; =&gt; clojure.lang.PersistentVector</span>
</pre></td></tr></tbody></table></code></pre></figure>

<p>Evaluating pmap on a <code class="language-plaintext highlighter-rouge">range</code> divides the range in 4 partitions that are evaluated in parallel (on this 4-core machine). However, when evaluating on the lazy sequence created with <code class="language-plaintext highlighter-rouge">iterate</code>, new elements are only created as needed, and therefore <code class="language-plaintext highlighter-rouge">pmap</code> is essentially sequential. Wrapping the lazy sequence in <code class="language-plaintext highlighter-rouge">doall</code> does not solve this issue! Converting it to a vector using <code class="language-plaintext highlighter-rouge">vec</code> first does.</p>

<p><strong>Take-away</strong>: <code class="language-plaintext highlighter-rouge">pmap</code> can have unexpected behavior on lazy sequences. Moreover, it is often difficult to predict what the underlying type is of a sequence in Clojure. It is usually better to manually implement your own parallel map that does exactly what you want it to do.</p>

<p>Relevant documentation: <a href="http://clojuredocs.org/clojure.core/pmap">pmap</a></p>

<h2 id="propagation-of-nil">Propagation of <code class="language-plaintext highlighter-rouge">nil</code></h2>

<p>In Clojure, many functions still work even when passed <code class="language-plaintext highlighter-rouge">nil</code>. For example, <code class="language-plaintext highlighter-rouge">(map inc nil)</code> returns <code class="language-plaintext highlighter-rouge">()</code>, just like <code class="language-plaintext highlighter-rouge">(map inc '())</code> would; and <code class="language-plaintext highlighter-rouge">(reduce + 0 nil)</code> returns <code class="language-plaintext highlighter-rouge">0</code>. However, this means that errors can often propagate quite far before an actual error is triggered.
Many proponents of Clojure consider this a feature. However, in my experience this often obscures errors. It can also make it hard to find the root cause of a problem, requiring you to trace back through many function calls to find out where the <code class="language-plaintext highlighter-rouge">nil</code> originated.</p>

<p><strong>Take-away</strong>: try to detect <code class="language-plaintext highlighter-rouge">nil</code> early.</p>

<h2 id="nil-and--sometimes-act-the-same-sometimes-differently"><code class="language-plaintext highlighter-rouge">nil</code> and <code class="language-plaintext highlighter-rouge">'()</code> sometimes act the same, sometimes differently</h2>

<p>There are some cases where <code class="language-plaintext highlighter-rouge">nil</code> and <code class="language-plaintext highlighter-rouge">()</code> act differently, as shown in the table below. In particular, <code class="language-plaintext highlighter-rouge">nil</code> is falsy while <code class="language-plaintext highlighter-rouge">()</code> is truthy. This is particularly annoying when combined with the problem above: when <code class="language-plaintext highlighter-rouge">nil</code> is used as input for <code class="language-plaintext highlighter-rouge">map</code>, an empty list is returned, and when using that empty list in next computations, things may not react as expected.</p>

<table>
  <thead>
    <tr>
      <th> </th>
      <th><code class="language-plaintext highlighter-rouge">x = nil</code></th>
      <th><code class="language-plaintext highlighter-rouge">x = '()</code></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(map inc x)</code></td>
      <td><code class="language-plaintext highlighter-rouge">()</code></td>
      <td><code class="language-plaintext highlighter-rouge">()</code></td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(reduce + 0 x)</code></td>
      <td><code class="language-plaintext highlighter-rouge">0</code></td>
      <td><code class="language-plaintext highlighter-rouge">0</code></td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(if x :true :false)</code></td>
      <td><code class="language-plaintext highlighter-rouge">:false</code> ⚠️</td>
      <td><code class="language-plaintext highlighter-rouge">:true</code> ⚠️</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(get x 2)</code></td>
      <td><code class="language-plaintext highlighter-rouge">nil</code></td>
      <td><code class="language-plaintext highlighter-rouge">nil</code></td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(nth x 2)</code></td>
      <td><code class="language-plaintext highlighter-rouge">nil</code></td>
      <td>error (index out of bounds)</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(contains? x 2)</code></td>
      <td><code class="language-plaintext highlighter-rouge">false</code></td>
      <td>error (illegal argument) ⚠️</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(.contains x 2)</code></td>
      <td>error (null pointer exception)</td>
      <td><code class="language-plaintext highlighter-rouge">false</code></td>
    </tr>
  </tbody>
</table>

<p><strong>Take-away</strong>: try to detect <code class="language-plaintext highlighter-rouge">nil</code> early, so you can return <code class="language-plaintext highlighter-rouge">'()</code> when that is expected.</p>

<h2 id="contains-vs-contains"><code class="language-plaintext highlighter-rouge">contains?</code> vs. <code class="language-plaintext highlighter-rouge">.contains</code></h2>

<p><code class="language-plaintext highlighter-rouge">(contains? collection key)</code> checks whether <code class="language-plaintext highlighter-rouge">key</code> is present in <code class="language-plaintext highlighter-rouge">collection</code>. For maps and sets, this works as expected, however, for vectors or lists the result is probably not what you expect:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">; Checking for KEY</span><span class="w">
</span><span class="p">(</span><span class="nb">contains?</span><span class="w"> </span><span class="p">{</span><span class="no">:a</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="mi">2</span><span class="p">}</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w"> </span><span class="c1">; = true  - on maps: as expected</span><span class="w">
</span><span class="p">(</span><span class="nb">contains?</span><span class="w"> </span><span class="p">{</span><span class="no">:a</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="mi">2</span><span class="p">}</span><span class="w"> </span><span class="no">:c</span><span class="p">)</span><span class="w"> </span><span class="c1">; = false</span><span class="w">

</span><span class="p">(</span><span class="nb">contains?</span><span class="w"> </span><span class="p">[</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">]</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w">  </span><span class="c1">; = false - on vector: unexpected ⚠️</span><span class="w">
</span><span class="p">(</span><span class="nb">contains?</span><span class="w"> </span><span class="p">[</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">]</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">   </span><span class="c1">; = true</span><span class="w">

</span><span class="p">(</span><span class="nb">contains?</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">)</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w"> </span><span class="c1">; error   - on list ⚠️</span><span class="w">
</span><span class="p">(</span><span class="nb">contains?</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">)</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">
</span><span class="c1">; Execution error (IllegalArgumentException).</span><span class="w">
</span><span class="c1">; contains? not supported on type: clojure.lang.PersistentList</span><span class="w">

</span><span class="p">(</span><span class="nb">contains?</span><span class="w"> </span><span class="o">#</span><span class="p">{</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">}</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w"> </span><span class="c1">; = true  - on set: as expected</span><span class="w">
</span><span class="c1">; #{:a :b :c} defines a set containing the elements :a, :b, and :c</span><span class="w">
</span></code></pre></div></div>

<p>What is happening here is that <code class="language-plaintext highlighter-rouge">contains?</code> checks for the presence of <em>a key, not a value</em>, and for vectors and lists this corresponds to the index, not the value. In other words, <code class="language-plaintext highlighter-rouge">contains?</code> does not search through the collection to check whether the value is present, but does a check whether the key exists (usually in O(1)). If <code class="language-plaintext highlighter-rouge">(contains? collection key)</code> returns <code class="language-plaintext highlighter-rouge">true</code>, that means <code class="language-plaintext highlighter-rouge">(get collection key)</code> will succeed.</p>

<p>If you want to check whether a vector or list contains a <em>value</em>, you can instead use the Java <code class="language-plaintext highlighter-rouge">.contains</code> method, which is defined on vectors, lists, and sets (but not maps, where you should first use <code class="language-plaintext highlighter-rouge">vals</code>!):</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">; Checking for VALUE</span><span class="w">
</span><span class="p">(</span><span class="nf">.contains</span><span class="w"> </span><span class="p">{</span><span class="no">:a</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="mi">2</span><span class="p">}</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w"> </span><span class="c1">; error   - on map ⚠️</span><span class="w">
</span><span class="c1">; Execution error (IllegalArgumentException).</span><span class="w">
</span><span class="c1">; No matching method contains found taking 1 args for class clojure.lang.PersistentArrayMap</span><span class="w">
</span><span class="p">(</span><span class="nf">.contains</span><span class="w"> </span><span class="p">(</span><span class="nb">vals</span><span class="w"> </span><span class="p">{</span><span class="no">:a</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="mi">2</span><span class="p">})</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w"> </span><span class="c1">; = false - using vals to check for value</span><span class="w">
</span><span class="p">(</span><span class="nf">.contains</span><span class="w"> </span><span class="p">(</span><span class="nb">vals</span><span class="w"> </span><span class="p">{</span><span class="no">:a</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="mi">2</span><span class="p">})</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">  </span><span class="c1">; = true</span><span class="w">

</span><span class="p">(</span><span class="nf">.contains</span><span class="w"> </span><span class="p">[</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">]</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w">  </span><span class="c1">; = true  - on vector: as expected</span><span class="w">
</span><span class="p">(</span><span class="nf">.contains</span><span class="w"> </span><span class="p">[</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">]</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">   </span><span class="c1">; = false</span><span class="w">

</span><span class="p">(</span><span class="nf">.contains</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">)</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w"> </span><span class="c1">; = true  - on list: as expected</span><span class="w">
</span><span class="p">(</span><span class="nf">.contains</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">)</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">  </span><span class="c1">; = false</span><span class="w">

</span><span class="p">(</span><span class="nf">.contains</span><span class="w"> </span><span class="o">#</span><span class="p">{</span><span class="no">:a</span><span class="w"> </span><span class="no">:b</span><span class="w"> </span><span class="no">:c</span><span class="p">}</span><span class="w"> </span><span class="no">:a</span><span class="p">)</span><span class="w"> </span><span class="c1">; = true  - on set: as expected</span><span class="w">
</span></code></pre></div></div>

<p><strong>Take-away</strong>: use the examples above to choose the appropriate function.</p>

<p>Relevant documentation: <a href="http://clojuredocs.org/clojure.core/contains_q">contains?</a></p>

<h2 id="firstrest-lastbutlast-peekpop-on-list-vs-vector"><code class="language-plaintext highlighter-rouge">first</code>/<code class="language-plaintext highlighter-rouge">rest</code>, <code class="language-plaintext highlighter-rouge">last</code>/<code class="language-plaintext highlighter-rouge">butlast</code>, <code class="language-plaintext highlighter-rouge">peek</code>/<code class="language-plaintext highlighter-rouge">pop</code> on list vs. vector</h2>

<p>Some functions on collections have different behavior on lists vs. vectors, as shown in the table below. There is a logical reason behind these, but this may not be immediately obvious. In particular, <code class="language-plaintext highlighter-rouge">peek</code> “efficiently” retrieves an element from a sequence, which means returning the first element for a linked list but the last element on a vector.</p>

<table>
  <thead>
    <tr>
      <th> </th>
      <th><code class="language-plaintext highlighter-rouge">[1 2 3]</code></th>
      <th><code class="language-plaintext highlighter-rouge">(list 1 2 3)</code></th>
      <th>Intuition</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(first ...)</code></td>
      <td><code class="language-plaintext highlighter-rouge">1</code></td>
      <td><code class="language-plaintext highlighter-rouge">1</code></td>
      <td>First element</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(next ...)</code></td>
      <td><code class="language-plaintext highlighter-rouge">(2 3)</code></td>
      <td><code class="language-plaintext highlighter-rouge">(2 3)</code></td>
      <td>Items after first, seq. <code class="language-plaintext highlighter-rouge">(next [1]) = nil</code></td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(rest ...)</code></td>
      <td><code class="language-plaintext highlighter-rouge">(2 3)</code></td>
      <td><code class="language-plaintext highlighter-rouge">(2 3)</code></td>
      <td>Items after first, seq. <code class="language-plaintext highlighter-rouge">(rest [1]) = ()</code></td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(last ...)</code></td>
      <td><code class="language-plaintext highlighter-rouge">3</code></td>
      <td><code class="language-plaintext highlighter-rouge">3</code></td>
      <td>Last element</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(butlast ...)</code></td>
      <td><code class="language-plaintext highlighter-rouge">(1 2)</code></td>
      <td><code class="language-plaintext highlighter-rouge">(1 2)</code></td>
      <td>Items before last, seq.</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(peek ...)</code></td>
      <td><code class="language-plaintext highlighter-rouge">3</code></td>
      <td><code class="language-plaintext highlighter-rouge">1</code></td>
      <td>⚠️ “Efficient” retrieval of element.</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">(pop ...)</code></td>
      <td><code class="language-plaintext highlighter-rouge">[1 2]</code></td>
      <td><code class="language-plaintext highlighter-rouge">(2 3)</code></td>
      <td>⚠️ Complement of peek.</td>
    </tr>
  </tbody>
</table>

<p>On vectors, <code class="language-plaintext highlighter-rouge">peek</code> is a lot more efficient than <code class="language-plaintext highlighter-rouge">last</code>, even though they return the same value.</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">large-vector</span><span class="w"> </span><span class="p">(</span><span class="nf">vec</span><span class="w"> </span><span class="p">(</span><span class="nb">range</span><span class="w"> </span><span class="mi">10000</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nb">peek</span><span class="w"> </span><span class="n">large-vector</span><span class="p">))</span><span class="w"> </span><span class="c1">; 9999 - Elapsed time: 0.046609 msecs</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nb">last</span><span class="w"> </span><span class="n">large-vector</span><span class="p">))</span><span class="w"> </span><span class="c1">; 9999 - Elapsed time: 12.9485 msecs</span><span class="w">
</span></code></pre></div></div>

<p>However, <code class="language-plaintext highlighter-rouge">peek</code> is not defined on some sequence types:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nb">peek</span><span class="w"> </span><span class="p">(</span><span class="nb">concat</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="p">]</span><span class="w"> </span><span class="p">[</span><span class="mi">3</span><span class="w"> </span><span class="mi">4</span><span class="p">]))</span><span class="w">
</span><span class="c1">; ⚠️ error: class clojure.lang.LazySeq cannot be cast to class clojure.lang.IPersistentStack</span><span class="w">
</span></code></pre></div></div>

<p><strong>Take-away</strong>: for some operations, you <em>need</em> to know whether you’re working on a list or a vector. Check the table above to see what each operation does. If necessary, convert the collection’s type:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">; List-&gt;vector</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nf">vec</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="nf">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">)))</span><span class="w">       </span><span class="c1">; =&gt; [1 2 3]</span><span class="w">
</span><span class="c1">; Vector-&gt;list - note the use of `apply`</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nb">apply</span><span class="w"> </span><span class="nb">list</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]))</span><span class="w"> </span><span class="c1">; =&gt; (1 2 3)</span><span class="w">
</span></code></pre></div></div>

<p>Relevant documentation: <a href="http://clojuredocs.org/clojure.core/first">first</a>, <a href="http://clojuredocs.org/clojure.core/rest">rest</a>, <a href="http://clojuredocs.org/clojure.core/last">last</a>, <a href="http://clojuredocs.org/clojure.core/butlast">butlast</a>, <a href="http://clojuredocs.org/clojure.core/peek">peek</a>, <a href="http://clojuredocs.org/clojure.core/pop">pop</a>, <a href="http://clojuredocs.org/clojure.core/vec">vec</a>, <a href="http://clojuredocs.org/clojure.core/list">list</a></p>

<h2 id="unexpected-conversion-to-other-sequence-type">Unexpected conversion to other sequence type</h2>

<p>Some operations on sequences convert their results to another sequence type, which can be unexpected, especially combined with the different behaviors shown above. For example, <code class="language-plaintext highlighter-rouge">next</code> and <code class="language-plaintext highlighter-rouge">rest</code> return a “chunked seq”, which does not have a <code class="language-plaintext highlighter-rouge">peek</code> function:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nb">peek</span><span class="w"> </span><span class="p">(</span><span class="nb">rest</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]))</span><span class="w">
</span><span class="c1">; ⚠️ error: class clojure.lang.PersistentVector$ChunkedSeq cannot be cast to class clojure.lang.IPersistentStack</span><span class="w">
</span><span class="p">(</span><span class="nf">type</span><span class="w"> </span><span class="p">(</span><span class="nb">rest</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="p">]))</span><span class="w"> </span><span class="c1">; = clojure.lang.PersistentVector$ChunkedSeq</span><span class="w">
</span></code></pre></div></div>

<p>In particular, <code class="language-plaintext highlighter-rouge">concat</code> is lazy and returns a <code class="language-plaintext highlighter-rouge">LazySeq</code> which is only realized when its result is needed. This can lead to unexpected bad performance when a vector is inadvertently converted to a lazy sequence:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">large-vector</span><span class="w"> </span><span class="p">(</span><span class="nf">vec</span><span class="w"> </span><span class="p">(</span><span class="nb">range</span><span class="w"> </span><span class="mi">10000</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nb">nth</span><span class="w"> </span><span class="n">large-vector</span><span class="w"> </span><span class="mi">5000</span><span class="p">))</span><span class="w">
</span><span class="c1">; "Elapsed time: 0.031142 msecs"</span><span class="w">
</span><span class="c1">; nth on vector is O(1)</span><span class="w">

</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">concatted</span><span class="w"> </span><span class="p">(</span><span class="nb">concat</span><span class="w"> </span><span class="n">large-vector</span><span class="w"> </span><span class="n">large-vector</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nb">nth</span><span class="w"> </span><span class="n">concatted</span><span class="w"> </span><span class="mi">5000</span><span class="p">))</span><span class="w">
</span><span class="c1">; ⚠️ "Elapsed time: 3.446098 msecs"</span><span class="w">
</span><span class="c1">; nth on lazy seq requires the sequence to be calculated before retrieving the element</span><span class="w">

</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">concatted-vector</span><span class="w"> </span><span class="p">(</span><span class="nf">vec</span><span class="w"> </span><span class="n">concatted</span><span class="p">))</span><span class="w">
</span><span class="c1">; converting the lazy seq to a vector takes several msec</span><span class="w">
</span><span class="p">(</span><span class="nb">time</span><span class="w"> </span><span class="p">(</span><span class="nb">nth</span><span class="w"> </span><span class="n">concatted-vector</span><span class="w"> </span><span class="mi">5000</span><span class="p">))</span><span class="w">
</span><span class="c1">; "Elapsed time: 0.023805 msecs"</span><span class="w">
</span><span class="c1">; but afterwards, nth is again O(1)</span><span class="w">
</span></code></pre></div></div>

<p>Worse, this can lead to a stack overflow when a lot of lists a first concatenated before they are used:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nf">type</span><span class="w"> </span><span class="p">(</span><span class="nb">concat</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="p">]</span><span class="w"> </span><span class="p">[</span><span class="mi">3</span><span class="w"> </span><span class="mi">4</span><span class="p">]))</span><span class="w"> </span><span class="c1">; = clojure.lang.LazySeq</span><span class="w">

</span><span class="p">(</span><span class="nb">repeat</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="nf">0</span><span class="w"> </span><span class="mi">1</span><span class="p">))</span><span class="w">
</span><span class="c1">; = ((0 1) (0 1) (0 1) (0 1) (0 1))</span><span class="w">
</span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="nb">concat</span><span class="w"> </span><span class="p">(</span><span class="nb">repeat</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="nf">0</span><span class="w"> </span><span class="mi">1</span><span class="p">)))</span><span class="w">
</span><span class="c1">; = (0 1 0 1 0 1 0 1 0 1)</span><span class="w">
</span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="nb">concat</span><span class="w"> </span><span class="p">(</span><span class="nb">repeat</span><span class="w"> </span><span class="mi">5000</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="nf">0</span><span class="w"> </span><span class="mi">1</span><span class="p">)))</span><span class="w">
</span><span class="c1">; ⚠️ error: StackOverflowError</span><span class="w">
</span><span class="c1">; This error occurs because the LazySeq is built up as a tree of its components;</span><span class="w">
</span><span class="c1">; here the tree is 5000 levels deep.</span><span class="w">
</span></code></pre></div></div>

<p><strong>Take-away</strong>: unfortunately, for some operations you need to know their return type. If you encounter bad performance in unexpected places, check the types of the involved data structures for laziness. Again, use <code class="language-plaintext highlighter-rouge">vec</code> or <code class="language-plaintext highlighter-rouge">list</code> to convert to the desired type.</p>

<p>Relevant documentation: <a href="http://clojuredocs.org/clojure.core/next">next</a>, <a href="http://clojuredocs.org/clojure.core/rest">rest</a>, <a href="http://clojuredocs.org/clojure.core/concat">concat</a>, <a href="http://clojuredocs.org/clojure.core/vec">vec</a>, <a href="http://clojuredocs.org/clojure.core/list">list</a></p>

<h2 id="60-second-wait-before-program-ends-shutdown-agents">60 second wait before program ends (<code class="language-plaintext highlighter-rouge">shutdown-agents</code>)</h2>

<p>When using futures (or features that use futures behind the scenes like agents or <code class="language-plaintext highlighter-rouge">pmap</code>), a program will keep running for 60 seconds at the end. You need to call <code class="language-plaintext highlighter-rouge">(shutdown-agents)</code> to force a shutdown of the thread pool immediately. For example:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">time </span>clj <span class="nt">--eval</span> <span class="s2">"(time 1)"</span>
<span class="s2">"Elapsed time: 0.014664 msecs"</span>
1

clj <span class="nt">--eval</span> <span class="s2">"(time 1)"</span>  1.81s user 0.16s system 185% cpu 1.062 total

<span class="nv">$ </span><span class="nb">time </span>clj <span class="nt">--eval</span> <span class="s2">"(time (future 1))"</span>
<span class="s2">"Elapsed time: 7.38667 msecs"</span>
<span class="c">#object[clojure.core$future_call$reify__8544 0x72bca894 {:status :ready, :val 1}]</span>

clj <span class="nt">--eval</span> <span class="s2">"(time (future 1))"</span>  1.96s user 0.20s system 3% cpu 1:01.10 total

<span class="nv">$ </span><span class="nb">time </span>clj <span class="nt">--eval</span> <span class="s2">"(time (do (future 1) (shutdown-agents)))"</span>
<span class="s2">"Elapsed time: 3.066247 msecs"</span>

clj <span class="nt">--eval</span> <span class="s2">"(time (do (future 1) (shutdown-agents)))"</span>  1.82s user 0.15s system 187% cpu 1.047 total
</code></pre></div></div>

<p>Here, we measure the total time to execute a Clojure program using <code class="language-plaintext highlighter-rouge">(time ...)</code> within our program, and using the shell command <code class="language-plaintext highlighter-rouge">time</code>. The relevant time is shown before <code class="language-plaintext highlighter-rouge">total</code>: the program that just returns <code class="language-plaintext highlighter-rouge">1</code> takes 1 second to execute, the one that creates a future takes one minute and one second. This is because the cached thread pool used for futures keeps running for one minute before shutting down automatically. By calling <code class="language-plaintext highlighter-rouge">(shutdown-agents)</code> at the end of our program, we avoid this one minute wait.</p>

<p>Relevant documentation: <a href="https://clojuredocs.org/clojure.core/shutdown-agents">shutdown-agents</a></p>

<h2 id="def-is-global-for-ex-schemers"><code class="language-plaintext highlighter-rouge">def</code> is global (for ex-Schemers)</h2>

<p>If you’ve programmed in Scheme before, you’re probably used to using <code class="language-plaintext highlighter-rouge">define</code> for local variables. However, that doesn’t work as expected in Clojure. For example:</p>

<div class="language-clj highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="p">[</span><span class="n">parameter</span><span class="p">]</span><span class="w">
  </span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">parameter</span><span class="p">)</span><span class="w">
  </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="n">x</span><span class="p">))</span><span class="w">

</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">one</span><span class="w"> </span><span class="p">(</span><span class="nf">f</span><span class="w"> </span><span class="mi">1</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nf">one</span><span class="p">))</span><span class="w"> </span><span class="c1">; =&gt; 1</span><span class="w">
</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">two</span><span class="w"> </span><span class="p">(</span><span class="nf">f</span><span class="w"> </span><span class="mi">2</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="p">(</span><span class="nf">one</span><span class="p">))</span><span class="w"> </span><span class="c1">; =&gt; 2</span><span class="w">
</span></code></pre></div></div>

<p>In this example, the second call to <code class="language-plaintext highlighter-rouge">f</code> overwrote the <em>global</em> variable <code class="language-plaintext highlighter-rouge">x</code>. In Clojure, <code class="language-plaintext highlighter-rouge">def</code> always defines a <em>global</em> variable, unlike in Scheme where <code class="language-plaintext highlighter-rouge">define</code> defines a local variable. In Clojure, you should always use <code class="language-plaintext highlighter-rouge">let</code> for local variables.</p>

<h2 id="general-conclusions">General conclusions</h2>

<p>Most of these problems have two root causes:</p>

<ol>
  <li>Clojure’s extreme <strong>dynamicity</strong>. This leads to ‘secret’ conversions between (seq) types, and faulty <code class="language-plaintext highlighter-rouge">nil</code> results being propagated because <code class="language-plaintext highlighter-rouge">nil</code> often passes as a seq. Note that this is not just the fact that Clojure is dynamically typed, but rather the automatic and careless conversions between types (e.g. Python avoids these issues).</li>
  <li>Clojure’s <strong>laziness</strong>. This often leads to things happening at unexpected times. It is also not always clear which sequence types are lazy. Haskell, another lazy language, partially avoids this by explicitly typing all IO operations with the <code class="language-plaintext highlighter-rouge">IO</code> monad, so that at least for those its often more obvious what happens.<sup id="fnref:HaskellLazyIO"><a href="#fn:HaskellLazyIO" class="footnote" rel="footnote" role="doc-noteref">1</a></sup> In my opinion, it is a mistake to mix laziness with a dynamically typed language (Haskell avoids that), and to mix laziness with multithreading (this applies to Haskell too). In a lazy language it is difficult to write multithreaded code, or even just single-threaded but efficient code.</li>
</ol>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:HaskellLazyIO">
      <p>In Haskell, <code class="language-plaintext highlighter-rouge">map print [1, 2, 3]</code> has type <code class="language-plaintext highlighter-rouge">[IO ()]</code>: it is cleary a list of IO actions that have not been executed yet. It is clear that a <code class="language-plaintext highlighter-rouge">do</code> is needed to combine those IO actions into one. <a href="#fnref:HaskellLazyIO" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Janwillem</name></author><category term="clojure" /><category term="programming" /><summary type="html"><![CDATA[After teaching the Multicore Programming course at Vrije Universiteit Brussel for quite a few years, I’ve compiled a list of some common gotchas students encounter when they first program in Clojure:]]></summary></entry><entry><title type="html">How to measure CO₂ using the MH-Z19B sensor and a Raspberry Pi</title><link href="http://jnwllm.be/blog/mh-z19" rel="alternate" type="text/html" title="How to measure CO₂ using the MH-Z19B sensor and a Raspberry Pi" /><published>2020-08-30T16:00:00+00:00</published><updated>2020-08-30T16:00:00+00:00</updated><id>http://jnwllm.be/blog/mh-z19</id><content type="html" xml:base="http://jnwllm.be/blog/mh-z19"><![CDATA[<p>In this post, we’ll delve into measuring CO₂ levels. We’ll use the MH-Z19B sensor, connected to a Raspberry Pi, and read out its measurements using Python.</p>

<h2 id="background-measuring-co-levels-in-house">Background: measuring CO₂ levels in house</h2>

<p>First off, some background information on indoor CO₂ levels. (Note: this is about CO₂ = carbon <strong>di</strong>oxide, not CO = carbon <strong>mon</strong>oxide.<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>)</p>

<p>CO₂ levels are measured in ppm or “parts per million”: out of a million molecules in the air, how many are CO₂ molecules? Currently, CO₂ levels in the atmosphere are <a href="https://en.wikipedia.org/wiki/Carbon_dioxide">about 420 ppm</a>, with slight seasonal variations. An interesting side note is that, before the Industrial Revolution, atmospheric CO₂ levels were <a href="https://en.wikipedia.org/wiki/Carbon_dioxide">around 280 ppm</a>, but we’re not here to talk about climate change.</p>

<p>Indoors, CO₂ levels can vary from 420 ppm (the atmospheric level is the minimum you can reach, when you have your windows open), to 1500–2000 ppm in closed rooms, and even higher in crowded rooms such as class rooms or meeting rooms. The largest contributor to this is the CO₂ humans and large pets exhale. (A large dog exhales about half the CO₂ a human exhales.)</p>

<p>Elevated CO₂ levels can lead to several health problems, such as headaches and drowsiness. Recommendations vary, with the United States goverment currently limiting exposure to 5000 ppm for an eight-hour period, but other studies showing possible negative effects starting at 1000 ppm. (A level of 40,000 ppm is considered “immediately dangerous to life and health”, but you’re unlikely to encounter that.) If two people sleep a whole night in a closed bedroom, it’s possible to get to <a href="https://www.reddit.com/r/dataisbeautiful/comments/cek4dm/co%E2%82%82_in_my_sons_bedroom_at_night/">levels of 2000 ppm</a>. Especially in modern, well-insulated homes and in schools and offices where many people gather, it is therefore important to have sufficient ventilation to keep CO₂ levels below these limits.</p>

<p>I got interested in measuring CO₂ at home to better configure my ventilation unit. This machine sucks in fresh air from outside and blows “used” air out (and at the same time has a heat exchanger that warms the incoming air using the outgoing air). Its air flow rate can be configured from 0 to 100%. Higher rates use more electricity and therefore cost more, but of course its rate should be high enough to keep CO₂ levels below safe limits.</p>

<h2 id="why-the-mh-z19b-sensor">Why the MH-Z19B sensor?</h2>

<p>You can find many CO₂ sensors online. There are roughly two kinds: those that measure VOC and those that actually detect CO₂ molecules. The first kind detect Volatile Organic Compounds (VOCs): organic compounds that we exhale together with CO₂ and can therefore be used as a proxy to estimate the CO₂ in the air. However, results of such sensors often have quite large errors, as shown for instance in <a href="https://www.youtube.com/watch?v=M6baDH4QNpc">this video</a>: a tumble dryer produces dust that contains VOCs, but does not produce CO₂. A VOC sensor will therefore incorrectly report high CO₂ levels when a tumble dryer is on. The usually cited advantage of VOC sensors is that they’re cheaper. The second kind of sensors measure CO₂ directly, but are often said to be more expensive. However, the MH-Z19B sensor is a cheap sensor of the second kind, so that’s why I chose to use that one.</p>

<p>The MH-Z19B sensor measures CO₂ using the <a href="https://en.wikipedia.org/wiki/Nondispersive_infrared_sensor">non-dispersive infrared (NDIR)</a> principle. It has an infrared lamp, of which the light is directed through two chambers. The first chamber is open to the air we want to measure; the second contains a reference gas, typically nitrogen. At the end is a detector. The gases in each chamber absorb certain wavelengths, and by comparing the results of the two chambers for the wavelengths absorbed by CO₂, we can calculate the CO₂ concentration in the air.</p>

<p>The MH-Z19B sensor has a reported accuracy of about 100ppm and a measuring range from 0 to 5000ppm. Its results should also be independent from the humidity. You can find the MH-Z19B sensor on AliExpress or European electronics websites (didn’t check others) for about $15 or €15.</p>

<h2 id="connecting-the-mh-z19b-sensor-to-a-raspberry-pi">Connecting the MH-Z19B sensor to a Raspberry Pi</h2>

<p>It is easy to connect the MH-Z19B sensor to a Raspberry Pi, as shown below. (Figure from <a href="https://github.com/UedaTakeyuki/mh-z19">here</a>.)</p>

<p><img src="/assets/images/mh-z19b-rpi-connection.jpg" alt="" style="max-width: 20rem;" /></p>

<p>This table shows which pins to connect: (for Pi models with 40 pins, that is all models since 1B+ up to at least 4B, <a href="https://www.raspberrypi.org/documentation/usage/gpio/">check here</a>)</p>

<table>
    <thead>
        <tr>
            <th>MH-Z19B</th>
            <th>Raspberry Pi GPIO</th>
            <th>Function</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>6 Vin</td>
            <td>4 (or 2) +5V</td>
            <td>5V current</td>
        </tr>
        <tr>
            <td>7 GND</td>
            <td>6 (or another ground pin) GND</td>
            <td>Ground</td>
        </tr>
        <tr>
            <td>2 Rx</td>
            <td>8 Tx (UART)</td>
            <td>Commands from Pi to sensor</td>
        </tr>
        <tr>
            <td>3 Tx</td>
            <td>10 Rx (UART)</td>
            <td>Data from sensor to Pi</td>
        </tr>
    </tbody>
</table>

<p>I found two versions of the MH-Z19B sensor online: one with just “holes” at the end and one with pins soldiered on already. I bought the ones with pins, so that I could use female-to-female <a href="https://en.wikipedia.org/wiki/Jump_wire">jump wires</a> to connect the sensor with the Pi.</p>

<h3 id="configuring-uart-on-pi">Configuring UART on Pi</h3>

<p>The Raspberry Pi communicates with the sensor over its UART (universal asynchronous receiver-transmitter) interface: a device for asynchronous serial communication. You need to enable this by following some tricky steps, as <a href="http://www.circuits.dk/setup-raspberry-pi-3-gpio-uart/">in this guide</a> or <a href="https://www.raspberrypi.org/documentation/configuration/uart.md">in the Raspberry Pi documentation</a>. I’ve copied the instructions for the Pi 3 below, assuming you’re using Raspbian.</p>

<h4 id="disable-linux-serial-console">Disable Linux serial console</h4>

<p>By default the serial port is assigned to the Linux console, allowing you to execute commands in a console over the serial port (using a service called getty). We want to disable as we want to use this port for the MH-Z19B sensor instead, do this as follows:</p>

<ol>
  <li>Start raspi-config on the command line: <code class="language-plaintext highlighter-rouge">sudo raspi-config</code></li>
  <li>Select “5 Interfacing Options”.</li>
  <li>Select “P6 Serial”.</li>
  <li>For the question “Would you like a login shell to be accessible over serial?” choose No.</li>
  <li>For the question “Would you like the serial port hardware to be enabled?” choose Yes.</li>
</ol>

<p>Raspbian also comes with a GUI to configure these options.</p>

<p>Furthermore, execute the command <code class="language-plaintext highlighter-rouge">sudo systemctl disable serial-getty@ttyS0.service</code> to disable the getty service from starting on boot.</p>

<h4 id="enable-uart-in-bootconfigtxt">Enable UART in /boot/config.txt</h4>

<p>At the bottom of <code class="language-plaintext highlighter-rouge">/boot/config.txt</code>, add the following line:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enable_uart=1
</code></pre></div></div>

<p>Reboot the Pi after these changes. The serial port is now available at <code class="language-plaintext highlighter-rouge">/dev/serial0</code> (RPi 3 as well as older versions).</p>

<h2 id="reading-co-levels-using-python">Reading CO₂ levels using Python</h2>

<p>Now that we’ve connected the sensor to the Pi, we can start reading its measurements. You can find the full code <a href="https://github.com/jswalens/sensors/blob/master/mh-z19.py">on my GitHub</a>. It’s a refactored and simplified of <a href="https://github.com/UedaTakeyuki/mh-z19">an existing library</a> for my use case. I’m using Python 3.</p>

<h3 id="reading-measurements">Reading measurements</h3>

<p>We are using the <a href="https://pythonhosted.org/pyserial/">pyserial</a> library to connect to the device. Its path is <code class="language-plaintext highlighter-rouge">/dev/serial0</code>. The following code will read the current measurements:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">serial</span>

<span class="n">DEVICE_PATH</span> <span class="o">=</span> <span class="sh">"</span><span class="s">/dev/serial0</span><span class="sh">"</span>
<span class="n">TIMEZONE</span> <span class="o">=</span> <span class="n">pytz</span><span class="p">.</span><span class="nf">timezone</span><span class="p">(</span><span class="sh">"</span><span class="s">Europe/Brussels</span><span class="sh">"</span><span class="p">)</span>

<span class="k">def</span> <span class="nf">current_time</span><span class="p">():</span>
    <span class="n">now</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">datetime</span><span class="p">.</span><span class="nf">now</span><span class="p">().</span><span class="nf">astimezone</span><span class="p">(</span><span class="n">TIMEZONE</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">now</span><span class="p">.</span><span class="nf">strftime</span><span class="p">(</span><span class="sh">"</span><span class="s">%Y-%m-%dT%H:%M:%S%z</span><span class="sh">"</span><span class="p">)</span>

<span class="k">def</span> <span class="nf">connect</span><span class="p">():</span>
    <span class="k">return</span> <span class="n">serial</span><span class="p">.</span><span class="nc">Serial</span><span class="p">(</span><span class="n">DEVICE_PATH</span><span class="p">,</span> <span class="n">baudrate</span><span class="o">=</span><span class="mi">9600</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mf">3.0</span><span class="p">)</span>

<span class="k">def</span> <span class="nf">read_all</span><span class="p">():</span>
    <span class="k">with</span> <span class="nf">connect</span><span class="p">()</span> <span class="k">as</span> <span class="n">ser</span><span class="p">:</span>
        <span class="n">ser</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="se">\xff\x01\x86\x00\x00\x00\x00\x00\x79</span><span class="sh">"</span><span class="p">)</span>
        <span class="n">r</span> <span class="o">=</span> <span class="n">ser</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="mi">9</span><span class="p">)</span>

        <span class="k">if</span> <span class="nf">len</span><span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="o">==</span> <span class="mi">9</span> <span class="ow">and</span> <span class="n">r</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="mh">0xff</span> <span class="ow">and</span> <span class="n">r</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="mh">0x86</span><span class="p">:</span>
            <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">time</span><span class="sh">"</span><span class="p">:</span> <span class="nf">current_time</span><span class="p">(),</span>
                    <span class="sh">"</span><span class="s">co2</span><span class="sh">"</span><span class="p">:</span> <span class="n">r</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="o">*</span><span class="mi">256</span> <span class="o">+</span> <span class="n">r</span><span class="p">[</span><span class="mi">3</span><span class="p">],</span>
                    <span class="sh">"</span><span class="s">temperature</span><span class="sh">"</span><span class="p">:</span> <span class="n">r</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span> <span class="o">-</span> <span class="mi">40</span><span class="p">,</span>
                    <span class="sh">"</span><span class="s">TT</span><span class="sh">"</span><span class="p">:</span> <span class="n">r</span><span class="p">[</span><span class="mi">4</span><span class="p">],</span> <span class="c1"># raw temperature
</span>                    <span class="sh">"</span><span class="s">SS</span><span class="sh">"</span><span class="p">:</span> <span class="n">r</span><span class="p">[</span><span class="mi">5</span><span class="p">],</span> <span class="c1"># status?
</span>                    <span class="sh">"</span><span class="s">Uh</span><span class="sh">"</span><span class="p">:</span> <span class="n">r</span><span class="p">[</span><span class="mi">6</span><span class="p">],</span> <span class="c1"># ticks in calibration cycle?
</span>                    <span class="sh">"</span><span class="s">Ul</span><span class="sh">"</span><span class="p">:</span> <span class="n">r</span><span class="p">[</span><span class="mi">7</span><span class="p">]}</span> <span class="c1"># number of performed calibrations?
</span>        <span class="k">else</span><span class="p">:</span>
            <span class="k">raise</span> <span class="nc">Exception</span><span class="p">(</span><span class="sh">"</span><span class="s">got unexpected answer %s</span><span class="sh">"</span> <span class="o">%</span> <span class="n">r</span><span class="p">)</span>
</code></pre></div></div>

<p>Here, the <code class="language-plaintext highlighter-rouge">connect</code> function connects to the device, returning an interface to the device. We then write nine bytes to the device: <code class="language-plaintext highlighter-rouge">FF 01 86 00 00 00 00 00 79</code>, which is the command to read the CO₂ concentration you’ll find <a href="https://github.com/jswalens/sensors/blob/master/docs/MH-Z19B.pdf">in the MH-Z19B documentation</a>. It’ll return nine bytes, which we’ll number 0 up to 8. What bytes 4 to 7 represent is not mentioned in the documentation, but people have tried to figure this out.</p>

<ul>
  <li>Bytes 0 and 1 are <code class="language-plaintext highlighter-rouge">FF 86</code>, which we check in the <code class="language-plaintext highlighter-rouge">if</code> condition to make sure we got the expected reply.</li>
  <li>The CO₂ concentration is byte 2*256 + byte 3. For instance, if <code class="language-plaintext highlighter-rouge">02 20</code> (in hexadecimal) is returned, this corresponds to 2 and 32 in decimal numbers, and so the concentration is 2*256 + 32 = 544 ppm.</li>
  <li>Byte 4 is probably the temperature in Celsius + 40. In my experience, this is often a few degrees above the actual room temperature.</li>
  <li>Byte 5 represents a status, and is always 0 in my case.</li>
  <li>Byte 6 counts ticks up to a new calibration. This counts from 0 to 142, incrementing every 10 minutes. Therefore, after 1440 minutes = 24 hours, it resets. Calibration of the sensor happens every 24 hours, at the moment this counter resests.</li>
  <li>Byte 7 counts the number of calibrations that have occurred, up to a maximum of 6.</li>
</ul>

<p>The function <code class="language-plaintext highlighter-rouge">read_all()</code> therefore returns, for example:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="nl">"time"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-08-23T15:36:24+0200"</span><span class="p">,</span><span class="w"> </span><span class="nl">"co2"</span><span class="p">:</span><span class="w"> </span><span class="mi">448</span><span class="p">,</span><span class="w"> </span><span class="nl">"temperature"</span><span class="p">:</span><span class="w"> </span><span class="mi">29</span><span class="p">,</span><span class="w"> </span><span class="nl">"TT"</span><span class="p">:</span><span class="w"> </span><span class="mi">69</span><span class="p">,</span><span class="w"> </span><span class="nl">"SS"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nl">"Uh"</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="p">,</span><span class="w"> </span><span class="nl">"Ul"</span><span class="p">:</span><span class="w"> </span><span class="mi">6</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p>There’s an easy and fun way to test whether the sensor works: pour out a can of Coca Cola, beer, or other carbonated beverage, and hold the sensor above the glass. It should immediately measure CO₂ levels of several 1000s ppm.</p>

<h3 id="calibration">Calibration</h3>

<p>The sensor must be calibrated. It’ll consider the “ambient environment” level to be 400 ppm, and calculates its levels relative to that. There are three ways to calibrate the sensor (see sensor documentation):</p>

<ol>
  <li>By connecting two pins when the ambient CO₂ level is 400 ppm.</li>
  <li>By sending the command <code class="language-plaintext highlighter-rouge">FF 01 87 00 00 00 00 00 78</code> when the ambient CO₂ level is 400 ppm.</li>
  <li>“Self-calibration”: every 24 hours, the sensor will calibrate by considering the lowest measured level to be 400 ppm. You can enable this with the command <code class="language-plaintext highlighter-rouge">FF 01 79 A0 00 00 00 00 E6</code>, although it should be enabled by default.</li>
</ol>

<p>I use the self-calibration because it is the easiest, as it does not require you to figure out when your sensor is in an environment with a CO₂ level of 400 or 440ppm. Note though that on its first use, you may have to have the sensor powered on for at least 24 hours before it’ll give accurate results. <a href="https://github.com/jswalens/sensors/blob/master/mh-z19.py">Using my script</a>, you can enable automatic baseline calibration with <code class="language-plaintext highlighter-rouge">python3 mh-z19.py --abc_on</code>.</p>

<h3 id="more">More</h3>

<p>You can find the full script shown in this post <a href="https://github.com/jswalens/sensors">in this GitHub project</a>. Run <code class="language-plaintext highlighter-rouge">sudo python3 mh-z19.py</code> to output the time and CO₂ level to the terminal. Run <code class="language-plaintext highlighter-rouge">sudo python3 mh-z19.py --all</code> to also output the undocumented parameters. You need to run the script as root using <code class="language-plaintext highlighter-rouge">sudo</code> to be able to use the serial connection.</p>

<p>There’s one more feature I haven’t mentioned: you can set the sensor’s detection range to vary from 0 to 2000, 5000, or 10000 ppm. According to the documentation, this setting should not influence the sensor’s accuracy.</p>

<h2 id="automatically-running-this-script-and-writing-results-to-file">Automatically running this script and writing results to file</h2>

<p>When <a href="https://github.com/jswalens/sensors/blob/master/mh-z19.py">my Python script</a> runs without any parameters, it performs the measurement and writes its output as JSON to the terminal. As a final step, let’s run this script continuously and write its results to a file. I created a shell script <a href="https://github.com/jswalens/sensors/blob/master/mh-z19.sh">mh-z19.sh</a>, containing:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">while </span><span class="nb">true</span><span class="p">;</span> <span class="k">do </span><span class="nb">sudo </span>python3 mh-z19.py <span class="nt">--all</span> | <span class="nb">tee</span> <span class="nt">-a</span> mh-z19.log<span class="p">;</span> <span class="nb">sleep </span>2m<span class="p">;</span> <span class="k">done</span>
</code></pre></div></div>

<p>This’ll run the script, and use <code class="language-plaintext highlighter-rouge">tee</code> to print its result to both the terminal and a log file. Every 2 minutes, a new measurement is added.</p>

<p>We’ve now succeeded in measuring the CO₂ in house and writing the results to a file. You can use this data to plot and examine the CO₂ levels over time, for instance using <a href="https://matplotlib.org/">Matplotlib</a> or Excel. You can also import the data into <a href="https://www.home-assistant.io/">Home Assistant</a> using the <a href="https://www.home-assistant.io/integrations/file">file integration</a>, but that’s for another time.</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1">
      <p>CO₂ = carbon <strong>di</strong>oxide is not the same as CO = carbon <strong>mon</strong>oxide. CO₂ is produced when humans exhale or during combustion, and although dangerous in high concentrations, normally you’ll notice and open a window before any permanent harm occurs. CO is produced from incomplete combustion and can lead to headache, dizziness, unconciousness, and death. However, these effects often occur without our body noticing, so CO is much more dangerous than CO₂. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Janwillem</name></author><category term="sensors" /><category term="python" /><category term="raspberry-pi" /><summary type="html"><![CDATA[In this post, we’ll delve into measuring CO₂ levels. We’ll use the MH-Z19B sensor, connected to a Raspberry Pi, and read out its measurements using Python.]]></summary></entry></feed>