After teaching the Multicore Programming course at Vrije Universiteit Brussel for quite a few years, I’ve compiled a list of some common gotchas students encounter when they first program in Clojure:

Note that although some of these are mentioned pretty clearly in the documentation, that doesn’t make them less of a gotcha for newcomers.

Unexpected laziness of for

Clojure’s laziness often results in unexpected code for beginners. For instance, what does the following code do?

(defn second [lst]
  (println "Elements:")
  (for [element lst]
    (println "*" element))
  (nth lst 1))

(println (second [1 2 3]))

We define a function second, which takes a list and returns the second element (position 1). While debugging, we use for to loop through the list and print its elements. If you’ve never programmed in Clojure before, you probably think this program prints the following:

Elements:
* 1
* 2
* 3
2

That’s incorrect! This program actually prints:

Elements:
2

This code snippet is confusing Clojure’s for with a for loop as you can find it in a traditional language. In Clojure, for returns a list with the results of its body, and is evaluated lazily. You can consider the for loop above syntactic sugar for (map (fn [element] (prinln "*" element)) lst), which is also evaluated lazily. To get the desired behavior, you should use doseq:

(defn second [lst]
  (println "Elements:")
  (doseq [element lst]
    (println "*" element))
  (nth lst 1))

(println (second [1 2 3]))

This code snippet will print the expected output. doseq is similar to for, but is evaluated strictly (= not lazily).

However, another important difference between doseq and for is that doseq returns nil. If you want to force evaluation of for, map, or another construct that generates a lazy sequence, you can surround it with doall.

Some examples:

(defn print-and-increment [i]
  (print i)
  (+ i 1))

(def a (map print-and-increment [1 2 3]))
; prints nothing - map is lazy
(println (last a))
; prints 1234 ⚠️
; the first 123 is the 'delayed' lazy code from inside the function,
; 4 is the value of (last a) (= 3 + 1).

(def b
  (for [i [1 2 3]]
    (print-and-increment i)))
; prints nothing - for is lazy
(println (last b))
; prints 1234 - same as 'map' above

(def c
  (doseq [i [1 2 3]]
    (print-and-increment i)))
; prints 123 - doseq is evaluated immediately
(println c)
; prints nil - doseq returns nil

(def d
  (doall (for [i [1 2 3]]
    (print-and-increment i))))
; prints 123 - doall forces evaluation of lazy sequence returned by for
(println (last d))
; prints 4 - the value of (last d).

Take-away: remember that for is lazy. Use doseq or doall to trigger strict evaluation.

Relevant documentation: for, doseq, doall

Unexpected laziness combined with multithreading

This is an extension of the problem above: laziness when combined with multithreading. For example, imagine we want to sum a large list in parallel. We might write:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

; Sum list, e.g. (sum [1 2 3]) = 1+2+3 = 6
(defn sum [lst]
  (reduce + lst))

(def large-list (range 10000))
; Divide large-list into 4 partitions of 2500 items:
; '((0 .. 2499) (2500 .. 4999) (5000 .. 7499) (7500 .. 9999))
(def partitions (partition-all 2500 large-list))
; Create 4 futures, in each future sum the partition: (⚠️ this is incorrect)
(def futures (map (fn [partition] (future (sum partition))) partitions))
; Wait until 4 futures have completed and get their results:
(def results (map deref futures))
; Sum 4 partial sums into one big sum:
(def result (sum results))

This code is incorrect! It will not actually execute the futures in parallel. The problem is again that map is lazy, and so on line 10, the futures are not actually created when that line is executed. The maps on both lines 10 and 12 are lazy. It is only on line 14, when reducing the final list, that each element of the the results (lazy) list is computed, needing the corresponding element in the futures list. Because reduce uses the elements in sequence, they are computed sequentially…

Here, the solution is to wrap the map on line 10 in doall:

(def futures (doall (map (fn [partition] (future (sum partiton))) partitions)))

This forces the computation of the futures to start at that point.

Take-away: when doing parallel computations on lists, check whether you aren’t using lazy operations.

Unexpected behavior of pmap with lazy sequences

pmap implements a parallel map. According to the documentation, pmap is “semi-lazy”: it evaluates lazily, but “tries to stay ahead of the consumption”. However, depending on whether its input is a lazy sequence or not, and whether it is “chunked”, its behavior will be different. Moreover, some types you might expect to be lazy are not (or only “partially”), in particular range. This is all quite complicated and confusing. Take a look at the following examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

(println (range 100))                ; => (0 1 .. 99)
(println (take 100 (iterate inc 0))) ; => (0 1 .. 99)
; Both look the same

(defn one-second [i]
  (Thread/sleep 1000)
  i)

(defn pmap-range []
  (let [x (pmap one-second (range 100))]
    (reduce + x)))

(defn pmap-lazy []
  (let [x (pmap one-second (take 100 (iterate inc 0)))]
    (reduce + x)))

(defn pmap-forced-lazy []
  (let [x (pmap one-second (doall (take 100 (iterate inc 0))))]
    (reduce + x)))

(defn pmap-vec-lazy []
  (let [x (pmap one-second (vec (take 100 (iterate inc 0))))]
    (reduce + x)))

; On a 4-core machine:
(time (pmap-range))       ; Elapsed time: 4036.040718 msecs
(time (pmap-lazy))        ; Elapsed time: 15108.031248 msecs ⚠️
(time (pmap-forced-lazy)) ; Elapsed time: 15063.460062 msecs ⚠️
(time (pmap-vec-lazy))    ; Elapsed time: 4017.902369 msecs

(println (type (range 100)))                      ; => clojure.lang.LongRange
(println (type (take 100 (iterate inc 0))))       ; => clojure.lang.LazySeq
(println (type (vec (take 100 (iterate inc 0))))) ; => clojure.lang.PersistentVector

Evaluating pmap on a range divides the range in 4 partitions that are evaluated in parallel (on this 4-core machine). However, when evaluating on the lazy sequence created with iterate, new elements are only created as needed, and therefore pmap is essentially sequential. Wrapping the lazy sequence in doall does not solve this issue! Converting it to a vector using vec first does.

Take-away: pmap can have unexpected behavior on lazy sequences. Moreover, it is often difficult to predict what the underlying type is of a sequence in Clojure. It is usually better to manually implement your own parallel map that does exactly what you want it to do.

Relevant documentation: pmap

Propagation of nil

In Clojure, many functions still work even when passed nil. For example, (map inc nil) returns (), just like (map inc '()) would; and (reduce + 0 nil) returns 0. However, this means that errors can often propagate quite far before an actual error is triggered. Many proponents of Clojure consider this a feature. However, in my experience this often obscures errors. It can also make it hard to find the root cause of a problem, requiring you to trace back through many function calls to find out where the nil originated.

Take-away: try to detect nil early.

nil and '() sometimes act the same, sometimes differently

There are some cases where nil and () act differently, as shown in the table below. In particular, nil is falsy while () is truthy. This is particularly annoying when combined with the problem above: when nil is used as input for map, an empty list is returned, and when using that empty list in next computations, things may not react as expected.

  x = nil x = '()
(map inc x) () ()
(reduce + 0 x) 0 0
(if x :true :false) :false ⚠️ :true ⚠️
(get x 2) nil nil
(nth x 2) nil error (index out of bounds)
(contains? x 2) false error (illegal argument) ⚠️
(.contains x 2) error (null pointer exception) false

Take-away: try to detect nil early, so you can return '() when that is expected.

contains? vs. .contains

(contains? collection key) checks whether key is present in collection. For maps and sets, this works as expected, however, for vectors or lists the result is probably not what you expect:

; Checking for KEY
(contains? {:a 1 :b 2} :a) ; = true  - on maps: as expected
(contains? {:a 1 :b 2} :c) ; = false

(contains? [:a :b :c] :a)  ; = false - on vector: unexpected ⚠️
(contains? [:a :b :c] 1)   ; = true

(contains? '(:a :b :c) :a) ; error   - on list ⚠️
(contains? '(:a :b :c) 1)
; Execution error (IllegalArgumentException).
; contains? not supported on type: clojure.lang.PersistentList

(contains? #{:a :b :c} :a) ; = true  - on set: as expected
; #{:a :b :c} defines a set containing the elements :a, :b, and :c

What is happening here is that contains? checks for the presence of a key, not a value, and for vectors and lists this corresponds to the index, not the value. In other words, contains? does not search through the collection to check whether the value is present, but does a check whether the key exists (usually in O(1)). If (contains? collection key) returns true, that means (get collection key) will succeed.

If you want to check whether a vector or list contains a value, you can instead use the Java .contains method, which is defined on vectors, lists, and sets (but not maps, where you should first use vals!):

; Checking for VALUE
(.contains {:a 1 :b 2} :a) ; error   - on map ⚠️
; Execution error (IllegalArgumentException).
; No matching method contains found taking 1 args for class clojure.lang.PersistentArrayMap
(.contains (vals {:a 1 :b 2}) :a) ; = false - using vals to check for value
(.contains (vals {:a 1 :b 2}) 1)  ; = true

(.contains [:a :b :c] :a)  ; = true  - on vector: as expected
(.contains [:a :b :c] 1)   ; = false

(.contains '(:a :b :c) :a) ; = true  - on list: as expected
(.contains '(:a :b :c) 1)  ; = false

(.contains #{:a :b :c} :a) ; = true  - on set: as expected

Take-away: use the examples above to choose the appropriate function.

Relevant documentation: contains?

first/rest, last/butlast, peek/pop on list vs. vector

Some functions on collections have different behavior on lists vs. vectors, as shown in the table below. There is a logical reason behind these, but this may not be immediately obvious. In particular, peek “efficiently” retrieves an element from a sequence, which means returning the first element for a linked list but the last element on a vector.

  [1 2 3] (list 1 2 3) Intuition
(first ...) 1 1 First element
(next ...) (2 3) (2 3) Items after first, seq. (next [1]) = nil
(rest ...) (2 3) (2 3) Items after first, seq. (rest [1]) = ()
(last ...) 3 3 Last element
(butlast ...) (1 2) (1 2) Items before last, seq.
(peek ...) 3 1 ⚠️ “Efficient” retrieval of element.
(pop ...) [1 2] (2 3) ⚠️ Complement of peek.

On vectors, peek is a lot more efficient than last, even though they return the same value.

(def large-vector (vec (range 10000)))
(time (peek large-vector)) ; 9999 - Elapsed time: 0.046609 msecs
(time (last large-vector)) ; 9999 - Elapsed time: 12.9485 msecs

However, peek is not defined on some sequence types:

(peek (concat [1 2] [3 4]))
; ⚠️ error: class clojure.lang.LazySeq cannot be cast to class clojure.lang.IPersistentStack

Take-away: for some operations, you need to know whether you’re working on a list or a vector. Check the table above to see what each operation does. If necessary, convert the collection’s type:

; List->vector
(println (vec '(1 2 3)))       ; => [1 2 3]
; Vector->list - note the use of `apply`
(println (apply list [1 2 3])) ; => (1 2 3)

Relevant documentation: first, rest, last, butlast, peek, pop, vec, list

Unexpected conversion to other sequence type

Some operations on sequences convert their results to another sequence type, which can be unexpected, especially combined with the different behaviors shown above. For example, next and rest return a “chunked seq”, which does not have a peek function:

(peek (rest [1 2 3]))
; ⚠️ error: class clojure.lang.PersistentVector$ChunkedSeq cannot be cast to class clojure.lang.IPersistentStack
(type (rest [1 2 3])) ; = clojure.lang.PersistentVector$ChunkedSeq

In particular, concat is lazy and returns a LazySeq which is only realized when its result is needed. This can lead to unexpected bad performance when a vector is inadvertently converted to a lazy sequence:

(def large-vector (vec (range 10000)))
(time (nth large-vector 5000))
; "Elapsed time: 0.031142 msecs"
; nth on vector is O(1)

(def concatted (concat large-vector large-vector))
(time (nth concatted 5000))
; ⚠️ "Elapsed time: 3.446098 msecs"
; nth on lazy seq requires the sequence to be calculated before retrieving the element

(def concatted-vector (vec concatted))
; converting the lazy seq to a vector takes several msec
(time (nth concatted-vector 5000))
; "Elapsed time: 0.023805 msecs"
; but afterwards, nth is again O(1)

Worse, this can lead to a stack overflow when a lot of lists a first concatenated before they are used:

(type (concat [1 2] [3 4])) ; = clojure.lang.LazySeq

(repeat 5 '(0 1))
; = ((0 1) (0 1) (0 1) (0 1) (0 1))
(reduce concat (repeat 5 '(0 1)))
; = (0 1 0 1 0 1 0 1 0 1)
(reduce concat (repeat 5000 '(0 1)))
; ⚠️ error: StackOverflowError
; This error occurs because the LazySeq is built up as a tree of its components;
; here the tree is 5000 levels deep.

Take-away: unfortunately, for some operations you need to know their return type. If you encounter bad performance in unexpected places, check the types of the involved data structures for laziness. Again, use vec or list to convert to the desired type.

Relevant documentation: next, rest, concat, vec, list

60 second wait before program ends (shutdown-agents)

When using futures (or features that use futures behind the scenes like agents or pmap), a program will keep running for 60 seconds at the end. You need to call (shutdown-agents) to force a shutdown of the thread pool immediately. For example:

$ time clj --eval "(time 1)"
"Elapsed time: 0.014664 msecs"
1

clj --eval "(time 1)"  1.81s user 0.16s system 185% cpu 1.062 total

$ time clj --eval "(time (future 1))"
"Elapsed time: 7.38667 msecs"
#object[clojure.core$future_call$reify__8544 0x72bca894 {:status :ready, :val 1}]

clj --eval "(time (future 1))"  1.96s user 0.20s system 3% cpu 1:01.10 total

$ time clj --eval "(time (do (future 1) (shutdown-agents)))"
"Elapsed time: 3.066247 msecs"

clj --eval "(time (do (future 1) (shutdown-agents)))"  1.82s user 0.15s system 187% cpu 1.047 total

Here, we measure the total time to execute a Clojure program using (time ...) within our program, and using the shell command time. The relevant time is shown before total: the program that just returns 1 takes 1 second to execute, the one that creates a future takes one minute and one second. This is because the cached thread pool used for futures keeps running for one minute before shutting down automatically. By calling (shutdown-agents) at the end of our program, we avoid this one minute wait.

Relevant documentation: shutdown-agents

def is global (for ex-Schemers)

If you’ve programmed in Scheme before, you’re probably used to using define for local variables. However, that doesn’t work as expected in Clojure. For example:

(defn f [parameter]
  (def x parameter)
  (fn [] x))

(def one (f 1))
(println (one)) ; => 1
(def two (f 2))
(println (one)) ; => 2

In this example, the second call to f overwrote the global variable x. In Clojure, def always defines a global variable, unlike in Scheme where define defines a local variable. In Clojure, you should always use let for local variables.

General conclusions

Most of these problems have two root causes:

  1. Clojure’s extreme dynamicity. This leads to ‘secret’ conversions between (seq) types, and faulty nil results being propagated because nil often passes as a seq. Note that this is not just the fact that Clojure is dynamically typed, but rather the automatic and careless conversions between types (e.g. Python avoids these issues).
  2. Clojure’s laziness. This often leads to things happening at unexpected times. It is also not always clear which sequence types are lazy. Haskell, another lazy language, partially avoids this by explicitly typing all IO operations with the IO monad, so that at least for those its often more obvious what happens.1 In my opinion, it is a mistake to mix laziness with a dynamically typed language (Haskell avoids that), and to mix laziness with multithreading (this applies to Haskell too). In a lazy language it is difficult to write multithreaded code, or even just single-threaded but efficient code.

  1. In Haskell, map print [1, 2, 3] has type [IO ()]: it is cleary a list of IO actions that have not been executed yet. It is clear that a do is needed to combine those IO actions into one. ↩