Support running against Basilisp by chrisrink10 · Pull Request #840 · jank-lang/clojure-test-suite

chrisrink10 · 2025-12-31T23:02:09Z

This PR adds support for running the test suite against Basilisp:

It adds a new CI job which runs the suite on PRs and merges into the default branch.
Added a new Babashka task for running the tests: bb test-lpy.

By virtue of running on the Python VM, there are a few idiosyncrasies which repeatedly appear in tests:

Python does not have fixed-size integer values and, thus, any tests involving integer overflow or underflow needed to be excluded for Basilisp as they do not apply.
Python's bool type is a subclass of ints so there are many instances where true and false have to be specially handled.
Basilisp does not implement any of the sorted collection types, so all tests against those data types had to be excluded.
Basilisp does not currently support standard clojure.test style fixtures.
I'm actually not sure why this is happening, but there are many tests in the suite which assert on exact ordered results when calling seq on maps. These seem to work consistently for all other implementations but not in Basilisp. I'm guessing this may have to do with the fact that Clojure (and perhaps others) have array maps for smaller maps and perhaps array maps have a consistent iteration order?

⚠️ There is one open issue in Basilisp that I haven't addressed: apply is not lazy right now, so I had to suppress 2 test cases in the mapcat tests which produced an infinite loop. I filed #844 to track that in this repo as well.

CLAassistant · 2025-12-31T23:02:15Z

All committers have signed the CLA.

E-A-Griffin · 2025-12-31T23:38:19Z

Thanks so much for adding this! I'll take a look tomorrow :)

chrisrink10 · 2025-12-31T23:43:33Z

Thanks so much for adding this! I'll take a look tomorrow :)

Thanks for the quick reply! I don't actually have this running quite yet, so you can hold off on reviewing.

If me opening the PR while I'm working on it is not preferred, I can close it again. Just let me know!

jeaye · 2025-12-31T23:46:23Z

Thanks so much for adding this! I'll take a look tomorrow :)

Thanks for the quick reply! I don't actually have this running quite yet, so you can hold off on reviewing.

If me opening the PR while I'm working on it is not preferred, I can close it again. Just let me know!

Ah, a draft PR is ok. I requested Emma's help since I thought you were ready. Feel free to hack on this. Once you're ready, please just request a review from me and I'll loop others in.

E-A-Griffin · 2026-01-19T18:09:22Z

test/clojure/core_test/number_range.cljc

 (def ^:const max-double #?(:clj Double/MAX_VALUE
                           :cljr Double/MaxValue
                           :cljs js/Number.MAX_VALUE
+                           :lpy (.-max sys/float-info)


If this is system dependent we should add a comment explaining that here

@E-A-Griffin just to confirm: by "system dependent" you mean like "the computer I'm using may have different sized floats", yes?

E-A-Griffin · 2026-01-19T18:11:58Z

test/clojure/core_test/plus.cljc

-           (is (instance? clojure.lang.BigInt (+ 1 5N)))
-           (is (instance? clojure.lang.BigInt (+ 1N 5)))
-           (is (instance? clojure.lang.BigInt (+ 1N 5N)))]))
+           #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:12:13Z

test/clojure/core_test/pop_bang.cljc

-    (testing "cannot pop! after call to persistent!"
-      (let [t (transient [0 1]), _ (persistent! t)]
-        (is (thrown? #?(:cljs js/Error :cljr Exception :default Error) (pop! t)))))
+    #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:12:31Z

test/clojure/core_test/pos_int_qmark.cljc

      false :-1
      false 'a-sym

+      #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:12:40Z

test/clojure/core_test/pos_qmark.cljc

      true  1.0M
      false -1.0M

+      #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:13:54Z

test/clojure/core_test/prn_str.cljc

     ;; `prn-str` adds it to the end of the string.
     (is (= (str "\"a\" \"string\"" nl) (prn-str "a" "string")))
     (is (= #?(:cljs (str "nil \"a\" \"string\" \"A\" \" \" 1 17 [:a :b] {:c :d} #{:e}" nl)
+               :lpy (str "nil \"a\" \"string\" \"A\" \" \" 1 17.0 [:a :b] {:c :d} #{:e}" nl)


Could you add the same

;; Basilisp does not have character types, but will print floats ;; with trailing decimal place.

comment here?

E-A-Griffin · 2026-01-19T18:14:18Z

test/clojure/core_test/remove_watch.cljc

    #?(:cljs
       nil

+       :lpy


comment would be good here

E-A-Griffin · 2026-01-19T18:15:03Z

test/clojure/core_test/repeat.cljc


      (testing "n not being a number"
        (are [n x] #?(:cljs    (= [] (repeat n x))
+                      :lpy     (thrown? Exception (vec (repeat n x)))


Is this because repeat is lazy? Do you know why wrapping this in vec is required?

I actually might have fixed this in Basilisp to check the value before creating the lazy seq. Let me double check.

This is actually not necessary any longer, so I will revert this change.

E-A-Griffin · 2026-01-19T18:15:24Z

test/clojure/core_test/reversible_qmark.cljc

-      true (sorted-map :a 1)
-      true (sorted-set :a)
+
+      #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:15:31Z

test/clojure/core_test/reversible_qmark.cljc

-      #?(:cljs true :default false) (seq [1 2 3]))))
+      #?(:cljs true :default false) (seq [1 2 3])
+
+      #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:15:45Z

test/clojure/core_test/seq.cljc

     '() nil
     nil nil
-     (sorted-set 3.0 1.0 -2.5 4.0) '(-2.5 1.0 3.0 4.0)
+     #?@(:lpy [] :default [(sorted-set 3.0 1.0 -2.5 4.0) '(-2.5 1.0 3.0 4.0)])


comment would be good here

E-A-Griffin · 2026-01-19T18:15:51Z

test/clojure/core_test/seq.cljc

+     #?@(:lpy [] :default [(sorted-set 3.0 1.0 -2.5 4.0) '(-2.5 1.0 3.0 4.0)])
     (range 5 10) '(5 6 7 8 9)
     #?@(:cljs [(int-array 3) '(nil nil nil)]
+         :lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:15:58Z

test/clojure/core_test/seq.cljc

                      :c 800
                      nil 40}
-           input-sorted-map (into (sorted-map) input-map)
+           #?@(:lpy [] :default [input-sorted-map (into (sorted-map) input-map)])


comment would be good here

E-A-Griffin · 2026-01-19T18:16:03Z

test/clojure/core_test/seq.cljc

       (is (= input (into #{} (seq input))))
       (is (= input-hash (into (hash-set) (seq input))))
-       (is (= input-sorted-map (into (sorted-map) (seq input-sorted-map))))
+       #?(:lpy nil :default (is (= input-sorted-map (into (sorted-map) (seq input-sorted-map)))))


comment would be good here

E-A-Griffin · 2026-01-19T18:16:10Z

test/clojure/core_test/seq_qmark.cljc

      true (range)
      true (rseq [1 2 3])

+      #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:16:16Z

test/clojure/core_test/seq_qmark.cljc

-      false (object-array 3))))
+      false (object-array 3)
+
+      #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:16:27Z

test/clojure/core_test/seqable_qmark.cljc

      true "a string"
      true (object-array 3)

+      #?@(:lpy []


comment would be good here

E-A-Griffin

Ton of small suggestions, mostly for comments, thanks for your hard work on this!

E-A-Griffin · 2026-01-19T18:17:49Z

test/clojure/core_test/set_qmark.cljc

  (deftest test-set?
    (are [expected x] (= expected (set? x))
-      true (sorted-set :a)
+      #?@(:lpy [] :default [true (sorted-set :a)])


comment would be good here

E-A-Griffin · 2026-01-19T18:17:58Z

test/clojure/core_test/set_qmark.cljc

      false \a
-      false (object-array 3))))
+      false (object-array 3)
+      #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:18:12Z

test/clojure/core_test/short.cljc

    ;; test whether it's a fixed-length integer of some sort.
    (is (int? (short 0)))
    #?@(:cljs []
+        :lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:18:29Z

test/clojure/core_test/special_symbol_qmark.cljc

    (testing "special symbols"
      (are [arg] (special-symbol? 'arg)
-                 &
+                 #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:19:35Z

test/clojure/core_test/subs.cljc

         (is (= "ab" (subs "abcde" nil 2)))
         (is (= "a" (subs "abcde" 1 nil)))]
+        :lpy
+        [(is (= "" (subs "abcde" 2 1)))


What is the behavior here?

E-A-Griffin · 2026-01-19T18:20:33Z

test/clojure/core_test/vals.cljc

      (is (= '(0.0) (vals {0 0.0})))
      (is (= '(:b) (vals {:a :b})))
-      (is (= '(:b :d) (vals {:a :b :c :d})))
+      (is (contains? #{'(:b :d) '(:d :b)} (vals {:a :b :c :d})))


interesting

E-A-Griffin · 2026-01-19T18:20:41Z

test/clojure/core_test/vec.cljc

                           [1 2 3] '(1 2 3)
                           [1 2 3] [1 2 3]
-                           [1 2 3] (sorted-set 1 2 3)
+                           #?@(:lpy [] :default [[1 2 3] (sorted-set 1 2 3)])


comment would be good here

E-A-Griffin · 2026-01-19T18:20:53Z

test/clojure/core_test/vector_qmark.cljc

      false \a
-      false (object-array 3))))
+      false (object-array 3)
+      #?@(:lpy []


comment would be good here

E-A-Griffin · 2026-01-19T18:22:36Z

test/clojure/string_test/ends_with_qmark.cljc

+      [(is (true? (str/ends-with? 'ab "b")))
+		   (is (false? (str/ends-with? 'ab "a")))
+       (is (true? (str/ends-with? :ab "b")))
+		   (is (false? (str/ends-with? :ab "a")))])


indentation is weird on this file

Some tabs ended up in there somehow. Fixed.

E-A-Griffin · 2026-01-19T18:23:19Z

test/clojure/string_test/starts_with_qmark.cljc

+		   (is (thrown? Exception (str/starts-with? :ab ":a")))
+		   (is (thrown? Exception (str/starts-with? 'a/b ":a")))
+		   (is (thrown? Exception (str/starts-with? :a/b ":a")))]
+		  :default


indentation is weird on this file

jeaye · 2026-01-19T18:37:52Z

I can add a section on Basilisp-specific differences to the README as part of this PR, if you all like. I do also maintain a Differences from Clojure documentation which mirrors the same document for Clojurescript. I can link to that for more a more comprehensive list.

No need to tackle that now. When we do it, I'd like to get the Clojure core team involved, so we can make a coordinated effort across dialects. For now, keeping that "Differences from Clojure" document up to date with anything you've learned by making this PR would be appreciated.

jeaye · 2026-01-19T21:10:40Z

Excellent review, @E-A-Griffin. I've gone through the whole PR but had nothing more to add than what she identified. Once we're good with each of these comments being addressed, and you feel good about the PR Chris, I think we're golden!

Going forward, Chris, if you have anyone from the Basilisp community who would be interested in helping us write these tests, I'd appreciate the call to action. It's a lot of work and we need to kick up some more momentum for 2026.

chrisrink10 · 2026-01-21T02:07:00Z

@jeaye @E-A-Griffin I believe I have addressed all of the comments.

In general I tried to add comments where requested, but in cases where I could easily wrap a test case with (for example) (when-var-exists sorted-set ...) I did that instead since that won't require any manual intervention whenever Basilisp finally adds those Vars. However, I couldn't do that in every case since many tests are defined with are.

Let me know if you have any other comments. Thanks again for the review!

jeaye · 2026-01-22T19:08:10Z

chrisrink10 added 2 commits December 30, 2025 22:01

Support running against Basilisp

9af5c9c

Support Basilisp

396e2d8

jeaye requested a review from E-A-Griffin December 31, 2025 23:24

chrisrink10 added 8 commits December 31, 2025 21:14

Fixes

d072e5f

README and Babashka

6430ba6

More updates

96c00a5

Fix string tests

16fea58

Fix test

aa31d40

More tests

a58718c

Parse tests

f03e8b9

Fixes

70a68c3

chrisrink10 mentioned this pull request Jan 3, 2026

Run each implementation's tests in separate Github Actions steps #841

Merged

chrisrink10 added 14 commits January 3, 2026 13:54

Fixes

0170cd1

Atom

0ae4c32

More changes

5bdb3b7

More

98371b8

Use pip/setuptools

94a4b06

fix bb.edn

ecbbbfb

Fix

ddde469

More fixes

809f945

More fixes

14cd918

More changes

4fd316d

More

866ed35

More changes

bfcdeb1

Merge branch 'main' into feature/support-basilisp

0216fd6

Ignore

58d431d