Differences from ClojureCLR

As much as possible, MAGIC is meant to be a drop in replacement for ClojureCLR. That means that code written for ClojureCLR is generally expected to work on MAGIC without modification. However, there are places where magic deviates from ClojureCLR for reasons that include performance and bootstrapping.

Dynamic vars control the runtime

There are a few places where the Clojure runtime needs to invoke a Clojure complier. These include:

• Macro expansion
• Evaluating expressions
• Loading files and namespaces

In the standard ClojureCLR implementation, These code paths are hard-coded to invoke the ClojureCLR complier. This makes using alternate compilers or bootstrapping difficult if not impossible. In place of this, MAGIC and its patched Clojure.Runtime use a collection of dynamic vars to control complier-centric functionality. The vars are:

• clojure.core/*load-paths* — a vector of strings indicating paths on the file system to search as namespace roots. This takes the place of the Java class path in ClojureJVM and the CLOJURE_LOAD_PATH environment variable used by ClojureCLR
• clojure.core/*load-fn* — a function invoked to load a relative path. Bound by default to a Clojure port of clojure.lang.RT.load from the original ClojureCLR runtime and exhibits the same semantics, except uses these dynamic vars. In MAGIC the default binding is fine.
• clojure.core/*eval-form-fn* — a function called to evaluate an s-expression. In MAGIC this should be bound to magic.api/eval.
• clojure.core/*compile-file-fn* — a function called to load a file, evaluating its contents, and compile it to disk. In MAGIC this should be bound to magic.api/runtime-compile-file.
• clojure.core/*load-file-fn* — a function called to load a file into memory, evaluating its contents. In MAGIC this should be bound to magic.api/runtime-load-file.
• clojure.core/*macroexpand-1-fn* — a function called to expand a macro. In MAGIC this should be bound to magic.api/runtime-macroexpand-1.

Different initialization sequence

ClojureCLR initializes itself using a static constructor, which causes a number of problems. MAGIC's Clojure.Runtime instead exposes a RT.Initialize method on clojure.lang.RT to initialize in a manner that is easier to reason about. The intended initialization sequence is as follows:

1. Load all .clj.dll files packaged alongside Clojure.dll. These include the pre-compiled MAGIC compiler and its dependencies. Loading them in does not evaluate any code, but only makes their namespaces available in memory.
2. Begin the initialization of the Clojure runtime by invoking RT.Initialize(doRuntimePostBoostrap: false).
3. Load the clojure.core and magic.api namespaces from memory.
4. Bind the runtime dynamic vars to MAGIC's functions
5. Finish initialization with RT.PostBootstrapInit()

As an example, taken from Nostrand:

// (1) load compiled clojure assemblies
var assemblyPath = Path.GetDirectoryName(Assembly.Load("Clojure").Location);
foreach(var cljDll in Directory.EnumerateFiles(assemblyPath, "*.clj.dll"))
    Assembly.LoadFile(cljDll);

// (2) begin initialization
RT.Initialize(doRuntimePostBoostrap: false);

// (3) load clojure.core and magic.api namespaces from their InitTypes loaded in (1)
RT.TryLoadInitType("clojure/core");
RT.TryLoadInitType("magic/api");

// (4) bind runtime dynamic vars
RT.var("clojure.core", "*load-fn*").bindRoot(RT.var("clojure.core", "-load"));
RT.var("clojure.core", "*eval-form-fn*").bindRoot(RT.var("magic.api", "eval"));
RT.var("clojure.core", "*load-file-fn*").bindRoot(RT.var("magic.api", "runtime-load-file"));
RT.var("clojure.core", "*compile-file-fn*").bindRoot(RT.var("magic.api", "runtime-compile-file"));
RT.var("clojure.core", "*macroexpand-1-fn*").bindRoot(RT.var("magic.api", "runtime-macroexpand-1"));

// (5) finish initialization
RT.PostBootstrapInit();

Type hints are annotations

This is the difference most likely to require modifications to existing code. Standard ClojureCLR treats type hints as hints. In some cases they may allow the compiler to avoid a dynamic call site, but in general they do not affect the compiled code in a significant way.

MAGIC treats type hints as proper type annotations, which are more similar to how type annotations are handled in C#. They affect the storage locations of locals and the parameter types of functions. They may even affect dispatch to overloaded C# methods. MAGIC's optimizations depend on this type information, but the side effect is that some code that would compile under ClojureCLR will fail to compile in MAGIC. For example, the following would compile with a warning in ClojureCLR, but MAGIC will refuse to compile it because the String type is known not to have a DoesNotExist property at compile time.

(defn some-function [^String x]
  (.DoesNotExist x))

It can also result in situations where type hints actually need to be removed, especially in cases of functions that are polymorphic. This occurred while porting the standard library (e.g. 59cb6dd, 4480947). As an example, take the name function in clojure.core. It is defined to work on strings or instances of types implementing the clojure.lang.Named interface. The original code includes the hint in the parameter and does a dynamic type check in the body.

;; original code, from ClojureCLR
(defn name [^clojure.lang.Named x]
  (if (string? x) 
    x 
    (. x (getName))))

This will compile in MAGIC, but will fail at runtime if you pass in a String because it will attempt to cast the String to clojure.lang.Named. This function does not in fact take an instance of clojure.lang.Named. It takes an instance of an unknown type, and performs a type check at runtime. To support this in MAGIC, we move the type hint out of the parameter and into the function body.

;; patched code, in MAGIC
(defn name [x]
  (if (string? x) 
    x 
    (. ^clojure.lang.Named x (getName))))

This is a significant difference, but in practice it does not break a lot of code and fixing it is fairly straightforward. Because it is a central part of our optimization strategy this is not a difference we expect to ever reconcile.

proxy is static

Our implementation of the proxy expression is more static than ClojureCLR's. In ClojureCLR proxy is backed by an updatable hash map of standard Clojure functions, so that you could change out the implementation of any proxy method at any time.

We avoided implementing this feature because we could not find any evidence of its use in practice, and the static approach we have adopted is over 100x faster. The result is that the following proxy related functions are not supported and will throw an exception of called: proxy-mappings, update-proxy, init-proxy, get-proxy-class, construct-proxy.

This is a feature that could be reintroduced in the future if it were determined to be useful.

Ambiguous method names must be qualified in method declarations

When implementing methods in a proxy, deftype, defrecord, or reify expression, there are situations where the programmer will have to explicitly disambiguate a method override implementation. In ClojureCLR this is done with a hint. In MAGIC, because we use type hint syntax for type annotations, this is done by qualifying the name. For example, the following expression will not compile in either MAGIC or ClojureCLR because assoc could be meant to overload either clojure.lang.IPersistentMap::assoc or clojure.lang.Associative::assoc (IPersistentMap extends Associative).

(reify
  clojure.lang.IPersistentMap
  (assoc [this k v] this)) ;; <== assoc here is ambiguous

The fix in ClojureCLR is a type hint, but the fix in MAGIC is a name qualification.

;; MAGIC fix
(reify
  clojure.lang.IPersistentMap
  (clojure.lang.IPersistentMap.assoc [this k v] this))

;; ClojureCLR fix
(reify
  clojure.lang.IPersistentMap
  (^clojure.lang.IPersistentMap assoc [this k v] this))

gvec not supported

Clojure supports a feature called gvec, a generic homogeneous vector that can hold one of a closed set of types. We decided not to spend time implementing this feature as we could not find evidence of its use in practice, and it is not a good fit for the CLR platform in any case (the CLR has real generics, there's no reason to constrain the types of a generic vector to a hard coded set of types).

This is a difference that could be reconciled in the future if it were determined to be useful.

Primitive type hints are supported

ClojureCLR has restrictions on the types that can be used as function parameter types. MAGIC does not have these restrictions, so it is possible to write code that compiles in magic but fails to compile in ClojureCLR:

;; works in MAGIC, fails in ClojureCLR
;; with System.ArgumentException:
;;   Only long and double primitives are supported
(defn some-function [^float x]
  (+ x 1))