Merge pull request #7 from JuliaAI/dev

Finish up traits and optional data interface

Author: ablaom

Project.toml

@@ -4,7 +4,6 @@ authors = ["Anthony D. Blaom <anthony.blaom@gmail.com>"]
 version = "0.1.0"
 
 [deps]
-InteractiveUtils = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
 Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 
 [extras]

README.md

@@ -1,16 +1,26 @@
 # LearnAPI.jl
 
-A Julia interface for training and applying machine learning models.
+A Julia interface for training and applying machine learning models. 
 
-**Status:** Proposal. 
 
+**Devlopement Status:**
 
-🚧
+- [X] Detailed proposal stage ([this
+      documentation](https://juliaai.github.io/LearnAPI.jl/dev/))
+- [ ] Initial feedback stage (opened mid-January, 2023)
+- [ ] Implement feedback and finish "To do" list (below)
+- [ ] Proof of concept implementation
+- [ ] Polish
+- [ ] Registration
 
+To do:
 
-[![Build Status](https://github.com/JuliaAI/LearnAPI.jl/workflows/CI/badge.svg)](https://github.com/JuliaAI/LearnAPI.jl/actions)  
-[![Coverage](https://codecov.io/gh/JuliaAI/LearnAPI.jl/branch/master/graph/badge.svg)](https://codecov.io/github/JuliaAI/LearnAPI.jl?branch=master) 
+- [ ] Add methods to create/save persistent representation of learned parameters
+- [ ] Add more repo tests
+- [ ] Add methods to test an implementation
+- [ ] Add user guide ("Common Implementation Patterns" section of manual)
+
+[![Build Status](https://github.com/JuliaAI/LearnAPI.jl/workflows/CI/badge.svg)](https://github.com/JuliaAI/LearnAPI.jl/actions)
+[![Coverage](https://codecov.io/gh/JuliaAI/LearnAPI.jl/branch/master/graph/badge.svg)](https://codecov.io/github/JuliaAI/LearnAPI.jl?branch=master)
 [![Docs](https://img.shields.io/badge/docs-dev-blue.svg)](https://juliaai.github.io/LearnAPI.jl/dev/)
 
-Please refer to the documentation for a detailed preview of what this package proposes to
-offer.

docs/Project.toml

@@ -1,6 +1,5 @@
 [deps]
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
-LearnAPI = "92ad9a40-7767-427a-9ee6-6e577f1266cb"
 ScientificTypesBase = "30f210dd-8aff-4c5f-94ba-8e64358c1161"
 Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
 

docs/make.jl

@@ -1,19 +1,23 @@
 using Documenter
 using LearnAPI
+using ScientificTypesBase
 
 const REPO="github.com/JuliaAI/LearnAPI.jl"
 
 makedocs(;
     modules=[LearnAPI,],
     format=Documenter.HTML(prettyurls = get(ENV, "CI", nothing) == "true"),
     pages=[
-        "Introduction" => "index.md",
+        "Overview" => "index.md",
         "Anatomy of an Implementation" => "anatomy_of_an_implementation.md",
         "Reference" => "reference.md",
         "Fit, update and ingest" => "fit_update_and_ingest.md",
         "Predict and other operations" => "operations.md",
+        "Accessor Functions" => "accessor_functions.md",
+        "Optional Data Interface" => "optional_data_interface.md",
         "Model Traits" => "model_traits.md",
         "Common Implementation Patterns" => "common_implementation_patterns.md",
+        "Testing an Implementation" => "testing_an_implementation.md",
     ],
     repo="https://$REPO/blob/{commit}{path}#L{line}",
     sitename="LearnAPI.jl"
@@ -24,4 +28,3 @@ deploydocs(
     devbranch="dev",
     push_preview=false,
 )
-

docs/src/accessor_functions.md

@@ -2,14 +2,15 @@
 
 > **Summary.** While byproducts of training are ordinarily recorded in the `report`
 > component of the output of `fit`/`update!`/`ingest!`, some families of models report an
-> itme that is likely shared by multiple model types, and it is useful to have common
+> item that is likely shared by multiple model types, and it is useful to have common
 > interface for accessing these directly. Training losses and feature importances are two
 > examples.
 
 ```@docs
 LearnAPI.feature_importances
-LearnAPI.training_labels
 LearnAPI.training_losses
 LearnAPI.training_scores
+LearnAPI.training_labels
 ```
 
+

docs/src/anatomy_of_an_implementation.md

@@ -6,8 +6,8 @@
 > `transform`). In this example we also implement an **accessor function**, called
 > `feature_importance`, returning the absolute values of the linear coefficients. The
 > ridge regressor has a target variable and `predict` makes literal predictions of the
-> target (rather than, say, probablistic predictions); this behaviour is flagged by the
-> `target_proxies` model trait.  Other traits articulate the model's training data type
+> target (rather than, say, probabilistic predictions); this behavior is flagged by the
+> `predict_proxy` model trait.  Other traits articulate the model's training data type
 > requirements and the input/output type of `predict`.
 
 We begin by describing an implementation of LearnAPI.jl for basic ridge
@@ -35,7 +35,7 @@ nothing # hide
 ```
 
 The subtyping `MyRidge <: LearnAPI.Model` is optional but recommended where it is not
-otherwise disruptive (it allows models to be displayed in a standard way, for example).
+otherwise disruptive.
 
 Instances of `MyRidge` are called **models** and `MyRidge` is a **model type**.
 
@@ -75,7 +75,7 @@ function LearnAPI.fit(model::MyRidge, verbosity, X, y)
         feature_importances =
                 [features[j] => abs(coefficients[j]) for j in eachindex(features)]
         sort!(feature_importances, by=last) |> reverse!
-        verbosity > 1 && @info "Features in order of importance: $(first.(feature_importances))"
+        verbosity > 0 && @info "Features in order of importance: $(first.(feature_importances))"
         report = (; feature_importances)
 
         return fitted_params, state, report
@@ -92,15 +92,15 @@ Regarding the return value of `fit`:
   or [`LearnAPI.ingest!`](@ref) method (see [Fit, update! and ingest!](@ref)).
 
 - The `report` is for other byproducts of training, apart from the learned parameters (the
-  ones will need to provide `predict` below).
+  ones we'll need to provide `predict` below).
 
-Our `fit` method assumes that `X` is a table (satifies the [Tables.jl
+Our `fit` method assumes that `X` is a table (satisfies the [Tables.jl
 spec](https://github.com/JuliaData/Tables.jl)) whose rows are the observations; and it
 will need need `y` to be an `AbstractFloat` vector. A model implementation is free to
 dictate the representation of data that `fit` accepts but articulates its requirements
 using appropriate traits; see [Training data types](@ref) below. We recommend against data
 type checks internal to `fit`; this would ordinarily be the responsibility of a higher
-level API, using those trasits. 
+level API, using those traits. 
 
 
 ## Operations
@@ -139,43 +139,39 @@ LearnAPI.feature_importances(::MyRidge, fitted_params, report) =
 nothing # hide
 ```
 
-Another example of an accessor function is [`training_losses`](@ref).
+Another example of an accessor function is [`LearnAPI.training_losses`](@ref).
 
 
 ## [Model traits](@id traits)
 
 Our model has a target variable, in the sense outlined in [Scope and undefined
 notions](@ref scope), and `predict` returns an object with exactly the same form as the
-target. We indicate this behaviour by declaring
+target. We indicate this behavior by declaring
 
 ```@example anatomy
-LearnAPI.target_proxies(::Type{<:MyRidge}) = (; predict=LearnAPI.TrueTarget())
+LearnAPI.predict_proxy(::Type{<:MyRidge}) = LearnAPI.TrueTarget()
 nothing # hide
 ```
 Or, you can use the shorthand
 
 ```@example anatomy
-@trait MyRidge target_proxies = (; predict=LearnAPI.TrueTarget())
+@trait MyRidge predict_proxy=LearnAPI.TrueTarget()
 nothing # hide
 ```
 
 More generally, `predict` only returns a *proxy* for the target, such as probability
 distributions, and we would make a different declaration here. See [Target proxies](@ref)
 for details.
 
-`LearnAPI.target_proxies` is an example of a **model trait**. A complete list of traits
+`LearnAPI.predict_proxy` is an example of a **model trait**. A complete list of traits
 and the contracts they imply is given in [Model Traits](@ref).
 
-> **MLJ only.** The values of all traits constitute a model's **metadata**, which is
-> recorded in the searchable MLJ Model Registry, assuming the implementation-providing
-> package is registered there.
-
 We also need to indicate that a target variable appears in training (this is a supervised
 model). We do this by declaring *where* in the list of training data arguments (in this
 case `(X, y)`) the target variable (in this case `y`) appears:
 
 ```@example anatomy
-@trait MyRidge position_of_target = 2
+@trait MyRidge position_of_target=2
 nothing # hide
 ```
 
@@ -184,7 +180,7 @@ As explained in the introduction, LearnAPI.jl does not attempt to define strict
 descriptors, as in
 
 ```@example anatomy
-@trait MyRidge descriptors = (:regression,)
+@trait MyRidge descriptors=(:regression,)
 nothing # hide
 ```
 
@@ -195,7 +191,7 @@ Finally, we are required to declare what methods (excluding traits) we have expl
 overloaded for our type:
 
 ```@example anatomy
-@trait MyRidge methods = (
+@trait MyRidge methods=(
         :fit,
         :predict,
         :feature_importances,
@@ -206,26 +202,25 @@ nothing # hide
 ## Training data types
 
 Since LearnAPI.jl is a basement level API, one is discouraged from including explicit type
-checks in an implementation of `fit`. Instead one uses traits to make promisises about the
+checks in an implementation of `fit`. Instead one uses traits to make promises about the
 acceptable type of `data` consumed by `fit`. In general, this can be a promise regarding
-the ordinary type of `data` and/or the [scientific
-type](https://github.com/JuliaAI/ScientificTypes.jl) of `data`. Alternatively, one may
-only make a promise about the type/scitype of *observations* in the data . See [Model
-Traits](@ref) for further details. In this case we'll be happy to restrict the scitype of
-the data:
+the ordinary type of `data` or the [scientific
+type](https://github.com/JuliaAI/ScientificTypes.jl) of `data` (but not
+both). Alternatively, one may only make a promise about the type/scitype of *observations*
+in the data . See [Model Traits](@ref) for further details. In this case we'll be happy to
+restrict the scitype of the data:
 
 ```@example anatomy
 import ScientificTypesBase: scitype, Table, Continuous
-@trait MyRidge fit_data_scitype = Tuple{Table(Continuous), AbstractVector{Continuous}}
+@trait MyRidge fit_scitype = Tuple{Table(Continuous), AbstractVector{Continuous}}
 nothing # hide
 ```
 
 This is a contract that `data` is acceptable in the call `fit(model, verbosity, data...)`
 whenever
 
-```@example anatomy
+```julia
 scitype(data) <: Tuple{Table(Continuous), AbstractVector{Continuous}}
-nothing # hide
 ```
 
 Or, in other words:
@@ -239,33 +234,23 @@ Or, in other words:
   AbstractVector{Continuous}` - meaning that it is an abstract vector with `<:AbstractFloat`
   elements.
 
-## Input/output types for operations
+## Input types for operations
 
-An optional promise that an operation, such as `predict`, returns an object of given
-scientific type is articulated in this way:
+An optional promise about what `data` is guaranteed to work in a call like
+`predict(model, fitted_params, data...)` is articulated this way:
 
 ```@example anatomy
-@trait output_scitypes = (; predict=AbstractVector{<:Continuous})
-nothing # hide
+@trait MyRidge predict_input_scitype = Tuple{AbstractVector{<:Continuous}}
 ```
 
-If `predict` had instead returned probability distributions that implement the
-`Distributions.pdf` interface, then one could instead make the declaration
+Note that `data` is always a `Tuple`, even if it has only one component (the typical
+case), which explains the `Tuple` on the right-hand side.
 
-```julia
-@trait MyRidge output_scitypes = (; predict=AbstractVector{Density{<:Continuous}})
-```
-
-Similarly, there exists a trait called [`output_type`](@ref) for making promises on the
-ordinary type resturned by an operation.
-
-Finally, we'll make a promise about what `data` is acceptable in a call like
-`predict(model, fitted_params, data...)`. Note that `data` is always a `Tuple`, even if it
-has only one component (the typical case).
+Optionally, we may express our promise using regular types, using the
+[`LearnAPI.predict_input_type`](@ref) trait.
 
-```example anatomy
-@trait MyRidge input_scitype = (; predict=Tuple{AbstractVector{<:Continuous}})
-```
+One can optionally make promises about the outut of an operation. See [Model Traits](@ref)
+for details.
 
 ## [Illustrative fit/predict workflow](@id workflow)
 
@@ -283,21 +268,21 @@ X = (; a, b, c) |> Tables.rowtable
 y = 2a - b + 3c + 0.05*rand(n)
 nothing # hide
 ```
-Instantiate a model with relevant hyperparameters:
+Instantiate a model with relevant hyperparameters (which is all the object stores):
 
 ```@example anatomy
 model = MyRidge(lambda=0.5)
 ```
 
-Train the model:
+Train the model (the `0` means do so silently):
 
 ```@example anatomy
 import LearnAPI: fit, predict, feature_importances
 
-fitted_params, state, fit_report = fit(model, 1, X[train], y[train])
+fitted_params, state, fit_report = fit(model, 0, X[train], y[train])
 ```
 
-Inspect the learned paramters and report:
+Inspect the learned parameters and report:
 
 ```@example anatomy
 @info "training outcomes" fitted_params fit_report

docs/src/common_implementation_patterns.md

@@ -3,10 +3,10 @@
 !!! warning
 
 	This section is only an implementation guide. The definitive specification of the
-	Learn API is given in [Reference](@ref).
+	Learn API is given in [Reference](@ref reference).
 
-This guide is intended to be consulted after reading [Anatomy of a Model
-Implementation](@ref), which introduces the main interface objects and terminology.
+This guide is intended to be consulted after reading [Anatomy of an Implementation](@ref),
+which introduces the main interface objects and terminology.
 
 Although an implementation is defined purely by the methods and traits it implements, most
 implementations fall into one (or more) of the following informally understood patterns or
@@ -21,7 +21,7 @@ implementations fall into one (or more) of the following informally understood p
 - [Incremental Models](@ref)
 
 - [Static Transformers](@ref): Transformations that do not learn but which have
-  hyper-parameters and/or deliver ancilliary information about the transformation
+  hyper-parameters and/or deliver ancillary information about the transformation
 
 - [Dimension Reduction](@ref): Transformers that learn to reduce feature space dimension
 

docs/src/fit_update_and_ingest.md

@@ -1,19 +1,21 @@
 # Fit, update! and ingest!
 
-> **Summary.** Models that learn, i.e., generalize to new data, must overload `fit`;
-> the fallback performs no operation and returns all `nothing`. Implement `update!` if
-> certain hyper-parameter changes do not necessitate retraining from scratch (e.g.,
-> increasing iteration parameters). Implement `ingest!` to implement incremental learning.
+> **Summary.** Models that learn, i.e., generalize to new data, must overload `fit`; the
+> fallback performs no operation and returns all `nothing`. Implement `update!` if certain
+> hyper-parameter changes do not necessitate retraining from scratch (e.g., increasing an
+> iteration parameter). Implement `ingest!` to implement incremental learning. All
+> training methods implemented must be named in the return value of the
+> `functions` trait.
 
 | method                     | fallback                                           | compulsory? | requires          |
 |:---------------------------|:---------------------------------------------------|-------------|-------------------|
-[`LearnAPI.fit`](@ref)       | does nothing, returns `(nothing, nothing, nothing)`| no          |                   |
-[`LearnAPI.update!`](@ref)   | calls `fit`                                        | no          | `LearnAPI.fit`    |
-[`LearnAPI.ingest!`](@ref)   | none                                               | no          | `LearnAPI.fit`    |
+| [`LearnAPI.fit`](@ref)     | does nothing, returns `(nothing, nothing, nothing)`| no          |                   |
+| [`LearnAPI.update!`](@ref) | calls `fit`                                        | no          | [`LearnAPI.fit`](@ref) |
+| [`LearnAPI.ingest!`](@ref) | none                                               | no          | [`LearnAPI.fit`](@ref) |
 
 All three methods above return a triple `(fitted_params, state, report)` whose components
 are explained under [`LearnAPI.fit`](@ref) below.  Items that might be returned in
-`report` include: feature rankings/importances, SVM support vectors, clustering centres,
+`report` include: feature rankings/importances, SVM support vectors, clustering centers,
 methods for visualizing training outcomes, methods for saving learned parameters in a
 custom format, degrees of freedom, deviances. Precisely what `report` includes might be
 controlled by model hyperparameters, especially if there is a performance cost to it's
@@ -26,10 +28,12 @@ as a basic DBSCAN clustering algorithm.
 
 The `update!` method is intended for all subsequent calls to train a model *using the same
 observations*, but with possibly altered hyperparameters (`model` argument). A fallback
-implementation simply calls `fit`. The main use cases for implementing `update` are: (i)
-warm-restarting iterative models, and (ii) "smart" training of composite models, such as
-linear pipelines. Here "smart" means that hyperparameter changes only trigger the
-retraining of downstream components.
+implementation simply calls `fit`. The main use cases for implementing `update` are: 
+
+- warm-restarting iterative models
+
+- "smart" training of composite models, such as linear pipelines; here "smart" means that
+  hyperparameter changes only trigger the retraining of downstream components.
 
 The `ingest!` method supports incremental learning (same hyperparameters, but new training
 observations). Like `update!`, it depends on the output a preceding `fit` or `ingest!`