What information is needed to reuse code?

[kaythaney]

(For more, see the full post on the blog: http://mozillascience.org/what-else-is-needed-for-code-reuse/)

When we first started discussions around our latest "Code as a research object" project, one of the main topics that arose was reuse. It's one thing for code and software to have an identifier that the community trusts so that it can be integrated into scholarly publishing systems. But what about the researchers looking to use that information to build or reuse that code in their own work? What information is needed for the code to be picked up, forked and run by someone else outside of their lab? What would the ideal README look like?

A few ideas:

• What does the code do? For what field? (Short descriptor)
• What's needed to get the code to run? Is it part of a larger codebase? Links to relevant repositories or tools used to run the code.
• Contributors.
• Link to the documentation on how the code is used.
• License*
• We recommend MIT or BSD, but for more options/clarity, visit: http://choosealicense.com/.

[ctb]

A few off the cuff thoughts:

at least minimal automated tests so we can be sure the software runs properly;
example data set with expected output;
a lot of the stuff that the Software Sustainability Institute puts up about sustainable projects;
how to cite the code;

@mr-c, what are we missing?

[gearmonkey]

a quick one, that I see missing (and sometimes keeps me from using) lots of code from research:

• Example use for core cases, ie. how do I run this code?

[JensTimmerman]

• ALL dependencies needed to compile the code.
• Versions of these dependencies that are KNOWN to work (including compiler)
• (unit) test cases
• examples and their expected output
• hash of the release, or git commit id, optionally signed by the release manager so the authenticity of the code can be checked.
• optionally benchmark figures for the examples on a certain platform to get an idea of how long a run of the code will take.

shameless plug:

• I've been involved in a framework to build (mainly) scientific software, we often run into trouble when building scientific software, so we figure out how to do it, and distribute the build procedure, in python code, so other people can benefit from our work. More information: http://hpcugent.github.io/easybuild/

[manics]

• How does it compare to other similar projects (if any)?
• Use cases
• Situations that it's not designed for

These help users make a decision, and benefits the authors since it reduces negative feedback from people expecting it to do something it's not designed for.

[Carreau]

I know if is more from the prospective of someone that want to publish code, but I'm often missing a good tool that generate all this boilerplate and set the distribution of software.

Beeing mostly a python dev, I'm still highly annoyed when I need to publish python code even after all this time, I'm much more impressed by language like julia that have a built-in mechanism to generate a package in minutes. (set up github unit-tests , performance test, register the name, set-up doc in less than a minute) an npm (javascript).

Then as a "user" of that code I shouldn't need to know how to install this package it should "just works". And this is true as code author, if I can't use language way of installing things on other computer in my lab in less than 10 minutes, I won't even try to make this work across computers.

This is (for me) more than 80% of what is needed to reuse code :

• A quick way to install test it unmodified on my data.
• A quick an simple way to publish it that don't asked me to read 10 pages on different website.

[kaythaney]

Thanks, @ctb! Only concern is that for researchers who don't indentify as "computational scientists" who may be doing something slightly more entry level, is that too onerous? Thinking the bare minimum here for someone to be able to get up and running with the code ...

[kaythaney]

Great stuff, all. Keep it coming. :)

[khinsen]

I think it's important to distinguish short-term goals (What can we do right now? What should we recommend as best practices?) from long-term goals (In which direction should we develop our infrastructures?) In between we have the category of "thin layers on top of our current toolstack that would make life easier". The comments I see are about short-term and mid-term tasks, so I'll tackle the long-term directions.

The number one long-term goal for me is a stable code representation for scientific software. This can't be source code: programming languages are a user interface, which we want ever better and adapted to our domain-specific needs. So we will always have multiple and evolving languages. Machine-level code can't be stable either, because hardware evolves as well. So a stable layer must be somewhere in the middle, at a level that no one cares about too much to reject it. JVM or CLR bytecode are at the right level, but not so well suited for scientific applications.

Why do we need a stable code representation? For two reasons:

1. Language interoperability. A scientific library should be reusable from a different language, to make investments into high-quality domain-specific code possible.
2. Long-term reusability. Sustainable software development is great, and should be a goal for big community codes, but it is not reasonable to require that every PhD student's explorative code should be maintained. Still, people should be able to read and run it, even years later.

[kaythaney]

Great point, @khinsen. For this, we're thinking more the 5 fields you fill in to go along with your code as you push a release to say, figshare, so that someone can meaningfully glean (without much pain) what the code does, how to run it, etc in minimal time. There's of course a longer term play here, but styling this in the ilk of some of the standards listed in the blog post, looking at a) whittling it down to the basics for this first instance (they're easier to implement, higher fill rates) and b) for those who do not identify as "computational researchers" necessarily. Stellar points.

[IanCal]

[edit - this is all with regards to the building, distributing and running part]

Docker would be an obvious candidate for this kind of thing: https://www.docker.io/

A public repository with a dockerfile allows people to build the software, and you can distribute built images easily and efficiently. It'd leave the explanation of how to just run the system as simple as

docker pull IanCal/experiment_1
docker run -t IanCal/experiment_1 run.sh

All dependencies would be contained in the image, which could run optimised BLAS libraries or python or whatever they want (as long as it works on linux).

Converting something that already runs into a docker image is as simple as providing a build script (which should read pretty much like a good README setup section), no changing language or anything like that.

If the original researcher uses this to manage their dependencies & build process, then you also know it will actually run. Reviewers could also easily run the software without fighting build systems.

[Carreau]

@IanCal for me this is reproductibility, not reuse, if you got 2 project having a docker image A, and B, how do you use A in docker image B ?

Agree that it might be helpfull, but it should be an extra to have something like that available. not the main point.

[IanCal]

@Carreau There's nothing stopping you from reusing the code within it. You'd publish your code, a script for building it, and a fully contained runnable system. This means you know all the dependencies and you know it'll build.

Many things may benefit from being structured such that they communicate over the network, in which case you can very simply link your docker container and theirs. This allows you to have isolated dependencies, so you're not stuck because you both require different, conflicting dependencies.

[mbjones]

Here's a diagram of a subset of EML metadata proposed back around 2000 for software for reuse on eco and enviro data:
http://knb.ecoinformatics.org/software/eml/eml-2.1.1/eml-software.png
Might be of use, could certainly be improved. I think @cboettig has this implemented for the R EML package from ROpenSci that is in progress.

[robldavidson]

It may help to consider what sort of endpoint could be achieved by truly re-usable code. To my mind, a good outcome would be to have a standardised system that could build complex software projects for you out of combinations of other software projects that were available in the unified, re-usable format.

There are a few examples of this type of thing that already exist:

Automatic installation systems for *nix systems e.g. MacPorts, HomeBrew, Yum, Apt-Get. There is a standardised 'make' format and a standardised meta-data format for storing your project in a repository that allows these systems to search for and install/build/utilise disparate and non-uniform projects.

Web documents and browsers. Each browser ought to be able to interpret the structure, layout and information present in billions of disparate projects. This is done via separating information from metadata and layout. Tags like 'document type definition' could be very useful for specifying programming language and version for non-web based projects.

The pipelining software, Galaxy (galaxyproject.org) or other variants e.g. Taverna. I've been using Galaxy to make my disparate conglomerate of R, Python and Matlab scripts and tools more accessible to the biologists in my lab. Each tool is 'wrapped' with an XML file that describes all possible inputs, outputs and command line instructions. By putting this uniform descriptor on each uniquely designed project, they are able to be lassooed together into one large complex project.

I honestly think that something like Galaxy is very close to where you want to go with this, but the next step would be to design a universal standard for this type of meta-data wrapper system so that a) it is optimised/advanced and b) anyone can design a Galaxy-style system for pulling projects together, without necessarily having to be a Galaxy expert first.

Finally, while that system is useful for merging commandline-callable standalone tools, i think it would utlimately be possible to have a system that built tools for you by wrapping libraries and packages in this way along with 'include and calling methods' rather than 'command line instructions'.

[cboettig]

@kaythaney I think we could use a bit more guidance here as to what problems this question seeks to address. I think you're asking what we should consider the minimal metadata fields for scientific software (e.g. title, version, maintainers, etc). Other things like code documentation, unit testing, functionalized code, good API, knowledge of the algorithms and their limitations, etc can all be necessary for reuse but aren't necessarily things we can treat as software metadata.

Regarding minimal metadata, I think it's instructive to consider what various software packaging systems have decided should be minimal metadata: e.g. R's DESCRIPTION files on CRAN, Perl's META.yml files on CPAN, Ruby's Gem::Specification, Debian's metadata for .deb packages, etc. (p.s. Python people, how do you guys handle this?)

While I don't think any of these provide an exhaustive list of what is needed to "re-use" the software, each has been developed and tested within it's own ecosystem to be a reasonably reliable source of information to (a) facilitate installation by handling dependencies, etc, and (b) provide enough information for users and developers to search for desired features (like "parse xml") within a repository, and (c) usually provide some indication of how to get documentation and/or support for the software. Notably too, none of these systems try to answer "can I re-use this software", or "can I trust the results" (questions best left to the user community to evaluate), and yet people build upon these systems all the time.

So if Debian etc have effectively answered what metadata they expect from package providers to help promote re-usable software on their platform, I suspect the question is what are the metadata elements (and format?) to promote reusable software in scientific research?

It seems to me that part of the answer is simply to meet whatever the platform / system specific standards are for software distribution (R software should use the package system with valid DESCRIPTION file, provided on CRAN, etc). Given that, we might then ask are there elements missing from any of these existing standards that we would expect should be part of the minimal metadata for a scientific package? (a DOI? Citation(s)? particular licenses? Bug tracker/mailing list/maintainer contact? Description? Keywords?) What fields should be optional, and what required?

p.p.s. A related but different question is whether or not there should be a standard metadata format for scientific software, such as one might collect into a central searchable repository akin to the language-specific ones. Personally, I think success in so doing is hard and we should instead build on the language-specific repositories, but I could be convinced otherwise.