Abstract data sources

.gitignore

@@ -38,7 +38,8 @@ QtSerialPort/build*
 build-*
 .idea
 CereStimGUI/matlab/cerestim_dbs/
-neuroport_dbs.egg-info
+*.egg-info
 *.whl
 
 /site
+build/

docs/README.md

@@ -1,6 +1,6 @@
 # OpenMER
 
-A collection of software developed in the Sachs Lab at the Ottawa Hospital for for deep brain stimulation (DBS) surgery intraoperative mapping with microelectrode recording (MER).
+OpenMER is a collection of software developed in the Sachs Lab at the Ottawa Hospital for deep brain stimulation (DBS) surgery intraoperative mapping with microelectrode recording (MER).
 
 ![Image of vis apps](https://github.com/SachsLab/OpenMER/blob/master/vis_apps_screenshot.PNG?raw=true)
 

docs/for-developers.md

@@ -1,8 +1,10 @@
 ## Repository Organization
 
-* Library and Application code in the /neuroport_dbs folder
-* Unit tests in the /tests folder. Note this uses the pytest framework and conventions.
-* Documentation in the /docs folder
+* `/open_mer`: Library and Application code
+  * `/open_mer/scripts`: Entry points 
+* `/tests` TODO: Unit tests
+* `/docs`: Documentation
+* `/scripts`: Scratch scripts
 
 ## Maintaining the Documentation
 
@@ -12,15 +14,17 @@ You will need to install several Python packages to maintain the documentation.
 
 The /docs/{top-level-section} folders contain a mix of .md and .ipynb documentation. The latter are converted to .md by the [mknotebooks plugin](https://github.com/greenape/mknotebooks/projects) during building.
 
-Run `mkdocs gh-deploy` to build the documentation, commit to the `gh-deploy` branch, and push to GitHub. This will make the documentation available at https://SachsLab.github.io/NeuroportDBS/
+Run `mkdocs gh-deploy` to build the documentation, commit to the `gh-deploy` branch, and push to GitHub. This will make the documentation available at https://SachsLab.github.io/OpenMER/
 
 ### Autogenerated Documentation
 
-The /docs/neuroport_dbs folder can hold stubs to tell the [mkdocstrings plugin](https://github.com/mkdocstrings/mkdocstrings) to build the API documentation from the docstrings in the library code itself. Currently this is empty. If any stubs are added then it's necessary to build the documentation from a Python environment that has the package installed. A stub takes the form
+The /docs/open_mer folder can hold stubs to tell the [mkdocstrings plugin](https://github.com/mkdocstrings/mkdocstrings) to build the API documentation from the docstrings in the library code itself. Currently, this is empty. If any stubs are added then it's necessary to build the documentation from a Python environment that has the package installed.
+
+A stub takes the form
 
 ```
 # Title
-::: neuroport_dbs.module.name
+::: open_mer.module.name
 ```
 
 ### Testing the documentation locally
@@ -33,92 +37,68 @@ If you build the docs locally then you'll also get the /site directory, but this
 
 TODO
 
-## Maintaining the Zip Distribution
-
-* Download the latest [WinPython release](https://github.com/winpython/winpython/releases/latest).
-    * These instructions were tested with Winpython64-3.8.5.0
-* Run the WinPython self-extracting executable. This will create a folder containing a full Python distribution with many useful packages installed (see full list [here](https://github.com/winpython/winpython/blob/master/changelogs/WinPython-64bit-3.8.5.0.md)).
-* [Edit the `WPy64-3850\settings\winpython.ini` file](https://sourceforge.net/p/winpython/wiki/Environment/) and add the following line: `PATH = %WINPYDIR%\Lib\site-packages\PyQt5\Qt\bin;%PATH%`
-* Download [MySQL Windows ZIP Archive](https://dev.mysql.com/downloads/mysql/)
-    * Tested with mysql-8.0.29-win64.zip
-* Next to the WinPython folder, extract the mysql zip and rename the extracted folder to `mysql`
-* In the WinPython folder, run "WinPython Command Prompt". This will open a Command Prompt with all the paths configured to use this new Python distribution.
-* Install all of the Python packages listed in the table below.
-    * Version numbers may not be important. Please try the latest version and report to us if it does not work.
-    * The method to install the packages isn't important. If you're on an internet-connected computer then you can use the pip commands. Otherwise you can first download the wheels then bring them to the development computer to pip install the wheels.
-    * If you wish to be able to modify any of the SachsLab packages that are pure python (mspacman, cerebuswrapper, serf, **neuroport_dbs**) then you may do so by first cloning the repository to get the source and installing the package in-place: Using the WinPython command prompt, run `pip install -e .` from within the cloned directory.
-    * The `cerebus` package may complain "DLL load failed". This happens when cerebus.cbpy can't find Qt5 or it finds the wrong version. This SHOULD be fixed by editing the PATH in the 3rd step above, but I also found it necessary to copy Qt5Core.dll and Qt5Xml.dll from the above path directly into the site-packages\cerebus folder. We hope to remove the qt dependency from cerebus to avoid this in the future.
-* In the command prompt, `cd` into the `bin` subfolder of the unzipped mysql folder.
-* Create a mysql\data folder along with the base databases: `mysqld --initialize-insecure --console`
-    * You can change the default data directory, username, and password. See the section below "Configuring MySQL Database Server"
-* Run `mysqld` in the `mysql\bin` folder. (Windows: `start /B mysqld.exe`; allow network access if asked.)
-* Back in the command prompt, run `mysqladmin --user=root create serf`
-* Install the serf databases with the following commands:
-    ```
-    serf-makemigrations
-    serf-migrate
-    ```
-* Make a batch file `WPy64-3850\scripts\NeuroportDBS.bat` with the following contents:
-    ```shell script
-    @echo off
-    call "%~dp0env_for_icons.bat"
-    start "" "%WINPYDIR%\Scripts\dbs-sweep.exe" /command:%1 /B
-    start "" "%WINPYDIR%\Scripts\dbs-raster.exe" /command:%1 /B
-    start "" "%WINPYDIR%\Scripts\dbs-waveform.exe" /command:%1 /B
-    start "" "%WINPYDIR%\Scripts\dbs-ddu.exe" /command:%1 /B
-    start "" "%WINPYDIR%\Scripts\dbs-features.exe" /command:%1 /B
-    ```
-* Jump ahead to [Usage Instructions](#usage-instructions) below.
-
-### Required Python Packages
-
-| Package        | Version    | Wheel | pip command |
-| -------        | -------    | ----- | ----------- |
-| pyFFTW         | 0.12.0     | [Link](https://files.pythonhosted.org/packages/2b/e4/822d4cf12cd907fb8e80844db48aef7adf9e888c89256feb510fa81ae83f/pyFFTW-0.12.0-cp38-cp38-win_amd64.whl)
-| mysqlclient    | 2.0.1      | [Link](https://files.pythonhosted.org/packages/b2/72/e205fcf877dd0ec05d71b975def8ecef3ae4bb7fee14434615140ebdc168/mysqlclient-2.0.1-cp38-cp38-win_amd64.whl)
-| Django         | 3.1        | [Link](https://files.pythonhosted.org/packages/2b/5a/4bd5624546912082a1bd2709d0edc0685f5c7827a278d806a20cf6adea28/Django-3.1-py3-none-any.whl)
-| quantities     | 0.12.4     | | |
-| python-neo     | 0.9.0       | | `pip install git+https://github.com/NeuralEnsemble/python-neo.git`
-| pylsl          | 1.13.6     | [Link](https://files.pythonhosted.org/packages/02/c2/7b58adda02dbfa8f76bf835879d36b83dfc1da2eaa50d124d13a515e148c/pylsl-1.13.6-py2.py3-none-win_amd64.whl)
-| pytf           | 0.1        | [Link](https://github.com/SachsLab/pytf/releases/download/v0.1/pytf-0.1-py2.py3-none-any.whl) |`pip install git+https://github.com/SachsLab/pytf.git`|
-| mspacman       | 0.1        | [Link](https://github.com/SachsLab/mspacman/releases/download/v0.1/mspacman-0.1-py2.py3-none-any.whl) |`pip install git+https://github.com/SachsLab/mspacman.git`|
-| cerebus        | 0.0.4      | [Link](https://github.com/dashesy/CereLink/releases/download/v7.0.5/cerebus-0.0.4-cp38-cp38-win_amd64.whl) |N/A - must use wheel|
-| cerebuswrapper | 0.1      | [Link](https://github.com/SachsLab/cerebuswrapper/releases/download/v0.1/cerebuswrapper-0.1.0-py3-none-any.whl) |`pip install git+https://github.com/SachsLab/cerebuswrapper.git`|
-| serf           | 1.1        | [Link](https://github.com/cboulay/SERF/releases/download/v1.1/serf-1.1-py3-none-any.whl) | `pip install git+https://github.com/cboulay/SERF.git#subdirectory=python`|
-| neurport_dbs   | 1.0        | [Link](https://github.com/SachsLab/NeuroportDBS/releases/download/v1.0/neuroport_dbs-1.0.0-py3-none-any.whl) | `pip install git+https://github.com/SachsLab/NeuroportDBS.git`|
-
-### Configuring MySQL Database Server
-
-* If you wish to use a different datadir then you must first create a `my.cnf` file in the root `mysql` folder with the following contents (commented out lines aren't necessary, just keeping them here for reference):
-    ```
-    [mysqld]
-    datadir=path/to/data
-    #port = 3306
-    #socket = /tmp/mysql.sock
-    #pid-file = /Volumes/STORE/eerfdata/Chadwicks-MacBook-Pro.local.pid
-    #default-storage-engine = MyISAM
-    #default_tmp_storage_engine = MyISAM
-    #query_cache_type = 1
-    #key_buffer_size = 2G
-    #query_cache_limit = 400M
-    ```
-* If you wish to secure the database then you'll need to give the root account a password. Do so with `mysql_secure_installation`.
-* If you change from the default username (`root`) and password (none) then you will have to tell `serf` what the username and password are. Create a file named `my_serf.cnf` and put it in the path identified by the following command: `python -c "import os; print(os.path.expanduser('~'))"` The file contents should be
-    ```
-    [client]
-    user = root
-    password = {password}
-    #port = 3306
-    #socket = /tmp/mysql.sock
-    ```
-
-## For experts who want to use their existing environment
-
-We assume you know how to work with conda environments and that you have a MySQL database server running and configured to your liking.
-
-* Install the Python packages from the table above.
-* Adapt the instructions at [Segmented Electrophys Recordings and Features Database (SERF)](https://github.com/cboulay/SERF) to prepare the database server for these tools.
-* If you have a hybrid distribution/system-MySQL environment (i.e., your name is Guillaume) then you may also wish to use some of the MySQL DB config settings from above.
+## Interprocess Communication
+
+This section is referring to communication among the applications within the OpenMER suite, including mysqld and ~8 Python applications. Communication to/from the data sources is out-of-scope in this section.
+
+The applications all run independently of each other, but most of them work better in combination. To communicate information between applications we use [ZeroMQ](https://zeromq.org/).
+
+| Publisher         | Port  | Topic              | Message                                                | Subscribers       |
+|-------------------|-------|--------------------|--------------------------------------------------------|-------------------|
+| ProcedureGUI      | 60001 | procedure_settings | json of settings. "procedure": {}, "running": bool     | Segments_Process  |
+| Segments_Process  | 60002 | snippet_status     | (startup, notrecording, recording, accumulating, done) | ProcedureGUI      |
+| SweepGUI          | 60003 | channel_select     | json with channel, range, highpass                     | FeaturesGUI       |
+| Features_Process  | 60004 | features           | refresh                                                | ProcedureGUI      |
+| DepthGUI          | 60005 | ddu                | float of depth                                         | Segments_Process  |
+
+We also have one LSL stream coming from the DepthGUI. Old version of the SERF Depth_Process might still be using it but they should be migrated. 
+
+| Stream Name     | Stream Type | Content                 | Inlets                     |
+|-----------------|-------------|-------------------------|----------------------------|
+| electrode_depth | depth       | 1 float32 of elec depth | *old* Depth_Process (SERF) |
+
+## Development Environment
+
+### Blackrock Neuroport
+
+When using Blackrock hardware, the following tools and SDKs are needed.
+
+The Blackrock NSP has its own [NeuroPort Central Suite](https://www.blackrockmicro.com/technical-support/software-downloads/) to manage the configuration of the device and to store data. However, its data visualization capabilities are rather limited and not suited for DBS MER.
+
+The NSP data stream is accessible via an open source API [CereLink](https://github.com/CerebusOSS/CereLink) which includes a Python interface called `cerebus.cbpy`. These are maintained by Sachs Lab member Chadwick Boulay. Most of our OpenMER software is written in Python and depends on `cerebus.cbpy` and a custom [cerebuswrapper](https://github.com/SachsLab/cerebuswrapper) to communicate with the NSP.
+
+#### nPlayServer
+
+For development, it is useful to playback previously recorded data, without need of hardware or patients.
+nPlayServer emulates a Blackrock Neuroport almost perfectly -- just with a different ip address and it plays back an old data file instead of provides new data. See below for platform-specific instructions.
+
+When using nPlayServer, you can follow the general [Usage Instructions](./usage-instructions.md) with one modification:
+Modify the [DepthGUI settings](settings.md#depthguiini) to use "cbsdk playback" as its source. The depth value might not update until the file playback emits a change in depth.
+
+**Windows**
+
+* Run "C:\Program Files (x86)\Blackrock Microsystems\NeuroPort Windows Suite\runNPlayAndCentral.bat"
+* Select a recording to play back
+* Use Central's hardware configuration tool to enable continuous recording and spike extraction on the recorded channels.
+
+**Nix**
+
+* Blackrock does not distribute nPlayServer on macOS or Linux. However, it does exist. Contact Chad directly.
+* Central is unavailable on these platforms. Use `pycbsdk` to quickly change the hardware configuration.
+
+### Playback XDF file
+
+If you have a correctly formatted file, it may be enough to use [pyxdf playback](https://github.com/xdf-modules/pyxdf/blob/main/pyxdf/examples/playback_lsl.py).
+
+TODO: More instructions needed.
+
+### Dependencies
+
+We assume you know how to work with conda / mamba environments and that you have a MySQL database server running and configured to your liking.
+
+* Create an `openmer` conda environment.
+* Install the Python packages from the [table](preparing-distribution.md#required-python-packages).
+* Adapt the instructions at [Segmented Electrophys Recordings and Features Database (SERF)](https://github.com/cboulay/SERF) to prepare the database server for your development environment.
 
 ## Future goal - installable package
 

docs/getting-started.md

@@ -1,25 +1,30 @@
-Our primary method of distributing the full OpenMER Suite is as a giant zip file. TODO: Link!
+The PC that runs our software is directly connected to the acquisition system, but it is never connected to the internet. Thus, we create a portable install on a thumb drive which we then copy to the clinical PC.
 
-If you want to setup the individual pieces on an internet-connected computer (e.g., for development or testing) then please look at the [For Developers](./for-developers.md) documentation. 
+> For development or testing on an internet-connected computer, or a non-Windows computer, please look at the [For Developers](for-developers.md) documentation.
 
-It is expected that the target computer is a standalone computer that has a dedicated connection to the data acquisition system, such as a manufacturer-provided PC which is usually not connected to the internet. Testing without the hardware is also possible using a signal generator source or a data playback source (see below for example).
+## Installation
 
-Extract the zip file to the target computer. Choose a destination with a lot of disk space because the data segments will be saved within.
+### Distribution
 
-Updates may come in the form of a smaller zip file to extract within a specific subfolder of the extracted distribution.
+Copy the `<distribution>` folder from the thumb drive to the instrument PC.
 
-Proceed with the [Usage Instructions](./usage-instructions.md)
+> Be sure to choose a destination with lots of disk space because many recording segments will be stored within the database located in this folder.
 
-## Test Environment - Without Hardware
+> If you do not have the `<distribution>` folder then follow the [Preparing Distribution](preparing-distribution.md) instructions to create it.
 
-### Emulate Blackrock NSP
+### Configure
 
-* Run "C:\Program Files (x86)\Blackrock Microsystems\NeuroPort Windows Suite\runNPlayAndCentral.bat"
-* Select a recording to play back
-* Use Central's hardware configuration tool to enable continuous recording and spike extraction on the recorded channels.
-* Follow the general [Usage Instructions](./usage-instructions.md) with one modification:
-    * When running `dbs-ddu`, choose "cbsdk playback" from the dropdown menu to reuse the depths from the recording. The value might not update until the file plays back a change in depth.
+The `<distribution>` folder is ready to use as-is. However, with some additional steps it can be more useful on the target PC.
 
-### Playback XDF file
+#### Shortcuts
 
-More instructions are needed. If you have a correctly formatted file, it may be enough to use [XDFStreamer](https://github.com/labstreaminglayer/App-XDFStreamer).
+* Make a desktop shortcut to `<distribution>\mysql\bin\mysqld.exe`.
+* Make a desktop shortcut to `<distribution>\<python>\scripts\OpenMER.bat`
+
+#### Settings files
+
+OpenMER functionality can be modified via INI files. See [Settings](settings.md) for more information.
+
+## Using OpenMER
+
+See [Usage Instructions](usage-instructions.md)