Including dll driven instruments

[BenediktBurger]

Dear @scasagrande what's your opinion about dll driven instruments? Pymeasure does not want to include those.
I'm looking for a good instruments framework to include my lab instruments (including tests). Right now I have my own simple collection of files.

I'm willing to put my devices in such a framework and publish them upstream.

Some devices use dlls (NKT lasers, High finesse wavemeter etc.), I'd like to include them in the framework, but understand, if it is not desired.

By the way, I like your better testing ability than in pymeasure and that you do not have a gui, while I look for instruments only.

[scasagrande]

I am open to the idea, but we'll need to integrate these DLL/SO files in an intelligent way. Ideally we build them from source for each platform as part of CI. @bmoneke do you have the source code for those DLLs? Or are you speaking to closed-source DLLs provided by a manufacturer?

[BenediktBurger]

Unfortunately I'm talking about closed-source DLLs provided by the manufacturer, as such, I do not have the source code.

I know, that it is a difficult question. On one hand, the repository would not be enough to run the device. On the other hand it allows to use the device easily with python, once you follow the manufacturer's installation instructions.

[scasagrande]

Okay good to know.

I've been investigating how to build a plugin system to allow others to write their own drivers. It can also be used for installing "optional" drivers, say from an instrumentkit hosted repo containing Python wrappers for closed-source drivers. That way the default set of drivers don't require any extra work, but if you need that extra functionality you can install them.

[BenediktBurger]

The idea of an extra repository sounds great.

I don't know, how the usage will look like afterwards, but that way there is a clear distinction between byte stream oriented devices with a complete driver and wrappers or more complex drivers.

[scasagrande]

If we assume a package of the name instrumentkit-extras, it would work like so.

First install either by specifying to install the package:

$ pip install "instrumentkit-extras"

or during the main package installation, but making use of the "extra-deps" functionality of setuptools:

$ pip install "instrumentkit[extras]"

This would then map to installing instrumentkit, and the version of instrumentkit-extras as defined in instrumentkit/setup.cfg. Others could be included in this.

Then using the drivers would then ideally be the same as it is right now.

[BenediktBurger]

That sounds great.

[untzag]

Consider using NiceLib [1], developed by the Hideo Mabuchi Group [2]. It's developed by the Instrumental [3] folks but perhaps it could be used in your new package as well.

[1] https://nicelib.readthedocs.io
[2] https://mabuchilab.org
[3] https://instrumental-lib.readthedocs.io

[trappitsch]

I like the ideas with putting drivers that use close-sourced dlls into an extras package. Testing should still be possible and straight forward. Even now we are using the loopback communicator for testing, so the test suite doesn't need the actual instrument. That should basically be the same with the dll, i.e., there should be a loopback such that the dll is mocked appropriately.

One more thing to keep in mind: IK is currently pretty much platform agnostic, however, loading DLLs with ctypes.WinDLL will change that. Currently we define numpy as an extra dependence that does not need to be installed, would probably be good to simply throw an error if an, e.g., win-only driver is being used from Linux... On the other hand, the test suite needs to mock that appropriately out, since testing currently happens on linux (if I'm not mistaken).

Since you mentioned the high finesse wavemeter above: I actually did write a very simple start for a driver a while ago for the WS6. The idea was to write put it in the instrumentkit spirit, and I'm actually using the ProxyList part directly from ik. Here's the source: https://github.com/RIMS-Code/high_finesse_ws6
No tests at this point, but might be interesting / helpful (?) to check. I'd be interested in helping with developing tests / drivers.

[BenediktBurger]

Since you mentioned the high finesse wavemeter above: I actually did write a very simple start for a driver a while ago for the WS6. The idea was to write put it in the instrumentkit spirit, and I'm actually using the ProxyList part directly from ik. Here's the source: https://github.com/RIMS-Code/high_finesse_ws6 No tests at this point, but might be interesting / helpful (?) to check. I'd be interested in helping with developing tests / drivers.

Interesting @trappitsch , thanks for sharing, I also had found the ws7 driver and based an (yet untested) API on that version.

[BenediktBurger]

One caveat for dll-driven drivers: It is really hard to write meaningful tests for them, as I learned at the pico-python repository for picoscope digital oscilloscopes.

I appreciate how easy it is writing tests for instrumentkit due to the great expected_protocol function.

[trappitsch]

This might be ignorant, but I think it should be possible to mock the ctypes.WinDLL() calls and create a context manager similar to expected_protocol to work. That way, we should be able to create a similar loopback_communicator as for ik to test the DLL drivers accordingly.