Python client

[ewhauser]

I have a working Python client for river which is a basically a port of the Ruby client. Is there any interest in hosting this as part of the riverqueue org? Happy to release it under my profile first and transfer later if preferred.

Thanks!

[ewhauser]

Here's the initial version: https://github.com/ewhauser/riverqueue-py

[brandur]

@ewhauser HOLY CRAP, this is awesome!!! Going through the code it looks like you got some of the "edge" features too, like unique job insertion.

I'm semi-functional in Python, but not super familiar with how projects are run and modern day standards. OOC:

• Is it using an automatic code formatting tool? I was in VSCode and presented with a couple different options, and wanted to make sure I installed the right one.
• This is a really basic question, but how do you run the test suite? I tried googling for this, but couldn't find a super clear/generalized answer.

[ewhauser]

This should help: https://github.com/ewhauser/riverqueue-py/blob/main/.github/workflows/build.yml

You should be able to run all of those commands locally using rye: https://github.com/astral-sh/rye

It used ruff for formatting/linting and I'm using mypy for type checking

[brandur]

Ah okay, that does help. TIL Rye — there is something a little funny about the idea of using a Rust program to manage your Python projects, lol.

I was able to get it installed successfully, and most of the targets worked. The only thing that didn't work are the tests because I don't even have Docker installed — may want to add a non-Docker-based test target, but shouldn't be too hard.

[brandur]

Oh and actually two more follow ups:

• If I'm reading this right, the project establishes connections via SQLAlchemy. In Python is that kind of the overwhelming default for database use? Are there any other libraries out there that a Python River driver might want to be compatible with?
• It looks like you used Sqlc internally and made a driver for that — in other projects I wouldn't really consider Sqlc as a "driver" per se — it's more like a code generation tool that plugs into drivers. Sqlc looks like it's generating code that targets SQLAlchemy, so SQLAlchemy would be the "driver" in this situation. I guess this is more like a comment than a question, but is related to the point above in wondering if we'd need to target any other driver-level Python libraries besides SQLAlchemy.

[ewhauser]

1. I don't know about overwhelming but it's mostly what I see. The driver has an interface so if someone wants to implement their own implementation, then they can: https://github.com/ewhauser/riverqueue-py/blob/main/src/riverqueue/driver.py#L6

2. Although sqlc in Python uses sqlalchemy for connections, it doesn't really use any of the ORM mapping APIs that sqlalchemy provides - it's very sqlc specific. That being said, I think it's probably reasonable to name the driver sqlalchemy.

[brandur]

1. I don't know about overwhelming but it's mostly what I see. The driver has an interface so if someone wants to implement their own implementation, then they can: https://github.com/ewhauser/riverqueue-py/blob/main/src/riverqueue/driver.py#L6

Cool, thanks. Yeah the reason I ask is that what I did for the Ruby drivers is implement multiple driver targets (Sequel and ActiveRecord), but packaged each driver separately so that if a user is only using ActiveRecord, they pull in just the driver for ActiveRecord, and don't pick up an accidental transitive dependency on Sequel through River.

If there's anything like this in Python world, it'd be a good idea to do the same, but if SQLAlchemy is practically speaking the only game in town, then it might not be necessary.

2. Although sqlc in Python uses sqlalchemy for connections, it doesn't really use any of the ORM mapping APIs that sqlalchemy provides - it's very sqlc specific. That being said, I think it's probably reasonable to name the driver sqlalchemy.

Cool. Yeah I think this is more obvious when you use Sqlc with Go where it can target multiple drivers like jackc/pgx or database/sql, and you can configure it via an option in sqlc.yaml. You're proximately using Sqlc, but it's really Pgx by way of code generated by Sqlc, and calling a driver "Sqlc" would be ambiguous because it could mean either generated target.

If I'm reading the docs correctly for Sqlc's Python plugin correctly, there are no non-SQLAlchemy targets, so things aren't as clear. Still, it's probably better to call the driver something like "SQLAlchemyDriver" to be more consistent with how drivers are named in other languages.

[ewhauser]

If I'm reading the docs correctly for Sqlc's Python plugin correctly, there are no non-SQLAlchemy targets, so things aren't as clear. Still, it's probably better to call the driver something like "SQLAlchemyDriver" to be more consistent with how drivers are named in other languages.

Done: ewhauser/riverqueue-py#2

[ewhauser]

If there's anything like this in Python world, it'd be a good idea to do the same, but if SQLAlchemy is practically speaking the only game in town, then it might not be necessary.

We can use python extras so that the package is installable without the transitive dependencies. i.e. you would depend on riverqueue[sqlalchemy] instead of just riverqueue. I can do that as part of setting up publishing.

I think you already own https://pypi.org/project/riverqueue/ ? Do you want to move this repo over to https://github.com/riverqueue/ ? If so, then I'll just need you to add the pypi publishing secret to the repo config and then I can have the existing CI job publish on commits to main (or add an explicit publish job).

[brandur]

Heads up that the Python package (all based on Eric's work) is now being formally published to PyPI:

https://pypi.org/project/riverqueue/

And is documented over on the River docs site:

https://riverqueue.com/docs/python