It would be great if faculty-distributed handled all platform requirements. A way to achieve this could be to create a Job when FacultyJobExecutor.map is called. The command of the job could also install faculty_distributed to avoid the need for a server environment: pip install -U git+ssh://git@github.com/facultyai/faculty-distributed.git && faculty_distributed_job $path $worker_id with the appropriate parameters. Resource requests and optional additional server environment IDs could be passed as constructor arguments.
Optionally, the job could deleted as part of the cleanup function.
It would be great if
faculty-distributedhandled all platform requirements. A way to achieve this could be to create a Job whenFacultyJobExecutor.mapis called. The command of the job could also installfaculty_distributedto avoid the need for a server environment:pip install -U git+ssh://git@github.com/facultyai/faculty-distributed.git && faculty_distributed_job $path $worker_idwith the appropriate parameters. Resource requests and optional additional server environment IDs could be passed as constructor arguments.Optionally, the job could deleted as part of the cleanup function.