-
Notifications
You must be signed in to change notification settings - Fork 0
bug: InferenceQueue is not priority-aware despite being documented as such #49
Description
Bug: InferenceQueue ignores Priority
File: internal/scheduler/queue.go
The package doc and struct comment both describe the queue as "priority-aware", and four Priority constants are defined (PriorityWorker, PriorityPlanner, PriorityCorrector, PriorityEvaluator). However, Submit() never uses the priority — it simply blocks on a semaphore channel:
func (q *InferenceQueue) Submit(ctx context.Context, req InferenceRequest) error {
// ...
select {
case q.slots <- struct{}{}: // FIFO channel — priority ignored
case <-ctx.Done():
return ctx.Err()
}
// ...
}The InferenceRequest.Priority field is never read. All requests are served FIFO regardless of their declared priority. This means PriorityEvaluator = 3 gets no advantage over PriorityWorker = 1.
Impact
- Evaluator and Corrector agents (higher priority) are not given inference slots ahead of Worker agents. Under load this can cause governance-critical evaluation to queue behind routine worker calls.
- Misleading documentation increases the chance of future callers setting priorities and expecting them to be honoured.
Suggested fix
Replace the semaphore channel with a proper heap-based priority queue (e.g. container/heap) that dequeues by descending priority. Or, if strict priority isn't needed yet, remove the Priority field from InferenceRequest and update the docs to say the queue is FIFO with a concurrency cap.