Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ Available addons
addon | version | maintainers | summary
--- | --- | --- | ---
[base_import_async](base_import_async/) | 18.0.1.0.0 | | Import CSV files in the background
[queue_job](queue_job/) | 18.0.2.0.2 | <a href='https://github.com/guewen'><img src='https://github.com/guewen.png' width='32' height='32' style='border-radius:50%;' alt='guewen'/></a> | Job Queue
[queue_job](queue_job/) | 18.0.2.0.6 | <a href='https://github.com/guewen'><img src='https://github.com/guewen.png' width='32' height='32' style='border-radius:50%;' alt='guewen'/></a> <a href='https://github.com/sbidoul'><img src='https://github.com/sbidoul.png' width='32' height='32' style='border-radius:50%;' alt='sbidoul'/></a> | Job Queue
[queue_job_batch](queue_job_batch/) | 18.0.1.0.0 | | Job Queue Batch
[queue_job_cron](queue_job_cron/) | 18.0.1.1.1 | | Scheduled Actions as Queue Jobs
[queue_job_cron_jobrunner](queue_job_cron_jobrunner/) | 18.0.1.0.1 | <a href='https://github.com/ivantodorovich'><img src='https://github.com/ivantodorovich.png' width='32' height='32' style='border-radius:50%;' alt='ivantodorovich'/></a> | Run jobs without a dedicated JobRunner
[queue_job_subscribe](queue_job_subscribe/) | 18.0.1.0.0 | | Control which users are subscribed to queue job notifications
[test_queue_job](test_queue_job/) | 18.0.2.0.0 | | Queue Job Tests
[test_queue_job](test_queue_job/) | 18.0.2.0.1 | <a href='https://github.com/sbidoul'><img src='https://github.com/sbidoul.png' width='32' height='32' style='border-radius:50%;' alt='sbidoul'/></a> | Queue Job Tests
[test_queue_job_batch](test_queue_job_batch/) | 18.0.1.0.0 | | Test Job Queue Batch

[//]: # (end addons)
Expand Down
24 changes: 6 additions & 18 deletions queue_job/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Job Queue
!! This file is generated by oca-gen-addon-readme !!
!! changes will be overwritten. !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! source digest: sha256:58f9182440bb316576671959b69148ea5454958f9ae8db75bccd30c89012676d
!! source digest: sha256:9294f4c715c0f0e10a55590082388776b34763472ac72b2b88d0d464b31f42a3
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

.. |badge1| image:: https://img.shields.io/badge/maturity-Mature-brightgreen.png
Expand Down Expand Up @@ -627,21 +627,6 @@ Known issues / Roadmap

- After creating a new database or installing ``queue_job`` on an
existing database, Odoo must be restarted for the runner to detect it.
- When Odoo shuts down normally, it waits for running jobs to finish.
However, when the Odoo server crashes or is otherwise force-stopped,
running jobs are interrupted while the runner has no chance to know
they have been aborted. In such situations, jobs may remain in
``started`` or ``enqueued`` state after the Odoo server is halted.
Since the runner has no way to know if they are actually running or
not, and does not know for sure if it is safe to restart the jobs, it
does not attempt to restart them automatically. Such stale jobs
therefore fill the running queue and prevent other jobs to start. You
must therefore requeue them manually, either from the Jobs view, or by
running the following SQL statement *before starting Odoo*:

.. code:: sql

update queue_job set state='pending' where state in ('started', 'enqueued')

Changelog
=========
Expand Down Expand Up @@ -715,10 +700,13 @@ promote its widespread use.
.. |maintainer-guewen| image:: https://github.com/guewen.png?size=40px
:target: https://github.com/guewen
:alt: guewen
.. |maintainer-sbidoul| image:: https://github.com/sbidoul.png?size=40px
:target: https://github.com/sbidoul
:alt: sbidoul

Current `maintainer <https://odoo-community.org/page/maintainer-role>`__:
Current `maintainers <https://odoo-community.org/page/maintainer-role>`__:

|maintainer-guewen|
|maintainer-guewen| |maintainer-sbidoul|

This module is part of the `OCA/queue <https://github.com/OCA/queue/tree/18.0/queue_job>`_ project on GitHub.

Expand Down
4 changes: 2 additions & 2 deletions queue_job/__manifest__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

{
"name": "Job Queue",
"version": "18.0.2.0.2",
"version": "18.0.2.0.6",
"author": "Camptocamp,ACSONE SA/NV,Odoo Community Association (OCA)",
"website": "https://github.com/OCA/queue",
"license": "LGPL-3",
Expand All @@ -29,7 +29,7 @@
},
"installable": True,
"development_status": "Mature",
"maintainers": ["guewen"],
"maintainers": ["guewen", "sbidoul"],
"post_init_hook": "post_init_hook",
"post_load": "post_load",
}
20 changes: 16 additions & 4 deletions queue_job/controllers/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,15 @@ def _enqueue_dependent_jobs(self, env, job):
tries = 0
while True:
try:
job.enqueue_waiting()
with job.env.cr.savepoint():
job.enqueue_waiting()
except OperationalError as err:
# Automatically retry the typical transaction serialization
# errors
if err.pgcode not in PG_CONCURRENCY_ERRORS_TO_RETRY:
raise
if tries >= DEPENDS_MAX_TRIES_ON_CONCURRENCY_FAILURE:
_logger.info(
_logger.error(
"%s, maximum number of tries reached to update dependencies",
errorcodes.lookup(err.pgcode),
)
Expand Down Expand Up @@ -177,6 +178,7 @@ def create_test_job(
description="Test job",
size=1,
failure_rate=0,
job_duration=0,
):
if not http.request.env.user.has_group("base.group_erp_manager"):
raise Forbidden(_("Access Denied"))
Expand All @@ -187,6 +189,12 @@ def create_test_job(
except (ValueError, TypeError):
failure_rate = 0

if job_duration is not None:
try:
job_duration = float(job_duration)
except (ValueError, TypeError):
job_duration = 0

if not (0 <= failure_rate <= 1):
raise BadRequest("failure_rate must be between 0 and 1")

Expand Down Expand Up @@ -215,6 +223,7 @@ def create_test_job(
channel=channel,
description=description,
failure_rate=failure_rate,
job_duration=job_duration,
)

if size > 1:
Expand All @@ -225,6 +234,7 @@ def create_test_job(
channel=channel,
description=description,
failure_rate=failure_rate,
job_duration=job_duration,
)
return ""

Expand All @@ -236,6 +246,7 @@ def _create_single_test_job(
description="Test job",
size=1,
failure_rate=0,
job_duration=0,
):
delayed = (
http.request.env["queue.job"]
Expand All @@ -245,7 +256,7 @@ def _create_single_test_job(
channel=channel,
description=description,
)
._test_job(failure_rate=failure_rate)
._test_job(failure_rate=failure_rate, job_duration=job_duration)
)
return f"job uuid: {delayed.db_record().uuid}"

Expand All @@ -259,6 +270,7 @@ def _create_graph_test_jobs(
channel=None,
description="Test job",
failure_rate=0,
job_duration=0,
):
model = http.request.env["queue.job"]
current_count = 0
Expand All @@ -281,7 +293,7 @@ def _create_graph_test_jobs(
max_retries=max_retries,
channel=channel,
description="%s #%d" % (description, current_count),
)._test_job(failure_rate=failure_rate)
)._test_job(failure_rate=failure_rate, job_duration=job_duration)
)

grouping = random.choice(possible_grouping_methods)
Expand Down
43 changes: 26 additions & 17 deletions queue_job/jobrunner/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -357,23 +357,26 @@ def _query_requeue_dead_jobs(self):
ELSE exc_info
END)
WHERE
id in (
SELECT
queue_job_id
FROM
queue_job_lock
WHERE
queue_job_id in (
SELECT
id
FROM
queue_job
WHERE
state IN ('enqueued','started')
AND date_enqueued <
(now() AT TIME ZONE 'utc' - INTERVAL '10 sec')
)
FOR UPDATE SKIP LOCKED
state IN ('enqueued','started')
AND date_enqueued < (now() AT TIME ZONE 'utc' - INTERVAL '10 sec')
AND (
id in (
SELECT
queue_job_id
FROM
queue_job_lock
WHERE
queue_job_lock.queue_job_id = queue_job.id
FOR UPDATE SKIP LOCKED
)
OR NOT EXISTS (
SELECT
1
FROM
queue_job_lock
WHERE
queue_job_lock.queue_job_id = queue_job.id
)
)
RETURNING uuid
"""
Expand All @@ -396,6 +399,12 @@ def requeue_dead_jobs(self):
However, when the Odoo server crashes or is otherwise force-stopped,
running jobs are interrupted while the runner has no chance to know
they have been aborted.

This also handles orphaned jobs (enqueued but never started, no lock).
This edge case occurs when the runner marks a job as 'enqueued'
but the HTTP request to start the job never reaches the Odoo server
(e.g., due to server shutdown/crash between setting enqueued and
the controller receiving the request).
"""

with closing(self.conn.cursor()) as cr:
Expand Down
22 changes: 17 additions & 5 deletions queue_job/models/queue_job.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

import logging
import random
import time
from datetime import datetime, timedelta

from odoo import _, api, exceptions, fields, models
Expand All @@ -16,6 +17,7 @@
from ..job import (
CANCELLED,
DONE,
ENQUEUED,
FAILED,
PENDING,
STARTED,
Expand Down Expand Up @@ -324,18 +326,26 @@ def _change_job_state(self, state, result=None):
raise ValueError(f"State not supported: {state}")

def button_done(self):
# If job was set to STARTED or CANCELLED, do not set it to DONE
states_from = (WAIT_DEPENDENCIES, PENDING, ENQUEUED, FAILED)
result = _("Manually set to done by {}").format(self.env.user.name)
self._change_job_state(DONE, result=result)
records = self.filtered(lambda job_: job_.state in states_from)
records._change_job_state(DONE, result=result)
return True

def button_cancelled(self):
# If job was set to DONE or WAIT_DEPENDENCIES, do not cancel it
states_from = (PENDING, ENQUEUED, FAILED)
result = _("Cancelled by {}").format(self.env.user.name)
self._change_job_state(CANCELLED, result=result)
records = self.filtered(lambda job_: job_.state in states_from)
records._change_job_state(CANCELLED, result=result)
return True

def requeue(self):
jobs_to_requeue = self.filtered(lambda job_: job_.state != WAIT_DEPENDENCIES)
jobs_to_requeue._change_job_state(PENDING)
# If job is already in queue or started, do not requeue it
states_from = (FAILED, DONE, CANCELLED)
records = self.filtered(lambda job_: job_.state in states_from)
records._change_job_state(PENDING)
return True

def _message_post_on_failure(self):
Expand Down Expand Up @@ -442,7 +452,9 @@ def related_action_open_record(self):
)
return action

def _test_job(self, failure_rate=0):
def _test_job(self, failure_rate=0, job_duration=0):
_logger.info("Running test job.")
if random.random() <= failure_rate:
raise JobError("Job failed")
if job_duration:
time.sleep(job_duration)
15 changes: 0 additions & 15 deletions queue_job/readme/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,2 @@
- After creating a new database or installing `queue_job` on an existing
database, Odoo must be restarted for the runner to detect it.
- When Odoo shuts down normally, it waits for running jobs to finish.
However, when the Odoo server crashes or is otherwise force-stopped,
running jobs are interrupted while the runner has no chance to know
they have been aborted. In such situations, jobs may remain in
`started` or `enqueued` state after the Odoo server is halted. Since
the runner has no way to know if they are actually running or not, and
does not know for sure if it is safe to restart the jobs, it does not
attempt to restart them automatically. Such stale jobs therefore fill
the running queue and prevent other jobs to start. You must therefore
requeue them manually, either from the Jobs view, or by running the
following SQL statement *before starting Odoo*:

``` sql
update queue_job set state='pending' where state in ('started', 'enqueued')
```
20 changes: 3 additions & 17 deletions queue_job/static/description/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -372,7 +372,7 @@ <h1>Job Queue</h1>
!! This file is generated by oca-gen-addon-readme !!
!! changes will be overwritten. !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! source digest: sha256:58f9182440bb316576671959b69148ea5454958f9ae8db75bccd30c89012676d
!! source digest: sha256:9294f4c715c0f0e10a55590082388776b34763472ac72b2b88d0d464b31f42a3
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -->
<p><a class="reference external image-reference" href="https://odoo-community.org/page/development-status"><img alt="Mature" src="https://img.shields.io/badge/maturity-Mature-brightgreen.png" /></a> <a class="reference external image-reference" href="http://www.gnu.org/licenses/lgpl-3.0-standalone.html"><img alt="License: LGPL-3" src="https://img.shields.io/badge/license-LGPL--3-blue.png" /></a> <a class="reference external image-reference" href="https://github.com/OCA/queue/tree/18.0/queue_job"><img alt="OCA/queue" src="https://img.shields.io/badge/github-OCA%2Fqueue-lightgray.png?logo=github" /></a> <a class="reference external image-reference" href="https://translation.odoo-community.org/projects/queue-18-0/queue-18-0-queue_job"><img alt="Translate me on Weblate" src="https://img.shields.io/badge/weblate-Translate%20me-F47D42.png" /></a> <a class="reference external image-reference" href="https://runboat.odoo-community.org/builds?repo=OCA/queue&amp;target_branch=18.0"><img alt="Try me on Runboat" src="https://img.shields.io/badge/runboat-Try%20me-875A7B.png" /></a></p>
<p>This addon adds an integrated Job Queue to Odoo.</p>
Expand Down Expand Up @@ -928,21 +928,7 @@ <h2><a class="toc-backref" href="#toc-entry-11">Known issues / Roadmap</a></h2>
<ul class="simple">
<li>After creating a new database or installing <tt class="docutils literal">queue_job</tt> on an
existing database, Odoo must be restarted for the runner to detect it.</li>
<li>When Odoo shuts down normally, it waits for running jobs to finish.
However, when the Odoo server crashes or is otherwise force-stopped,
running jobs are interrupted while the runner has no chance to know
they have been aborted. In such situations, jobs may remain in
<tt class="docutils literal">started</tt> or <tt class="docutils literal">enqueued</tt> state after the Odoo server is halted.
Since the runner has no way to know if they are actually running or
not, and does not know for sure if it is safe to restart the jobs, it
does not attempt to restart them automatically. Such stale jobs
therefore fill the running queue and prevent other jobs to start. You
must therefore requeue them manually, either from the Jobs view, or by
running the following SQL statement <em>before starting Odoo</em>:</li>
</ul>
<pre class="code sql literal-block">
<span class="k">update</span><span class="w"> </span><span class="n">queue_job</span><span class="w"> </span><span class="k">set</span><span class="w"> </span><span class="k">state</span><span class="o">=</span><span class="s1">'pending'</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="k">state</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="p">(</span><span class="s1">'started'</span><span class="p">,</span><span class="w"> </span><span class="s1">'enqueued'</span><span class="p">)</span>
</pre>
</div>
<div class="section" id="changelog">
<h2><a class="toc-backref" href="#toc-entry-12">Changelog</a></h2>
Expand Down Expand Up @@ -1008,8 +994,8 @@ <h3><a class="toc-backref" href="#toc-entry-19">Maintainers</a></h3>
<p>OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.</p>
<p>Current <a class="reference external" href="https://odoo-community.org/page/maintainer-role">maintainer</a>:</p>
<p><a class="reference external image-reference" href="https://github.com/guewen"><img alt="guewen" src="https://github.com/guewen.png?size=40px" /></a></p>
<p>Current <a class="reference external" href="https://odoo-community.org/page/maintainer-role">maintainers</a>:</p>
<p><a class="reference external image-reference" href="https://github.com/guewen"><img alt="guewen" src="https://github.com/guewen.png?size=40px" /></a> <a class="reference external image-reference" href="https://github.com/sbidoul"><img alt="sbidoul" src="https://github.com/sbidoul.png?size=40px" /></a></p>
<p>This module is part of the <a class="reference external" href="https://github.com/OCA/queue/tree/18.0/queue_job">OCA/queue</a> project on GitHub.</p>
<p>You are welcome to contribute. To learn how please visit <a class="reference external" href="https://odoo-community.org/page/Contribute">https://odoo-community.org/page/Contribute</a>.</p>
</div>
Expand Down
Loading