Skip to content

Release kq mutex when waiting for events #162

@arr2036

Description

@arr2036

libkqueue has multiple issues:

  1. If multiple threads attempt to modify a kqueue that is waiting for events, then the mutex
    cannot be acquired until the waiting is done, which causes threads to block, and makes
    user events useless.
  2. kqueue behaviour is that when one thread closes a kqueue fd, all threads waiting,
    immediately wake up and register an error. libkqueue does not do this.

Common code changes changes:

  1. An enum of states should be added to the common kqueue code

KQ_ACTIVE = 0
KQ_SHUTDOWN,
KQ_FREE //!< For debugging, set at the end of kqueue_free

  1. An additional kqop should be added for kqueue_shutdown. This frees the platform specific
    port, iocp, fd handle used for waiting on events, which in turn wakes up anything waiting
    and causes it to return an error.

  2. A "waiting" counter needs to be added to the common kqueue structure. This tracks
    how many kqueues are waiting on events.
    kevent() should acquire the kq mutex, then check the kq state.
    If kevent() is called with a kq in the KQ_SHUTDOWN state, it should return EBADF.
    if kevent() is called with a kq in the KQ_FREE state, it should assert or return EBADF.
    If kevent() is called with a kq in the KQ_ACTIVE state, it should increment a waiting
    counter, release the mutex, and call the platform wait function.
    When the platform wait function completes, kevent() should lock the kq mutex and
    decrement the waiting counter, then check the kq state.

    • KQ_ACTIVE continue with copy_out.
    • KQ_SHUTDOWN call kqueue_free() return EBADF.
    • KQ_FREE assert or return EBADF.
  3. Calls to kqueue_free should be replaced with calls to kqueue_shutdown.
    In kqueue_shutdown if the waiting counter is > 0, kqops.kqueue_shutdown function
    is called which should free the eventing handle, and the kqueue state should be
    set to KQ_SHUTDOWN.
    If the waiting count == 0, then kqops.kqueue_shutdown is called, then kqueue_free()
    should be called to release resources.

  4. kqueue_free() should be modified to not close the eventing handle and to set the state
    to KQ_FREE. If kqueue_free() is called with a kq in any state except KQ_SHUTDOWN
    it must assert.

Platform specific changes:

  1. Code that previously closed the eventing handle should be moved to kqueue_shutdown

  2. kevent_wait should be modified to handle errors generated by closing the eventing
    handle gracefully.

Testing:

  1. Multi-threaded user test should be added to verify signalling occurs when one kqueue
    is in the waiting state.

  2. Multi-threaded test should be added to verify one thread closing a kqueue terminates
    the waiting of another.

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions