Skip to content

free-threadable Arrow library #55

@1fish2

Description

@1fish2

At some point we'll want to parallelize Vivarium processes, mostly one thread per process but some Vivarium processes might be able to use multiple threads. Then we'd want the Arrow extension library to support free-threading.

Design alternatives:

  1. Leave the responsibility to the calling Python code to keep each StochasticSystem instance within a Python thread, like the rest of a Vivarium processes's state. This is the easiest approach. (Vivarium stores would need synchronization.)
  2. Ensure thread safety by making each public method of the StochasticSystem instance lock the instance state so multiple threads can't simultaneously access it.
  3. Get aggressive by allowing each Arrowhead instance to do its own internal parallelism, but I don't recall that it's a parallelizable algorithm.

Looking over the Arrow code, I don't see any thread-safety bugs like mutable global variables. All the state is kept in an instance variable of the StochasticSystem Python object, including an instance of the Arrowhead extension object. Good.

It doesn't look like it needs any atomic sections for the thread-local model (1). But we still have to:

  • Read up on Python & Cython free-threading,
  • Turn on Cython free-threading,
  • Write a test.

When this becomes worthwhile depends on when we're ready to parallelize Vivarium processes.

Furthermore, I'm optimistic about the prospect of using Mojo to compile down and parallelize the Vivarium code and models, making use of multi-cores, SIMD, GPUs, and other parallel hardware. It could be easier and more effective to move Python code (gillespy2?) to Mojo than to support free-threading in this C code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions