Best Approach for brokerless Python workers?
I'm developing a custom tool that needs to coordinate multiple long-running worker processes for task execution without relying on external brokers like Redis or RabbitMQ. We want to avoid multiprocessing.Pool due to the setup time required for spawning new processes. Instead, we prefer persistent, warmed-up worker processes that can handle tasks efficiently.
Would ZeroMQ still be the best choice for this, or are there other alternatives with minimal overhead? Ok, kinda sounds like mixing apples and oranges. Are the multiple long-running worker processes working on distinct tasks (aka each is the equivalent of starting a distinct individual process/program with no dependencies) ?? worker process that are awaiting data to process ?? simplified related bash shell examples : aka guarantee sequencial process running : https://unix.stackexchange.com/questions/305039/pausing-a-ba... vs. wait for all processes to end (can be modified to continual request for next thing(s) to do) : https://stackoverflow.com/questions/356100/how-to-wait-in-ba...