Requires: Python >=3.5
However, it was proven that shared memory tends to violate the laws of Physics.
<- Joe Armstrong, the creator of Erlang.
The solution presented by Erlang, (and ZMQ) is the notion of
message passing for achieving parallelism.
However, Message passing can be tedious, and often un-pythonic.
This is where ZProc comes in.
It provides you a middle ground between message passing and shared memory.
It does message passing, without you ever knowing that it's doing it.
It provides you a global
state is not a shared object.
It works purely on message passing.
It also supports a fair bit of reactive programming, (kind of like ReactJS) using state watchers.
Behind the covers, it simulates the Actor Model.
ZProc doesn't blindly follow it, but you can think of it as such.
It also borrows the
autoretry feature of Celery, but unlike
Celery it doesn't need a broker.
The zen of zero
The Ø in ØMQ is all about trade-offs. On the one hand, this strange
name lowers ØMQ’s visibility on Google and Twitter. On the other hand,
it annoys the heck out of some Danish folk who write us things like
“ØMG røtfl”, and “Ø is not a funny-looking zero!” and “Rødgrød med
Fløde!” (which is apparently an insult that means “May your neighbours
be the direct descendants of Grendel!”). Seems like a fair trade.
Originally, the zero in ØMQ was meant to signify “zero broker” and
(as close to) “zero latency” (as possible). Since then, it has come
to encompass different goals: zero administration, zero cost, zero
waste. More generally, “zero” refers to the culture of minimalism
that permeates the project. We add power by removing complexity
rather than by exposing new functionality.
- The state only gets updated if you do it directly.
This means that if you mutate objects inside the state,
they wont get reflected in the global state.
- The state should be pickle-able
- It runs an extra Process for managing the state.
Its fairly lightweight though, and shouldn't add too
much weight to your application.
- Above all, ZProc is written for safety and ease of use.
- However, since its written using ZMQ, it's plenty fast for most stuff.
- Run -> for a taste.
- The code itself is stable, but the API is quite unstable.
- Please don't use it in production right now.
Zproc uses a Server, which is responsible for storing and communicating the state.
- This isolates our resource (state), eliminating the need for locks.
The process(s) communicate through ZMQ sockets, over
- The clients (Proceses) use a
- The Server uses a
If a Process wishes to watch the state, it subscribes to a global publish message.
- The zproc server publishes the state at every state update.
- A Process may subscribe to this message and
filter out the event it needs (using
- zmq sockets block your application efficiently
till an update occurs, eliminating the need for Busy Waiting.
(Busy waiting refers to the
while True: <check condition> approach).
git clone https://github.com/pycampers/zproc.git
Assuming you have sphinx installed (Linux)
pipenv run ./build.sh
- Thanks to pieter hintjens,
for his work on the ZeroMQ library
and for his amazing book.
- Thanks to tblib,
ZProc can raise First-class Exceptions from the zproc server!
- Thanks to psutil,
ZProc can handle nested procesess!
- Thanks to Kennith Rietz.
His setup.py was used to host this project on pypi.
Plus lot of documentation is blatantly copied
from his documentation on requests