Introduction¶
Runnel is a distributed event processing framework for Python based on Redis Streams.
Whereas traditional job queues do not provide ordering guarantees, Runnel is designed to process partitions of your event stream strictly in the order events are created.
Basic Usage¶
from datetime import datetime
from runnel import App, Record
app = App(name="myapp", redis_url="redis://127.0.0.1")
class Order(Record):
order_id: int
created_at: datetime
amount: int
orders = app.stream("orders", record=Order, partition_by="order_id")
@app.processor(orders)
async def printer(events):
async for order in events.records():
print(order.amount)
Meanwhile, run the worker (assuming code in example.py and PYTHONPATH is set):
$ runnel worker example:app
And send some events:
$ runnel send example:orders "{\"order_id\": 1, \"created_at\": \"2020-07-21T22:09:37Z\" , \"amount\": 99}"
Features¶
Runnel was designed to support a similar paradigm to Kafka Streams, but on top of Redis.
At least once processing semantics
Automatic partitioning of events by key
Each partition maintains strict ordering
Dynamic rebalance algorithm distributes partitions among workers on-the-fly
Support for nested Record types with custom serialisation and compression
Background tasks, including timers and cron-style scheduling
User-defined middleware for exception handling, e.g. dead-letter-queueing
A builtin batching mechanism to efficiently process events in bulk
A
runnel[fast]bundle for C or Rust extension dependencies (uvloop, xxhash, orjson, lz4)