Skip to main content

About

Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP (custom binary specification) and HTTP (regular REST API) transport protocols. Currently, running as a single server, it allows creating streams, topics, partitions and segments, and send/receive messages to/from them. The messages are stored on disk as an append-only log, and are persisted between restarts.

The goal of the project is to make a distributed streaming platform (running as a cluster), which will be able to scale horizontally and handle millions of messages per second.

Features​

  • Highly performant, persistent append-only log for the message streaming
  • Very high throughput for both writes and reads
  • Low latency and predictable resource usage thanks to the Rust compiled language (no GC)
  • Users authentication and authorization with granular permissions
  • Support for multiple streams, topics and partitions
  • Support for multiple transport protocols (QUIC, TCP, HTTP)
  • Fully operational RESTful API which can be optionally enabled
  • Available client SDK in multiple languages
  • Works directly with the binary data (lack of enforced schema and serialization/deserialization)
  • Configurable server features (e.g. caching, segment size, data flush interval, transport protocols etc.)
  • Possibility of storing the consumer offsets on the server
  • Multiple ways of polling the messages:
    • By offset (using the indexes)
    • By timestamp (using the time indexes)
    • First/Last N messages
    • Next N messages for the specific consumer
  • Possibility of auto committing the offset (e.g. to achieve at-most-once delivery)
  • Consumer groups providing the message ordering and horizontal scaling across the connected clients
  • Message expiry with auto deletion based on the configurable retention policy
  • Additional features such as server side message deduplication
  • TLS support for all transport protocols (TCP, QUIC, HTTPS)
  • Optional server-side as well as client-side data encryption using AES-256-GCM
  • Optional metadata support in the form of message headers
  • Built-in CLI to manage the streaming server
  • Built-in benchmarking app to test the performance
  • Single binary deployment (no external dependencies)
  • Running as a single node (no cluster support yet)

Supported languages SDK​

CLI​

The brand new, rich, interactive CLI is being implemented under the cmd project, to provide the best developer experience. This will be a great addition to the Web UI, especially for all the developers who prefer using the console tools.

Web UI​

There's an ongoing effort to build the administrative web UI for the server, which will allow to manage the streams, topics, partitions, messages and so on. Check the Web UI repository

Docker​

You can find the Dockerfile and docker-compose in the root of the repository. To build and start the server, run: docker compose up. The official images can be found here, simply type docker pull iggyrs/iggy.

Additionally, you can run the client which is available in the running container, by executing: docker exec -it iggy-server /client.