Shardex

Copy Markdown View Source

Route keys — and batches of items — to sharded pools: Ecto repos, Redis, Elasticsearch, or anything with a child spec.

  • Backend-agnostic adapters: Shardex.Adapter.Ecto, Shardex.Adapter.Redix, Shardex.Adapter.Generic (Elasticsearch, Finch, ...).
  • Batch-first routing: group thousands of items by shard with one strategy call.
  • Roles per shard (:primary, :replica, ...) with fallback.
  • Pluggable strategies: hash (default), jump consistent hash, static maps, or your own (e.g. a database directory), with an optional cache.
  • Maintenance (:drain / :stop) and runtime add_shard / remove_shard.
  • Telemetry for routing, runs, batches and status changes.

Installation

def deps do
  [
    {:shardex, "~> 0.1"},
    # optional, depending on your pools:
    {:ecto_sql, "~> 3.12"},
    {:redix, "~> 1.5"}
  ]
end

Quick start

defmodule MyApp.Shards do
  use Shardex, otp_app: :my_app
end

# config/runtime.exs
config :my_app, MyApp.Shards,
  strategy: Shardex.Strategy.JumpHash,
  shards: [
    shard_1: [
      primary: {Shardex.Adapter.Ecto, repo: MyApp.Repo, config: [database: "app_s1"]},
      replica: {Shardex.Adapter.Ecto, repo: MyApp.Repo, config: [database: "app_s1", hostname: "replica-1"]}
    ],
    shard_2: {Shardex.Adapter.Ecto, repo: MyApp.Repo, config: [database: "app_s2"]}
  ]

# application.ex
children = [MyApp.Shards]
# One key
{:ok, contacts} = MyApp.Shards.run(org_id, fn repo -> repo.all(Contact) end)
MyApp.Shards.run(org_id, &read_report/1, role: :replica, fallback: :primary)

# Many items, one routing pass, one call per shard
{:ok, results, errors} =
  MyApp.Shards.run_batch(messages, & &1.org_id, fn repo, msgs ->
    repo.insert_all(Event, Enum.map(msgs, &to_row/1))
  end)

# Maintenance
:ok = MyApp.Shards.maintenance(:shard_2, :drain)
:ok = MyApp.Shards.activate(:shard_2)

Guides

License

MIT