Maintenance & topology

Maintenance

MyApp.Shards.maintenance(:shard_2, :drain)                 # stop routing, keep pools
MyApp.Shards.maintenance(:shard_2, :stop)                  # stop routing, then stop pools
MyApp.Shards.maintenance(:shard_1, :drain, role: :replica) # one role only
MyApp.Shards.activate(:shard_2)                            # start pools, then route
MyApp.Shards.status(:shard_2)
#=> {:ok, %{status: :active, roles: %{primary: :active}}}
MyApp.Shards.shards()                                      # includes meta.pids

Calls are idempotent. Taking a shard out publishes the new status before stopping pools; activating starts pools before publishing. Calls already in flight are not tracked: :stop relies on the pool's own shutdown.

Adding and removing shards

MyApp.Shards.add_shard(:shard_9, primary: {Shardex.Adapter.Ecto, repo: MyApp.Repo, config: [database: "s9"]})
# warm up first: pools start, but the shard is not routed until activate/1
MyApp.Shards.add_shard(:shard_10, {Shardex.Adapter.Ecto, repo: MyApp.Repo, config: [database: "s10"]},
  status: :drain
)
MyApp.Shards.remove_shard(:shard_9)

With Hash, adding or removing a shard remaps most keys (a warning is logged). With JumpHash, appending moves ~1/n keys to the new shard. With a directory strategy, nothing moves until you assign keys to the new shard.

Persistence

Runtime changes live in memory. If the instance restarts, the boot config wins. Keep your source of truth (database, config service) and re-apply it at boot and when it changes:

for %{shard: shard, mode: mode} <- MyApp.Metadata.shards_in_maintenance() do
  MyApp.Shards.maintenance(shard, mode)
end

The Coordinator can also restart on its own after a crash. It then rebuilds from the boot config: drained or stopped shards come back :active and shards added or removed at runtime are reverted. Shardex emits [:shardex, :coordinator, :init] after every successful boot, so attach a handler that re-applies your source of truth. The handler runs inside the Coordinator while it boots, so do the work in another process:

defmodule MyApp.ShardexReapply do
  def attach do
    :telemetry.attach("myapp-shardex-reapply", [:shardex, :coordinator, :init], &__MODULE__.handle/4, nil)
  end

  def handle(_event, _measurements, %{instance: MyApp.Shards}, _config) do
    Task.Supervisor.start_child(MyApp.TaskSupervisor, fn ->
      for %{shard: shard, mode: mode} <- MyApp.Metadata.shards_in_maintenance() do
        MyApp.Shards.maintenance(shard, mode)
      end
    end)
  end

  def handle(_event, _measurements, _metadata, _config), do: :ok
end

Call MyApp.ShardexReapply.attach() before the instance starts (e.g. in Application.start/2) so it also covers the initial boot.

A pool that keeps failing to start also affects restarts: if the Coordinator crashes and cannot rebuild (with start_failure: :raise, any boot-config pool failing to start stops it), the instance supervisor keeps restarting it until its restart intensity is exhausted, and then the whole instance goes down.