Dowser.Migration behaviour (Dowser.Client v0.5.0)

View Source

A migration changes the indices of a repo, the way an Ecto.Migration changes the tables of a database.

Migrations live in priv/dowser/migrations/ (see Dowser.Migrator for another place), one per file, named after the UTC time they were generated at — mix dowser.gen.migration create_events writes 20261104120000_create_events.exs. That timestamp is the migration's version.

defmodule MyApp.Search.Migrations.CreateEvents do
  use Dowser.Migration

  def change do
    create_index "events", %{
      settings: %{number_of_shards: 3, number_of_replicas: replicas()},
      mappings: %{
        dynamic: :strict,
        properties: %{
          client_id: %{type: :keyword},
          body: %{type: :text},
          comments: %{type: :nested, properties: %{body: %{type: :text}}}
        }
      }
    }
  end

  # No replicas on a development machine's single node.
  defp replicas do
    if Application.get_env(:my_app, :env) == :dev, do: 0, else: 1
  end
end

mix dowser.migrate runs the migrations not run yet, mix dowser.rollback reverts the last one.

Indices are aliases

The name a migration gives — "events" — is always an alias. The index behind it is named after the version that created it, events_20261104120000, and its mapping records the version that last changed it in _meta.dowser_version. A change the cluster can't make in place — a field removed, renamed or retyped, a static setting — copies the documents into a new index (events_<version>) and moves the alias onto it.

The separator between the name and the version is _ unless the index is created with another — create_index "events", body, separator: "-" gives events-20261104120000. It is the index's own: every later copy of it, and the index a rollback recreates, uses it too.

Operations

The bodies are what the engine takes — the closest form to the final JSON — with atom or string keys. A migration is frozen history: give literal maps, or functions defined in the migration itself, rather than the application's own modules, which keep changing after the migration is written.

Functions

Everything in a migration runs when the migration runs, not when it compiles: any value can come from a function, so a migration can differ from one cluster to another — locally and in production, from one repo or tenant to the next. The function can depend on the repo being migrated — repo/0 returns it — or on your configuration. Don't use Mix.env/0: it doesn't exist in a release; record the environment in your configuration instead (config :my_app, env: config_env()).

change, up and down

change/0 describes the migration forward, and is reverted by running its operations backwards, each one inverted. An operation that can't be inverted without more information — removing a field without its definition, a setting without from: — makes the rollback fail before anything is changed, with a Dowser.Migration.IrreversibleError.

Or define up/0 and down/0; they are used as they are.

Copying documents

When an index is copied, each operation changing documents — rename/2, remove/2, add/3 with a :default — becomes a step of one painless script that dowser ships: the field names and values travel as the script's parameters, never as script source. It walks each document's _source as stored: a field inside a list of objects is changed in every object, at any depth, a missing or null parent is skipped, and a field stored with a dotted name ("address.zip": ...) is found too. script/2 adds your own painless code between those steps.

Summary

Types

field()

A field: its name, or the path to it through object and nested fields ([:address, :zip]).

setting()

A setting, relative to index (:number_of_replicas, or [:analysis, :analyzer, :folded]); a leading :index is accepted.

Callbacks

change()

The migration forward; reverted by inverting its operations.

down()

The migration backward.

up()

The migration forward.

Functions

add(field, definition, opts \\ [])

Adds a field, given its mapping definition.

alter_index(name, list)

Changes the index behind the alias name with the operations in its block.

alter_index(name, opts, list)
create_index(name, body, opts \\ [])

Creates the index behind the alias name, from body — the body of the engine's create-index request (settings, mappings, aliases).

drop_index(name, body \\ nil)

Deletes the index behind the alias name.

modify(field, definition, opts \\ [])

Changes a field's mapping definition; the documents are copied, so the engine parses them again with the new one.

remove(field, definition \\ nil)

Removes a field; the documents are copied without it.

rename(field, to)

Renames a field, keeping it under the same parent; the documents are copied, their value moved to the new name.

repo()

The repo being migrated — the module, or {module, name} for a dynamic instance. Only available while a migration runs.

script(source, opts \\ [])

Runs painless source on each document while it is copied — for what the other operations don't do, a value converted, a field computed.

set(setting, value, opts \\ [])

Sets a setting to value — a map for a group of settings.

unset(setting, opts \\ [])

Resets a setting to the engine's default.

Types

field()

@type field() :: atom() | String.t() | [atom() | String.t()]

A field: its name, or the path to it through object and nested fields ([:address, :zip]).

setting()

@type setting() :: atom() | String.t() | [atom() | String.t()]

A setting, relative to index (:number_of_replicas, or [:analysis, :analyzer, :folded]); a leading :index is accepted.

Callbacks

change()

(optional) 
@callback change() :: term()

The migration forward; reverted by inverting its operations.

down()

(optional) 
@callback down() :: term()

The migration backward.

up()

(optional) 
@callback up() :: term()

The migration forward.

Functions

add(field, definition, opts \\ [])

@spec add(field(), map(), keyword()) :: :ok

Adds a field, given its mapping definition.

add :status, %{type: :keyword}
add [:address, :zip], %{type: :keyword}

Its parent must exist. Added to the live index, unless default: is given: then, the documents are copied and every object holding the field's parent gets default where the field is missing. Reverted by removing it.

alter_index(name, list)

(macro)

Changes the index behind the alias name with the operations in its block.

alter_index "events" do
  add :status, %{type: :keyword}, default: "active"
  rename [:comments, :body], :content
  remove :legacy, %{type: :long}
  set :number_of_replicas, 2, from: 1
end

from: names the version the index must be at — the migration that last changed it. Without it, the migrator works it out from the migrations it has run.

Operations the engine applies in place — a field added without a default, a dynamic setting — run on the live index. Any other — a field modified, renamed, removed, added with a default, a static setting, a script — needs the documents copied into a new index, which only happens when the migrator is told to (mix dowser.migrate --reindex).

alter_index(name, opts, list)

(macro)

create_index(name, body, opts \\ [])

@spec create_index(String.t() | atom(), map(), keyword()) :: :ok

Creates the index behind the alias name, from body — the body of the engine's create-index request (settings, mappings, aliases).

create_index "events", %{
  settings: %{number_of_shards: 3},
  mappings: %{dynamic: :strict, properties: %{title: %{type: :text}}}
}

The index is named <name><separator><version>; its mapping's _meta gets dowser_version. Reverted by drop_index/2.

Options:

  • :separator — between the name and the version (default "_"), e.g. "-" for events-20261104120000. It can't hold digits, uppercase letters, or a character index names don't allow. Every later copy of the index keeps it.

drop_index(name, body \\ nil)

@spec drop_index(String.t() | atom(), map() | nil) :: :ok

Deletes the index behind the alias name.

In change/0, give the index's body — as create_index/2 takes it — to make it reversible: a rollback creates the index again, empty.

modify(field, definition, opts \\ [])

@spec modify(field(), map(), keyword()) :: :ok

Changes a field's mapping definition; the documents are copied, so the engine parses them again with the new one.

modify :group_id, %{type: :keyword}, from: %{type: :keyword, ignore_above: 256}

from: is the definition it has: checked against the index's mapping before anything changes, and needed to revert it.

remove(field, definition \\ nil)

@spec remove(field(), map() | nil) :: :ok

Removes a field; the documents are copied without it.

remove :legacy, %{type: :long}

Its definition is checked against the index's mapping before anything changes, and makes it reversible — a rollback maps the field again, but can't bring its values back.

rename(field, to)

@spec rename(field(), atom() | String.t()) :: :ok

Renames a field, keeping it under the same parent; the documents are copied, their value moved to the new name.

rename :body, :content
rename [:comments, :body], :content

A document already holding a field under the new name fails the copy rather than lose one of the two values. Reverted by renaming it back.

repo()

@spec repo() :: Dowser.Repo.t()

The repo being migrated — the module, or {module, name} for a dynamic instance. Only available while a migration runs.

script(source, opts \\ [])

@spec script(String.t(), keyword()) :: :ok

Runs painless source on each document while it is copied — for what the other operations don't do, a value converted, a field computed.

script "ctx._source.amount = ctx._source.amount / 100.0",
  reverse: "ctx._source.amount = (long) (ctx._source.amount * 100)"

Options:

  • :params — the script's parameters, as params.<name>; shared by all the scripts of one copy.
  • :reverse — the painless source reverting it; without it, the script makes change/0 irreversible.

set(setting, value, opts \\ [])

@spec set(setting(), term(), keyword()) :: :ok

Sets a setting to value — a map for a group of settings.

set :number_of_replicas, 2, from: 1
set [:analysis, :analyzer, :folded], %{tokenizer: :standard, filter: [:asciifolding]}, from: nil

A dynamic setting changes on the live index; a static one needs the documents copied.

from: is the value it has — nil for none — checked against the index before anything changes, the migration aborting when they differ; and needed to revert it.

unset(setting, opts \\ [])

@spec unset(setting(), keyword()) :: :ok

Resets a setting to the engine's default.

unset :max_result_window, from: 50_000

from: is the value it has, as for set/3.