Skip to main content

Migrations

Once you are content with the changes you've made to your data platform, it's time to deploy it.

With Datoria, deployment is a two-step process.

  • First, you run migrations, which will update the schemas and SQL of all the relations and routines in the data platform.
  • Then you run the jobs with the new logic

About migrations​

We took a lot of cues from tools such as Flyway and Liquibase in how software engineers typically handle database migrations.

Schema definitions you create in code are the authoritative source for the database structure. It's also completely declarative, meaning you don't have to write incremental migration scripts.

Migrations are roll-forward only, meaning you can't roll back a migration. If you need to revert a change, you need to create a new migration that undoes the change. Enforcing backwards-compatible changes makes this much less of a problem.

Safe changes by comparing to prod​

When you ask Datoria to plan a migration, it will compare the changes you've made to the current state of the data platform.

If the changes are backwards compatible, Datoria can apply them. If they are not, Datoria will stop and ask you to fix the changes.

You ask Datoria for the list of changes it will apply to production by running

datoria migrate --plan

This will do a real diff against the metadata in prod to determine if your changes are backwards compatible.

Example: Backwards compatible change​

If you have done a backwards compatible change it'll look something like this:

Shows what happens when changes are compatible

Example: Backwards incompatible change​

On the other hand, if you try to do a backwards incompatible change, Datoria will stop and ask you to fix it.

Shows what happens when changes are NOT compatible

Example: Forcing a backwards incompatible change​

Like the error message says, you can force the change

datoria migrate --plan --force
Shows what happens when changes are NOT compatible but you force the change

Actually running the migrations​

datoria migrate

This is what it can look like when running while forcing an incompatible change:

Shows what happens when you run migrations ith changes which are NOT compatible but you force the change