Running Datoria
This section covers the operational aspects of Datoria - how to run your data platform from development through testing to production deployment. We'll walk through the workflow stages that take your code and turn it into a running data pipeline.
Development Workflow Overview
Building and maintaining a data platform with Datoria follows a structured workflow:
1. Define Your Data Platform
Start by defining your tables, transformations, and dependencies in code. The Virtual Data Platform validates your definitions in real-time, catching issues before execution.
2. Test with Unit Tests
Validate your transformations with unit tests that run against the real database engine. These tests provide confidence that your code behaves as expected.
3. Run in Sandboxes
Test your transformations in isolated environments that don't affect shared resources. Sandboxes allow you to work with real data while keeping your changes contained.
4. Deploy to Dev/Stage
Once you're confident in your changes, deploy them to a shared development or staging environment for broader testing and stakeholder review.
5. Migrate to Production
Apply your changes to production schemas with Datoria's migration system, which ensures backward compatibility and prevents breaking changes.
6. Run in Production
Execute your transformations in production, leveraging Datoria's partitioning and invalidation system to process only what needs processing.
7. Compare Changes
View the differences between versions of your data platform using Datoria's diffing capabilities.
Tools for Each Stage
Datoria provides specific tools for each stage of the workflow:
| Stage | Primary Tool | Key Capabilities |
|---|---|---|
| Development | VS Code + Datoria LSP | Real-time validation, SQL assistance |
| Unit Testing | UnitTest framework | Test transformations without affecting the database |
| Sandbox Testing | datoria sandbox | Run in isolated environments |
| Dev/Stage | datoria migrate/run --dataset | Deploy to shared environments |
| Migration | datoria migrate | Apply schema changes safely |
| Production | datoria run | Execute transformations efficiently |