Skip to main content

Virtual Data Platform (VDP) Architecture

You can read about practical applications for VDP in core concepts

Technical Properties​

Comprehensive Modeling​

The VDP models all aspects of your data platform, including:

  • Partitioning strategies (day, month, integer, etc.)
  • Schema definitions with detailed field information
  • Transformation queries with their inputs and outputs
  • Routine definitions with parameters and return types
  • Custom metadata for additional context

Immutability​

Each version of the VDP is immutable - when changes are made, a new version is created rather than modifying the existing one. This provides:

  • A complete record of previous states
  • Safe rewrites of references for different environments
  • Clear tracking of what has changed between versions

Digest-Based Change Detection​

Each component in the VDP has a digest (hash) computed from its definition, enabling:

  • Identification of which components have changed
  • Selective revalidation of only affected parts
  • Precise detection of conflicts (e.g., duplicate names with different content)

Source-Location Awareness​

The VDP can track the source code location of each component, powering features like:

  • Hover documentation showing a table's schema in your IDE
  • Error messages tied to specific lines of code
  • Navigation from components to their source definitions

Open-Ended Metadata​

Every component in the VDP can have custom metadata attached as arbitrary JSON, allowing for:

  • Team ownership or cost-center information
  • Security classifications (e.g., PII markers)
  • Integration points like webhook URIs
  • Any custom information relevant to your organization

Serialization​

The entire VDP can be serialized to formats like JSON, enabling:

  • Storage in version control
  • Comparison of different versions
  • Sharing between environments