dbt Integration

Honeydew dbt integration allows to materialize Honeydew-managed datasets via dbt.

Usage

Dynamic Datasets in dbt

Given a dynamic dataset named my_dataset_name create a dbt model called my_dataset_name.sql.

Put the following in the model code:

-- Optional: Set up various materialization parameters
{{ config(materialized='table') }}

-- Set up any additional dependencies in dbt with
-- depends_on: {{ ref('upstream_parent_model') }}

-- Dynamic dataset materialization
{{ honeydew.get_dataset_sql('my_dataset_name') }}

Setup

Installation

Honeydew runs in dbt projects connected to Snowflake. To interact with Honeydew, dbt can call the Honeydew Native Application.

Install the Honeydew dbt package by adding to packages.yml of your dbt project:

packages:
  - git: "https://github.com/honeydew-ai/honeydew-dbt.git"
    revision: "latest"

Run dbt deps to apply.

Set the following variables in the dbt project configuration (dbt_project.yml):

NameMeaning
honeydew:workspaceWorkspace in which to operate.
honeydew:branchBranch in which to operate. Default=prod
honeydew:snowflake_native_app_nameSnowflake native app installed name. Default=SEMANTIC_LAYER_ENTERPRISE_EDITION

An example dbt_project.yml configuration:

vars:
  "honeydew:workspace": "tpch_demo"
  "honeydew:branch": "prod"
  "honeydew:snowflake_native_app_name": "SEMANTIC_LAYER_ENTERPRISE_EDITION"

Metadata Sync

dbt Metadata Sync is coming soon.

Honeydew dbt metadata sync supports syncing with dbt the following metadata:

  • Source dbt model references (when source dbt model is set for entities)
  • Column descriptions
  • Labels (as tags)
  • General purpose metadata (if set)

Honeydew and different dbt offerings

dbt Core

Honeydew provides a dbt package that integrates with dbt. Using that package, Honeydew can manage dbt models in dbt Core.

Running dbt is based on an external orchestrator such as Airflow.

dbt Cloud

Works in the same way as dbt Core without limitations. There is no need for an external orchestrator. Additional integration is available to link dbt documentation built on a dbt Cloud job and the Honeydew UI .

dbt Semantic Layer

There is no need to use the dbt Semantic Layer when using Honeydew. Honeydew metrics can connect to any BI tool directly. There are a number of core differences:

  1. Composable metrics: Honeydew enables a high degree of metric composition, including applying filters, partial aggregations, time-based logic, etc.
  2. Multi-Join metrics: Honeydew metrics can include multi-hop joins.
  3. Level of detail metrics: Honeydew metrics can include partial level of details.
  4. Using metrics as filters: Honeydew metrics can be used to filter data.
  5. Use metrics in data marts: Honeydew metrics can be used directly to build columns in other tables.
  6. BI connectivity and Metadata Sync: Honeydew has a standard integration that works with all major BI tools.

FAQ

Honeydew enables complex modeling using a number of advanced modeling techniques:

  1. Virtual entities for a derived level of granularity
  2. Modeling a many-to-many relation through a connecting entity
  3. Role-playing
  4. Basing entities on top of SQL queries or Views in Snowflake

While these are possible with bespoke SQL code, building those in Honeydew provides a higher level of abstraction for the data model, and allows incremental expansion of the data model, without duplicating code and logic.

Honeydew provides a full component-level lineage within the Honeydew platform, as well as dbt lineage integration.

Honeydew does not produce dbt Semantic Layer models. The reason is that the dbt Semantic Layer models are more limited and do not support all the semantics that Honeydew supports.

Honeydew has internal automated tools for migrating logic from dbt to Honeydew. Honeydew team is happy to assist customers with this initial migration. This can be an iterative process - choosing one analytical domain at a time, and then re-modeling the logic kept in dbt models into Honeydew. The usual ratio between dbt models and Honeydew entities is 3:1, so at the end of this process there would be much less entities to maintain.