Usage
Given a dynamic dataset namedmy_dataset_name
create a dbt model called my_dataset_name.sql
.
Put the following in the model code:
Honeydew dbt Package
Honeydew provides a dbt package to integrate Honeydew with dbt.High-Level Flow
dbt connects to Honeydew via a Snowflake connection and uses the Honeydew Snowflake Native Application to communicate with Honeydew.
- dbt connects to Snowflake with the current profile.
-
Within a dbt model, the Honeydew dbt macro initiates a Snowflake call to Honeydew API.
The macro uses the workspace and branch based on the Honeydew variable values set up in the dbt project
-
The Honeydew API client in Snowflake communicates with Honeydew to receive the correct compiled SQL for that branch.
The Snowflake user and role set up in the dbt profile will be used to access Honeydew.Make sure that the role you use for dbt has access to the Honeydew native application.
A role can be added withSee installation guide for more details. -
The compiled result of the macro call is the SQL for a Honeydew-based dbt model.
The compiled SQL for a model based on Honeydew is generated for each dbt run, keeping it up to date with the semantic model definitions.
Installation
Honeydew runs in dbt projects connected to Snowflake. To interact with Honeydew, dbt can call the Honeydew Native Application. Install the Honeydew dbt package by adding topackages.yml
of your dbt project:
dbt deps
to apply.
Set the following variables in the dbt project configuration (dbt_project.yml
):
Name | Meaning |
---|---|
honeydew:workspace | Workspace in which to operate. |
honeydew:branch | Branch in which to operate. Default=prod |
honeydew:snowflake_native_app_name | Snowflake native app installed name. Default=SEMANTIC_LAYER_ENTERPRISE_EDITION |
dbt_project.yml
configuration:
Here is an example repository for dbt integration with Honeydew.
Testing the integration
Communication details between dbt and Honeydew can be tested with the following call:snowflake
profile with your dbt profile and my_dataset_name
with the dataset name to test.
Extending the integration
Any Honeydew API operation can be added in the following way:Metadata Sync
dbt Metadata Sync is coming soon.
- Source dbt model references (when source dbt model is set for entities)
- Column descriptions
- Labels (as tags)
- General purpose metadata (if set)
Honeydew and different dbt offerings
dbt Core
Honeydew provides a dbt package that integrates with dbt. Using that package, Honeydew can manage dbt models in dbt Core. Running dbt is based on an external orchestrator such as Airflow.dbt Cloud
Works with Honeydew in the same way as dbt Core. There is no need for an external orchestrator. Additional integration is available to link dbt documentation built on a dbt Cloud job and the Honeydew UI .dbt Semantic Layer
There is no necessity to use the dbt Semantic Layer when using Honeydew. Honeydew metrics can connect to any BI tool directly. Both can be used in parallel. There are a number of core differences, such as:- Composable complex metrics: Honeydew enables a high degree of metric composition, including applying filters, partial aggregations, fan-out joins, multi-hop joins, multiple aggregation steps, time-based logic, and more.
- Context-aware metrics: Honeydew metrics can adjust the user grouping, for example for dynamic ratios or dynamic nested aggregation.
- Reuse metrics in data marts: Honeydew metrics can be directly used to build columns in other tables.
- BI connectivity and Metadata Sync: Honeydew has standard SQL and metadata integrations that support a live connection with most major BI tools.
- Security Model: Honeydew is designed around a user-based security model supporting row-level security and SSO.
- Performance: Honeydew has a number of semantic-aware performance optimization such as aggregate awareness.
FAQ
I have a complex data model, that does not easily fit into Honeydew modeling schema. How can I still use Honeydew with it?
I have a complex data model, that does not easily fit into Honeydew modeling schema. How can I still use Honeydew with it?
Honeydew enables complex modeling using a number of advanced modeling techniques:
- Virtual entities for a derived level of granularity
- Modeling a many-to-many relation through a connecting entity
- Basing entities on top of SQL queries or Views in Snowflake
With dbt, I have full column lineage of the data, from the raw tables and up to the final datasets. With Honeydew, I lose this visibility.
With dbt, I have full column lineage of the data, from the raw tables and up to the final datasets. With Honeydew, I lose this visibility.
Honeydew provides a full component-level lineage within the Honeydew platform, as well as dbt lineage integration.
Can Honeydew produce dbt Semantic Layer models?
Can Honeydew produce dbt Semantic Layer models?
Honeydew does not produce dbt Semantic Layer models. The reason is that the dbt Semantic Layer models are more limited and do not support all the semantics that Honeydew supports.
We already have a dbt project with a number of models. How do we move our analytical workload to Honeydew?
We already have a dbt project with a number of models. How do we move our analytical workload to Honeydew?
Honeydew has internal automated tools for migrating logic from dbt to Honeydew. Honeydew team is happy to assist customers with this initial migration. This can be an iterative process - choosing one analytical domain at a time, and then re-modeling the logic kept in dbt models into Honeydew. The usual ratio between dbt models and Honeydew entities is 3:1, so at the end of this process there would be much less entities to maintain.