Data Pipeline Architecture

​

Honeydew Managed Pipelines

Honeydew is a repository of shared business logic such as standard metrics and entity definitions.

Honeydew integrates with dbt to allow an end-to-end analytics pipeline.

​

Pipeline Stages

A typical data pipeline of Honeydew with Snowflake and dbt consists of the following stages:

1. Ingestion: Bringing raw data into Snowflake. Usually, with a tool such as Fivetran or similar.

2. Staging: An incremental periodic process that transforms raw data into analytic-ready granular data. The following transformations are typically performed at this stage:

   1. Data cleanups, normalization, deduplication
   2. Data merging, change data capture deconstruction
   3. Dimensional modeling, incremental updates

   Implemented using dbt or other transformation tools. Executed by an orchestrator (dbt Cloud, Airflow, etc.) based on a period (i.e. hourly) or change triggers.

3. Entity Layer (Data Mart): A periodic process that transforms granular data into a domain data mart with business entities, metrics and aggregated datasets.

   The following transformations are typically performed at this stage:

   1. Business Entities: Extend granular data with logic properties, such as
      1. Calculated columns (build an “Age Group” category out of “date of birth”)
      2. Aggregated properties (first order data, total revenue per customer, etc.)
   2. Metric Datasets: Build datasets such as:
      1. Aggregated tables (“Monthly KPIs”)
      2. Denormalized fact tables

   Both are implemented using Honeydew metrics and can be materialized in Snowflake by Honeydew.

   For materialization, Honeydew can leverage your dbt pipeline by creating Honeydew-managed dbt models, in the same dbt repository as your staging models.

   

4. Metric Layer: On-the fly transformation of user queries to JOINs and aggregations over entity tables using standard metrics like “Revenue” or “Active Customer Count”.

​

Example

1. Ingestion: raw customer tables and raw order events are loaded into Snowflake.
2. Staging: dbt models that transform those into two tables:
   1. dim_customers: A customer dimension table, with a row per known customer. Dbt preparation includes clean-up and merging from raw data.
   2. fact_orders: An incremental model, with a row per unique order event. Dbt preparation includes handling backfilling and time stamp conversion.
3. Data Mart: Honeydew-managed models:
   1. Entity models, such as customers: A customer entity table. Includes additional properties:
      1. “Age Group” - a CASE WHEN that maps ages to categories
      2. “First Order Data” - an aggregation of order facts that finds first order
      3. “Lifetime Revenue” - the revenue standard metric by customer
   2. Dataset models, such as monthly_kpis: An aggregated table:
      1. “Order Month” - based on order timestamp
      2. “Monthly Revenue” - the revenue standard metric by order month
      3. “Monthly # Customers” - a count of customers active in a month