​
Workflow

The workflow for Honeydew AI helps data teams ensure the stability and improve the quality of AI responses, in view of user demands.

Data teams using Honeydew to build production-grade AI systems typically follow three key approaches:

  1. Metadata: Define AI-specific metadata to clarify intent and resolve ambiguities or inconsistencies in data.
  2. Evaluation: Regularly validate AI outputs by testing them against a curated set of reference questions.
  3. Monitoring: Continuously track new AI responses to gauge overall quality and identify specific modeling issues.

​
AI Domain

Honeydew AI operates on a dedicated domain, within a workspace and a branch.

The aim of the AI domain is to control the metrics and attributes that are approved for AI usage, as well as a business context.

You can create multiple AI domains within the same workspace, serving different categories of business questions.

​
Setting up a new business domain

Setting up any new business domain will include:

  1. Set up Semantic Layer Definitions: Map data sources to a semantic layer; create relevant metrics or entities; add them to an AI domain.
  2. Create AI Metadata: Descriptions, sample values and other hints.
  3. Test Business Questions: A set of business questions to evaluate the AI Analyst.

You can use the AI modeling helper at every step:

  1. Use it to suggest required metrics or attributes given a question.
  2. Ask it about places where additional metadata can be useful, or where metadata is ambiguous.
  3. Check its judging response on tested business questions.

​
Maintaining an active business domain

The more frequently a domain is used, the more critical it becomes to actively monitor its quality and refine the semantic layer as necessary.

Activities on an active domain will include:

  1. Monitor AI Responses to improve the semantic model when there are gaps.
  2. Add New Business Questions to the evaluation set to cover new business demands.
  3. Keep testing Known Business Questions as things change to avoid regression.

​
Metadata

Metadata helps AI to understand the context of your business. You can add AI-specific metadata to various semantic objects. This metadata provides AI with additional context to be able to accurately answer the user questions.

The most important metadata for an object is its name.

Always prefer descriptive names for objects (entities, attributes, metrics) over explanations about what they do.

AI metadata serves to amplify the context for a semantic object that is not obvious from its name.

​
Domain

Domain metadata is the place to teach AI about the context of your business domain as a whole. A typical explanation might include who you are, what you do, and limitations of your data to ground it in your business domain.

For example, “We are ACME corp., an export-import provider in the Northwest. This includes all our sales data for the last seven years for financial analysis. NOTE: Data is up to previous day only.”

Domain metadata can be changed in Honeydew Studio, or within the domain YAML structure:

...
metadata:
  - name: honeydew_ai
    metadata:
      - name: description
        value: This is a domain-level AI-specific context for the domain

​
Attributes and Metrics

You can add AI-specific metadata to attributes and metrics that specifies the AI context for these fields. When an AI-specific description is not provided, the general field description is used as a fallback.

Typical hints can include:

  • Synonyms: “also known as GMV”
  • Data structure: “US state, abbreviated to two letters as NY”
  • Usage rules: “Use case-insensitive ILIKE to filter”
  • Other hints for when to prefer a field: “Key indicator for performance”

Field metadata can be changed in Honeydew Studio, or within the metric or attribute YAML structure:

...
metadata:
  - name: honeydew_ai
    metadata:
      - name: description
        value: This is an AI-specific description for the attribute
      - name: values
        value:
          - value1
          - value2
          - value3

​
Entities

You can add AI-specific metadata to entities that specifies the AI context for these entities as a whole. When an AI-specific description is not provided, the general field description is used as a fallback.

For example, “Only includes customers with an active subscription” for a customers entity.

Entity metadata can be changed in Honeydew Studio, or within the entity YAML structure:

...
metadata:
  - name: honeydew_ai
    metadata:
      - name: description
        value: This is an AI-specific description for the entity

​
Evaluation

An evaluation set is a group of known questions that can be used to evaluate the performance of AI.

Sets can be of any size or structure, and can be set to run periodically or after schema changes.

​
Running evaluations

You can use the Honeydew Native Application for Snowflake to automate the AI evaluation workflow.

  1. Create a table containing the business questions you want to answer using AI. The table should have a column for the question text.

  2. Run the evaluation flow using the Honeydew Native Application.

     WITH response AS (
     SELECT
         *,
         SEMANTIC_LAYER_ENTERPRISE_EDITION.API.ASK_QUESTION(
             workspace => $WORKSPACE,    -- Replace with your workspace name
             branch => $BRANCH,          -- Replace with your branch name
             question => question,
             domain => $AI_DOMAIN,       -- Replace with your AI domain name
             include_judge => TRUE
         ) AS api_response
     FROM ${YOUR_QUESTIONS_TABLE}        -- Replace with your table containing questions
 )
 SELECT
     question,
     api_response,
     api_response::object:"judge"::object:"is_correct"::string AS judge_verdict,
     api_response::object:"judge"::object:"explanation"::string AS judge_explanation,
     api_response::object:"llm_response_json"::string AS llm_response_json,
     api_response::object:"llm_response"::string AS llm_response,
     api_response::object:"perspective"::string AS perspective,
     api_response::object:"error"::string AS error_in_response
 FROM response;

  3. Review the results of the evaluation, including the judge verdict and explanation. The judge_verdict column will indicate whether the AI response is correct or not. The judge_explanation column will provide an explanation for the verdict.

The Honeydew AI Judge is an agent tasked with evaluating answers given by AI to business users for completeness, relevance, and overall correctness