The workflow for Honeydew AI helps data teams ensure the stability and improve the quality of AI responses, in view of user demands.Data teams using Honeydew to build production-grade AI systems typically follow three key approaches:
Metadata: Define AI-specific metadata to clarify intent and resolve ambiguities or inconsistencies in data.
Evaluation: Regularly validate AI outputs by testing them against a curated set of reference questions.
Monitoring: Continuously track new AI responses to gauge overall quality and identify specific modeling issues.
Honeydew AI operates on a dedicated domain, within a workspace and a branch.The aim of the AI domain is to control the metrics and attributes that are approved for AI usage, as well as a business context.
You can create multiple AI domains within the same workspace, serving different categories of business questions.
The more frequently a domain is used, the more critical it becomes to actively monitor its quality and refine the semantic layer as necessary.Activities on an active domain will include:
Monitor AI Responses to improve the semantic model when there are gaps.
Add New Business Questions to the evaluation set to cover new business demands.
Keep testing Known Business Questions as things change to avoid regression.
Metadata helps AI to understand the context of your business. You can add AI-specific metadata to various semantic objects.
This metadata provides AI with additional context to be able to accurately answer the user questions.
The most important metadata for an object is its name.Always prefer descriptive names for objects (entities, attributes, metrics) over explanations about what they do.AI metadata serves to amplify the context for a semantic object that is not obvious from its name.
Domain metadata is the place to teach AI about the context of your business domain as a whole. A typical explanation
might include who you are, what you do, and limitations of your data to ground it in your business domain.
For example, “We are ACME corp., an export-import provider in the Northwest. This includes all our sales data
for the last seven years for financial analysis. NOTE: Data is up to previous day only.”
Domain metadata can be changed in Honeydew Studio, or within the domain YAML structure:
Copy
Ask AI
...metadata: - name: honeydew_ai metadata: - name: description value: This is a domain-level AI-specific context for the domain
You can add AI-specific metadata to attributes and metrics that specifies the AI context
for these fields. When an AI-specific description is not provided, the general field description is used as a fallback.
Typical hints can include:
Synonyms: “also known as GMV”
Data structure: “US state, abbreviated to two letters as NY”
Usage rules: “Use case-insensitive ILIKE to filter”
Other hints for when to prefer a field: “Key indicator for performance”
Field metadata can be changed in Honeydew Studio, or within the metric or attribute YAML structure:
Copy
Ask AI
...metadata: - name: honeydew_ai metadata: - name: description value: This is an AI-specific description for the attribute - name: values value: - value1 - value2 - value3
You can add AI-specific metadata to entities that specifies the AI context
for these entities as a whole. When an AI-specific description is not provided, the general field description is used as a fallback.
For example, “Only includes customers with an active subscription” for a customers entity.
Entity metadata can be changed in Honeydew Studio, or within the entity YAML structure:
Copy
Ask AI
...metadata: - name: honeydew_ai metadata: - name: description value: This is an AI-specific description for the entity
An evaluation set is a group of known questions that can be used to evaluate the performance of AI.Sets can be of any size or structure, and can be set to run periodically or after schema changes.
Create a table containing the business questions you want to answer using AI.
The table should have a column for the question text.
Run the evaluation flow using the Honeydew Native Application.
Copy
Ask AI
WITH response AS ( SELECT *, SEMANTIC_LAYER_ENTERPRISE_EDITION.API.ASK_QUESTION( workspace => $WORKSPACE, -- Replace with your workspace name branch => $BRANCH, -- Replace with your branch name question => question, domain => $AI_DOMAIN, -- Replace with your AI domain name include_judge => TRUE ) AS api_response FROM ${YOUR_QUESTIONS_TABLE} -- Replace with your table containing questions ) SELECT question, api_response, api_response::object:"judge"::object:"is_correct"::string AS judge_verdict, api_response::object:"judge"::object:"explanation"::string AS judge_explanation, api_response::object:"llm_response_json"::string AS llm_response_json, api_response::object:"llm_response"::string AS llm_response, api_response::object:"perspective"::string AS perspective, api_response::object:"error"::string AS error_in_response FROM response;
Review the results of the evaluation, including the judge verdict and explanation.
The judge_verdict column will indicate whether the AI response is correct or not.
The judge_explanation column will provide an explanation for the verdict.
The Honeydew AI Judge is an agent tasked with evaluating answers given by AI to business users
for completeness, relevance, and overall correctness