Skip to main content

Challenge 38: Custom Question Answering

Estimated Time

50 min | Cost: $2-5 (estimated) | Domain: Implement NLP Solutions (15-20%)

Exam skills covered​

  • Create a custom question answering project
  • Add QA pairs manually and import from sources
  • Create multi-turn conversations
  • Add alternate phrasings and chit-chat
  • Train, test, and publish the knowledge base

Overview​

Custom Question Answering (replacement for QnA Maker) builds knowledge bases that answer questions from your content:

FeatureDescription
QA pairsQuestion-answer pairs (manually added or auto-extracted)
SourcesImport from URLs, files (PDF, DOCX, TSV), or SharePoint
Multi-turnFollow-up prompts creating dialogue trees
Alternate questionsMultiple phrasings mapping to same answer
Chit-chatPre-built personality responses (professional, friendly, etc.)
Precise answeringExtract exact answer span from long answers

The service is part of Azure AI Language: https://{endpoint}.cognitiveservices.azure.com/language/

Prerequisites​

  • Azure subscription
  • Azure AI Language resource with Custom Question Answering enabled
  • Azure AI Search resource (required for indexing)
  • Python 3.9+ with azure-ai-language-questionanswering

Implementation​

Task 1: Create Resources​

az group create --name rg-ai102-qna --location eastus2

# Create Azure AI Search (required for QA indexing)
az search service create \
  --name search-ai102-qna \
  --resource-group rg-ai102-qna \
  --sku basic \
  --location eastus2

# Create Language resource with custom QA feature
az cognitiveservices account create \
  --name language-ai102-qna \
  --resource-group rg-ai102-qna \
  --kind TextAnalytics \
  --sku S \
  --location eastus2 \
  --custom-domain language-ai102-qna

Task 2: Create QA Project and Add Pairs​

import os
import requests
import time

endpoint = os.environ["AZURE_AI_ENDPOINT"]
key = os.environ["AZURE_AI_KEY"]
project_name = "faq-knowledge-base"

headers = {
    "Ocp-Apim-Subscription-Key": key,
    "Content-Type": "application/json"
}

# Create QA project
create_url = f"{endpoint}/language/authoring/query-knowledgebases/projects/{project_name}?api-version=2023-04-01"

project_body = {
    "description": "Product FAQ knowledge base",
    "language": "en",
    "multilingualResource": False,
    "settings": {
        "defaultAnswer": "I'm sorry, I don't have information about that. Please contact support."
    }
}

response = requests.patch(create_url, headers=headers, json=project_body)
print(f"Project created: {response.status_code}")

# Add QA pairs with sources
update_url = f"{endpoint}/language/authoring/query-knowledgebases/projects/{project_name}/sources?api-version=2023-04-01"

# Add manual QA pairs
qna_pairs = [
    {
        "op": "add",
        "value": {
            "id": 1,
            "answer": "Azure AI Services (formerly Cognitive Services) is a collection of cloud-based AI APIs that help developers build intelligent applications without direct AI expertise. It includes Vision, Language, Speech, Decision, and OpenAI services.",
            "source": "manual",
            "questions": [
                "What is Azure AI Services?",
                "What are Cognitive Services?",
                "Tell me about Azure AI",
                "What AI services does Azure offer?"
            ],
            "metadata": {"category": "overview"},
            "dialog": {
                "isContextOnly": False,
                "prompts": [
                    {
                        "displayOrder": 1,
                        "displayText": "What services are included?",
                        "qnaId": 2
                    },
                    {
                        "displayOrder": 2,
                        "displayText": "How much does it cost?",
                        "qnaId": 3
                    }
                ]
            }
        }
    },
    {
        "op": "add",
        "value": {
            "id": 2,
            "answer": "Azure AI Services includes:\n- **Vision**: Image analysis, face detection, OCR\n- **Language**: Text analytics, translation, CLU\n- **Speech**: Speech-to-text, text-to-speech, translation\n- **Decision**: Content moderator, anomaly detector\n- **OpenAI**: GPT-4, DALL-E, embeddings",
            "source": "manual",
            "questions": [
                "What services are included?",
                "List Azure AI services",
                "What are the categories of AI services?"
            ],
            "metadata": {"category": "services"}
        }
    },
    {
        "op": "add",
        "value": {
            "id": 3,
            "answer": "Azure AI Services uses a pay-as-you-go pricing model. Most services offer a free tier:\n- Free tier: Limited transactions/month\n- Standard (S0): Pay per transaction\n- Commitment tiers: Discounted rates for high volume\n\nVisit https://azure.microsoft.com/pricing/details/cognitive-services/ for current prices.",
            "source": "manual",
            "questions": [
                "How much does it cost?",
                "What is the pricing?",
                "Is there a free tier?",
                "Azure AI pricing plans"
            ],
            "metadata": {"category": "pricing"}
        }
    }
]

# Submit QA pairs
update_sources_url = f"{endpoint}/language/authoring/query-knowledgebases/projects/{project_name}/qnas?api-version=2023-04-01"
response = requests.patch(update_sources_url, headers=headers, json=qna_pairs)
operation_url = response.headers.get("operation-location")
print(f"QA pairs submitted: {response.status_code}")

# Wait for completion
while True:
    status_resp = requests.get(operation_url, headers=headers)
    status = status_resp.json().get("status", "unknown")
    print(f"  Status: {status}")
    if status in ["succeeded", "failed"]:
        break
    time.sleep(2)

Task 3: Import from URL Source​

# Import QA pairs from a web page
sources_url = f"{endpoint}/language/authoring/query-knowledgebases/projects/{project_name}/sources?api-version=2023-04-01"

source_body = [
    {
        "op": "add",
        "value": {
            "displayName": "Azure AI FAQ",
            "sourceUri": "https://learn.microsoft.com/en-us/azure/ai-services/what-are-ai-services",
            "sourceKind": "url"
        }
    }
]

response = requests.patch(sources_url, headers=headers, json=source_body)
print(f"URL source added: {response.status_code}")
# Poll operation-location for completion

Task 4: Deploy and Query​

from azure.ai.language.questionanswering import QuestionAnsweringClient
from azure.core.credentials import AzureKeyCredential

# Deploy the project
deploy_url = f"{endpoint}/language/authoring/query-knowledgebases/projects/{project_name}/deployments/production?api-version=2023-04-01"
response = requests.put(deploy_url, headers=headers)
print(f"Deployment started: {response.status_code}")

# Wait for deployment
operation_url = response.headers.get("operation-location")
if operation_url:
    while True:
        status_resp = requests.get(operation_url, headers=headers)
        status = status_resp.json().get("status", "unknown")
        if status in ["succeeded", "failed"]:
            print(f"Deployed: {status}")
            break
        time.sleep(3)

# Query the knowledge base using SDK
qa_client = QuestionAnsweringClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(key)
)

# Ask questions
test_questions = [
    "What is Azure AI?",
    "How much does it cost?",
    "What vision services are available?"
]

for question in test_questions:
    response = qa_client.get_answers(
        question=question,
        project_name=project_name,
        deployment_name="production",
        confidence_threshold=0.3,
        top=3,
        include_unstructured_sources=True
    )
    
    print(f"\nQ: {question}")
    for answer in response.answers:
        print(f"  A: {answer.answer[:100]}...")
        print(f"     Confidence: {answer.confidence:.4f}")
        if answer.dialog and answer.dialog.prompts:
            print(f"     Follow-ups: {[p.display_text for p in answer.dialog.prompts]}")

Expected Output​

Project created: 201
QA pairs submitted: 202
  Status: running
  Status: succeeded
URL source added: 202
Deployment started: 200
Deployed: succeeded

Q: What is Azure AI?
  A: Azure AI Services (formerly Cognitive Services) is a collection of cloud-based AI APIs that hel...
     Confidence: 0.9534
     Follow-ups: ['What services are included?', 'How much does it cost?']

Q: How much does it cost?
  A: Azure AI Services uses a pay-as-you-go pricing model. Most services offer a free tier:
- Free...
     Confidence: 0.9123

Q: What vision services are available?
  A: Azure AI Services includes:
- **Vision**: Image analysis, face detection, OCR...
     Confidence: 0.8234

Break & fix​

ScenarioSymptomRoot CauseFix
No answers returnedEmpty or default answerConfidence below thresholdLower confidenceScoreThreshold or add more alternate phrasings
URL import failsSource shows errorsPage not accessible or poorly structuredVerify URL is public; use well-structured FAQ pages
Multi-turn not workingFollow-up prompts missingDialog prompts not configured on QA pairAdd dialog.prompts with correct qnaId references
Duplicate answersSame answer repeatedSimilar QA pairs with overlapping questionsMerge duplicate pairs; use alternate questions on single pair
Deployment fails400 errorNo training data or invalid pairsEnsure at least one QA pair exists; validate all pair IDs

Knowledge Check​

1. What Azure resource is required in addition to the Language resource for Custom Question Answering?

2. How do you create multi-turn conversations in Custom QA?

3. What is the purpose of alternate questions on a QA pair?

4. What happens when no answer meets the confidence threshold?

5. What content sources can Custom Question Answering import from?

Cleanup​

az group delete --name rg-ai102-qna --yes --no-wait

Learn More​