Pular para o conteúdo principal

Desafio 19: Otimizar Soluções de IA Generativa

Tempo Estimado

45-60 min | Custo: ~$5.00 (estimado, fine-tuning) | Domínio: Soluções de IA Generativa (15-20%)

Habilidades do exame cobertas​

  • Configurar parâmetros para otimizar a saída de IA generativa
  • Implementar monitoramento e observabilidade para soluções de IA generativa
  • Otimizar escalabilidade e desempenho
  • Implementar rastreamento com Application Insights e OpenTelemetry
  • Preparar e enviar jobs de fine-tuning

Visão Geral​

Otimizar soluções de IA generativa requer atenção à latência, custo, qualidade e observabilidade. Respostas com streaming melhoram a latência percebida entregando tokens de forma incremental em vez de esperar pela geração completa. Otimização de tokens usando bibliotecas como tiktoken permite estimativa precisa de custos e compressão de prompts. Juntas, essas técnicas reduzem tanto o tempo de resposta real quanto o percebido.

Observabilidade é crítica para sistemas de IA em produção. O Azure OpenAI integra-se com o Application Insights através do OpenTelemetry, fornecendo rastreamento de ponta a ponta de requisições, uso de tokens, distribuições de latência e taxas de erro. Spans e atributos customizados permitem rastrear métricas específicas do negócio, como uso de templates de prompt e scores de qualidade de resposta.

Fine-tuning permite a personalização de modelos base com dados específicos do domínio. O fluxo de trabalho envolve preparar dados de treinamento em formato JSONL (com pares de mensagens system/user/assistant), fazer upload dos arquivos, criar um job de fine-tuning e implantar o modelo customizado resultante. Modelos com fine-tuning podem alcançar melhor desempenho em tarefas específicas com prompts mais curtos, reduzindo tanto a latência quanto o custo por requisição.

Arquitetura​

Este desafio implementa respostas com streaming, configura rastreamento OpenTelemetry, otimiza o uso de tokens e prepara um fluxo de trabalho de fine-tuning.

Challenge 19 topology

Pré-requisitos​

  • Recurso Azure OpenAI com GPT-4o implantado
  • Recurso Application Insights (connection string)
  • Python 3.9+ com pacotes openai, tiktoken, azure-monitor-opentelemetry
  • .NET 8 SDK com pacotes NuGet Azure.AI.OpenAI, Azure.Monitor.OpenTelemetry.AspNetCore
  • Dados de treinamento em formato JSONL (para a tarefa de fine-tuning)

Implementação​

Tarefa 1: Implementar Respostas com Streaming​

import os
import time
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_KEY"],
    api_version="2024-10-21"
)

# Non-streaming: wait for complete response
start = time.time()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain 5 Azure AI services in detail."}],
    max_tokens=500
)
non_stream_time = time.time() - start
print(f"Non-streaming: {non_stream_time:.2f}s total wait")
print(f"Response length: {len(response.choices[0].message.content)} chars\n")

# Streaming: receive tokens incrementally
start = time.time()
first_token_time = None
full_response = ""

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain 5 Azure AI services in detail."}],
    max_tokens=500,
    stream=True
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        if first_token_time is None:
            first_token_time = time.time() - start
        content = chunk.choices[0].delta.content
        full_response += content
        print(content, end="", flush=True)

total_stream_time = time.time() - start

print(f"\n\nStreaming: first token in {first_token_time:.2f}s, "
      f"total {total_stream_time:.2f}s")
print(f"Response length: {len(full_response)} chars")
print(f"Time to first token improvement: {non_stream_time - first_token_time:.2f}s faster")

Tarefa 2: Configurar Rastreamento OpenTelemetry​

import os
from openai import AzureOpenAI
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

# Configure Application Insights export
configure_azure_monitor(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)

tracer = trace.get_tracer(__name__)

client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_KEY"],
    api_version="2024-10-21"
)

@tracer.start_as_current_span("chat_completion")
def get_completion(user_message: str, template_name: str = "default") -> str:
    """Traced chat completion with custom attributes."""
    span = trace.get_current_span()
    span.set_attribute("ai.prompt_template", template_name)
    span.set_attribute("ai.model", "gpt-4o")
    span.set_attribute("ai.user_message_length", len(user_message))

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": user_message}
        ],
        max_tokens=200
    )

    # Record usage metrics as span attributes
    span.set_attribute("ai.prompt_tokens", response.usage.prompt_tokens)
    span.set_attribute("ai.completion_tokens", response.usage.completion_tokens)
    span.set_attribute("ai.total_tokens", response.usage.total_tokens)
    span.set_attribute("ai.finish_reason", response.choices[0].finish_reason)

    return response.choices[0].message.content

# Execute traced requests
with tracer.start_as_current_span("user_interaction") as parent_span:
    parent_span.set_attribute("user.session_id", "session-12345")

    result1 = get_completion("What is Azure OpenAI?", "knowledge_qa")
    result2 = get_completion("Summarize in one sentence.", "summarization")

    parent_span.set_attribute("interaction.total_requests", 2)

print("Traces exported to Application Insights")
print(f"Result 1: {result1[:100]}...")
print(f"Result 2: {result2[:100]}...")

Tarefa 3: Contar e Otimizar Tokens​

import os
import tiktoken
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_KEY"],
    api_version="2024-10-21"
)

# Get the tokenizer for GPT-4o (uses o200k_base encoding)
encoding = tiktoken.encoding_for_model("gpt-4o")

def count_message_tokens(messages: list, model: str = "gpt-4o") -> int:
    """Count tokens for a list of chat messages."""
    enc = tiktoken.encoding_for_model(model)
    tokens_per_message = 3  # Every message has <|start|>role/name\n content<|end|>\n
    tokens_per_name = 1

    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(enc.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # Every reply is primed with <|start|>assistant<|message|>
    return num_tokens

# Example: Compare verbose vs. optimized prompts
verbose_messages = [
    {"role": "system", "content": "You are a very helpful and knowledgeable assistant who always provides detailed, comprehensive, and thorough answers to any questions that users might ask about Azure cloud computing services and their various features and capabilities."},
    {"role": "user", "content": "Could you please explain to me what Azure Cognitive Services is and what it does and how it works?"}
]

optimized_messages = [
    {"role": "system", "content": "Azure technical assistant. Be concise."},
    {"role": "user", "content": "What is Azure Cognitive Services?"}
]

verbose_tokens = count_message_tokens(verbose_messages)
optimized_tokens = count_message_tokens(optimized_messages)

print(f"Verbose prompt: {verbose_tokens} tokens")
print(f"Optimized prompt: {optimized_tokens} tokens")
print(f"Token savings: {verbose_tokens - optimized_tokens} tokens "
      f"({(1 - optimized_tokens/verbose_tokens)*100:.0f}% reduction)")

# Verify with actual API call
response = client.chat.completions.create(
    model="gpt-4o",
    messages=optimized_messages,
    max_tokens=100
)
print(f"\nActual prompt tokens (API): {response.usage.prompt_tokens}")
print(f"Local estimate: {optimized_tokens}")

# Truncation strategy for long contexts
def truncate_to_token_limit(text: str, max_tokens: int = 4000) -> str:
    """Truncate text to fit within token limit."""
    tokens = encoding.encode(text)
    if len(tokens) <= max_tokens:
        return text
    truncated_tokens = tokens[:max_tokens]
    return encoding.decode(truncated_tokens)

long_text = "Azure provides many services. " * 500  # Simulate long input
truncated = truncate_to_token_limit(long_text, max_tokens=100)
print(f"\nOriginal: {len(encoding.encode(long_text))} tokens")
print(f"Truncated: {len(encoding.encode(truncated))} tokens")

Tarefa 4: Preparar Dados de Fine-Tuning e Enviar Job​

import os
import json
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_KEY"],
    api_version="2024-10-21"
)

# Step 1: Prepare training data in JSONL format
training_data = [
    {
        "messages": [
            {"role": "system", "content": "You are an Azure billing assistant."},
            {"role": "user", "content": "What is a Reserved Instance?"},
            {"role": "assistant", "content": "A Reserved Instance (RI) is a billing discount applied when you commit to a 1-year or 3-year term for specific Azure resources like VMs or SQL databases. RIs can save up to 72% compared to pay-as-you-go pricing."}
        ]
    },
    {
        "messages": [
            {"role": "system", "content": "You are an Azure billing assistant."},
            {"role": "user", "content": "How do I view my Azure spending?"},
            {"role": "assistant", "content": "Navigate to Cost Management + Billing in the Azure portal. Use Cost Analysis for detailed breakdowns by resource, service, or time period. Set up budgets and alerts to monitor spending proactively."}
        ]
    },
    {
        "messages": [
            {"role": "system", "content": "You are an Azure billing assistant."},
            {"role": "user", "content": "What's the difference between PAYG and EA?"},
            {"role": "assistant", "content": "Pay-As-You-Go (PAYG) charges per-usage with no commitment. Enterprise Agreement (EA) is a volume licensing contract with upfront monetary commitment, offering lower rates and centralized billing for organizations."}
        ]
    }
]

# Write training file (minimum 10 examples required for fine-tuning)
with open("training_data.jsonl", "w") as f:
    for entry in training_data:
        f.write(json.dumps(entry) + "\n")

# Step 2: Upload training file
with open("training_data.jsonl", "rb") as f:
    training_file = client.files.create(
        file=f,
        purpose="fine-tune"
    )
print(f"Uploaded file ID: {training_file.id}")
print(f"Status: {training_file.status}")

# Step 3: Create fine-tuning job
fine_tuning_job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18",  # Base model to fine-tune
    hyperparameters={
        "n_epochs": 3,
        "batch_size": 1,
        "learning_rate_multiplier": 1.0
    },
    suffix="azure-billing"  # Custom model name suffix
)

print(f"\nFine-tuning job created: {fine_tuning_job.id}")
print(f"Status: {fine_tuning_job.status}")

# Step 4: Monitor fine-tuning progress
import time
while fine_tuning_job.status not in ("succeeded", "failed", "cancelled"):
    time.sleep(30)
    fine_tuning_job = client.fine_tuning.jobs.retrieve(fine_tuning_job.id)
    print(f"Status: {fine_tuning_job.status}")

if fine_tuning_job.status == "succeeded":
    print(f"Fine-tuned model: {fine_tuning_job.fine_tuned_model}")

Saída Esperada​

Non-streaming: 3.45s total wait
Response length: 892 chars

Streaming: first token in 0.42s, total 3.51s
Response length: 892 chars
Time to first token improvement: 3.03s faster

Traces exported to Application Insights

Verbose prompt: 68 tokens
Optimized prompt: 21 tokens
Token savings: 47 tokens (69% reduction)

Actual prompt tokens (API): 21
Local estimate: 21

Uploaded file ID: file-abc123def456
Fine-tuning job created: ftjob-xyz789
Status: running
Status: succeeded
Fine-tuned model: ft:gpt-4o-mini-2024-07-18:azure-billing:abc123

Quebra & conserta​

CenárioSintomaCausa RaizCorreção
Streaming retorna chunks vaziosSem conteúdo no deltaNormal — alguns chunks contêm apenas role/metadataFiltrar chunks onde delta.content não é None/null
Contagem de tokens divergenteContagem local difere da APIVersão do tokenizer incompatível ou overhead de mensagemUsar tiktoken com o modelo correto; considerar overhead de 3 tokens por mensagem
Job de fine-tuning falhaStatus: failedFormato de dados de treinamento inválido ou menos de 10 exemplosValidar formato JSONL; garantir mínimo de 10 exemplos de treinamento
Traces não aparecemSem dados no Application InsightsConnection string mal configurada ou atraso na ingestãoVerificar connection string; aguardar 2-5 minutos para ingestão
Modelo com fine-tuning com alta latênciaMais lento que o modelo baseModelo customizado não otimizado para deploymentAumentar capacidade do SKU; considerar se fine-tuning é necessário vs. few-shot

Verificação de Conhecimento​

1. Qual é o principal benefício de respostas com streaming no Azure OpenAI?

2. Qual é o número mínimo de exemplos de treinamento necessários para fine-tuning no Azure OpenAI?

3. Qual biblioteca é usada para contar tokens localmente para GPT-4o antes de fazer chamadas à API?

4. Ao configurar rastreamento OpenTelemetry para Azure OpenAI, qual métrica é mais importante para monitoramento de custos?

5. Qual formato os dados de treinamento devem usar para fine-tuning no Azure OpenAI?

Limpeza​

az group delete --name rg-ai102-challenge19 --yes --no-wait

# Delete fine-tuning artifacts
# curl -X DELETE "https://${AZURE_OPENAI_ENDPOINT}/openai/files/{file-id}?api-version=2024-10-21" \
#   -H "api-key: ${AZURE_OPENAI_KEY}"

Saiba Mais​