Building Production-Grade Semantic Search with GPT-5 and Microsoft Foundry, From Scratch

Most "RAG tutorials" stop at a single embedding query against a single index. That works for a demo and falls over the moment a real user asks something like "compare our Q3 and Q4 vendor contracts and flag anything that changed" — a question that needs multiple sub-queries, reasoning about what's still missing, and synthesis across documents. Microsoft Foundry's answer to this is Foundry IQ: an agentic retrieval layer built on Azure AI Search that treats retrieval as a reasoning task rather than a single keyword or vector lookup. This is a from-scratch build of a semantic search pipeline using GPT-5 for query planning/synthesis and Foundry IQ for retrieval. Architecture Instead of one query hitting one index once, Foundry IQ's knowledge base plans sub-queries, executes them in parallel against one or more knowledge sources, evaluates whether it has enough signal, and iterates before synthesizing a final, cited answer. The knowledge base sits between your agent and the underlying content. Your Foundry agent doesn't talk to Azure AI Search directly — it calls the knowledge base's MCP endpoint, which handles planning, retrieval, and synthesis behind a single tool call. Step 1: Create a knowledge source A knowledge source is a reusable reference to your underlying content — in this example, a Blob Storage container of documents. Creating it also triggers Azure AI Search to generate the index, skillset, and indexer needed to chunk and vectorize the content automatically. from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import ( SearchIndexKnowledgeSource, SearchIndexKnowledgeSourceParameters, ) from azure.identity import DefaultAzureCredential index_client = SearchIndexClient( endpoint="https://.search.windows.net", credential=DefaultAzureCredential(), ) knowledge_source = SearchIndexKnowledgeSource( name="vendor-contracts-ks", search_index_parameters=SearchIndexKnowledgeSourceParameters( search_index_name="vendor-contracts-index", ), ) index_client.create_or_update_knowledge_source(knowledge_source=knowledge_source) Enter fullscreen mode Exit fullscreen mode Step 2: Create a knowledge base with GPT-5 for planning and synthesis The knowledge base ties one or more knowledge sources together with an LLM deployment that handles query planning and answer synthesis. from azure.search.documents.indexes.models import ( KnowledgeBase, KnowledgeBaseAzureOpenAIModel, AzureOpenAIVectorizerParameters, ) knowledge_base = KnowledgeBase( name="vendor-contracts-kb", knowledge_sources=[{"name": "vendor-contracts-ks"}], models=[ KnowledgeBaseAzureOpenAIModel( azure_open_ai_parameters=AzureOpenAIVectorizerParameters( resource_url="https://.openai.azure.com", deployment_name="gpt-5-mini", model_name="gpt-5-mini", ) ) ], output_configuration={ "modality": "answerSynthesis", # verbatim extractive data is the alternative }, ) index_client.create_or_update_knowledge_base(knowledge_base=knowledge_base) Enter fullscreen mode Exit fullscreen mode output_configuration is the key lever here: answerSynthesis returns a pre-generated, cited answer, while extractive mode returns verbatim source chunks and leaves reasoning entirely to your agent's own model. Extractive mode costs less and gives your agent more control; synthesis mode does more work up front at the retrieval layer. Step 3: Wire the knowledge base into a Foundry agent via MCP Each knowledge base exposes a standalone MCP endpoint. Any MCP-compatible client — including Foundry Agent Service, but also GitHub Copilot or other MCP clients — can call its knowledge_base_retrieve tool directly. from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential project = AIProjectClient( endpoint="https://", credential=DefaultAzureCredential(), ) agent = project.agents.create_agent( model="gpt-5-mini", name="contract-search-agent", instructions="Answer questions about vendor contracts using the knowledge base tool. Always cite sources.", tools=[{ "type": "mcp", "server_url": "https://.search.windows.net/knowledgebases/vendor-contracts-kb/mcp?api-version=2026-05-01-preview", "allowed_tools": ["knowledge_base_retrieve"], }], ) Enter fullscreen mode Exit fullscreen mode Reasoning effort and cost/latency tradeoffs The knowledge base's retrieval_reasoning_effort setting controls how much LLM-driven planning happens before retrieval, and it's the main dial for balancing latency, cost, and answer quality. Reasoning effort What happens Best for Minimal Bypasses LLM query planning entirely; direct hybrid search Simple factual lookups, latency-sensitive paths Medium LLM reformulates and may decompose the query into sub-queries Multi-part or ambiguous questions High Full iterative planning, evaluates sufficiency, re-queries as needed Complex, multi-hop questions across many documents Lowering reasoning effort is also the primary way to control the number of GPT-5 tokens consumed per query — fewer planning passes and less iteration directly reduce both latency and inference cost, without needing to touch the underlying index. Multi-hop queries in practice The reason this matters versus classic RAG: a query like "which vendors had payment terms that changed between the Q3 and Q4 renewals" can't be answered by a single embedding lookup. With medium or high reasoning effort, the knowledge base decomposes it into sub-queries (find Q3 renewals, find Q4 renewals, compare payment terms fields), runs them in parallel against the knowledge source, and only synthesizes a final answer once it judges the retrieved context sufficient — re-querying automatically if it isn't. References Agentic Retrieval Overview — Azure AI Search, Microsoft Learn Foundry IQ: Unlock knowledge retrieval for agents — Microsoft Community Hub Tutorial: Build an Agentic Retrieval Solution — Azure AI Search, Microsoft Learn Connect Agents to Foundry IQ Knowledge Bases — Microsoft Foundry, Microsoft Learn GPT-5 in Azure AI Foundry: The future of AI apps and agents starts here — Microsoft Azure Blog