Build a Conversational AI with LlamaIndex

What We’re Building

In this tutorial, we will construct an agentic conversational AI companion using LlamaIndex. Instead of using primitive hardcoded logic to route user intent, we will use a ReAct (Reasoning and Acting) agent loop. This workflow allows the core engine to evaluate a user’s prompt and dynamically choose whether to delegate the task to OpenAI’s GPT-4o or Anthropic’s Claude 3.5 Sonnet.

Setup and Installation

Modern versions of LlamaIndex are modular. We must install the core framework alongside the specific multi-model provider integration packages.

pip install llama-index llama-index-llms-openai llama-index-llms-anthropic python-dotenv

Next, configure your environment variables securely. Create a .env file in your root project directory:

# .env file
OPENAI_API_KEY=your_openai_api_key
ANTHROPIC_API_KEY=your_anthropic_api_key
  

Step 1: Initializing LLMs with LlamaIndex Abstractions

First, we configure our environment and initialize our language models using the official LlamaIndex wrappers. This standardizes their inputs and outputs.

import os
import asyncio
from dotenv import load_dotenv
from llama_index.llms.openai import OpenAI
from llama_index.llms.anthropic import Anthropic

load_dotenv()

# Instantiate the respective LLM configurations
gpt_model = OpenAI(model="gpt-4o")
claude_model = Anthropic(model="claude-3-5-sonnet-20241022")
  

Step 2: Defining Functional Agent Tools

To let our agent interact with these models dynamically, we wrap the model invocations inside LlamaIndex FunctionTool structures. The docstrings act as the prompt-hints the agent reads to make its structural decisions.

from llama_index.core.tools import FunctionTool

def call_gpt_engine(prompt: str) -> str:
    """Useful when queries require raw logic, structured JSON formatting, or math calculations."""
    response = gpt_model.complete(prompt)
    return str(response)

def call_claude_engine(prompt: str) -> str:
    """Useful when queries require deeply creative writing, code generation, or nuanced tonal analysis."""
    response = claude_model.complete(prompt)
    return str(response)

# Convert functions to native tools
gpt_tool = FunctionTool.from_defaults(fn=call_gpt_engine)
claude_tool = FunctionTool.from_defaults(fn=call_claude_engine)
  

Step 3: Creating the ReAct Agent

Now, we construct the ReActAgent, passing our custom model tools directly into its execution layout. We will use GPT-4o as our central engine coordinator.

from llama_index.core.agent import ReActAgent

# Bind tools to the orchestration framework
tools = [gpt_tool, claude_tool]
agent = ReActAgent.from_tools(tools, llm=gpt_model, verbose=True)
  

Testing Your Implementation

Execute queries targeting different tasks to see LlamaIndex evaluate the text, pick a tool, format its variables, and cleanly handle responses.

async def main():
    # This should trigger the Claude tool based on your docstring hints
    creative_res = await agent.achat("Write a short, moody poem about artificial intelligence.")
    print(f"Creative Task:\n{creative_res}\n")

    # This should route to the GPT tool
    structured_res = await agent.achat("Generate a structured list of 3 fake user profiles with keys: id, name.")
    print(f"Structured Task:\n{structured_res}\n")

if __name__ == "__main__":
    asyncio.run(main())
  

What to Build Next

• Add a VectorStoreIndex to supply localized RAG context windows straight to your tool execution pathways.
• Incorporate persistent database storage to preserve chat histories across multiple app sessions.
• Expose your LlamaIndex orchestration wrapper via a robust FastAPI framework backend.