Skip to content

AI Tools and Retrievers

This document provides detailed information about OmniButler's AI tools and retriever components, which extend the core language model's capabilities.

Overview

OmniButler's AI system is enhanced with specialized tools and retrievers that enable it to:

  1. Access and search transaction data
  2. Retrieve relevant email information
  3. Interact with external services
  4. Perform specific financial operations
  5. Access real-time information

These capabilities transform the AI from a simple language model into a comprehensive financial assistant with domain-specific knowledge and capabilities.

Architecture Diagram

```mermaid graph TD subgraph LLM System LLM[Language Model] TE[Tool Executor] end

subgraph Tools
    TT[Transaction Tool]
    MT[Messaging Tool]
    GT[Google Places Tool]
    CT[Time Tool]
    FT[Finance Tool]
end

subgraph Retrievers
    TR[Transaction Retriever]
    ER[Email Retriever]
    MR[Merger Retriever]
end

subgraph Data Sources
    TD[Transaction DB]
    ED[Email DB]
    EX[External APIs]
end

User[User] --> |Query| LLM
LLM --> |Tool Call| TE

TE --> TT
TE --> MT
TE --> GT
TE --> CT
TE --> FT

TT --> TR
TR --> TD

ER --> ED

MR --> TR
MR --> ER

MT --> EX
GT --> EX
FT --> EX

TR --> |Results| TE
ER --> |Results| TE
MT --> |Results| TE
GT --> |Results| TE
CT --> |Results| TE
FT --> |Results| TE

TE --> |Tool Results| LLM
LLM --> |Response| User

style LLM System fill:#f9f,stroke:#333,stroke-width:2px
style Tools fill:#dfd,stroke:#333,stroke-width:2px
style Retrievers fill:#bbf,stroke:#333,stroke-width:2px
style Data Sources fill:#fdd,stroke:#333,stroke-width:2px

```

Tools System

The tools system extends the AI's capabilities by allowing it to perform specific actions and access external services.

Integration with LangChain

OmniButler uses LangChain's tool framework to define and implement tools:

def get_tools(user_id: str):
    return [
        Tool(
            name="WhatsappMessagingTool",
            func=WhatsappMessagingTool(),
            description="Use this tool to send a WhatsApp message...",
        ),
        Tool(
            name="Search",
            func=search.run,
            description="useful for when you need to answer questions about current events",
        ),
        transactions_retriever_tool(user_id),
        GooglePlacesTool(),
        GoogleFinanceQueryRun(
            api_wrapper=GoogleFinanceAPIWrapper(
                serp_api_key=app_config.serpapi_api_key.get_secret_value(),
            )
        ),
        get_current_time,
    ]

Tools are registered with the LLM system and made available for the AI to use when needed.

Core Tools

1. Transaction Retriever Tool

The Transaction Retriever Tool enables the AI to search and analyze financial transactions:

def transactions_retriever_tool(app_user_id: str) -> Tool:
    retriever = transaction_retriever(app_user_id, use_structured_query=True)

    return create_retriever_tool(
        retriever=retriever,
        name="TransactionRetrieverTool",
        description="Use this retriever tool to search transactions",
    )

Features: - Searches transactions by date, amount, merchant, category - Uses structured queries for precise filtering - Returns relevant transaction details including metadata - User-specific data access with proper authentication

2. WhatsApp Messaging Tool

The WhatsApp Messaging Tool enables the AI to send WhatsApp messages:

class WhatsappMessagingTool(BaseTool):
    name: str = "WhatsappMessagingTool"
    description: str = (
        "Use this tool to send a WhatsApp message. "
        "If you don't know the recipient's number, ask for it first before using this tool."
    )

    def _run(self, body: str, to: str) -> str:
        try:
            message_sid = whatsapp.run(body=body, to=to)
            return f"WhatsApp message sent successfully. Message SID: {message_sid}"
        except Exception as e:
            return f"Error: {e!s}"

Features: - Sends formatted WhatsApp messages via Twilio - Validates recipient phone numbers - Returns confirmation with message SID - Handles error conditions gracefully

3. Google Places Tool

The Google Places Tool allows the AI to search for location information:

GooglePlacesTool()

Features: - Searches for places by name, type, or address - Returns detailed place information - Provides location coordinates and business details - Useful for financial context like merchant locations

4. Current Time Tool

The Current Time Tool provides accurate time information:

@tool
def get_current_time() -> str:
    """Returns the current time in UTC and local time."""
    utc_time = datetime.datetime.now(datetime.timezone.utc)
    local_time = datetime.datetime.now()

    return (
        f"UTC time: {utc_time.strftime('%Y-%m-%d %H:%M:%S %Z')}\n"
        f"Local time: {local_time.strftime('%Y-%m-%d %H:%M:%S')}"
    )

Features: - Provides current time in both UTC and local timezone - Used for time-sensitive financial queries - Helps with scheduling and deadline-related questions

5. Google Finance Tool

The Google Finance Tool provides financial information:

GoogleFinanceQueryRun(
    api_wrapper=GoogleFinanceAPIWrapper(
        serp_api_key=app_config.serpapi_api_key.get_secret_value(),
    )
)

Features: - Retrieves stock prices and market data - Provides financial news and updates - Supports investment-related queries - Enhances the AI's financial knowledge

Retriever System

The retriever system enables the AI to access and search through various data sources using semantic search and structured queries.

Retriever Types

1. Transaction Retriever

The Transaction Retriever enables semantic search across financial transactions:

def transaction_retriever(app_user_id: str, use_structured_query: bool = False):
    """Create a transaction retriever for a specific user"""
    # Load user's transaction data
    db_client = get_user_db_client(app_user_id)
    transactions = load_user_transactions(db_client)

    if use_structured_query:
        # Return a structured query retriever
        return create_structured_query_retriever(transactions)
    else:
        # Return a standard semantic search retriever
        return create_semantic_search_retriever(transactions)

Features: - User-specific transaction access - Semantic search based on natural language queries - Optional structured query support for precise filtering - Returns relevant transactions with metadata

2. Email Retriever

The Email Retriever provides access to email content:

def email_retriever():
    """Create an email retriever"""
    # Initialize email data source
    email_data = initialize_email_data_source()

    # Create and return email retriever
    return create_email_retriever(email_data)

Features: - Searches email content semantically - Filters by sender, recipient, date, subject - Returns relevant email snippets - Integrates with Gmail

3. Merger Retriever

The Merger Retriever combines multiple retrievers for unified search:

def merge_retriever_tool(app_user_id: str) -> Tool:
    retriever = MergerRetriever(
        retrievers=[email_retriever(), transaction_retriever(app_user_id)]
    )

    return create_retriever_tool(
        retriever=retriever,
        name="TransactionAndEmailRetrieverTool",
        description="Use this retriever tool to search emails and transactions",
    )

Features: - Combines results from multiple retrievers - Provides unified search across different data types - Ranks results by relevance - Returns formatted results with source attribution

Integration with Memory System

The retrievers integrate with the memory system to maintain context:

  1. Vector Store Retrieval: Transactions and emails are embedded and stored in vector databases
  2. Context Enrichment: Retrieved data is used to enrich the AI's context
  3. Memory Management: Frequently accessed information is prioritized

Tool Descriptions in System Prompt

The tools are described in the system prompt to guide the AI in their usage:

**Available Tools:**
- `get_current_time`: Use this to fetch the current time for queries related to events, transactions, deadlines, etc.
- `TransactionRetrieverTool`: Use this to search transactions based on the user's query. This is your go-to tool and you should always consider using this first before any other tool
- `WhatsappMessagingTool`: Use this to send WhatsApp messages to users.
- `GooglePlacesTool`: Use this to find places based on user queries.

Prompt Engineering for Tool Use

The system uses prompt engineering to guide the AI in effective tool usage:

**Guidelines:**
- **Accuracy & Detail**: Provide detailed and specific information based on the context and available data.
- **Assumptions**: Minimize assumptions. If you require more information, ask clarifying questions.
- **Internal Reasoning:** Do not include any internal reasoning steps, thoughts, actions, action inputs, structured queries, code, or any JSON in your final response. Present only the final answer to the user in a clear and concise manner.

Transaction Data Retrieval

The Transaction Retriever is particularly important, providing structured access to financial data:

  1. Loading Transactions: Transactions are loaded from the user's database
  2. Embedding Generation: Transactions are embedded for semantic search
  3. Query Processing: User queries are processed to extract search parameters
  4. Result Formatting: Results are formatted for optimal AI consumption
# Example formatted transaction result
"""
Aug 31, 2024: $500.00 at KFC
Transaction Type: Purchase
Account: Chase Freedom
Category: Dining
Reference: TX123456789
"""

Future Enhancements

  1. Expanded Tool Set
  2. Investment management tools
  3. Budget planning assistants
  4. Tax calculation helpers

  5. Document processing tools

  6. Enhanced Retrievers

  7. Multi-vector retrieval for improved accuracy
  8. Hybrid retrieval combining semantic and keyword search
  9. User feedback-based retrieval improvement

  10. Real-time financial data retrieval

  11. Tool Orchestration

  12. Multi-step tool execution planning
  13. Parallel tool execution for efficiency
  14. Tool result verification and correction
  15. Learning from tool usage patterns