dspy-simple-demo.md

Small Demo of Ollama + DSPy

Installing DSPy and setting up Ollama in mac:

1. Install DSPy:

pip install dspy-ai

2. Install Ollama:

• Visit https://ollama.ai/download
• Download the macOS installer
• Open the downloaded file and follow installation instructions OR
• Use Homebrew:

brew install ollama

3. Start Ollama service:

ollama serve

By default, Ollama runs on port 11434. To run on a different port:

OLLAMA_HOST=127.0.0.1:3000 ollama serve

Replace 3000 with your desired port number.

4. Pull a model (in a new terminal window):

ollama pull llama2

You can replace llama2 with any other model like:

• mistral
• codellama
• gemma
• neural-chat

5. Test if Ollama is running:

curl http://localhost:11434/api/tags

(Replace 11434 with your custom port if you changed it)

6. To stop Ollama:

• Use Control+C in the terminal where Ollama is running
• Or use Activity Monitor to quit the Ollama process

Common issues and solutions:

• If port is already in use: Check what's using the port with lsof -i :11434
• If permission denied: Try running with sudo
• If model download is slow: Check your internet connection or try a smaller model

Code Example in Python

import dspy
from typing import Literal

# 1. Set up Ollama endpoint
lm = dspy.LM('ollama_chat/llama2', api_base='http://localhost:3333', api_key='')

# 2. Configure DSPy to use Ollama
dspy.configure(lm=lm)

# 3. Define a basic signature for completion
class SimpleCompletion(dspy.Signature):
    """Basic completion task"""
    input = dspy.InputField()
    output = dspy.OutputField(desc="Completed response")

def search_wikipedia(query: str) -> list[str]:
    results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3)
    return [x['text'] for x in results]

# 4. Create a predictor using the signature
predictor = dspy.Predict(SimpleCompletion)
rag = dspy.ChainOfThought('context, question -> response')

# 5. Example usage with basic completion
def run_example():
    # Simple question answering
    result = predictor(input="What is the capital of France?")
    print("Response:", result.output)
    
    # More complex completion
    result = predictor(input="Write a haiku about programming:")
    print("Response:", result.output)

    question = "Who is Narendra Modi?"
    result = rag(context=search_wikipedia(question), question=question)
    print("RAG RESULT:")
    print(result)


if __name__ == "__main__":
    run_example()

Code Breakdown

The search_wikipedia function performs a search operation using the dspy.ColBERTv2 model to retrieve relevant information from a Wikipedia dataset. Here's a breakdown of the function:

Function Breakdown

def search_wikipedia(query: str) -> list[str]:

• Inputs:
  • query: A string representing the search query you want to look up in the Wikipedia dataset.
• Outputs:
  • Returns a list of strings, where each string is a relevant snippet or abstract retrieved from the Wikipedia dataset.

---

results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3)

1. dspy.ColBERTv2:

   • This is a retrieval model from the DSPy library.
   • ColBERTv2 is a dense retrieval model designed to retrieve relevant passages or documents from large datasets based on the given query.

2. url='http://20.102.90.50:2017/wiki17_abstracts':

   • Specifies the endpoint or database where the Wikipedia dataset is hosted. In this case, it appears to point to a locally or remotely hosted ColBERTv2 retrieval service.

3. query:

   • The function takes the user-provided query and passes it to the ColBERTv2 model.

4. k=3:

   • This parameter specifies that the model should retrieve the top 3 most relevant results for the query.

5. results:

   • The variable results stores the output from the model, which is typically a list of dictionaries, where each dictionary represents a retrieved result (e.g., text and associated metadata).

---

return [x['text'] for x in results]

• Extracts only the 'text' field from each dictionary in results and returns them as a list of strings.

---

Overall Purpose

This function:

1. Sends a query to a ColBERTv2 model hosted at a specific URL containing Wikipedia abstracts.
2. Retrieves the top 3 most relevant passages based on the query.
3. Extracts and returns the text of those passages as a list of strings.

---

Example Usage

Suppose the dataset contains abstracts from Wikipedia. If you call:

search_wikipedia("What is quantum mechanics?")

It might return:

[
    "Quantum mechanics is a fundamental theory in physics that describes nature at the smallest scales.",
    "It provides a mathematical framework for understanding atomic and subatomic particles.",
    "Key principles of quantum mechanics include superposition, entanglement, and wave-particle duality."
]