Agent Switching in Notebook Chat Panel

HISTORY_MARKDOWN_AGENT_SWITCHING.md

Agent Switching in Notebook Chat Panel

Problem

The notebook chat panel hardcodes agent_type = "notebook_assistant". The notebook assistant is good at document editing but knows nothing about Galaxy's visualization plugin system, Vega-Lite spec generation, or chart type recommendations. Users doing data exploration want AI help picking and configuring visualizations — a different skill set from "rewrite my Methods section."

Current State (Why This Is Easy)

The architecture already supports multi-agent at every layer except one frontend constant:

Layer	Status
DB schema	agent_type stored per-message in JSON blob, not a column. No constraint on mixing.
Chat API	POST /api/chat?agent_type=X — already a dynamic query param routed by AgentService.route_and_execute()
Agent registry	AgentType enum + _registry dict. New agents register with a decorator.
Agent config	Per-agent model, temperature, prompt, tools via galaxy.yml cascade
Notebook context	Chat API loads history_id + notebook_content for any agent when notebook_id present
Frontend display	ChatMessageCell already renders per-message agentType with distinct icons/labels from agentTypes.ts
Diff views	ProposalDiffView/SectionPatchView only render when edit_mode present in agent response metadata. Non-edit agents gracefully render as plain chat.

The one hardcoded line: NotebookChatPanel.vue:34 → const AGENT_TYPE = "notebook_assistant"

Minimal Agent Switcher (Phase 1)

Frontend (~50-100 lines)

1. Replace const AGENT_TYPE with const selectedAgent = ref("notebook_assistant")
2. Add agent picker dropdown in chat header — filter agentTypes.ts registry to notebook-compatible agents
3. Pass selectedAgent.value in submitQuery() instead of constant
4. Show agent icon/label per-message in conversation thread (already works via ChatMessageCell)
5. Persist selected agent in userLocalStorage per-notebook

Backend (zero changes)

Nothing. The routing, context injection, and storage all work already.

Test additions

• Vitest: agent picker renders, switching changes submitted agent_type
• Vitest: messages from different agents render with correct icons
• API test: send messages with different agent_types to same notebook exchange, verify both stored

Visualization Agent (Phase 2)

Registration

# lib/galaxy/agents/visualization.py
class VisualizationAgent(BaseGalaxyAgent):
    agent_type = AgentType.VISUALIZATION
    name = "visualization"

Add VISUALIZATION = "visualization" to AgentType enum in base.py.

Tools

Tool	Purpose
list_history_datasets	Reuse from history_tools.py — same dataset discovery
get_dataset_peek	Reuse — preview tabular data
get_column_metadata	NEW — column names, types, cardinality for a tabular dataset
search_visualizations	NEW — query /api/plugins?embeddable=True, return available viz types with descriptions
get_visualization_schema	NEW — given a plugin name, return its config schema (from XML <param> definitions)
resolve_hid	Reuse — HID → directive ID translation

Structured Output Types

class VegaSpecProposal(BaseModel):
    mode: Literal["vega_spec"] = "vega_spec"
    description: str          # what the chart shows
    spec: dict                # Vega-Lite JSON spec
    target_section: str | None  # heading to insert under (optional)

class PluginVisualizationProposal(BaseModel):
    mode: Literal["plugin_viz"] = "plugin_viz"
    description: str
    plugin_name: str
    dataset_hid: int
    config: dict              # plugin-specific config

class DirectiveInsert(BaseModel):
    mode: Literal["directive_insert"] = "directive_insert"
    directive: str            # e.g. "history_dataset_display(history_dataset_id=42)"
    target_section: str | None

Plain str for conversational responses (same pattern as notebook assistant).

System Prompt Focus

• "You help users visualize their data"
• Knows Vega-Lite spec syntax and common chart patterns (bar, scatter, line, heatmap, histogram)
• Knows Galaxy plugin visualization types and their configs
• Knows Galaxy markdown directive syntax for embedding
• Can read dataset peeks to suggest appropriate chart types based on column structure
• Produces insertable directives or Vega-Lite specs ready to paste into the notebook

Frontend Proposal Views

New component: VisualizationProposalView.vue

• For vega_spec: render a live Vega-Lite preview (reuse VegaWrapper.vue) + "Insert into notebook" button
• For plugin_viz: render plugin description + config summary + "Insert directive" button
• For directive_insert: render directive preview + insert button

The NotebookChatPanel would check agent_response.mode and dispatch to the appropriate view — same pattern as existing isProposalVisible / isSectionPatch checks.

Context Sharing Between Agents

Problem

If a user talks to the notebook assistant about their ChIP-seq analysis, then switches to the visualization agent, the viz agent has no memory of the prior conversation.

Options

1. Separate context (simplest): Each agent sees only its own prior messages. Conversation history filtering by agent_type in the JSON blob. Users switch agents like switching tools.

2. Shared context (richer): All messages in the exchange are visible to both agents. The viz agent sees the notebook assistant's conversation and vice versa. Risk: prompt confusion, wasted tokens on irrelevant context.

3. Summary handoff (middle ground): When switching agents, inject a one-paragraph summary of the prior conversation into the new agent's context. Could be auto-generated or just the last N messages.

Recommendation: start with option 1 (separate context). It's the simplest, avoids prompt pollution, and matches user mental model of "I'm now talking to a different assistant." Revisit if users report friction.

File Changes Estimate

Phase 1 (agent switcher only)

File	Change
NotebookChatPanel.vue	~30 lines: ref, dropdown, pass to API
agentTypes.ts	~5 lines: add notebook_compatible: true filter flag
historyNotebookStore.ts	~5 lines: persist selected agent per-notebook
Tests (3-4 vitest files)	~80 lines
Total	~120 lines

Phase 2 (visualization agent)

File	Change
lib/galaxy/agents/visualization.py	~300 lines: agent class, tools, output types
lib/galaxy/agents/prompts/visualization.md	~100 lines: system prompt
lib/galaxy/agents/base.py	+1 line: enum value
client/src/components/HistoryNotebook/VisualizationProposalView.vue	~150 lines
NotebookChatPanel.vue	~20 lines: proposal view dispatch
agentTypes.ts	~5 lines: new entry
Backend tests	~200 lines
Frontend tests	~150 lines
Total	~930 lines

Unresolved Questions

• Should switching agents mid-exchange create a new ChatExchange row, or continue the same one? Per-message agent_type in JSON supports either, but "New Chat" semantics may be cleaner.
• Should the auto router agent be offered in the notebook picker? It doesn't have notebook tools registered — would need wiring.
• Should the visualization agent be able to produce SectionPatchEdit (reuse notebook assistant's output type) for inserting directives, or always use its own output types?
• For get_column_metadata tool: should it use the dataset API or parse the peek directly? Peek is already loaded; API gives richer metadata but requires additional request.
• Should the visualization agent know about the Charts external plugin (charts.galaxyproject.org), or only plugins registered in the local Galaxy instance?