Skip to content

Instantly share code, notes, and snippets.

@scottcain

scottcain/galaxy-workflow-execution-analysis.md

Last active February 17, 2026 21:30
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save scottcain/5e480674b4285a22cac30103aa9a8f85 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save scottcain/5e480674b4285a22cac30103aa9a8f85 to your computer and use it in GitHub Desktop.
Download ZIP
Claude generated details of issues with using Galaxy API and workflow execution
Raw
galaxy-workflow-execution-analysis.md

Galaxy Workflow Execution Behavior Analysis

Note: All API calls in this document use [YOUR_API_KEY] as a placeholder. Replace this with your actual Galaxy API key when reproducing these calls.

Issue Summary

When invoking workflows via Galaxy API with a specified target history, workflows consistently create new histories instead of executing in the specified target history. This occurs even with user-owned (imported) workflows, not just public workflows.

Context

  • Galaxy Instance: https://usegalaxy.org
  • User: scottcain
  • Target History: bbd44e69cb8906b5b37974f851b5cfe1 ("A fumigatus antifungal resistance analysis - STEP1 - Iteration2")
  • Goal: Execute "Parallel accession download ENA" workflow on two input datasets within the target history
  • Problem: All workflow invocations create new histories regardless of API parameters

Target History Analysis

History Status Check

curl -H "X-API-KEY: [YOUR_API_KEY]" \
  "https://usegalaxy.org/api/histories/bbd44e69cb8906b5b37974f851b5cfe1"

Response:

{
  "model_class": "History",
  "id": "bbd44e69cb8906b5b37974f851b5cfe1",
  "name": "A fumigatus antifungal resistance analysis - STEP1 - Iteration2",
  "create_time": "2026-02-16T22:16:10.906935",
  "update_time": "2026-02-16T22:25:36.726067",
  "published": false,
  "importable": false,
  "deleted": false,
  "purged": false,
  "archived": false,
  "size": 0,
  "user_id": "5969b674c84a81c9",
  "username": "scottcain",
  "email": "scottcain@users.noreply.github.com",
  "nice_size": "0 bytes",
  "annotation": null,
  "tags": [],
  "state": "error",          # ← IMPORTANT: History in error state
  "state_details": {},
  "state_ids": {
    "discarded": [],
    "empty": [],
    "error": [53, 54, 55, 57, 60, 61, 62],
    "failed_metadata": [],
    "new": [],
    "ok": [32, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 56, 58, 59, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72],
    "paused": [],
    "queued": [],
    "running": [],
    "setting_metadata": [],
    "upload": []
  }
}

Key Observations:

  • History exists and is accessible
  • History state: "error"
  • Error datasets: HIDs 53, 54, 55, 57, 60, 61, 62
  • Many datasets in "ok" state including our input files (HIDs 71, 72)

Workflow Import Process

Public Workflow Discovery

Target Workflow: "Parallel accession download ENA"

Failed Import Attempts

Attempt 1: Direct Workflow Creation

curl -X POST -H "X-API-KEY: [YOUR_API_KEY]" -H "Content-Type: application/json" \
  --data @workflow_definition.json \
  "https://usegalaxy.org/api/workflows"

Result: {"err_msg": "Uncaught exception in exposed API method:", "err_code": 0}

Note: workflow_definition.json = raw workflow JSON downloaded from public workflow (same content as workflow_def.json shown above)

Attempt 2: Workflow Upload Endpoint

curl -X POST -H "X-API-KEY: [YOUR_API_KEY]" -H "Content-Type: application/json" \
  --data @workflow_definition.json \
  "https://usegalaxy.org/api/workflows/upload"

Result: {"err_msg": "Uncaught exception in exposed API method:", "err_code": 0}

Note: Same raw workflow JSON, different endpoint

Attempt 3: Import from URL

curl -X POST -H "X-API-KEY: [YOUR_API_KEY]" -H "Content-Type: application/json" \
  -d '{"workflow_url": "https://usegalaxy.org/api/workflows/36e89888fca50cdd/download"}' \
  "https://usegalaxy.org/api/workflows/import"

Result: {"err_msg": "Missing required parameter 'workflow_id'.", "err_code": 400005}

Successful Import Method

Workflow Definition Download

curl -H "X-API-KEY: [YOUR_API_KEY]" \
  "https://usegalaxy.org/api/workflows/36e89888fca50cdd/download" > workflow_def.json

Result: ✅ Success (19,776 bytes, valid JSON with "a_galaxy_workflow": "true")

Workflow Definition Structure (workflow_def.json):

{
    "a_galaxy_workflow": "true",
    "annotation": "This Galaxy workflow automates the parallel download of sequencing data (FASTQ files) from the European Nucleotide Archive (ENA) .",
    "comments": [],
    "creator": [
        {
            "class": "Person",
            "identifier": "https://orcid.org/0000-0002-5987-8032",
            "name": "Anton Nekrutenko",
            "url": "https://orcid.org/0000-0002-5987-8032"
        }
    ],
    "format-version": "0.1",
    "license": "MIT",
    "name": "Parallel  accession download ENA",
    "readme": "[Long README content...]",
    "report": {
        "markdown": "[Report template...]"
    },
    "steps": {
        "0": {
            "annotation": "This is the list of accessions downloaded from NCBI or EBA. It should contain one accession per line",
            "content_id": null,
            "errors": null,
            "id": 0,
            "input_steps": {},
            "inputs": [
                {
                    "description": "This is the list of accessions downloaded from NCBI or EBA. It should contain one accession per line",
                    "name": "List of accessions"
                }
            ],
            "label": "List of accessions",
            "name": "Input dataset",
            "outputs": [],
            "position": {...},
            "tool_id": null,
            "tool_state": "{...}",
            "tool_version": null,
            "type": "data_input",
            "uuid": "48559016-e44c-47ad-ac54-1f71003cbce8",
            "when": null,
            "workflow_outputs": []
        },
        "1": {
            "annotation": "By default this number is 1, which may create too many jobs...",
            "content_id": null,
            "errors": null,
            "id": 1,
            "input_steps": {},
            "inputs": [
                {
                    "description": "By default this number is 1...",
                    "name": "Number of accessions for a single download job"
                }
            ],
            "label": "Number of accessions for a single download job",
            "name": "Input parameter",
            "outputs": [],
            "position": {...},
            "tool_id": null,
            "tool_state": "{...}",
            "tool_version": null,
            "type": "parameter_input",
            "uuid": "00442617-161d-4a84-aae7-cdb373295ce1",
            "when": null,
            "workflow_outputs": []
        },
        [... additional steps 2-6 with tools: split_file_to_collection, fastq_dl, __APPLY_RULES__ ...]
    },
    "tags": [],
    "uuid": "c60cfa31-3eaa-4067-aa46-abcd1c8f315b",
    "version": 1
}

Note: Full workflow definition is 19,776 bytes containing 7 steps with complete tool configurations.

Successful Import with Wrapped Payload

# Create payload: {"workflow": <workflow_definition>}
echo '{"workflow": ' > workflow_payload.json
cat workflow_def.json >> workflow_payload.json
echo '}' >> workflow_payload.json

curl -X POST -H "X-API-KEY: [YOUR_API_KEY]" -H "Content-Type: application/json" \
  --data @workflow_payload.json \
  "https://usegalaxy.org/api/workflows"

Successful Payload Structure (workflow_payload.json):

{
  "workflow": {
    "a_galaxy_workflow": "true",
    "annotation": "This Galaxy workflow automates the parallel download of sequencing data (FASTQ files) from the European Nucleotide Archive (ENA) .",
    "comments": [],
    "creator": [...],
    "format-version": "0.1",
    "license": "MIT",
    "name": "Parallel  accession download ENA",
    "readme": "[Long README content...]",
    "steps": {
      "0": {...},
      "1": {...},
      [... all 7 workflow steps ...]
    },
    "tags": [],
    "uuid": "c60cfa31-3eaa-4067-aa46-abcd1c8f315b",
    "version": 1
  }
}

Key Discovery: The API requires workflow definition to be wrapped in {"workflow": ...} object, not sent as raw workflow JSON.

Response:

{
  "model_class": "StoredWorkflow",
  "id": "b8f98b64537f2324",
  "latest_workflow_id": 690514,
  "name": "Parallel  accession download ENA",
  "create_time": "2026-02-17T20:24:47.978537",
  "update_time": "2026-02-17T20:24:48.053845",
  "published": false,
  "importable": false,
  "deleted": false,
  "hidden": false,
  "tags": [],
  "latest_workflow_uuid": "c60cfa31-3eaa-4067-aa46-abcd1c8f315b",
  "creator_deleted": false,
  "annotations": ["This Galaxy workflow automates the parallel download of sequencing data (FASTQ files) from the European Nucleotide Archive (ENA) ."],
  "url": "/api/workflows/b8f98b64537f2324",
  "owner": "scottcain",                    # ← Successfully imported to user account
  "number_of_steps": 7
}

Result: ✅ SUCCESS - Workflow imported with ID b8f98b64537f2324

Input Dataset Preparation

Dataset Information

  • Resistant accessions (HID 71): Dataset ID f9cad7b01a472135e129a439da02d6d9
  • Susceptible accessions (HID 72): Dataset ID f9cad7b01a472135c7e27f4dfb4144b3

Both datasets confirmed in target history and in "ok" state.

Workflow Execution Attempts

Iteration 2: Direct Public Workflow Invocation

Resistant Accessions Execution

curl -X POST -H "X-API-KEY: [YOUR_API_KEY]" -H "Content-Type: application/json" \
  -d '{
    "history": "bbd44e69cb8906b5b37974f851b5cfe1",
    "workflow_id": "36e89888fca50cdd",
    "inputs": {
      "0": {
        "src": "hda",
        "id": "f9cad7b01a472135e129a439da02d6d9"
      },
      "1": 1
    }
  }' \
  "https://usegalaxy.org/api/workflows/36e89888fca50cdd/invocations"

Response:

{
  "model_class": "WorkflowInvocation",
  "id": "8301eb013921921b",
  "create_time": "2026-02-17T16:33:18.680863",
  "update_time": "2026-02-17T16:33:18.680871",
  "workflow_id": "9f8dceaf295f8450",
  "history_id": "bbd44e69cb8906b5a8ca2202862977a7",  # ← NEW history, not target
  "uuid": "5dc04d36-0c1e-11f1-b9d7-bc24110164aa",
  "state": "new",
  "landing_uuid": null
}

Susceptible Accessions Execution

curl -X POST -H "X-API-KEY: [YOUR_API_KEY]" -H "Content-Type: application/json" \
  -d '{
    "history": "bbd44e69cb8906b5b37974f851b5cfe1",
    "workflow_id": "36e89888fca50cdd",
    "inputs": {
      "0": {
        "src": "hda",
        "id": "f9cad7b01a472135c7e27f4dfb4144b3"
      },
      "1": 5
    }
  }' \
  "https://usegalaxy.org/api/workflows/36e89888fca50cdd/invocations"

Response:

{
  "model_class": "WorkflowInvocation",
  "id": "a9f8fce9d4fb6c44",
  "create_time": "2026-02-17T16:33:26.315779",
  "update_time": "2026-02-17T16:33:26.315787",
  "workflow_id": "9f8dceaf295f8450",
  "history_id": "bbd44e69cb8906b5642ce18146a5a534",  # ← ANOTHER new history, not target
  "uuid": "624cf3ea-0c1e-11f1-984d-bc24110164aa",
  "state": "new",
  "landing_uuid": null
}

Key Observations:

  • Both executions specified target history in API request
  • Both created entirely new histories instead
  • Different new history IDs for each execution
  • Workflows executed successfully in the new histories

Iteration 3: User-Owned (Imported) Workflow Invocation

Resistant Accessions Execution

curl -X POST -H "X-API-KEY: [YOUR_API_KEY]" -H "Content-Type: application/json" \
  -d '{
    "history": "bbd44e69cb8906b5b37974f851b5cfe1",
    "workflow_id": "b8f98b64537f2324",
    "inputs": {
      "0": {
        "src": "hda",
        "id": "f9cad7b01a472135e129a439da02d6d9"
      },
      "1": 1
    }
  }' \
  "https://usegalaxy.org/api/workflows/b8f98b64537f2324/invocations"

Response:

{
  "model_class": "WorkflowInvocation",
  "id": "771990f464943137",
  "create_time": "2026-02-17T20:25:07.048443",
  "update_time": "2026-02-17T20:25:07.048453",
  "workflow_id": "608c63cc71b7d215",
  "history_id": "bbd44e69cb8906b56088eacab460fc5d",  # ← NEW history (different from target)
  "uuid": "bfc8d492-0c3e-11f1-b2e2-bc24110164aa",
  "state": "new",
  "landing_uuid": null
}

Susceptible Accessions Execution

curl -X POST -H "X-API-KEY: [YOUR_API_KEY]" -H "Content-Type: application/json" \
  -d '{
    "history": "bbd44e69cb8906b5b37974f851b5cfe1",
    "workflow_id": "b8f98b64537f2324",
    "inputs": {
      "0": {
        "src": "hda",
        "id": "f9cad7b01a472135c7e27f4dfb4144b3"
      },
      "1": 1
    }
  }' \
  "https://usegalaxy.org/api/workflows/b8f98b64537f2324/invocations"

Response:

{
  "model_class": "WorkflowInvocation",
  "id": "cb27f80447dc4781",
  "create_time": "2026-02-17T20:25:15.004570",
  "update_time": "2026-02-17T20:25:15.004579",
  "workflow_id": "608c63cc71b7d215",
  "history_id": "bbd44e69cb8906b5f06f16bc37098c96",  # ← ANOTHER new history (different from target)
  "uuid": "c480caa8-0c3e-11f1-bd3c-bc24110164aa",
  "state": "new",
  "landing_uuid": null
}

Summary of Behaviors

Consistent Pattern

  1. API Request: Always specifies "history": "bbd44e69cb8906b5b37974f851b5cfe1"
  2. Actual Behavior: Always creates new history, never uses specified target
  3. Workflow Success: Workflows execute successfully in the new histories
  4. Input Access: Workflows can access input datasets from target history despite executing in new history

History IDs Generated

Execution Target History Actual History Workflow Type
Resistant (Public) bbd44e69cb8906b5b37974f851b5cfe1 bbd44e69cb8906b5a8ca2202862977a7 Public
Susceptible (Public) bbd44e69cb8906b5b37974f851b5cfe1 bbd44e69cb8906b5642ce18146a5a534 Public
Resistant (Imported) bbd44e69cb8906b5b37974f851b5cfe1 bbd44e69cb8906b56088eacab460fc5d User-owned
Susceptible (Imported) bbd44e69cb8906b5b37974f851b5cfe1 bbd44e69cb8906b5f06f16bc37098c96 User-owned

Key Questions for Galaxy Experts

  1. Is the error state in the target history preventing workflow execution in that history?

    • Target history state: "error" with 7 datasets in error state
    • Is this documented behavior?

  2. Should user-owned workflows behave differently from public workflows regarding history targeting?

    • Both show identical behavior (creating new histories)
    • Expected: user-owned workflows should respect history parameter

  3. Are there additional API parameters needed to force execution in a specific history?

    • Current payload structure may be incomplete
    • Missing parameters for history enforcement?

  4. Is this expected behavior for workflows with collection inputs/outputs?

    • Workflow creates collections, may require new history for organization
    • Could be related to collection handling requirements

  5. How should cross-history dataset access work?

    • Workflows access input datasets from target history
    • Outputs go to new history
    • Is this the intended workflow execution model?

Technical Environment

  • Galaxy Version: usegalaxy.org (production instance)
  • API Authentication: X-API-KEY header method
  • Request Format: JSON payload via curl
  • User Account: scottcain (verified active and authenticated)
  • Workflow: "Parallel accession download ENA" (fastq-dl based, 7 steps)

Additional Context

This issue was discovered during A. fumigatus genomic analysis where maintaining all data in a single history is important for project organization and downstream analysis workflows.

Contact: This analysis was prepared for consultation with Galaxy community experts and developers to understand the root cause of workflow execution history targeting behavior.%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment