You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Feature Request #21591: Batch export of histories, workflow invocations, and datasets
URL: galaxyproject/galaxy#21591Author: M Bernt (@bernt-matthias)
Created: 2026-01-15
Labels: feature-request
Reactions: 1 thumbs up
Description
It would be great to be able to export multiple histories, invocations, or datasets at once.
For histories I could imaging that users select all histories they want to export and then the same export dialog opens where a destination can be selected. It would be perfectly sufficient for my needs to have a separate export job per history. But one could also imagine that users might want to have all histories in a single export.
Invocations currently miss the possibility to select multiple ones. But I could imagine a similar approach as for histories is feasible.
For datasets we can only download single datasets at the moment, i.e. there is no export (eg. to a file source). Selecting multiple datasets and having the possibility to export them to a file source could be a nice thing to have as well.
# Model store export with file sourceuser_context=ModelStoreUserContext(self._app, user)
export_store=model.store.get_export_store_factory(
self._app, model_store_format, export_files=export_files, user_context=user_context
)(target_uri)
# In lib/galaxy/celery/tasks.py@galaxy_task(action="batch export histories to file source")defbatch_write_histories_to(model_store_manager, request: BatchWriteHistoriesTo, ...):
forhistory_idinrequest.history_ids:
# Export each history to target_uri/{history_name}.{format}
Backend - New API Endpoints
# In lib/galaxy/webapps/galaxy/api/histories.py@router.post("/api/histories/batch_write_store")defbatch_write_store(self, payload: BatchWriteStoreToPayload) ->AsyncTaskResultSummary:
...
# In lib/galaxy/webapps/galaxy/api/workflows.py@router.post("/api/invocations/batch_write_store")defbatch_write_store(self, payload: BatchWriteInvocationsStoreToPayload) ->AsyncTaskResultSummary:
...
# In lib/galaxy/webapps/galaxy/api/history_contents.py (or datasets.py)@router.post("/api/datasets/batch_write_store")defbatch_write_store(self, payload: BatchWriteDatasetsToPayload) ->AsyncTaskResultSummary:
...
Long-standing need: Multiple independent requests over 9+ years
Workaround adoption: Users actively using collection-based workarounds (create collection then download)
Training/GTN demand: #16974 shows trainers need simpler export workflows
VGP collaboration need: #20269 demonstrates real production use case for complex exports
User Quotes
"It would be really useful if I could group these histories to one project... To my knowledge there is no option to see and share already in the 'All histories' view this." - @joachimwolff (#8378)
"I find myself a bit frustrated by having to do imports one at a time... And for history tarball, I want to upload multiple potentially, not just one." - @hexylena (#10695)
"Given how hard it is to initially train users to just learn how to share their histories via a link, pointing out yet another place with scary words like RO-Crate... is really tough." - @nomadscientist (#10695)
"This came up on our side today too" - @hexylena on download throttling (#2968)
Pain Points Identified
Time-consuming manual process: Users report 4+ minutes per history to share/export
Scalability issues: Users with 14+ project histories struggle with current workflow
Training barrier: Non-developers find current export tools intimidating
No unified interface: Export scattered across multiple menus/tools
Feature Request Patterns
Component 1: Batch History Export
Direct requests: #21591, #8378, #15345
Related: #14771 (archiving), #5389 (archival)
Estimated demand: MEDIUM-HIGH
Component 2: Batch Invocation Export
Direct requests: #21591, #21592
Related: #20269, #16974
Estimated demand: MEDIUM
Component 3: Dataset Export to File Sources
Direct requests: #21591
Related: #3195, #18245, #2968
Estimated demand: HIGH (oldest and most requested)
Duplicate/Related Issues
Potential Duplicates
#3195 vs #18245: Both request multi-dataset download
#8378 vs #15345: Both request batch history operations
Should Consider Closing/Linking
If #21591 is implemented, could partially address:
#3195 (multi-download - if export to local added)
#18245 (download selected datasets)
#8378 (batch history operations)
Parts of #2968 (more download options)
Implementation Considerations
Existing Workarounds
Create collection from selected datasets, download collection as ZIP
Use export_remote tool for individual datasets
Manual one-by-one export
Technical Foundation
History export to file sources already exists
Invocation export exists (RO-Crate, BCO)
Export datasets tool exists (but individual only)
Multi-select UI patterns exist in history panel
Overall Demand Assessment
Aspect
Rating
Justification
User Interest
MEDIUM
1 reaction but new issue; related issues show consistent demand
Longevity
HIGH
Related requests span 9+ years
Core Team Interest
HIGH
Core maintainer (@mvdbeek) immediately supported
Related Activity
HIGH
Multiple related issues, active discussions
Workaround Availability
MEDIUM
Collections partially address download need
Implementation Complexity
MEDIUM
Foundation exists but UI/UX work needed
OVERALL DEMAND: MEDIUM-HIGH
Rationale: While the specific issue is new with limited direct engagement, the underlying need has been expressed repeatedly over 9 years across multiple issues. The thumbs-up from a core maintainer suggests technical alignment with project direction. The existence of workarounds (collections) indicates users actively want this functionality. The feature fits within Galaxy's ongoing efforts to improve export/archival capabilities (see #14771 implementation).
Recommendations
Link related issues: Connect #21591 to #3195, #18245, #2968, #8378 for visibility
Scope definition: Consider implementing in phases:
Phase 1: Batch dataset export to file sources
Phase 2: Batch history export
Phase 3: Batch invocation export
Close duplicates: After implementation, close older related issues
Promote issue: Given long-standing demand, consider adding to project roadmap
classBatchHistoryWriteStoreToPayload(StoreExportPayload):
history_ids: List[DecodedDatabaseIdField]
target_uri: str# Base directory URI - each history written as subfile# Inherits from StoreExportPayload:# - model_store_format: ModelStoreFormat (tar.gz, rocrate.zip)# - include_files: bool# - include_deleted: bool# - include_hidden: bool
Response
classBatchExportResult(Model):
task_ids: List[str] # One celery task UUID per historytotal_count: int
In HistoryList.vue, add to existing bulk operations dropdown alongside "Delete Selected", "Purge Selected":
<!-- In HistoryList.vue operations dropdown -->
<BDropdownItem @click="exportSelected">
<FontAwesomeIcon :icon="faFileExport" class="mr-1" />
Export Selected to File Source
</BDropdownItem>
How Selection Works
Already implemented via useSelectedItems composable:
Checkbox selection per history
Shift+click range select
"Select All" button
Selection state stored in selectedItems ref
Export Flow
User selects multiple histories via checkboxes
Clicks "Export Selected to File Source" in dropdown
HistoryExportWizard opens in batch mode
Wizard shows:
Count of selected histories
File source destination picker
Export format options (model_store_format, include_files, etc.)
User confirms, API called
Toast notification with link to file source destination
2.4 Key Implementation Steps
Schema & Models (backend)
Add BatchHistoryWriteStoreToPayload to schema.py
Add BatchWriteHistoriesTo to tasks.py
Celery Task (backend)
Add batch_write_histories_to task that iterates history_ids
Each history exported to {target_uri}/{name}_{id}.{format}
Return aggregated result
Service & API (backend)
Add batch_write_store() to HistoriesService
Add endpoint to histories.py router
API Client (frontend)
Add batchExportHistoriesToFileSource() to histories.export.ts
Wizard Modification (frontend)
Update HistoryExportWizard.vue props to accept historyIds: string[]
Show batch preview step when multiple IDs provided
Call batch endpoint when in batch mode
HistoryList Integration (frontend)
Add dropdown item
Wire up to open wizard with selected IDs
3. Testing Strategy
3.1 Unit Tests
Test
Location
Description
Schema validation
test/unit/schema/
Validate BatchHistoryWriteStoreToPayload
Service method
test/unit/webapps/
Mock manager, test batch_write_store service
API serialization
test/unit/webapps/api/
Test endpoint response format
3.2 Integration Tests
Test
Location
Description
Batch export task
test/integration/
Full celery task execution with mock file source
API endpoint
test/integration/
POST /api/histories/batch_write_store with auth
File source write
test/integration/
Verify files written to destination
Recommended test file: test/integration/test_batch_history_export.py
deftest_batch_export_to_file_source(self):
# Create 3 test histories# Configure test file source# POST to batch endpoint# Verify 3 files created at destination# Verify naming conventiondeftest_batch_export_partial_failure(self):
# Create histories where one will fail (e.g., no access)# Verify other exports succeed# Verify error reported for failed onedeftest_batch_export_empty_list(self):
# POST with empty history_ids# Expect 400 Bad Requestdeftest_batch_export_unauthorized(self):
# Attempt to export histories user doesn't own# Expect 403 for those histories
3.3 Manual Testing Scenarios
Happy path: Select 3 histories, export to configured S3 file source, verify all 3 archives created
Mixed access: Select history you own + shared history, verify appropriate handling
Large batch: Select 10+ histories, verify performance acceptable
This feature requests batch export capabilities for histories, invocations, and datasets to file sources. While the issue itself is new (1 reaction), demand analysis reveals this is a 9+ year request with 19+ reactions across related issues (#3195, #2968, #8378). The recommended approach is phased implementation starting with batch history export (~1 week), which has the best existing infrastructure (selection UI, export wizard) and clearest user story (archive all project histories at once). Core maintainer @mvdbeek has already signaled support.
Importance Assessment Summary
Dimension
Rating
Notes
User Demand
Medium-High
1 direct reaction, but 19+ across related 9-year-old issues
Strategic Value
Medium-High
Aligns with file source ecosystem, reproducibility goals
Effort
Large
~3-4 weeks total (3 features), ~1 week for Phase 1 only
Risk
Medium
Security (rate limiting) and performance (bulk I/O) need attention
Priority
Backlog
Good feature, phased approach can deliver value incrementally
Key Questions for Discussion
Scope for v1: Should we start with batch history export only, or commit to all three features?
Error handling: If one history fails to export, should the batch abort or continue with others?
Rate limiting: What's an acceptable max batch size? (Suggest 50, configurable)
Progress feedback: Is a simple toast notification sufficient, or do users need a progress bar?
File naming: Use {history_name}_{id}.{format} to avoid collisions, or let users choose structure?
Related issues: Should we link/close older duplicates (#3195, #8378, #18245) when implemented?
Concerns
Scope Creep
The request covers three distinct features. Risk of expanding scope to include:
Combined export (all histories in single archive)
Scheduled exports
Export templates
Mitigation: Explicitly scope v1 to batch-to-directory pattern, one file per item.
Long-term Maintenance
Batch operations add complexity to error handling, progress tracking, and resource management
Each export type (history, invocation, dataset) needs testing against all file source types
Performance
Large batch exports could strain object store I/O and network bandwidth
Need configurable limits and potentially staggered execution
Recommended Next Steps
Label: Add backlog (or enhancement if actively considering)
Link: Connect to related issues #3195, #2968, #8378, #18245
Scope decision: Confirm Phase 1 (batch history export) as starting point
Clarify questions: Get input on error handling, rate limits before implementation
Assign: Good candidate for contributor with Galaxy UI/API experience
