19-06 - Discussion.md

Sharding Config, Queue - Creation and Deletion on Topic Operations

Configuration Decisions

Sharding Level

• Config level rather than topic level
• Provides consistency across all topics

Number of Shards

• Keep num_shards configurable at config level
• Maintain constant number for simplicity (no resharding)

Topic Operations

Topic Creation

Topic Already Exists (Persistent)

• Sharded → Sharded: Do nothing
• Non-sharded → Sharded: Create new shards
  • Question: How to handle existing queue?
• Sharded → Non-sharded: Create single queue
  • Question: Do we support this? How to handle existing shards?
• Non-sharded → Non-sharded: Do nothing

New Topic

• Persistent sharded: Create shards (handled)
• Persistent non-sharded: Create single queue (handled)
• Non-persistent: Standard handling (handled)

Topic Deletion

• Topic doesn't exist: No change (handled)
• Non-persistent topic: No change (handled)
• Persistent topic: Delete queues/shards (handled)

Attribute Changes

• Non-persistent → Persistent (sharded): Initialize shards
• Non-persistent → Persistent (non-sharded): Initialize queue (handled)
• Persistent (sharded) → Persistent (non-sharded): Question: Support this?
• Persistent (non-sharded) → Persistent (sharded): Create shards
  • Question: How to handle existing queue?

Open Questions

1. Data Migration: How to handle existing queues when transitioning sharding states?
2. Configuration: Use rgw_yaml.in for config-level params?
3. Config Location: Keep shard options at rgw_pubsub_dest level?
4. Request Lifecycle: What's the function trigger order?