Design Decisions

How to divide server responsibilities

Stateless playlist API, logical owner partitions, dedicated WebSocket gateways, and asynchronous fanout workers.

Request servers scale with HTTP traffic, owners scale with mutation partitions, gateways scale with connection count, and workers scale with event fanout. A failure in one role has a bounded effect.

A small deployment can combine these roles in one service while preserving the same logical boundaries.

Which database holds canonical playlist state

Relational primary sharded by playlist_id. Playlist, membership, entries, version, operation deduplication, mutation history, and outbox rows share one shard.

One transaction enforces authorization, ordering, stable identity, version advancement, and publication intent. Indexed range scans serve ordered entries efficiently.

• Document store: compact reads for small lists with larger rewrites and document-size constraints.
• Key-value store: high scale with application-managed secondary access and multi-record transaction logic.
• Pure event store: rich history with an additional materialized view and membership transaction design.

How to publish a committed mutation reliably

Transactional outbox in SQL followed by Kafka publication. The mutation and its unpublished event commit together.

A direct SQL-plus-Kafka dual write can leave durable state and broadcast state inconsistent after a partial failure. The outbox publisher retries safely, and consumers deduplicate by event ID.

Kafka delay postpones fanout while the playlist commit remains durable. Reconnect replay restores clients after publication resumes.

How to accelerate reads

Versioned Redis snapshots for hot public playlists, with SQL as durable authority.

Popular playlists create repeated identical reads. A cache projector consumes committed events, advances only contiguous versions, and can rebuild from SQL.

Collaborative clients carry a version cursor. The read service uses a cache entry at least as recent as that cursor or reads the primary shard.

How to identify a list item

Stable playlist-entry ID. Every add creates a new occurrence identity. Move and remove address that entry ID.

Track identity permits duplicates. Index identity changes after every reorder. A stable occurrence identity survives movement and uniquely targets deletion.

How to represent order

Fractional position keys between stable neighbors. A move updates one entry in the common case. A background pass rebalances crowded key ranges.

• Dense integers: simple queries with write amplification on insertion and movement.
• Linked neighbors: small mutations with more complex pagination and integrity repair.
• List CRDT: strong multi-replica convergence and richer offline editing with greater protocol cost.

How to handle concurrent writes

Serialize per playlist and rebase operations by stable identity. The owner shard accepts a mutation against a base version, validates current targets, and assigns the next version.

Concurrent inserts survive. Delete dominates a later move of the same entry. Concurrent moves follow canonical server order. The acknowledgement returns the accepted placement.

How to deliver updates

HTTP mutation plus WebSocket event stream. Both transports carry the operation ID and resulting playlist version.

Each client persists its last applied version. Reconnect replays later events or supplies a fresh snapshot when replay is impractical.

How to maintain history

Append mutation log plus materialized snapshot. The log supports replay, audit, repair, and optional undo. The snapshot makes playlist opening and pagination efficient.

Periodically create a validated snapshot at version N and expire earlier log records according to audit and restore policy.