aws-event-store.md

AWS Lambda Event Sourcing Architecture

I'd like to have a AWS lambda / event system that allows for a certain kind of flexability, here's what I need.

• Events come in via HTTP post webhook with headers and body
  1. Store {headers, body} in one main events table where the event is named what the action is e.g. 'orderCreate', this acts as a ledger of events, with the main columns name, data, created_at.
  2. I'd like to build up the state of individual object, in their own table so for orders a table would exist with that name the event of a orderCreated the row is created and subsiquent orderDeleted the row is deleted.
  3. I need a way to tap into this event and trigger some side-effect event, for instance if a orderCreated comes in to the system I also need to fire something when that comes in such as invoiceCreated I need a way to access the stream of all events and subscribe to the orderCreated and fire a invoiceCreated which ideally should repeat this process all over again.

The database technology really shouldn't matter, the queuing and messaging system does. In the example described above there is only a real need to have the following lambdas.

• event-process where events come in via HTTP or some queueing system and stores the data in the initial events table from there each event needs to enter the queue
• event-state where events come in from the queue and CRUD operations are set accross different tables and databases example would be if orderCreated comes in an order is inserted into the orders table and if an orderDeleted event comes in the order with the specified id is deleted. Object state is maintained here.
• create-invoice-on-order a side-effect lambad listening to the orderCreated event, creates and invoice and then emits it's own invoiceCreated event back into the queue, and the cycle begins again.

Questions

• What AWS service should I use for this queue?
• How do you process events and allow the possibilities for many side-effects to operate without removing things from the queue?

Cascading Events

Let's say we've got a new Service Webhook that came into a Lambda via a HTTP POST when an Order was Created for a specific shop.

We can split this one event into different streams.

• serviceWebhook
• serviceWebhookShopName
• serviceWebhookShopNameOrderCreate

Now you can have three streams for each action getting more specific as you go down the list.

An alternative to this idea could be just to fire the event for the top level serviceWebhook and then have a your Lambda filter out when to run based on the payload. So if this hook fires for ShopAlpha and ShopBeta the payload has a speicific property {"shop": "ShopAlpha"} you can just throw an error if it's not the one you want and continue on with your intended Lambda operation.

There's a split between where the logic lies, and cost.

1. Multiple arbitrary data pipes. (cost per pipe)
2. Having the lambda fire and filter out events it doesn’t need to take action for. (cost per running functions unnecessarily)

What AWS service should I use for this queue?

SQS Aws' simple queue service.

How do you process events and allow the possibilities for many side-effects to operate without removing things from the queue?

Read about the visibility window for SQS messages. Basically for SQS message when it's read by a client it become not visible for a fixed period of time (extension is possible) then you either delete it to remove the message from the queue or leave it alone to become visible again.

Now thinking about side effect operating on things sounds like making multiple queue might also be a better option to simplify the programming model such that every side-effect operation might have it's own queue.

@JefStat thanks for your comment!

I guess there’s two camps.

1. events should be fired and many lambdas can tap into them
2. events should be deliberate and fired to trigger a specific lambda

I just added a bit above about cascading events.

If I'm understand correctly your in camp one. Where this Order Create event should add a message into the queue to do exactly what it needs for instance triggerEmailCustomer and have these items enter the queue, and be removed.

Another thing, you can't currently trigger a lambda based on a SQS message AFAIK.