Blameless Postmortem Template

postmortem-template.md

Postmortem: Flashsale Grey Screen Problem

Flashsale grey screen in homepage, product list, product detail

Date: 2022-06-21

Authors: @Tech Sirka

Status: issue: solved, postmortem report: in progress review

Background

Flashsale feature released or deployed in the app version v4.11.3 (29-May-2023)

since then several flash sale campaign has been launched.

Problem

detail:

However this flashsale imposed issue, it showed grey screen on several pages:

1. In commerce homepage

provide screenshot

2. In product list page - Lihat Semua

provide screenshot

3. In product detail page - happened for each product with grey screen

provide screenshot

This issue happened on several products under the flashsale campaign (not all products)

Summary

After investigating the issue, we found out that for there are misunderstanding of data definition or data anomalies for each "grey products" :

• in response debug
• in gsheet master campaign (for grey products see highlighted in yellow)
• in database (same case as gsheet)
• Future enhancements -already socialized to commerce ops team @Indriani and Laela

Impact

• Users failed to checkout ➝ buy the selected product with grey screen
  • @Product Sirka please provide detail data users & how many product selected ➝ not checked out during flashsale period

Root Causes

5 Whys Analysis

1. Why did the UI broken in flashsale?

   1. possible case, invalid data handling (null, or data anomalies)
   2. defect on app mobile side
   3. defect on backend side

2. Based on three possibilities, did the issue happened only on production environment? on latest app version only?

3. Are there any differences in data format?

   1. response data comparison (no data difference in format/standard)

4. re-check on gsheet master campaign

5. What data are different? And what are the logics behind it?

   1. Data format is still the same, however found the anomaly in ...
   2. Based on the logic in code
   3. usage error due to minus percentage % (0-100%) this has caused the UI broken (grey screen)
   4. Re-test result in staging

Action Items

1. Issue discussion with commerce ops team
2. Internal debugging by engineering team
   1. Mobile team: Niki
   2. QA: Faida
   3. BE: Nanda
3. Found the root cause
4. Socialize to commerce ops team

Lessons Learned

What went well

• New feature socialization or demo (usage in client side, dynamic config, operational concern -create a new campaign or product-)

What went wrong

• Missing in detail Flashsale operational concern
  • missing out campaign detail or product operational detail
  • e.g. field definition, rule, logic, etc.

Where we got lucky

• There is workaround for this case: user still able to checkout through website sirka.io commerce

Timeline

2023-06-21 19.00-21.00 WIB

• This issue first happening

2023-06-22 09.45 WIB

• Data and operational discussion with commerce ops: Laela, Indri
• BE: Nanda
• Mobile: Niki
• QA: Faida

2023-06-22 15.00 WIB

• Found root cause
• discussion and socialization to commerce ops: Laela, Indri

Estimated Amount of Financial Impact

Total Estimated Financial Impacts Rp3,996,500

Total users affected 14

• Since when this issue actually happening or should be detected?
  • we assumed from 1-March-2023
• How many users failed to checkout using Next Day option?
  • 14 users
• How much estimated of financial impacts?
  • Rp3,996,500 of users failed to checkout during the time period (1-March ~ 13-March)

Assumption

1. The following data is calculated based on the total value of items that are still in the 'cart'.
2. The following data is calculated based on the user_id who attempted to make a payment based on event 'Payment Summary Page - Pay Now Button'.
3. Users who fail to select a delivery duration cannot be identified due to the limitations of the data we have, so assumption number 2 is used as the closest assumption to the actual conditions.