fix.txt

The message flow is:

  Remediation Lambdas → SNS (encrypted with CMK) → SQS → Splunk

  When you encrypt the SNS topic with a CMK, two things must happen that don't happen automatically:

  1. Lambdas Publishing to SNS

  The remediation Lambda execution roles need KMS permissions on the CMK to publish to the encrypted
  topic:

  - kms:GenerateDataKey
  - kms:Decrypt

  Without this, the Lambdas fail to publish finding status to SNS — remediation still runs, but you get
   zero alerts in Splunk. Silent failure.

  2. SNS Delivering to SQS (This is the real gotcha)

  When SNS delivers a message to an SQS subscriber, the SNS service itself must decrypt the message
  from the CMK. This requires the CMK Key Policy to grant the SNS service principal access:

  {
    "Sid": "AllowSNSToDecrypt",
    "Effect": "Allow",
    "Principal": {
      "Service": "sns.amazonaws.com"
    },
    "Action": [
      "kms:Decrypt",
      "kms:GenerateDataKey*"
    ],
    "Resource": "*"
  }

  Without this statement in the CMK key policy, SNS silently fails to deliver to SQS. No error, no
  retry, messages just vanish.

  Why the SQS Resource Policy Doesn't Help

  The SQS resource policy controls SQS actions (sqs:SendMessage, etc.). Having kms:Encrypt/kms:Decrypt
  in the SQS resource policy is irrelevant — KMS permissions are enforced by the KMS Key Policy, not by
   the SQS resource policy.

  The trust chain looks like this:

  CMK Key Policy  →  controls who can use the key
  SQS Resource Policy  →  controls who can send/receive SQS messages
                          (has NOTHING to do with KMS key access)

  What Breaks and How

  ┌──────────────────────┬─────────────────────────────────────┬──────────────────────────────────┐
  │      Component       │            What Happens             │             Visible?             │
  ├──────────────────────┼─────────────────────────────────────┼──────────────────────────────────┤
  │ Lambda → SNS publish │ Fails with KMSAccessDeniedException │ Yes, Lambda errors in CloudWatch │
  ├──────────────────────┼─────────────────────────────────────┼──────────────────────────────────┤
  │ SNS → SQS delivery   │ Silently drops messages             │ No — this is the dangerous one   │
  ├──────────────────────┼─────────────────────────────────────┼──────────────────────────────────┤
  │ SQS → Splunk         │ Never receives messages             │ Looks like "no findings"         │
  └──────────────────────┴─────────────────────────────────────┴──────────────────────────────────┘

  The worst outcome: remediation runs fine, but you think nothing is happening because Splunk gets zero
   events. You lose all audit/alerting visibility.

  Fix — What You Need

  A. CMK Key Policy (in IAM Central Account)

  {
    "Sid": "AllowSNSServiceToUseKey",
    "Effect": "Allow",
    "Principal": {
      "Service": "sns.amazonaws.com"
    },
    "Action": [
      "kms:Decrypt",
      "kms:GenerateDataKey*"
    ],
    "Resource": "*"
  },
  {
    "Sid": "AllowLambdaRolesToPublish",
    "Effect": "Allow",
    "Principal": {
      "AWS": [
        "arn:aws:iam::<ACCOUNT_ID>:role/<remediation-resource-policy-role>",
        "arn:aws:iam::<ACCOUNT_ID>:role/<remediation-non-resource-policy-role>",
        "arn:aws:iam::<ACCOUNT_ID>:role/<remediation-s3-role>"
      ]
    },
    "Action": [
      "kms:Decrypt",
      "kms:GenerateDataKey*"
    ],
    "Resource": "*"
  }

  B. Lambda IAM Roles (all 3 remediation Lambdas)

  {
    "Effect": "Allow",
    "Action": [
      "kms:Decrypt",
      "kms:GenerateDataKey*"
    ],
    "Resource": "arn:aws:kms:<region>:<ACCOUNT_ID>:key/<CMK-KEY-ID>"
  }

  C. If SQS is Also Encrypted with a CMK

  Then SNS also needs kms:GenerateDataKey and kms:Decrypt on the SQS queue's KMS key to re-encrypt the
  message when sending to SQS.

  TL;DR

  ┌────────────────────────────────────┬───────────────────────────────────────────┐
  │              Question              │                  Answer                   │
  ├────────────────────────────────────┼───────────────────────────────────────────┤
  │ Will it break?                     │ Yes, unless you update the CMK key policy │
  ├────────────────────────────────────┼───────────────────────────────────────────┤
  │ Will SQS resource policy save you? │ No — wrong trust boundary                 │
  ├────────────────────────────────────┼───────────────────────────────────────────┤
  │ Is the failure obvious?            │ No — SNS → SQS drops silently             │
  ├────────────────────────────────────┼───────────────────────────────────────────┤
  │ What's at risk?                    │ All Splunk alerting/audit goes dark       │
  └────────────────────────────────────┴───────────────────────────────────────────┘

  The fix is straightforward — grant sns.amazonaws.com and the Lambda roles access in the CMK key
  policy. Everything else stays the same.

Since SQS is Already Encrypted with the Same CMK

The delivery chain becomes:

Lambda → SNS (encrypt with CMK) → SNS decrypts (CMK) → SQS re-encrypts (same CMK)

SNS needs to decrypt the message from the topic AND re-encrypt it when pushing to the SQS queue —
both using the same CMK.

What Your CMK Key Policy Needs

Single statement covers both SNS encrypt and decrypt since it's the same key:

{
"Sid": "AllowSNSServiceToUseCMK",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey*"
],
"Resource": "*"
}

• kms:Decrypt — SNS reads from the encrypted topic
• kms:GenerateDataKey* — SNS writes to the encrypted SQS queue

Without kms:GenerateDataKey*, SNS can decrypt from the topic but cannot deliver to the encrypted SQS
queue — messages silently drop.

Quick Check — Does Your CMK Already Grant SNS Access?

Get the CMK key policy

aws kms get-key-policy
--key-id <CMK_KEY_ID>
--policy-name default
--output text | python3 -m json.tool

Look for sns.amazonaws.com in the Principal block. If it's not there, that's your problem waiting to
happen.

Also Verify the Splunk Consumer

Since SQS is encrypted with the CMK, whatever IAM role Splunk uses to poll SQS also needs kms:Decrypt
on the same CMK — but if Splunk is already reading from the encrypted SQS queue today and working
fine, that's already in place.

Summary

┌───────────────────────────────────┬───────────────────────┬────────────────────────────────────┐
│ Permission │ Why │ Without It │
├───────────────────────────────────┼───────────────────────┼────────────────────────────────────┤
│ SNS → kms:Decrypt on CMK │ Read from encrypted │ Can't deliver to subscribers │
│ │ SNS topic │ │
├───────────────────────────────────┼───────────────────────┼────────────────────────────────────┤
│ SNS → kms:GenerateDataKey* on CMK │ Write to encrypted │ Silent message drop │
│ │ SQS queue │ │
├───────────────────────────────────┼───────────────────────┼────────────────────────────────────┤
│ Lambda roles → │ Publish to encrypted │ Lambda throws │
│ kms:GenerateDataKey* on CMK │ SNS topic │ KMSAccessDeniedException │
├───────────────────────────────────┼───────────────────────┼────────────────────────────────────┤
│ Splunk role → kms:Decrypt on CMK │ Read from encrypted │ Already working if SQS is │
│ │ SQS │ encrypted today │
└───────────────────────────────────┴───────────────────────┴────────────────────────────────────┘

Same CMK means one key policy to update, but you need both Decrypt and GenerateDataKey* for the SNS
service principal — that's the part most people miss.

https://excalidraw.com/#json=k7zgt2okv0em5_o8hC3bF,8c1IMwEzXKq_K0UDETrdkg