bucket-logging.md

Basic Bucket Logging Testing

• to enable our extension to the API when using python (boto3 or aws CLI) the following file has to be placed under: ~/.aws/models/s3/2006-03-01/ (the directory should be created if it dioes not exist)
• currently there is no generic solution for other client SDKs

• start a vstart cluster
• create a bucket:

aws --endpoint-url http://localhost:8000 s3 mb s3://fish

• create a log bucket:

aws --endpoint-url http://127.0.0.1:8000 s3 mb s3://all-logs

• create a matching log bucket policy file (policy.json):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3ServerAccessLogsPolicy",
            "Effect": "Allow",
            "Principal": {
                "Service": "logging.s3.amazonaws.com"
            },
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::all-logs/logs",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:s3:::fish"
                },
                "StringEquals": {
                  "aws:SourceAccount": "testid"
                }
            }
        }
    ]
}

• add policy to log bucket:

aws --endpoint-url http://127.0.0.1:8000 s3api put-bucket-policy --bucket all-logs --policy file://policy.json

• create bucket logging rule on bucket:

aws --endpoint-url http://localhost:8000 s3api put-bucket-logging --bucket fish --bucket-logging-status '{"LoggingEnabled": {"TargetBucket": ":all-logs", "TargetPrefix": "fish/"}}'

• upload/download files to/from the bucket (assuming myfile1/2/3/4/5 exist i nthe directory):

aws --endpoint-url http://localhost:8000 s3 cp myfile1 s3://fish
aws --endpoint-url http://localhost:8000 s3 cp myfile2 s3://fish
aws --endpoint-url http://localhost:8000 s3 cp myfile3 s3://fish
aws --endpoint-url http://localhost:8000 s3 cp myfile4 s3://fish
aws --endpoint-url http://localhost:8000 s3 cp myfile5 s3://fish
aws --endpoint-url http://localhost:8000 s3 cp s3://fish/myfile1 .
aws --endpoint-url http://localhost:8000 s3 cp s3://fish/myfile2 .
aws --endpoint-url http://localhost:8000 s3 cp s3://fish/myfile3 .
aws --endpoint-url http://localhost:8000 s3 cp s3://fish/myfile4 .
aws --endpoint-url http://localhost:8000 s3 cp s3://fish/myfile5 .
aws --endpoint-url http://localhost:8000 s3 ls s3://fish

• flush the logs (you must use out boto3 extension file for that command):

aws --endpoint-url http://localhost:8000 s3api post-bucket-logging --bucket fish

• list objects on the log bucket:

aws --no-cli-pager --endpoint-url http://localhost:8000 s3api list-objects --bucket all-logs

e.g. result:

{
    "Contents": [
        {
            "Key": "fish/2024-10-08-11-16-51-M14V9SSPFWX8YGF3",
            "LastModified": "2024-10-08T11:22:03.786000+00:00",
            "ETag": "\"99a2c3f71c17420be10e59cfdbf4e453\"",
            "Size": 1304,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "testid"
            }
        },
        {
            "Key": "fish/2024-10-08-11-22-03-55QTC8W16UL53YK4",
            "LastModified": "2024-10-08T11:24:56.492000+00:00",
            "ETag": "\"90b7929fd4e3bebffcb4c1b7de795306\"",
            "Size": 2410,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "testid"
            }
        }
    ],
    "RequestCharged": null
}

• view one of the log objects (the format is described here):

aws --no-cli-pager --endpoint-url http://localhost:8000 s3api get-object --bucket all-logs --key "fish/2024-10-08-11-16-51-M14V9SSPFWX8YGF3" tmp && cat tmp