Ihor Hadzera

Do you have a backup of your database? Is it stored in safe place? Do you have a plan B if someone will delete your database and backup by accident or intentionally?

This is the list of questions I faced after recent security breach. Let’s imagine a situation when we have a database in cloud and someone accidentally removed that database and all backups/snapshot. Sounds like impossible disaster right? Well, even if it is pretty hard to do accidentally, that doesn’t mean it’s impossible. Or if someone “from outside” will get an access to your cloud prod environment and intentionally remove your database, oh, that sounds more realistic. Well, all we know that due to Moore’s law that question is rather “When?” and not “What if?”.

Regardless of scenario, it is always good plan to have a plan B? So what we can do with that – probably the most obvious scenario is to save the database snapshots in a different place than your database lives. Yes, I think it is a good option. Nevertheless, it is always good to remember that every additional environment, framework or technology will require additional engineering time to support.

Another way is that we can to restrict anyone (regardless of permissions) from removing database snapshots. So even if someone intentionally will want to remove database, it is always a possibility of (relatively) quick restore. Fortunately Amazon S3 bucket provides a possibility to set object lock for S3 bucket. There are 2 retention modes for object lock in AWS S3 service (AWS doc source):

• Governance mode
• Compliance mode

Governance allow to delete objects if you have a special permissions and Compliance mode does not allow to remove this object at all (even for root user) for particular period of time.

In addition to that, recently AWS introduced possibility to export RDS database snapshots to S3 bucket (AWS doc source). From AWS Console we can easily do that by clicking to Export to Amazon S3 button:

So we can combine Exporting Snapshots and Non-Deletable files to protect database snapshots from deleting.

The last thing, is that even if it is pretty simple to do manually through AWS Console, it is always better to have such important process automated, so our database will be safely stored even when we are chilling on the beach during our long waited summer holidays 🏝 . For doing that, we can subscribe to RDS Snapshot creating events and via lambda execution initiate exporting newly created snapshot to non-deletable S3 bucket.

The architecture of that will look like that:

Bellow you could find serverless framework file for creating that infrastructure:

service:
 name: rds-s3-exporter

plugins:
 - serverless-pseudo-parameters
 - serverless-plugin-lambda-dead-letter
 - serverless-prune-plugin

provider:
 name: aws
 runtime: nodejs14.x
 timeout: 30
 stage: dev${env:USER, env:USERNAME}
 region: eu-west-1
 deploymentBucket:
   name: ${opt:stage, self:custom.default-vars-stage}-deployments
 iamRoleStatements:
   - Effect: Allow
     Action:
       - "KMS:GenerateDataKey*"
       - "KMS:ReEncrypt*"
       - "KMS:GenerateDataKey*"
       - "KMS:DescribeKey"
       - "KMS:Encrypt"
       - "KMS:CreateGrant"
       - "KMS:ListGrants"
       - "KMS:RevokeGrant"
     Resource: "*"
   - Effect: Allow
     Action:
       - "IAM:Passrole"
       - "IAM:GetRole"
     Resource:
       - { Fn::GetAtt: [ snapshotExportTaskRole, Arn ] }
   - Effect: Allow
     Action:
       - ssm:GetParameters
     Resource: "*"
   - Effect: Allow
     Action:
       - sqs:SendMessage
       - sqs:ReceiveMessage
       - sqs:DeleteMessage
       - sqs:GetQueueUrl
     Resource:
       - { Fn::GetAtt: [ rdsS3ExporterQueue, Arn ] }
       - { Fn::GetAtt: [ rdsS3ExporterFailedQ, Arn ] }
   - Effect: Allow
     Action:
       - lambda:InvokeFunction
     Resource:
       - "arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${opt:stage}-rds-s3-exporter"
   - Effect: Allow
     Action:
       - rds:DescribeDBClusterSnapshots
       - rds:DescribeDBClusters
       - rds:DescribeDBInstances
       - rds:DescribeDBSnapshots
       - rds:DescribeExportTasks
       - rds:StartExportTask
     Resource: "*"

 environment:
   CONFIG_STAGE: ${self:custom.vars.configStage}
   REDEPLOY: "true"
custom:
 stage: ${opt:stage, self:provider.stage}
 region: ${opt:region, self:provider.region}
 default-vars-stage: ppe
 vars: ${file(./vars.yml):${opt:stage, self:custom.default-vars-stage}}
 version: ${env:BUILD_VERSION, file(package.json):version}
 rdsS3ExporterQ: ${self:custom.stage}-rds-s3-exporter
 rdsS3ExporterFailedQ: ${self:custom.stage}-rds-s3-exporterFailedQ
 databaseSnapshotCreatedTopic: ${self:custom.stage}-database-snapshotCreated
 rdsS3ExporterBucket: "${self:custom.stage}-database-snapshot-backups"

functions:
 backup:
   handler: dist/functions/backup.main
   reservedConcurrency: ${self:custom.vars.lambdaReservedConcurrency.backup}
   timeout: 55
   events:
     - sqs:
         arn: "arn:aws:sqs:#{AWS::Region}:#{AWS::AccountId}:${self:custom.rdsS3ExporterQ}"
         batchSize: 1
   environment:
     CONFIG_STAGE: ${self:custom.vars.configStage}
     DATABASE_BACKUPS_BUCKET: ${self:custom.rdsS3ExporterBucket}
     IAM_ROLE: "arn:aws:iam::#{AWS::AccountId}:role/${opt:stage}-rds-s3-exporter-role"
     KMS_KEY_ID: alias/lambda
     REGION: "eu-west-1"

resources:
 Description: Lambda to handle upload database backups to S3 bucket
 Resources:
   rdsS3ExporterQueue:
     Type: AWS::SQS::Queue
     Properties:
       QueueName: "${self:custom.rdsS3ExporterQ}"
       MessageRetentionPeriod: 1209600 # 14 days
       RedrivePolicy:
         deadLetterTargetArn:
           Fn::GetAtt: [ rdsS3ExporterFailedQ, Arn ]
         maxReceiveCount: 5
       VisibilityTimeout: 60
   rdsS3ExporterFailedQ:
     Type: "AWS::SQS::Queue"
     Properties:
       QueueName: "${self:custom.rdsS3ExporterFailedQ}"
       MessageRetentionPeriod: 1209600 # 14 days
   databaseSnapshotCreatedTopic:
     Type: AWS::SNS::Topic
     Properties:
       TopicName: ${self:custom.databaseSnapshotCreatedTopic}
   snapshotCreatedTopicQueueSubscription:
     Type: "AWS::SNS::Subscription"
     Properties:
       TopicArn: arn:aws:sns:#{AWS::Region}:#{AWS::AccountId}:${self:custom.databaseSnapshotCreatedTopic}
       Endpoint:
         Fn::GetAtt: [ rdsS3ExporterQueue, Arn ]
       Protocol: sqs
       RawMessageDelivery: true
     DependsOn:
       - rdsS3ExporterQueue
       - databaseSnapshotCreatedTopic

   snapshotCreatedRdsTopicSubscription:
     Type: "AWS::RDS::EventSubscription"
     Properties:
       Enabled: true
       EventCategories : [ "creation"]
       SnsTopicArn : arn:aws:sns:#{AWS::Region}:#{AWS::AccountId}:${self:custom.databaseSnapshotCreatedTopic}
       SourceType : "db-snapshot"
     DependsOn:
       - databaseSnapshotCreatedTopic

   rdsS3ExporterQueuePolicy:
     Type: AWS::SQS::QueuePolicy
     Properties:
       Queues:
         - Ref: rdsS3ExporterQueue
       PolicyDocument:
         Version: "2012-10-17"
         Statement:
           - Effect: Allow
             Principal: "*"
             Action: [ "sqs:SendMessage" ]
             Resource:
               Fn::GetAtt: [ rdsS3ExporterQueue, Arn ]
             Condition:
               ArnEquals:
                 aws:SourceArn: arn:aws:sns:#{AWS::Region}:#{AWS::AccountId}:${self:custom.databaseSnapshotCreatedTopic}
  
   rdsS3ExporterBucket:
     Type: AWS::S3::Bucket
     DeletionPolicy: Retain
     Properties:
       BucketName: ${self:custom.rdsS3ExporterBucket}
       AccessControl: Private
       VersioningConfiguration:
         Status: Enabled
       ObjectLockEnabled: true
       ObjectLockConfiguration:
         ObjectLockEnabled: Enabled
         Rule:
           DefaultRetention:
             Mode: COMPLIANCE
             Days: "${self:custom.vars.objectLockRetentionPeriod}"
       LifecycleConfiguration:
         Rules:
           - Id: DeleteObjectAfter31Days
             Status: Enabled
             ExpirationInDays: ${self:custom.vars.expireInDays}
       PublicAccessBlockConfiguration:
         BlockPublicAcls: true
         BlockPublicPolicy: true
         IgnorePublicAcls: true
         RestrictPublicBuckets: true
       BucketEncryption:
         ServerSideEncryptionConfiguration:
           - ServerSideEncryptionByDefault:
               SSEAlgorithm: AES256

   snapshotExportTaskRole:
     Type: AWS::IAM::Role
     Properties:
       RoleName: ${opt:stage}-rds-s3-exporter-role
       Path: /
       AssumeRolePolicyDocument:
         Version: '2012-10-17'
         Statement:
           - Effect: Allow
             Principal:
               Service:
                 - rds-export.aws.internal
                 - export.rds.amazonaws.com
             Action:
               - "sts:AssumeRole"
       Policies:
           - PolicyName: ${opt:stage}-rds-s3-exporter-policy
             PolicyDocument:
               Version: '2012-10-17'
               Statement:
                 - Effect: Allow
                   Action:
                     - "s3:PutObject*"
                     - "s3:ListBucket"
                     - "s3:GetObject*"
                     - "s3:DeleteObject*"
                     - "s3:GetBucketLocation"
                   Resource:
                     - "arn:aws:s3:::${self:custom.rdsS3ExporterBucket}"
                     - "arn:aws:s3:::${self:custom.rdsS3ExporterBucket}/*"

package:
 include:
   - dist/**
   - package.json
 exclude:
   - "*"
   - .?*/**
   - src/**
   - test/**
   - docs/**
   - infrastructure/**
   - postman/**
   - offline/**
   - node_modules/.bin/**

Now it’s time to add lambda handler code:

import { Handler } from "aws-lambda";
import { v4 as uuidv4 } from 'uuid';
import middy = require("middy");
import * as AWS from "aws-sdk";

export const processEvent: Handler<any, void> = async (request: any) => {
	console.info("rds-s3-exporter.started");

	const record = request.Records[0];
	const event = JSON.parse(record.body);

	const rds = new AWS.RDS({region: process.env.REGION});
	await rds.startExportTask({
		ExportTaskIdentifier: `database-backup-${uuidv4()}`,
		SourceArn: event["Source ARN"],
		S3BucketName: process.env.DATABASE_BACKUPS_BUCKET || "",
		IamRoleArn: process.env.IAM_ROLE || "",
		KmsKeyId: process.env.KMS_KEY_ID || ""
	}).promise().then(data =>{
		console.info("rds-s3-exporter.status", data);
		console.info("rds-s3-exporter.success");
	})
};

export const main = middy(processEvent)
	.use({
		onError: (context, callback) => {
			console.error("rds-s3-exporter.error", context.error);
			callback(context.error);
		}
	})

As there is no built-in possibility to create a flexible filter of databases we want to export, it is always possible to add some custom filtering in the lambda execution itself. You could find example of such logic as well as all codebase in GitHub repository.

Right before publication this post I’ve found that AWS actually recently (Oct 2021) has implemented AWS Backup Vault Lock which does the same thing out of the box. You could read more about that at AWS Doc website. Nevertheless, at the time of publication this post, AWS Backup Vault Lock has not been certified by third-party organisations SEC 17a-4(f) and CFTC.