Serverless ETL for Post-Acquisition Compliance

Following a merger, compliance teams must audit all incoming user databases to ensure compliance with strict privacy regulations (GDPR/CCPA). Doing this manually or via a single server instance is slow and expensive. We built a fully serverless, scalable ETL pipeline to automate legal discovery and sanitize data.

Pipeline Orchestration with AWS Step Functions

AWS Step Functions coordinated our serverless flow: processing databases exported from target systems, sanitizing PII (Personally Identifiable Information), and loading compliant structures into corporate data lakes.

{
  "StartAt": "TriggerBatchScanner",
  "States": {
    "TriggerBatchScanner": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:BatchScanner",
      "Next": "IsScanComplete"
    },
    "IsScanComplete": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.status",
          "StringEquals": "COMPLETED",
          "Next": "EncryptDataLake"
        }
      ],
      "Default": "WaitTenSeconds"
    }
  }
}

The serverless setup scaled seamlessly to process over 40 million rows in under 20 minutes, reducing server maintenance costs to zero.