Protect Personally Identifiable Data with CloudWatch Log Masking

Logging is one of the essential components of software development. We use different types of logs to identify issues in applications, track application events, analyze the performance of applications etc. Even though logs are very beneficial, they also introduce a huge risk to your business. For example, logging sensitive data of the customers, exposing security credentials with logs etc are some of the serious issues.

In this post, I will show how to use the CloudWatch log masking feature to protect personally identifiable (PII) data such as email address, date of birth, zip code etc in CloudWatch logs.

Use Case

We have an application that logs employee name, email, date of birth and zip code into CloudWatch. So we need to mask all these sensitive data when the log is written. Also, we want to notify a Slack channel when data protection operations happen.

Steps

• Create a log group for application logs
• Create a log group for audit findings destination
• Create a data protection policy for the application log group
• Setup a CloudWatch alarm to monitor data protection operations
• Setup AWS ChatBot to notify a Slack channel
• Testing

Create a log group for application logs

Our application sends all the logs to this CloudWatch log group.

resource "aws_cloudwatch_log_group" "employee_logs" {
  name              = "employee-logs"
  retention_in_days = 7
}

Create a log group for audit findings destination

When we set up the data protection policy in the next step, we can specify a destination to send audit findings. Currently, we can specify destinations such as log groups, Amazon Kinesis Data Firehose streams and S3 buckets. In this example, we use a log group as the destination.

resource "aws_cloudwatch_log_group" "employee_audit_finding_logs" {
  name              = "employee-finding-audit-logs"
  retention_in_days = 7
}

Create a data protection policy for the application log group

In this step, we define a data protection policy and attach it to the CloudWatch log group (employee-logs). CloudWatch uses this policy to mask sensitive data. In this example, we only consider personally identifiable data. But you can find more details about other types in the documentation.

You can attach one data protection policy to a log group. Each data protection policy can audit multiple types of information.

Data Protection Policy

As you can see in the below code, the data protection policy consists of two main blocks.

First Block

The first block must have a DataIdentifer array and an Operation property with Audit action.

DataIdentifer: define the types of data that you want to mask.

Operation with Audit: find the sensitive data terms.

Second Block

The second block must have a DataIdentifer array and an Operation property with Deidentify action.

DataIdentifer: must exactly match the DataIdentifer array in the first block.

Operation with Deidentify: masks the data, and it must contain the "MaskConfig": {} object.

data "aws_cloudwatch_log_data_protection_policy_document" "log_data_protection_policy_document" {
  name = "employee-logs-protection-policy"
  statement {
    sid = "Audit"
    data_identifiers = local.pii_data_identifiers
    operation {
      audit {
        findings_destination {
          cloudwatch_logs {
            log_group = aws_cloudwatch_log_group.employee_audit_finding_logs.name
          }
        }
      }
    }
  }
  statement {
    sid              = "Deidentify"
    data_identifiers = local.pii_data_identifiers
    operation {
      deidentify {
        mask_config {}
      }
    }
  }
}

locals {
  pii_data_identifiers = [
    "arn:aws:dataprotection::aws:data-identifier/Name",
    "arn:aws:dataprotection::aws:data-identifier/Address",
    "arn:aws:dataprotection::aws:data-identifier/DateOfBirth",
    "arn:aws:dataprotection::aws:data-identifier/EmailAddress",
    "arn:aws:dataprotection::aws:data-identifier/ZipCode-US",
  ]
}

Attach data protection policy to CloudWatch log group.

resource "aws_cloudwatch_log_data_protection_policy" "log_data_protection_policy" {
  log_group_name  = aws_cloudwatch_log_group.employee_logs.name
  policy_document = data.aws_cloudwatch_log_data_protection_policy_document.log_data_protection_policy_document.json
}

Setup a CloudWatch alarm to monitor data protection operations

In this step, we set up a CloudWatch alarm to monitor data protection operations in the application log group (employee-logs). This alarm will be triggered when the application sends any sensitive data to the CloudWatch log group. We can use CloudWatch LogEventsWithFindings as the metric for this.

We use an SNS topic as the alarm action and the next step AWS ChatBot will use this topic for sending alarms to a slack channel.

resource "aws_cloudwatch_metric_alarm" "log_events_with_pii_findings" {
  alarm_name          = "log-events-with-pii-findings"
  comparison_operator = "GreaterThanThreshold"
  period  = 60
  evaluation_periods = 1
  metric_name = "LogEventsWithFindings"
  namespace = "AWS/Logs"
  
  dimensions = {
    DataProtectionOperation = "Audit"
    LogGroupName = aws_cloudwatch_log_group.employee_logs.name
  }

  datapoints_to_alarm = 1
  statistic = "Sum"
  alarm_actions = [aws_sns_topic.slack_notification.arn]
}

resource "aws_sns_topic" "slack_notification" {
  name = "employee-audit-logs-notification"
  display_name = "Employee Audit Log Notification"
}

Setup AWS ChatBot to notify a Slack channel

First, we need to configure AWS ChatBot with a Slack workspace. This part has to be done manually as it requires some manual actions. So please follow the instructions given in the documentation.

Once we configured the ChatBot, we can configure a Slack channel to receive alarm notifications. In this step, we use the same SNS topic that we used for alarm actions in the previous step.

Please refer to the source code for complete implementation.

resource "awscc_chatbot_slack_channel_configuration" "slack_channel_config" {
  configuration_name = var.configuration_name
  iam_role_arn       = var.iam_role_arn != "" ? var.iam_role_arn : aws_iam_role.chatbot_default_iam_role.arn
  slack_channel_id   = var.slack_channel_id
  slack_workspace_id = var.slack_workspace_id
  sns_topic_arns     = var.sns_topic_arns
  logging_level      = var.logging_level
  guardrail_policies = var.guardrail_policy_arns
}

resource "aws_iam_role" "chatbot_default_iam_role" {
  assume_role_policy = data.aws_iam_policy_document.chatbot_trust_policy.json
}

resource "aws_iam_role_policy" "chatbot_default_iam_role_policy" {
  policy = data.aws_iam_policy_document.chatbot_default_permission_policy.json
  role   = aws_iam_role.chatbot_default_iam_role.id
}

Testing

You can download the complete code from this repository.

Sample log event 1

{

  "fullName": "Nick Jhon",
  "dateOfBirth": "1984-06-19",
  "email": "nickjon@gmail.com",
  "country": "USA"
  "zipcode": "99501"

}

Sample log event 2

His name is Mark Anthony and his dob is 1984–03–12. His email is marka@gmail.com

Masked log events

Triggered alarm

Audit logs

Slack notification

References

Help protect sensitive log data with masking

Create a data protection policy for a log group

Personally identifiable information (PII)

Terraform Registry