Logging in AWS: Security and Compliance

AWS provides a host of services designed to help organizations manage their security and compliance, particularly through logging capabilities. These services allow you to track, store, and analyze all activity within your AWS environment. This blog post will explore some key services that you should be aware of: CloudTrail, Config, CloudWatch, VPC Flow Logs, ELB/ALB/NLB Access Logs, CloudFront Logs, and WAF Logs. We'll also discuss how to use Athena to analyze these logs stored in S3.

AWS CloudTrail

CloudTrail is a service that logs all API calls made in your AWS account. By enabling CloudTrail, you can trace all activity, such as who made a request, the services used, the actions performed, and the parameters for the actions. It offers continuous monitoring and post-event forensics to analyze operational and security incidents.

AWS Config

AWS Config is a fully managed service that gives you an AWS resource inventory, configuration history, and configuration change notifications. It simplifies compliance auditing, security analysis, change management, and operational troubleshooting. By defining Config rules, you can continuously track changes over time and ensure your resources are in compliance with your policies.

CloudWatch Logs

CloudWatch Logs help you to monitor, store, and access your log files from Amazon EC2 instances, AWS CloudTrail, and other sources. It enables you to centralize the logs from all your systems, applications, and AWS services that you use, in a single, highly scalable service. It is perfect for an environment that requires full data retention for various operational or regulatory compliance needs.

VPC Flow Logs

VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your Virtual Private Cloud (VPC). This allows you to have a deep understanding of the network traffic patterns and inspect traffic anomalies within your environment.

Load Balancer Access Logs

Access logs for Elastic Load Balancers, whether they're Application Load Balancers (ALBs), Network Load Balancers (NLBs), or Classic Load Balancers (ELBs), capture detailed information about requests made to your load balancer. This includes the time a request was received, client's IP address, latencies, request paths, and server responses. By analyzing access logs, you can learn about traffic patterns and identify issues.

CloudFront Logs

CloudFront logs are web distribution access logs that provide detailed records about every user request that CloudFront receives. These logs include information such as the requester's IP address, the object that was requested, the response code that CloudFront returned, and more.

AWS WAF Logs

WAF, or Web Application Firewall, helps protect your applications from common web exploits. With WAF logs, you can store and monitor the full logging of all your requests analyzed by the service, which provides insights into traffic patterns and potential threats.

Analyzing Logs with Athena

With all these logs stored in Amazon S3, AWS Athena can be used to analyze these logs. Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.

Here is an example of how you can use Athena to analyze your logs:

Query for CloudTrail Logs

Detecting root account usage:

SELECT useridentity.type, eventtime, eventsource, eventname, sourceipaddress
FROM cloudtrail_logs
WHERE useridentity.type = 'Root'
AND eventtime > '2023-01-01T00:00:00Z';

This query retrieves all activities performed by the root account after January 1, 2023.

Query for VPC Flow Logs

Detecting unusual data transfer:

SELECT interfaceid, sourceaddress, destinationaddress, SUM(bytes) as total_bytes
FROM vpc_flow_logs
WHERE starttime BETWEEN 1596240000 AND 1596326399
GROUP BY interfaceid, sourceaddress, destinationaddress
HAVING SUM(bytes) > 1000000;

This query checks for any IP pairs that have transferred more than 1,000,000 bytes of data in a specified time range, which might indicate unusual or suspicious data transfer.

Query for GuardDuty Findings

Detecting threats with GuardDuty:

SELECT type, COUNT(*) as count
FROM guardduty_logs
WHERE updatedat > '2023-01-01T00:00:00Z'
GROUP BY type
ORDER BY count DESC;

This query returns the types and counts of threats detected by GuardDuty since January 1, 2023, ordered by the number of occurrences.

Query for Access Logs

Detecting failed access attempts:

SELECT requestip, requesturi, statuscode, COUNT(*) as count
FROM access_logs
WHERE statuscode = '403'
AND time > '2023-01-01T00:00:00Z'
GROUP BY requestip, requesturi, statuscode
ORDER BY count DESC;

This query checks for IP addresses and URIs that have had multiple failed access attempts (403 status code), which might indicate a potential brute-force attack.

These queries can provide valuable insights for security monitoring and incident response in your AWS environment. Please note that these queries are examples and may need to be adjusted based on the structure of your logs and your specific use cases.

Conclusion

Logging is a critical component of maintaining security and compliance in the cloud. AWS provides a wealth of services to help you manage your logging needs, from capturing API calls with CloudTrail, maintaining configuration histories with Config, to tracking IP traffic within your VPC via VPC Flow Logs, and many more. Moreover, with Athena, you can conveniently analyze these logs using standard SQL, making it a powerful tool for deriving insights from your log data.

It's important to understand and effectively use these services to ensure that your AWS environment is secure, compliant, and optimally performing. Remember, logging not only helps in identifying and troubleshooting issues but also aids in improving the overall performance and efficiency of your cloud infrastructure. So, start exploring these services and make the most out of your AWS environment.