Introduction

In the labyrinth of modern cloud infrastructure, effective log management is a cornerstone for ensuring operational security and compliance. However, small and medium-sized enterprises often find themselves at a crossroads — balancing budget constraints against the need for comprehensive log analysis tools. Even larger organizations aren't spared, as they battle against restrictive API throttling and limited log retention policies enforced by major cloud providers such as AWS and Azure.

Introducing CloudTail

CloudTail is an open-source tool designed to enhance the long-term retention and searchability of cloud logs. It offers the flexibility to selectively preserve a curated subset of significant cloud events from AWS and Azure, all without the need for expensive SIEM solutions.

Key Features

Multi-Cloud Support:

Compatible with various cloud providers including AWS and Azure.

Config-Based Event Filtering

Allows users to specify and retrieve detailed events using a config-based approach that interacts seamlessly with native cloud APIs.

Flexible Data Storage

Stores logs in their raw format locally while extracting and normalizing key properties to facilitate easier and more powerful searching capabilities.

Automated and Scheduled Execution

Designed to be run as a scheduled task, maintaining an execution history to ensure consistent log collection without duplicate events.

How CloudTail Works

CloudTail utilizes a JSON configuration file that offers a straightforward way for users to specify exactly how they want to monitor cloud events. Within this file, users can fine-tune the parameters to focus on specific events or expand their scope to encompass a broader range of activities across AWS and Azure. This flexibility allows users to tailor the monitoring to specific needs, whether they are tracking a narrow set of actions or overseeing a wide array of cloud operations.

The default configuration file we provide contains a curated list of high-value events that we have identified as most important for security operations. These include key events related to user management, resource changes, and security alerts. However, users can easily modify this configuration to suit their specific needs, adding more detailed or customized event filters as required.

The JSON configuration file in CloudTail is designed for precision in event monitoring across multiple cloud platforms including AWS and Azure:

• Data Sources Configuration: Users define which cloud environments to monitor by listing specific data sources such as AWS CloudTrail or Azure Activity Logs. Each entry can support multiple accounts or subscriptions, offering a scalable approach for organizations operating across various cloud services.
• Event Filtering Mechanics: Within each data source, the file allows users to determine key event attributes through lookup_Attributes. This is essential for identifying which events to capture, focusing on specific attributes like EventName to ensure only pertinent data is collected, optimizing both resource usage and security monitoring efforts.
• Rule-based Event Selection: The configuration supports the creation of rules that specify how events are filtered and captured. For instance, setting a rule to monitor all CreateUser events helps organizations keep tabs on new user account creations, a vital element in user management and security protocols.
• Local Storage and Data Normalization: Once events are retrieved, they are stored locally in their original format and also undergo a normalization process. This approach not only maintains the integrity of the original data but also simplifies later searches and analysis, boosting both the usability and investigative capabilities of the system.

Usage Examples

Example #1: Detecting S3 Activity

Monitor Amazon S3 activities by setting EventSource as the AttributeKey with AttributeValue set to s3.amazonaws.com. This configuration hones in on S3-related events.

Example #2: Tracking Deletion Events

To capture all deletion-related events, utilize wildcard matching (e.g. Delete*). This is crucial for overseeing critical changes and maintaining security.

Example #3: Filtering User Creation Events

For monitoring specific user creation events, use EventName as the AttributeKey with AttributeValue set to CreateUser. Include CloudTrail’s client-side JMESPath filtering to focus on particular users, enhancing your monitoring capabilities.

Output

CloudTail processes and stores the extracted event data in two SQLite databases (aws_events.db for AWS and azure_events.db for Azure), ensuring that event metadata is readily available for further analysis or reporting. In addition to storing events in SQLite databases, CloudTail allows you to export the processed events as JSON files for easier viewing and external use.

Conclusion

Permiso Security is continually hard at work to show our customers the most important activity across all of their identity and cloud environments, always linking the context of the originating identity (human or non-human). We hope CloudTail helps many organizations to simplify their multi-cloud log collection and retention process as a first step in ensuring they retain the highest-value events relevant to their security needs.

You can access CloudTail on Github: https://github.com/Permiso-io-tools/cloudtail