Back to Projects
Cloud Infrastructure
Last updated June 2026Cloud Log Aggregation Pipeline
Event-driven AWS pipeline that ingests, validates, and indexes service logs for querying
Overview
Logs scattered across services are hard to search, and processing them synchronously means a burst of events can back up or drop data.
Producer services publish structured log events to an SQS queue. A Lambda function drains the queue, validates and normalizes each event, archives the raw payload to S3, and writes queryable metadata to DynamoDB behind a REST API for filtering by service, level, and time range.
Architecture
Log Producers
SQS Queue
Lambda Processor
S3 (raw) + DynamoDB (index)
REST Query API
Engineering Challenges
- Decoupling ingestion from processing so a burst of log events doesn't block producers — SQS buffers events and Lambda drains the queue asynchronously.
- Keeping durable storage and queryable metadata separate — raw events are archived as individual S3 objects (the source of truth), while DynamoDB indexes them by service, level, and time for fast lookups.
- Isolating malformed events — invalid payloads are routed to a dead-letter queue instead of blocking the rest of the batch.
Design Decisions
- SQS over Kafka: matches the AWS-native architecture and keeps operations simple at the throughput this project targets, instead of running a Kafka cluster for a portfolio-scale workload.
- DynamoDB as an index in front of S3 as the source of truth, rather than one system doing both — keeps queries fast without duplicating the full event payload.
Reliability & Scalability
- Failed or invalid events go to a dead-letter queue for inspection instead of being silently dropped.
- Local development runs the full pipeline against LocalStack; a SAM template defines the path to a real AWS deployment (IAM roles, Lambda packaging, CloudWatch).
Technology
Python
AWS SQS
AWS Lambda
Amazon S3
DynamoDB
FastAPI