Distributed Log Analyzer
Go master-worker system that parses a 3.3 GB, 10M-line access log. 36s single worker, 6s at 8-11 workers.
A Go master-worker log parser that cut a 10M-line job from 36 s to 6 s, with the scaling ceiling measured rather than guessed.
- Role
- Solo: designed and built
- Timeframe
- Jan 2025
- repo created 2025-01-08
- Team
- 1
- Problem
- Parsing a 3.3 GB access log on one core takes long enough to be worth distributing, but distributing it badly makes it slower.
- Approach
- A master splits the file into byte ranges and hands them to workers over gRPC; workers parse and return status-code counts the master merges. Profiled and tuned buffers to find where the speedup stops.
- Result
- Near-linear scaling to about 8 workers, flat after that because the job becomes I/O and coordination bound. Adding workers past the knee makes it slower.
Every figure above is measured or read from the running system. Where a figure has a source line, that is how and when it was read.
Technology Stack
What it does
A master process splits a large web server access log into ranges and hands them to worker nodes over gRPC. Workers parse their range and return HTTP status code counts, which the master merges.
The numbers above are measured on a real 3.3 GB / 10 million line log, not projected: 36 seconds with one worker, 6 seconds with 8 to 11 workers.
Where the speedup stops
Scaling is close to linear up to roughly 8 workers and then flattens. Past that point the job is no longer parse-bound. It is bound by reading the file and by the coordination chatter of handing out and collecting ranges.
The interesting part of this project was not writing the parser, it was finding that ceiling with profiling and buffer tuning instead of guessing at it. Adding workers past the knee makes the run slower, which is the kind of result that only shows up if you actually measure.