Log analysis
Parses 5,000 synthetic security events and summarises them into an incident report with four tables.
Security · Python · Browsable demo
The problem
Any server writes thousands of lines a day, and an attack hides right there, in the noise. The first job of a security operations centre is not responding: it is separating signal, counting failed attempts and ranking addresses by persistence.
What it does
- Generates a synthetic log from a fixed seed, mixing normal and malicious traffic, so the published report always matches the published log.
- Reads every line with regular expressions and classifies the event type.
- Ranks the five addresses with the most failed logins and counts events by type, including the lines it could not read.
- Writes a threat report in Markdown, with not a single external dependency.
What was decided while building it
- Counting and formatting are two different things
- The figures are computed in one function and the report is laid out in another. When they lived together, the only way to reach a number was to parse it back out of the finished text, so the dashboard and the report each recounted the file on their own and drifted apart the moment a rule changed.
- An unreadable line is a finding, not a gap
- Lines the parser does not understand are counted and published. Dropping them silently is what turns «I could not read half the log» into «there are no threats», and those two sentences look very alike in a report and nothing alike in reality.
- Fixed clock and seed, so the report matches the log
- The time window used to start from the current clock, so every run produced a different file and the committed report stopped matching the committed log. The start date and the seed are now fixed: the same code produces the same log and the same report.
- Not a single dependency
- The whole analysis runs on the standard library: regular expressions and counters. A log analyser that has to be installed is a log analyser that does not run on the machine where the problem is, which is usually exactly where it is needed.
How far it goes
- The five thousand events are synthetic and generated by the repository itself from a fixed seed. There is not one line from a real network, and the addresses are invented private-range ones.
- Brute-force events arrive already labelled by the generator: the analysis counts them, it does not discover them. What is genuinely computed from the data is the ranking of addresses with the most failed logins.
- It understands its own log format and nothing else. A real syslog, a Linux journal or a Windows event log would land entirely in the unreadable-lines counter.
- There are no alerts, no configurable thresholds and no connection to a firewall or an event manager. The output is a report and some data files.
Built with
- Python
Where the data comes from
Synthetic data generated by the repository itself: no client information involved.