Processing and rotation of Nginx log files

The Nginx web server saves requests to the file /var/log/nginx/access.log. Every day, rotation occurs, so in the directory /var/log/nginx there are several files:

• /var/log/nginx/access.log – the current log;
• /var/log/nginx/access.log.1 – the log from the previous day;
• /var/log/nginx/access.log.2 – the log from two days ago;
• /var/log/nginx/access.log.3 and so on.

Rotation is triggered at midnight, but file sizes can vary greatly depending on the load during the day. It is required to write a utility that scans all log files (access.log and *access.log.**) in the specified directory, searches for lines matching a given pattern (for example, with the pattern “favicon.ico”, it should output all lines where “favicon.ico” appears).

Found lines should be saved to the file /var/log/nginx/parsed/lines.txt. The size of this file should not exceed 200 MB. Once the limit is reached, the file should be renamed to /var/log/nginx/parsed/lines.txt.1 (and in subsequent rotations – to .2, .3, etc.), then a new empty /var/log/nginx/parsed/lines.txt should be created, and writing should continue.

Thus, each time the 200 MB limit is reached, a new rotation of the result occurs, forming a sequential numbering of files in the parsed directory.