- Thread Author
- #1
Types and modules of logs. Apache access log format.
Table of contents
1. Types and modules of journals. Apache access log format[/B]
1.1 Types of Apache logs
1.2 Apache log modules
1.3 Mod_log_config module
1.4 Access Log
1.5 How to customize the Apache log format. Custom Log Formats
1.6 BufferedLogs Directive
1.7 CustomLog Directive
1.8 GlobalLog Directive
1.9 LogFormat Directive
1.10 TransferLog Directive
1.11 Apache Log Formats
1.11.1 Common Log Format
1.11.2 Combined Log Format
1.11.3 Multiple Access Logs (Multiple Access Logs)
1.11.4 Conditional Logs
1.12 Rotating Logs
1.13 Piped Logs
1.14 Virtual Hosts
1.15 Security Issues
2. Format of error logs. Module event log
3. Programs for analyzing Apache logs
4. Forensic logs
5. Additional configurable debug logs. CGI script execution logs
Note: this series of articles is devoted to the log files (logs) of the Apache web server, their configuration, format, commands, and also special programs for analyzing the web server logs are considered. The information is presented in detail for a deep study of the topic, as well as for use as a reference material. For more concise information, it is recommended that you refer to the article "Apache log (logs): how to configure and analyze web server logs".
To effectively manage your web server, you need to get feedback on the activity and performance of the server, as well as any issues that may arise. Apache HTTP Server provides very rich and flexible logging capabilities.
Apache HTTP Server provides many different mechanisms for logging everything that happens to your server, from the initial request and the URL mapping process, to the final resolution of the connection, including any errors that may have occurred in the process. In addition to this, third-party modules can provide logging capabilities or insert records into existing log files, and applications such as CGI programs, PHP scripts, or other handlers can also send messages to the server error log.
The web server logs contain tons of interesting information! Using the server access logs, you can compose a collective portrait of the audience: in which countries and cities they live, what operating systems they use, which browsers they view the site, what time they are most active, from which sites they came to you, which search engines prefer, how many pages are viewed for every visit to the site. And no less important are logs for monitoring the state of the web server and sites: which pages were not found, web server errors, the degree of congestion, detecting bot activity, detecting malicious activity, searching for traces of hacking, identifying hacking paths.
In general, server access logs should be understood, configured, and used primarily by the webmasters and system administrators serving the server. At the same time, an attacker, or someone who is investigating the consequences of an attacker's actions, also needs to understand what exactly is saved in the web server logs, how they can benefit from them, or how to disguise their traces, or how to analyze access log files for searching. problems, attacks and traces of hacking.
The first part is about configuring the format of the logs in the Apache web server. This information will be useful to you even if you do not have sites and you are not engaged in maintaining a web server. You will need it in order to be able to analyze web logs using tools (as a rule, they need to specify the format of the analyzed log, for which the same specifiers are used as in the Apache config).
Types of Apache logs
Different types of Apache logs are managed by different web server modules and have different control directives and the ability to specify the format of the log string.
The following types of Apache web server logs are available:
Error Log
The server error log is the most important log file. This is where Apache httpd will send diagnostic information and record any errors it encounters while processing requests. This is the first place to look when a server startup or server problem occurs, as it often contains details of what went wrong and how to fix it.
Per-module logging
The LogLevel directive allows you to specify the log severity level for each module. Thus, if you are troubleshooting a problem with only one specific module, you can increase its size in the log, while not getting unnecessary information about other modules that you are not interested in. This is especially useful for modules like mod_proxy or mod_rewrite where you want to know the details of what it is trying to do.
Access Log
The server access log records all requests processed by the server.
Additional configurable debug logging
This directive causes a custom message to be logged in the error log. The message can use variables and functions from the ap_expr syntax. References to HTTP headers do not result in header names being added to the Vary header. Messages are logged at the log level.
Forensic (forensic logs)
Logging is done before and after the request is processed, so the forensic journal contains two journal lines for each request. Differs in increased severity.
CGI script execution logs
If ScriptLog is not specified, no error log is generated. If ScriptLog is set, then any CGI errors are logged in the file specified as an argument.
Apache log modules
Apache has several modules that are responsible for web logs:
The mod_log_config module provides flexible logging of client requests. Logs are written in a customizable format and can be written directly to a file or to an external program. Conditional logging is provided, that is, individual queries can be included or excluded from the log based on the characteristics of the query. This module is key to making the Access Log work .
This module supports the following directives:
Access Log
The server access log records all requests processed by the server. The location and contents of the access log are controlled by the CustomLog directive. The LogFormat directive can be used to simplify the selection of log content. This section describes how to configure the server to write information to the access log.
Of course, keeping information in the access log is just the beginning of log management. The next step is to analyze this information to obtain useful statistics. The analysis of logs in general is not part of the work of the web server itself, but will be discussed in a subsequent article in this series.
Various versions of Apache httpd used different modules and directives to manage the access log, including mod_log_referer, mod_log_agent, and the TransferLog directive. The CustomLog directive now includes the functionality of all the old directives.
The access log format is highly customizable. The format is specified using a format string, which is very similar to the C-style printf (1) format string. Some examples are provided below. For a complete list of the possible contents of the format string, see the next section, How to Customize the Format of Apache Access Logs. Custom log formats”.
Table of contents
1. Types and modules of journals. Apache access log format[/B]
1.1 Types of Apache logs
1.2 Apache log modules
1.3 Mod_log_config module
1.4 Access Log
1.5 How to customize the Apache log format. Custom Log Formats
1.6 BufferedLogs Directive
1.7 CustomLog Directive
1.8 GlobalLog Directive
1.9 LogFormat Directive
1.10 TransferLog Directive
1.11 Apache Log Formats
1.11.1 Common Log Format
1.11.2 Combined Log Format
1.11.3 Multiple Access Logs (Multiple Access Logs)
1.11.4 Conditional Logs
1.12 Rotating Logs
1.13 Piped Logs
1.14 Virtual Hosts
1.15 Security Issues
2. Format of error logs. Module event log
3. Programs for analyzing Apache logs
4. Forensic logs
5. Additional configurable debug logs. CGI script execution logs
Note: this series of articles is devoted to the log files (logs) of the Apache web server, their configuration, format, commands, and also special programs for analyzing the web server logs are considered. The information is presented in detail for a deep study of the topic, as well as for use as a reference material. For more concise information, it is recommended that you refer to the article "Apache log (logs): how to configure and analyze web server logs".
To effectively manage your web server, you need to get feedback on the activity and performance of the server, as well as any issues that may arise. Apache HTTP Server provides very rich and flexible logging capabilities.
Apache HTTP Server provides many different mechanisms for logging everything that happens to your server, from the initial request and the URL mapping process, to the final resolution of the connection, including any errors that may have occurred in the process. In addition to this, third-party modules can provide logging capabilities or insert records into existing log files, and applications such as CGI programs, PHP scripts, or other handlers can also send messages to the server error log.
The web server logs contain tons of interesting information! Using the server access logs, you can compose a collective portrait of the audience: in which countries and cities they live, what operating systems they use, which browsers they view the site, what time they are most active, from which sites they came to you, which search engines prefer, how many pages are viewed for every visit to the site. And no less important are logs for monitoring the state of the web server and sites: which pages were not found, web server errors, the degree of congestion, detecting bot activity, detecting malicious activity, searching for traces of hacking, identifying hacking paths.
In general, server access logs should be understood, configured, and used primarily by the webmasters and system administrators serving the server. At the same time, an attacker, or someone who is investigating the consequences of an attacker's actions, also needs to understand what exactly is saved in the web server logs, how they can benefit from them, or how to disguise their traces, or how to analyze access log files for searching. problems, attacks and traces of hacking.
The first part is about configuring the format of the logs in the Apache web server. This information will be useful to you even if you do not have sites and you are not engaged in maintaining a web server. You will need it in order to be able to analyze web logs using tools (as a rule, they need to specify the format of the analyzed log, for which the same specifiers are used as in the Apache config).
Types of Apache logs
Different types of Apache logs are managed by different web server modules and have different control directives and the ability to specify the format of the log string.
The following types of Apache web server logs are available:
- Error Log
- Per-module logging
- Access Log
- Additional configurable debug logging
- Forensic (forensic logs)
- CGI script execution logs
Error Log
The server error log is the most important log file. This is where Apache httpd will send diagnostic information and record any errors it encounters while processing requests. This is the first place to look when a server startup or server problem occurs, as it often contains details of what went wrong and how to fix it.
Per-module logging
The LogLevel directive allows you to specify the log severity level for each module. Thus, if you are troubleshooting a problem with only one specific module, you can increase its size in the log, while not getting unnecessary information about other modules that you are not interested in. This is especially useful for modules like mod_proxy or mod_rewrite where you want to know the details of what it is trying to do.
Access Log
The server access log records all requests processed by the server.
Additional configurable debug logging
This directive causes a custom message to be logged in the error log. The message can use variables and functions from the ap_expr syntax. References to HTTP headers do not result in header names being added to the Vary header. Messages are logged at the log level.
Forensic (forensic logs)
Logging is done before and after the request is processed, so the forensic journal contains two journal lines for each request. Differs in increased severity.
CGI script execution logs
If ScriptLog is not specified, no error log is generated. If ScriptLog is set, then any CGI errors are logged in the file specified as an argument.
Apache log modules
Apache has several modules that are responsible for web logs:
- mod_log_config. Keeps a log of requests made to the server. This is the main module, which is enabled by default and it is he who stores information about requests. Basically, here we will consider this particular module and its settings. Provides the Access Log.
- mod_log_debug. Additional configurable debug logs. Enables Additional configurable debug logging. Has experimental status.
- mod_log_forensic. Forensic registration of requests to the server. Provides Forensic (forensic logs).
- mod_logio. Registration of input and output bytes of each request. This module must be included in the Apache configuration if you want to log information about the amount of transmitted and / or received data. Provides the functionality of the Access Log format.
- Apache Core Features - Apache HTTP Server core features that are always available. Including provides the operation of Error Log (error log) and Per-module logging (logging of module events).
- mod_cgi and mod_cgid. Provides the operation of the CGI script execution log.
The mod_log_config module provides flexible logging of client requests. Logs are written in a customizable format and can be written directly to a file or to an external program. Conditional logging is provided, that is, individual queries can be included or excluded from the log based on the characteristics of the query. This module is key to making the Access Log work .
This module supports the following directives:
- TransferLog to create a log file,
- LogFormat to set a custom format. This directive is followed by a log format string, as well as a name that can be used as an alias for this string. After setting an alias with this directive, it can be specified in CustomLog.
- CustomLog to define log file and format in one step. Specifies how the log file is saved (for example, to a file) and the format to use. The format can be either an alias set using the LogFormat directive, or a format string.
- BufferedLogs. Keep log entries in memory before writing to disk
- GlobalLog. Sets the file name and format of the log file
Access Log
The server access log records all requests processed by the server. The location and contents of the access log are controlled by the CustomLog directive. The LogFormat directive can be used to simplify the selection of log content. This section describes how to configure the server to write information to the access log.
Of course, keeping information in the access log is just the beginning of log management. The next step is to analyze this information to obtain useful statistics. The analysis of logs in general is not part of the work of the web server itself, but will be discussed in a subsequent article in this series.
Various versions of Apache httpd used different modules and directives to manage the access log, including mod_log_referer, mod_log_agent, and the TransferLog directive. The CustomLog directive now includes the functionality of all the old directives.
The access log format is highly customizable. The format is specified using a format string, which is very similar to the C-style printf (1) format string. Some examples are provided below. For a complete list of the possible contents of the format string, see the next section, How to Customize the Format of Apache Access Logs. Custom log formats”.