Home
Forums
New posts
Search forums
What's new
New posts
New profile posts
Latest activity
Members
Current visitors
New profile posts
Search profile posts
Log in
Register
What's new
Search
Search
Search titles only
By:
New posts
Search forums
Menu
Log in
Register
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Reply to thread
Home
Forums
CARDING & HACKING
HOSTING & BOTNET
Apache logs 2023
Message
<blockquote data-quote="Cupper" data-source="post: 578" data-attributes="member: 22"><p>If you have a special format that does not fit any of the above, then you can configure the processing of any format in the configuration file, for this see the section " How to configure goaccess.conf".</p><p></p><p>In order to collect statistics on the countries that accessed the site (geolocation), you need to specify the path to the GeoIP <strong>database</strong> with the <strong>--geoip-database</strong> option, for example <strong>GeoLiteCity.dat</strong> or <strong>GeoLite2-City.mmdb</strong> .</p><p></p><p>If GeoIP2 is used, you need to download the <strong>GeoLite2-City.mmdb</strong> or <strong>GeoLite2-Country.mmdb database</strong>. These databases can be downloaded from the MaxMind.com website - download is free, but requires obtaining an API, so you need to register on the site - all this is free.</p><p></p><p>So, my large combined Apache log is located in the <strong>biglog.txt</strong> file, it is in <strong>COMBINED</strong> format, I want to save the generated report to the <strong>logs_report.html</strong> file and use geolocation using the <strong>GeoLite2-City.mmdb</strong> database <strong>for analysis</strong>, then the command is as follows:</p><p>Code:</p><p>cat biglog.txt | goaccess - --log-format = COMBINED --output = logs_report.html --geoip-database = GeoLite2-City.mmdb</p><p></p><p>You can open the generated report in any browser:</p><p>Code:</p><p>firefox logs_report.html</p><p></p><p>Example of a report:</p><p></p><p>If you are interested in a detailed description of each item, then see the article "Why and How to Analyze Web Server Logs".</p><p></p><p>Various output formats can be specified: <strong>-o --output = <path / file. [Json | csv | html]></strong>:</p><ul> <li data-xf-list-type="ul">/ path / file. <strong>csv</strong> - Comma Separated Values (CSV)</li> <li data-xf-list-type="ul">/ path / file. <strong>json</strong> - JSON (JavaScript Object Notation)</li> <li data-xf-list-type="ul">/ path / file. <strong>html</strong> - HTML</li> </ul><p>That is, the format is determined by the file extension, so you can specify any name, and the file extension can be selected from one of the three presented.</p><p></p><p>To analyze referrers (referring sites), you can exclude the analyzed site itself, as well as various incorrect values, this is done with the <strong>--hide-referer</strong> option, which can be used many times:</p><p>Code:</p><p>cat biglog.txt | goaccess - --log-format = COMBINED --output = logs_report.html --hide-referer = "hackware.ru" --hide-referer = "-" --geoip-database = GeoLite2-City.mmdb</p><p></p><p>If search engines are not of interest among the referring sites, then they can also be added to the exclusions:</p><p>Code:</p><p>cat biglog.txt | goaccess - --log-format = COMBINED --output = logs_report.html --hide-referer = "hackware.ru" --hide-referer = "-" --hide-referer = "* google *" --hide -referer = "* yandex *" --geoip-database = GeoLite2-City.mmdb</p><p></p><p>Another option that improves the readability of the results is <strong>-d</strong> or the long version <strong>--with-output-resolver</strong>, this option enables the conversion of IP addresses to host names, only works for HTML and JSON formats.</p><p></p><p>Please note that when using the <strong>-d</strong> option, a large number of DNS lookups are performed and the generation of the log file may be slower.</p><p></p><p>By the way, you can use the online service GoAccess to analyze web server logs: <a href="https://suip.biz/?act=goaccess" target="_blank">https://suip.biz/?act=goaccess</a></p><p></p><p>This service accepts log files as unpacked text files or in .gz archives.</p><p></p><p>It says that this is an Apache log analyzer, but in fact, any log format that GoAccess supports is accepted.</p><p></p><p><strong>LORG</strong></p><p>LORG - Apache log file security analyzer, is a tool for advanced security analysis of HTTPD logs. It aims to implement various modern approaches to detecting web application attacks in HTTP traffic logs (such as Apache access logs (access_log files)), including signature-based, statistics, and machine learning techniques. Detected incidents are subsequently grouped into sessions that are classified as “manual” or automated to determine if the attacker is human or machine. In addition, geo-targeting and DNSBL lookups can be performed to see if attacks are originating from a specific geolocation or botnet. Additionally, attacks can be quantified in terms of success or failure based on anomalies within the size of HTTP responses.</p><p></p><p>A detailed description of LORG, a complete list of options and installation instructions can be found on this page: <a href="https://kali.tools/?p=4852" target="_blank">https://kali.tools/?p=4852</a></p><p></p><p>Run command:</p><p>Code:</p><p>./lorg OPTIONS input_file [output_file]</p><p></p><p>I will use the following options in the command:</p><ul> <li data-xf-list-type="ul"><strong>-i input format</strong>. Options: <strong>common combined vhost logio cookie</strong></li> <li data-xf-list-type="ul"><strong>-o output format</strong>. Variants: <strong>html json xml csv</strong></li> <li data-xf-list-type="ul"><strong>-u</strong> perform URL decode for encoded requests (only affects reports)</li> <li data-xf-list-type="ul"><strong>-g</strong> enable geotag</li> </ul><p>The Apache log file is located in the <strong>~ / access_log file</strong>, I want to save the report to the current folder in a file named <strong>report.htm</strong>, then the command is as follows:</p><p>Code:</p><p>./lorg -u -i combined -g -o html ~ / access_log report.htm</p><p></p><p>The report can be opened in a web browser:</p><p>Code:</p><p>firefox report.htm</p><p><strong>How to edit log formats in LORG</strong></p><p>In fact, the format of my web server log does not fit any of the suggested LORG (<strong>common combined vhost logio cookie</strong>) formats. The format of my file is very similar to <strong>combined,</strong> with the difference that at the end of the line is the hostname (site domain). You can edit the existing supported log formats or add your own. To do this, open the executable file of the program:</p><p>Code:</p><p>gedit ./lorg</p><p></p><p>We find the lines there:</p><p>Code:</p><p>static $ allowed_input_types = array (</p><p> 'common' => '% h% l% u% t \ "% r \"%> s% b',</p><p> 'combined' => '% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \"',</p><p> 'vhost' => '% v% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \"',</p><p> 'logio' => '% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \% I% O"' ,</p><p> 'cookie' => '% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \" \ "% {Cookie } i \ "'</p><p>);</p><p></p><p>To these lines, I will add the new format of my Hostland host:</p><p>Code:</p><p>'combined_hostland' => '% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \"% v',</p><p></p><p>It turned out like this:</p><p><img src="https://sun9-23.userapi.com/impg/c858532/v858532275/1e4dde/Gem3zGUU8ds.jpg?size=807x182&quality=96&sign=6fc72a81aca521e61312450741680bad&type=album" alt="Gem3zGUU8ds.jpg" class="fr-fic fr-dii fr-draggable " style="" /></p><p></p><p></p><p></p><p>We save and close the file.</p><p></p><p>I run the command again to analyze the logs, but in this case I specify <strong>combined_hostland</strong> as the format type:</p><p>Code:</p><p>./lorg -u -i combined_hostland -g -o html ~ / biglog.txt report2.htm</p><p></p><p>Although there are almost 6 million entries in the biglog.txt file, the analysis was pretty quick.</p><p></p><p><img src="https://sun9-70.userapi.com/impg/c858532/v858532275/1e4de5/8R1vaRBNBMg.jpg?size=389x129&quality=96&sign=1c73ff001a16a8e4c90641227a92ef19&type=album" alt="8R1vaRBNBMg.jpg" class="fr-fic fr-dii fr-draggable " style="" /></p><p></p><p></p><p>When the program terminates, it displays generalized statistics - how many incidents were found and how many users are involved in them:</p><p></p><p><img src="https://sun9-63.userapi.com/impg/c858532/v858532275/1e4ded/GM8_7YQg7L0.jpg?size=758x368&quality=96&sign=dc97468a619e67bc8625ba601d8735ba&type=album" alt="GM8_7YQg7L0.jpg" class="fr-fic fr-dii fr-draggable " style="" /></p><p></p><p></p><p>Open the generated report:</p><p>Code:</p><p>firefox report2.htm</p><p></p><p>Above is a diagram with summarized information:</p><p><img src="https://sun9-11.userapi.com/impg/c858532/v858532275/1e4df6/s0k4MVg-8Jw.jpg?size=807x600&quality=96&sign=7ad609439317d1c687029aa71d27b90e&type=album" alt="s0k4MVg-8Jw.jpg" class="fr-fic fr-dii fr-draggable " style="" /></p><p></p><p></p><p>Detailed information can be viewed for each incident:</p><p><img src="https://sun9-69.userapi.com/impg/c858532/v858532275/1e4dff/9iXImJguMmI.jpg?size=807x618&quality=96&sign=e253f78157f934364f55a7619335e156&type=album" alt="9iXImJguMmI.jpg" class="fr-fic fr-dii fr-draggable " style="" /></p></blockquote><p></p>
[QUOTE="Cupper, post: 578, member: 22"] If you have a special format that does not fit any of the above, then you can configure the processing of any format in the configuration file, for this see the section " How to configure goaccess.conf". In order to collect statistics on the countries that accessed the site (geolocation), you need to specify the path to the GeoIP [B]database[/B] with the [B]--geoip-database[/B] option, for example [B]GeoLiteCity.dat[/B] or [B]GeoLite2-City.mmdb[/B] . If GeoIP2 is used, you need to download the [B]GeoLite2-City.mmdb[/B] or [B]GeoLite2-Country.mmdb database[/B]. These databases can be downloaded from the MaxMind.com website - download is free, but requires obtaining an API, so you need to register on the site - all this is free. So, my large combined Apache log is located in the [B]biglog.txt[/B] file, it is in [B]COMBINED[/B] format, I want to save the generated report to the [B]logs_report.html[/B] file and use geolocation using the [B]GeoLite2-City.mmdb[/B] database [B]for analysis[/B], then the command is as follows: Code: cat biglog.txt | goaccess - --log-format = COMBINED --output = logs_report.html --geoip-database = GeoLite2-City.mmdb You can open the generated report in any browser: Code: firefox logs_report.html Example of a report: If you are interested in a detailed description of each item, then see the article "Why and How to Analyze Web Server Logs". Various output formats can be specified: [B]-o --output = <path / file. [Json | csv | html]>[/B]: [LIST] [*]/ path / file. [B]csv[/B] - Comma Separated Values (CSV) [*]/ path / file. [B]json[/B] - JSON (JavaScript Object Notation) [*]/ path / file. [B]html[/B] - HTML [/LIST] That is, the format is determined by the file extension, so you can specify any name, and the file extension can be selected from one of the three presented. To analyze referrers (referring sites), you can exclude the analyzed site itself, as well as various incorrect values, this is done with the [B]--hide-referer[/B] option, which can be used many times: Code: cat biglog.txt | goaccess - --log-format = COMBINED --output = logs_report.html --hide-referer = "hackware.ru" --hide-referer = "-" --geoip-database = GeoLite2-City.mmdb If search engines are not of interest among the referring sites, then they can also be added to the exclusions: Code: cat biglog.txt | goaccess - --log-format = COMBINED --output = logs_report.html --hide-referer = "hackware.ru" --hide-referer = "-" --hide-referer = "* google *" --hide -referer = "* yandex *" --geoip-database = GeoLite2-City.mmdb Another option that improves the readability of the results is [B]-d[/B] or the long version [B]--with-output-resolver[/B], this option enables the conversion of IP addresses to host names, only works for HTML and JSON formats. Please note that when using the [B]-d[/B] option, a large number of DNS lookups are performed and the generation of the log file may be slower. By the way, you can use the online service GoAccess to analyze web server logs: [URL]https://suip.biz/?act=goaccess[/URL] This service accepts log files as unpacked text files or in .gz archives. It says that this is an Apache log analyzer, but in fact, any log format that GoAccess supports is accepted. [B]LORG[/B] LORG - Apache log file security analyzer, is a tool for advanced security analysis of HTTPD logs. It aims to implement various modern approaches to detecting web application attacks in HTTP traffic logs (such as Apache access logs (access_log files)), including signature-based, statistics, and machine learning techniques. Detected incidents are subsequently grouped into sessions that are classified as “manual” or automated to determine if the attacker is human or machine. In addition, geo-targeting and DNSBL lookups can be performed to see if attacks are originating from a specific geolocation or botnet. Additionally, attacks can be quantified in terms of success or failure based on anomalies within the size of HTTP responses. A detailed description of LORG, a complete list of options and installation instructions can be found on this page: [URL]https://kali.tools/?p=4852[/URL] Run command: Code: ./lorg OPTIONS input_file [output_file] I will use the following options in the command: [LIST] [*][B]-i input format[/B]. Options: [B]common combined vhost logio cookie[/B] [*][B]-o output format[/B]. Variants: [B]html json xml csv[/B] [*][B]-u[/B] perform URL decode for encoded requests (only affects reports) [*][B]-g[/B] enable geotag [/LIST] The Apache log file is located in the [B]~ / access_log file[/B], I want to save the report to the current folder in a file named [B]report.htm[/B], then the command is as follows: Code: ./lorg -u -i combined -g -o html ~ / access_log report.htm The report can be opened in a web browser: Code: firefox report.htm [B]How to edit log formats in LORG[/B] In fact, the format of my web server log does not fit any of the suggested LORG ([B]common combined vhost logio cookie[/B]) formats. The format of my file is very similar to [B]combined,[/B] with the difference that at the end of the line is the hostname (site domain). You can edit the existing supported log formats or add your own. To do this, open the executable file of the program: Code: gedit ./lorg We find the lines there: Code: static $ allowed_input_types = array ( 'common' => '% h% l% u% t \ "% r \"%> s% b', 'combined' => '% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \"', 'vhost' => '% v% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \"', 'logio' => '% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \% I% O"' , 'cookie' => '% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \" \ "% {Cookie } i \ "' ); To these lines, I will add the new format of my Hostland host: Code: 'combined_hostland' => '% h% l% u% t \ "% r \"%> s% b \ "% {Referer} i \" \ "% {User-Agent} i \"% v', It turned out like this: [IMG alt="Gem3zGUU8ds.jpg"]https://sun9-23.userapi.com/impg/c858532/v858532275/1e4dde/Gem3zGUU8ds.jpg?size=807x182&quality=96&sign=6fc72a81aca521e61312450741680bad&type=album[/IMG] We save and close the file. I run the command again to analyze the logs, but in this case I specify [B]combined_hostland[/B] as the format type: Code: ./lorg -u -i combined_hostland -g -o html ~ / biglog.txt report2.htm Although there are almost 6 million entries in the biglog.txt file, the analysis was pretty quick. [IMG alt="8R1vaRBNBMg.jpg"]https://sun9-70.userapi.com/impg/c858532/v858532275/1e4de5/8R1vaRBNBMg.jpg?size=389x129&quality=96&sign=1c73ff001a16a8e4c90641227a92ef19&type=album[/IMG] When the program terminates, it displays generalized statistics - how many incidents were found and how many users are involved in them: [IMG alt="GM8_7YQg7L0.jpg"]https://sun9-63.userapi.com/impg/c858532/v858532275/1e4ded/GM8_7YQg7L0.jpg?size=758x368&quality=96&sign=dc97468a619e67bc8625ba601d8735ba&type=album[/IMG] Open the generated report: Code: firefox report2.htm Above is a diagram with summarized information: [IMG alt="s0k4MVg-8Jw.jpg"]https://sun9-11.userapi.com/impg/c858532/v858532275/1e4df6/s0k4MVg-8Jw.jpg?size=807x600&quality=96&sign=7ad609439317d1c687029aa71d27b90e&type=album[/IMG] Detailed information can be viewed for each incident: [IMG alt="9iXImJguMmI.jpg"]https://sun9-69.userapi.com/impg/c858532/v858532275/1e4dff/9iXImJguMmI.jpg?size=807x618&quality=96&sign=e253f78157f934364f55a7619335e156&type=album[/IMG] [/QUOTE]
Name
Verification
Post reply
Home
Forums
CARDING & HACKING
HOSTING & BOTNET
Apache logs 2023
Top