0%

nignx日志实时分析工具-ngxtop

ngxtop是一款python模块,可实时展示nginx的日志情况,支持自定义日志格式、过滤。

官网:https://github.com/lebinh/ngxtop
(2015年之后就没有再更新了,有些可惜)

安装:

1
# pip install ngxtop

使用方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# ngxtop --help
ngxtop - ad-hoc query for nginx access log.

Usage:
ngxtop [options]
ngxtop [options] (print|top|avg|sum) <var> ...
ngxtop info
ngxtop [options] query <query> ...

Options:
-l <file>, --access-log <file> access log file to parse.
-f <format>, --log-format <format> log format as specify in log_format directive. [default: combined]
--no-follow 默认情况下ngxtop只会分析新增日志(类似tail -f);使用这个选项使ngxtop分析日志文件中的已有日志
-t <seconds>, --interval <seconds> report interval when running in follow mode [default: 2.0]

-g <var>, --group-by <var> group by variable [default: request_path]
-w <var>, --having <expr> having clause [default: 1]
-o <var>, --order-by <var> order of output for default query [default: count]
-n <number>, --limit <number> limit the number of records included in report for top command [default: 10]
-a <exp> ..., --a <exp> ... add exp (must be aggregation exp: sum, avg, min, max, etc.) into output

-v, --verbose more verbose output
-d, --debug print every line and parsed record
-h, --help print this help message.
--version print version information.

Advanced / experimental options:
-c <file>, --config <file> allow ngxtop to parse nginx config file for log format and location.
-i <filter-expression>, --filter <filter-expression> filter in, records satisfied given expression are processed.
-p <filter-expression>, --pre-filter <filter-expression> in-filter expression to check in pre-parsing phase.

Examples:
All examples read nginx config file for access log location and format.
If you want to specify the access log file and / or log format, use the -f and -a options.

"top" like view of nginx requests
$ ngxtop

Top 10 requested path with status 404:
$ ngxtop top request_path --filter 'status == 404'

Top 10 requests with highest total bytes sent
$ ngxtop --order-by 'avg(bytes_sent) * count'

Top 10 remote address, e.g., who's hitting you the most
$ ngxtop --group-by remote_addr

Print requests with 4xx or 5xx status, together with status and http referer
$ ngxtop -i 'status >= 400' print request status http_referer

Average body bytes sent of 200 responses of requested path begin with 'foo':
$ ngxtop avg bytes_sent --filter 'status == 200 and request_path.startswith("foo")'

Analyze apache access log from remote machine using 'common' log format
$ ssh remote tail -f /var/log/apache2/access.log | ngxtop -f common

实例展示

自定义日志格式

1
# ngxtop -f '$http_x_forwarded_for $http_ip_source $remote_user [$time_local] "$request" $status $body_bytes_sent "$ht_referer" "$http_user_agent" "$cookie" "$cookie2" "$cookie3" "$request_time"' -l /var/log/httpd/access_log

ngxtop_default.png

过滤

1
# ngxtop -f '$http_x_forwarded_for $http_ip_source $remote_user [$time_local] "$request" $status $body_bytes_sent "$ht_referer" "$http_user_agent" "$cookie" "$cookie2" "$cookie3" "$request_time"' -l /var/log/httpd/api_access_log -i 'status == 200'

top

1
# ngxtop -f '$http_x_forwarded_for $http_ip_source $remote_user [$time_local] "$request" $status $body_bytes_sent "$ht_referer" "$http_user_agent" "$cookie" "$cookie2" "$cookie3" "$request_time"' -l /var/log/httpd/api_access_log top http_x_forwarded_for

ngxtop_top.png

group-by

1
# ngxtop -f '$http_x_forwarded_for $http_ip_source $remote_user [$time_local] "$request" $status $body_bytes_sent "$ht_referer" "$http_user_agent" "$cookie" "$cookie2" "$cookie3" "$request_time"' -l /var/log/httpd/api_access_log --group-by http_x_forwarded_for

ngxtop_group_by.png

分析已有日志

自定义日志格式,分析已有日志,而非tail -f模式,按访问次数展示前20条数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# ngxtop -f '$http_x_forwarded_for $http_ip_source $remote_user [$time_local] "$request" $status $body_bytes_sent "$ht_referer" "$http_user_agent" "$cookie" "$cookie2" "$cookie3" "$request_time"' -l /var/log/httpd/access_log -n 20 --no-follow
running for 15 seconds, 4430 records processed: 297.95 req/sec

Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 4430 | 1354.862 | 1730 | 2464 | 236 | 0 |

Detailed:
| request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------------------------------------+---------+------------------+-------+-------+-------+-------|
| /xxx.php | 1746 | 489.269 | 121 | 1624 | 1 | 0 |
| /xxx.php | 1723 | 1760.109 | 871 | 814 | 38 | 0 |
| /xxx | 717 | 2797.664 | 710 | 0 | 7 | 0 |
| /xxx.css | 89 | 432.011 | 0 | 0 | 89 | 0 |
| /xxx.css | 89 | 436.528 | 0 | 0 | 89 | 0 |
| | 25 | 5.000 | 0 | 25 | 0 | 0 |
| /xxx.php | 14 | 0.000 | 14 | 0 | 0 | 0 |
| /xxx.js | 11 | 489.909 | 0 | 0 | 11 | 0 |
| /xxx.php | 9 | 1272.889 | 9 | 0 | 0 | 0 |
| /xxx.php | 2 | 999.500 | 2 | 0 | 0 | 0 |
| / | 1 | 563.000 | 0 | 0 | 1 | 0 |
| /xxx.php | 1 | 5.000 | 0 | 1 | 0 | 0 |
| /xxx.php | 1 | 852.000 | 1 | 0 | 0 | 0 |
| /xxx/ | 1 | 10045.000 | 1 | 0 | 0 | 0 |
| /xxx.php | 1 | 1449.000 | 1 | 0 | 0 | 0 |