\u6982\u8981<\/h2>\n
Logstash 5.1.1 \u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3002(Elasticsearch 5.1.1 \u3068 Kibana 5.1.1 \u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u306f\u524d\u30d6\u30ed\u30b0\u3067\u3084\u3063\u305f\u306e\u3067\u5272\u611b)
\nElasticsearch 5.1.1 \u306e\u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u6295\u5165\u3002
\nKibana \u306e Visualize \u3067 Tile map \u3092\u8868\u793a\u3002<\/p>\n
\u5b9f\u884c\u74b0\u5883<\/h2>\n
Amazon Linux AMI release 2016.09
\nt2.medium (\u4eca\u56de\u3082\u8d05\u6ca2\u306b\u884c\u304d\u307e\u3059)<\/p>\n
\u524d\u63d0\u3068\u3057\u3066\u3001Elasticsearch 5.1.1 \u3068 Kibana 5.1.1 \u306f\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u6e08\u307f\u3068\u3057\u307e\u3059\u3002(\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u65b9\u6cd5\u306f\u904e\u53bb\u30d6\u30ed\u30b0\u3092\u898b\u3066\u304f\u3060\u3055\u3044\u3002https:\/\/www.skyarch.net\/blog\/?p=9668<\/a> )<\/p>\n \u3044\u3064\u3082\u306e\u3088\u3046\u306b\u516c\u5f0f\u30b5\u30a4\u30c8\u3078\u3002 \u4eca\u56de\u306e\u304a\u76ee\u5f53\u3066 Logstash\uff01 ELK \u3067\u5168\u3066\u30d0\u30fc\u30b8\u30e7\u30f3\u7d71\u4e00\u3055\u308c\u3066\u3044\u308b\u3093\u3067\u3059\u306d\u30fc\u3002 Java \u306f\u524d\u56de\u5165\u308c\u305f\u306e\u3067\u98db\u3070\u3057\u307e\u3059\u3002<\/p>\n Logstash \u3092\u8a2d\u5b9a\u3002 \u3092\u8a18\u8f09\u3057\u3066\u4fdd\u5b58\u3002 \u3044\u3064\u3082 Elasticsearch \u306b\u30de\u30c3\u30d4\u30f3\u30b0\u304a\u9858\u3044\u3057\u3066\u3044\u307e\u3059\u304c\u3001\u4eca\u56de\u306f\u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u3092\u521d\u3081\u3066\u6295\u5165\u3057\u307e\u3059\u3002(\u30c9\u30ad\u30c9\u30ad) \u307e\u305a\u306f\u3058\u3081\u306b\u3084\u3063\u305f\u4e8b\u306f\u3001\"%{COMBINEDAPACHELOG}\" \u304c\u3069\u306e\u3088\u3046\u306b\u30de\u30c3\u30d4\u30f3\u30b0\u3055\u308c\u305f\u304b\u3092\u4e0b\u8a18\u30b3\u30de\u30f3\u30c9\u306b\u3066\u78ba\u8a8d\u3002<\/p>\n \u305a\u3089\u305a\u3089\u3068\u51fa\u529b\u3055\u308c\u3066\u3044\u307e\u3059\u306d\u3002 \u3057\u3063\u304b\u308a\u3068 type \u304c float \u306b\u306a\u3063\u3066\u307e\u3059\u306d\u3002 \u3068\u308a\u3042\u3048\u305a\u3001geoip.location \u306e type \u3092 geo_point \u306b\u5909\u66f4\u3057\u307e\u3057\u305f\u3002\u305d\u308c\u4ee5\u5916\u306f\u4f55\u3082\u5909\u3048\u3066\u307e\u305b\u3093\u3002(\u3053\u308c\u3067\u3044\u3051\u308b\u306e\u304b\uff1f) \u3067\u4e00\u6383\u3067\u304d\u307e\u3059\u3002\u6016\u3044\u30b3\u30de\u30f3\u30c9\u3067\u3059\u304b\u3089\u3054\u5229\u7528\u306f\u8a08\u753b\u7684\u306b\u3002 Kibana \u3092\u8997\u3044\u3066\u307f\u3088\u3046\uff01\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/h2>\n
\nhttps:\/\/www.elastic.co\/downloads<\/a><\/p>\n
\nhttps:\/\/artifacts.elastic.co\/downloads\/logstash\/logstash-5.1.1.rpm<\/strong><\/p>\n
\nfluentd \u306b\u30b5\u30e8\u30ca\u30e9\u3092\u544a\u3052\u306a\u304c\u3089\u00a0yum install !<\/p>\n\r\nyum install https:\/\/artifacts.elastic.co\/downloads\/logstash\/logstash-5.1.1.rpm\r\n<\/pre>\n
Logstash \u8a2d\u5b9a<\/h2>\n
\n\u8a2d\u5b9a\u30d5\u30a1\u30a4\u30eb\u306f \/etc\/logstash\/conf.d\/ <\/strong>\u914d\u4e0b\u306b\u8a2d\u7f6e\u3057\u307e\u3059\u3002\u8a2d\u5b9a\u30d5\u30a1\u30a4\u30eb\u306e\u8aad\u307f\u8fbc\u307f\u306b\u95a2\u3059\u308b\u8a2d\u5b9a\u306f \/etc\/logstash\/logstash.yml \u306b\u8a18\u8f09\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n\r\nvi \/etc\/logstash\/conf.d\/10_apache_access_log.conf\r\n\r\n-------------------------------------------------\r\ninput {\r\n file {\r\n path => "\/tmp\/access_log"\r\n start_position => "beginning"\r\n }\r\n}\r\n\r\nfilter {\r\n grok {\r\n match => { "message" => "%{COMBINEDAPACHELOG}" }\r\n }\r\n date {\r\n match => ["timestamp", "dd\/MMM\/YYYY:HH:mm:ss Z"]\r\n }\r\n geoip {\r\n source => ["clientip"]\r\n }\r\n}\r\n\r\noutput {\r\n elasticsearch {\r\n hosts => ["localhost:9200"]\r\n index => "access_log-%{+YYYYMMdd}"\r\n }\r\n}\r\n<\/pre>\n
\n\u4eca\u56de\u306f grok-pattern \"%{COMBINEDAPACHELOG}\" \u3092\u4f7f\u7528\u3057\u3066 match \u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002\u65e2\u306b Apache \u306e combined \u5f62\u5f0f\u304c\u30b5\u30dd\u30fc\u30c8\u3055\u308c\u3066\u3044\u308b\u306a\u3093\u3066\u3001\u5b09\u3057\u3044\u3067\u3059\u306d\uff01
\n\u3042\u3068\u306f\u5fd8\u308c\u3061\u3083\u3044\u3051\u306a\u3044 geoip \u3092\u4f7f\u3044\u307e\u3059\u3002(\u8a73\u7d30\u306f https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/plugins-filters-geoip.html \u3092\u53c2\u7167\u3002)
\nApache Log \u306f logstash \u30e6\u30fc\u30b6\u30fc\u304b\u3089\u3067\u3082\u8aad\u307f\u8fbc\u3081\u308b\u3088\u3046\u306b \/tmp \u3078\u653e\u308a\u8fbc\u3093\u3067\u3044\u307e\u3059\u3002<\/p>\n\u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u6295\u5165<\/h2>\n
\n\u307e\u305a\u30ad\u30e2\u3068\u3057\u3066\u3001geoip.location (\u7d4c\u5ea6\u7def\u5ea6) \u3068\u3044\u3046 Field \u306e type \u304c geo_point \u3067\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u70b9<\/strong>\u304c\u3042\u308a\u307e\u3059\u3002
\ngeoip \u3092\u6709\u52b9\u306b\u3059\u308b\u3068\u3001\"%{COMBINEDAPACHELOG}\" \u3067\u30d1\u30fc\u30b9\u3055\u308c\u305f clientip (\u63a5\u7d9a\u5143 IP\u30a2\u30c9\u30ec\u30b9) \u3092\u5143\u306b\u3001geoip \u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u3067\u89e3\u6790\u3057\u3066\u304f\u308c\u307e\u3059\u3002
\ncity_name, country_code, timezone \u306a\u3069\u591a\u69d8\u306a\u60c5\u5831\u304c\u53d6\u5f97\u3055\u308c\u307e\u3059\u3002
\n\u53d6\u5f97\u3055\u308c\u305f\u60c5\u5831\u306f Field \u3068\u3057\u3066\u6271\u3046\u4e8b\u304c\u3067\u304d\u308b\u306e\u3067\u3059\u304c\u3001\u305d\u306e\u307e\u307e Elasticsearch \u306b\u30de\u30c3\u30d4\u30f3\u30b0\u3092\u304a\u9858\u3044\u3059\u308b\u3068 type \u304c float \u3068\u306a\u3063\u3066\u3057\u307e\u3044\u307e\u3059\u3002
\n\u305d\u3053\u3067\u3001geoip.location \u306e type \u3092 geo_point \u3068\u3057\u3066\u6271\u3048\u308b\u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u3092\u6295\u5165\u3057\u3088\u3046\u3068\u3044\u3046\u306e\u304c\u4eca\u56de\u306e\u6d41\u308c\u3067\u3059\u3002(\u521d\u5fc3\u8005\u306a\u306e\u3067\u9593\u9055\u3063\u3066\u305f\u3089\u3059\u3044\u307e\u305b\u3093\u3002)<\/p>\n\r\ncurl -XGET localhost:9200\/_mapping?pretty\r\n<\/pre>\n
\n\u809d\u5fc3\u306e geoip.location \u306f\u3068\u3044\u3046\u3068<\/p>\n\r\n-----------------------\r\n "geoip" : {\r\n "properties" : {\r\n "city_name" : {\r\n "type" : "text",\r\n "fields" : {\r\n "keyword" : {\r\n "type" : "keyword",\r\n "ignore_above" : 256\r\n }\r\n }\r\n },\r\n "continent_code" : {\r\n "type" : "text",\r\n "fields" : {\r\n "keyword" : {\r\n "type" : "keyword",\r\n "ignore_above" : 256\r\n }\r\n }\r\n~~~~~~~~~~~~\u7701\u7565~~~~~~~~~~~~\r\n "latitude" : {\r\n "type" : "float"\r\n },\r\n "location" : {\r\n "type" : "float" \u2190 \u3053\u3053\u2605\r\n },\r\n "longitude" : {\r\n "type" : "float"\r\n },\r\n "postal_code" : {\r\n "type" : "text",\r\n "fields" : {\r\n "keyword" : {\r\n "type" : "keyword",\r\n "ignore_above" : 256\r\n }\r\n }\r\n~~~~~~~~~~~~\u7701\u7565~~~~~~~~~~~~\r\n "timezone" : {\r\n "type" : "text",\r\n "fields" : {\r\n "keyword" : {\r\n "type" : "keyword",\r\n "ignore_above" : 256\r\n }\r\n }\r\n }\r\n }\r\n},\r\n-----------------------\r\n<\/pre>\n
\n\u4e0a\u8a18\u30b3\u30de\u30f3\u30c9\u304b\u3089\u5f97\u305f\u7d50\u679c\u3092\u5143\u306b\u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u3092\u4f5c\u3063\u3066\u307f\u307e\u3057\u305f\u3002<\/p>\n\r\nvi es_templete.json\r\n-------------------------\r\n{\r\n "template" : "access_log-*",\r\n "mappings" : {\r\n "logs" : {\r\n "properties" : {\r\n "@timestamp" : { "type" : "date" },\r\n "@version" : { "type" : "text" },\r\n "agent" : { "type" : "text" },\r\n "auth" : { "type" : "text" },\r\n "bytes" : { "type" : "text" },\r\n "clientip" : { "type" : "text" },\r\n "geoip" : {\r\n "properties" : {\r\n "city_name" : { "type" : "text" },\r\n "continent_code" : { "type" : "text" },\r\n "country_code2" : { "type" : "text" },\r\n "country_code3" : { "type" : "text" },\r\n "country_name" : { "type" : "text" },\r\n "ip" : { "type" : "text" },\r\n "latitude" : { "type" : "float" },\r\n "location" : { "type" : "geo_point" },\r\n "longitude" : { "type" : "float" },\r\n "postal_code" : { "type" : "text" },\r\n "region_code" : { "type" : "text" },\r\n "region_name" : { "type" : "text" },\r\n "timezone" : { "type" : "text" }\r\n }\r\n },\r\n "host" : { "type" : "text" },\r\n "httpversion" : { "type" : "text" },\r\n "ident" : { "type" : "text" },\r\n "message" : { "type" : "text" },\r\n "path" : { "type" : "text" },\r\n "referrer" : { "type" : "text" },\r\n "request" : { "type" : "text" },\r\n "response" : { "type" : "text" },\r\n "timestamp" : { "type" : "text" },\r\n "verb" : { "type" : "text" }\r\n }\r\n }\r\n }\r\n}\r\n,\r\n<\/pre>\n
\n\u6295\u5165\u3059\u308b\u969b\u306f\u3001\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u524a\u9664\u3057\u3066\u304b\u3089\u306b\u3057\u307e\u3057\u3087\u3046\u306d\u3002
\n\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u524a\u9664\u306f<\/p>\n\r\ncurl -X DELETE 'localhost:9200\/*'\r\n<\/pre>\n
\n\u3044\u3056\u6295\u5165\uff01<\/p>\n\r\ncurl -XPUT localhost:9200\/_template\/apache_access_log --data @es_templete.json\r\n<\/pre>\n
\u78ba\u8a8d<\/h2>\n
![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222100441_kibana.png\")
<\/a>Index name or pattern \u3078\u4eca\u56de\u6307\u5b9a\u3057\u305f\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u540d\u3092\u5165\u308c\u307e\u3059\u3002
\nTime-field name \u306f @timestamp \u3092\u9078\u629e\u3057\u3066 Create\uff01<\/p>\n![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222100518_kibana.png\")
<\/a>\u3057\u3063\u304b\u308a\u3068 fields \u304c\u767b\u9332\u3055\u308c\u307e\u3057\u305f\u306d\u3002\u3053\u306e\u753b\u9762\u3067\u306f\u3001field \u306e\u9805\u76ee\u304c analyzed \u304b\u3069\u3046\u304b\u3082\u78ba\u8a8d\u3059\u308b\u4e8b\u304c\u3067\u304d\u307e\u3059\u3002
\n\u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u3092\u4f5c\u6210\u3059\u308b\u969b\u3001\u3053\u3053\u306e\u9805\u76ee\u3092\u898b\u306a\u304c\u3089\u5fae\u8abf\u6574\u3059\u308b\u3068\u826f\u3044\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u3002(\u4eca\u56de\u306f\u3055\u3063\u3071\u308a\u3068\u3057\u305f\u5185\u5bb9\u306a\u306e\u3067\u6df1\u6398\u308a\u306f\u3057\u307e\u305b\u3093\u3002)
\n\u307e\u305f\u3001geoip.location \u306e type \u304c geo_point \u306b\u306a\u3063\u3066\u3044\u308b\u4e8b\u3082\u308f\u304b\u308a\u307e\u3059\u3002(\u3042\u308c\u3060\u3051\u30ad\u30e2\u3060\u3068\u8aac\u660e\u3057\u3066\u304a\u3044\u3066 geoip.location \u898b\u5207\u308c\u3066\u3066\u3059\u3044\u307e\u305b\u3093\u3002)<\/p>\n![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222100610_kibana.png\")
<\/a>\u65e9\u901f Visualize \u306b\u79fb\u308a\u3001Tile map \u3092\u9078\u629e\u3057\u307e\u3059\u3002<\/p>\n![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222100647_kibana.png\")
<\/a>\u4eca\u56de\u4f5c\u6210\u3057\u305f \"access_log-*\" \u3092\u9078\u629e\u3002<\/p>\n![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222100732_kibana.png\")
<\/a>\u4e16\u754c\u5730\u56f3\u3067\u307e\u3057\u305f\u3002\u3082\u3046\u3053\u3053\u307e\u3067\u6765\u308b\u3068\u697d\u3057\u3044\u3002
\nbuckets \u306e select buckets type \u3067 Geo Coordinates \u3092\u9078\u629e\u3002<\/p>\n![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222100820_kibana.png\")
<\/a>Field \u306b geoip.location \u304c\u8a8d\u8b58\u3055\u308c\u3066\u3044\u308b\u4e8b\u304c\u78ba\u8a8d\u3067\u304d\u307e\u3059\u3002
\n\u53f3\u5411\u304d\u4e09\u89d2 (Apply changes) \u3092\u30af\u30ea\u30c3\u30af\u3057\u3066\u5909\u66f4\u3092\u9069\u7528\u3002<\/p>\n![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222100849_kibana.png\")
<\/a>\u304a\u3001\u304a\u304a\u304a\u304a\u304a\u304a\u304a\u304a\uff01\u65e5\u672c\u306e IP \u30a2\u30c9\u30ec\u30b9\u304b\u3089\u30a2\u30af\u30bb\u30b9\u304c\u6765\u3066\u3044\u308b\u4e8b\u304c\u308f\u304b\u308a\u307e\u3059\uff01<\/p>\n![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222101001_kibana.png\")
\n<\/a>\u30ba\u30fc\u30e0\u3002<\/p>\n![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222101030_kibana.png\")
<\/a>\u30ba\u30fc\u30fc\u30fc\u30fc\u30e0\u3002w<\/p>\n
![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222101347_kibana.png\")
<\/a>Heatmap \u3082\u4f7f\u304a\u3046\u3068\u601d\u3044\u904a\u3093\u3067\u3044\u308b\u3068\u3001\u3075\u3068\u6c17\u3065\u304f\u3002![](\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2016\/12\/20161222101442_kibana.png\")
<\/a>Heatmap \u306b\u3059\u308b\u3068\u3053\u3093\u306a\u611f\u3058\u3067\u3059\u3002