轉(zhuǎn)帖|使用教程|編輯:龔雪|2014-08-21 09:31:35.000|閱讀 1015 次
概述:在眾多JavaScript圖表工具中,amcharts以其易操作性受到廣大用戶好評,我們可以使用amcharts結(jié)合ElasticSearch做webserver 日志分析,amcharts強大的數(shù)據(jù)可視化效果,可以讓我們的分析更直觀、更有效。
# 界面/圖表報表/文檔/IDE等千款熱門軟控件火熱銷售中 >>
之前有一篇從 ElasticSearch 官網(wǎng)摘下來的博客《【翻譯】用ElasticSearch和Protovis實現(xiàn)數(shù)據(jù)可視化》。不過一來 Protovis 已經(jīng)過時,二來 不管是 Protovis 的進化品 D3 還是 Highchart什么的,我覺得在多圖方面都還不如 amcharts 好用。所以在最后依然選擇了老牌的 amcharts 完成。
展示品的大概背景還是 webserver 日志,嗯,這個需求應(yīng)該是最有代表性的了。我們需要對webserver的性能有所了解。之前有一篇文章《Tatsumaki框架的小demo一個》,講的是通過terms_stats 獲取固定時段內(nèi)請求時間的平均值。其實這個demo是可以參照官網(wǎng)博客修改成純js應(yīng)用的。因為 Tatsumaki 在這里除了處理 HTTP 請求參數(shù),什么都沒干。而且這個demo目的是展示 perl 框架的處理,所以amchart方面直接就寫死了各種變量。
但是還有一種需求,比如你需要的是針對某個情況超過某個百分比的分時走勢統(tǒng)計。這時候必須多次請求 ES 來做運算,再讓 js 做,不是說不行,但是多一倍數(shù)據(jù)在網(wǎng)絡(luò)中傳輸,就不如在服務(wù)器端封裝 API 了 —— 其實是我 js 太爛這種事情,我會告訴你們么。。。
先上兩張效果圖,其實這個布局我是從 facetgrapher 項目偷來的,但這個項目只適合比較不同 index 之間同時間段的數(shù)據(jù),我建議作者修改,作者說”我自己js也是半吊子水平”。。。
查詢的 ES 庫情況如下:
$ curl "//10.4.16.68:9200/demo-photo/log/_mapping?pretty=1" { "log" : { "properties" : { "brower" : { "type" : "string" }, "date" : { "type" : "date", "format" : "dateOptionalTime" }, "fromArea" : { "type" : "string", "index" : "not_analyzed" }, "hasErr" : { "type" : "string" }, "requestUrl" : { "type" : "string", "index" : "not_analyzed" }, "timeCost" : { "type" : "long" }, "userId" : { "type" : "string" }, "xnforword" : { "type" : "string" } } } } $ curl "//10.4.16.68:9200/demo-photo/log/_search?pretty=1&size=1" -d '{"query":{"match_all":{}}}' { "took" : 14, "timed_out" : false, "_shards" : { "total" : 10, "successful" : 10, "failed" : 0 }, "hits" : { "total" : 2330679, "max_score" : 1.0, "hits" : [ { "_index" : "demo-photo", "_type" : "log", "_id" : "iSI5xic7Qg2p9Sqk5yp-pQ", "_score" : 1.0, "_source" : {"hasErr":"false","date":"2012-12-06T15:04:21,983","userId":"123456789","requestUrl":"//photo.demo.domain.com/path/to/your/app/test.jpg","brower":"chrome17.0.963.84","timeCost":750,"xnforword":["192.168.1.123","10.10.10.10"],"fromArea":"CN-UNI-OTHER"} } ] } }
然后后臺是我慣用的 Dancer 框架:
package AnalysisDemo; use Dancer ':syntax'; use Dancer::Plugin::Ajax; use ElasticSearch; use POSIX qw(strftime); no warnings; my $elsearch = ElasticSearch->new( { %{ config->{plugins}->{ElasticSearch} } } ); my $index_prefix = 'demo-'; my $type = 'log'; # 這里是對ip庫的歸類。數(shù)據(jù)是需要提前導(dǎo)入ES的,這可以是logstash發(fā)揮作用 my $default_provider = { yidong => [qw(CN-CRN CN-CMN)], jiaoyu => [qw(CN-CER CN-CST)], dianxin => [qw(CN-CHN)], liantong => [qw(CN-UNI CN-CNC)], guangdian => [qw(CN-SCN)], haiwai => [qw(OS)], }; get '/' => sub { # 通過 state API 獲取 ES 集群現(xiàn)有的所有index列表 # 因為是一個域名一個index,這樣就有了前段頁面上的域名下拉選擇框 my $indices = $elsearch->cluster_state->{routing_table}->{indices}; template 'demo/chart', { providers => [ sort keys %$default_provider ], datasources => [ grep { /^$index_prefix/ && s/$index_prefix// } keys %$indices ], inputfrom => strftime("%F\T%T", localtime(time()-864000)), inputto => strftime("%F\T%T", localtime()), }; }; # 這里把 api 拆成服務(wù)商和區(qū)域兩個,沒啥特殊原因,因為是分兩回寫的,汗 # 其實可以看到最開始的請求參數(shù)類似,最后json的field名字都一樣 ajax '/api/provider' => sub { my $param = from_json(request->body); my $index = $index_prefix . $param->{'datasource'}; my $from = $param->{'from'} || 'now-10d'; my $to = $param->{'to'} || 'now'; my $providers = $param->{'provider'}; my ( $pct, $chartData ); for my $provider ( sort @{$providers} ) { my $provider_pct; # 這里是比較麻煩的一點,因為一個區(qū)域在ip庫里可能標記成多個,比如鐵通和移動,現(xiàn)在都是移動 for my $area ( @{ $default_provider->{$provider} } ) { my $res = pct_count( $index, $area, $from, $to ); for my $time ( sort keys %{$res} ) { $provider_pct->{$time}->{count} += $res->{$time}->{count}; $provider_pct->{$time}->{error} += $res->{$time}->{error}; $provider_pct->{$time}->{slow} += $res->{$time}->{slow}; } } # 這里因為可能沒有錯誤,所以前面關(guān)閉了常用的 warnings 警告 for my $time ( sort keys %{$provider_pct} ) { my $right_pct = 100; $right_pct = 100 - $provider_pct->{$time}->{slow} / $provider_pct->{$time}->{count} * 100; $pct->{$time}->{$provider} = sprintf "%.2f", $right_pct; $pct->{$time}->{"${provider}Err"} = sprintf "%.2f", $provider_pct->{$time}->{error} / $provider_pct->{$time}->{count} * 100; $pct->{$time}->{"${provider}Size"} = sprintf "%.0f", $pct->{$time}->{"${provider}Err"}; } }; for my $time ( sort keys %$pct ) { my $data->{date} = $time; for my $provider ( @$providers ) { $data->{$provider} = $pct->{$time}->{$provider} || 100; $data->{"${provider}Err"} = $pct->{$time}->{"${provider}Err"} || 0; # 百分比太低,所以翻 5 倍來作為 bullet 的大小 $data->{"${provider}Size"} = $pct->{$time}->{"${provider}Size"} * 5 || 0; }; push @$chartData, $data; }; my $res = { type => "line", categoryField => "date", graphList => $providers, chartData => $chartData, }; return to_json($res); }; ajax '/api/area' => sub { my $param = from_json(request->body); my $index = $index_prefix . $param->{'datasource'}; my $limit = $param->{'limit'} || 50; my $from = $param->{'from'} || 'now-10d'; my $to = $param->{'to'} || 'now'; # 這是后來寫的,盡可能把 sub 拆分了,所以 ajax 這里就很簡略 # 當(dāng)然因為不考慮多運營商的問題,本身也容易一些 my $res = pct_terms( $index, $limit, $from, $to ); return to_json($res); }; sub pct_terms { my ( $index, $limit, $from, $to ) = @_; my $area_all_count = area_terms( $index, 0, $limit, $from, $to ); my $area_err_count = area_terms( $index, 2000, $limit, $from, $to ); my ( $error, $chartData ); for ( @{$area_err_count} ) { $error->{ $_->{term} } = $_->{count}; } for ( @{$area_all_count} ) { push @$chartData, { area => $_->{term}, error => $error->{ $_->{term} } || 0, right => $_->{count} - $error->{ $_->{term} }, }; } my $res = { type => "column", categoryField => "area", graphList => [qw(right error)], chartData => $chartData, }; return $res; } sub pct_count { my ( $index, $area, $from, $to ) = @_; my $level = $area eq 'OS' ? 3000 : 2000; my $all_count = histo_count( $index, 0, $area, $from, $to ); my $slow_count = histo_count( $index, $level, $area, $from, $to ); my $err_count = histo_count( $index, 'hasErr', $area, $from, $to ); my $res; for ( @{$slow_count} ) { $res->{ $_->{time} }->{slow} = $_->{count}; } for ( @{$err_count} ) { $res->{ $_->{time} }->{error} = $_->{count}; } for ( @{$all_count} ) { $res->{ $_->{time} }->{count} = $_->{count}; } return $res; } # 下面開始的兩個才是真正發(fā) ES 請求的地方 sub area_terms { my ( $index, $level, $limit, $from, $to ) = @_; my $data = $elsearch->search( index => $index, type => $type, size => 0, facets => { area => { facet_filter => { and => [ { range => { date => { from => $from, to => $to }, }, }, { numeric_range => { timeCost => { gte => $level, }, }, }, ], }, # 使用最簡單的 terms facets API,因為只用計數(shù)就好了 terms => { field => "fromArea", size => $limit, } } } ); return $data->{facets}->{area}->{terms}; } sub histo_count { my ( $index, $level, $area, $from, $to ) = @_; # 根據(jù) level 參數(shù)判斷使用 hasErr 還是 timeCost 列數(shù)據(jù) my $level_ref = $level eq 'hasErr' ? { term => { hasErr => 'true' } } : { numeric_range => { timeCost => { gt => $level } } }; my $facets = { pct => { facet_filter => { # 這里條件比較多,所以要用 bool API,不能用 and 了 bool => { # must 可以提供多個條件作為 AND 數(shù)組 # 此外還有 must_not 作為 AND NOT 數(shù)組 # should 作為 OR 數(shù)組 must => [ { range => { date => { from => $from, to => $to }, }, }, { prefix => { fromArea => $area } }, $level_ref, ], }, }, # 這里是需要針對專門的時間列做匯總,所以用 date_histogram 了,具體說明之前有博客 date_histogram => { field => "date", interval => "1h", } } }; my $data = $elsearch->search( index => $index, type => $type, facets => $facets, size => 0, ); return $data->{facets}->{pct}->{entries}; }
其實把里面請求的hash拆開來一個個定義,然后根據(jù)情況組合,但是不方便察看作為 demo 的整體情況。
然后看template里怎么寫。這里雖然有兩個效果圖,但是只有一個template喲:
<link rel="stylesheet" href="[% $request.uri_base %]/amcharts/style.css" type="text/css"> <script src="[% $request.uri_base %]/amcharts/amcharts.js" type="text/javascript"></script> <script type="text/javascript"> var chart; function createAmChart(data) { // 清空原有圖形 $("#chartdiv").empty(); // 如果是時間軸線圖,需要把date字符轉(zhuǎn)成Date對象 if ( data.categoryField == "date" ) { for ( var j = 0; j < data.chartData.length; j++ ) { data.chartData[j].date = new Date(Number(data.chartData[j].date)); } } chart = new AmCharts.AmSerialChart(); // 拖動條等圖片的路徑 chart.pathToImages = "/amcharts/images/"; chart.dataProvider = data.chartData; chart.categoryField = data.categoryField; // 如果是柱狀圖,可以顯示 3D 效果 if ( data.type == 'column' ) { // chart.rotate = true; chart.depth3D = 20; chart.angle = 30; } var categoryAxis = chart.categoryAxis; categoryAxis.fillAlpha = 1; categoryAxis.fillColor = "#FAFAFA"; categoryAxis.axisAlpha = 0; categoryAxis.gridPosition = "start"; // 時間軸需要解析Date對象 if ( data.categoryField == "date" ) { categoryAxis.parseDates = true; categoryAxis.minPeriod = "hh"; } var valueAxis = new AmCharts.ValueAxis(); valueAxis.dashLength = 5; valueAxis.axisAlpha = 0; // 指定柱狀圖為疊加模式,這里有多種模式可以看文檔 if ( data.type == 'column' ) { valueAxis.stackType = "regular"; } chart.addValueAxis(valueAxis); // 這里有個有趣的事情,如果不把graph當(dāng)數(shù)組直接循環(huán),效果也沒問題 // 我只能猜測是 addGraph 后數(shù)據(jù)其實已經(jīng)緩存到 chart 了 var graph = []; var colors = ['#FF6600', '#FCD202', '#B0DE09', '#0D8ECF', '#2A0CD0', '#CD0D74', '#CC0000', '#00CC00', '#0000CC', '#DDDDDD', '#999999', '#333333', '#990000']; for ( var i = 0; i < data.graphList.length; i++ ) { graph[i] = new AmCharts.AmGraph(); graph[i].title = data.graphList[i]; graph[i].valueField = data.graphList[i]; graph[i].type = data.type; if ( data.type == 'column' ) { graph[i].lineAlpha = 0; graph[i].fillAlphas = 1; } else { graph[i].valueField = data.graphList[i]; graph[i].descriptionField = data.graphList[i] + "Err"; graph[i].bulletSizeField = data.graphList[i] + "Size"; graph[i].bullet = "round"; // 設(shè)定為空心圓圈 graph[i].bulletColor = "#ffffff"; graph[i].bulletBorderAlpha = 1; // amchart 本來有默認顏色,不過前面因為修改了圓內(nèi)的顏色,所以其他顏色無法繼承默認設(shè)定了 graph[i].bulletBorderColor = colors[i]; graph[i].lineColor = colors[i]; graph[i].lineAlpha = 1; graph[i].lineThickness = 1; graph[i].balloonText = "[[value]]% / hasErr:[[description]]%"; } chart.addGraph(graph[i]); } // 加圖例,這樣可以在圖上隨時勾選察看具體某個數(shù)據(jù),也方便某數(shù)據(jù)異常的時候影響察看其他 var legend = new AmCharts.AmLegend(); legend.position = "right"; legend.horizontalGap = 10; legend.switchType = "v"; chart.addLegend(legend); // 加拖拉軸,這樣可以拖動察看細節(jié),這個功能很贊 var scrollbar = new AmCharts.ChartScrollbar(); scrollbar.graph = graph[0]; scrollbar.graphType = "line"; scrollbar.height = 30; chart.addChartScrollbar(scrollbar); var cursor = new AmCharts.ChartCursor(); chart.addChartCursor(cursor); chart.write("chartdiv"); }; function drawChart() { var provider = []; $("#provider :selected").each(function(){ provider.push( $(this).val() ); }); var datasource = $("#datasource :selected").val(); var apitype = $(":radio:checked").val(); var from = $("#from").val(); var to = $("#to").val(); $.ajax({ processData: false, url: "[% $request.uri_base %]/demo/api/" + apitype, data: JSON.stringify({"provider":provider, "datasource":datasource, "from":from, "to":to}), type: "POST", dataType: "json", success : createAmChart }); }; function showselect() { $("#providers").show(); }; function hideselect() { $("#providers").hide(); }; </script> <div class="well"> <div class="span8"> <input type="text" class="input-medium" id="from" name="from" value="[% $inputfrom %]"> <input type="text" class="input-medium" id="to" name="to" value="[% $inputto %]"> <select class="input-medium" id="datasource"> %% for $datasources -> $datasource { <option value="[% $datasource %]">[% $datasource %]</option> %% } </select> </div> <div class="span2"> <label class="radio"> <input type="radio" name="querytype" value="provider" onclick="showselect()">服務(wù)商趨勢 </label> <label class="radio"> <input type="radio" name="querytype" value="area" checked onclick="hideselect()">分地區(qū)統(tǒng)計 </label> </div> <button type="submit" class="btn btn-primary" onclick="drawChart()">查詢</button> <div id ="providers" class="controls hide"> <select class="input-medium" id="provider" multiple="mulitiple"> %% for $providers -> $provider { <option value="[% $provider %]" selected>[% $provider %]</option> %% } </select> </div> </div><!--/well--> <div id="chartdiv" style="width: 100%; height: 400px;"> </div>
本站文章除注明轉(zhuǎn)載外,均為本站原創(chuàng)或翻譯。歡迎任何形式的轉(zhuǎn)載,但請務(wù)必注明出處、不得修改原文相關(guān)鏈接,如果存在內(nèi)容上的異議請郵件反饋至chenjj@fc6vip.cn