Pages

Showing posts with label elk. Show all posts
Showing posts with label elk. Show all posts

Tuesday, June 6, 2017

ELK : Json Data not Logged Correctly in Elastic Search

Symptom
Data written to S3 form logstash is in Format
2016-12-08T21:55:36.381Z %{host} %{message}
2016-12-08T21:55:36.385Z %{host} %{message}
2016-12-08T21:55:36.385Z %{host} %{message}
2016-12-08T21:55:36.390Z %{host} %{message}
2016-12-08T21:55:36.391Z %{host} %{message}
2016-12-08T21:55:36.421Z %{host} %{message}
2016-12-08T21:55:36.421Z %{host} %{message}
2016-12-08T21:55:36.421Z %{host} %{message}
Cause
What happens here is that the default plain codec is being used for the S3 output from Logsearch. In the configuration for Custom Logstash outputs, you should use the JSON Lines Codec. There are more codecs you can use which are listed here.
Resolution
You can add the codec by adding the json_lines codec to your Custom Logstash Outputs Configuration in the Logstash tile settings. Your configuration should look like the following:
output {
...
    s3 {
access_key_id => "****************"
secret_access_key => "*********************"
region => "region name"
bucket => "bucket-name"
time_file => 15
codec => "json_lines"
}
...
After adding the json_lines codec, your S3 bucket Logstash entries should look more like this:
{"@timestamp":"2016-12-12T15:58:37.000Z","port":34854,"@type":"CounterEvent","@message":"{\"cf_origin\":\"firehose\",\"delta\":65,\"deployment\":\"cf\",\"event_type\":\"CounterEvent\",\"index\":\"9439da9a-fb72-4064-839f-934d4e8a6a5c\",\"ip\":\"192.0.2.1\",\"job\":\"router\",\"level\":\"info\",\"msg\":\"\",\"name\":\"udp.sentMessageCount\",\"origin\":\"MetronAgent\",\"time\":\"2016-12-12T15:58:37Z\",\"total\":5257491}","syslog_pri":"6","syslog_pid":"6229","@raw":"<6>2016-12-12T15:58:37Z f7643aae-c011-4715-a88b-2333aaf770ab doppler[6229]: {\"cf_origin\":\"firehose\",\"delta\":65,\"deployment\":\"cf\",\"event_type\":\"CounterEvent\",\"index\":\"9439da9a-fb72-4064-839f-934d4e8a6a5c\",\"ip\":\"192.0.2.1\",\"job\":\"router\",\"level\":\"info\",\"msg\":\"\",\"name\":\"udp.sentMessageCount\",\"origin\":\"MetronAgent\",\"time\":\"2016-12-12T15:58:37Z\",\"total\":5257491}","tags":["syslog_standard","firehose","CounterEvent"],"syslog_severity_code":6,"syslog_facility_code":0,"syslog_facility":"kernel","syslog_severity":"informational","@source":{"host":"f7643aae-c011-4715-a88b-2333aaf770ab","deployment":"cf","job":"router","ip":"192.0.2.1","program":"doppler","index":9439,"vm":"router/9439"},"@level":"INFO","CounterEvent":{"delta":65,"name":"udp.sentMessageCount","origin":"MetronAgent","total":5257491}}
Additional Information

Tuesday, January 17, 2017

Kibana Authentication with Nginx on Centos


Kibana doesn’t support authentication or restricting access to dashboards by default.We can restrict access to Kibana 4 using nginx as a proxy in front of Kibana.

Install nginx server:
To install Nginx using yum we need to include the Nginx repository, install the Nginx repository using,
1
rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
Install Nginx and httpd-tools by issuing the following command,
1
yum -y install nginx httpd-tools
Create a password file for basic authentication of http users, this is to enable the password protected access to kibana portal. Replace “admin” with your own user name
1
htpasswd -c /etc/nginx/conf.d/kibana.htpasswd adin
Configure Nginx:
Create a confiiguration file with the name kibana.conf in /etc/nginx/conf.d directory
1
vi /etc/nginx/conf.d/kibana.conf
Place the following content to the kibana.conf file, assuming that both kibana and Nginx are installed on same server

server {
listen *:8080;
server_name 192.168.01;
access_log /var/log/nginx/kibana-access.log;
error_log /var/log/nginx/kibana-error.log;
location / {
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/conf.d/kibana.htpasswd;
proxy_pass http://192.168.01:5601;
#proxy_connect_timeout 150;
#proxy_send_timeout 100;
#proxy_read_timeout 100;
}
}
Restart nginx server:
1
sudo service nginx restart
Go to the URL : http://192,168.01:8080, we should get an authentication screen as below on successful setup,
6
If nothing is showing up check the logs and see whether you have encountered an error as below,
2015/08/11 22:31:13 [crit] 80274#0: *3 connect() to 192.168.1.5:5601 failed (13: Permission denied) while connecting to upstream, client: 10.200.100.29, server: 10.242.126.73, request: "GET / HTTP/1.1", upstream: "http://192.168.1.5:5601/", host: "192.168.1.5:8080"
Error Resolution:
This is happening because we have selinux enabled on our machine.
Disable the selinux by running the command
1
sudo setsebool -P httpd_can_network_connect 1
Restart nginx:
1
sudo service nginx restart