Using Logstash for large scale log analysis along with MongoDB

This simple demo will give you an idea on how to use the logstash for large scale analysis of logs along with MongoDB. MongoDB can be used to store the large amount of logs. These logs further can be used to perform analytic by bringing into SQL DB as per requirement or as per time frame.

Step 1:Installing the Contrib package which has MongoDB plugin 
Option 1:  Using Script
   vaibhav@ubuntu:~/logstash-1.4.2$ bin/plugin install contrib 

  As per (https://github.com/elasticsearch/logstash/issues/1658) if this fails there is option 2 to install contrib package.


Option 2: Install the contrib package in same directory, in which the Logstash is installed.
  • wget http://download.elasticsearch.org/logstash/logstash/logstash-contrib-1.4.2.tar.gz
  •  tar zxf logstash-contrib-1.4.2.tar.gz



Step 2: Modify the configuration file for the logstash with with the credentials for the MongoDB as output. Input to the logstash is simple project.log file
Here I have used MongoLab (MongoDB as a Service)

temp.conf

input 
{
  file
 {
    path => "/home/vaibhav/project2.log"
    start_position => beginning
  }
}

output 
{ 
 mongodb 
{
    collection => "CollectionName"
    database => "DBname"
    uri => "mongodb://username:pwd@ds053090.mongolab.com:53090/DBname" 
  }
}

Step 3: Run the logstash with configuration file


Step 4: For testing enter manually log line into the log file

project2.log

    




Step 5: As soon as we write in the log file, on every write event logstash will make parsed structured log entry in MongoDB



Hence we saw simple Demo of how we can use the MongoDB to store parsed logs from the Logstash, Which can be further used to perform analytic.

Comments

Post a Comment

Popular posts from this blog

Elasticsearch JIRA Alert and Email Notification using ElastAlert Library

In this post, I will be giving you sample example of how to create alerts and notifications on top of Elasticsearch using ElastAlert Library(opensource library from Yelp, https://github.com/Yelp/elastalert ). Requirements:   Python 2.6/2.7   pip Step 1: clone the ElastAlert library             git clone  https://github.com/Yelp/elastalert.git Step 2 : python setup.py install Step 3: pip install -r requirements.txt Step 4: Now start the elasticsearch and run below command from ElastAlert Library folder              elastalert-create-index   This will create index called "elastalert_status"  in elasticsearch, where all the alerts send and debugging information is stored. Step 5: make sure you have config.yaml file properly configured with your elasticsearch host-port and rules folder where all alert rules file are located Step 6: Now yo...
Read more

Web Application using Spring Tool Suite,Gradle and Scala

Image
Step 1 : Install Gradle on your windows system.             Download link:  http://gradle.org/  (version 2.2.1) Set the Path environment variable to: your path\gradle-2.2.1-all\gradle-2.2.1\bin Step 2 : Install Scala on your system.            Download Link:  http://www.scala-lang.org/download/   (version 2.11.4) As above set the Path environment variable to :  C:\Program Files\scala\bin; Step 3: Install  Spring STS with scala IDE plug in              Download Link :  http://spring.io/tools/sts Step 4 : clone this Hello-World github code in your system              Link : git clone   https://github.com/vaibhavtupe/hello-world Important files: build.gradle Mention your gradle version correctly in this file. And also mention spring boot application main class file. buil...
Read more