Friday, October 15, 2010

How to configure syslog server or a centralized log server in redhat linux rhel5 or centos

Advertisements

This article will explain installing and configuring a syslog log server in redhat enterprise linux. It'll work in other redhat distributions like centos, fedora etc.
Centralized log server (syslog server)
Suppose we have a server and 5 client machines. And we want to monitor the logs of all those client machines. In situations like this, we will use centralized server as a log server. Whatever events are happening in client machines, the logs will be sent to the server. So that we can monitor all the logs from a centralized server. We make use of syslog service for this. You can also check out another log server like syslog-ng
Features of syslog:
1. Logs the daemon information to localhost
2. Logs the daemon information to Remote host
3. Logs the daemon information to List of users
4. Logs the daemon information to console


Instaltion Package is sysklogd
[root@apache ~]# rpm -q sysklogd
sysklogd-1.4.1-44.el5
[root@apache ~]#

Or you can check as follows:
[root@apache ~]# rpm -qf /etc/syslog.conf
sysklogd-1.4.1-44.el5
[root@apache ~]#

Starting the syslog daemon
[root@apache ~]# /etc/init.d/syslog start
Starting system logger:                                    [  OK  ]
Starting kernel logger:                                    [  OK  ]
[root@apache ~]#

Checking the process name. it is syslogd
[root@apache ~]# ps -ax | grep syslog
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
 5190 ?        Ss     0:00 syslogd -m 0
 5210 pts/0    S+     0:00 grep syslog
[root@apache ~]#

Configuration of server machine(syslog server)

Service name: syslog
configuration file: /etc/sysconfig/syslog

Steps:

1. Open the /etc/sysconfig/syslog file and add "-r" option to the variable SYSLOGD_OPTIONS as shown below.

[root@server ~]# cat /etc/sysconfig/syslog
# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS="-r -m 0"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
# once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details
KLOGD_OPTIONS="-x"
#
SYSLOG_UMASK=077
# set this to a umask value to use for all log files as in umask(1).
# By default, all permissions are removed for "group" and "other".
[root@server ~]#

2. Restart the syslog service.

[root@server ~]# service syslog restart
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
[root@server ~]#

Configuration for client machines

service name: syslog
Configuration file: /etc/syslog.conf


The configuration file /etc/syslog.conf has two parts
Eg:
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
[selector field(Facility.priority)]      [action field]

They are selector field and actions field. Selector field is again divided into two. Facilities and priorities.
Facility examples are (authpriv,kern,mail,local7 etc)
The priority is one of the following in ascending order: debug(0), info, notice, warning(warn), error(err), crit, alert, emerg(panic(7))
Actions can be regular files, console, list of users, remote machine ip etc.

Steps:

1. Open the configuration file /etc/syslog.conf and add an entry to redirect the logs to the remote server.

[root@vm1 ~]# cat /etc/syslog.conf
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

*.* @192.168.0.19

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages


# The authpriv file has restricted access.
##authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

[root@vm1 ~]#

2. Restart the service

[root@vm1 ~]# service syslog restart
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
[root@vm1 ~]#

Checking:

In server open a terminal and watch /var/log/messages and restart syslog service in client. You can see the log from clinet coming to server.

[root@server ~]# tail -f /var/log/messages

Oct 15 14:42:30 vm1 kernel: Kernel logging (proc) stopped.
Oct 15 14:42:30 vm1 kernel: Kernel log daemon terminating.
Oct 15 14:42:31 vm1 exiting on signal 15
Oct 15 14:42:31 vm1 syslogd 1.4.1: restart.
Oct 15 14:42:31 vm1 kernel: klogd 1.4.1, log source = /proc/kmsg started.

Fields in log from remote machine:

Date Hostname Name_of_the_application: Actual_log_message


Recommended Reading

1. Logging and Log Management: The Authoritative Guide to Dealing with Syslog, Audit Logs, Events, Alerts and other IT 'Noise'
2. Security Monitoring: Proven Methods for Incident Detection on Enterprise Networks
3. SYSLOG I
4. Linux Bible 
5. The Linux Command Line: A Complete Introduction
6. Amazon Web Services For Dummies 

18 comments:

  1. Hi Randeep,
    Interesting setup,just i ant a clarification from you like can we seperate the logs by server ?
    Ex ; if we have 5 clients, can we get client wise ogs like /var/log/client1, /var/log/client2

    Expecting your comments on this.....

    ReplyDelete
    Replies
    1. We can make all logs to come to one server location............

      Delete
  2. I think its possible. I'm lil busy these days. I'll update the post when I work it out. Thanks for commenting. Hv fun.

    ReplyDelete
  3. Hi Randeep,
    This comment is not related to your post.After reading your blog,i got confidence that you can help me to sort out my problem.

    I am working on syslogd which prints a message(contains tabs,spaces
    etc...).When i observe the syslogd file in /var/log/ directory,message
    is not having any tabs.Instead it is printing ^I.

    Running this program on ubuntu 10.04 LTS and sysklogd.

    Java program i used is:

    import org.apache.log4j.Appender;
    import org.apache.log4j.Level;
    import org.apache.log4j.Logger;
    import org.apache.log4j.PatternLayout;
    import org.apache.log4j.net.SyslogAppender;

    public class syslogPrinter {

    private static int sysLogFacility = SyslogAppender.LOG_LOCAL3;
    private static String syslogHost = "localhost";
    private static String layoutPattern = "%m%n";
    public static Logger log2static =
    Logger.getLogger(syslogPrinter.class);

    public static void main(String[] args) {

    log2static.setLevel(Level.INFO);
    PatternLayout layout = new PatternLayout(layoutPattern);

    Appender syslogdAppender = new SyslogAppender(layout,
    syslogHost, sysLogFacility);

    log2static.addAppender(syslogdAppender);

    String str2 = "this is a tab: ";
    str2 = str2.concat("\t");
    str2 = str2.concat(" after space");
    log2static.error(str2);
    }

    }



    Output is coming like : this is a tab: ^I after space

    expected out put is : this is a tab: after space(tab space should
    come)

    Please tell me how to print tab in the message instead of ^I.......

    Do i need to configure syslogd.conf file?


    I don't find any error with java log4j .After debugging the code,i
    found that it is sending message as expected.But in the logs message
    is being formatted with ^I instead of tab.

    I have viewed the logs with gnome editor,gedit,vi.............But no
    change.........


    Help to how to fix it.....................

    ReplyDelete
  4. @Rajeev

    I have no idea about java. If you are using any shell scripts you can use '\t' to get a tab. Else check the syslogd.conf to see if there is a log file format. Some applications let us create custom logs.

    ReplyDelete
  5. Hello Randeep -
    Nice post, but my question is ....

    I want to configure syslog server in linux box and configure client setup in windows box... so how to do can you please explain in details....

    i would be much appreciated for your help brother......

    thanks a ton in advance

    ReplyDelete
  6. Hello Vasim. Thanks for the comment. But unfortunately I'm not familiar with Windows. There must be something to be done on the windows client side. Google it. All the best.

    ReplyDelete
  7. I have done all the steps.But i am not able to get the logs of Client on Server.

    ReplyDelete
  8. I have performed all the steps. But yet not able to get the logs of my client on server

    ReplyDelete
    Replies
    1. did u give
      *.* @192.168.0.19 on cat /etc/syslog.conf of client?

      in both the servers syslog is running?

      Delete
    2. hello shina,

      this post regarding ur..problem
      did u open ur syslog port(514)
      check it's open or not
      #netstat -an|grep 514
      udp 0 0 0.0.0.0:514 0.0.0.0:*

      >>Verify that port is open

      >>Run following command:
      # netstat -tulpn | less (if open, it ll show output like this way)
      Active Internet connections (only servers)
      Proto Recv-Q Send-Q Local Address Foreign Address State
      PID/Program name
      tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN
      -
      tcp 0 0 0.0.0.0:46147 0.0.0.0:* LISTEN
      -
      tcp 0 0 0.0.0.0:645 0.0.0.0:* LISTEN
      3858/rpc.mountd
      tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN
      3716/snmpd
      tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
      4013/mysqld
      tcp 0 0 0.0.0.0:7210 0.0.0.0:* LISTEN
      3899/vserver
      tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
      3366/portmap
      tcp 0 0 0.0.0.0:659 0.0.0.0:* LISTEN
      3445/rpc.statd
      tcp 0 0 0.0.0.0:982 0.0.0.0:* LISTEN
      3770/rpc.rquotad
      tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
      3742/cupsd
      tcp 0 0 0.0.0.0:27000 0.0.0.0:* LISTEN
      4284/lmgrd
      tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
      4049/sendmail: acce
      tcp 0 0 :::22 :::* LISTEN
      3731/sshd
      udp 0 0 0.0.0.0:2049 0.0.0.0:*
      -
      udp 0 0 0.0.0.0:514 0.0.0.0:*
      5655/syslogd
      udp 0 0 0.0.0.0:642 0.0.0.0:*
      3858/rpc.mountd
      udp 0 0 0.0.0.0:653 0.0.0.0:*
      3445/rpc.statd



      if not then use the below command to open 514.

      #iptables -A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT

      #iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 514 -j ACCEPT

      #service iptables save


      >>and for flushhing iptables us
      #iptables -F

      #service syslog restart

      Delete
  9. hi all,

    i need to configure syslog server.. i have some devices from where i want to get log via syslog server.
    1. 2-cisco router
    2. 2-cisco switch
    3. 5- server

    plz share me ur great idea
    would i appreciate ur reply

    ReplyDelete
    Replies
    1. First of all, you have to create a syslog server. You have to specify in cisco devices the ip(syslog server) and configuration for forwarding the logs to syslog server.I haven't worked with these devices. But I think the info is available in many sites. Google it. You will find it easily. For other servers you can use the method we discussed above.

      Delete
  10. Hi randeep;

    i configured syslog server & client also getting logs from client server,

    when i restart the client server, syslog server having log of client server ....but it was only restart log.How can i get debug log , user information like what command user doing...,kernel log, boot log,...

    how can i do it in advance way....Share ur great information.

    ReplyDelete
    Replies
    1. Have you given *.* @server in ur client? kern.* etc can be used to forward particular logs.

      Delete
  11. yaa...but still not getting user info, debug info .

    that log info should come in /var/log/messages in syslog server...

    or some other folder..of syslog server contain these log

    ReplyDelete
  12. Hey guys..
    I configured the same in two of my servers. I configure one as server and other as client. But i am not getting any updates in logs from client. 514 Ports are open.. Actually tell me where I can configure the ports??? Actually I am not getting any updates from clients.

    ReplyDelete
    Replies
    1. Check ur server configuration part /etc/sysconfig/syslog in Linux(Redhat and cenOS)

      SYSLOGD_OPTIONS="-r -m 0" same should be there:)

      Delete

Be nice. That's all.