Finding First Instance of the Pattern Using Grep in Linux

Grep in Linux
Have you ever faced a problem grepping (Parsing) for a pattern from a file, Definately being a Linux Administrator you have done it several times but have you came across a situation where from a huge file you only have to parse the first instance of the pattern, grepping the whole file for a pattern will be tedious and will be using more system resources especially when you are doing it using a script (Using Loop for, while, until)

The easy solution to tackle the problem is to use grep with inbuilt option which will display only the first instances instead of checking the whole file and than printing the first line using head command.

Command :
grep -m 1 "pattern" filename.txt

The "-m" option when passed with grep will search for the first matching pattern from the file and print the output same-way if you want to look for first 2 instances pass option "grep -m 2 "pattern" yourfilename.log"

Enjoy...

Packet Capturing and Analayzing packets Using TCPDUMP on Centos, Fedora, Ubuntu and other Linux Distros

Many a times Linux System Admin2 face problem in network latency, connectivity the servers hampers the services which impact to the business. Using linux basic tools can not really solve it but you can identify the issues faced by the host to host or host to the device.

Here we will be using tcpdump tool to capture packet after which you can analyse the data and conclude with errors host.
If you have installed linux properly tcpdump is by default installed on to your system, and if not you can install it finding the relevant packages for your distros

RPM Based Linux can google for "tcpdump +rpm" and Debian can google for "tcpdump +deb", you will get enough results to finish installing the TCPDUMP package onto your system.

After installing, check if tcpdump is installed successfully executing the below command

 #>which tcpdump

Should return the binary location of the package installed

/usr/sbin/tcpdump

After confirming you can run tcpdump with various options passing to the command to check for the options

[root@ashwin ~]# tcpdump --help
tcpdump version 3.9.8
libpcap version 0.9.8
Usage: tcpdump [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]
		[ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]
		[ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
		[ -W filecount ] [ -y datalinktype ] [ -Z user ]
		[ expression ]

We will be capturing packets for 2 minutes and analyze for any issues with hosts connecting to and from.

Running the below command will capture all the packets transferring between the hosts through the Network Interface Cards and will be redirected to a file

tcpdump -n -i bond0 -e -vvv > output.log

Once you have captured packets for 5 minutes press Ctrl + z to end and proceed with analyzing the file "output.log"


Example :

2.39.omserv > 11.240.240.141.nfs: ., cksum 0x1eba (correct), 156:156(0) ack 28961 win 30782
18:41:21.397883 00:a0:98:10:7a:05 > 00:21:5a:4c:a6:3e, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 26738, offset 0, flags [DF], proto: TCP (6), length: 1500) 11.240.240.141.2049 > 11.240.240.39.33554432: reply ERR 1448

In the above line its clear that a partition is mounted on the host using NFS protocol from other hosts having IP 11.240.240.141 which is having issues while transferring data to host bearing IP 11.240.240.39 Error "reply ERR 1448"

You can find great detail in the output of tcpdump, Also you can use wireshark to read the output.txt and get a better clarity of the data transmission hence lot of network latency can be resolved.

Enjoy.

Packet Capturing and Analayzing packets Using TCPDUMP on Centos, Fedora, Ubuntu and other Linux Distros

Solved : Lock table is out of available locker entries - Yum install, yum upgrade

Recently faced a issue while installing a package using yum

[root@back ~]# yum install mod-security
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 3077
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 85, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 163, in getOptionsConfig
    disabled_plugins=self.optparser._splitArg(opts.disableplugins))
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 164, in _getConfig
    self._conf = config.readMainConfig(startupconf)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 685, in readMainConfig
    yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.distroverpkg)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 752, in _getsysver
    idx = ts.dbMatch('provides', distroverpkg)
TypeError: rpmdb open failed

Googling and Searching on this topic got it resolved

[root@back ~]# ll /var/lib/rpm/
Basenames       __db.000        __db.002        Dirnames        Group           Name            Providename     Pubkeys         Requireversion  Sigmd5          
Conflictname    __db.001        __db.003        Filemd5s        Installtid      Packages        Provideversion  Requirename     Sha1header      Triggername     
[root@back ~]# ll /var/lib/rpm/
Basenames       __db.000        __db.002        Dirnames        Group           Name            Providename     Pubkeys         Requireversion  Sigmd5          
Conflictname    __db.001        __db.003        Filemd5s        Installtid      Packages        Provideversion  Requirename     Sha1header      Triggername     

[root@back ~]# rm /var/lib/rpm/_db.00*
rm: cannot lstat `/var/lib/rpm/_db.00*': No such file or directory
[root@back ~]# rm /var/lib/rpm/__db.00*
rm: remove regular empty file `/var/lib/rpm/__db.000'? y
rm: remove regular file `/var/lib/rpm/__db.001'? y
rm: remove regular file `/var/lib/rpm/__db.002'? y
rm: remove regular file `/var/lib/rpm/__db.003'? y

[root@back ~]# rpm --rebuilddb

[root@back ~]# yum install mod-security
utterramblings            100% |=========================|  951 B    00:00     
primary.xml.gz            100% |=========================|  22 kB    00:00     
utterrambl: ################################################## 64/64
extras                    100% |=========================| 2.1 kB    00:00     
primary.sqlite.bz2        100% |=========================| 184 kB    00:38     
updates                   100% |=========================| 1.9 kB    00:00     
primary.sqlite.bz2        100% |=========================| 254 kB    00:28     
base                      100% |=========================| 2.1 kB    00:00     
primary.sqlite.bz2        100% |=========================| 1.6 MB    00:17     
c5-testing                100% |=========================| 1.9 kB    00:00     
addons                    100% |=========================|  951 B    00:00     
Setting up Install Process
Parsing package install arguments
No package mod-security available.
Nothing to do
[root@back ~]# yum install mod_security
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package mod_security.i386 0:2.5.9-1.jason.1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Installing:
 mod_security            i386       2.5.9-1.jason.1  utterramblings    1.2 M

Transaction Summary
=============================================================================
Install      1 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         

Total download size: 1.2 M
Is this ok [y/N]: y
Downloading Packages:
(1/1): mod_security-2.5.9 100% |=========================| 1.2 MB    00:03     
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 0d4306ef
Importing GPG key 0x0D4306EF "Jason Litka (http://www.jasonlitka.com) " from http://www.jasonlitka.com/media/RPM-GPG-KEY-jlitka
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: mod_security                 ######################### [1/1] 

Installed: mod_security.i386 0:2.5.9-1.jason.1
Complete!

This problem occurs when rpm accesses the Berkeley database files, it makes temporary locker entries within the tables while it searches for data. If you control-c your rpm processes often, this issue will occur much sooner because the locks are never cleared.

Solved Lock table is out of available locker entries - Yum install, yum upgrade

Howto Add New User, New Group and modify Users, Groups in Linux - Centos5.4, Fedora 11 - Part 1

Adding users in Linux helps you from giving all user Administrative (root access) rights if the system is used by Multiple people. Its one of the basic command but still very important which need to studied in much details, The More detail you know about adding user and adding group modifying existing users and groups may lead with excellent system administration skills and also help you with securing system. There are many attributes to read when we learn about useradd and groupadd command in linux. In this article we will go indepth of Adding, Modifying User and groups and also discussing about various parameters while using these commands.

Adding New User in Linux

Adding a new user to your linuxbox is as simple as running a command with a argument which is the username to be created

[root@ashwin ~]# useradd ashwin

Great You have created a new user but now into the technical details how does the user gets the home directory, how is the UID,GID set, how the default shell is assigned and other different features gets assigned when addding a user.

When you add a user immediately some files are checked if the user is already present Files

1. /etc/passwd
2. /etc/groups
3. /etc/shadows

If the user is not present in the above 3 files the next step is to look for another set of files which by default assign values to a newly added user the file which is looked is

 # cat /etc/login.defs 

The above file contains all the default parameter like, UID, GID to be set for new user, Home Directory (If not specified explicitly). Now once the user is added the home directory is created in /home/username by default and assigned a unique UID,GID. When you see the contents of /home/username you will find files

[ashwin@ashwin ~]$ ll -a /home/ashwin/
-rw-r--r--.   1 root root    18 2009-04-08 16:16 .bash_logout
-rw-r--r--.   1 root root   176 2009-04-08 16:16 .bash_profile
-rw-r--r--.   1 root root   124 2009-04-08 16:16 .bashrc
-rw-r--r--.   1 root root   500 2009-04-10 17:22 .emacs
drwxr-xr-x.   2 root root  4096 2009-03-18 06:22 .gnome2
drwxr-xr-x.   4 root root  4096 2010-03-19 12:20 .mozilla
-rw-r--r--.   1 root root   658 2009-03-02 22:50 .zshrc

Now from where these files came from, yes this files are copied to every newly added user from the /etc/skel folder.

[ashwin@ashwin ~]$ ll -a /etc/skel/
-rw-r--r--.   1 root root    18 2009-04-08 16:16 .bash_logout
-rw-r--r--.   1 root root   176 2009-04-08 16:16 .bash_profile
-rw-r--r--.   1 root root   124 2009-04-08 16:16 .bashrc
-rw-r--r--.   1 root root   500 2009-04-10 17:22 .emacs
drwxr-xr-x.   2 root root  4096 2009-03-18 06:22 .gnome2
drwxr-xr-x.   4 root root  4096 2010-03-19 12:20 .mozilla
-rw-r--r--.   1 root root   658 2009-03-02 22:50 .zshrc

Now a Bit more depth on useradd, we'll now discuss some parameters while adding a new user to change the default home directory, UID, GID, Default Shell Login, Password

The Following arguments can be used to the command useradd

  -b, --base-dir BASE_DIR       base directory for the new user account
                                home directory
  -c, --comment COMMENT         set the GECOS field for the new user account
  -d, --home-dir HOME_DIR       home directory for the new user account
  -D, --defaults                print or save modified default useradd
                                configuration
  -e, --expiredate EXPIRE_DATE  set account expiration date to EXPIRE_DATE
  -f, --inactive INACTIVE       set password inactive after expiration to INACTIVE
  -g, --gid GROUP               force use GROUP for the new user account
  -G, --groups GROUPS           list of supplementary groups for the new  user account
  -h, --help                    display this help message and exit
  -k, --skel SKEL_DIR           specify an alternative skel directory
  -K, --key KEY=VALUE           overrides /etc/login.defs defaults
  -l,                           do not add the user to the lastlog and  faillog databases
  -m, --create-home             create home directory for the new user account
  -M, 	                  do not create user's home directory(overrides /etc/login.defs)
  -N, --no-user-group           do not create a group with the same name as  the user
  -o, --non-unique              allow create user with duplicate (non-unique) UID
  -p, --password PASSWORD       use encrypted password for the new user account
  -r, --system                  create a system account
  -s , --shell SHELL             the login shell for the new user account
  -u, --uid UID                 force use the UID for the new user account
  -U, --user-group              create a group with the same name as the user
  -Z, --selinux-user SEUSER     use a specific SEUSER for the SELinux user mapping
 

Examples

1. Add user with home directory in /opt/username

[root@ashwin ~]# useradd -d /opt/ username
 

2. Add user with defined home directory UID, GID

[root@ashwin ~]# useradd -d /opt/ -u 800 -g 800 username
 

3. Defining password while adding a new user

[root@ashwin ~]# useradd -d /opt/ -u 800 -g 800 -p password username 
 

4. Defining the login shell for the new user account

[root@ashwin ~]# useradd -d /opt/ -u 800 -g 800 -p password -s /bin/sh username 

[root@ashwin ~]# useradd -d /opt/ -u 800 -g 800 -p password -s /bin/nologin username 
 

Explore around with the parameters you would find many features in it.

Also Look For Parts Two to know more depth knowledge of User and Group adding in Linux

Howto Install RED5 Flash Media Server in Fedora 11, Centos 5.4, Redhat

Red5 is an Open Source Flash Media Server written in Java that supports:

* Streaming Audio/Video (FLV and MP3)
* Recording Client Streams (FLV only)
* Shared Objects
* Live Stream Publishing
* Remoting (AMF)

This is a reverse engineered open source project which aims to produce a feature-complete implementation written in Java. It os one of the Full implementation RTMP servers other than:

Adobe Flash Media Server,
Onlinelib VCS Video Communication Server
Wowza Media Server
WebORB Integration Server
haXeVideo

In this tutorial we'll learn howto installRED5 on Centos 5.4,Fedora, Redhat and other Linux Distribution, There are many ways to achieve this but this is how i installed RED5 Media Server on Fedora 11.

Red5 Requires Java Development Kit (JDK) to run so you have to install Java onto the system,You can acheive this in 2 ways
1. Either Use Yellow DOG Updater

# yum -y install java-1.6.0-openjdk java-1.6.0-openjdk-devel

OR

Download JDK and Compile

cd /usr/java 
IF /usr/java cannot be found
mkdir /usr/java
Go to the Link and Download http://java.sun.com/javase/downloads/widget/jdk6.jsp and Download jdk-6u19-linux-i586.bin

mv /home/ashwin/Download/jdk-6u19-linux-i586.bin /usr/java/

chmod +x jdk-6u19-linux-i586.bin
./jdk-6u19-linux-i586.bin

You will see 
JAVA_HOME="/usr/java/jdk1.6"
export JAVA_HOME
echo $JAVA_HOME

Great Now you have installed JAVA now proceed

Another Important requirement is ANT (It is a Apache Project) which we require to compile the RED5 Code, Good News is you dont need to compile ANT as it is already in binary format Just extract and install it in /usr/java

#cd /home/ashwin/
#w.get http://mirrors.kahuki.com/apache/ant/binaries/apache-ant-1.8.0-bin.tar.bz2
#tar -jxf apache-ant-1.8.0-bin.tar.bz2
#mv apache-ant-1.8.0 /usr/local/ant

Now we need to Export the JAVA and ANT Environment Variables which would help us in installing and proper running if RED5 Server

export ANT_HOME=/usr/local/ant
export JAVA_HOME=/usr/lib/jvm/java
export PATH=$PATH:/usr/local/ant/bin
export CLASSPATH=.:$JAVA_HOME/lib/classes.zip

vi /etc/bashrc

# ALSO Put the Above Here
export ANT_HOME=/usr/local/ant
export JAVA_HOME=/usr/lib/jvm/java
export PATH=$PATH:/usr/local/ant/bin
export CLASSPATH=.:$JAVA_HOME/lib/classes.zip

Now download the latest RED5 CODE from the site to build

cd /home/ashwin/
svn checkout http://red5.googlecode.com/svn/java/server/trunk/ red5
mv red5 /usr/local/
cd /usr/local/red5
ant prepare
ant dist

You would you screen scrolling and you are not able to understand anything "CHILL" All is well and RED5 Media Server is getting compiled, AT the END you would SEE

BUILD SUCCESSFUL

If you see this you have made it you have done 95% of the work. Now copy the conf directory from dist/ and test the red5 installation.

cp -r dist/conf .
./red5.sh

STARTING, STOPPING,RESTARTING RED5 Media Server
I have created a Init Script Click Here

INIT Script to Start,STOP, RESTART RED5 Media Server

Start RED5

/etc/init.d/red5 start

TESTING RED5

Go TO your Browser Address Bar and Type

http://your.ip.address:5080

Check Out the Demos
http://your.ip.address:5080/demos/

ISSUE

When you run ./red5.sh, it will show you Installer service created. Thats mean everything runs fine and red5 server is up. But if you went to port_tester.swf using demos above or your application shows connections FAILS, this is an issue of RTMPT and RTMPTS. You can see it by running

/usr/local/red5/red5.sh

output trancated

[INFO] [main] org.red5.server.tomcat.TomcatLoader – RTMPT server bean was not found
[INFO] [main] org.red5.server.tomcat.TomcatLoader – RTMPS server bean was not found

output truncated

If you see this you have to uncomment the RTMPT and RTMPTS TomcatLoader in/usr/local/red5/conf/red5-core.xml

vi /usr/local/red5/conf/red5-core.xml

Search for a lines



Remove the from end of

Same goes for RTMPS



Remove the from end of



Restart the red5 services and connection fails problem will be fixed.

Init Script to Start, Stop, Restart RED5 Media Server

Click Here Howto Install RED5 Flash Media Server in Fedora 11, Centos 5.4, Redhat

RED5 is Open Source Media Flash Server, Highly Scalable and Customizable

Just Copy the Code in

vi /etc/init.d/red5

COPY the Below Code

    #!/bin/sh
    # Author : Ashwin 
    # Description: Red5 Media Flash streaming server

    PROG=red5
    RED5_HOME=/usr/local/red5
    DAEMON=$RED5_HOME/$PROG.sh
    PIDFILE=/var/run/$PROG.pid

    # Source function library
    . /etc/rc.d/init.d/functions

    [ -r /etc/sysconfig/red5 ] && . /etc/sysconfig/red5

    RETVAL=0

    case “$1″ in
    start)
    echo -n $”Starting $PROG: ”
    cd $RED5_HOME
    $DAEMON >/dev/null 2>/dev/null &
    RETVAL=$?
    if [ $RETVAL -eq 0 ]; then
    echo $! > $PIDFILE
    touch /var/lock/subsys/$PROG

    fi
    [ $RETVAL -eq 0 ] && success $”$PROG startup” || failure $”$PROG startup”
    echo
    ;;
    stop)
    echo -n $”Shutting down $PROG: ”
    killproc -p $PIDFILE
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$PROG
    ;;
    restart)
    $0 stop
    $0 start
    ;;
    status)
    status $PROG -p $PIDFILE
    RETVAL=$?
    ;;
    *)
    echo $”Usage: $0 {start|stop|restart|status}”
    RETVAL=1
    esac

    exit $RETVAL

chmod 755 /etc/init.d/red5

Test It

/etc/init.d/red5 start
/etc/init.d/red5 stop
/etc/init.d/red5 restart

Howto setup local yum repository on RedHat Fedora12 Centos5.4

This article is about Setting up a Local yum repository to be used for local network. Local Yum Repository can save internet bandwidth downloading all your packages from the local repository over FAST LAN Network.

If You want to use it in your network the best option to be used is using Httpd - Apache so the yum repository will be acessible via web.

Step 1
Check if Httpd is installed and running

[root@localhost ~]# ps ax | grep http

 3617 ?        Ss     0:00 /usr/sbin/httpd
 3621 ?        S      0:00 /usr/sbin/httpd
 3622 ?        S      0:00 /usr/sbin/httpd
 3623 ?        S      0:00 /usr/sbin/httpd
 3624 ?        S      0:00 /usr/sbin/httpd
 3625 ?        S      0:00 /usr/sbin/httpd
 3626 ?        S      0:00 /usr/sbin/httpd
 3627 ?        S      0:00 /usr/sbin/httpd
 3628 ?        S      0:00 /usr/sbin/httpd
 3629 ?        S      0:00 /usr/sbin/httpd

If it is not running then Start HTTPD

[root@localhost ~]# /etc/init.d/httpd start

OR 

[root@localhost ~]# service httpd start

and if you haven't installed httpd yet then you can do it by

[root@localhost ~]# yum install httpd

Now make httpd start at every system boot automatically by putting it in system Startup.

[root@localhost ~]# chkconfig --levels 235 httpd on

Start apache
[root@localhost ~]# /etc/init.d/httpd start
                  OR 
[root@localhost ~]# service httpd start

How to Build the Repository
We will use Apache's default Document root to store the packages, Default DocumentRoot for apache is "/var/www/html".

Create a dir in the Document Root named yum. We will create two more directories
base-pkg : This would contain all the packages of the Linux Distribution you are making repository for.
updates : This would contain all the update packages for your Linux Distribution

[root@localhost ~]# mkdir /var/www/html/yum
[root@localhost ~]# mkdir /var/www/html/yum/base-pkg
[root@localhost ~]# mkdir /var/www/html/yum/updates

BUILDING YOU REPOSITORY

To build you yum repository the easy way is copy all the contents from the Linux Distribution DVD/CD Media to the repository location in this case /var/www/html/yum, Make sure that you copy all the contents (Packages) into the base-pkg directory. If you have a Media follow the Steps Below

[root@localhost ~]# mount /dev/cdrom /mnt
[root@localhost ~]# cp -v /mnt/Packages/* /var/www/html/yum/base-pkg/
[root@localhost ~]# umount /mnt

Now we need a tool "createrepo" which would create a repository for the linux distibution. By default it is comes installed with all Distribution of linux, but if its not installed use yum to install it.
Check if createrepo is installed

[root@localhost ~]# which createrepo
/usr/bin/createrepo

If you get the above result skip the next step and start Building the repository if its not installed then install it

[root@localhost ~]# yum install createrepo

Now Run the createrepo tool

[root@localhost ~]# createrepo /var/www/html/yum/base-pkg/

[root@localhost ~]# ls -l /var/www/html/yum/base-pkg/repodata/
-rw-r--r-- 1 root root 2227275 2010-02-18 11:51 filelists.xml.gz
-rw-r--r-- 1 root root 6487453 2010-02-18 11:51 other.xml.gz
-rw-r--r-- 1 root root  747714 2010-02-18 11:51 primary.xml.gz
-rw-r--r-- 1 root root     951 2010-02-18 11:51 repomd.xml

The Second step building repository is updating your updates directory for all new packages. Go to the Main website of the Linux Distribution you are preparing repostory for and check the updates link or use the below links

CENTOS : rsync://centos.arcticnetwork.ca/centos/5.4/updates/i386/RPMS/
FEDORA : rsync rsync://mirror.aarnet.edu.au/fedora/linux/updates/12/i386/

As per you distribution rsync the data to the updates directory

FOR CENTOS RUN
[root@localhost ~]# rsync -avrt rsync://centos.arcticnetwork.ca/centos/5.4/updates/i386/RPMS/ --exclude=debug/ /var/www/html/yum/updates/
FOR CENTOS RUN
[root@localhost ~]# rsync -avrt rsync://mirror.aarnet.edu.au/fedora/linux/updates/12/i386/ --exclude=debug/ /var/www/html/yum/updates/

To get the updates regularly you can put the above command inthe crontab so that you updates directories are updated periodically

[root@localhost ~]# crontab -e 
* * 2 * * rsync -avrt rsync://centos.arcticnetwork.ca/centos/5.4/updates/i386/RPMS/ --exclude=debug/ /var/www/html/yum/updates/

You have created local yum repository, now to make clients on your network to use the local repository, create a file mylocal.repo in /etc/yum.repos.d

[root@localhost ~]# vim /etc/yum.repos.d/mylocal.repo
Add the Following Lines

[base-local]
name=Centos $releasever - $basearch
failovermethod=priority
baseurl=http://192.168.1.1/yum/base-pkg/
enabled=1
gpgcheck=0

[updates-local]
name=Centos $releasever - $basearch - Updates
failovermethod=priority
baseurl=http://192.168.1.1/yum/updates/
enabled=1
gpgcheck=0

THAT's IT, Now you can update all your packages over high Speed Lan connectivity instead of Internet.

Quick Understanding Linux Cron in under 3 Minutes

What is Cron?
In Layman Languages, cron is a Task scheduler used in Linux operating System (but with Great Powers if used with it Complete functionality) developed by Paul Vixie. You can schedule a specific task to be run every minute,hour,day,month.

Majorly cron is used to automate jobs daily without human intervention or manual running of command. Cron can be configured using configuration files which resides in "/etc", changes to which can be done only by the Administrator. Cron also has the feature to run commands using particular user.

| --------------------------------------------------------------    Minutes (00 - 59)
|    ------------------------------------------------------    Hours (00 - 24)
|    |    ----------------------------------------------    Day of Month (01 - 30)
|    |    |    --------------------------------------    Month (01 - 12)
|    |    |    |    ------------------------------  Day of Week (0 - 6) (Sunday = 0)
|    |    |    |    |    ----------------------    User to the Run the Command
|    |    |    |    |    |        ------    Command / Script to be Executed
|    |    |    |    |    |        |

*    *    *    *    *    root    /home/linuxmaza.com/htdocs/scripts/Stats_Update.sh

If you are a lazy System Adminstrator and Dont want to write the whole entry, cron has several special entries which are shortcuts for specifying a Complete entry.
You can specify entries as below

Entry       ||    Description          ||    Equivalent To
=====================================================
@reboot     Run once, at startup.     None
@yearly     Run once a year     0 0 1 1 *
@annually     (same as @yearly)     0 0 1 1 *
@monthly     Run once a month     0 0 1 * *
@weekly     Run once a week     0 0 * * 0
@daily         Run once a day         0 0 * * *
@midnight     (same as @daily)     0 0 * * *
@hourly     Run once an hour     0 * * * *

Operators in Cron
There are way of specifying multiple date and time in a field which can be overcomed using Operators in cron.

• The comma (',') operator specifies a list of values, for example: "1,3,4,7,8" (Spaces between the values are not accepted)


• The dash ('-') operator specifies a range of values, for example: "1-6", which is equivalent to "1,2,3,4,5,6"


• The asterisk ('*') operator specifies all possible values for a field. For example, an asterisk in the hour time field would be equivalent to 'every hour' (depending on other values entered in other fields).


• The Forward Slash ('/') operator is used to SKIP a given number of values. "Say the value we used is 5" "For Example, "*/5" in the hour time field is equivalent to "0,5,10,15,20" and so on.

Cron Advantages :
1. Can Handle Multiuser
2. Cron enables users to schedule jobs to run automatically at a certain time or date.
3. Automate daily System Administration tasks
4. Operators in Cron saves time and confusion while creating many crons.
5. Support for Special entries using (@reboot, @yearly, @monthly, @weekly, @hourly, @midnight, @daily).


Some Examples
1. If you want to run a task every Hour

0 * * * * root w > /var/log/mysystemstats.log
OR
@hourly root w > /var/log/mysystemstats.log

Above entry would run the "w" command at the start of every hour and redirect the output to "/var/log/mysystemstats.log"

2. Scheduling a cron job running every 15 minutes to delete apache Logs

15 * * * * echo " " > /var/log/httpd/access_log

3. Run a job every 5 hours
* 0,5,10,15,20 * * * echo " " > /var/log/httpd/access_log

Script to Check IP on SPAM Sources and getting EMAIL and SMS alerts.

As a System Administrators "The Key to a good performing Server is Good IP Reputation"

Maintaining a MAIL Servers or Web Servers is big time for System Admins,there is always a big fear for your IP getting Blacklisted on different SPAM sources due to which mails originated from your server either are not accepted or deffered by the Recipient Server.It would of great help if we get timely EMAIL as well as SMS alerts of IP getting blacklisted. This would make servers perform well as BOUNCES do use system resources which impact on system overall performance (Explaining Which in this article is OUT OF SCOPE.)

You can get this by Paying a Handsome amount of money to a Service Provider or DO MY way. I have written a Shell script using which you can get timely alerts as your IP gets blacklisted. You can put the script in crontab to run every Hour or so.

To START with Copy the script onto you system

# mkdir dnsbl && cd dnsbl

# vim dnsbl_check.sh

Copy the Below Script

#!/bin/bash

# Date: Mar 20,2010
# Author: Ashwin Muni
# Purpose: Check the IP Against Major SPAM Sources.

## Uncomment to Debug
# set -x

# Variables
tmp_file='/tmp/dnsbl'

#IN_DNSBL=127.0.0.[2-6]
#IN_DNSBL=127.0.0.
IN_DNSBL='127.0.0.2|127.0.0.3|127.0.0.4|127.0.0.5|127.0.0.6|127.0.0.7|127.0.0.8|127.0.0.9|127.0.0.10|127.0.0.'
DIG=`which dig`
MAIL_ADMIN="test@example.com ashwin@linuxmaza.com"

###################################################

# SCRIPT START

> $tmp_file

echo "Below IPs are Listed" >> $tmp_file

if [ "$#" == 1 ]; then

for i in `cat rbllist.txt`; do
IP_REV=`echo $1 | awk -F\. '{ print $4"."$3"."$2"."$1 }'`
$DIG $IP_REV.$i | grep $IN_DNSBL

if [ $? == '0' ]; then
#echo "$1 Listed on $i"
echo -e "\033[31m \033[1m PROBLEM : Listed on $i \033[0m \033[22m"
echo "################################ Attention : $1 Listed on $i" >> $tmp_file
else
echo -e "Not Listed on $i : \033[32m \033[1m OK \033[22m \033[0m "
echo "$1 Not Listed on $i" >> $tmp_file
fi
done

echo -e "\033[31m \033[1m ===================$1 is LISTED ON BELOW SPAM SOURCES====================== \033[0m \033[22m"

cat $tmp_file | mail -s "DNSBL REPORT FOR $1" $MAIL_ADMIN

else

echo -e "\t\t\t\t\033[31m \033[1m Enter Proper Arguments:\n Script Usage :\n /bin/sh $0 IP.ADD.RE.SS \033[0m \033[22m"

# EOF

################################################

Save the file Using ":wq"

Make necessary changes in the Script like the System Admin email address to sent Emails.

You will need the SPAM sources to check which you can find Here MAJOR SPAM SOURCES

Copy all the SPAM Sources and paste it in a txt file named "rbllist.txt"

# vi rbllist.txt

Should show you all the Major SPAM Sources for Checking your IPs.

Note: The script and the rbllist.txt should exist in the Same directory.

Once done we will give executable permission to the script which allows us to run it.

# chmod 755 dnsbl_check.sh

OR

# chmod +x dnsbl_check.sh

Now Run the Script

#./dnsbl_check.sh 100.200.100.200

You can put the script in crontab to run it regularly.

MAJOR SPAM Sources where your IPs can get blacklisted which could affect your service.

MAJOR SPAM Sources where your IPs can get blacklisted which could affect your service. Most of the Players in the market like Yahoo, Google, Hotmail use this for stopping huge SPAM to their servers. Many appliances do the same to STOP SPAM.

SPAM Sources

3y.spam.mrs.kithrup.com
access.redhawk.org
all.rbl.kropka.net
all.spamblock.unit.liu.se
assholes.madscience.nl
blackholes.five-ten-sg.com
blackholes.intersil.net
blackholes.mail-abuse.org
blackholes.sandes.dk
blackholes.uceb.org
blackholes.wirehub.net
blacklist.sci.kun.nl
blacklist.spambag.org
bl.borderworlds.dk
bl.csma.biz
block.dnsbl.sorbs.net
blocked.hilli.dk
blocklist2.squawk.com
blocklist.squawk.com
bl.redhatgate.com
bl.spamcannibal.org
bl.spamcop.net
bl.starloop.com
bl.technovision.dk
cart00ney.surriel.com
cbl.abuseat.org
dev.null.dk
dews.qmail.org
dialup.blacklist.jippg.org
dialup.rbl.kropka.net
dialups.mail-abuse.org
dialups.visi.com
dnsbl-1.uceprotect.net
dnsbl-2.uceprotect.net
dnsbl-3.uceprotect.net
dnsbl.ahbl.org
dnsbl.antispam.or.id
dnsbl.cyberlogic.net
dnsbl.kempt.net
dnsbl.njabl.org
dnsbl.solid.net
dnsbl.sorbs.net
dsbl.dnsbl.net.au
duinv.aupads.org
dul.dnsbl.sorbs.net
dul.ru
dun.dnsrbl.net
dynablock.njabl.org
dynablock.wirehub.net
fl.chickenboner.biz
forbidden.icm.edu.pl
form.rbl.kropka.net
hil.habeas.com
http.dnsbl.sorbs.net
http.opm.blitzed.org
intruders.docs.uu.se
ip.rbl.kropka.net
korea.services.net
l1.spews.dnsbl.sorbs.net
l2.spews.dnsbl.sorbs.net
lame-av.rbl.kropka.net
list.dsbl.org
mail-abuse.blacklist.jippg.org
map.spam-rbl.com
misc.dnsbl.sorbs.net
msgid.bl.gweep.ca
multihop.dsbl.org
no-more-funn.moensted.dk
ohps.bl.reynolds.net.au
ohps.dnsbl.net.au
omrs.bl.reynolds.net.au
omrs.dnsbl.net.au
opm.blitzed.org
op.rbl.kropka.net
orbs.dorkslayers.com
orid.dnsbl.net.au
or.rbl.kropka.net
orvedb.aupads.org
osps.bl.reynolds.net.au
osps.dnsbl.net.au
osrs.bl.reynolds.net.au
osps.dnsbl.net.au
osrs.bl.reynolds.net.au
osrs.dnsbl.net.au
owfs.bl.reynolds.net.au
owfs.dnsbl.net.au
owps.bl.reynolds.net.au
owps.dnsbl.net.au
pdl.dnsbl.net.au
probes.dnsbl.net.au
proxy.bl.gweep.ca
psbl.surriel.com
pss.spambusters.org.ar
rbl.cluecentral.net
rblmap.tu-berlin.de
rbl.rangers.eu.org
rbl.schulte.org
rbl.snark.net
rbl.triumf.ca
rdts.bl.reynolds.net.au
rdts.dnsbl.net.au
relays.bl.gweep.ca
relays.bl.kundenserver.de
relays.dorkslayers.com
relays.mail-abuse.org
relays.nether.net
relays.visi.com
ricn.bl.reynolds.net.au
ricn.dnsbl.net.au
rmst.bl.reynolds.net.au
rmst.dnsbl.net.au
rsbl.aupads.org
satos.rbl.cluecentral.net
sbl.csma.biz
sbl.spamhaus.org
sbl-xbl.spamhaus.org
smtp.dnsbl.sorbs.net
socks.dnsbl.sorbs.net
socks.opm.blitzed.org
sorbs.dnsbl.net.au
spam.dnsbl.sorbs.net
spam.dnsrbl.net
spamguard.leadmon.net
spam.olsentech.net
spamsites.dnsbl.net.au
spamsources.dnsbl.info
spamsources.fabel.dk
spamsources.yamta.org
spam.wytnij.to
spews.dnsbl.net.au
t1.bl.reynolds.net.au
t1.dnsbl.net.au
ucepn.dnsbl.net.au
unconfirmed.dsbl.org
vbl.messagelabs.com
vox.schpider.com
web.dnsbl.sorbs.net
whois.rfc-ignorant.org
will-spam-for-food.eu.org
wingate.opm.blitzed.org
xbl.spamhaus.org
zombie.dnsbl.sorbs.net
ztl.dorkslayers.com

Check you IP if is blacklisted using a script SPAM-IP Reputation Check Script

How to sent Message through Terminal to a Login users In Fedora, Centos, RedHat, Ubuntu.

You can broadcast a message using WALL -- How to broadcast a Message through Terminal to Login users How To but what if you want to sent message to only a particular user logged into the system or different Messages to Different users. Yes its possible using write command.

Before using write command Check if it is avaliable in you system. From the terminal Run

[root@localhost ~]# which write

which should give
[root@localhost ~]# which write
/usr/bin/write

If write is not installed Install bsdmainutil package

Once Installed to sent message to a particular user we need to know which users are currently logged into the System

[root@localhost ~]# w

USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
ashwin :0 - 15Feb10 ?xdm? 7:56m 0.21s /usr/bin/gnome-session
ashwin pts/1 :0.0 11:34 22.00s 1.08s 0.01s gnome-terminal
demo1 pts/2 :0.0 11:58 11.00s 1.07s 1.2s gnome-terminal
demo2 pts/3 :0.0 01:58 8.00s 0.16s 1.8s gnome-terminal
ashwin pts/4 :0.0 12:58 1:47 4.02s 0.02s bash


Now you have the list of users logged in.

METHOD 1
[root@localhost ~]# write ashwin pts/4
Your Message 1
Your Message 2
Your Message 3
Once Done

“Terminate the write by ctrl+D.”

The User "ashwin" would see the message into the console as

[ashwin@localhost ~]$
Message from ashwin@localhost (as root) on pts/2 at 12:49 ...
Your Message 1
Your Message 2
Your Message 3
EOF

METHOD 2
[ashwin@localhost ~]$ echo "Test Message Using Write from http://linuxmaza.com" > /dev/pts/3

The above command will display a message to User "demo2" on the console.

Write Command Runs on all linux distros and installed by default in Fedora, Centos. Please refer the documentation if it is not installed under you linux distribution.

How to broadcast a Message through Terminal to Login users In Fedora, Centos, RedHat, Ubuntu

Do you want to sent message to all logged in users in your system (This works great when you are a System Administrator and many users are logged into same System)?

If you are Running Any Linux Distribution like Fedora, Centos, RedHat, Ubuntu you can send or broadcast message to all user logged in using "WALL" command. To use it on the command line / Terminal.

METHOD 1
========

[root@localhost ~]# wall "Your message Goes here"

Now all users can see the Message Broadcasted by You

[ashwin@localhost tmp]$
Broadcast message from root (pts/2) (Mon Feb 22 12:10:35 2010):

Your message Goes here

METHOD 2
========
[shankar@localhost ~]$ wall [Press Enter]
Write You Message Here.
Testing Wall OutPut On All Terminals.
Finished

Press Ctrl + D for EOF (End of File)

[root@localhost ~]#
Broadcast message from shankar (pts/3) (Mon Feb 22 12:14:55 2010):

Write You Message Here.
Testing Wall OutPut On All Terminals.
Finished

All Users can view the Message on their Terminal as below

[root@localhost ~]#
Broadcast message from shankar (pts/3) (Mon Feb 22 12:14:55 2010):

Write You Message Here.
Testing Wall OutPut On All Terminals.
Finished

Limitations of Wall :
1. The total length of the message is limited to 22 lines.
2. Broadcasted Messages are limited to The System Itself.

Advantages of Wall
1. During an Activity / Crisis on the system all users can be notified immediately
2. No need to Send Email to the users who are Logged into the System.

Mounting SSH Server Using FUSE

FUSE which stands for Filesystem in Userspace has brought a host of features to mount a remote filesystem using various protocols like SSH, FTP. FUSE allows non privileged users to develop their own filesystem without even modifying the Kernel, basically fuse is module which play a role of bridge between user space and the kernel.

To Mount a remote partition locally. Run the Command (You Dont Need To be Root User to Run this Command).

root# sshfs user@hostname:path_of_directory_to_be_mounted mount_to_point

root# sshfs ashwin@192.168.1.10:/home/ashwin /mnt/

If You are using a specific port for connecting to your SSH Server use -p Option

root# sshfs -p 10234 ashwin@192.168.1.10:/home/ashwin /mnt/

where 10234 is the port running SSH Service
Important Note : To Mount other Partition like /var, /opt, /root you have to mount being a root user ofthe remote machine.

root# cd /mnt

root# ls

msrv2826d0.1.tmp
keyring-0sIxHo keyring-K0zJSJ msrv2832d0.1.tmp
keyring-16cIRa keyring-kJqkFX mysqlbin.000085
keyring-1h8sea keyring-kzkqpE mysqlbin.000086
keyring-1M5nUH keyring-lyNVMO mysqlbin.000087
keyring-2i7woO keyring-m0dxjd mysqlbin.000088
keyring-4hSFtb keyring-m1xNam mysqlbin.000089
keyring-68PwBN keyring-mTgrKf mysqlbin.000092
keyring-7JY4q3 keyring-NDLFUc mysqlbin.000093
keyring-7zjMDV keyring-nt2Ygk mysqlbin.index


Enjoy, you have mounted a remote directory locally that to on secured transmission.

How to mount RAMFS, TMPFS in Centos, RHEL, Fedora, Ubuntu, Linux

How to Use and mount RAMFS / TMPFS in Centos, RHEL, Fedora, Ubuntu?
Definition :
RAMFS Random Access Memory Filing System
TMPFS Temporary File System / Temporary File Storage

RAMFS / TMPFS is used to assign or allocate a part of physical memory to be used as a individual partition which can be used for storing data (Reading, Writing) like all other Disk partition on a system. As the data is stored in RAM (Random Access Memory) it is much faster than that of writing or reading from a System Physical disk Partition.
Performance can be measured when writing or Reading a large file from this partition (Small File size wont give you the performance). Mounting RAMFS and TMPFS is same on almost all linux distribution RHEL, Centos, Ubuntu, Fedora.

1. How to mount Tmpfs

# mkdir -p /mnt/tmp
# mount -t tmpfs -o size=200m tmpfs /tmp/ashfs

The last line in the following df -h shows the above mounted /tmp/tmpfs tmpfs file system.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda2 24G 11G 12G 47% /
/dev/hda1 99M 12M 83M 12% /boot
tmpfs 248M 0 248M 0% /dev/shm
tmpfs 200M 0 200M 0% /tmp/ashfs

2. How do I mount Ramfs on the System?

# mkdir -p /tmp/ashfs

# mount -t ramfs -o size=200m ramfs /tmp/ashfs

You can change the size of the partition on the Fly.

# mount -o remount,size=2G /tmp/ashfs
It grows dynamically
You can verify the mount using

# mount
/dev/hda2 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/hda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
tmpfs on /tmp/ashfs type tmpfs (rw,size=200m)

By adding the partition in /etc/fstab they RAMFS/TMPFS can be mounted on boot time.

RAMFS Uses RAM of a System It grows dynamically Allows to write data above the partition created After Full Usage system may hang or stop working. Data would be deleted once system is rebooted or incase of crashing. You have to make a process to transfer data from RAMFS to Physical disk periodically to loose data Suitable to have less data and read intensive

TMPFS Uses Swap of a System. It doesn't grow dynamically. Doesn't allow to write data above the partition created. System would give an error display “No space left on device”. Data would be deleted once system is rebooted or incase of crashing. You have to make a process to transfer data from RAMFS to Physical disk periodically to loose data. Suitable if you want to write very fast Write intensive.

Example : You have a system with 2GB RAM and 2 GB Swap Space
RAMFS : When a partition of 1 GB is created and mounted users can even write more than 1GB till the total RAM size is exhausted as the parition is made on RAM.

TMPFS : When a partition of 1 Gb is created and mounted users wont be able to use more than 1GB space as designated on the partition. If required the space can be extended on a fly. Useses dont need to worry to control the process that writes on tmpfs partition