Linux commands for daily usage

File/Basic commands
These are some useful basic linux commands. For every command I followed this convention.
1)The name of the command with a short description is given.
2)Some of the useful options that can be used with the command is given after the command.

ls – list the current directory contents.
-l long listing
-a list including hidden files
-t list by last modified

cat – show the content of a file.
format: cat [options] [file name(s)]
-n show the line numbers also.

head – output the first 10 lines of each FILE to standard output
format: head [options] [file name(s)]
–bytes=5k or -c 5k
-n specify the number of lines.

tail – output last part of file
same as head

sort – sort lines of text files
sort [OPTION]… [FILE]…
-n numerical sort
-f ingnore case
-r reverse the results
-R random
-d in dictinary order

free – Display amount of free and used memory in the system
free [-b|-k|-m|-g]
-b bytes
-k kilo bytes
-m mega bytes
-g giga bytes

Search commands:

find – search for files in a directory hierarchy
find [directory to search] [option] [filename]
eg:- find . -name abc.py

find [directory] -group gname (File belongs to group gname)

grep – print lines matching a pattern
grep [options] [pattern] [filename]
eg:
grep ‘head’ words.txt
grep ‘^head’ words.txt
grep ‘head$’ words.txt
grep ‘^head$’ words.txt
grep ‘^….$’ words.txt
grep ‘^h.*d$’ words.txt

sed – stream editor for filtering and transforming text
sed [OPTION]… {script-only-if-no-other-script} [input-file]…
two examples:
sed ‘s/cat/dog/g’ data ( substitute all cat with dog globally )
sed 2,4d data ( delete lines from 2 to 4 in file data)

Let’s Write some simple shell scripts

A shell is a command line interpreter. It takes commands and executes them. As such, it implements a programming language.

Let’s look how to write shell script to print hello world
Steps:
1)write your shell script on some editor like vim.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ vim first

#content of my first shell script
clear
echo "Hello world"

Description:
‘#’ is used to denote comment line.
‘clear’ clear the screen when shell executes.
‘echo’ prints the output on screen.

if we try to run script directly we get the following result.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ ./first
bash: ./first: Permission denied

2)so our second step is to set permission for the shell script.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ chmod 755 first

3)run the script using ‘./ script-name’

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ ./first

Hello world

Now lets look on another example. This script print the user information, current date and calender.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ vim ginfo

#script to print user information who currently login, current date & time

clear
echo "Hello $USER"
echo "Today is  ";date
echo "Number of user login : " ; who |wc -l
echo "Calender"
cal
exit 0

Setting permission and run script.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ chmod 755 ginfo
jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ ./ginfo

Outpt:

Hello jinesh
Today is  
Sun Aug  7 21:21:44 IST 2011
Number of user login : 
2
Calender
    August 2011       
Su Mo Tu We Th Fr Sa  
    1  2  3  4  5  6  
 7  8  9 10 11 12 13  
14 15 16 17 18 19 20  
21 22 23 24 25 26 27  
28 29 30 31

Linux make command

The make command helps us to manage large programs or groups of programs. When we are working with large files, the compilation/recompilation time is much longer. Most of the time in recompilation we might have changed some part of program. In such cases we don’t need to recompile the program. ‘make’ command helps us in this aspect of compilation.

The make program aids you in developing your large programs by keeping track of which portions of the entire program have been changed, compiling only those parts of the program which have changed since the last compile.

By creating a descriptor file containing dependency rules, macros and suffix rules, you can instruct ‘make’ to automatically rebuild your program when one of the program’s component files is modified. Make is smart enough to only recompile the files that were affected by changes thus saving compile time.

Lets look at what happens when we compile a C program. The three different tasks required to produce the executable program are as follows:

Compile green.o: cc -c green.c
Compile blue.o: cc -c blue.c
Link the parts together: cc green.o blue.o

Now lets see how ‘make’ command works with an example:
Suppose we have three source files file1.c, file2.c, and file3.c. An include file, mydefs.h, is required by files file1.c and file2.c. We have a descriptor file called ‘prog1’ If you wanted to compile this file from the command line using C++ the command would be

% CC -c prog1 file1.cc file2.cc file3.cc

This command line is rather long to be entered many times as a program is developed and is prone to typing errors. A descriptor file could run the same command better by using the simple command

% make prog1

or if prog1 is the first target defined in the descriptor file

% make

Now lets look how to write a descriptor file. To operate ‘make’ needs to know the relationship between your program’s component files and the commands to update each file. This information is contained in a descriptor file (prog1) you must write called Makefile or makefile.

	   
     OBJS = file1.o file2.o file3.o

     prog1 : $(OBJS)
           CC -c prog1 $(OBJS)

     file1.o : file1.c mydefs.h
             CC -c file1.cc

     file2.o : file2.c mydefs.h
             CC -c file2.c

     file3.o : file3.cc
             CC -c file3.c
		 clean :
     rm $(OBJS)

Let’s go through the example to see what make does by executing with the command make prog1 and assuming the program has never been compiled.

1)make finds the target prog1 and sees that it depends on the object files file1.o file2.o file3.o
2)make next looks to see if any of the three object files are listed as targets. They are so make looks at each target to see what it depends on. make sees that file1.o depends on the files file1.c and mydefs.h.

3)Now make looks to see if either of these files are listed as targets and since they aren’t it executes the commands given in file1.o’s rule and compiles file1.c to get the object file.
4)make looks at the targets file2.o and file3.o and compiles these object files in a similar fashion.
5)make now has all the object files required to make prog1 and does so by executing the commands in its rule.

References:

http://frank.mtsu.edu/~csdept/FacilitiesAndResources/make.htm
http://www.eng.hawaii.edu/Tutor/Make/index.html

sed & awk commands

In this post I would like to introduce linux’s powerful commands ‘sed’ and ‘awk’. These are two versatile commands to handle files. With sed, you can automate commands on text files. awk allows you to  manipulate files that are structured as columns of data and strings.

Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).While in some ways similar to an editor which permits scripted edits
(such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed’s ability to filter text in a pipeline which particularly distinguishes it from other types of
editors.

The general format of sed is:
sed options sed-script filename

sed is a command used to read text files line by line and search, substitute, remove lines matching a pattern and does many other interesting text manipulation functions. It outputs the transformed text to the console by default, although you can redirect it to another file or selectively supress it using one of the sed options. Most applications of sed command were in text substitution, but its not limited to it.

Some of the common uses of sed is given below.

1)Displaying lines

suppose I have a file users:

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ cat users
lora
louri
jinesh
linda
mathew
ajay
arun
anoop
joy
john

The following command display first two lines of file. With this command, you tell sed to show two lines, and then quit (q)

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed 2q users
lora
louri

Searches for lines that contain string ‘or’ and then execute an operation.
‘n’ is used to suppress the output. Otherwise the matching pattern print twice. P is used to print the output. following the file name.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed -n /or/p users
lora

following command searches for lines that starts with j, and print output.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed -n /^j/p users
jinesh
joy
john

following command searches for all lines, because ‘.’ represents anycharacter.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed -n /./p users
lora
louri
jinesh
linda
mathew
ajay
arun
anoop
joy
john

2)Deleting arbitary lines in the input file
inorder to delete first line of file.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed '1d' users
louri
jinesh
linda
mathew
ajay
arun
anoop
joy
john

inorder to delete last line of file:

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed '$d' users
lora
louri
jinesh
linda
mathew
ajay
arun
anoop
joy

3) Replace or substitute the occurrences of a pattern

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed ‘s/joy/jyothi/g’ users lora louri jinesh linda mathew ajay arun anoop jyothi john

4) Delete all occurrences of a pattern in a file

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed '/mathew/d' users
lora
louri
jinesh
linda
ajay
arun
anoop
joy
john

AWK command
Short for Aho, Weinberger, and Kernighan, awk is a script processing language also known as oawk, gawk, mawk and nawk allows for pattern scanning and processing. Awk has two faces: it is a utility for performing simple text-processing tasks, and it is a programming language for performing complex text-processing tasks. More complex tasks should be written as awk programs (“scripts”) to a file.

basic format:
awk ‘/pattern/{action}’ file

when using script file basic format is:

awk [program | -f program file] [flags / variables] [files]

To print first column of a file.
awk ‘{print $1}’ file

Renaming file:
ls users | awk ‘{print “mv “$1” “$1″.new”}’ |sh
Above command change file name ‘users ‘ to  ‘users.new’