sed & awk commands

In this post I would like to introduce linux’s powerful commands ‘sed’ and ‘awk’. These are two versatile commands to handle files. With sed, you can automate commands on text files. awk allows you to  manipulate files that are structured as columns of data and strings.

Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).While in some ways similar to an editor which permits scripted edits
(such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed’s ability to filter text in a pipeline which particularly distinguishes it from other types of
editors.

The general format of sed is:
sed options sed-script filename

sed is a command used to read text files line by line and search, substitute, remove lines matching a pattern and does many other interesting text manipulation functions. It outputs the transformed text to the console by default, although you can redirect it to another file or selectively supress it using one of the sed options. Most applications of sed command were in text substitution, but its not limited to it.

Some of the common uses of sed is given below.

1)Displaying lines

suppose I have a file users:

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ cat users
lora
louri
jinesh
linda
mathew
ajay
arun
anoop
joy
john

The following command display first two lines of file. With this command, you tell sed to show two lines, and then quit (q)

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed 2q users
lora
louri

Searches for lines that contain string ‘or’ and then execute an operation.
‘n’ is used to suppress the output. Otherwise the matching pattern print twice. P is used to print the output. following the file name.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed -n /or/p users
lora

following command searches for lines that starts with j, and print output.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed -n /^j/p users
jinesh
joy
john

following command searches for all lines, because ‘.’ represents anycharacter.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed -n /./p users
lora
louri
jinesh
linda
mathew
ajay
arun
anoop
joy
john

2)Deleting arbitary lines in the input file
inorder to delete first line of file.

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed '1d' users
louri
jinesh
linda
mathew
ajay
arun
anoop
joy
john

inorder to delete last line of file:

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed '$d' users
lora
louri
jinesh
linda
mathew
ajay
arun
anoop
joy

3) Replace or substitute the occurrences of a pattern

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed ‘s/joy/jyothi/g’ users lora louri jinesh linda mathew ajay arun anoop jyothi john

4) Delete all occurrences of a pattern in a file

jinesh@jinesh-Compaq-Presario-CQ40-Notebook-PC:~$ sed '/mathew/d' users
lora
louri
jinesh
linda
ajay
arun
anoop
joy
john

AWK command
Short for Aho, Weinberger, and Kernighan, awk is a script processing language also known as oawk, gawk, mawk and nawk allows for pattern scanning and processing. Awk has two faces: it is a utility for performing simple text-processing tasks, and it is a programming language for performing complex text-processing tasks. More complex tasks should be written as awk programs (“scripts”) to a file.

basic format:
awk ‘/pattern/{action}’ file

when using script file basic format is:

awk [program | -f program file] [flags / variables] [files]

To print first column of a file.
awk ‘{print $1}’ file

Renaming file:
ls users | awk ‘{print “mv “$1” “$1″.new”}’ |sh
Above command change file name ‘users ‘ to  ‘users.new’

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s