SHELL SCRIPTING

One Step Towards Automation

Objective of the Training

This training is designed to introduce shell utilities, shell scripting and its concepts to users having some experience of working on unix / linux platforms .

Lets begin
% cat > hello.sh #!/bin/sh echo Hello, world ### now we exit exit 0 <ctrl-d> % chmod +x hello.sh % ./hello.sh

% echo $?
3

Overview of Shell Scripts

Set of shell commands Execution flow can be changed Uses flow control constructs and filters Shell scripts are interpreted The status is stored on $?

Filters

Filters
1- Filters take input from the standard i/p files piped o/p 2- Perform an operation on the data . 3- Send the o/p to standard o/p or files .

 cat : concatenate and display the contents of

files.
$> cat > tempfile.txt This is a temporary file. Delete it afterwards. <ctrl-d> $> $> cat existingfile.txt This is an existing file. $> $> cat tempfile.txt existingfile.txt > new.txt $> cat new.txt This is a temporary file. Delete it afterwards. This is an existing file. $>
7

 cut : extracts a list of columns or fields from files.

-c for characters selection -f for fields selection -d for delimeter (valid only with f , default is tab) -s suppresses lines with no delimiter characters in case of f option $> myFriends=alok,sachin,amir,ghazni,amar

$> echo $myFriends alok,sachin,amir,ghazni,amar

$> echo $myFriends | cut d , f 2 sachin $> echo $myFriends |cut d, f 3, 5 amir,amar $> echo $myFriends |cut c 3-5 Ok,
8

paste : combines files by columns.

d delimeter list <default tab>

$>paste tempfile.txt existingfile.txt > see.txt $> cat see.txt This is a temporary file. This is an existing file. Delete it afterwards. $>

$> paste d * tempfile.txt existingfile.txt This is a temporary file.*Delete it afterwards. This is an existing file.* $>

 head : displays first few lines of a file.

$>head myfile.txt This shows, the first 10 lines of myfile.txt

$>head n myfile.txt This displays the first n lines of myfile.txt.

tail :

shows the last few lines of a file.

$> tail myfile.txt $>tail n myfile.txt This displays the last n lines of myfile.txt.
10

more : displays a text file or output of any

command one full screen at a time $> more aVeryLargeFile

less : same as more

but - more efficient and with more options $> less aVeryLargeFile

11

sort :

sorts alphabetically -n sort numerically -r reverse order -u uniq -t specifies delimeter ( default tab) -k field selection

$> sort fruitsName $> sort u fruitsName $> sort -nr classMarksFile

$> sort -n -t: -k 3 classMarksFile

12

tr :
translates inputs to output as specified.

$> cat file1.txt | tr [A-Z] [a-z] > file2.txt $> tr [A-Z] [a-z] < file1.txt > file2.txt $> tr aeiou AEIOU < file1.txt > file2.txt

13

Common Filters (cont.)

wc :
counts -> characters , words and lines eg : wc , wc m ,wc w ,wc l $> wc l fileName

14

Regular Expressions
. ^ $ [a-z] a single character start of a line end of a line any single character from a to z

[ab^c] x* xy a

a, b ^ or c zero or more occurrence of x x followed by y Character a

[^a-z]

any single character not in given range

15

Regular Expressions (cont. )

[.*] matches . or *

Supported only by egrep a? a+ a|b (a|b)c zero or one occurrence of a one or more occurrence of a either a or b either ac or bc

16

Grep Family
grep - prints lines that match pattern
grep [ options ] < pattern > < filename >

cmd | grep [ option ] < pattern > Here cmd is any unix command that displays o/p
Options: -n: print line number -v: invert the search criteria -i : makes search criteria case insensitive -c: display only the number of occurence
17

Usage of grep family

$> cat nf.txt
unix is a better OS minix is mini unix in St we have Solaris. ST has developed st pc. $>grep ^unix nf.txt unix is a better OS

$>egrep (u|mi)nix nf.txt unix is a better OS minix is mini unix

$>grep i st nf.txt in St we have Solaris. ST has developed st pc.
19

$> cat nf.txt unix is a better OS minix is mini unix in St we have Solaris. ST has developed st pc.

$>grep S[tT] nf.txt in St we have Solaris. ST has developed st pc $>grep nix.*nix nf.txt minix is mini unix
$>grep mi[^ ]*ix nf.txt minix is mini unix $>
20

Shell Variables
21

To set a sh variable VAR=value

To use the value of the variable $VAR or ${VAR} example #!/bin/sh COLOR=yellow echo This looks $COLORish echo This seems ${COLOR}ish prints This looks This seems yellowish There is only one type of variable in sh: strings.
22

 Local vs. environment variables

Environment variables are passed to subprocesses. Local variables are not. By default, variables are local. local variable -> environment variable through export export VAR To unexport a variable unset VAR
23

Shell Meta-Characters
* Matches any string including null string ? [xyz] [a-z] [^xyz] Matches a single character matches x or y or z matches any character from a to z Matches any characters except x, y, z

[^a-z]

Matches any character except from the range a-z

24

Usage of Meta-Characters
$>ls file1.txt file2.txt new.txt now.txt $>ls * file1.txt file2.txt new.txt now.txt $>ls file?.txt file1.txt file2.txt $>ls n[a-z]w.* new.txt now.txt

$>ls [^f-h]*.* new.txt now.txt

25

Quoting
Backslash a backslash (``\'') removes any special meaning from the character that follows. echo \wow\ wow Single quotes Single quotes, such as echo '* MAKE $$$ FAST *' * MAKE $$$ FAST *

26

Quoting (cont..)
Double quotes
Double quotes, such as "foo" foo $HOME /home/ramesh

Backquotes
If you have an expression within backquotes (also known as backticks), e.g., `cmd` echo You are `whoami` prints You are ramesh

27

Different Shell Symbols

Symbol
cmd > file cmd >> file cmd < file cmd1 | cmd2

Meaning to shell
redirects the o/p of cmd to file appends the stdout to file redirects stdin from file stdout from cmd1 redirected to stdin of cmd2

cmd1 &
cmd1 ; cmd2 `cmds`

cmd1 goes to background

executes comd1 and then cmd2 cmds are executed and output of final cmd replaces the backquoted expression
28

Different Shell Symbols ( cont. )

Symbol
( cmds )

Meaning to shell
executes commands in sub-shell

{ cmds ; } \char string string

execute cmds in current shell takes char literally takes string literally Takes string after recognizing $, `cmds` and \

#cmd

takes as comment, does not perform any operation

29

Illustration of Shell Symbols

$>x=1 $>echo $x 1 $>( x=5 ) $>echo $x 1 $>{ x=5 ; } $>echo $x 5
30

Error Handling 0 1 2 cmd >& n cmd <& n cmd 2> file cmd 2>& n cmd 1> file 2>file2 file descriptor for standard input file descriptor for standard output file descriptor for standard error redirects o/p from cmd to fd n redirects i/p from file (fd n) to cmd redirects standard error from cmd to file redirects standard error from cmd to file with fd n redirects standard o/p from cmd to file and standard error to file2
31

Error Handling in Command line

$>x=5
$>expr $x 5 $>expr $y Usage: expr expression $>expr $y > op1 Usage: expr expression

$>cat op1
$>expr $y 2> op1

$>cat op1 Usage: expr expression $>expr $y 2> /dev/null

32

Argument Passing to Script

Arguments are referred as $n (n <10 ) shift helps when n>9

$n -- nth positional parameter

$0 -- name of the executing file
33

Symbol $n $* $@

Meaning to shell scripts value of nth positional parameter holds all the positional parameters same as $*, but when quoted $* is $1 $2 $3 $4 $5.. but $@ $1 $2 $3 etc. Total no. of positional parameters passed to script excluding $0

$#

$$
$! $$?

process id of the current shell

process id of last background job option flag set for the shell exit status of the previous command
34

Example
$> cat printArguments #!/bin/sh echo "first argument is $1 " echo "second argument is $2 " echo "third argument is $3 " echo "numberof arguments -- $# " echo "all arguments together -- $* " echo "program name is $0 " exit 0 $> chmod 755 printArguments $> printArguments How are you Mr Goblin first argument is How second argument is are third argument is you numberof arguments -- 5 all arguments together -- How are you Mr Goblin program name is printArguments 35

$>cat script.sh #!/bin/sh echo $0 echo $# echo $1 $2 $3 $4 shift echo $1 $2 $3 $4

$>script.sh first second third fourth fifth script 5 first second third fourth second third fourth fifth
36

Shell Built-in Functions

37

 eval - arguments are read, evaluated , concatenated and

then executed
$>ls myfile.txt nf.txt $>var1=ls $>var2= | more $>$var1 $var2 |: No such file or directory more: No such file or directory

Built-in Functions

$>eval $var1 $var2 myfile.txt nf.txt

38

 exec
- Does not create a child process. - for the new process it keeps the same PID - the current process is terminated with it $> cat testExec.sh
#!/bin/bash exec echo "leaving this script forever $0" # Exit from script here. # ---------------------------------# The following line never happens echo "This echo will never echo.

$> testExec.sh leaving this script forever testExec.sh

39

 read
- is used to read from the standard input

- if read is used in a shell script, it waits there till input is given by the keyboard.

40

$> read name Peter $> echo $name Peter

$> read name Saint Peter

$> echo $name Saint Peter $>read first_name second_name Saint Peter

$>echo $first_name -- $second_name Saint -- Peter

41

Shell Built-in Functions: readonly

readonly
- this is used to make a variable readonly, so in a script you dont change it accidentally. - variables can be read but cant be overwritten - if used without arguments it gives a list of readonly variables in the current shell
42

 set

- is used to set options to the shell (used in debugging )

- displays all the variables and their values

- by default it maps to the positional parameters

43

$> set dr 56 78 $>echo $1 $2 $3 dr 56 78 $> echo $# 3

44

Debugging Shell Script

Debugging option set while invoking, #!/bin/sh options set options inside the script. Options are x, -v, -n, -e, -u
-x : prints command and their arguments as they are read -v: prints shell input lines as they are read -n: check syntax but dont execute -e: exit whenever return status is non zero -u: treat unset variables as errors

Shell variables $- tells the option passed to shell.

45

test : Built-in Function

test
- test command used for variables comparison - used for string comparison - used for checking file status test is used as test [option] <expression>
46

 string comparison
test z string - true if length of string is zero test n string - true if length of string is non zero test string1 = string2 - true if string1 is equal to string2

test string1 != string2 - true if string1 is not equal to string2

47

 File status checking

test f file - true if file exists and an ordinary file test d file - true if file exists and is a directory test r file - true if file has read permission test w file - true if file has write permission
51

File status checking test x file -true if file is executable test file1 nt file2 - true if file1 is newer than file2 test file1 ot file2 - true if file1 is older than file2
52

Examples of test ( file check)

$> ls dt1 dt2 $>cd dt1 $>ls file1.txt file2.txt $>test f file1.txt $>echo $? 0 $>
53

$>test f file3.txt $>echo $? 1 $>cd ../ $>test f dt1 $>echo $? 1 $>test d dt1 $>echo $? 0 $>
54

expr : Built-in Function

Main operators String Matching Operators

Integer Arithmetic Operators

58

Matching operator.
Compares the characters in arg1 with the characters in arg2. Returns the number of characters in arg1 matched by characters in arg2. Arg1 should be a string and arg2 should be a regular expression ( with the exception of the ^ character, because all expressions start at the beginning of arg2)

$> expr string : str 3

$> expr string : ing 0

$> expr string : strg 0 $> expr string : '.*i' 4

59

use the \(...\) notation to buffer and print

$> expr string : '\(.*\)' string $> expr string : '..\(..\)' ri $> expr string : '\(...\)' str $> expr string : '.*\(...\)' ing $> expr string : '\(stx\) $> expr string : 'st\(.*\)' ring

60

Integer Arithmetic Operators.

arg1 \* arg2 $> expr 80 \* 24 1920 arg1 / arg2 $> expr 15 / 4 3

arg1 % arg2 $> expr 15 % 4 3

arg1 + arg2 $> NUM=0 $> expr $NUM + 1 1 arg1 - arg2 $ expr $NUM - 1 0 61

Logic: test
test 1 lt 10

[ 1 lt 10 ]

62

Logic: test
[ [ [ [ -f /etc/passwd ] ! f /etc/passwd ] -f /etc/passwd a f /etc/shadow ] -f /etc/passwd o f /etc/shadow ]

63

Using bc inside shell script

$> x=`echo 1.5 * 1.2 | bc` $>cmp=`echo 2.0089 >0.003|bc` if [ $cmp eq 1 ] ; then echo 2.0089 is greater fi
64

Flow Constructs
65

Conditional Flow Constructs

If
if condition ; then
expressions else expressions fi

66

Conditional Flow Constructs (cont.)

We can also have multi condition checking if cond1 then statements else if cond2 then statements. else statements fi fi
67

# see if a file exists if [ -e /etc/passwd ] then echo /etc/passwd exists else echo /etc/passwd not found! fi

68

Multi-way Branching
case
case word in pattern1) commands ;; pattern2 | pattern3) commands ;; *) commands ;; esac
69

Loop Constructs : for

for i in 1 2 3 do echo $i done
70

for ..
for i in * do echo Listing $i: ls -l $i done

71

Logic: while
while condition do : done

72

Logic: while
a=0; LIMIT=10 while [ "$a" -lt "$LIMIT" ] do echo -n "$a a=`expr $a + 1 ` done
73

Displaying the arguments

#!/bin/sh i=1 while [ $# -ge 1 ] ; do echo $i -> $1 i=` expr $i +1` shift done exit 0
74

Logical Operators
|| and && operator

&& operator can be used to execute a command and if it is successful, execute the next command in the list. cmd1 && cmd2 if cmd1 succeeds, execute cmd2 else nothing

75

|| operator is used to execute a 2nd command if it gets a non zero exit status in the 1st command

cmd1 || cmd2 if cmd1 succeeds, then only cmd1 if cmd1 fails, execute cmd2

76

Function
sh supports a function call in which the value of variables are treated as global The General Syntax is function () { expressions }

77

 In the body of shell script, it will be called as function not function() Function should have been defined before being called . Arguments are passed to a function in the same way as passed to a script
78

file descriptors
How to create a file descriptors exec 6</tmp/foo opens the file /tmp/foo for input on the file descriptor #6. This is equivalent to the system call open("/tmp/foo", O_RDONLY) exec 7>/tmp/bar opens the file /tmp/bar for output on the file descriptor #7. If the file already exist it is recreated. This is equivalent to the system call open("/tmp/bar", O_WRONLY|O_CREAT). exec 7>>/tmp/bar opens the file /tmp/bar for appending on the file descriptor #7. If the file does not exist it is created. This is equivalent to the system call open("/tmp/bar", O_APPEND|O_CREAT). exec 6<&close the file descriptor.
79

#!/bin/bash cat /tmp/foo | while read a ; do echo $a >>/tmp/bar done The input file is a file containing 2.000.000 lines. This means that the shell script is going to open, write, and close /tmp/bar 2 million times. Now using the same shell script with file descriptors. #!/bin/bash # open the files exec 6</tmp/foo exec 7>/tmp/bar # data "processing" cat <&6 | while read a ; do echo $a >&7 done exec 6&exec 7&the performance increase to almost 4.5.
80

SED
81

 works line by line throws the o/p on console the original file is left unchanged.

82

sed
The syntax for the utility is: sed [options] '{command} [filename]

83

Sed for line printing

$>cat op1 er1 file1.txt file2.txt new.txt now.txt op1 $>sed $q op1 er1 file1.txt file2.txt new.txt now.txt op1

84

Sed for line printing (cont.)

$>sed 3q op1

er1 file1.txt
file2.txt $>sed /n[a-z]w/q op1 er1 file1.txt file2.txt new.txt $>sed n $p op1 op1 $>

n option is used turn off the automatic printing option of sed

85

Sed used for printing

sed n m, np file
- prints m to n lines of the file

sed n /pattern/,$p file

- prints the lines that contains pattern to the end of file
86

$>sed n /file/p op1 file1.txt file2.txt $>sed n /file/,/n[a-z]w/p op1 file1.txt file2.txt new.txt

The bang ( ! ) above reverses the matched criteria

$>sed n /n[a-z]w/!p op1 er1 file1.txt file2.txt op1

87

 d option is used in place of p for deletion

$>sed /file/,/n[a-z]w/d op1 er1 now.txt op1 $>

88

 sed is extensively used to match a pattern and replace

sed s/pattern1/pattern2/ file pattern1 is searched in file and replaced by pattern2 at the 1st occurrence of each line sed s/patter1/pattern2/g file pattern1 is replaced by pattern2

globally

89

Examples: search and replace

$>cat op1 er file1.txt file3.txt file2.txt file4.txt $>sed s/file/changed_file/ op1 er changed_file1.txt file3.txt changed_file2.txt file4.txt $>
90

$>cat op1 er file1.txt file3.txt file2.txt file4.txt

$>sed s/file/changed_file/g op1 er changed_file1.txt changed_file3.txt changed_file2.txt chnaged_file4.txt

91

Matching Patterns in sed

pattern /^/ matches in sed All lines

/./
/^$/ /st/ /^st/ /st$/ /^st$/ /^st.$/ /st\.$/

Non-empty i.e.atleast 1 character

Empty or blank lines st anywhere in the line st at the beginning of the line st at the end of the line line containing only st line containing only st plus any 1 char Line containing st. at the end
92

Matching Patterns in sed (cont.)

pattern
/[sS]t/ /st[0-9]/ /st[^0-9]/ /st[0-9][^0-9]/

matches in sed
st or St anywhere in the line st followed by a digit st followed by a non-digit st followed by a digit and then followed by a non digit sgs followed by any string and then by st

/sgs.*st/

/sgs.*st$/

sgs followed by any string and then by st at the end of the line

/\/st\//

line containing /st/ any where in the line

93

 References The ampersand (&) holds the matched string: $> sed 's/[aeiou]/&&/g'

echo okay | sed s/[aeiou]/&&/g

94

 Referencing a substring Substrings enclosed with "\(" and "\)" can be referenced with "\n" (n is a digit from 1 to 9) $> sed 's/^\(\).*\(\)$/\2:\1/'

95

awk
awk runs through a text file. awk reads and process one record at a time. each field is separated by a variable FS

96

awk
General syntax of awk is
awk BEGIN { ##begin section (optional ) } { ### main section } END { ### end section (optional) } filename
97

Features
inbuilt variables
inbuilt functions flow control

hash tables
98

Special Variable Description

FILENAME FS NF NR RS OFS current filename field separator, default: blank no. of field in the current record no. of current input record record separator, default: new line o/p field separator, default: blank

ORS

output record separator, default: newline

$0 $n

entire input record nth field of current record, the fields are separated by FS
99

Math operator in awk

x+y x-y x*y x/y x%y x=y adds x to y adds x - y multiplies with x with y divides x by y gives the remainder assign value of y to x

x++
x-x += y

increments x by 1
decrements by 1 x=x+y

x -= y

x=x-y
100

Math operator and functions :

x *= y x /= y x %= y x=x*y x=x/y x=x%y

int(x)
rand cos(x) sin(x) atan2(x, y) exp(x) log(x)

truncates x to a whole no
a random no. between 0 and 1 cosine of x sine of x arc tangent of y/x e raised to the power of x logarithm of x

sqrt(x)

non negative square root of x

101

Example
$>cat op1 er file1.txt file3.txt now.txt File4.txt new.txt

$>awk {print $0} op1 er file1.txt file3.txt now.txt File4.txt new.txt

102

Example
$>awk /[Ff]ile/ {print $0} op1 file1.txt file3.txt now.txt File4.txt $>

$>awk /file/,/File/ {print $0} op1 File1.txt file3.txt Now.txt File4.txt

103

Example
$>cat pr.txt main 20 printf:pow sort 120 return:log max 40 swap:return $>awk {print $2, $1} pr.txt 20 main 120 sort 40 max $>
104

Example
$>cat pr.txt main 20 printf:pow sort 120 return:log max 40 swap:return

$>awk F: { print $1 } pr.txt main 20 printf sort 120 return max 40 swap

105

 BEGIN
- In awk begin and end are two keywords. BEGIN{actions} perform the actions before reading any record - BEGIN is mainly used to initialize some variables e.g. FS

END - Similarly END{actions} perform the actions after all record has been processed - END is normally used to calculate some end result e.g. calculating average
106

Example: BEGIN in awk

$>cat /etc/passwd root:x:0:0:SU:/:/bin/sh ritesh:x:200:200:U:/home/ritesh:/bin/bash ashu:x:201:200:U:/home/ashu:/bin/csh $>awk BEGIN{FS=: } { print $1} root ritesh Ashu $>
107

Example: END in awk

$>awk -F\: {print NR $1} END { printf \n There are %d users\n NR} 1. root 2. ritesh 3. ashu There are 3 users $>

108

if constructs in awk
awk flow construct if has following syntax
if ( condition1 )

statements else if ( condition2 ) statements else statements

109

Use of for, while in awk

Syntax for for loop is
for ( looinitializer; loop condition; loop modifier) { statements statements }

Syntax for while loop

while ( condition ) { statements statements }

110

ARRAYS
ASSOCIATIVE ARRAYS ARRAY_NAME[key]=value

Key alphanumeric type

Value-> alphanumeric type

To iterate through the keys for (keyName in ARRAY_NAME ) { print key "--" ARRAY_NAME [keyName] }

111

$>cat questions where are you? why are you here? Can you lend me 1000 Rs ? $>awk BEGIN { ARRAY["where are you?"]="NOIDA"; ARRAY["why are you here?"]="had nothing else to do }{ if(ARRAY[$0]=="") { value="sorry ! I didnt get the question } else { value=ARRAY[$0]; } print $0"\t\n\t"value }' questions where are you? NOIDA why are you here? had nothing else to do Can you lend me 1000 Rs ? sorry ! I didnt get the question

112

Example: for in awk

$>awk BEGIN { FS=: ;key=ST} { count=0 for ( j = 1; j<=NF; j++) { if ( $j == key ) count++ } print NR , count } textFile $>

113

Logical operators in awk

x == y x != y x>y x >= y x<y x <= y Is x equals to y ? Is x not equal to y ? Is x greater than y ? Is x greater or equal to y ? Is x less than y ? Is x less or equal to y ?

x~y
x !~ y && || !(x = y)

does x contain expression y ?

Is x not containing expr y ? ( x < y) && ( y < z) (x < y) || (y < z) Same as x != y
114

Sample of Built-in Functions

splt(str, arr, [sep])
- Splits the string str into fields using the separator
sep - Puts these field into the array arr - if sep is not mentioned default is to take FS

length(str)
- returns the length of the string
115

Want to get more info ?

Manual pages of Sh THE UNIX PROGRAMMING ENVIRONMENT by Kernighan and Pike Any Shell Script Book.

116