BABU BANARASI DAS UNIVERSITY

UNIX AND SHELL PROGRAMMING

ASSIGNMENT-3

SUBMITTED BY - SUBMITTED TO-

Srishti gupta Mam Nilu

MCA -2

IV semester

2190212054
Q1) State general structure of an awk and discuss about its
content.
AWK is abbreviated from the names of the developers – Aho, Weinberger, and
Kernighan.

AWK is a domain-specific language designed for text processing and typically used as

a data extraction and reporting tool in most Unix-like operating systems. The AWK
programming language requires no compiling, and allows the users to use variables,
numeric functions, string functions, and logical operators.
AWK is a utility that enables a programmer in the form of statements that define text
patterns that are to be searched for in each line of a document and the action that is to
be taken when a match is found within line.
Used for pattern scanning and processing.
Searches one or more files to see if they contain lines that matches with the specified
pattern and then performs the associated actions.

Contents of AWK-
The AWK language is a data-driven scripting language  consisting of a set of actions to
be taken against streams of textual data – either run directly on files or used as part of
a pipeline – for purposes of extracting or transforming text, such as producing formatted
reports. The language extensively uses the string datatype, associative arrays (that is,
arrays indexed by key strings), and regular expressions. While AWK has a limited
intended application domain and was especially designed to support one-liner
programs, the language is Turing-complete.

GENERAL STRUCTURE OF AN AWK-

AWK reads the input a line at a time. A line is scanned for each pattern in the program,
and for each pattern that matches, the associated action is executed.
An AWK program is a series of pattern action pairs, written as:
condition {action}
condition {action}
…..
where condition is typically an expression and action is a series of commands.

AWK is line oriented. That is, the pattern specifies a test that is performed with each line
read as input. If the condition is true, then the action is taken. The default pattern is
something that matches every line. This is the blank or null pattern. Two other important
patterns are specified by the keywords "BEGIN" and "END". These two words specify
actions to be taken before any lines are read, and after the last line is read. The AWK
program below:

BEGIN { print "START" }

{ print }
END { print "STOP" }

adds one line before and one line after the input file. A typical AWK program:

BEGIN { print "File\tOwner"}

{ print $8, "\t", $3}
END { print " - DONE -" }

The input is split into records, where by default records are separated by newline
characters so that the input is split into lines.
The program tests each record against each of the conditions in turn, and executes
the action for each expression that is true. Either the condition or the action may be
omitted. The condition defaults to matching every record. The default action is to print
the record. This is the same pattern-action structure as sed.
What can we do by using AWK?

1. AWK operations-
(a) Scans a line by line
(b) Splits each input line into fields
(c) Compares input line/fields to pattern
(d) Performs action(s) on matched lines
2. Useful for:
(a) Transform data files
(b) Produce formatted reports
3. Programming constructs:
(a) Format output lines
(b) Arithmetic and string operations
(c) Conditionals and loops

SYNTAX:

awk options ‘selection_criteria { action }’ input-file > output line

OPTIONS:

-f program-file : Reads the AWK program source from the file program-file, instead of
from the first command line argument.
-F fs : use for the input field separator.

BUILT IN VARIABLES IN AWK(COMMANDS)-

AWK’s built-in variables include the field variables- $1, $2, $3, that breaks a
line of text into individual words or pieces called fields.
 Number of fields variable (NF)
 Number of records variable (NR)
 Record separator variable (RS)
 Output record separator variable (ORS)
 Current filename variable (FILENAME)

Q2) How to debug a shell script?

When a script does not work properly, we need to determine the location of the
problem. The UNIX/Linux shells provide a debugging mode. Run the entire script in
debug mode or just a portion of the script. To run an entire script in debug mode, add -
x after the #!/bin/[shell] on the first line:

For Example :

#!/bin/sh –x

To run an entire script in debug mode from the command line, add a -x to the sh
command used to execute the script: $ sh -x script_name

Debug Statement Options-

Run several portions of a script in debug mode by placing the set -x option at the point
where debugging is to begin and the set +x option where you want it to stop. The debug
options are shown in Table below:

set –x : Prints the statements after interpreting meta characters and variables

set +x : Stops the printing of statements

set –v : Prints the statements before interpreting meta characters and variables

set –f : Disables file name generation (using meta-characters).

Example:

# cat script.sh
#!/bin/sh
set -x
echo "Your home is : $HOME"
set +x

Q3) What is a directory? Explain significance of Sticky bit permissions on a

directory.

Directory-

A directory is defined as an organizational unit, or container, used to organize folders

and files into a hierarchical structure. A directory contains bookkeeping information
about files that are, figuratively speaking, beneath them in the hierarchy.

Significance of Sticky bit permissions on a directory-

Suppose if we create a Linux directory that can be used by all the users of the Linux
system for creating files and users can create, delete or rename files according to their
convenience in this directory. There exists, for example, /tmp directory in the Linux
system that can be used by different Linux users to create temporary files. What will
happen If a user accidentally or deliberately deletes (or rename) a file created by some
other user in this directory. Well, to avoid these kind of issues, the concept of sticky bit
is used.

A Sticky bit is a permission bit that is set on a file or a directory that lets only the
owner of the file/directory or the root user to delete or rename the file. No other
user is given privileges to delete the file created by some other user.

1) First introduced in 1974 in the Unix operating system.

2) Earlier, it was introduced to minimize the time delay introduced every time when
a program is executed. When a program is executed, it takes time to load the
program into memory before the user can actually start using it. For this purpose,
to improve this time delay, the sticky bit was introduced. If some kind of patch
was applied to the executable as a bug fix or a new feature, a lot of overhead was
required, so it became obsolete later.

How to set and unset sticky bit-

Create a directory and provide all the users read-write-execute access to it:
#mkdir allAccess

#chmod 777 allAccess/

#ls –ld allAccess/

drwxrwxrwx 2 srishti srishti 4096 Oct 24 15:43 allAccess/

So we see that a directory named ‘allAccess’ is created and read-write-execute access to

this directory is given to all the users through chmod command.
Now, create multiple files in this directory (with different users) such that all users have
read-write-execute access to them.

For example:

# ls -l allAccess/

total 0

-rwxrwxrwx 1 srishti srishti 0 Oct 24 15:48 user1

-rwxrwxrwx 1 guest guest 0 Oct 24 16:11 user_file_0

-rwxrwxrwx 1 guest-2 guest-2 0 Oct 24 16:15 user_file_1

The files user_file_0 and user_file_1 are created by different users but have read-write-
execute access on for all the users. This means that the user ‘guest’ can delete or
rename the file created by user ‘guest-2’.In order to avoid this, sticky bit can be set on
the directory allAccess.

Now, turn ON the sticky bit on the directory by using +t flag of chmod command.

# chmod +t allAccess/

# ls -ld allAccess/
drwxrwxrwt 2 srishti srishti 4096 Oct 24 16:19 allAccess/

As can be observed, a permission bit ‘t’ is introduced in the permission bits of the
directory.

Now, if the user ‘guest’ tries to rename the file ‘user_file_1’, here is what happens :

$ mv /home/srishti/allAccess/user_file_1 /home/himanshu/allAccess/user_file_0

mv: cannot move `/home/srishti/allAccess/user_file_1' to

`/home/srishti/allAccess/user_file_0': Operation not permitted

So we see that the operation was not permitted.

Remove sticky bit using -t option

Sticky bit can be removed from a directory permissions through the -t option of the
chmod command.

Here is an example :

# chmod -t allAccess/

# ls -ld allAccess/

drwxrwxrwx 2 srishti srishti 4096 Oct 24 16:19 allAccess/

So we see that the permission bit ‘t’ is removed from directory.

Different OS behave differently with sticky bits as explained in this wikipedia article. For
example, Linux only looks for sticky bit if a user tries to rename a file. It will not check
the sticky bit if a file is being deleted.

Q4) Explain grep family utility with one example each.

grep command-
The grep filter searches a file for a particular pattern of characters, and displays all lines
that contain that pattern. The pattern that is searched in the file is referred to as the
regular expression (grep stands for globally search for regular expression and print out).

Syntax:

grep [options] pattern [files]

Options Description-

-c : This prints only a count of the lines that match a pattern

-h : Display the matched lines, but do not display the filenames.

-i : Ignores, case for matching

-l : Displays list of a filenames only.

-n : Display the matched lines and their line numbers.

-v : This prints out all the lines that do not matches the pattern

-e exp : Specifies expression with this option. Can use multiple times.

-f file : Takes patterns from file, one per line.

-E : Treats pattern as an extended regular expression (ERE)

-w : Match whole word

-o : Print only the matched parts of a matching line, with each such part on a separate output
line.

EXAMPLES-
$cat > geekfile.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
Unix linux which one you choose.
uNix is easy to learn.unix is a multiuser os.Learn unix .unix is a
powerful.

1. Case insensitive search : The -i option enables to search for a string case
insensitively in the give file. It matches the words like “UNIX”, “Unix”, “unix”.
$grep -i "UNix" geekfile.txt
Output:

unix is great os. unix is open source. unix is free os.

Unix linux which one you choose.
uNix is easy to learn.unix is a multiuser os.Learn unix. unix is a
powerful.

2. Displaying the count of number of matches : We can find the number of lines that
matches the given string/pattern.

$grep -c "unix" geekfile.txt

Output:
2

3. Display the file names that matches the pattern : We can just display the files that
contains the given string/pattern.
$grep -l "unix" *

or

$grep -l "unix" f1.txt f2.txt f3.xt f4.txt

Output:
geekfile.txt

4. Checking for the whole words in a file : By default, grep matches the given
string/pattern even if it found as a substring in a file. The -w option to grep makes it
match only the whole words.
$ grep -w "unix" geekfile.txt
Output:
unix is great os. unix is opensource. unix is free os.
uNix is easy to learn.unix is a multiuser os.Learn unix .unix is a
powerful.

5. Displaying only the matched pattern : By default, grep displays the entire line
which has the matched string. We can make the grep to display only the matched string
by using the -o option.

$ grep -o "unix" geekfile.txt

Output:

unix
unix
unix
unix
unix
unix

6. Show line number while displaying the output using grep -n : To show the line
number of file with the line matched.

$ grep -n "unix" geekfile.txt

Output:
1:unix is great os. unix is opensource. unix is free os.
4:uNix is easy to learn.unix is a multiuser os.Learn unix .unix is a
powerful.

7. Inverting the pattern match : You can display the lines that are not matched with
the specified search sting pattern using the -v option.

$ grep -v "unix" geekfile.txt

Output:
learn operating system.
Unix linux which one you choose.

8. Matching the lines that start with a string : The ^ regular expression pattern
specifies the start of a line. This can be used in grep to match the lines which start with
the given string or pattern.

$ grep "^unix" geekfile.txt

Output:
unix is great os. unix is opensource. unix is free os.

9. Matching the lines that end with a string : The $ regular expression pattern
specifies the end of a line. This can be used in grep to match the lines which end with
the given string or pattern.
$ grep "os$" geekfile.txt

10.Specifies expression with -e option. Can use multiple times :

$grep –e "Agarwal" –e "Aggarwal" –e "Agrawal" geekfile.txt
11. -f file option Takes patterns from file, one per line.
$cat pattern.txt
Agarwal
Aggarwal
Agrawal
$grep –f pattern.txt geekfile.txt

Q5) Discuss about various modes of vi editor.

Various modes of vi editor-

Modes of Operation in vi editor There are three modes in vi:

1) Command Mode
 When vi starts up, it is in Command Mode.
 This mode is where vi interprets any characters we type as commands and
thus does not display them in the window. This mode allows us to move
through a file, and to delete, copy, or paste a piece of text.

 To enter into Command Mode from any other mode, it requires pressing
the [Esc] key. If we press [Esc] when we are already in Command Mode,
then vi will beep or flash the screen.
 2) Insert mode: 
 This mode enables you to insert text into the file.
 Everything that’s typed in this mode is interpreted as input and finally, it is
put in the file.
 The vi always starts in command mode.
 To enter text, you must be in insert mode. To come in insert mode
you simply type i. To get out of insert mode, press the Esc key,
which will put you back into command mode.

3) Last Line Mode(Escape Mode): 

 Line Mode is invoked by typing a colon [:], while vi is in Command Mode.
 The cursor will jump to the last line of the screen and vi will wait for a
command.
 This mode enables you to perform tasks such as saving files, executing
commands.

Q6) What are the possible redirection operators? Give an example for each.

Redirection is a feature in Unix/Linux such that when executing a command, you can
change the standard input/output devices. The basic workflow of any Linux command is
that it takes an input and give an output.

 The standard input (stdin) device is the keyboard.

 The standard output (stdout) device is the screen.

With redirection, the above standard input/output can be changed.

It is one of the following symbols:

< > >| << >> <& >& <<- <>

These allow you to control the input and output of your commands. They can appear
anywhere within a simple command or may follow a command. Redirections are
processed in the order they appear, from left to right.

 < : Gives input to a command.

Example- command < file.txt
The above will execute command on the contents of file.txt.
 <> : same as above, but the file is open in read+write mode instead of read-only:
Example- command <> file.txt
If the file doesn't exist, it will be created.

 > : Directs the output of a command into a file.

Example- command > out.txt
The above will save the output of command as out.txt. If the file exists, its contents
will be overwritten and if it does not exist it will be created.
 This operator is also often used to choose whether something should be printed
to standard error or standard output:
Example- command >out.txt 2>error.txt
In the example above, > will redirect standard output and 2> redirects standard
error. Output can also be redirected using 1> but, since this is the default, the 1 is
usually omitted and it's written simply as >.
So, to run command on file.txt and save its output in out.txt and any error messages
in error.txt you would run:
Example- command < file.txt > out.txt 2> error.txt
 >| : Does the same as >, but will overwrite the target, even if the shell has been
configured to refuse overwriting (with set -C or set -o noclobber).
Example- command >| out.txt
If out.txt exists, the output of command will replace its content. If it does not exist it
will be created.
 >> : Does the same as >, except that if the target file exists, the new data are
appended.
Example- command >> out.txt
If out.txt exists, the output of command will be appended to it, after whatever is
already in it. If it does not exist it will be created.
 &>, >&, >>& and &>> : (non-standard). Redirect both standard error and
standard output, replacing or appending, respectively.
Example- command &> out.txt
Both standard error and standard output of command will be saved in out.txt,
overwriting its contents or creating it if it doesn't exist.
Example- command &>> out.txt
As above, except that if out.txt exists, the output and error of command will be
appended to it.

 << : A here document. It is often used to print multi-line strings.

Example- command << WORD
Text
WORD
Here, command will take everything until it finds the next occurrence of WORD, Text in the
example above, as input . If you want to pipe the output of command << WORD ...
WORD directly into another command or commands, you have to put the pipe on the same
line as << WORD, you can't put it after the terminating WORD or on the line following. For
example:
command << WORD | command2 | command3...
Text
WORD
 <<< : Here strings, similar to here documents, but intended for a single line.
These exist only in the Unix port or rc (where it originated), zsh, some
implementations of ksh, yash and bash.
Example- command <<< WORD
Whatever is given as WORD is expanded and its value is passed as input to command.
This is often used to pass the content of variables as input to a command. For example:
$ foo="bar"
$ sed 's/a/A/' <<< "$foo"
bAr
# as a short-cut for the standard:
$ printf '%s\n' "$foo" | sed 's/a/A/'
bAr
# or
sed 's/a/A/' << EOF
$foo
EOF

A few other operators (>&-, x>&y x<&y) can be used to close or duplicate file

descriptors.
Redirection operators-
1) pgm > file : Output of pgm is redirected to the file
2) pgm >> file : Output of pgm is appended to file
3) n > file : Output from stream with descriptor n redirected to file
4) n >> file : Output from stream WITH descriptor n appended to file
5) n >& m : Merges output from stream n with stream m
6) n <& m : Merges input from stream n with stream m
7) << tag : Standard input comes from here through next tag at the start of line
8) | : Takes output from one program, on process, and sends it to another.

Q7) Write the difference between sed and awk.

What is sed?
The command sed allows modifying and filtering text files.
EXAMPLES-
sed ‘1,3 d’ test1.txt
The above command will display the lines after line 3. It will not display 1, 2, 3 lines of
the test1.txt.

sed   ‘ s /abc / cab/’ test1.txt

The above command will substitute cab instead of abc in the test1.txt file.

sed  ‘s/one/1/’  test1.txt

The above command will substitute 1 instead of one in the test1.txt file.

sed ‘1,2 ! d’  test1.txt

This will only display line 1 and line 2.
What is awk?
The command awk allows searching data in a file and printing data on the console.  If a
file contains multiple columns, then it is also possible to find data in specific columns.
Furthermore, it helps tasks such as searching, conditional execution, updating and
filtering.
EXAMPLES-

awk  ‘{ print }’ file1.txt

The above command will display all the content in the file1.txt.  Assume that the file1.txt
file contains multiple columns.

awk   ‘{print $1}’ file1.txt

This command prints the first column of the file1.txt.  By default, the awk command
considers a white space as the separator.

awk  ‘{print $1 “  ”  $3}’ file1.txt

The above command prints the first and the third column of the file1.txt with a space in
between.

DIFFERENCE BETWEEN sed AND awk-

sed awk
The sed is a command line utility that The awk is a command line utility designed
parses and transforms text, using a for text processing that allows writing
simple, compact programming language.  effective programs in the form of
statements.

 The awk is more powerful and robust. Less powerful.

Similar to programming language.

Simple and limited. Complex and versatile.

Q8) Draw and explain the typical architecture of UNIX.

Unix has a graphical user interface similar to the Windows operating system that makes
it easy for navigation and a good supportive environment. The internal design view of
this operating system can be known from its architecture. The architecture of this
operating system is four layered. It consists of Hardware, Kernel, System Call
interface(shell) and application libraries/tools, utilities, etc…The kernel controls the
hardware of the computer and resides at the core of the architecture. System calls acts
as the interface between the kernel and other libraries. These libraries include general
functions and built on top of the system calls. Shell is a special application that provides
an interface to the other applications of the architecture.

Kernel

For this operating system, Kernel is the central core that interacts directly with the
hardware of the system. The main functions of Kernel are-
 Computer hardware such as memory, disc, printers, etc.. are controlled by the
kernel.
 The kernel schedules the processes, control and executes various user-defined
tasks.
 Manages the data storage and control the computer accesses by several users.
 The kernel is composed of several sub-components such as configurations
including boot code, device drivers to control hardware, header files.

Shell

It is the interface between the user and the kernel. Users can interact with the shell
using shell commands. Shell has two main responsibilities which include interpreting the
commands given by the users and execute them using the kernel, providing
programming ability to the users to write shell commands for a shell script to perform
specific tasks.

Commands

Some of the major categories of commands used by the Unix operating system are –
‘sh’ – shell commands providing a primary user interface, ‘utilities’ forming the core
toolkit of Unix commands includes sub-categories such as system utilities supporting
administrative tools and User utilities for environment management tools.