UNIX INTERVIEW QUESTIONS

UNIX INTERVIEW QUESTIONS ON AWK COMMAND

Awk is powerful tool in Unix. Awk is an excellent tool for processing the files which have data arranged in rows and columns format. It is a good filter and report writer.
1. How to run awk command specified in a file?
awk -f filename

2. Write a command to print the squares of numbers from 1 to 10 using awk command
awk 'BEGIN { for(i=1;i<=10;i++) {print "square of",i,"is",i*i;}}'

3. Write a command to find the sum of bytes (size of file) of all files in a directory.
ls -l | awk 'BEGIN {sum=0} {sum = sum + $5} END {print sum}'

4. In the text file, some lines are delimited by colon and some are delimited by space. Write a command to print the third field of each line.

awk '{ if( $0 ~ /:/ ) { FS=":"; } else { FS =" "; } print $3 }' filename

5. Write a command to print the line number before each line?
awk '{print NR, $0}' filename

6. Write a command to print the second and third line of a file without using NR.
awk 'BEGIN {RS="";FS="\n"} {print $2,$3}' filename

7. Write a command to print zero byte size files?
ls -l | awk '/^-/ {if ($5 !=0 ) print $9 }'

8. Write a command to rename the files in a directory with "_new" as postfix?
ls -F | awk '{print "mv "$1" "$1".new"}' | sh

9. Write a command to print the fields in a text file in reverse order?
awk 'BEGIN {ORS=""} { for(i=NF;i>0;i--) print $i," "; print "\n"}' filename

10. Write a command to find the total number of lines in a file without using NR
awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' filename

Another way to print the number of lines is by using the NR. The command is
awk 'END{print NR}' filename

UNIX INTERVIEW QUESTIONS ON GREP COMMAND

The grep is one of the powerful tools in unix. Grep stands for "global search for regular expressions and print". The power of grep lies in using regular expressions mostly.

The general syntax of grep command is
grep [options] pattern [files]

1. Write a command to print the lines that has the the pattern "july" in all the files in a particular directory?

grep july *
This will print all the lines in all files that contain the word “july” along with the file name. If any of the files contain words like "JULY" or "July", the above command would not print those lines.

2. Write a command to print the lines that has the word "july" in all the files in a directory and also suppress the filename in the output.

grep -h july *

3. Write a command to print the lines that has the word "july" while ignoring the case.

grep -i july *
The option i make the grep command to treat the pattern as case insensitive.

4. When you use a single file as input to the grep command to search for a pattern, it won't print the filename in the output. Now write a grep command to print the filename in the output without using the '-H' option.
grep pattern filename /dev/null
The /dev/null or null device is special file that discards the data written to it. So, the /dev/null is always an empty file.

Another way to print the filename is using the '-H' option. The grep command for this is
grep -H pattern filename

5. Write a Unix command to display the lines in a file that do not contain the word "july"?
grep -v july filename
The '-v' option tells the grep to print the lines that do not contain the specified pattern.

6. Write a command to print the file names in a directory that has the word "july"?
grep -l july *
The '-l' option make the grep command to print only the filename without printing the content of the file. As soon as the grep command finds the pattern in a file, it prints the pattern and stops searching other lines in the file.

7. Write a command to print the file names in a directory that does not contain the word "july"?
grep -L july *
The '-L' option makes the grep command to print the filenames that do not contain the specified pattern.

8. Write a command to print the line numbers along with the line that has the word "july"?
grep -n july filename
The '-n' option is used to print the line numbers in a file. The line numbers start from 1

9. Write a command to print the lines that starts with the word "start"?
grep '^start' filename
The '^' symbol specifies the grep command to search for the pattern at the start of the line.

10. Write a command to print the lines which end with the word "end"?
grep 'end$' filename
The '$' symbol specifies the grep command to search for the pattern at the end of the line.

11. Write a command to select only those lines containing "july" as a whole word?
grep -w july filename
The '-w' option makes the grep command to search for exact whole words. If the specified pattern is found in a string, then it is not considered as a whole word. For example: In the string "mikejulymak", the pattern "july" is found. However "july" is not a whole word in that string.

UNIX INTERVIEW QUESTIONS ON SED COMMAND

SED is a special editor used for modifying files automatically.

1. Write a command to replace the word "bad" with "good" in file?
sed s/bad/good/ < filename

2. Write a command to replace the word "bad" with "good" globally in a file?
sed s/bad/good/g < filename

3. Write a command to replace the character '/' with ',' in a file?
sed 's/\//,/' < filename
sed 's|/|,|' < filename

4. Write a command to replace the word "apple" with "(apple)" in a file?
sed s/apple/(&)/ < filename

5. Write a command to switch the two consecutive words "apple" and "mango" in a file?
sed 's/$apple$ $mango$/\2 \1/' < filename

6. Write a command to replace the second occurrence of the word "bat" with "ball" in a file?
sed 's/bat/ball/2' < filename

7. Write a command to remove all the occurrences of the word "jhon" except the first one in a line with in the entire file?
sed 's/jhon//2g' < filename

8. Write a command to remove the first number on line 5 in file?
sed '5 s/[0-9][0-9]*//' < filename

9. Write a command to remove the first number on all lines that start with "@"?
sed '\,^@, s/[0-9][0-9]*//' < filename

10. Write a command to replace the word "gum" with "drum" in the first 100 lines of a file?
sed '1,00 s/gum/drum/' < filename

11. write a command to replace the word "lite" with "light" from 100th line to last line in a file?
sed '100,$ s/lite/light/' < filename

12. Write a command to remove the first 10 lines from a file?
sed '1,10 d' < filename

13. Write a command to duplicate each line in a file?
sed 'p' < filename

14. Write a command to duplicate empty lines in a file?
sed '/^$/ p' < filename

15. Write a sed command to print the lines that do not contain the word "run"?
sed -n '/run/!p' < filename

UNIX INTERVIEW QUESTIONS ON CUT COMMAND

The cut command is used to used to display selected columns or fields from each line of a file. Cut command works in two modes:

Delimited selection: The fields in the line are delimited by a single character like blank,comma etc.
Range selection: Each field starts with certain fixed offset defined as range.

1. Write a command to display the third and fourth character from each line of a file?

cut -c 3,4 filename

2. Write a command to display the characters from 10 to 20 from each line of a file?

cut -c 10-20 filename

3. Write a command to display the first 10 characters from each line of a file?

cut -c -10 filename

4. Write a comamnd to display from the 10th character to the end of the line?

cut -c 10- filename

5. The fields in each line are delimited by comma. Write a command to display third field from each line of a file?

cut -d',' -f2 filename

6. Write a command to print the fields from 10 to 20 from each line of a file?

cut -d',' -f10-20 filename

7. Write a command to print the first 5 fields from each line?

cut -d',' -f-5 filename

8. Write a command to print the fields from 10th to the end of the line?

cut -d',' -f10- filename

9. By default the cut command displays the entire line if there is no delimiter in it. Which cut option is used to supress these kind of lines?

The -s option is used to supress the lines that do not contain the delimiter.

10. Write a cut command to extract the username from 'who am i' comamnd?

who am i | cut -f1 -d' '

UNIX INTERVIEW QUESTIONS ON FIND COMMAND

Find utility is used for searching files using the directory information.

1. Write a command to search for the file 'test' in the current directory?
find -name test -type f

2. Write a command to search for the file 'temp' in '/usr' directory?
find /usr -name temp -type f

3. Write a command to search for zero byte size files in the current directory?
find -size 0 -type f

4. Write a command to list the files that are accessed 5 days ago in the current directory?
find -atime 5 -type f

5. Write a command to list the files that were modified 5 days ago in the current directory?
find -mtime 5 -type f

6. Write a command to search for the files in the current directory which are not owned by any user in the /etc/passwd file?
find . -nouser -type f

7. Write a command to search for the files in '/usr' directory that start with 'te'?
find /usr -name 'te*' -type f

8. Write a command to search for the files that start with 'te' in the current directory and then display the contents of the file?
find . -name 'te*' -type f -exec cat {} \;

9. Write a command to list the files whose status is changed 5 days ago in the current directory?
find -ctime 5 -type f

10. Write a command to list the files in '/usr' directory that start with 'ch' and then display the number of lines in each file?
find /usr -name 'ch*' -type f -exec wc -l {} \;

TOP EXAMPLES OF AWK COMMAND IN UNIX

Awk is one of the most powerful tools in Unix used for processing the rows and columns in a file. Awk has built in string functions and associative arrays. Awk supports most of the operators, conditional blocks, and loops available in C language.

One of the good things is that you can convert Awk scripts into Perl scripts using a2p utility.

The basic syntax of AWK:

awk 'BEGIN {start_action} {action} END {stop_action}' filename

Here the actions in the begin block are performed before processing the file and the actions in the end block are performed after processing the file. The rest of the actions are performed while processing the file.

Examples:

Create a file input_file with the following data. This file can be easily created using the output of ls -l.

-rw-r--r-- 1 center center  0 Dec  8 21:39 p1
-rw-r--r-- 1 center center 17 Dec  8 21:15 t1
-rw-r--r-- 1 center center 26 Dec  8 21:38 t2
-rw-r--r-- 1 center center 25 Dec  8 21:38 t3
-rw-r--r-- 1 center center 43 Dec  8 21:39 t4
-rw-r--r-- 1 center center 48 Dec  8 21:39 t5

From the data, you can observe that this file has rows and columns. The rows are separated by a new line character and the columns are separated by a space characters. We will use this file as the input for the examples discussed here.

1. awk '{print $1}' input_file

Here $1 has a meaning. $1, $2, $3... represents the first, second, third columns... in a row respectively. This awk command will print the first column in each row as shown below.

-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--

To print the 4th and 6th columns in a file use awk '{print $4,$5}' input_file

Here the Begin and End blocks are not used in awk. So, the print command will be executed for each row it reads from the file. In the next example we will see how to use the Begin and End blocks.

2. awk 'BEGIN {sum=0} {sum=sum+$5} END {print sum}' input_file

This will prints the sum of the value in the 5th column. In the Begin block the variable sum is assigned with value 0. In the next block the value of 5th column is added to the sum variable. This addition of the 5th column to the sum variable repeats for every row it processed. When all the rows are processed the sum variable will hold the sum of the values in the 5th column. This value is printed in the End block.

3. In this example we will see how to execute the awk script written in a file. Create a file sum_column and paste the below script in that file

#!/usr/bin/awk -f
BEGIN {sum=0} 
{sum=sum+$5} 
END {print sum}

Now execute the the script using awk command as

awk -f sum_column input_file.

This will run the script in sum_column file and displays the sum of the 5th column in the input_file.

4. awk '{ if($9 == "t4") print $0;}' input_file

This awk command checks for the string "t4" in the 9th column and if it finds a match then it will print the entire line. The output of this awk command is

-rw-r--r-- 1 pcenter pcenter 43 Dec  8 21:39 t4

5. awk 'BEGIN { for(i=1;i<=5;i++) print "square of", i, "is",i*i; }'

This will print the squares of first numbers from 1 to 5. The output of the command is

square of 1 is 1
square of 2 is 4
square of 3 is 9
square of 4 is 16
square of 5 is 25

Notice that the syntax of “if” and “for” are similar to the C language.

Awk Built in Variables:

You have already seen $0, $1, $2... which prints the entire line, first column, second column... respectively. Now we will see other built in variables with examples.

FS - Input field separator variable:

So far, we have seen the fields separted by a space character. By default Awk assumes that fields in a file are separted by space characters. If the fields in the file are separted by any other character, we can use the FS variable to tell about the delimiter.

6. awk 'BEGIN {FS=":"} {print $2}' input_file
OR
awk -F: '{print $2} input_file

This will print the result as

OFS - Output field separator variable:

By default whenever we printed the fields using the print statement the fields are displayed with space character as delimiter. For example

7. awk '{print $4,$5}' input_file

The output of this command will be

center 0
center 17
center 26
center 25
center 43
center 48

We can change this default behavior using the OFS variable as

awk 'BEGIN {OFS=":"} {print $4,$5}' input_file

center:0
center:17
center:26
center:25
center:43
center:48

Note: print $4,$5 and print $4$5 will not work the same way. The first one displays the output with space as delimiter. The second one displays the output without any delimiter.

NF - Number of fileds variable:

The NF can be used to know the number of fields in line

8. awk '{print NF}' input_file
This will display the number of columns in each row.

NR - number of records variable:
The NR can be used to know the line number or count of lines in a file.

9. awk '{print NR}' input_file
This will display the line numbers from 1.

10. awk 'END {print NR}' input_file
This will display the total number of lines in the file.

String functions in Awk:
Some of the string functions in awk are:

index(string,search)
length(string)
split(string,array,separator)
substr(string,position)
substr(string,position,max)
tolower(string)
toupper(string)

Advanced Examples:

1. Filtering lines using Awk split function

The awk split function splits a string into an array using the delimiter.

The syntax of split function is
split(string, array, delimiter)

Now we will see how to filter the lines using the split function with an example.

The input "file.txt" contains the data in the following format

1 U,N,UNIX,000
2 N,P,SHELL,111
3 I,M,UNIX,222
4 X,Y,BASH,333
5 P,R,SCRIPT,444

Required output: Now we have to print only the lines in which whose 2nd field has the string "UNIX" as the 3rd field( The 2nd filed in the line is separated by comma delimiter ).
The ouptut is:

1 U,N,UNIX,000
3 I,M,UNIX,222

The awk command for getting the output is:

awk '{ 
        split($2,arr,","); 
        if(arr[3] == "UNIX") 
        print $0 
} ' file.txt

EXAMPLES OF AWK COMMAND IN UNIX - PART 2

1. Inserting a new line after every 2 lines

We will see how to implement this using the awk command with an example.
The input "file.txt" contains the below data:

1 A
2 B
3 C
4 D
5 E
6 F

Let say, we want to insert the new line "9 Z" after every two lines in the input file. The required output data after inserting a new line looks as

1 A
2 B
9 Z
3 C
4 D
9 Z
5 E
6 F
9 Z

The awk command for getting this output is

awk '{ 
 if(NR%2 == 0) 
 {
  print $0"\n9 Z";
 } 
 else 
 {
  print $0
 }
     }' file.txt

2. Replace the Nth occurrence of a pattern

The input file contains the data.

AAA 1
BBB 2
CCC 3
AAA 4
AAA 5
BBB 6
CCC 7
AAA 8
BBB 9
AAA 0

Now we want to replace the fourth occurrence of the first filed "AAA" with "ZZZ" in the file.
The required output is:

AAA 1
BBB 2
CCC 3
AAA 4
AAA 5
BBB 6
CCC 7
ZZZ 8
BBB 9
AAA 0

The awk command for getting this output is

awk 'BEGIN {count=0} 
 { 
  if($1 == "AAA") 
  { 
   count++
  } 
  if(count == 4) 
  { 
   sub("AAA","ZZZ",$1) 
  } 
 } 
 {
  print $0
 }' file.txt

3. Find the sum of even and odd lines separately
The input file data:

A 10
B 39
C 22
D 44
E 75
F 89
G 67

You have to get the second field and then find the sum the even and odd lines.
The required output is

174, 172

The awk command for producing this output is

awk '{ 
 if(NR%2 == 1) 
 {
  sum_e = sum_e + $2
 } 
 else 
 {
  sum_o = sum_o + $2
 }
      } 
      END { print sum_e,sum_o }' file.txt

4. Fibonacci series using awk command

Now we will produce the Fibonacci series using the awk command.

awk ' BEGIN{ 
 for(i=0;i<=10;i++) 
 { 
  if (i <=1 )
  {
   x=0;
   y=1;
   print i;
  }
  else
  {
   z=x+y;
   print z; 
   x=y;
   y=z;
  }
 } 
    }'

The output is

5. Remove leading zeros from a file using the awk command. The input file contains the below data.

After removing the leading zeros, the output should contain the below data.

The awk command for this is.

awk '{print $1 + 0}' file.txt
awk '{printf "%d\n",$0}' file.txt

SED COMMAND IN UNIX AND LINUX EXAMPLES

Sed is a Stream Editor used for modifying the files in unix (or linux). Whenever you want to make changes to the file automatically, sed comes in handy to do this. Most people never learn its power; they just simply use sed to replace text. You can do many things apart from replacing text with sed. Here I will describe the features of sed with examples.

Consider the below text file as an input.

>cat file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.

Sed Command Examples

1. Replacing or substituting string

Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.

>sed 's/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

Here the "s" specifies the substitution operation. The "/" are delimiters. The "unix" is the search pattern and the "linux" is the replacement string.

By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line.

2. Replacing the nth occurrence of a pattern in a line.

Use the /1, /2 etc flags to replace the first, second occurrence of a pattern in a line. The below command replaces the second occurrence of the word "unix" with "linux" in a line.

>sed 's/unix/linux/2' file.txt
unix is great os. linux is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.

3. Replacing all the occurrence of the pattern in a line.

The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.

>sed 's/unix/linux/g' file.txt
linux is great os. linux is opensource. linux is free os.
learn operating system.
linuxlinux which one you choose.

4. Replacing from nth occurrence to all occurrences in a line.

Use the combination of /1, /2 etc and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... "unix" word with "linux" word in a line.

>sed 's/unix/linux/3g' file.txt
unix is great os. unix is opensource. linux is free os.
learn operating system.
unixlinux which one you choose.

5. Changing the slash (/) delimiter

You can use any delimiter other than the slash. As an example if you want to change the web url to another url as

>sed 's/http:\/\//www/' file.txt

In this case the url consists the delimiter character which we used. In that case you have to escape the slash with backslash character, otherwise the substitution won't work.

Using too many backslashes makes the sed command look awkward. In this case we can change the delimiter to another character as shown in the below example.

>sed 's_http://_www_' file.txt
>sed 's|http://|www|' file.txt

6. Using & as the matched string

There might be cases where you want to search for the pattern and replace that pattern by adding some extra characters to it. In such cases & comes in handy. The & represents the matched string.

>sed 's/unix/{&}/' file.txt
{unix} is great os. unix is opensource. unix is free os.
learn operating system.
{unix}linux which one you choose.

>sed 's/unix/{&&}/' file.txt
{unixunix} is great os. unix is opensource. unix is free os.
learn operating system.
{unixunix}linux which one you choose.

7. Using \1,\2 and so on to \9

The first pair of parenthesis specified in the pattern represents the \1, the second represents the \2 and so on. The \1,\2 can be used in the replacement string to make changes to the source string. As an example, if you want to replace the word "unix" in a line with twice as the word like "unixunix" use the sed command as below.

>sed 's/\(unix\)/\1\1/' file.txt
unixunix is great os. unix is opensource. unix is free os.
learn operating system.
unixunixlinux which one you choose.

The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is

>sed 's/\(unix\)\(linux\)/\2\1/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxunix which one you choose.

Another example is switching the first three characters in a line

>sed 's/^\(.\)\(.\)\(.\)/\3\2\1/' file.txt
inux is great os. unix is opensource. unix is free os.
aelrn operating system.
inuxlinux which one you choose.

8. Duplicating the replaced line with /p flag

The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.

>sed 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
linuxlinux which one you choose.

9. Printing only the replaced lines

Use the -n option along with the /p print flag to display only the replaced lines. Here the -n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.

>sed -n 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linuxlinux which one you choose.

If you use -n alone without /p, then the sed does not print anything.

10. Running multiple sed commands.

You can run multiple sed commands by piping the output of one sed command as input to another sed command.

>sed 's/unix/linux/' file.txt| sed 's/os/system/'
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.

Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.

>sed -e 's/unix/linux/' -e 's/os/system/' file.txt
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.

11. Replacing string on a specific line number.

You can restrict the sed command to replace the string on a specific line number. An example is

>sed '3 s/unix/linux/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

The above sed command replaces the string only on the third line.

12. Replacing string on a range of lines.

You can specify a range of line numbers to the sed command for replacing a string.

>sed '1,3 s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

Here the sed command replaces the lines with range from 1 to 3. Another example is

>sed '2,$ s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

Here $ indicates the last line in the file. So the sed command replaces the text from second line to last line in the file.

13. Replace on a lines which matches a pattern.

You can specify a pattern to the sed command to match in a line. If the pattern match occurs, then only the sed command looks for the string to be replaced and if it finds, then the sed command replaces the string.

>sed '/linux/ s/unix/centos/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
centoslinux which one you choose.

Here the sed command first looks for the lines which has the pattern "linux" and then replaces the word "unix" with "centos".

14. Deleting lines.

You can delete the lines a file by specifying the line number or a range or numbers.

>sed '2 d' file.txt
>sed '5,$ d' file.txt

15. Duplicating lines

You can make the sed command to print each line of a file two times.

>sed 'p' file.txt

16. Sed as grep command

You can make sed command to work as similar to grep command.

>grep 'unix' file.txt
>sed -n '/unix/ p' file.txt

Here the sed command looks for the pattern "unix" in each line of a file and prints those lines that has the pattern.

You can also make the sed command to work as grep -v, just by using the reversing the sed with NOT (!).

>grep -v 'unix' file.txt
>sed -n '/unix/ !p' file.txt

The ! here inverts the pattern match.

17. Add a line after a match.

The sed command can add a new line after a pattern match is found. The "a" command to sed tells it to add a new line after a match is found.

>sed '/unix/ a "Add a new line"' file.txt
unix is great os. unix is opensource. unix is free os.
"Add a new line"
learn operating system.
unixlinux which one you choose.
"Add a new line"

18. Add a line before a match

The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.

>sed '/unix/ i "Add a new line"' file.txt
"Add a new line"
unix is great os. unix is opensource. unix is free os.
learn operating system.
"Add a new line"
unixlinux which one you choose.

19. Change a line

The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.

>sed '/unix/ c "Change line"' file.txt
"Change line"
learn operating system.
"Change line"

20. Transform like tr command

The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.

>sed 'y/ul/UL/' file.txt
Unix is great os. Unix is opensoUrce. Unix is free os.
Learn operating system.
UnixLinUx which one yoU choose.

Here the sed command transforms the alphabets "ul" into their uppercase format "UL"

GREP COMMAND IN UNIX AND LINUX EXAMPLES

Grep is the frequently used command in Unix (or Linux). Most of us use grep just for finding the words in a file. The power of grep comes with using its options and regular expressions. You can analyze large sets of log files with the help of grep command.

Grep stands for Global search for Regular Expressions and Print.

The basic syntax of grep command is

grep [options] pattern [list of files]

Let see some practical examples on grep command.

1. Running the last executed grep command

This saves a lot of time if you are executing the same command again and again.

!grep

This displays the last executed grep command and also prints the result set of the command on the terminal.

2. Search for a string in a file

This is the basic usage of grep command. It searches for the given string in the specified file.

grep "Error" logfile.txt

This searches for the string "Error" in the log file and prints all the lines that has the word "Error".

3. Searching for a string in multiple files.

grep "string" file1 file2
grep "string" file_pattern

This is also the basic usage of the grep command. You can manually specify the list of files you want to search or you can specify a file pattern (use regular expressions) to search for.

4. Case insensitive search

The -i option enables to search for a string case insensitively in the give file. It matches the words like "UNIX", "Unix", "unix".

grep -i "UNix" file.txt

5. Specifying the search string as a regular expression pattern.

grep "^[0-9].*" file.txt

This will search for the lines which starts with a number. Regular expressions is huge topic and I am not covering it here. This example is just for providing the usage of regular expressions.

6. Checking for the whole words in a file.

By default, grep matches the given string/pattern even if it found as a substring in a file. The -w option to grep makes it match only the whole words.

grep -w "world" file.txt

7. Displaying the lines before the match.

Some times, if you are searching for an error in a log file; it is always good to know the lines around the error lines to know the cause of the error.

grep -B 2 "Error" file.txt

This will prints the matched lines along with the two lines before the matched lines.

8. Displaying the lines after the match.

grep -A 3 "Error" file.txt

This will display the matched lines along with the three lines after the matched lines.

9. Displaying the lines around the match

grep -C 5 "Error" file.txt

This will display the matched lines and also five lines before and after the matched lines.

10. Searching for a sting in all files recursively

You can search for a string in all the files under the current directory and sub-directories with the help -r option.

grep -r "string" *

11. Inverting the pattern match

You can display the lines that are not matched with the specified search sting pattern using the -v option.

grep -v "string" file.txt

12. Displaying the non-empty lines

You can remove the blank lines using the grep command.

grep -v "^$" file.txt

13. Displaying the count of number of matches.

We can find the number of lines that matches the given string/pattern

grep -c "sting" file.txt

14. Display the file names that matches the pattern.

We can just display the files that contains the given string/pattern.

grep -l "string" file.txt

15. Display the file names that do not contain the pattern.

We can display the files which do not contain the matched string/pattern.

grep -l "string" file.txt

16. Displaying only the matched pattern.

By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.

grep -o "string" file.txt

17. Displaying the line numbers.

We can make the grep command to display the position of the line which contains the matched string in a file using the -n option

grep -n "string" file.txt

18. Displaying the position of the matched string in the line

The -b option allows the grep command to display the character position of the matched string in a file.

grep -o -b "string" file.txt

19. Matching the lines that start with a string

The ^ regular expression pattern specifies the start of a line. This can be used in grep to match the lines which start with the given string or pattern.

grep "^start" file.txt

20. Matching the lines that end with a string

The $ regular expression pattern specifies the end of a line. This can be used in grep to match the lines which end with the given string or pattern.

grep "end$" file.txt

FIND COMMAND IN UNIX AND LINUX EXAMPLES

Find is one of the powerful utility of Unix (or Linux) used for searching the files in a directory hierarchy. The syntax of find command is

find [pathnames] [conditions]

Let see some practical exercises on using find command.

1. How to run the last executed find command?

!find

This will execute the last find command. It also displays the last find command executed along with the result on the terminal.

2. How to find for a file using name?

find -name "sum.java"
./bkp/sum.java
./sum.java

This will find all the files with name "sum.java" in the current directory and sub-directories.

3. How to find for files using name and ignoring case?

find -iname "sum.java"
./SUM.java
./bkp/sum.java
./sum.java

This will find all the files with name "sum.java" while ignoring the case in the current directory and sub-directories.

4. How to find for a file in the current directory only?

find -maxdepth 1 -name "sum.java"
./sum.java

This will find for the file "sum.java" in the current directory only

5. How to find for files containing a specific word in its name?

find -name "*java*"
./SUM.java
./bkp/sum.java
./sum.java
./multiply.java

It displayed all the files which have the word "java" in the filename

6. How to find for files in a specific directory?

find /etc -name "*java*"

This will look for the files in the /etc directory with "java" in the filename

7. How to find the files whose name are not "sum.java"?

find -not -name "sum.java"
.
./SUM.java
./bkp
./multiply.java

This is like inverting the match. It prints all the files except the given file "sum.java".

8. How to limit the file searches to specific directories?

find -name "sum.java"
./tmp/sum.java
./bkp/var/tmp/files/sum.java
./bkp/var/tmp/sum.java
./bkp/var/sum.java
./bkp/sum.java
./sum.java

You can see here the find command displayed all the files with name "sum.java" in the current directory and sub-directories.

a. How to print the files in the current directory and one level down to the current directory?

find -maxdepth 2 -name "sum.java"
./tmp/sum.java
./bkp/sum.java
./sum.java

b. How to print the files in the current directory and two levels down to the current directory?

find -maxdepth 3 -name "sum.java"
./tmp/sum.java
./bkp/var/sum.java
./bkp/sum.java
./sum.java

c. How to print the files in the subdirectories between level 1 and 4?

find -mindepth 2 -maxdepth 5 -name "sum.java"
./tmp/sum.java
./bkp/var/tmp/files/sum.java
./bkp/var/tmp/sum.java
./bkp/var/sum.java
./bkp/sum.java

9. How to find the empty files in a directory?

find . -maxdepth 1 -empty
./empty_file

10. How to find the largest file in the current directory and sub directories

find . -type f -exec ls -s {} \; | sort -n -r | head -1

The find command "find . -type f -exec ls -s {} \;" will list all the files along with the size of the file. Then the sort command will sort the files based on the size. The head command will pick only the first line from the output of sort.

11. How to find the smallest file in the current directory and sub directories

find . -type f -exec ls -s {} \; | sort -n -r | tail -1

Another method using find is

find . -type f -exec ls -s {} \; | sort -n  | head -1

12. How to find files based on the file type?

a. Finding socket files

find . -type s

b. Finding directories

find . -type d

c. Finding hidden directories

find -type d -name ".*"

d. Finding regular files

find . -type f

e. Finding hidden files

find . -type f -name ".*"

13. How to find files based on the size?

a. Finding files whose size is exactly 10M

find . -size 10M

b. Finding files larger than 10M size

find . -size +10M

c. Finding files smaller than 10M size

find . -size -10M

14. How to find the files which are modified after the modification of a give file.

find -newer "sum.java"

This will display all the files which are modified after the file "sum.java"

15. Display the files which are accessed after the modification of a give file.

find -anewer "sum.java"

16. Display the files which are changed after the modification of a give file.

find -cnewer "sum.java"

17. How to find the files based on the file permissions?

find . -perm 777

This will display the files which have read, write, and execute permissions. To know the permissions of files and directories use the command "ls -l".

18. Find the files which are modified within 30 minutes.

find . -mmin -30

19. Find the files which are modified within 1 day.

find . -mtime -1

20. How to find the files which are modified 30 minutes back

find . -not -mmin -30

21. How to find the files which are modified 1 day back.

find . -not -mtime -1

22. Print the files which are accessed within 1 hour.

find . -amin -60

23. Print the files which are accessed within 1 day.

find . -atime -1

24. Display the files which are changed within 2 hours.

find . -cmin -120

25. Display the files which are changed within 2 days.

find . -ctime -2

26. How to find the files which are created between two files.

find . -cnewer f1 -and ! -cnewer f2

So far we have just find the files and displayed on the terminal. Now we will see how to perform some operations on the files.

1. How to find the permissions of the files which contain the name "java"?

find -name "*java*"|xargs ls -l

Alternate method is

find -name "*java*" -exec ls -l {} \;

2. Find the files which have the name "java" in it and then display only the files which have "class" word in them?

find -name "*java*" -exec grep -H class {} \;

3. How to remove files which contain the name "java".

find -name "*java*" -exec rm -r {} \;

This will delete all the files which have the word “java" in the file name in the current directory and sub-directories.

REDIRECT OUTPUT TO MULTIPLE FILES

The tee command in unix writes the output to multiple files and also displays the output on terminal.

Example:
date | tee -a file1 file2 file3

For more details look at "man tee"

CONVERT LOWER CASE TO UPPER CASE

'tr' command will convert one set of characters to another set. The following command converts lower case alphabets in to upper case.

echo "apple" | tr [a-z] [A-Z]

Similarly to convert from upper case to lower case, use the following command

echo "APPLE" | tr [A-Z] [a-z]

For more details on tr look at "man tr".

CONVERT STRING TO INITCAP IN UNIX

The following unix command converts the first letter in a string to upper case and the remaining letters to lower case.

echo apple | awk '{print toupper(substr($1,1,1)) tolower(substr($1,2))}'

WHAT IS A UNIX OPERATING SYSTEM

Unix is a multi-tasking, multi-user operating system. It is a layer between the hardware and the applications that run on the computer. It has functions which manage the hardware and applications.

The structure of Unix operating system can be divided into three parts.

Kernel is the core part of Unix which interacts with the hardware for low level functions.
Shell is the outer unit of Unix which interacts with the user to perform the functions.
File System.

UNIX CORE UNIT - KERNEL

The kernel is the heart of a UNIX system and manages the hardware, executing processes etc. When the computer is booted, kernel is loaded into the computer's main memory and it remains there until the computer is shut down. The kernel performs many low-level and system-level functions.

The tasks of kernel include

Interpreting and sending basic instructions to the computer's processor.
Running and scheduling the processes.
Allocating the necessary hardware.
Controlling the I/O operations.

UNIX OUTER UNIT - SHELL

The instructions to the kernel are complex and highly technical. To protect the kernel from the short comings of user, a shell is built around the kernel. The Shell acts like a mediator between the user and the kernel. Whenever a user run a command, the shell interprets the command and passes the command to the kernel.

Three types of shell are standard in Unix

Bourne shell is developed by Stephen Bourne. It is the most widely used shell and is a program with name sh. The bourne shell prompts with $ symbol
Korn shell is developed by David Korn. The korn shell has additional features than bourne shell and is called by the name ksh.
C shell is developed by Bill Joy and is called by the name csh.

FUNCTIONS OF UNIX SHELL

Some of the basic functions of shell are:

Command line interpretation
Program initiation
Input-output redirection
Pipeline connection
Substitution of filenames
Maintenance of variables
Environment control
Shell programming

UNIX USER LOGIN PROGRAMS - GETTY AND LOGIN

The Kernel should know who the user is logging in and how to communicate with the user. To do this the kernel invokes two programs, getty and login.

The kernel invokes the getty program for every user terminal. When the getty program receives input from the user, it invokes the login program. The login program verifies the identity of the user by checking the password file. If the user fails to provide valid password, the login program returns the control back to the getty program. If the user enters a valid password, the login program takes the user to the shell prompt.

UNIX FILE SYSTEM

The strength of the Unix lies in treating the files in a consistent way. For Unix a file is a file. This consistency makes it easy to work with files and the user does not have to learn special commands for new tasks. The user can write Unix programs easily without worrying about whether he’s communicating to a terminal, a printer, or an ordinary file on a disk drive.

For example a "cat" command can be used to display the contents of a file on terminal screen and can also send the file to a printer. As far as Unix is concerned the terminal and the printer are files just as other files.

DIFFERENT TYPES OF UNIX FILES

There are mainly three types of Unix files. They are

Regular files
Directories
Special or Device files

Regular Files

Regular files hold data and executable programs. Executable programs are the commands (ls) that you enter on the prompt. The data can be anything and there is no specific format enforced in the way the data is stored.
The regular files can be visualized as the leaves in the UNIX tree.

Directories

Directories are files that contain other files and sub-directories. Directories are used to organize the data by keeping closely related files in the same place. The directories are just like the folders in windows operating system.
The kernel alone can write the directory file. When a file is added to or deleted from this directory, the kernel makes an entry.
A directory file can be visualized as the branch of the UNIX tree.

Special Or Device Files

These files represent the physical devices. Files can also refer to computer hardware such as terminals and printers. These device files can also refer to tape and disk drives, CD-ROM players, modems, network interfaces, scanners, and any other piece of computer hardware. When a process writes to a special file, the data is sent to the physical device associated with it. Special files are not literally files, but are pointers that point to the device drivers located in the kernel. The protection applicable to files is also applicable to physical devices.

UNIX FILE STRUCTURE (FILE TREE)

The Unix file structure is organized in a reverse tree structure manner. The following figure shows a typical organization of files in Unix system.

The diagram looks like any upside-down tree. The slash (/) indicates the root directory. Names like etc, usr, local are directories and science.txt is a file. The regular files in Unix are the leaves in a tree structure.

SWAPPING AND PAGING IN UNIX

Swapping

The whole process in swapping is moved from the swap device to the main memory for execution. The process size must be less than or equal to the available main memory. It is easier to implementation and overhead to the system. Swapping systems does not handle the memory more flexibly as compared to the paging systems.

Paging

Only the required memory pages are moved to main memory from the swap device for execution. The process size does not matter. Paging gives the concept of the virtual memory. It provides greater flexibility in mapping the virtual address space into the physical memory of the machine. It allows more number of processes to fit in the main memory simultaneously and allows the greater process size than the available physical memory. Demand paging systems handle the memory more flexibly.

===========================================================

FIND LENGTH OF STRING IN UNIX / LINUX BASH SCRIPT

Q) How to find the length of the given string in unix or linux operating system?

Length of the string is the number of characters in the strings content. There are many different ways to compute the length of the string. Here we will see some of the ways to find the length of the string using the shell script.

> cat length.sh
#!/bin/bash

str="Linux dedicated server"

#1. finding size of string using the wc command
len=`echo $str|wc -c`
echo "length of $str=$len"

#2.caluclating the length using the awk command
len=`echo $str|awk '{print length}'`

#3.length using the expr command
len=`expr length "$str"`
echo "length of $str=$len"

#4.length using the shell built in hash(#)
echo "length of $str=$len"
echo "length of $str=${#str}"

GREP / PRINT LINES BETWEEN TWO MATCHING PATTERNS/STRINGS - UNIX/LINUX

Q) How to print the lines between two matching strings using the linux or unix commands?

Create the following sample file in the unix operating system:

> cat sample.dat
unix operating system
linux dedicated server
debian virtual server
fedora system
windows gui system

Let say, we want to grep the lines between the two matching strings linux and fedora. After grepping, the output should be:

linux dedicated server
debian virtual server
fedora system

Solution:

1. Using the sed command.

To the sed command, we can specify the starting pattern and ending pattern to print the lines. The syantax and the example is shown below:

Syntax:
sed -n '/start-pattern/,/end-pattern/p' filename

Example:
> sed -n '/linux/,/fedora/p' sample.dat

2. Using the awk command

Similar to the sed command, you can specify the starting strind and ending string in the awk command. The syntax and the example is shown eblow:

Syntax:
awk '/start-string/,/end-string/' filename

Example:
> awk '/unix/,/debian/' sample.dat

HOW TO GET HOSTNAME FROM IP ADDRESS - UNIX /LINUX

The host command in unix or linux operating system can be used to get the hostname from an IP address. An example is shown below:

> host 4.59.125.171
171.125.59.4.in-addr.arpa is an alias for 171.128-254.125.59.4.in-addr.arpa.
171.128-254.125.59.4.in-addr.arpa domain name pointer 4-59-125-171-hou.servercraft.co.

The host command gets the hostname listed in the /etc/resolv.conf file by querying the nameserver.

Another way to get the hostname is to user the nslookup command.

> nslookup 4.59.125.171
Server:         192.168.172.100
Address:        192.168.172.100#53

Non-authoritative answer:
171.125.59.4.in-addr.arpa       canonical name = 171.128-254.125.59.4.in-addr.arpa.
171.128-254.125.59.4.in-addr.arpa       name = 4-59-125-171-hou.servercraft.co.

Authoritative answers can be found from:
128-254.125.59.4.in-addr.arpa   nameserver = ns1.p23.dynect.net.
128-254.125.59.4.in-addr.arpa   nameserver = ns5.p23.dynect.net.
128-254.125.59.4.in-addr.arpa   nameserver = ns3.p23.dynect.net.
128-254.125.59.4.in-addr.arpa   nameserver = ns2.p23.dynect.net.
128-254.125.59.4.in-addr.arpa   nameserver = ns4.p23.dynect.net.
ns4.p23.dynect.net      internet address = 204.13.251.23
ns3.p23.dynect.net      internet address = 208.78.71.23
ns1.p23.dynect.net      internet address = 208.78.70.23
ns2.p23.dynect.net      internet address = 204.13.250.23

Here the hostname is highlighted in the green color. The alias name for the hostname is highlighted in the red color. If you are connected the remote host, you can get the hostname of the remote machine by using the arp command. It will list all the hostnames with the IP address. Another way is to simply type the hostname command on the remote server to know its host name.

EXECUTE MYSQL COMMAND IN BASH / SHELL SCRIPT

Q) How to connect to mysql database from a bash script in unix or linux and run sql queries?

Bash scripting helps in automating things. We can automate running sql queries by connecting to the mysql database through a shell script in unix or linux system.

Here we will see how to run a small sql query in mysql database through a script. The bash script code is shown below:

#!/usr/bin/bash

#Script to run automated sql queries

#Declaring mysql DB connection 

MASTER_DB_USER='username'
MASTER_DB_PASSWD='password'
MASTER_DB_PORT=3160
MASTER_DB_HOST='mysql.hostname'
MASTER_DB_NAME='mysqlDbName'

#Prepare sql query

SQL_Query='select * from tablename limit 10'

#mysql command to connect to database

MYSQL -u$MASTER_DB_USER -p$MASTER_DB_PASSWD -P$MASTER_DB_PORT -h$MASTER_DB_HOST -D$MASTER_DB_NAME <<EOF 
$SQL_Query
EOF
echo "End of script"

Here in the above script, the first part declares the mysql db variables and assigns the DB details. The second part prepares sql query. And the final part executes the mysql command.

PASTE COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

Paste command is one of the useful commands in unix or linux operating system. The paste command merges the lines from multiple files. The paste command sequentially writes the corresponding lines from each file separated by a TAB delimiter on the unix terminal.

The syntax of the paste command is

paste [options] files-list

The options of paste command are:

-d : Specify of a list of delimiters.
-s : Paste one file at a time instead of in parallel.
--version : version information
--help : Help about the paste command.

Paste Command Examples:

Create the following three files in your unix or linux servers to practice to practice the examples:

> cat file1
Unix
Linux
Windows

> cat file2
Dedicated server
Virtual server

> cat file3
Hosting
Machine
Operating system

1. Merging files in parallel

By default, the paste command merges the files in parallel. The paste command writes corresponding lines from the files as a tab delimited on the terminal.

> paste file1 file2
Unix    Dedicated server
Linux   Virtual server
Windows

> paste file2 file1
Dedicated server  Unix
Virtual server    Linux
                  Windows

2. Specifying the delimiter

Paste command uses the tab delimiter by default for merging the files. You can change the delimiter to any other character by using the -d option.

> paste -d"|" file1 file2
Unix|Dedicated server
Linux|Virtual server
Windows|

In the above example, pipe delimiter is specified

3. Merging files in sequentially.

You can merge the files in sequentially using the -s option. The paste command reads each file in sequentially. It reads all the lines from a single file and merges all these lines into a single line.

> paste -s file1 file2
Unix    Linux   Windows
Dedicated server        Virtual server

The following example shows how to specify a delimiter for sequential merging of files:

> paste -s -d"," file1 file2
Unix,Linux,Windows
Dedicated server,Virtual server

4. Specifying multiple delimiters.

Multiple delimiters come in handy when you want to merge more than two files with different delimiters. For example I want to merge file1, file2 with pipe delimiter and file2, file3 with comma delimiter. In this case multiple delimiters will be helpful.

> paste -d"|," file1 file2 file3
Unix|Dedicated server,Hosting
Linux|Virtual server,Machine
Windows|,Operating system

5. Combining N consecutive lines

The paste command can also be used to merge N consecutive lines from a file into a single line. The following example merges 2 consecutive lines into a single line

> cat file1 | paste - -
Unix    Linux
Windows

HOW TO DELETE SYMBOLIC LINK - UNIX / LINUX

Q) I have created a new symbolic link to a directory. Now i want to remove it. How can i remove the symbolic link?

Before we actually see how to delete a symbolic link, let see creating the symbolic to a directory. Follow the below steps for creating and deleting symbolic link:

1. Create New Directory

I am creating a new directory, “linkdir”, in /var/tmp. Later I will create a symbolic link to this directory.

> pwd
/home/mark

>mkdir /var/tmp/linkdir

2. Creating symbolic link

The ln command in unix or linux is used to create a symbolic link.

> pwd
/home/mark

> ln -s /var/tmp/linkdir linkname

>ls -l
total 16
-rw-rw-r-- 1 mark mark  0 Aug 29 23:45 doc
lrwxrwxrwx 1 mark mark 16 Aug 30 00:43 linkname -> /var/tmp/linkdir
-rw-rw-r-- 1 mark mark 14 Aug 29 22:31 sample

You can observe from the output of the ls -l command, that a symbolic link has been created.

3. Deleting a symbolic link

In Unix or linux system everything is a file. Just as we remove the files using the rm command, in the same way remove the link.

> rm linkname

> ls -l
total 12
-rw-rw-r-- 1 pcenter pcenter  0 Aug 29 23:45 doc
-rw-rw-r-- 1 pcenter pcenter 14 Aug 29 22:31 sample

Note that, the rm command just removes the link. However it will not delete the target directory /var/tmp/linkdir. Another way of removing the symbolic link is to use the unlink command.

unlink linkname

HOSTNAME COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

Hostname is the name of the system or server you are logged into. The hostname can also refer to the sitename or computer name. As an example, if an organization domain name is "google.com" and a specific computer name in that doman is "unix-box", then the hostname of the computer is "unix-box.google.com".
The syntax of hostname command in unix or linux system is

hostname [options] [file]

The options of hostname command are:

-a : Prints the alisa name of the host if created any.
-d : prints the domain name
-i : prints the ip address of the host
-s : prints the shortname of the host.
-v : verbose data
-V : version information
-h : help about hostname command

Hostname Command Examples:

1. Print the hostname of the system The basic functionality of the hostname command is to display the name of the system on the terminal. Just type the hostname on the unix terminal and press enter to print the hostname.

> hostname
unix-box.google.com

2. Ip address of the computer You can find the ip address of the computer by using the -i option with hostname command.

> hostname -i
125.20.223.69

3. Print the domain name To know the domain name where the computer resides, use the -d option with hostname command.

> hostname -d
google.com

4. Short hostname By default the hostname command prints the complete name of the computer. You can print a short name by using the -s option. This prints the name upto the first dot in the full hostname.

> hostname -s
unix-box

5. Getting help To get help about the hostanme command either use the man command or the -h option with hostname command.

> man hostname
> hostname -h

MAIL COMMAND EXAMPLES IN UNIX / LINUX TUTORIAL

The Mail command in unix or linux system is used to send emails to the users, to read the received emails, to delete the emails etc. Mail command will come in handy especially when writing automated scripts. For example, you have written an automated script for taking weekly backup of oracle database. How to know the status of backup, whether it is succeeded or not? In this case, sending an email from the automated script at the end of the backup will be helpful in knowing the status.

The syntax of mail command is:

mail [options] to-address [-- sendmail-options]

The options of mail command are listed below:

-v : Verbose mode. Delivery details are displayed on the terminal.
-s : Specify the subject of the mail
-c : Send carbon copies of the mail to the list of users. This is like cc option in Microsoft outlook.
-b : Send blind copies of the mail to the list of users. This is like bcc option in outlook.
-f : Read the contents of the mailbox
-r : Specify the from address in send mail options.

Mail Command Examples - Sending Emails:

1. Sending sample email to user

The basic functionality of the mail command in unix or linux system is to send an email to the user.

echo "Mail body" | mail -s "Mail subject" to@example.com

Here the echo statement is used for specifying the body of the email. The -s option is used for specifying the mail subject. The mail command sends the email to the user to@example.com

2. Specifying the body in a file

You want to compose a mail which contains 100 lines in the body. Specifying the body with the echo statement is a tedious process. So write the contents of the body in a file and send the mail using one of the following options:

Using cat statement:

cat body.txt | mail -s "Mail subject" to@example.com

Using input redirection operator

mail -s "Mail subject" to@example.com < body.txt

Here the body.txt file contains the body of the email. 3. Send mail to more than one user You can send email to more than one user by specifying the users in comma separated list.

mail -s "Mail subject" "user1@example.com,user2@example.com" < body.txt

4. Using the cc and bcc option You can copy the emails to more number of users by using the -c and -b options. An example is shown below:

mail -s "Mail subject" -c "ccuser@gmail.com" -b "bccuser@yahoo.com" "user@example.com" < body.txt

5. Specifying the from address So far the above examples send the emails with from address as the logged in user. You can explicitly specify the from-address using the -r option.

cat body.txt | mail -s "Mail subject" "to-user@example.com" -- -r "from-user@example.com"

6. Attaching files. The mail command does not provide an option for attaching files. There is a workaround for attaching files using the uuencode command. Pipe the output of uuencode command for attaching files.

uuencode attachment-file | mail -s "Mail subject" "to-user@example.com" < body.txt

Mail Command Examples - Reading Emails:

1. Viewing all the received emails Simply type the mail and then press enter to view the received emails.

mail

Another way of viewing the emails is using the -f option. This is shown below:

> mail -f /var/spool/mail/user

Mail version 8.1 6/6/93.  Type ? for help.
"/var/spool/mail/user": 2 messages 2 new
>N  1 root@hostname  Tue May 17 00:00  21/1013  "Mail subject 1"
 N  2 root@hostname  Wed May 18 00:00  21/1053  "Mail subject 2"
&

From the above output, you can see that, it displays the from-address, date and subject of the emails in the inbox. It also displays the ampersand (&) prompt at the end. To go back to the main prompt, type CTRL+z or CTRL+d depending on your operating system and press enter. The ampersand prompt allows you to read, reply, navigate and delete the emails.

2. Reading an email.

To read the Nth email, just enter the mail number at the ampersand prompt and press enter. This is shown below:

> mail -f /var/spool/mail/user

Mail version 8.1 6/6/93.  Type ? for help.
"/var/spool/mail/user": 2 messages 2 new
>N  1 root@hostname  Tue May 17 00:00  21/1013  "Mail subject 1"
 N  2 root@hostname  Wed May 18 00:00  21/1053  "Mail subject 2"
&2
Message 2:
From root@hostname  Wed May 18 00:00  21/1053
---------------
Subject: Mail subject 2
------------

This displays the second email details.

3. Navigating through inbox emails. To go to the next email, enter the + symbol. To go back to the previous email, enter the - symbol at the ampersand prompt.

&-
Message 1:
From root@hostname  Tue May 17 00:00  21/1013
---------------
Subject: Mail subject 1
------------

4. Replying email. Once you have read an email, you can give reply to the mail by typing "reply" and pressing enter.

&reply
To: root@hostname
    root@hostname
Subject: Re: Mail subject1

5. Deleting emails. You can delete a read email by typing the d and pressing enter. You can also specify the email numbers to d option for deleting them.

To delete read email
&d
To delete emails 1 and 2
&d 1 2
To delete range emails from 10 to 30
&d 10-30
To delete all emails in the mbox (mail box)
&d *

ZIP COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

zip is used to compress the files to reduce file size and also used as file package utility. zip is available in many operating systems like unix, linux, windows etc.

If you have a limited bandwidth between two servers and want to transfer the files faster, then zip the files and transfer.

The syntax of zip command is

zip [options] zipfile files_list

The options of zip command are:

-d : Removes the file from the zip archive
-u : Updates the file in the zip archive
-m : Deletes the original files after zipping.
-r : Recursively zips the files in a directory
-x : Exclude the files in creating the zip
-v : verbose mode
-1 : Compresses the files faster
-9 : Compresses the files better
-f : freshen only changed files.
zipfile : creates the zip file with name as zipfile.zip
files_list : list of files to be zipped.

Zip Command Examples:

The files in my current directory are listed below:

docs/linux.pdf
docs/oracle.pdf
docs/unix.pdf
linux-virtual-server.bat
unix-server.dat

Here docs is a directory which contains the files linux.pdf, unix.pdf and oracle.pdf. We will see how to use zip command with examples.

1. Creating a zip file

The zip command in unix or linux system creates an archive with the specified files. This is shown below:

> zip archive linux-virtual-server.bat unix-server.dat
  adding: linux-virtual-server.bat (deflated 80%)
  adding: unix-server.dat (deflated 80%)
> ls
archive.zip  docs  linux-virtual-server.bat  unix-server.dat

The above command creates the zip file with name archive.zip

2. Extracting files from zip

To extract files from the zip, use the unzip command in unix system. This is shown below:

> unzip archive.zip
Archive:  archive.zip
  inflating: linux-virtual-server.bat
  inflating: unix-server.dat
> ls
archive.zip  linux-virtual-server.bat  unix-server.dat

3. Removing file from a zip file

After creating a zip file, you can remove a file from the archive using the -d option. To remove the file unix-server.dat from the archive, run the below zip command:

> zip -d archive.zip unix-server.dat
deleting: unix-server.dat

> unzip archive.zip
Archive:  archive.zip
  inflating: linux-virtual-server.bat

4. Update existing zip file

You can update the files in already created zip file. If any of the files are modified after zipping, you can fresh the zip file with only those modified files using the -f option.

> zip -f archive.zip
freshening: linux-virtual-server.bat (stored 0%)

Another way is using the -u option. This option can be used to update the specified list of files or add new files to the existing zip file.

> zip -u archive.zip  linux-virtual-server.bat temp
updating: linux-virtual-server.bat (deflated 79%)
  adding: temp (stored 0%)

5. Recursively zip files in directory.

To zip a directory recursively, use the -r option with the zip command. This example is shown below:

> zip -r dir_archive docs
  adding: docs/ (stored 0%)
  adding: docs/unix.pdf (stored 0%)
  adding: docs/oracle.pdf (stored 0%)
  adding: docs/linux.pdf (stored 0%)

6. Excluding files in zipping

Let say you are zipping all the files in the current directory and want to exclude some unwanted files. You can exclude these unwanted files using the -x option.

zip exclude_archive * -x linux-virtual-server.bat

The above command zips all the files in the current directory except the file linux-virtual-server.bat

7. Faster compressing

You can compress the files very fast using the -1 option with zip command. An example is shown below with and without using fast compression.

> zip -1 fast_archive linux-virtual-server.bat
  adding: linux-virtual-server.bat (deflated 79%)

>zip normal_archive linux-virtual-server.bat
  adding: linux-virtual-server.bat (deflated 80%)

If you use fast compression, the archive file created will occupy more space (size) when compared to normal compression.

8. Better compression.

To reduce more amount of size the files occupied, you can use the -9 option with the zip command. This gives a better compression.

> zip -9 better_archive linux-virtual-server.bat
  adding: linux-virtual-server.bat (deflated 81%)

SORT COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

Sort command in unix or linux system is used to order the elements or text. Sort command has the capability of sorting numerical values and strings. The sort command can order the lines in a text file.

The syntax of sort command is:

sort [options] filename

The options are:

-b : Ignores leading spaces in each line
-d : Uses dictionary sort order. Conisders only spaces and alphanumeric characters in sorting
-f : Uses case insensitive sorting.
-M : Sorts based on months. Considers only first 3 letters as month. Eg: JAN, FEB
-n : Uses numeric sorting
-R : Sorts the input file randomly.
-r : Reverse order sorting
-k : Sorts file based on the data in the specified field positions.
-u : Suppresses duplicate lines
-t : input field separator

Sort Command Examples:

Before practicing the examples create the below two files in your unix system:

> cat order.txt
Unix distributed 05 server
Linux virtual 3 server
Unix distributed 05 server
Distributed processing 6 system

> cat delim_sort.txt
Mayday|4
Janmon|1
Declast|12

1. Sorting lines of text

The default sort command uses alphabetical order (ASCII order) to sort the file. It treats each line as a string and then sorts the lines.

> sort order.txt
Distributed processing 6 system
Linux virtual 3 server
Unix distributed 05 server
Unix distributed 05 server

2. Sorting based on the field positions.

You can specify the field postions using the -k option of sort command. The sort command uses the space or tab as the default delimiter. To sort based on the data in the second field, run the below command:

> sort -k2 order.txt
Unix distributed 05 server
Unix distributed 05 server
Distributed processing 6 system
Linux virtual 3 server

You can also pecify more than field with k option as a comma separated list. The below command uses the second and fourth fields to sort the data.

> sort -k2,4 order.txt

3. Numeric sorting

Instead of the default alphabetical sorting order, you can make the sort command to sort in numeric order using the -n option. This is shown below:

> sort -nk3 order.txt
Linux virtual 3 server
Unix distributed 05 server
Unix distributed 05 server
Distributed processing 6 system

4. Sort in reverse order

By default, the sort command sorts the data in ascending order. You can change this to descending order using the -r option.

> sort -nrk3 order.txt
Distributed processing 6 system
Unix distributed 05 server
Unix distributed 05 server
Linux virtual 3 server

5. Suppressing duplicates or Print only unique values

You can produce only unique values in the output using the - u option of the sort command.

> sort -u order.txt
Distributed processing 6 system
Linux virtual 3 server
Unix distributed 05 server

Another way is piping the output of sort command to uniq command.

> sort order.txt | uniq

6. Delimited file input

In the second, third and fourth examples we have sorted the data based on the field positions. Here the fields are separted by space or tab character. What if the fields are specifed by any other character? In such cases, we have to specify the input delimiter with the -t option. An example is shown below:

> sort -t'|' -nrk2 delim_sort.txt
Declast|12
Mayday|4
Janmon|1

7. Sorting on months.

We can sort the data in the monthwise using the -M option of the sort command. This is shown below:

> sort -M delim_sort.txt
Janmon|1
Mayday|4
Declast|12

Treats the first 3 characters in the string as month and then sorts in months order.

HOW TO FIND PRIME NUMBER BASH (SHELL) SCRIPT IN UNIX / LINUX

Q) How to find whether the give number is prime number or not using the bash script in unix or linux system?

A prime number is the one which is divisible by one and itself. Examples of prime numbers are 2,3,5 etc. Here we will see how to find whehter a number is prime or not using the bash script in unix or linux operating system. Before going to the script, let see the factor command in unix.

The factor command is used to find the number of factors a given number have. Examples of factor command are listed below:

> factor 1
1:
> factor 2
2: 2
> factor 3
3: 3
> factor 4
4: 2 2
> factor 20
20: 2 2 5

From the above output of the factor command, you can observe that the output contains two words for prime numbers. We will use the factor and wc command to find whether the number is prime or not. The complete bash script for finding the prime number is shown below:

#!/bin/bash
#Assign the first argument from the command line to the variable
Input_Number=$1

if [ $Input_Number -eq 1 ]
then
echo "$Input_Number is neither prime nor composite number"
exit 0;
fi

Num_Factors=`factor $Input_Number|wc -w`
echo $Num_Factors

if [ $Num_Factors -eq 2 ]
then
echo "$Input_Number is a prime number"
else
echo "$Input_Number is not a prime number"
fi

Save the script with name as prime.sh and run the bash script with the below inputs:

> bash prime.sh 1
1 is neither prime nor composite number
> bash prime.sh 7
7 is a prime number
> bash prime.sh 10
10 is not a prime number

FTP BATCH (BASH) SCRIPT EXAMPLE IN UNIX / LINUX TUTORIAL

The FTP is a useful command in unix or linux system used for transfering files between a local server and remote server. You can transfer files between unix systems and also non-unix systems like windows operating system using FTP.

We will see a sample bash script which transfers files from the local machine to the remote machine.

The following script reads the ftp instructions from the dat file and executes them on the remote machine.

#!/usr/bin/bash
echo "A sample Ftp batch script to transfer files."
echo "start ftp script"
ftp -s:ftp_instructions.dat remote-host
echo "End ftp script"

The contents of the instructions.dat file is

user
password
cd /var/tmp
put unix_server.rpm
quit

This script copies the unix_server.rpm file to the remote host. Know more about the ftp command: Ftp command examples in unix and linux

GREP UNIQUE OCCURRENCES IN UNIX / LINUX TUTORIALS

Q) How to grep for unique occurrences of a pattern in a file using the unix or linux commands?

First let me explain the problem. I have a file in my unix operating system. Sample content from the file is shown below:

> cat OS-Versions.dat

linux version 1.1
linux version 1.2
unix distributed system version 2.1
linux virtual server version 2.1
debian server version 1.1

When I grep for the word linux from the above file i will get three lines in the output. This is shown below:

> grep "linux" OS-Versions.dat

linux version 1.1
linux version 1.2
linux virtual server version 2.1

However I want only one occurrence of the linux to be printed on my terminal. That is the output should contain only the first line. How to get this result using unix or linux commands?

Solution:

1. Using grep and head command.

Pipe the output of grep command to head command to get the first line.

> grep "linux" OS-Versions.dat | head -1

2. Using m option of grep command.

The m option can be used to display the number of matching lines. To display the first matching line:

> grep -m 1 "linux" OS-Versions.dat

3. Using the sed command

We can also use the sed command to print unique occurrence of a pattern. This is shown below:

> sed -n "/linux/p;q" OS-Versions.dat

Here the n option duplicates each matched line. The p suppresses the duplicates. The q prints the first matched line.

4. Using awk command.

The awk command to display the first matched line is

> awk '/linux/ {print $0; exit }' OS-Versions.dat

Here the awk command searches for the pattern linux in each line. Once the pattern is found, it prints the line and exits searching in further lines.

TAR COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

In windows operating system, you might have used the winzip and winrar softwares for extracting and archiving the files. Similarly in unix or linux operating system, the tar command is used for creating archive files and also extracting files from the archives.

With the tar command, you can also create compressed archive files. In unix or linux operating system, there are many other commands like gzip and gunzip for creating and extracting archive files. Here we will see the important tar command examples in unix and linux systems which are used frequently in our daily work.

The syntax of tar command is

tar [options] [Archive file] [files list]

The options of tar command are:

c : creates a tar file.
v : verbose. Displays the files information.
f : Specify the tar file name.
r : updates the tar file with new files.
x : Extracts files from the archive (tar file).
t : view contents of tar file.
z : Specify the tar command to create a tar file using gzip in unix.
j : uses bzip2 to create the tar file.

Tar Command Examples:

1. Creating a tar file

Let see a sample example by archiving all the files in my current directory. The ls -l command displays the files and directories in the current directory.

> ls -l 
drwxr-xr-x 2 user group 4096 Aug  8 03:23 debian
-rw-r--r-- 1 user group  174 Aug  2 23:39 file
-rw-r--r-- 1 user group    0 Aug  8 03:22 linux_server.bat
-rw-r--r-- 1 user group   76 Aug  2 02:21 test.sh
-rw-r--r-- 1 user group    0 Aug  8 03:22 unix_distro

We see how to tar all these files using the -c option with the tar command. This is shown below:

> tar -cvf archive.tar *
debian/
file
linux_server.bat
test.sh
unix_distro

> ls
archive.tar  debian  file  linux_server.bat  test.sh  unix_distro

Observe the output of ls command and see the archive.tar file is created.

2. Printing the contents of tar file

We have created the tar file and we dont know whether it contains the actual files or not. To view the contents of the tar file use the -t option as

> tar -tvf archive.tar
drwxr-xr-x user/group   0 2012-08-08 03:23:07 debian/
-rw-r--r-- user/group 174 2012-08-02 23:39:51 file
-rw-r--r-- user/group   0 2012-08-08 03:22:19 linux_server.bat
-rw-r--r-- user/group  76 2012-08-02 02:21:32 test.sh
-rw-r--r-- user/group   0 2012-08-08 03:22:09 unix_distro

3. Updating the tar file with new contents.

You can add new files to the existing archive (tar) file using the -r option.

>touch red-hat-linux.dat

>tar -rvf archive.tar red-hat-linux.dat
red-hat-linux.dat

>tar -tvf archive.tar
drwxr-xr-x pcenter/pcenter   0 2012-08-08 03:23:07 debian/
-rw-r--r-- pcenter/pcenter 174 2012-08-02 23:39:51 file
-rw-r--r-- pcenter/pcenter   0 2012-08-08 03:22:19 linux_server.bat
-rw-r--r-- pcenter/pcenter  76 2012-08-02 02:21:32 test.sh
-rw-r--r-- pcenter/pcenter   0 2012-08-08 03:22:09 unix_distro
-rw-r--r-- pcenter/pcenter   0 2012-08-08 04:00:00 red-hat-linux.dat

Here the touch command creates a new file. The first tar command adds the new file to the existing archive file. The second command displays the contents of the tar file.

4. Extracting the contents of tar file

In the first example, we have created the archive file. Now we will see how to extract the set of files from the archive. To extract the contents of the tar file use the -x option.

> tar -xvf archive.tar
debian/
file
linux_server.bat
test.sh
unix_distro

5. Creating compressed tar file

So far we have created a uncompressed tar file in the above examples. We can create a compressed tar file using the gzip or bzip2.

Compressing files using gzip

> tar -zcvf new_tar_file.tar.gz *

Compressing files using bzip2

> tar -jcvf new_tar_file.tar.bz2 *

To extract or to view the files in a compressed tar file use the appropriate compression option (z or j).

To view files in a gzip compressed tar file
> tar -ztvf new_tar_file.tar.gz

To extract files from a gip compressed tar file
> tar -zxvf new_tar_file.tar.gz

To view files in a bzip2 compressed tar file
> tar -jtvf new_tar_file.tar.bz2

To extract files from a bzip2 compressed tar file
> tar -jxvf new_tar_file.tar.bz2

6. Creating tar file with specified list of files

You can specify a list of files to be included in the newly created tar file.

> tar -cvf unix_files.tar unix_server.bat unix_system.dat

Here the tar command creates the unix_files.tar file which contains only the files unix_server.bat and unix_system.dat

7. Extracting specific files from the tar

You can extract a specific file or a set of files from the archived file.

To extract a specifi file

> tar -xvf unix_files.tar unix_server.bat

To extract all files that start with name unix

> tar -xvf unix_files.tar --wildcards "unix*"

8. Extracting files from multiple archive files.

To extract the files from multiple archive files use the -M option with each -f option. This is shown below:

> tar -xv -Mf archive.tar -Mf unix_files.tar

GREP COMMAND IN UNIX SHELL SCRIPT

Q) How to use the grep command in unix or linux bash scripts to search for a pattern match?

You might have used the grep command to search for a string in a file on the unix command line. Here we will see how to use the grep command in a bash script.

Consider the below file with data

> cat sample_file.txt
linux storage
unix distributed system
linux file server
debian server
fedora backup server

Let see the bash script that prints the lines which contain the word server in it. The bash script code is shown below:

>vi grep_script.sh

#!/bin/bash

WORD=server
INPUT_FILE=sample_file.txt
grep "$WORD" $INPUT_FILE

Now we will execute this script and see what the output is

> bash grep_script.sh

linux file server
debian server
fedora backup server

Here in the script we have hardcoded the filename and the search pattern. We will see how to pass these as arguments from the command line to the script. The bash script is:

>vi grep_script_command_line.sh

#!/bin/bash

INPUT_FILE=$1
WORD=$2
grep "$WORD" $INPUT_FILE

Now run the script as shown below:

> bash grep_script_command_line.sh sample_file.txt server

FTP (FILE TRANSFER PROTOCOL) COMMAND EXAMPLES

The FTP (file transfer program) utility is used to transfer files between a local machine and remote network machine Using the File Transfer protocol. In simple terms it transfers / copies files between two computers. You can transfer files between unix systems and also non-unix systems like windows operating system using FTP.

The FTP command is simple to use and easy to learn. Let see useful examples of FTP command in detail.

FTP Command Examples:

If you are using windows operating system, open the command prompt and practice the below FTP commands. If you are using unix or linux operating systems, just simply type the ftp command on the terminal.

1. Connecting to Remote Host

First you need to connect to a remote host before doing any operations. You can use any one of the following methods to connect to a remote host. First method is

> ftp remote-server-name
connected to remote-server-name
User-Name:
Password:
ftp>

Once the ftp connects to the remote server name, it will prompt you to enter the user name and password. After successful login, your terminal or prompt changes to "ftp>".

Another method is to use the open option with ftp command. This is shown below:

>ftp
ftp>open remote-server-name
connected to remote-server-name
User-Name:
Password:
ftp>

If the ftp command fails to connect to the remote server, then you will get the below error:

ftp: connect: Connection refused

2. Copy file from remote machine to local machine.

The get option is used to download or transfer a file from the remote system to the local system.

ftp> get windows-cleveland.bat

This will download the specified file (windows-cleveland.bat) from the remote systems current directory.

3. Copying multiple files from remote machine to local machine.

You can use the mget to transfer multiple files from the remote host to local host.

ftp>mget *.png

This will download all the png images to the local machine.

4. Transferring file from local server to remote server

The put option is used to copy the file from the local host to the remote host.

ftp>put linux-virtual-server.rpm

This command puts the rpm file into the remote machine.

5. Transferring multiple files to the remote server.

You can use the mput option to transfer more than one file from local system to the remote system.

ftp>put *.rpm

6. Executing commands in remote machine.

After connecting to the remote network machine using the ftp, you can run commands like ls to list the files, cd to change directory and many more.

ftp> ls

This will list the files and directories in the remote machines current directory.

7. Executing commands in local machine.

Once you have connected to the remote host, to run the commands on local machine you need to exit from the ftp connection. Instead of this, there is a way to run commands on local host without exiting from the ftp connection. Use the ! symbol before the command you want to run.

ftp> !ls

Now this will list the files in the local machines current directory.

8. Changing the file transferring mode.

You can change the file transfer modes to ascii and binary modes. Use the below commands to change the mode.

ftp>ascii
ftp>binary

9. Deleting files on remote machine

You can use the delete or mdelete to remove a single file or multiple files in the remote machine.

ftp>delete linux-dedicated-server.dat
ftp>mdelete *.dat

10. Disconnecting from ftp connection.

Use the quit command to close the ftp connection.

ftp>quit

11. Using FTP command in batch scripts

The following script reads the instructions from the dat file and executes them on the remote machine.

echo "Ftp command batch script"
echo "start"
ftp -s:instructions.dat remote-host
echo "End"

The contents of the instructions.dat file is

user
password
cd /var/tmp
put oracle_storage.exe
quit

12. Getting the help about ftp command.

To know more about the ftp command, just type the help on the prompt. It will display the options/commands that you can use with ftp command.

ftp>help
Commands may be abbreviated.  Commands are:

!       disconnect mdelete  preserve  runique
$       edit       mdir     progress  send
account exit       mget     prompt    sendport
append  form       mkdir    proxy     site
ascii   ftp        mls      put       size
bell    get        mode     pwd       sndbuf
binary  gate       modtime  quit      status
bye     glob       more     quote     struct
case    hash       mput     rcvbuf    sunique
cd      help       msend    recv      system
cdup    idle       newer    reget     tenex
chmod   image      nlist    rename    trace
close   lcd        nmap     reset     type
cr      less       ntrans   restart   umask
debug   lpwd       open     rhelp     user
delete  ls         page     rmdir     verbose
dir     macdef     passive  rstatus   ?

CVS COMMAND EXAMPLES - UNIX / LINUX TUTORIALS

CVS (concurrent Version Control system) is a version controlling system used to record the history of the files. Whenever a code is changed in software, there might be chance of bugs creeping into that. With CVS, you can easily get the old version of the code and see what part of the code exactly created the bug.

In CVS, we can save every version of the file. The CVS only stores the differences between the files. This saves a huge amount of disk space.

The general syntax of CVS command is

cvs [option] filename

CVS Command Examples:

1. Checking out file

You can check out a file from the CVS repository with the checkout (co) option. This is shown below:

cvs co oracle_storage.dat

2. Adding a file to the repository

Use the add option to add a new file to the cvs repository.

cvs add mysql.bat

This will not commit the file to the cvs. It just simply adds the file.

3. Committing the file.

Once you have added a file to the CVS repository, you have to commit the file. Use the commit option with cvs command for committing a file.

cvs commit msql.bat

This will open an editor. Enter the comments and save by using the :wq.

4. Difference between files

You can find the differences between the local file with the latest version of the file in the cvs repository using the diff option.

cvs diff wireless.php

5. Update the file

You can update the local file with the latest version of the file from CVS repository using the update option.

cvs update -A network.dat

6. Update to particular version

You can get a particular version of the file from the cvs. Specify the version number of the file with -j option.

cvs update -j version-number network.bat

7. Adding binary files

You can add binary or image files to the CVS repository. Use the -kb option to add binary files.

cvs add -kb unix.png

8. Removing file from CVS

You can remove unwanted files permanently from the CVS repository using the remove option.

cvs remove linux_system.dat

After issuing this command you have to do a cvs commit. Otherwise the file will not be removed from the repository.

KILL COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

The Kill command in unix or linux operating system is used to send a signal to the specified process or group. If we dont specify any signal, then the kill command passes the SIGTERM signal. We mostly use the kill command for terminating or killing a process. However we can also use the kill command for running a stopped process.

The syntax of kill command is

kill [-s signal] pid
kill -l

The options to the kill command are:

pid : list of process that kill command should send a signal
-s signal : send the specified signal to the process
-l : list all the available signals.

Let see some of the useful kill command examples in unix or linux system.

Kill Command Examples:

1. Listing all the signal names.

Run the kill command with -l option to list all the available signal names.

> kill -l
HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM USR1 USR2
CLD PWR WINCH URG POLL STOP TSTP CONT TTIN TTOU VTALRM PROF XCPU XFSZ WAITING
LWP FREEZE THAW CANCEL LOST RTMIN RTMIN+1 RTMIN+2 RTMIN+3 RTMAX-3 RTMAX-2
RTMAX-1 RTMAX

Some the important signals which we use in our daily usage ar listed below:

Number	Signal Name	Description
0	SIGNULL	Used to check access to the process id
1	SIGHUP	Hup signal. Terminates the process.
2	SIGNINT	Interrupt signal. Terminating the process
3	SIGQUIT	Quit signal. Terminate process with core dump
9	SIGKILL	Forcibly killing a process
24	SIGSTOP	Pausing the process
26	SIGCONT	Runs a stopped process

To know more about a signal, check in man pages. To know about the signal 9, run the below man command:

man 5 signal

2. Getting the process id

To know the process id of a process running in the unix system, use the ps command as

ps -aef
root  4529   657   0   Jul 20 ?           0:06 /usr/local/sbin/sshd -R
....
....

The second field in the output is the process Id. Here the /usr/local/sbin/sshd -R is running with the process id 4529.

3. Killing a process.

To kill processes simply pass the process id to the kill command. This is shown below:

kill 4529

4. Forcefully killing a process.

Use the -9 option with the kill command to kill a process force fully. The following kill command terminates the process forcefully:

kill -9 1567
kill -SIGKILL 1567
kill -KILL 1567
kill -s SIGKILL 1567
kill -s KILL 1567

Avoid using the kill -9 for terminating a process. This will cause memory leaks in the operating system and leads to many other issues.

SSH COMMAND EXAMPLES - UNIX / LINUX TUTORIALS

SSH client utility in unix or linux server is used to logging into a remote host and execute commands on the remote machine. The rlogin and rsh commands can also be used to login into the remote machine. However these are not secure. The ssh command provides a secure connection between two hosts over a insecure network.

The syntax ssh command is

ssh [-l username] hostname | user@remote-hostname [command]

Let see the examples of ssh command.

SSH Command Examples:

1. Logging to a remote server

You can login to a remote server from the local host as shown below:

localhost:[~]> ssh -l username remote-server
username@remote-server password:
remote-server:[~]>

Alternatively you can use the below ssh command for connecting to remote host:

localhost:[~]> ssh username@remote-server
username@remote-server password:
remote-server:[~]>

Note: If you are logging for the first time, then it will prints a message that host key not found and you can give yes to continue. The host key of the remote server will be cached and added to the .ssh2/hostkeys directory in your home directory. From second time onwards you just need to enter the password.

2. Logging out from remote server

Simply enter the exit command on the terminal to close the connection. This is shown below:

remote-server:[~]>exit
logout
Connection to remote-server closed.
localhost:[~]>

3. Running remote commands from local host

Sometimes it is necessary to run the unix commands on the remote server from the local host. An example is shown below:

localhost:[~]> ssh user@remote-host "ls test"
online-backup.dat
oracle-storage.bat
unix-dedicated-server.txt

The ssh command connects to the remote host, runs the ls command, prints the output on the local host terminal and exits the connection from remote host.

Let see whether the ls command actually displayed the correct result or not by connecting to the remote host.

localhost:[~]> ssh user@remote-host
user@remotehost password:
remotehost:[~]> cd test
remotehost:[~/test]> ls
online-backup.dat
oracle-storage.bat
unix-dedicated-server.txt

4. Version of the SSH command

We can find the version of SSH installed on the unix system using the -V option to the ssh. This is shown below:

> ssh -V
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

5. Debugging the SSH Client

When we are not able to connect to the remote host, it is good to debug and find the exact error messages that causing the issue. Use the -v option for debugging the ssh client.

ssh -v user@remote-host
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to remote-host [172.22.200.140] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/identity type -1
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_dsa type 2
debug1: loaded 3 keys
..........
..........

6. Copying files between remote host and local host.

We can use the scp command to copy the files securely between the local host and remote host using the ssh authentication.

To copy the file from local host to remote hosts /var/tmp/ directory, run the below scp command.

scp filename user@remote-host:/var/tmp/

To copy the file from remote hosts /usr/local/bin/ directory to local hosts current directory, run the below scp command.

scp user@remote-host:/usr/local/bin/add.sh .

WC COMMAND EXAMPLES - COUNT OF LINES, WORDS, CHARACTERS - UNIX / LINUX

WC command in unix or linux is used to find the number of lines, words and characters in a file. The syntax of wc command is shown below:

wc [options] filenames

You can use the following options with the wc command.

-l : Prints the number of lines in a file.
-w : prints the number of words in a file.
-c : Displays the count of bytes in a file.
-m : prints the count of characters from a file.
-L : prints only the length of the longest line in a file.

Let see how to use the wc command with few examples. Create the following file in your unix or linux operating system.

> cat unix_wc.bat
Oracle Storage
unix distributed system
linux file server
debian server
Oracle backup server

WC Command Examples:

1. Printing count of lines

This is the most commonly used operation to find the number of lines from a file. Run the below command to display the number of lines:

wc -l unix_wc.bat
5 unix_wc.bat

Here in the output, the first field indicates count and second field is the filename

2. Displaying the number of words.

Just use the -w option to find the count of words in a file. This is shown below:

wc -w unix_wc.bat
13 unix_wc.bat

3. Print count of bytes, count of characters from a file

We can use the -c and -m options to find the number of bytes and characters respectively in a file.

> wc -c unix_wc.bat
92 unix_wc.bat
> wc -m unix_wc.bat
92 unix_wc.bat

4. Print the length of longest line

The -L option is used to print the number of characters in the longest line from a file.

wc -L unix_wc.bat
23 unix_wc.bat

In this example, the second line is the longest line with 23 characters.

5. Print count of lines, words and characters.

If you dont specify any option to the wc command, by default it prints the count of lines, words and characters. This is shown below:

wc unix_wc.bat
 5 13 92 unix_wc.bat

6. Wc help

For any help on the wc command, just run the wc --help on the unix terminal.

SCP COMMAND EXAMPLES - LINUX / UNIX TUTORIALS

SCP stands for secure copy is used to copy data (files or directories) from one unix or linux system to another unix or linux server. SCP uses secured shell (ssh) to transfer the data between the remote hosts. The features of SCP are:

Copies files within in the same machine
Copies files from local machine to remote machine.
Copies files from remote machine to local machine.
Copies files between two different remote servers.

SCP Command Syntax:

The syntax of SCP command is

scp [Options] [[User@]From_Host:]Source_File [[User@]To_Host:][Destination_File]

Each element of the scp command is explained in detail below:

User is the one who have the permissions to access the files and directories. User should have read permissions if it is a source and write permissions if it is the destination.
From_Host: hostname or Ip address where the source file or directory resides. This is optional if the from host is the host where you are running the scp command.
Source_File: Files or directories to be copied to the destination.
To_Host: Destination host where you want to copy the files. You can omit this when you want to copy the files to the host where you are issuing the scp command.
Destination_File: Name of the file or directory in the target host.

SCP Command Options:

The important SCP command options are listed below:

-r : Recursively copies the contents of source files or directories.
-p : Preserves the access time, modification time, permissions of the source files in the destination.
-q : Progress bar in not displayed
-v : verbose mode. Displays debugging messages.
-P : copy files using the specified port number.

SCP Command Examples:

Let see the examples of scp command in unix or linux system.

1. Copying with in the same system

You can use the scp command just like the cp command to copy files from one directory to another directory.

scp Unix-storage.dat /var/tmp/

This command copies the file unix-storage.dat from current directory to the /var/tmp directory.

2. Copy file from local host to remote server

This is most frequently used operation to transfer files in unix system.

scp filename user@remotehost:/remote/directory/

This command connects to the remote host and copies the specified file to the /remote/directory/.

3. Copy files from remote host to local server.

This operation is used when taking backup of the files in remote server.

scp user@remotehost:/usr/backup/oracle_backup.dat .

This command copies the oracle backup file in the remote host to the current directory.

4. Copying files between two remote servers

The scp command can also be used to copy files between two remote hosts.

scp source_user@source_remote_host:/usr/bin/mysql_backup.sh target_user@target_remote_host:/var/tmp/

The above command copies the mysql bakup shell script from the source remote host the /var/tmp directory of target remote host.

5. Copying a directory.

To copy all the files in a directory, use the -r option with the scp command. This makes the scp command to copy the directory recursively.

scp -r directory user@remotehost:/var/tmp/

The above command copies the directory from local server to the remote host.

6. Improving performance of scp command

By default the scp command uses the Triple-DES cipher/AES-128 to encrypt the data. Using the blowfish or arcfour encryption will improve the performance of the scp command.

scp -c blowfish filename  user@remoteserver:/var/
scp -c arcfour  localfile user@remoteserver:/var/

7. Limit bandwidth

You can limit the bandwidth used by the scp command using the -l option.

scp -l bandwidth_limit filename user@hostname:/usr/backup/
Here bandwidth_limit is numeric to be specified in kilobits per second.

8. Specifying the port number

We can make the scp command to copy the files over a specified port number using the -P option.

scp -P 6001 storage_backup.bat username@hostname:/tmp/

XARGS COMMAND EXAMPLES IN UNIX / LINUX TUTORIAL

Xargs command in unix or linux operating system is used to pass the output of one command as an argument to another command. Some of the unix or linux commands like ls and find produces a long list of filenames. We want to do some operation on this list of file names like searching for a pattern, removing and renaming files etc. The xargs command provide this capability by taking the huge list of arguments as input , divides the list into small chunks and then passes them as arguments to other unix commands.

Unix Xargs Command Examples:

1. Renaming files with xargs

We have to first list the files to be renamed either by using the ls or find command and then pipe the output to xargs command to rename the files. First list the files which end with ".log" using the ls command.

ls *.log
oracle.log  storage.log

> ls *.log | xargs -i mv {} {}_bkp
> ls *_bkp
oracle.log_bkp  storage.log_bkp

You can see how the log files are renamed with backup (bkp) suffix. Here the option "i" tells the xargs command to replace the {} with the each file returned by the ls command.

2. Searching for a pattern

We can combine the grep command with xargs to search for a pattern in a list of files returned by another unix command (ls or find). Let’s list out all the bash files in the current directory with the find command in unix.

find . -name "*.bash"
./sql_server.bash
./mysql_backup.bash
./oracle_backup.bash

Now we grep for the "echo" statements from the list of files returned by the find command with the help of xargs. The command is shown below:

find . -name "*.bash" |xargs grep "echo"

If you don’t use xargs and piped the output of find command to grep command directly, then the grep command treats each file returned by the find command as a line of string and searches for the word "echo" in that line rather in that file.

3. Removing files using xargs

We can remove the temporary files in a directory using the rm command along with the xargs command. This is shown below:

ls "*.tmp" | xargs rm

This removes all the files with ".tmp" suffix.

4. Converting Multi-line output to Single line output.

If you run the ls -1 command, it will list each file on a separate line. This is shown below:

ls -1
oracle.txt
online_backup.dat
mysql_storage.bat

We can convert this multi-line output to single line output using the xargs command. This is shown below:

ls -1 | xargs
oracle.txt online_backup.dat mysql_storage.bat

5. Handling spaces in file names

By default the xargs command treats the space as a delimiter and sends each item as an argument to the unix command. If the file name contains a space (example: "oracle storage"), then each item will be treated as a separate file and will be passed as an argument. This will cause an issue. Let see how to handle the spaces in file names with an example.

Creating a file which contains space in the name
> touch "oracle storage"

> ls oracle\ storage | xargs grep "log"
grep: oracle: No such file or directory
grep: storage: No such file or directory

You can see that grep command is treating oracle as separate file and storage as separate file. This is because of xargs treats space as delimiter. To avoid this kind of errors use the -i option with braces as shown in below:

> ls oracle\ storage | xargs -i grep "log" {}

If you want to know what command the xargs is executing use the -t option with xargs. This will print the command on the terminal before executing it.

6. Passing subset of arguments

We can pass only a subset of arguments from a long list of arguments using the -n option with xargs command. This is shown in below.

> ls -1
backup
mysql
network
online
oracle
storage
wireless

> ls -1 | xargs -n 3 echo
backup mysql network
online oracle storage
wireless

You can see from the above output that 3 arguments are passed at a time to the echo statement.

Important Notes on Xargs Command:

1. Xargs directly cannot handle files which contain new lines or spaces in their names. To handle this kind of files use the -i option with xargs command. Another way to handle these characters is to treat the new line or spaces as null characters using th -0 option with xargs. However this requires that the input to xargs should also use the null as separator. An example is shown below

find . -print0 | xargs -0 rm

The print0 in find command makes the newline or space characters as null separator.

2. By default the xargs uses the end of the file string as "_". If this string appears in input string, then xargs command stops reading the input and rest of the input is ignored. You can change the end of file string by using the "-eof" option.

3. To know more about xargs command, run the xargs --help on the unix or linux terminal.

BASH SHELL SCRIPT TO READ / PARSE COMMA SEPARATED (CSV) FILE - UNIX / LINUX

Q) How to parse CVS files and print the contents on the terminal using the bash shell script in Unix or Linux system?

It is the most common operation in Unix system to read the data from a delimited file and applying some operations on the data. Here we see how to read the Comma separated value (CSV) file using the while loop in shell script and print these values on the Unix terminal.

Consider the below CSV file as an example:

> cat os_server.csv
Unix, dedicated server
Linux, virtual server

This file contains two fields. First field is operating system and the second field contains the hosting server type. Let see how to parse this CVS file with simple bash script shown below:

#!/usr/bin/bash

INPUT_FILE='unix_file.csv'

IFS=','

while read OS HS
do

echo "Operating system - $OS"
echo "Hosting server type - $HS"

done < $INPUT_FILE

Here IFS is the input field separator. As the file is comma delimited, the IFS variable is set with comma. The output of the above script is

Operating system - Unix
Hosting server type - dedicated server
Operating system - Linux
Hosting server type - virtual server

Here in the code, the fourth line (IFS=',') and sixth line (while) can be merged into a single statement as shown below:

while IFS=',' read OS HS

ADD JOB TO CRON (CRONTAB COMMAND EXAMPLES) - UNIX / LINUX TUTORIALS

Unix or Linux operating system provides a feature for scheduling the jobs. You can setup command or scripts which will run periodically at the specified time. The Crontab is command used to add or remove jobs from the cron. The cron service is a daemon runs in the background and checks for /etc/crontab file, /etc/con.*/ directories and /var/spool/cron/ directory for any scheduled jobs.

Each user has a separate /var/spool/cron/crontab file. Users are not allowed directly to modify the files. The crontab command is used for setting up the jobs in the cron.

The format of crontab command is

* * * * * command to be executed

You can easily remember this command in the below format

MI HH DOM MON DOW command

The field descriptions of the crontab are explained below:

MI  : Minutes      from 0 to 59
HH  : Hours        from 0 to 23
DOM : Day of month from 0 to 31
MON : Months       from 1 to 12
DOW : Day of week  from 0 to 7   (0 or 7 represents Sunday)

Command: Any command or script to be scheduled

Let see the usage of crontab command with examples.

1. List crontab entries

You can list out all the jobs which are already scheduled in cron. Use "crontab -l" for listing the jobs.

crontab -l
0 0 * * *  /usr/local/bin/list_unix_versions.sh

The above contab command displays the cron entries. Here the shell script for listing the unix versions (list_unix_version.sh) is scheduled to run daily at midnight.

2. List crontab entries of other users

To list the corntab entries of other user in the unix, use the -u option with crontab. The syntax is shown below:

crontab -u username -l

3. Removing all crontab entries

You can un-schedule all the jobs by removing them from the crontab. The syntax for removing all the crontab entries is

crontab -r

For removing other user’s crontab entries:
crontab -u username -r

4. Editing the crontab

You can edit the crontab and add a new job to it. You can also remove an existing job from the crontab. Use the -e option for editing the crontab.

crontab -e

For editing other user’s crontab entries:
crontab -u username -e

This will open a file in VI editor. Now use the VI commands for adding, removing the jobs and for saving the crontab entries.

5. Schedule a job to take oracle backup on every Sunday at midnight

Edit crontab using "crontab -e" and append the following entry in the file.

0 0 * * 0 /usr/local/bin/oracle_backup.sh

6. Schedule a job to run every six hours in a day

You can schedule a job to run more than once in a day. As an example the following crontab entry takes the mysql backup more than once in a day.

0 0,6,12,18 * * * /usr/bin/mysql_backup.sh

Here the list 0,6,12,18 indicates midnight, 6am, 12pm and 6pm respectively.

7. Schedule job to run for the first 15 days of the month.

You can schedule a job by specifying the range of values for a field. The following example takes the sql server backup daily at midnight for the first 15 days in a month.

0 0 * 1-15 * /usr/bin/sql_server_backup.sh

8. Schedule job to run every minute.

The following crontab command runs the command to send emails to group of users for every minute.

* * * * * /bin/batch_email_send.sh

9. Taking backup of cron entries

Before editing the cron entries, it is good to take backup of the cron entries. So that even if you do mistake you can get back those entries from the backup.

crontab -l > /var/tmp/cron_backup.dat

10. Restoring the cron entries

You can restore the cron entries from the backup as

crontab cron_backup.dat

Understanding the Operators:

There are three operators allowed for specifying the scheduling times. They are:

Asterisk (*) : Indicates all possible values for a field. An asterisk in the month field indicates all possible months (January to December).
Comma (,) : Indicates list of values. See example 6 above.
Hyphen (-): Indicates range of values. See example 7 above.

Disabling Emails:

By default the crontab sends emails to the local user if the commands or scripts produce any output. To disable sending of emails redirect the output of commands to /dev/null 2>&1.

0 0 * 20 * /usr/bin/online_backup.sh > /dev/null 2>&1

Note: you cannot schedule a job to run at second’s level as the minimum allowed scheduling is at minute level.

YUM COMMAND EXAMPLES - INSTALL, UPDATE, UNINSTALL - LINUX TUTORIALS

Yum (Yellowdog Updater Modified) is one of the package manager utility in Linux operating system. Yum command is used for installing, updating and removing packages on Linux environment. Some other package manger utilities in linux system are apt-get, dpkg, rpm etc.

By default yum is installed on some of the linux distributions like CentOS, Fedora, Redhat. Let see some of the mostly used yum commands with examples.

1. Listing available packages

You can list all the available packages in the yum repository using the list.

yum list
yum list httpd

Display list of updated softwares
yum list updates

2. View installed packages

To print all the packages which are installed on your linux system, execute the following command.

yum list installed

3. Search for package

Searching for a package to be installed helps you when the exact package name is not known in advance. The syntax for searching a package is

yum search package_to_be_searched

If you want to search for mysql package, then execute the following yum command.

yum search mysql

However this yum command matches only in the name and summary. Use "search all" for everything.

4. How to install package using yum

"Yum install package_name" will install the specified package name in the linux operating system. The yum command will automatically finds the dependencies and also installs them in the linux machine.

yum install firefox.x86_64

Yum will prompt for the user to accept or decline before installing the package. If you want yum to avoid prompting the user, then use -y option with yum command.

yum -y install package_name

5. Check package is installed or not.

After installing, you don’t know whether the package is installed or not. To check whether a package is installed or not, run the below yum command.

yum list installed package_name
yum list installed firefox.x86_64

6. Updating existing package using yum

You can upgrade older version of package to the newer version by using the yum command. To check for the updates and for upgrading a package to the current version, run the following yum command.

yum update firefox.x86_64

7. Uninstalling a package.

You can remove (uninstall) a package with all dependencies using the "yum remove package_name". This is shown below:

yum remove firefox.x86_64

The yum remove prompts the user to accept or decline the uninstalling of package.

8. Information about the package.

You can print and check for the information about the package before installing it. Execute the following yum command to get info about the package.

yum info firefox.x86_64

9. Print what package provides the file.

You can know the name of the package that file belongs to. For example, if you want to know the file '/etc/passwd' belongs to which file, run the below yum command.

yum whatprovides /etc/passwd

10. Print List of group software

Basically, the related softwares are grouped together. You can install all the packages belong to a single group at one shot. This will help you in saving time in installing each individual package. To print the list of available package groups, run the below yum command.

yum grouplist

11. Installing a software group.

You can install the group software, by running the following command.

yum groupinstall 'software_group_name'

12. Update a software group.

You can update the installed software group from a older version to latest version. The yum command for this is

yum groupupdate 'software_group_name'

13. Removing a software group

You can uninstall (delete) existing software group using 'yum groupremove'. This is shown below:

yum groupremove 'software_group_name'

14. Print yum repository

You can display all the packages in the yum repository using the below command.

yum repolist [all]

Here "all" is optional. If you provide "all", then it displays enabled and disabled repositories. Otherwise it displays only enabled repositories.

15. More info about yum command

If you want to know more information about the yum command, then run the man on yum as

man yum

REPLACE STRING ON NTH LINE - UNIX / LINUX

Q) How to search for a string (or pattern) in a file and replace that matched string with another string only on the specified line number in the file?

Let see this with the help of an example. Consider the sample file with the following content:

> cat fedora_overview.dat
Fedora is an operating system based on linux kernal.
Fedora applications include LibreOffice, Empathy and GIMP.
Security features of Fedora is Security-Enhanced Linux.
Implements variety of security policies, access controls.
SELinux operating system was introduces in Fedora Core 2.

First we will see how to replace a pattern with another patter using sed command. The sed command for replacing the string "Fedora" with "BSD" is:

sed 's/Fedora/BSD/' fedora_oveview.dat

The above sed command will replace the string on all the matched lines. In this example, it replaces the string on first, second, third and fifth lines.

If we want to replace the pattern on a specific line number, then we have to specify the line number to the sed command. The sed command syntax for replacing the pattern on Nth line is:

sed 'n s/search_pattern/replace_pattern/' filename

To replace "Fedora" with "BSD" on second line, run the below sed command on the unix terminal:

sed '2 s/Fedora/BSD/' fedora_oveview.dat

DELETE RANGE OF LINES - UNIX / LINUX

Q) How to delete range of lines from a file using unix or inux command?

Unix provides simple way to delete lines whose number are line numbers are between m and n. This feature is not directly available in windows operating system.

We can use the sed command for removing the lines. The syntax for removing range of lines (between m and n) is:

sed 'm,nd' filename

Here the number n should be greater than m. Let see this with an example. Consider the sample file with the following contents:

> cat linux_range.dat
Linux is just like unix operating system.
Linux is leading os on servers like mainframes and super computers.
Linux also runs on embeded systems like network routers, mobiles etc.
Android is built on using linux kernal system.
Variants of linux are debian, fedora and open SUSE.

From the above file if we want to delete the lines from number 2 to 4, then run the below sed command in unix:

sed '2,4d' linux_range.dat

However this command just prints the lines on the terminal and did not remove from the file. To delete the lines from the source file itself use the -i option to the sed command.

sed -i '2,4d' linux_range.dat

You can negate this operation and can delete lines that are not in the specified range.. This is shown in the following sed command:

sed -i '2,4!d' linux_range.dat

REMOVE LAST LINE OR FOOTER LINE - UNIX / LINUX

Q) How to delete the trailer line (first line) from a file using the unix or inux command?

Let see how to remove the last line from a file with an example. Consider the file with sample content as shown below:

> cat unix_file.txt
Unix is a multitasking, multi-user operating system.
Unix operating system was first developed in assembly language.
In later periods unix is developed with C programming.
In academics BSD variant of unix is used.
Mostly used unix flavors are Solaris, HP-UX and AIX.

Unix Sed command is popularly used for searching a pattern and then replacing the matched pattern with another string. However we can also use the sed command for deleting the lines from a file.

To remove the last line from a file, run the below sed command:

sed '$d' unix_file.txt

Here $ represents the last line in a file. d is for deleting the line. The above command will display the contents of the file on the unix terminal excluding the footer line. However it does not delete the line from the source file. If you want the line to be removed from the source file itself, then use the -i option with sed command. This command is shown below:

sed -i '$d' unix_file.txt

If you want only the footer line to be present in the file and remove other lines from the line, then you have to negate the delete operation. For this use the exclamation (!) before the d. This is shown in the following sed command:

sed -i '$!d' unix_file.txt

DELETE FIRST LINE OR HEADER LINE - UNIX / LINUX

Q) How to remove the header line (first line) from a file using the unix or inux command?

Let see how to delete the first line from a file with an example. Consider the file with sample content as shown below:

> cat linux_file.txt
First line a file is called header line.
Remaining lines are generally called as detail or data lines.
Another detail line which is in the third row.
The last line in a unix file is called footer line.

Mostly we see the sed command for replacing the strings in a file. We can also use the sed command for removing the lines in a file. To delete the first line from a file, run the following sed command:

sed '1d' linux_file.txt

Here 1 represents the first line. d is for deleting the line. The above command will print the contents of the file on the unix terminal by removing the first line from the file. However it does not remove the line from the source file. If you want to changes in the source file itself, then use the -i option with sed command. this command is shown below:

sed -i '1d' linux_file.txt

You can keep only the first line and remove the remaining lines from the file by negating the above sed command. You have to use the exclamation (!) before the d command. The following sed command keeps only the frist line in the file and removes the other lines:

sed -i '1!d' unix_file.txt

PRINT LINES ENDING WITH STRING - UNIX / LINUX

Q) How to display the lines from a file that ends with specified string ( or pattern) using unix or linux commands?

Printing the lines that ends with specified pattern on the terminal is most commonly used operation in unix environment. Grep is the frequently used command in unix for searching a pattern in a file and printing the lines that contains the specified pattern.

We will see how to print the lines that end with the specified pattern with an example. Consider the Sample log file data as an example, which is shown below:

> cat unix_os_install.dat
How to install unix virtual machine on windows operating system
First download virtual box or vmware and install it on windows
Next place the ubuntu CD and follow the instructions in vmware
For running ubuntu allocate at least 512MB RAM in vmware
After installing, start the unix operating system

Now if we want to print the lines that end with the string "vmware", then use the grep command with dollar ($) in the pattern. The complete unix grep command is shown below:

grep "vmware$" unix_os_install.dat

Here $ is used to indicate the end of the line in a file. This grep command prints the lines that end with the word "vmware". In this example, the third and fourth lines are printed on the unix terminal.

If you want to display the lines that end with the word "system", then run the following grep command on unix command prompt:

grep "windows$" unix_os_install.dat

SEARCH (MATCH) FOR WHOLE WORDS IN FILE - UNIX / LINUX

Q) How to print the lines from a file that contain the specified whole word using unix or linux command.

Whole words are complete words which are not part of another string. As an example consider the sentence, "How to initialize shell". Here the words "how, to, shell, initialize" are whole words. However the word "initial" is not a whole word as it is part of another string (initialize).

Let see this in detail with the help of an example. Consider the following sample data in a file:

> cat unix_word.txt
matching for whole words in file is easy with unix
use the unix grep command match for a pattern.
Another example of whole word is: boy's
Here boy is a whole word.

Now we have the sample file. First we will see how to search for a word and print the lines with the help of grep command in unix. The following grep command prints the lines that have the word "match" in the line:

grep "match" unix_word.txt

The above command displays the first two lines on the unix terminal. Even though the first line does not contain the whole word "match", the grep command displays the line as it matches for the word in the string "matching". This is the default behavior of grep command.

To print only the lines that contain the whole words, you have to use the -w option to the grep command. The grep command for this is:

grep -w "match" unix_word.txt

Now the above command only displays the second line on the unix terminal. Another example for matching the whole word “boy” is shown below:

Grep -w "boy" unix_word.txt

PRINT NON MATCHING LINES (INVERSE OF GREP COMMAND) - UNIX / LINUX

Q) How to print the lines from a file that does not contain the specified pattern using unix / linux command?

The grep command in unix by default prints the lines from a file that contain the specified pattern. We can use the same grep command to display the lines that do not contain the specified pattern. Let see this with the help of an example.

Consider the following sample file as an example:

> cat unix_practice.txt
You can practice unix commands by installing the unix operating system.
There so many unix flavors available in the market.
ubuntu is one of the used operating system. it is available for free.
Go to the ubuntu website and download the OS image.
Alternatively you can order for a free CD. 
The question is how to start learning unix.
First know about the unix operating system in detail.
Then start slowly learning unix commands one by one.
Practice these unix command daily to have a grip.

First we will see how to display the lines that match a specified pattern. To print the lines that contain the word "ubuntu", run the below grep command on unix terminal:

grep "ubuntu" unix_practice.txt

The above command displays the third and fourth lines from the above sample file.

Now we will see how to print non matching lines which means the lines that do not contain the specified pattern. Use the -v option to the grep command for inverse matching. This is shown below:

grep -v "ubuntu" unix_practice.txt

This command prints the first, second and fifth lines from the example file.

PRINT LINES STARTING WITH STRING - UNIX / LINUX

Q) How to print the lines from a file that starts with specified string (pattern) using unix or linux commands?

Displaying lines that starts with specified pattern is most commonly used when processing log files in unix environment. Log files are used to store the messages of shell scripts (echo statements). We can search for errors in the log file using grep command. Generally, the error keyword will appear at the start of the line.

Sample log file data is shown below:

> cat linux_log_file.dat
Success: Shell script execution started.
Success: Exported the environment variables in shell environment.
Success: Able to connect to oracle DB from bash script.
Success: Run the SQL statement and inserted rows into table.
Error: Unable to process the stored procedure.
Message: Processing the statements in shell scripts stopped due to Error.
Message: Script failed. Aborting the bash script to avoid further errors.

Now if we want to get the lines that start with the string "Error", then use the grep command with anchor (^) in the pattern. The complete unix grep command is shown below:

grep "^Error" linux_log_file.dat

Here ^ is used to specify the start of the line in a file. This grep command will displays the lines that start with the word "Error". In this example, the third line starts with the specified pattern (Error).

FIND THE COUNT OF LINES MATCHING THE PATTERN- UNIX /LINUX

Q) How to print the count of number of lines from a file that match the specified pattern in unix or linux operating system?

Let say you are looking for the word "unix" in a file and want to display the count of lines that contain the word "unix". We will see how to find the count with an example.

Assume that i have a file (unix_sample.dat) in my unix operating system. Sample data from the file is shown below:

> cat unix_sample.dat
Monitoring your hosting on unix server is very important.
Otherwise you don’t know whether the unix server is running fine or not.
Use monitoring tools or email alerts to get the status of the unix server.
Send the status, logs in the email to your email Id.

In the sample data, the word "unix" appears in two lines. Now we will print this count on unix terminal using the commands in unix.

1. Using wc command.

We can pipe the output of grep command to wc command to find the number of lines that match a pattern. The unix command is

grep "unix" unix_sample.dat | wc -l

2. Using grep -c option

The grep command has the -c option to find the count of lines that match a pattern. The grep command for this is

grep -c "unix" unix_sample.dat

REMOVE EMPTY LINES FROM A FILE IN UNIX / LINUX

Unix commands can be used to remove the empty (blank) lines from a file. Let see this with the help of an example.

Consider the following data file as an example:

> cat linux_hosting.dat
Hosting a website on linux operating system

helps your site guarding from viruses. Host website on a

dedicated linux server to reduce the load on the server.

This improves the performance and provides good uptime of the site.

The above sample file contains three empty lines. We will see how to remove these blank lines with the help of unix / linux commands.

1. Remove empty lines with Grep command.

The grep command can be used to delete the blank lines from a file. The command is shown below:

grep -v "^$" linux_hosting.dat

Here ^ specifies the start of the line and $ specifies the end of the line. The -v option inverses the match of the grep command.

2. Delete blank lines with Sed command

The sed command can also be used to remove the empty lines from the file. This command is shown below:

sed '/^$/d' linux_hosting.dat

The difference between the sed and grep here is: Sed command removes the empty lines from the file and prints on the unix terminal. Where as grep command matches the non-empty lines and displays on the terminal.

PRINT N LINES AFTER A PATTERN MATCH - UNIX / LINUX

The unix grep command can be used to print the lines from a file that match specified pattern. The grep command has an option for printing the lines around the line that match the pattern. Here we will see how to display N line after a matching line with the help of an example.

Consider the below data file:

> cat linux_enterpise.dat
The advantage of linux operating system is its flexibility.
You can use linux in various systems from mobile phones to space crafts.
You have to choose a best linux operating system.
Things involved in choosing are hardware, scale of software program etc.
Some Os are ubuntu, fedora, linux mint, puppy linux etc.

First of all we will see the general syntax for displaying N lines after the matched line. The syntax of grep command is:

grep -A N "pattern" filename

This grep command will display N lines after the matched line and also prints the matched line. Now we will try to display the 2 lines after the line that contains the word "flexibility". The grep command for this is

grep -A 2 "flexibility" linux_enterpise.dat

This grep command will print first, second and third lines from the above file.

PRINT LINES BEFORE A PATTERN MATCH - UNIX / LINUX

We know how to use the unix grep command to display the lines from a file that match a pattern. However we can use the grep command to display the lines around the line that match the pattern. We will see this with the help of an example.

Consider the below data file which talks about importance of online backup:

> cat online_backup.dat
The most important concern is to keep your documents safe and secure 
in a protected place. There are so many companies which offer online 
backup services. However selecting a good online backup service is 
important. The companies should offer a free trial of backup service. 
Use a backup software to take backup and restore of your data.

First, we will see the general syntax for displaying N lines before the matched line. The syntax of grep command is:

grep -B N "pattern" filename

This grep command will display N lines before the matched line and also prints the matched line. Now we will try to print the 2 lines before the line that contains the word "important". The grep command for this is

grep -B 2 "important" online_backup.dat

This will display second, third and fourth lines from the above file.

GREP STRING IN MULTIPLE FILES - UNIX / LINUX

The Unix Grep command is used to search for a pattern in a line from a file and if it founds the pattern displays the line on the terminal. We can also use the grep command to match for a pattern in multiple files.

We will see this with the help of an example. Let’s consider two files shown below:

> cat webhost_online.dat
There are so many web hosting companies
which provides services for hosting a website

> cat webhost_trail.dat
You can go for trail before choosing a web hosting company.
Once you are happy with the free web hosting trail
then you can host your website there.

To grep for the word "hosting" from these two files specify both the file names as space separated list in grep command. The complete command is

grep hosting webhost_online.dat webhost_trail.dat

The output of the above grep command is

webhost_online.dat:There are so many web hosting companies
webhost_online.dat:which provides services for hosting a website
webhost_trail.dat:You can go for trail before choosing a web hosting company.
webhost_trail.dat:Once you are happy with the free web hosting trail

This will display the filename along with the matching line. Instead of specifying each file name, you can specify a pattern (regular expressions) for the filename. Let say you want to grep for the word "company" in all the files whose name starts with "webhost_", you can use the below grep command:

grep hosting webhost_*

CASE INSENSITIVE GREP COMMAND - UNIX / LINUX

Q) How to make the grep command case in-sensitive and search for a pattern in a file?

Let see how to do this with an example. Consider the below "Car insurance" data file:

> cat car_insurance.dat
Tom bought a new car and confused about car insurance quotes.
He is worrying which car insurance policy he should take.
So tom went to an insurance company and asks for clear explanation of the policies.
The insurance guy then explains about various policies in detail.
Now TOM gets an idea and chooses the right insurance for him.

In the above file, you can see the name "tom" appears in different cases (upper case, lower case and mixed case).

If I want to display the lines that contain the pattern tom with ordinary grep command, it will display only the third line. The grep command is shown below:

grep tom car_insurance.dat

To make this grep command case insensitive use the -i option to the command. Now it will display the first, third and fifth lines from the file. The case in sensitive grep command is

grep -i tom car_insurance.dat

HOW TO SEND MAIL FROM SHELL SCRIPT

We write automated scripts to perform scheduled tasks and put them in crontab. These automated scripts run at their scheduled times. However we don’t know whether the scripts are succeeded or not. So sending an email from automated bash scripts in unix host helps us to know whether the script is succeeded or not.

Here we will see simple bash script to send emails using the mail command in linux operating system.

#!/bin/bash

TO_ADDRESS="recipient@domain.com"
FROM_ADDRESS="sender"
SUBJECT="Mail Server Hosting Demo"
BODY="This is a linux mail system. Linux is one of the email operating systems which can be used to send and receive emails."

echo ${BODY}| mail -s ${SUBJECT} ${TO_ADDRESS} -- -r ${FROM_ADDRESS}

From the name of the variables you can easily understand the significance of each. In the mail command -s represents the subject. Here for the address by default the logged in unix / linux hostname is used as the sent address. For example if you have logged into unix host which is "example.com" and specified the from address as "test". Then your complete from address will be "test@example.com".

In the above bash script we specified the body from a file and did not specified any attachments. We will enhance the above script to attach files, to read body from a file and specifying a list of users in CC. The enhanced mail script is shown below:

#!/bin/bash

TO_ADDRESS="recipient@domain.com"
FROM_ADDRESS="sender"
SUBJECT="linux mail send attachment example"
BODY_FILE="script.dat"
ATTACHMENT_FILE="logfile.txt"
CC_LIST="user1@gmail.com;user2@yahoomail.com;user3@earthlink.com;user4@cheetahmail.com"

uuencode ${ATTACHMENT_FILE} | mail -s ${SUBJECT} -c ${CC_LIST} ${TO_ADDRESS} -- -r ${FROM_ADDRESS} < ${BODY_FILE}

The uuencode is used to attach files using the mail command. Here -c option in mail command is used to specify the list of users in cc list.

CONNECT TO ORACLE DATABASE IN UNIX SHELL SCRIPT

Q) How to connect to oracle database and run sql queries using a unix shell script?

The first thing you have to do to connect to oracle database in unix machine is to install oracle database drivers on the unix box. Once you installed, test whether you are able to connect to the database from command prompt or not. If you are able to connect to the database, then everything is going fine.

Here i am not going to discuss about how to install oracle database drivers. I am just providing the shell script which can be used to connect to the database and run sql statements.

The following Shell script connects to the scott schema of the oracle database and writes the database to the "query.log" file.

#!/bin/bash

LogDirectory='/var/tmp/logs'
DataDirectory='/var/tmp/data'

DBUSER='scott'
DBUSERPASSWORD='tiger'
DB='oracle'

sqlplus -s <<EOF  > ${LogDirectory}/query.log
${DBUSER}/${DBUSERPASSWORD}@${MYDB}
set linesize 32767
set feedback off
set heading off

select * from dual;

EOF

If the sql statements are failed to run, then the errors are written to the same "query.log" file. A better solution is to write the sql statements output to one file and the errors to another file. The below script uses the spooling concept in oracle to write to data to another file:

#!/bin/bash

LogDirectory='/var/tmp/logs'
DataDirectory='/var/tmp/data'

DBUSER='scott'
DBUSERPASSWORD='tiger'
DB='oracle'

sqlplus -s <<EOF  > ${LogDirectory}/query.log
${DBUSER}/${DBUSERPASSWORD}@${MYDB}
set linesize 32767
set feedback off
set heading off

spool ${DataDirectory}/query_output.dat
SELECT * from dual
spool off

EOF

Here the output of the select statement is written to the "query_output.dat" file.

DELETE ALL LINES IN VI / VIM EDITOR - UNIX / LINUX

Q) How to delete all the lines in a file when opened in a VI editor or VIM editor?

Those who are new to unix will use the dd to delete each and every line to empty the file. There is an easy way to delete all the lines in a file when opened in an Editor.

Follow the below steps to empty a file:

Go to command mode in the editor by pressing ESC key on the keyboard.
Press gg. It will take to the first line of the file.
Then press dG. This will delete from the first line to the last line.

See how simple it is to remove all the lines in a file. We will see how to empty the file when not opened in an editor. In unix /dev/null is any empty stream, you can use that to empty a file. The following commands shows how to empty a file

cat /dev/null > file

HOW TO READ LINES USING LOOPS IN SHELL SCRIPTING

Q) How to read each line from a file using loops in bash scripting?

Reading lines from files and then processing on each line is a basic operation in shell scripting. We will see here how to read each line from a file using for and while loop in bash scripting.

Read Line using While Loop:

The below bash script reads line from the file, "logfile.txt", using while loop and prints the line on the terminal:

#!/bin/bash
i=1
while read LINE
do
echo $i $LINE
i=`expr $i+ 1`
done < logfile.txt

Here the variable i is just used to represent the line number.

Read Line using For Loop:

The following shell script reads line using for loop from the file errors.txt:

#!/usr/bin/bash
n=1
for y in `cat errors.txt`
do
echo $n $y
n=`expr $n+ 1`
done

EXAMPLES OF ARRAYS IN AWK COMMAND - UNIX / LINUX

Awk command in unix has rich set of features. One of the feature is it can store the elements in arrays and can process the data in the elements. Here we will see how to use arrays in awk command with examples.

Examples of Arrays in Awk Command:

1. Finding the sum of values

I want to find the sum of values in the first column of all the lines and display it on the unix or linux terminal. Let say my file has the below data:

> cat file.txt
10
20
30

After summing up all the values, the output should be 60. The awk command to sum the values without using the arrays is shown below:

awk 'BEGIN {sum=0} {sum=sum+$1} END {print sum}' text

Here i have used a variable to store the sum of values. At the end after summing up all the values, the sum is printed on the terminal.

The awk command to find the sum of values by using arrays is shown below:

awk '{arr[NR]=arr[NR-1]+$1} END {print arr[NR]}' text

Here an array is used to store the sum of values. Basically this array will store the cumulative sum of values, at the end it contains the total and it is displayed on the terminal.

2. Ranking values in a file.

Let say I have a source file which contains the employees data. This file has three fields first field is department_id, second one is employee name and third one is salary. Sample data from the file is shown below:

> cat emp.dat
10, AAA, 6000
10, BBB, 8000
10, CCC, 6000
20, DDD, 4000
20, EEE, 2000
20, FFF, 7000

Now i want to assign ranks to the employees in each department based on their salary. The output should look as

20, FFF, 7000, 1
20, DDD, 4000, 2
20, EEE, 2000, 3
10, BBB, 8000, 1
10, AAA, 6000, 2
10, CCC, 6000, 2

Here Employees AAA and CCC got same rank as their salaries are same.

To solve this problem first we have to sort the data and then pipe it to awk command. The complete command is shown below:

sort -nr -k1 -k3 text | 

awk -F"," '{  
department_array[NR]=$1; 
salary_array[NR]=$3; 
if (department_array[NR] != department_array[NR-1]) 
{
  rank_array[NR]=1;
} 
else if (salary_array[NR] == salary_array[NR-1] )
{
  rank_array[NR] =rank_array[NR-1];
} 
else 
{
  rank_array[NR] = rank_array[NR-1]+1;
} 
print department_array[NR]","$2","salary_array[NR]","rank_array[NR];
}'

For readability purpose the above command is written in multiple lines. You have to write the above command in single line to make it work in unix.

CREATE TUNNEL IN UNIX USING PUTTY

Q) How to create tunnel to access network resources (Internet) in a remote unix machine using putty client?

You might have faced situations where you want to open a website from your browser and the website URL is blocked by your company. Especially this happens in software companies. Here i will show you how to open a blocked website by creating a tunnel.

The software’s required to create a tunnel are:

Putty client tool
Mozilla firefox browser
Access to remote unix server. This server should be capable of opening any website.

Creating Tunnle in Unix

Follow the below steps to create a tunnel

Open the putty client tool. Enter the remote unix hostname in the "Host Name (or IP address)". In this demo i have entered the hostname as "example.hostname.com".
To save this hostname, enter a name like "Tunnel" in the "Saved Sessions" place. This is shown in the below image:

On the left side of the client, you can see a navigation panel. Go to SSH-> Tunnels.
Again enter the remote hostname (example.hostname.com) in "Destination" section.
Enter the source port as 1100 (any value you prefer) and check the Dynamic option. This is shown below:

Now click on Add. Go back to the previous window by clicking on the Session in the left side pan. Here clik on save. I will save your tunnel details.
Open this tunnel and enter your remote machine login details. Do not close this unix session. If you close it, your tunneling won’t work.
Open the Mozilla fire fox browser. Go to Tools->Options->Advanced->Network->Settings.
In the settings, Check the manual proxy configurations, enter the Socket host as localhost and port as 1100 (Same port which is specified in tunnel configuration) and click on Ok. This is shown in below image.

Now you can open any website with this approach provided your remote host has access.

VIM EDITOR - SAVE AND QUIT OPTIONS

VIM is a powerful editor in unix or linux. The VIM editor got so many features. Here we will see the options for saving and quitting from the vim editor.

The following options works in command mode of VIM editor. To go to the command mode press ESC key on the keyboard and then type the below commands:

:w ->Saves the contents of the file without exiting from the VIM editor
:wq ->Saves the text in the file and then exits from the editor
:w filename -> Saves the contents of the opened file in the specified filename. However it won’t save the contents of the current file.
:x -> Saves changes to the current file and then exits. Similar to the :wq
:m,nw filename -> Here m and n are numbers. This option will write the lines from the specified numbers m and n to the mentioned filename.
:q -> Exits from the current file only if you did not do any changes to the file.
:q! -> Exits from the current file and ignores any changes that you made to the file.

DIRNAME COMMAND EXAMPLES IN UNIX / LINUX

The unix dirname command strips non-directory suffix from a file name.

The syntax of dirname command is

dirname NAME

The dirname command removes the trailing / component from the NAME and prints the remaining portion. If the NAME does not contain / component then it prints '.' (means current directory).

Dirname command is useful when dealing with directory paths in unix or linux operating systems. Some examples on dirname command are shown below:

Dirname Command Examples:

1. Remove the file name from absolute path.

Let say my directory path is /usr/local/bin/add.sh. Now i want to remove /add.sh and display only /usr/local/bin, then we can use the dirname command.

> dirname /usr/local/bin/add.sh
/usr/local/bin

2. dirname sum.pl

Here you can see that the NAME does not contain the / component. In this case the dirname produces '.' as the output.

> dirname sum.pl
.

Note: The directories and filename which i have passed as arguments to dirname command in the above examples are just strings. There is no need of these directories or files to exist in the unix machine.

SPLIT COMMAND EXAMPLES IN UNIX / LINUX

The Split command in unix or linux operating system splits a file into many pieces (multiple files). We can split a file based on the number of lines or bytes. We will see how to use the split command with an example.

As an example, let’s take the below text file as the source file which we want to split:

> cat textfile
unix linux os
windows mac os
linux environment

There are three lines in that file and the size of the file is 47 bytes.

Split Command Examples:

1. Splitting file on number of lines.

The Split command has an option -l to split the file based on the number of lines. Let say i want to split the text file with number of lines in each file as 2. The split command for this is

split -l2 textfile

The new files created are xaa and xab. Always the newly created (partitioned) file names start with x. We will see the contents of these files by doing a cat operation.

> cat xaa
unix linux os
windows mac os

> cat xab
linux environment

As there only three lines in the source file we got only one line in the last created file.

2. Splitting file on the number of bytes

We can use the -b option to specify the number of bytes that each partitioned file should contains. As an example we will split the source files on 10 bytes as

split -b10 textfile

The files created are xaa, xab, xac, xad, xae. The first four files contain 10 bytes and the last file contains 7 bytes as the source file size is 47 bytes.

3. Changing the newly created file names from character sequences to numeric sequences.

So far we have seen that the newly created file names are created in character sequences like xaa, Xab and so on. We can change this to numeric sequence by using the -d option as

split -l2 -d textfile

The names of the new files created are x00 and x01.

4. Changing the number of digits in the sequence of filenames.

In the above example, you can observe that the sequences have two digits (00 and 01) in the file names. You can change the number of digits in the sequence by using the -a option as

split -l2 -d -a3 textfile

Now the files created are x000 and x001

SWAP FIELDS (STRINGS) IN A FILE - UNIX / LINUX

First we will see how to swap two strings in a line and then we will see how to swap two columns in a file.

As an example, consider the text file with below data:

unix linux os
windows mac os

Swap Strings using Sed command:

Let see how to swap the words unix and linux using sed command in unix or linux environment. The sed command to swap the strings is shown below:

> sed 's/\(unix\) \(linux\)/\2 \1/' textfile
linux unix os
windows mac os

The parentheses are used to remember the pattern. \1 indicates first pattern and \2 indicates second pattern.

Swap Fields using Awk command:

From the above file structure, we can observe that the file is in format of rows and columns where the columns are delimited by space.

Awk command can be used to process delimited files. Awk command to swap the first two fields in a file is

> awk '{$0=$2." "$1" "$3; print $0}' textfile
linux unix os
mac windows os

Another way using awk is

awk '{print $2" "$1" "$3}' textfile

HOW TO REPLACE BRACES SYMBOLS IN UNIX / LINUX

Q) My log file contains the braces symbols '(' and ')'. I would like to replace the braces with empty string. Sample data in the log file is shown below:

> cat logfile
Error - (unix script failed)

The output should not contain the braces and the data should look as

Error - unix script failed

How can i achieve this using unix or linux commands?

Solution:

1. Replacing using tr command

We can use the tr command to delete characters in a file. The deleting of strings using tr command is shown below:

tr -d '()' < logfile

2. Replacing using sed command Sed command is popularly used for replacing the text in a file with another text. The sed command is

sed 's/[()]//g' logfile

Another way of replacing is using sed with pipes in unix:

sed 's/(//' logfile| sed 's/)//'

SPLIT FILE DATA INTO MULTIPLE FILES - UNIX / LINUX

Q) I have file with 10000 lines in unix or linux operating system. I want to split this file and create 10 files such that each file has 1000 lines. What I mean is the first 100 lines should go into one file; next 100 lines should go into another file and so on. How to do this using unix commands.

Solution:

Unix has the split command which can be used to partition the data in a file into multiple files. The command to split a file based on the number of lines is shown below:

split -l 1000 filename

The above split command splits the file such that each file has 1000 lines. Here the option l indicates the number of lines. You can split the file based on number of bytes using the -b option.

split -b 1024 filename

By default, the partitioned filenames starts with x like xab, xac, xad and so on. Instead of alphabetical sequences, you can use numeric sequences in filenames like x01, x02 using the -d option.

split -l 1000 -d filename

You can specify the number of digits to be used in the numeric sequences with the help of -a option.

split -l 1000 -d -a 3 filename

Examples: Let say i have a text file with 4 lines. The data in the file is shown below:

> cat textfile
unix is os
linux environment
centos
red hat linux

We will run the split command for each of the points discussed above and see what files will be created.

> split -l 2 textfile
Files: xaa, xab

> split -b 10 textfile
Files: xaa, xab, xac, xad, xae

> split -l 2 -d textfile
Files: x00, x01

> split -l 2 -d -a 3 textfile
Files: x000, x001

REMOVE LAST CHARACTER IN STRING - UNIX / LINUX

Q) I have a file with bunch of lines. I want to remove the last character in each line from that file. How can i achieve this in unix or linux environment.

Solution:

1. SED command to remove last character

You can use the sed command to delete the last character from a text. The sed command is

sed s/.$// filename

2. Bash script

The below bash script can be used to remove the last character in a file.

#! /bin/bash
while read LINE
do
echo ${LINE%?}
done < filename

3. Using Awk command We can use the built-in functions length and substr of awk command to delete the last character in a text.

awk '{$0=substr($0,1,length($0)-1); print $0}' filename

4. Using rev and cut command We can use the combination of reverse and cut command to remove the last character. The command is shown below:

rev products.txt | cut -c2- |rev

CONVERT MULTIPLE ROWS INTO SINGLE ROW - UNIX/LINUX

Q) I have a products data in the text file. The data in the file look as shown below:

> cat product.txt
iphone
samsung
nokia
yahoo
google
aol
amazon
ebay
walmart

Now my requirement is to group each 3 consecutive rows into a single row and produce a comma separated list of products. The output should look as

iphone,samsung,nokia
yahoo,google,aol
amazon,ebay,walmart

I want this to be implemented using unix or linux commands in different ways?

Solution:

1. One way we can implement this is using the awk command. The complete awk command is shown below:

awk '{printf("%s%s",$0,NR%3?",":"\n")}' products.txt

2. Another way is using the paste command. The solution using the paste command is shown below.

paste -d, - - - < products.txt

AWK COMMAND TO SPLIT COLUMN INTO ROW - UNIX/LINUX

Awk command to split list data in a column into multiple rows - Unix/Linux

Q) I have a flat file in the unix or linux environment. The data in the flat file looks as below

> cat students.txt
Mark Maths,Physics,Chemistry
Chris Social
Henry Biology, Science

The flat file contains the list of subjects that were taken by the students in their curriculum. I want the subjects list in each column to be splitted into multiple rows. After splitting the data in the target should look as:

Mark Maths
Mark Physics
Mark Chemistry
Chris Social
Henry Science
Henry Science

Write a command in UNIX or LINUx operating system to produce the result?

Solution:

We can use the AWK command which can process the files with table like structures. The solution to the problem using Awk command is

awk '{n=split($2,s,",");for (i=1;i<=n;i++) {$2=s[i];print}}' students.txt

SEARCH AND GREP FOR TEXT IN UNIX/LINUX

We will see how to search for files and then grep for a string of text in those files. First i will use the find command in unix or linux to search for the regular files in the current directory. The grep command to search for the normal files in the current directory is shown below:

> find . -type f
./docs/sum.pl
./add.sh
./sample

Now we will grep for a particular word in these files and display only the filenames that has the matching word. The unix command is shown below:

> find . -type f -exec grep -l word {} \;

The above command just displays the filenames that has the specified word. Now we will try to display the lines from the files that have the matching word. The unix command for this is:

> find . -type f -exec grep -l word {} \; -exec grep word {} \;

If you want to put space between the results of the above command, display the line using echo. The complete unix command is

> find . -type f -exec grep -l word {} \; -exec grep word {} \; -exec echo \;

The above example shows how to use multiple grep’s with the find command in unix or linux.

JOIN COMMAND IN UNIX/LINUX EXAMPLES

Join command is one of the text processing utility in Unix/Linux. Join command is used to combine two files based on a matching fields in the files. If you know SQL, the join command is similar to joining two tables in a database.

The syntax of join command is

join [options] file1 file2

The join command options are

-1 field number : Join on the specified field number in the first file
-2 field number : Join on the specified field number in the second file
-j field number : Equivalent to -1 fieldnumber and -2 fieldnumber
-o list : displays only the specified fields from both the files
-t char : input and output field delimiter
-a filenumber : Prints non matched lines in a file
-i : ignore case while joining

Unix Join Command Examples

1. Write a join command to join two files on the first field?

The basic usage of join command is to join two files on the first field. By default the join command matches the files on the first fields when we do not specify the field numbers explicitly. Let's say we have two files emp.txt and dept.txt

> cat emp.txt
10 mark
10 steve
20 scott
30 chris
> cat dept.txt
10 hr
20 finance
30 db

Here we will join on the first field and see the output. By default, the join command treats the field delimiter as space or tab.

> join emp.txt dept.txt
10 mark hr
10 steve hr
20 scott finance
30 chris db

Important Note: Before joining the files, make sure to sort the fields on the joining fields. Otherwise you will get incorrect result.

2. Write a join command to join the two files? Here use the second field from the first file and the first field from the second file to join.

In this example, we will see how to join two files on different fields rather than the first field. For this consider the below two files as an example

> cat emp.txt
mark 10 1
steve 10 1
scott 20 2
chris 30 3
> cat dept.txt
10 hr 1
20 finance 2
30 db 3

From the above, you can see the join fields are the second field from the emp.txt and the first field from the dept.txt. The join command to match these two files is

> join -1 2 -2 1 emp.txt dept.txt
10 mark 1 hr 1
10 steve 1 hr 1
20 scott 2 finance 2
30 chris 3 db 3

You can also see that the two files can also be joined on the third filed. As the both the files have the matching join field, you can use the j option in the join command.

Here -1 2 specifies the second field from the first file (emp.txt) and -2 1 specifies the first field from the second file (dept.txt)

> join -j 3 emp.txt dept.txt
1 mark 10 10 hr
1 steve 10 10 hr
2 scott 20 20 finance
3 chris 30 30 db

3. Write a join command to select the required fields from the input files in the output? Select first filed from first file and second field from second file in the output.

By default, the join command prints all the fields from both the files (except the join field is printed once). We can choose what fields to be printed on the terminal with the -o option. We will use the same files from the above example.

> join -o 1.1 2.2 -1 2 -2 1 emp.txt dept.txt
mark hr
steve hr
scott finance
chris db

Here 1.1 means in the first file select the first field. Similarly, 2.2 means in the second file select the second field

4. Write a command to join two delimited files? Here the delimiter is colon (:)

So far we have joined files with space delimiter. Here we will see how to join files with a colon as delimiter. Consider the below two files.

> cat emp.txt
mark:10
steve:10
scott:20
chris:30
> cat dept.txt
10:hr
20:finance
30:db

The -t option is used to specify the delimiter. The join command for joining the files is

> join -t: -1 2 -2 1 emp.txt dept.txt
10:mark:hr
10:steve:hr
20:scott:finance
30:chris:db

5. Write a command to ignore case when joining the files?

If the join fields are in different cases, then the join will not be performed properly. To ignore the case in join use the -i option.

> cat emp.txt
mark,A
steve,a
scott,b
chris,C
> cat dept.txt
a,hr
B,finance
c,db

> join -t, -i -1 2 -2 1 emp.txt dept.txt
A,mark,hr
a,steve,hr
b,scott,finance
C,chris,db

6. Write a join command to print the lines which do not match the values in joining fields?

By default the join command prints only the matched lines from both the files which means prints the matched lines that passed the join condition. We can use the -a option to print the non-matched lines.

> cat P.txt
A 1
B 2
C 3
> cat Q.txt
B 2
C 3
D 4

Print non pairable lines from first file.

> join -a 1 P.txt Q.txt
A 1
B 2 2
C 3 3

Print non pairable lines from second file.

> join -a 2 P.txt Q.txt
B 2 2
C 3 3
D 4

Print non pairable lines from both file.

> join -a 1 -a 2 P.txt Q.txt
A 1
B 2 2
C 3 3
D 4

MOVE / RENAME FILES, DIRECTORY - MV COMMAND IN UNIX / LINUX

Q. How to rename a file or directory in unix (or linux) and how to move a file or directory from the current directory to another directory?

Unix provides a simple mv (move) command which can be used to rename or move files and directories. The syntax of mv command is

mv [options] oldname newname

The options of mv command are

f : Do not prompt before overwriting a file. 
i : Prompts for the user input before overwriting a file.

If the newname already exists, then the mv command overwrites that file. Let see some examples on how to use mv command.

Unix mv command examples

1. Write a unix/linux command to rename a file?

Renaming a file is one of the basic features of the mv command. To rename a file from "log.dat" to "bad.dat", use the below mv command

> mv log.dat bad.dat

Note that if the "bad.dat" file already exists, then its contents will be overwritten by "log.dat". To avoid this use the -i option, which prompts you before overwriting the file.

mv -i log.dat bad.dat
mv: overwrite `bad.dat'?

2. Write a unix/linux command to rename a directory?

Just as renaming a file, you can use the mv command to rename a directory. To rename the directory from docs to documents, run the below command

mv docs/ documents/

If the documents directory already exists, then the docs directory will be moved in to the documents directory.

3. Write a unix/linux command to move a file into another directory?

The mv command can also be used to move the file from one directory to another directory. The below command moves the sum.pl file in the current directory to /var/tmp directory.

mv sum.pl /var/tmp/

If the sum.pl file already exists in the /var/tmp directory, then the contents of that file will be overwritten.

4. Write a unix/linux command to move a directory in to another directory?

Just as moving a file, you can move a directory into another directory. The below mv command moves the documents directory into the tmp directory

mv documents /tmp/

5. Write a unix/linux command to move all the files in the current directory to another directory?

You can use the regular expression pattern * to move all the files from one directory to another directory.

mv * /var/tmp/

The above command moves all the files and directories in the current directory to the /var/tmp/ directory.

6. mv *

What happens if you simply type mv * and then press enter?

It depends on the files you have in the directory. The * expands to all the files and directories. Three scenarios are possible.

If the current directory has only files, then the contents of all the files (except one file) will be written in to the one file. The one file is the last file which depends on the pattern *.
If the current directory contains only directories, then all the directories (except one directory) will be moved to another directory.
If the current directory contains both files and directories, then it depends on the expansion of the *. If the pattern * gives the last one as directory then all the files will be moved to that directory. Otherwise the mv command will fail.

Some Tips:

Try to avoid mv *
Avoid moving large number of files.

SED -I COMMAND EXAMPLES IN UNIX AND LINUX

Sed is great tool for replacing the text in a file. sed is a stream editor which means edit the file as a stream of characters. To replace a text using the unix sed command, you have to pass the search string and replacement string. By default the sed command does not edit the file and displays the output on the terminal.

We will see the usage of -i command with an example

Consider the below text file with data

> cat file.txt
linux sed command tutorial

We will replace the word "tutorial" with "example" in the file using the sed command.

> sed 's/tutorial/example/' file.txt
linux sed command example

> cat file.txt
linux sed command tutorial

The sed command replaced the text in the file and displayed the result on the terminal. However it did not changed the contents of the file. You can redirect the output of sed command and save it in a file as

> sed 's/tutorial/example/' file.txt > new_file.txt

The -i option comes in handy to edit the original file itself. If you use the -i option the sed command replaces the text in the original file itself rather than displaying it on the terminal.

> sed -i 's/tutorial/example/' file.txt
> cat file.txt
linux sed command example

Be careful in using the -i option. Once you changed the contents of the file, you cannot revert back to the original file. It is good to take the backup of the original file. You can provide a suffix to the -i option for taking the backup of the file. Now we will replace the "example" with "tutorial" and at the same time will take the backup of the file.

> sed -i_bkp 's/example/tutorial/' file.txt

> ls file.txt*
file.txt  file.txt_bkp

> cat file.txt_bkp
linux sed command example

> cat file.txt
linux sed command tutorial

See the backup file created with the contents of the original file.

DELETE DIRECTORY, FILES - RM, RMDIR COMMAND IN UNIX / LINUX

Q. How to delete directories and files in unix/linux

Unix provides rmdir and rm commands to remove the directories and files. Let see each command in detail.

Unix rmdir command syntax

The syntax of rmdir command is

rmdir [options] directories

The rmdir command options are

-p : Removes directory and its parent directories
-v : Provides the diagnostic information of the directory processed

Unix rmdir command examples

1. Write a unix/linux command to remove a directory?

The rmdir command deletes only the empty directories. If a directory contains files or sub directories, then the rmdir command fails.

rmdir docs/
rmdir: docs/: Directory not empty

Here the docs directory is not empty, that is why the rmdir command failed to remove the directory. To remove the docs directory first we have to make the directory empty and then delete the directory.

rm doc/*
rmdir docs/

We will see later how to remove non-empty directories with a single command.

2. Write a unix/linux command to remove the directory and its parent directories?

As mentioned earlier the -p option allows the rmdir command to delete the directory and also its parent directories.

rmdir -p docs/entertainment/movies/

This rmdir command removes the docs directory completely. If you don’t use the -p option, then it only deletes the movies directory.

3. Write a unix/linux command to remove directories using pattern matching?

You can specify the directory names using the regular expressions and can delete them.

rm doc*

This rm command deletes the directories like doc, documents, doc_1 etc.

Now we will see the rm command in unix.

Unix rm command syntax

The syntax of rm command is

rm [options] [directory|file]

The rm command options are

f : Removes all files in a directory without prompting the user.
i : Interactive: prompts the user for confirmation before deleting a file.
R or r : Recursively remove directories and sub directories.

The rm command can be used to delete both the files and directories. The rm command also deletes the non-empty directories.

Unix rm command examples

1. Write a unix/linux command to remove a file?

This is the basic feature of rm command. To remove a file, logfile.dat, in the current directory use the below rm command

rm logfile.dat

2. Write a unix/linux command to remove all the files in a directory?

use the * regular pattern as the file list in rm command for deleting all the files in the current directory.

rm *

3. Write a unix/linux command to delete empty directory?

The rm command can also be used to delete the empty directory. The command for this is

rm docs/

If the directory is non-empty, then the above command fails to remove the directories.

4. Write a unix/linux command to delete directories recursively (delete non empty directories)?

As mentioned earlier, the -r option can be used to remove the directories and sub directories.

rm -r docs

This removes the docs directory even if it is non-empty.

CUT COMMAND IN UNIX ( LINUX) EXAMPLES

Cut command in unix (or linux) is used to select sections of text from each line of files. You can use the cut command to select fields or columns from a line by specifying a delimiter or you can select a portion of text by specifying the range or characters. Basically the cut command slices a line and extracts the text.

Unix Cut Command Example

We will see the usage of cut command by considering the below text file as an example

> cat file.txt
unix or linux os
is unix good os
is linux good os

1. Write a unix/linux cut command to print characters by position?

The cut command can be used to print characters in a line by specifying the position of the characters. To print the characters in a line, use the -c option in cut command

cut -c4 file.txt
x
u
l

The above cut command prints the fourth character in each line of the file. You can print more than one character at a time by specifying the character positions in a comma separated list as shown in the below example

cut -c4,6 file.txt
xo
ui
ln

This command prints the fourth and sixth character in each line.

2.Write a unix/linux cut command to print characters by range?

You can print a range of characters in a line by specifying the start and end position of the characters.

cut -c4-7 file.txt
x or
unix
linu

The above cut command prints the characters from fourth position to the seventh position in each line. To print the first six characters in a line, omit the start position and specify only the end position.

cut -c-6 file.txt
unix o
is uni
is lin

To print the characters from tenth position to the end, specify only the start position and omit the end position.

cut -c10- file.txt
inux os
ood os
good os

If you omit the start and end positions, then the cut command prints the entire line.

cut -c- file.txt

3.Write a unix/linux cut command to print the fields using the delimiter?

You can use the cut command just as awk command to extract the fields in a file using a delimiter. The -d option in cut command can be used to specify the delimiter and -f option is used to specify the field position.

cut -d' ' -f2 file.txt
or
unix
linux

This command prints the second field in each line by treating the space as delimiter. You can print more than one field by specifying the position of the fields in a comma delimited list.

cut -d' ' -f2,3 file.txt
or linux
unix good
linux good

The above command prints the second and third field in each line.

Note: If the delimiter you specified is not exists in the line, then the cut command prints the entire line. To suppress these lines use the -s option in cut command.

4. Write a unix/linux cut command to display range of fields?

You can print a range of fields by specifying the start and end position.

cut -d' ' -f1-3 file.txt

The above command prints the first, second and third fields. To print the first three fields, you can ignore the start position and specify only the end position.

cut -d' ' -f-3 file.txt

To print the fields from second fields to last field, you can omit the last field position.

cut -d' ' -f2- file.txt

5. Write a unix/linux cut command to display the first field from /etc/passwd file?

The /etc/passwd is a delimited file and the delimiter is a colon (:). The cut command to display the first field in /etc/passwd file is

cut -d':' -f1 /etc/passwd

6. The input file contains the below text

> cat filenames.txt
logfile.dat
sum.pl
add_int.sh

Using the cut command extract the portion after the dot.

First reverse the text in each line and then apply the command on it.

rev filenames.txt | cut -d'.' -f1

DELETE EMPTY LINES USING SED / GREP COMMAND IN UNIX (OR LINUX)

In Unix / Linux you can use the Sed / Grep command to remove empty lines from a file. For example, Consider the below text file as input

> cat file.txt
Remove line using unix grep command

Delete lines using unix sed command

How it works

Now we will see how to remove the lines from the above file in unix / linux

1. Remove lines using unix sed command

The d command in sed can be used to delete the empty lines in a file.

sed '/^$/d' file.txt

Here the ^ specifies the start of the line and $ specifies the end of the line. You can redirect the output of above command and write it into a new file.

sed '/^$/d' file.txt > no_empty_lines.txt

2. Delete lines using unix grep command

First we will see how to search for empty lines using grep command.

grep '^$' file.txt

Now we will use the -v option to the grep command to reverse the pattern matching

grep -v '^$' file.txt

The output of both sed and grep commands after deleting the empty lines from the file is

Remove line using unix grep command
Delete lines using unix sed command
How it works

CHANGE DIRECTORY (CD) EXAMPLES | UNIX AND LINUX COMMAND

The Change directory (cd) command is one of the simple commands in Unix (or Linux) and it is very easy to use. The cd command is used to change from the current directory to another directory. The syntax of cd command is

cd [directory]
Here directory is the name of the directory where you wish to go.

CD Command Examples

1. Write a unix/linux cd command to change to home directory?

Just simply type cd command on the unix terminal and then press the enter key. This will change your directory to home directory.

> pwd
/usr/local/bin

Now i am in the /usr/local/bin directory. After typing the cd command and unix window, you will go to your home directory.

> cd
> pwd
/home/matt

Here pwd command displays the present working directory.

2. Write a unix/linux cd command to go back to one directory?

The cd .. changes the directory to its parent directory by going back one level. The space between the cd and .. is must.

> pwd
/var/tmp
> cd ..
> pwd
/var

3. Write a unix/linux cd command to go back to two directories?

The cd ../../ takes you back to two directories. You can extend this cd command to go back to n number of directories.

> pwd
/usr/local/bin
> cd ../../
> pwd
/usr

4. Write a unix/linux cd command to change the directory using the absolute path?

In case of changing directory using absolute path you have to specify the full directory path. Absolute path directories always start with a slash (/). An example is changing your directory to /usr/bin from your home directory.

> cd /usr/bin

5. Write a unix/linux cd command to change the directory using the relative path?

In relative path, you have to specify the directory path relative to your current directory. For example, you are in /var/tmp directory and you want to go to /var/lib directory, then you can use the relative path.

> pwd
/var/tmp
> cd ../lib
> pwd
/var/lib

Here the cd ../lib, first takes you to the parent directory which is /var and then changes the directory to the lib.

6. Write a unix/linux cd command to change back to previous directory.

As an example, i am in the directory /home/matt/documents and i changed to a new directory /home/matt/backup. Now i want to go back to my previous directory /home/matt/documents. In this case, you can use the cd - command to go back to the previous directory.

> pwd
/home/matt/documents
> cd /home/matt/backup
>pwd
/home/matt/backup
> cd -
> pwd
/home/matt/documents

UNIX TIMESTAMP COMMAND

What is Unix Timestamp

Unix timestamp is the representation of time as the running total of number of seconds since the unix epoch time on January 1st, 1970. Simply the Unix timestamp is the number of seconds between the particular date and the Unix Epoch.

The unix timestamp become standard in computer systems for tracking the information especially in distributed processing system like hadoop, cloud computing etc.

Here we will see how to convert the unix date to timestamp and unix timestamp to date. We will also see how to generate the unix current timestamp. Let see each one:

1. Unix Current Timestamp

To find the unix current timestamp use the %s option in the date command. The %s option calculates unix timestamp by finding the number of seconds between the current date and unix epoch.

date '+%s'
1327312578

You will get a different output if you run the above date command.

2. Convert Unix Timestamp to Date

You can use the -d option to the date command for converting the unix timestamp to date. Here you have to specify the unix epoch and the timestamp in seconds.

date -d "1970-01-01 956684800 sec GMT"
Tue Apr 25 10:46:40 PDT 2000

3. Convert Unix Date to Timestamp

You have to combine the -d option and the %s option for converting the unix date to timestamp.

date -d "2000-01-01 GMT" '+%s'
946684800

COPY (CP) FILE AND DIRECTORY EXAMPLES | UNIX AND LINUX COMMAND

Copy (cp) is the frequently used command in Unix (or Linux). The cp Command is used to copy the files from one directory to another directory. The cp command can also be used to copy the directories also. The syntax of cp command is

cp [options] source destination

Examples of cp Command

1. Write a unix/linux cp command to copy file in to a directory?

The basic usage of cp command is to copy a file from the current directory to another directory.

cp sum.pl tmp/

The cp command copies the file sum.pl into the tmp directory. The cp command does not remove the source file. It just copies the file into a new location. If a file with the same name as the source exists in the destination location, then by default the cp command overwrites that new file

2. Write a unix/linux cp to prompt for user before overwriting a file ( Interactive cp command)?

The -i option to the cp command provides the ability to prompt for a user input whether to overwrite the destination file or not.

> cp sum.pl tmp/
cp: overwrite `tmp/sum.pl'?

If you enter y, then the cp command overwrites the destination file, otherwise the cp command does not copy the file.

3. Write a unix/linux cp command to copy multiple files in to a new directory?

You can specify multiple files as the source and can copy to the new location.

cp log.dat bad.dat tmp/

The cp command copies the log.dat, bad.dat files in the current directory to the tmp directory.

4. Write a unix/linux cp command to do a Regular expression copy?

You can copy a set of files by specifying a regular expression pattern.

cp *.dat tmp/

Here the cp command copies all the files which has "dat" as suffix to the destination directory.

5. Write a unix/linux cp command to copy a file in to the current directory?

You can copy a file from a different directory to the current directory.

cp /usr/local/bin/multiply.sh .

Here the cp command copies the multiply.sh file in the /usr/local/bin directory the current directory. The dot (.) indicates the current directory.

6. Write a unix/linux cp command to copy all the files in a directory?

The cp command can be used to copy all the files in directory to another directory.

cp docs/* tmp/

This command copies all the files in the docs directory to the tmp directory.

7. Write a unix/linux cp command to copy files from multiple directories?

You can copy the files from different directories into a new location.

cp docs/* scripts/* tmp/

The command copies the files from docs and script directories to the destination directory tmp.

8. Write a unix/linux cp command to Copy a directory.

You can recursively copy a complete directory and its sub directory to another location using the cp command

cp -r docs tmp/

This copies the complete directory docs into the new directory tmp

9. Write a unix/linux cp command to Forcibly copy a file with -f option?

You can force the cp command to copy an existing destination file even it cannot be opened.

cp -f force_file.txt /var/tmp/

LS COMMAND IN UNIX AND LINUX EXAMPLES

ls is the most widely used command in unix or linux. ls command is used to list the contents of a directory. Learn the power of ls command to make your life easy. The syntax of ls command is

ls [options] [pathnames]

1. Write a unix/linux ls command to display the hidden files and directories?

To display the hidden files and directories in the current directory use the -a option of the ls command.

> ls -a
.  ..  documents  .hidden_file  sum.pl

Hidden files are the one whose name starts with dot (.). The las -a displays the current directory (.) and parent directory (..) also. If you want to exclude the current directory, parent directory, then use -A option.

> ls -A
documents  .hidden_file  sum.pl

2. Write a unix/linux ls command to classify the files with special characters

The -F option to ls command classifies the files. It marks the

Directories with trailing slash (/)
Executable files with trailing asterisk (*)
FIFOs with trailing vertical bar (|)
Symbolic links with trailing at the rate sign (@)
Regular files with nothing

> ls -F
documents/  sum.pl link@

3. Write a unix/linux ls command to print each file in a separate line?

The -1 option to the ls command specifies that each file should be displayed on a separate line

> ls -1
documents
sum.pl

4. Write a unix/linux ls command to display the inode number of file?

In some cases, you want to know the inode number of a file. Use -i option to the ls command to print the inode number of a file.

> ls -i1
10584066 documents
3482450 sum.pl

5. Write a unix/linux ls command to display complete information about the files?

The -l option provides lots of information about the file type, owner, group, permissions, file size, last modification date.

> ls -l
total 16
drwxr-xr-x 2 matt db 4096 Jan 30 23:08 documents
-rw-r--r-- 1 matt db   49 Jan 31 01:17 sum.pl

The first character indicates the type of the file. - for normal file, d for directory, l for link file and s for socket file
The next 9 characters in the first field represent the permissions. Each 3 characters refers the read (r), write (w), execute (x) permissions on owner, group and others. - means no permission.
The second field indicates the number of links to that file.
The third field indicates the owner name.
The fourth field indicates the group name.
The fifth field represents the file size in bytes.
The sixth field represents the last modification date and time of the file.
And finally the seventh field is the name of the file.

6. Write a unix/linux ls command to sort the files by their modification time?

The -t option allows the ls command to sort the files in descending order based on the modification time.

> ls -t1
sum.pl
documents

7. Write a unix/linux ls command to sort the files in ascending order of modification time?

The -r option reverses the order of the files displayed. Combine the -t and -r options to sort the files in ascending order.

> ls -rt1
documents
sum.pl

8. Write a unix/linux ls command to print the files recursively?

So far the ls command prints the files in the current directory. Use the -R option to recursively print the files in the sub-directories also.

> ls -R
.:
documents  sum.pl

./documents:
file.txt

9. Write a unix/linux ls command to print the files in a specific directory?

You can pass a directory to the ls command as an argument to print for the files in it.

> ls /usr/local/bin

10. Write a unix/linux ls command to display files in columns?

The -x option specifies the ls command to display the files in columns.

> ls -x

SED COMMAND IN UNIX AND LINUX EXAMPLES

>cat file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.

Sed Command Examples

1. Replacing or substituting string

Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.

>sed 's/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

>sed 's/unix/linux/2' file.txt
unix is great os. linux is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.

3. Replacing all the occurrence of the pattern in a line.

The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.

>sed 's/unix/linux/g' file.txt
linux is great os. linux is opensource. linux is free os.
learn operating system.
linuxlinux which one you choose.

>sed 's/unix/linux/3g' file.txt
unix is great os. unix is opensource. linux is free os.
learn operating system.
unixlinux which one you choose.

5. Changing the slash (/) delimiter

You can use any delimiter other than the slash. As an example if you want to change the web url to another url as

>sed 's/http:\/\//www/' file.txt

>sed 's_http://_www_' file.txt
>sed 's|http://|www|' file.txt

>sed 's/unix/{&}/' file.txt
{unix} is great os. unix is opensource. unix is free os.
learn operating system.
{unix}linux which one you choose.

>sed 's/unix/{&&}/' file.txt
{unixunix} is great os. unix is opensource. unix is free os.
learn operating system.
{unixunix}linux which one you choose.

>sed 's/\(unix\)/\1\1/' file.txt
unixunix is great os. unix is opensource. unix is free os.
learn operating system.
unixunixlinux which one you choose.

The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is

>sed 's/\(unix\)\(linux\)/\2\1/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxunix which one you choose.

Another example is switching the first three characters in a line

>sed 's/^\(.\)\(.\)\(.\)/\3\2\1/' file.txt
inux is great os. unix is opensource. unix is free os.
aelrn operating system.
inuxlinux which one you choose.

>sed 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
linuxlinux which one you choose.

>sed -n 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linuxlinux which one you choose.

>sed 's/unix/linux/' file.txt| sed 's/os/system/'
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.

Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.

>sed -e 's/unix/linux/' -e 's/os/system/' file.txt
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.

11. Replacing string on a specific line number.

You can restrict the sed command to replace the string on a specific line number. An example is

>sed '3 s/unix/linux/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

The above sed command replaces the string only on the third line.

12. Replacing string on a range of lines.

You can specify a range of line numbers to the sed command for replacing a string.

>sed '1,3 s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

Here the sed command replaces the lines with range from 1 to 3. Another example is

>sed '2,$ s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

>sed '/linux/ s/unix/centos/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
centoslinux which one you choose.

>sed '2 d' file.txt
>sed '5,$ d' file.txt

15. Duplicating lines

You can make the sed command to print each line of a file two times.

>sed 'p' file.txt

16. Sed as grep command

You can make sed command to work as similar to grep command.

>grep 'unix' file.txt
>sed -n '/unix/ p' file.txt

>grep -v 'unix' file.txt
>sed -n '/unix/ !p' file.txt

>sed '/unix/ a "Add a new line"' file.txt
unix is great os. unix is opensource. unix is free os.
"Add a new line"
learn operating system.
unixlinux which one you choose.
"Add a new line"

18. Add a line before a match

The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.

>sed '/unix/ i "Add a new line"' file.txt
"Add a new line"
unix is great os. unix is opensource. unix is free os.
learn operating system.
"Add a new line"
unixlinux which one you choose.

19. Change a line

The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.

>sed '/unix/ c "Change line"' file.txt
"Change line"
learn operating system.
"Change line"

20. Transform like tr command

The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.

>sed 'y/ul/UL/' file.txt
Unix is great os. Unix is opensoUrce. Unix is free os.
Learn operating system.
UnixLinUx which one yoU choose.

Here the sed command transforms the alphabets "ul" into their uppercase format "UL"

BASIC UNIX AND LINUX COMMANDS WITH EXAMPLES

Learning unix operating system is very easy. It is just that you need to understand the unix server concepts and familiar with the unix commands. Here I am providing some important unix commands which will be used in daily work.

Unix Commands With Examples:

1. Listing files

The first thing after logging into the unix system, everyone does is listing the files in a directory. The ls command is used to list the files in a directory.

>ls

add.sh
logfile.txt
prime.pl

If you simply execute ls on the command prompt, then it will display the files and directories in the current directory.

>ls /usr/local/bin

You can pass a directory as an argument to ls command. In this case, the ls command prints all the files and directories in the specific directory you have passed.

2. Displaying the contents of a file.

The next thing is to display the contents of a file. The cat command is used to display the contents in a file.

>cat file.txt
This is a sample unix file
Learning about unix server is awesome

3. Displaying first few lines from a file.

The head command can be used to print the specified number of lines from the starting of a file. The below head command displays the first five lines of file.

>head -5 logfile.dat

4. Displaying last few lines from a file.

The tail command can be used to print the specified number of lines from the ending of a file. The below tail command displays the last three lines of file.

>tail -3 logfile.dat

5. Changing the directories

The cd command can be used to change from one directory to another directory. You need to specify the target directory where you want to go.

>cd /var/tmp

After typing this cd command you will be in /var/tmp directory.

6. Creating a file.

The touch command simply creates an empty file. The below touch command creates a new file in the current directory.

touch new_file.txt

7. copying the contents of one file into another.

The cp command is used to copy the content of source file into the target file. If the target file already have data, then it will be overwritten.

>cp source_file target_file

8. Creating a directory.

Directories are a way of organizing your files. The mkdir command is used to create the specified directory.

>mkdir backup

This will create the backup directory in the current directory.

9. Renaming and moving the files.

The mv command is used to rename the files and it also used for moving the files from one directory into another directory.

Renaming the file.

>mv file.txt new_file.txt

Moving the file to another directory.

>mv new_file.txt tmp/

10. Finding the number of lines in a file

The wc command can be used to find the number of line, words and characters in a file.

>wc logfile.txt
21  26 198 logfile.txt

To know about the unix command, it is always good to see the man pages. To see the man pages simply pass the command as an argument to the man.

REPLACE STRING WITH AWK/SED COMMAND IN UNIX

Replace String With Awk/Sed Command In Unix:

You might have used the Sed Command often to replace the text in file. Awk can also be used to replace the strings in a file.

Here i will show you how to replace the string with awk command. To learn about replacing text with sed command go though the link, Replace String with Sed Command

Replace text with Awk command

1. First we will see a simple example of replacing the text. The source file contians the below data

>cat file.txt
Learn unix
Learn linux

We want to replace the word "unix" with "fedora". Here the word "unix" is in the second field. So, we need to check for the word "unix" in the second field and replace it with workd "fedora" by assigning the new value to the second field. The awk command to replace the text is

awk '{if($2=="unix") {$2="fedora"} print $0}' file.txt
Learn fedora
Learn linux

2. Now we will see a bit complex example.Consider the text file with the below data

>cat file.txt

left
(
In left
)
right
(
In top
)
top
(
In top
)
bottom
(
In bottom
)

Now replace the string, "top" in right section with the string "right". The output should look as

left
(
In left
)
right
(
In right
)
top
(
In top
)
bottom
(
In bottom
)

Here the delimiter in the text file is brace. We have to specify the delimiters in awk command with the record separators. The below awk command can be used to replace the string in a file

awk -vRS=")" '/right/{ gsub(/top/,"right"); }1' ORS=")" file.txt

Here RS is the input record separator and ORS is the output record separator.

DATE COMMAND IN UNIX AND LINUX EXAMPLES

Date command is used to print the date and time in unix. By default the date command displays the date in the time zone that the unix operating system is configured.

Now let see the date command usage in unix

Date Command Examples:

1. Write a unix/linux date command to print the date on the terminal?

>date
Mon Jan 23 01:37:51 PST 2012

This is the default format in which the date command print the date and time. Here the unix server is configured in pacific standard time.

2. Write a unix/linux date command to print the date in GMT/UTC time zone?

>date -u
Mon Jan 23 09:40:21 UTC 2012

The -u option to the date command tells it to display the time in Greenwich Mean Time.

3. Write a unix/linux date command to sett the date in unix?

You can change the date and time by using the -s option to the date command.

>date -s "01/01/2000 12:12:12"

4. Write a unix/linux date command to display only the date part and ignore the time part?

>date '+%m-%d-%Y'
01-23-2012

You can format the output of date command by using the %. Here %m for month, %d for day and %Y for year.

5. Write a unix/linux date command to display only the time part and ignore the date part?

>date '+%H-%M-%S'
01-48-45

Here %H is for hours in 24 hour format, %M is for minutes and %S for seconds

6. Write a unix/linux date command to format both the date and time part.

>date '+%m-%d-%Y %H-%M-%S'
01-23-2012 01-49-59

7. Write a unix/linux date command to find the number of seconds from unix epoch.

>date '+%s'
1327312228

Unix epoch is the date on January 1st, 1970. The %s option is used to find the number of seconds between the current date and unix epoch.

UNIX SEARCH FILE

One of the basic feature of any operating system is to search for files. Unix operating system also provides this feature for searching the files. The Find Command in Unix is used for searching files and directories in Unix, Linux and other Unix like operating systems.

You can specify search criteria for searching files and directories. If you do not specify any criteria, the find command searches for the files in the current directory.

Unix Search Command Examples:

1. Searching for the files in the current directory.

find . -name '*.sh'

The dot(.) represents the current directory and -name option specifies the name of the file to be searched. This find command searches for all the files with ".sh" as the suffix.

2. Searching for the file in all the directories.

find / -type f -name 'job.xml'

The / specifies the home directory of the user, which is at the highest level and the -type option specifies the type of file. This command searches for the regular file,"job.xml", in all the directories.

3. Searching for the file in a particular directory.

find /usr/local/bin/ -type f -name '*.java'

This find command searches for all the java files in the /usr/local/bin directory.

4. Searching for a directory.

find . -type d -name 'tmp'

The -type d indicates the directory. This find command searches for the tmp directory in the current directory.

5. Searching for a directory in another directory

find /var/tmp/ -typd d -name 'personal'

This find command searches for the personal directory in the /var/tmp directory.

FIND COMMAND TO DELETE FILES AND DIRECTORIES

Most of you might have used find command to search for files and directories. The find command can also be used to delete the files and directories. The find command has -delete option which can be used to delete files, directories etc. Be careful while using the -delete option of the find command especially when using recursive find. Otherwise you will end up in deleting the important files.

1. The basic find command to delete a file in the current directory is

find . -name filename -delete

2. The find command to remove empty files in the current directory is

find . -type f -empty -delete

3. The find command to delete empty directories is

find . -type d -empty -delete

MAKE AWK COMMAND CASE INSENSITIVE

Awk command is used to parse the files which have delimited data. By default, awk command does a case-sensitive parsing. The awk command has a IGNORECASE built-in variable to do a case insensitive parsing. We will see about the IGNORECASE in detail here.

Consider a sample text file with the below data

>cat file.txt
mark iphone
jhon sony
peter Iphone
chrisy motorola

The below awk command can be used to display the lines which have the word "iphone" in it.

awk '{if($2 == "iphone") print $0 }' file.txt

This awk command looks for the word "iphone" in the second column of each line and if it finds a match, then it displays that line. Here it just matches the word "iphone" and did not match the word "Iphone". The awk did a case sensitive match.

The output of the above command is

mark iphone

You can make the awk to do case insensitive and match even for the words like "Iphone" or "IPHONE" etc.

The IGNORECASE is a built in variable which can be used in awk command to make it either case sensitive or case insensitive.

If the IGNORECASE value is 0, then the awk does a case sensitive match. If the value is 1, then the awk does a case insensitive match.

The awk command for case insensitive match is

awk 'BEGIN {IGNORECASE = 1} {if($2 == "iphone") print $0 }' file.txt

The ouput is

mark iphone
peter Iphone

PARSING /ETC/PASSWD UNIX FILE WITH AWK COMMAND

Awk command parses the files which have delimited structure. The /etc/passwd file is a delimited file. Using the Awk command is a good choice to parse the /etc/passwd file. Sample /etc/passwd file looks like as below

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync

The /etc/passwd file contains the data in the form of row and columns. The columns are delimited by a colon (:) character.

Now we will see how to write an Awk command which reads the /etc/passwd file and prints the names of the users who have the /bin/bash program as their defaualt shell command.

awk -F: '$7 == "/bin/bash" { print $1 }' /etc/passwd

The -F option is used to specify the filed delimiter.

The output of the above awk command is

root

GREP COMMAND IN UNIX AND LINUX EXAMPLES

!grep

grep "Error" logfile.txt

This searches for the string "Error" in the log file and prints all the lines that has the word "Error".

3. Searching for a string in multiple files.

grep "string" file1 file2
grep "string" file_pattern

grep -i "UNix" file.txt

5. Specifying the search string as a regular expression pattern.

grep "^[0-9].*" file.txt

grep -w "world" file.txt

7. Displaying the lines before the match.

Some times, if you are searching for an error in a log file; it is always good to know the lines around the error lines to know the cause of the error.

grep -B 2 "Error" file.txt

This will prints the matched lines along with the two lines before the matched lines.

8. Displaying the lines after the match.

grep -A 3 "Error" file.txt

This will display the matched lines along with the three lines after the matched lines.

9. Displaying the lines around the match

grep -C 5 "Error" file.txt

grep -r "string" *

11. Inverting the pattern match

You can display the lines that are not matched with the specified search sting pattern using the -v option.

grep -v "string" file.txt

12. Displaying the non-empty lines

You can remove the blank lines using the grep command.

grep -v "^$" file.txt

13. Displaying the count of number of matches.

We can find the number of lines that matches the given string/pattern

grep -c "sting" file.txt

14. Display the file names that matches the pattern.

We can just display the files that contains the given string/pattern.

grep -l "string" file.txt

15. Display the file names that do not contain the pattern.

We can display the files which do not contain the matched string/pattern.

grep -l "string" file.txt

16. Displaying only the matched pattern.

By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.

grep -o "string" file.txt

17. Displaying the line numbers.

We can make the grep command to display the position of the line which contains the matched string in a file using the -n option

grep -n "string" file.txt

18. Displaying the position of the matched string in the line

The -b option allows the grep command to display the character position of the matched string in a file.

grep -o -b "string" file.txt

grep "^start" file.txt

grep "end$" file.txt

METHODS TO FIND SUM OF NUMBERS IN A FILE - UNIX

In Unix, there are different ways to find the sum of numbers in a file. I will explain this with help of an example.

I have an input file (num.txt) which contains the below data

>cat num.txt
1
2
3
4
5

I have to sum the values in each line from the file and print the result on the terminal. The required output is

Method1: Finding the sum using the bash script. Save the below bash script in a file and execute the file.

#!/bin/bash

SUM=0
while read LINE
do
SUM=`expr $SUM + $LINE`
done < num.txt
echo $SUM

Method2: Another way of implementing in bash is

SUM=0
for num in $(cat num.txt)
    do
        ((SUM+=num))
done
echo $SUM

Method3: You can use “Awk” command to find the sum of numbers in a file.

awk 'BEGIN{sum=0} {sum=sum+$1} END {print sum}' num.txt

Method4: The "bc" command can be used to do math operations. We will use the "tr" command along with the "bc" to find the sum of numbers.

tr '\n' '+' < num.txt | echo $(sed 's/+$//') | bc

Here the "tr" command converts the new line characters into "+" character. The output of tr '\n' '+' < num.txt is 1+2+3+4+5+. The "sed" command removes the addition "+" sign at the end. Finally the "bc" command finds the sum.

Method5: Using "bc" with "paste" command.

paste -sd+ num.txt | bc

The "paste" command works same as the "tr" command except it does not produce the additional "+" at the end.

Method6: Using "bc" with "sed" command.

echo $(sed 's/$/+/' num.txt) 0 | bc

Here the “sed” command adds the "+" sign at the end of each line.

GROUPING A SET OF LINES AS A PARAGRAPH - UNIX AWK COMMAND

The awk command can be used to group a set of lines into a paragraph. We will also use a bash shell script to group the lines into a paragraph. As an example, consider the file, group.txt, with the below data

>cat group.txt
A one
B two
C three
D four
E five
F six
G seven

We want to group 3 lines in the file as a paragraph. The required output is

A one
B two
C three

D four
E five
F six

G seven

The bash script for achieving this is

#!/bin/bash
line_count=1
while read line
do
 S=`expr $line_count % 3`
 if [ "$S" -eq 0 ]
 then
   echo -e $line"\n"
 else
   echo $line
 fi
 line_count=`expr $line_count + 1`
done < group.txt

Now we will see how to achieve this using the Awk command in Unix. The awk command for this is

awk '!( NR % 3 ) {$0 = $0"\n"} 1' group.txt

METHODS TO CONVERT HEXADECIMAL TO DECIMAL IN UNIX

We will see how to convert hexadecimal numbers into decimal numbers using Unix command bash scripting.

The bc command can be used to convert hexadecimal number into decimal number
Example:

>echo "ibase=16;ABCD"|bc
43981

The hexadecimal number "ABCD" is converted into decimal number 43981.

The bc command can also be used convert the decimal numbers back to hexadecimal numbers.

Exmple:

>echo "obase=16;43981"|bc
ABCD

Converting hexadecimal to decimal using bash script:

Consider the below file with hexadecimal numbers.

>cat hexa.txt
ABCD
125A
F36C
E962

The bash script shown below converts the hexadecimal numbers into decimal numbers

#!/bin/bash
while read line
do
printf "%d\n" "0x"$line
done < hexa.txt

The output after running this script is

Converting decimal to hexadecimal using bash script:

The input file contains the decimal numbers with below data.

>cat deci.txt
43981
4698
62316
59746

Now we will do the reverse process. The bash script for converting the decimal numbers to hexadecimal numbers is

#!/bin/bash
while read line
do 
printf "%0X\n" $line
done < deci.txt

After running this script the output will be

ABCD
125A
F36C
E962

PRINT FILE IN REVERSE USING UNIX COMMAND

Q) How to print the lines in a file in reverse order? Which means we have to print the data of file from last line to the first line.

We will see different methods to reverse the data in a file. As an example, consider the file with the below data.

>cat file.txt
Header
line2
line3
line4
Footer

We need to display the lines in a file in reverse order. The output data is

Footer
line4
line3
line2
Header

1.The tac command in unix can be used to print the file in reverse. The tac command is

tac file.txt

2. The sed command for reversing the lines in a file is.

sed '1!G;h;$!d' file.txt

3. Another usage of sed command is

sed -n '1!G;h;$p' file.txt

METHODS TO REVERSE A STRING USING UNIX COMMANDS

This topic will cover different methods to reverse each character in a string and reversing the tokens in a string.

Reversing a string:

1. The sed command can be used to reverse a string. The sed command for this is

echo "hello world" | sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'

2. Aother usage of sed command with tac and tr is

echo "hello world" |sed 's/./&\n/g' |tac |tr -d '\n'

3. The awk command for reversing the string is

echo "hello world" | awk '{ 
    n=split($0,arr,""); 
    for(i=1;i<=n;i++) 
     s=arr[i] s
     } 
     END
     {
    print s
     }'

4. In this method, a bash script will be written to reverse the string. The bash script is

#!/bin/bash
str="hello world"
len=`echo $str | wc -c`
len=`expr $len - 1`
rev=""
while test $len -gt 0
do
rev1=`echo $str | cut -c$len`
rev=$rev$rev1
len=`expr $len - 1`
done
echo $rev

The output of all the above four methods is the reverse of the string "hello world", which is

dlrow olleh

5. Using the rev command

We can use the rev command to reverse the string which is shown below:

echo "hello world"|rev

Reversing the tokens in a string:

1. The awk command can be used to reverse the tokens in a string. The awk command for this is

echo "hello world" | awk '{
    n=split($0,A);
    S=A[n];
    for(i=n-1;i>0;i--)
     S=S" "A[i]
     }
     END
     {
    print S
       }'

2. Using the tac and tr command we can reverse the tokens in a string. The unix command is

echo "hello world"|tac -s " "| tr  "\n" " "

3. The bash script for reversing the tokens in a string is.

#!/bin/bash
TOKENS="hello world"
for i in $TOKENS
do STR="$i $STR"
done
echo $STR

The output of the above two methods is

world hello

REMOVE TRAILING ZEROS USING UNIX COMMAND

Q) How to remove the trailing zeros from each line of a file?

Trailing zeros are the zero which appear only at the end of the line. There are many ways to remove the trailing zeros. Let us assume that the source file contains the below data.

>cat file.txt
12345
67890
10100
10000

The required output should not contain the trailing zeros. The output should be

The unix command for producing this result is

rev file.txt | awk '{print $1*1}'|rev

Here the rev command will reverse the string in each line. Now the trailing zeros will become leading zeros. In the awk command the string is converted into a number and the leading zeros will be removed. At the end, the rev command again reverses the string.

REMOVE THE LINES FROM A FILE WHICH ARE SAME AS THE FIRST LINE - UNIX AWK

Q) How to remove the lines which are same as the first line.

Awk command can be used to remove the lines which are same as the first line in a file. I will also show you another method of removing the file. As an example, consider the file with the below data.

Header
line2
line3
Header
line5
line6
line7
Header

The first line contains the text "Header". We need to remove the lines which has the same text as the first line.

The required output data is

Header
line2
line3
line5
line6
line7

The awk command can be used to achieve this. The awk command for this is

awk '{ 
 if(NR==1) 
 {
  x=$0; 
  print $0
 } 
 else if(x!=$0) 
  print $0 
     }' file.txt

The other way to get the output is using bash script.

x=`head -1 file.txt`; echo $x; cat file.txt | grep -v \^"${x}"\$

EXAMPLES OF BASENAME COMMAND IN UNIX

The basename utility is used to

Remove any prefix ending in /.
Remove the suffix from a string.

Syntax of basename command:

basename [string] [suffix]

Here 'string' is the input string and suffix is the string which needs to removed from the input string.

Examples:

1. basename /usr/bin/perlscript
This will remove the prefix, /usr/bin/, and prints only the string 'perlscript'

2. basename perlscript script
This will remove the suffix 'script' from 'perlscript' and prints only 'perl'

3. basename /usr/bin/perlscript script
This will remove both the prefix and suffix and prints only 'perl'

basename command is mostly used in shell scripts to get the name of the shell script file you are running. Sample shell script code is shown below

#!/usr/bin/sh
filename=`basename $0`
echo $filename

EXAMPLES OF ALIAS COMMAND IN UNIX

Alias command is an alternative name used for long strings that are frequently used. It is mostly used for creating a simple name for a long command.

Syntax of alias command:

alias [alias_name=['command']]

For more information on alias utility see the man pages. Type 'man alias' on the command prompt.

Examples:

1. alias

If you simply type alias on the command prompt and then enter, it will list all the aliases that were created.

2. alias pg='ps -aef'

The ps -aef command will list all the running processes. After creating the alias pg for ps -aef, then by using the pg on command prompt will display the running processes. The pg will work same as the ps -aef.

By creating an alias for a command on the command prompt will be present only for that session. Once you exit from the session, then the aliases won’t take effect. To make the aliases to remain permanent, place the alias command in the ".profile" of the user. Open the user ".profile" and place the command alias pg="ps -aef", save the file and then source the ".profile" file. Now the alias pg will remain forever.

To remove an alias use the unalias command
Example: unalias pg

CONVERTING AWK SCRIPT TO PERL SCRIPT - EXAMPLES OF A2P UNIX COMMAND

Unix provides the a2p (awk to perl) utility for converting the awk script to perl script. The a2p command takes an awk script and produces a comparable perl script.

Syntax of a2p:
a2p [options] [awk_script_filename]

Some of the useful options that you can pass to a2p are:

-D<number> Sets debugging flags.
-F<character> This will tell a2p that awk script is always invoked with -F option.
-<number> This makes a2p to assume that input will always have the specified number of fields.

For more options see the man pages; man a2p

Example1:

The awk script which prints the squares of numbers up to 10 is shown below. Call the below script as awk_squares.

#!/bin/awk -f
BEGIN 
{
    for (i=1; i <= 10; i++) 
    {
        print "The square of ", i, " is ", i*i;
    }
exit;
}

Run this script using awk command; awk -f awk_squares. This will produce squares of numbers up to 10.

Now we will convert this script using the a2p as
a2p awk_squares > perl_squares

The content of converted perl script, perl_squares, is shown below:

#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"mailto:$@%22%7D'
if $running_under_some_shell;
# this emulates #! processing on NIH machines.
# (remove #! line above if indigestible)
eval '$'.$1.'$2;' while $ARGV[0] =~ /^([A-Za-z_0-9]+=)(.*)/ && shift;
# process any FOO=bar switches
$, = ' ';               # set output field separator
$\ = "\n";              # set output record separator

for ($i = 1; $i <= 10; $i++) {
    print 'The square of ', $i, ' is ', $i * $i;
}
last line;

Run the perl script as: perl perl_squares. This will produce the same result as the awk.

Example2:

We will see an awk script which prints the first field from a file. The awk script for this is shown below. Call this script at awk_first_field.

#!/bin/awk -f
{
    print $1;
}

Run this script using awk command by passing a file as input: awk -f awk_first_field file_name. This will prints the first field of each line from the file_name.

We will convert this awk script into per script using the a2p command as
a2p awk_first_field > perl_first_field

The content of converted perl script, perl_first_field, is shown below:

#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"mailto:$@%22%7D'
if $running_under_some_shell;
# this emulates #! processing on NIH machines.
# (remove #! line above if indigestible)
eval '$'.$1.'$2;' while $ARGV[0] =~ /^([A-Za-z_0-9]+=)(.*)/ && shift;
# process any FOO=bar switches
$, = ' ';               # set output field separator
$\ = "\n";              # set output record separator
while (<>) {
    ($Fld1) = split(' ', $_, -1);
    print $Fld1;
}

Now run the perl script as: perl perl_first_field file_name. This will produce the same result as awk command.

======================================================

LINUX CHECK DISK DRIVE FOR BADBLOCKS AND ERRORS

Bad Sectors are common in a computers hard disk drive. Bad sectors are caused due to damage to the hard drive surface or operating system inability to access the sectors. These bad sectors can be identified by the disk utility softwares. Periodically checking for bad sectors prevents losing of valuable data.

In windows operating system CHKDSK or SCANDISK are used to detect the bad sectors. In unix or linux like operating systems (ubuntu, fedora, debian etc) badblocks utility is used to check for the bad sectors on a hard disk drive.

Here we will see about the badblocks and the fsck utilities. The badblocks utility scans the disk for errors and crates a list of bad sectors. This bad sectors list can be used by programs like fsck and mkfs, so that the OS do not use them in future and thus do not cause any corruption of data.

The following command creates a file with a list of bad sectors:

sudo badblocks -v /dev/disk > bad-blocks.dat

The fsck command is used to record the bad blocks.

sudo fsck -t ext3 -l bad-blocks.dat /dev/disk

Another way to record the bad blocks is to use the e2fsck command with -c option. This prevents the data from being stored on these bad blocks.

e2fsck -c /dev/drive

GREP / PRINT LINES BEFORE AND AFTER MATCH - UNIX / LINUX COMMAND

Q) How to display the lines that are above and below the matched line from a file using unix or linux commands?

The grep command in unix or linux system is used to print the lines that match a given pattern. By default the grep command displays only the matching lines. We can change the behaviour of the grep command to print the lines that are above and below the matched line.

Create the following file in linux or unix server:

> cat sample.dat
unix operating system
linux virtual server
fedora dedicated server
debian system
ubuntu host

Let see the below grep command examples"

1. Print only matched lines

As said earlier the grep command by default only displays the matched lines.

> grep "linux" sample.dat
linux virtual server

2. Print lines after the match

Use the -A option with grep command to print the lines after matched line. The syntax and the example are shown below:

syntax:
grep -An "pattern" filename

Here n is the number of lines to print after the matched line.

Example:

> grep -A1 "fedora" sample.dat
fedora dedicated server
debian system

3. Display lines before the match

To print the lines before the matched line use the -B option with grep command. The syntax and the example are shown below:

Syntax:

grep -Bn "pattern" filename

Example:

> grep -B1 "ubuntu" sample.dat
debian system
ubuntu host

4. Print lines before and after match

We can print both the lines above and below the matched line. Use the -a option with the grep command.

> grep -a1 "fedora" sample.dat
linux virtual server
fedora dedicated server
debian system

READ COMMAND LINE ARGUMENTS - UNIX / LINUX BASH SCRIPT

Q) How to read the arguments or parameters passed to a shell script from the command line?

In general the command line arguments are passed to the bash or shell script to change the behavior of the script. In this article, I am going to provide a simple shell script to read and print the command line parameters.

Take a look at the following unix shell script:

> cat OS_Print.sh
#!/bin/bash
echo "Script execution starts"
echo "$@"
echo "$0"
echo "$1"
echo "$2"
echo "$#"
echo "Script execution ends"

The basic functionality of the above script is to print the values stored in the $ variables. Now we will run the above script by passing some arguments.

> OS_Print.sh unix linux
Script execution starts
unix linux
OS_Print.sh
unix
linux
2
Script execution ends

You can see, the command line arguments passed here are unix and linux. Command line arguments are a list of parameters separated by space delimiters passed to the shell script.

Explanation of $ variables:

$@ : contains all the arguments
$0 : contains script name
$1 : First argument
$2 : Second argument
$n : Nth argument
$# : Count of arguments passed.

Examples:

1. Script to iterate through arguments.

The following script prints the parameters using for loop.

#!/bin/bash

for value in $@
do
  echo $value
done

2. Print only the last argument.

There are many ways to display the last argument. The following script shows the different ways of printing the last argument.

#!/bin/bash

echo "${@: -1}"
echo "${BASH_ARGV[0]}"
echo "${@: $#}"
echo "${!#}"

for value in $@; do :; done
echo $value

BC COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

Arithmetic operations are the most common in any kind of programming language. Unix or linux operating system provides the bc command and expr command for doing arithmetic calculations. You can use these commands in bash or shell script also for evaluating arithmetic expressions.

Here we will see only about the bc command. The bc command evaluates expressions similar to the c programming language. The bc command supports the following features.

Arithmetic operators
Increment and decrement operators
Assignment operators
Comparision or Relational Operators
Logical or Boolean operators
Math Functions
Conditional statements
Iterative statements
Functions

Arithmetic operator Examples:

The following example shows how to use various arithmetic operators. The examples are pretty straight forward. So, I will provide explanation only when required. In most of the examples the echo statment is used to provide the expressions to the bc command.

1. Finding Sum of Two expressions

> echo "2+5" | bc
7

2. Difference of Two numbers

> echo "10-4" | bc
6

3. Multiplying two numbers

> echo "3*8" | bc
24

4. Dividing two numbers

When you divide two numbers, the bc command Ignores the decimal part and returns only the integral part as the output. See the below examples

> echo "2/3" | bc
0

> echo "5/4" | bc
1

Use the scale function to specify the number of decimal digits that the bc command should return.

> echo "scale=2;2/3" | bc
.66

5. Finding the remainder using modulus operator

> echo "6%4" | bc
2

6. Using exponent operator

> echo "10^2" | bc
100

Here the expression is evaluated as 10 to the power of 2.

Assignment Operator Examples:

Assignment operators are used to assign a value to the variable. The following example shows how to use the assignment operators:

Assigns 10 to the variable and prints the value on the terminal.
> echo "var=10;var" | bc

Increment the value of the variable by 5
> echo "var=10; var+=5;var | bc
15

The lists of assignment operators supported are:

var = value : Assign the value to the variable
var += value : similar to var = var + value
var -= value : similar to var = var - value
var *= value : similar to var = var * value
var /= value : similar to var = var / value
var ^= value : similar to var = var ^ value
var %= value : similar to var = var % value

Increment Operator Examples:

There are two kinds of increment operators. They are pre increment and post increment operators.

++var : Pre increment operator. The variable is incremented first and then the result of the variable is used.
var++ : Post increment operator. The result of the variable is used first and then the variable is incremented.

> echo "var=5;++var" | bc
6

> echo "var=5;var++" | bc
5

Here, in the second example the value of var is printed first and then it is incremented. See the below example, to see the complete incremental effect.

> echo "var=5;var++;var" | bc
5
6

Decrement Operator Examples:

Similar to the increment operators, there are two types of decrement operators.

--var : Pre decrement operator. The variable is decremented first and then the result of the variable is used.
var-- : Post decrement operator. The result of the variable is used first and then the variable is decremented.

> echo "var=5;--var"| bc
4
> echo "var=5;var--"| bc
5

Relational Operators Examples:

Relational operators are used to compare two numbers. If the comparison is true, then it returns 1. Otherwise (false), it returns 0. The relational operators are mostly used in conditional statements like if. The list of relational operators supported in bc command are shown below:

expr1 < expr2 : Result is 1 if expr1 is strictly less than expr2.
expr1 <= expr2 : Result is 1 if expr1 is less than or equal to expr2.
expr1 > expr2 : Result is 1 if expr1 is strictly greater than expr2.
expr1 >= expr2 : Result is 1 if expr1 is greater than or equal to expr2.
expr1 == expr2 : Result is 1 if expr1 is equal to expr2.
expr1 != expr2 : Result is 1 if expr1 is not equal to expr2.

> echo "10 > 5" | bc
1

> echo "1 == 2" | bc
0

Logical Operator Examples:

Logical operators are also mostly used in conditional statements. The result of the logical operators is either 1 (True) or 0 (false) ! expr : Result is 1 if expr is 0.

expr && expr : Result is 1 if both expressions are non-zero.
expr || expr : Result is 1 if either expression is non-zero.

> echo "4 && 10" | bc
1
> echo "0 || 0" | bc
0

Math Functions:

The built-in math functions supported are:

s (x) : The sine of x, x is in radians.
c (x) : The cosine of x, x is in radians.
a (x) : The arctangent of x, arctangent returns radians.
l (x) : The natural logarithm of x.
e (x) : The exponential function of raising e to the value x.
j (n,x): The bessel function of integer order n of x.
sqrt(x): Square root of the number x.

In addition to the math functions, the following functions are also supported.

length(x) : returns the number of digits in x
read() : Reads the number from the standard input.

Conditional Statement Examples:

Conditional statements are used to take decisions and execute statements based on these decisions. Bc command supports the if condition. The syntax of if statement is

if(condition) { statements} else {statements}

The following example shows show to use the if condition

> echo 'if(1 == 2) print "true" else print "false"' | bc
false

Iterative Statements:

Bc command supports the for and while loop for doing iterations. The syntax of for and while loop are shown below:

for (assignment; condition; increment) {
statements
}

while (condition) {
statements
}

The following examples prints numbers from 1 to 10 using the for and while loops

> echo "for(i=1;i<=10;i++) {i;}" | bc
> echo "i=1; while(i<=10) { i; i+=1}" | bc

Functions:

A function is a code block which executes logically related functionality and returns a value. The syntax of creating a function is

define function-name(comma separated parameters list) {
statements
return statement
}

So far we have provided the arithmetic expressions to the bc command by using the echo statement. We can write these arithmetic expressions in a file and then execute those statements by providing the filename to the bc command. This is shown below:

> cat arth_expr.dat
2+5;
var = 10*3
var
print var
define sum(a,b) {
return a+b
}
sum(3,5)
quit

Now see how to execute these statements:

> bc arth_expr.dat
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
7
30
30
8

Be default the bc command prints the welcome message(version, copyright message. You can suppress this welcome message by using the -q option with bc command

> bc -q arth_expr.dat

Important Points:

Bc command treats the semicolon (;) or newline as the statement separator.
To group statements use the curly braces. Use with functions, if statement, for and while loops.
If only an expression is specified as a statement, then bc command evaluates the expression and prints the result on the standard output.
If an assignment operator is found. Bc command assigns the value to the variable and do not print the value on the terminal.
A function should be defined before calling it. Always the function definition should appear first before the calling statements.
If a standalone variable is found as a statement, bc command prints the value of the variable. You can also Use the print statement for displaying the list of values on the terminal.

EXPR COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

This continuation to my previous post on bc command - calculator. In this article we will see how to use the expr command in unix or linux system for doing arithmetic operations.

The syntax of expr command is

expr [expression]

Let see how to use the expr command in unix or linux with examples. Most of the examples are straightforward. I will provide description as and when required.

Note: You have to provide the space between the values and the operands. Otherwise the expr command may throw error or print them as a string.

Arithmetic Operator Examples:

1. Sum of numbers

$ expr 5 + 3

8

$ expr 1 + 2 + 3

6

$ expr 5+3

5+3

Here in the third expr command, space is not provided between the literals. The expr command treated it as a string and printed on the terminal.

2. Difference between two numbers

$ expr 10 - 6

4

3. Multiplying numbers

$ expr 7 \* 9

63

Here the * is shell builtin operator, that is why it needs to escaped with backslash.

4. Dividing numbers

$ expr 6 / 4

1

The division operator returns only the arithmetic quotient.

5. Remainder or modulus

$ expr 6 % 4

2

Comparision or Relational Operator Examples:

You can use the following comparision operators with the expr command:

Val1 < Val2 : Returns 1 if val1 is less than val2. otherwise zero.
Val1 <= Val2 : Returns 1 if val1 is less than or equal to val2. otherwise zero.
Val1 > Val2 : Returns 1 if val1 is greater than val2. otherwise zero.
Val1 >= Val2 : Returns 1 if val1 is greater than or equal to val2. otherwise zero.
Val1 = Val2 : Returns 1 if val1 is equal to val2. otherwise zero.
Val1 != Val2 : Returns 1 if val1 is equal to val2. otherwise zero.
val1 | val2 : Returns val1 if val1 is neither null nor zero. Otherwise val2.
val1 & val2 : Returns val1 if both val1 and val2 is neither null nor zero. Otherwise 0.

Note: You have to escape most of the operators with backslash as they are shell built in.

$ expr 1 \< 2

1

$ expr 1 \<= 1

1

$ expr 2 \> 5

0

$ expr 2 \>= 5

0

$ expr 7 = 7

1

$ expr 9 != 18

1

$ expr 2 \| 5

2

$ expr 0 \| 5

5

$ expr 2 \& 5

2

$ expr 6 \& 3

6

$ expr 6 \& 0

0

$ expr 0 \& 3

0

String Function Examples:

1. Length of string

The length function is used to find the number of characters in a string.

$ expr length linux

5

$expr length linux\ system

12

$expr length "linux system"

If you have spaces in your string escape them with backslash or quote them with double quotes.

2. Find Substring

You can extract a portion of the string by using the substr function. The syntax of substr function is

substr string position length

Here position is the character position in the string. length is the number of chracters to extract from the main string. An example is shown below:

$ expr substr unixserver 5 6
server

3. Index of the substring

You can find the position of a string in the main string using the index function. The syntax of index function is shown below:

index string chars

If the chars string is found in the main string, then the index function returns the position of the chars. Otherwise it returns 0. See the following examples:

$ expr index linux nux
3

$expr index linux win
0

4. Matching a regexp

The match function is used to find anchored pattern match of regexp in the string. The syntax of match function is shown below:

match string pattern

The match function returns the number of characters in the pattern is a match is found. Otherwise, it returns 0. Alternative synatx is

string : pattern

The following examples shows how to use the match function:

$ expr match linuxserver lin
3

$ expr match linuxserver server
0

Here in the second expr, the pattern (server) exists in the main string. However the pattern does not start from the beggining of the main string. Thats why the match function returns 0.

ARITHMETIC OPERATIONS / EXPRESSIONS IN UNIX / LINUX BASH SCRIPT

Unix or linux operating systems provides the bc and expr commands for doing arithmetic calculations. In this article, we will see how to do arithmetic operations in the unix shell script. I recommend the following articles before going to use the bc and expr commands in the bash script:

bc command tutorial
expr command tutorial

The following basic examples shows how to do arithmetic calculations in the shell scripts using the bc and expr command:

1. Sum of numbers

#!/bin/bash
#assigning values to the variables
a=9
b=7
res=`expr $a + $b`
echo $res
res=`echo "$a+$b" | bc`
echo $res
#using the let command to find the sum
let z=$a+$b
echo $z
#using braces
echo $(($a+$b))

2. Difference between two numbers

#!/bin/bash
#assign values to the variables
a=10
b=7
res=`expr $a - $b`
echo $res
res=`echo "$a-$b" | bc`
echo $res

3. Multiplication and division

#!/bin/bash
#assign values to the variables
a=2
b=3
res=`expr $a * $b`
echo $res
res=`echo "$a*$b" | bc`
echo $res

4. Length of the string

#!/bin/bash

str="linux dedicated server"
len=`expr length "$str"`
echo $len

5. Printing numbers from 1 to 10 using for loop

#!/bin/bash

echo 'for(start=1;start <= 10;start++) {start;}' | bc

FILE TEST OPERATORS / OPERATIONS EXAMPLES IN UNIX / LINUX SHELL SCRIPT

In linux and unix operating systems every thing is a file. When you are using files in your shell or bash script, it is a good idea to do some tests on the file before using it.

The file tests include:

Checking for existence of the file.
File is readable, writeable or executable.
Type of the file and so on.

The file test operators are mostly used in the if clause of the bash script. The syntax is shown below:

if [ -option filename ]
then
  do something
else
  do something
fi

The different file test operators are listed below:

a : True if the file exists.
b : True if the file exists and is a block special file.
c : True if the file exists and is a character special file.
d : True if the file exists and is a directory.
e : True if the file exists.
f : True if the file exists and is a regular file.
g : True if the file exists and its SGID bit is set.
h : True if the file exists and is a symbolic link.
k : True if the file exists and its sticky bit is set.
p : True if the file exists and is a named pipe (FIFO).
r : True if the file exists and is readable.
s : True if the file exists and has a size greater than zero.
t : True if file descriptor is open and refers to a terminal.
u : True if the file exists and its SUID (set user ID) bit is set.
w : True if the file exists and is writable.
x : True if the file exists and is executable.
O : True if the file exists and is owned by the effective user ID.
G : True if the file exists and is owned by the effective group ID.
L : True if the file exists and is a symbolic link.
N : True if the file exists and has been modified since it was last read.
S : True if the file exists and is a socket.

File Test Operator Example:

The following shell script checks for the existence of a regular file:

#!/bin/bash
#assign file name to the variable
FILE="linux-server.dat"

if [ -f $FILE ]
then
  echo "$FILE exists and is a regular file"
else
  echo "Either $FILE does not exist or is not a regular file"
fi

REMOVE DUPLICATE STRINGS / WORDS FROM LINE IN UNIX / LINUX

Q) I have the following file with data in my linux system:

$ cat sample.dat
unix,linux,linux,server
unix,unix,dedicated server

Here in the first line the word linux is duplicated. In the second line the pattern unix is duplicated. Now in the output, I want to suppress the duplicates and print the strings only once. The output should look as

unix,linux,server
unix,dedicated server

Solution:

Here I am providing an awk solution. The below awk command supress the duplicate patterns and prints the pattern only once in each line.

awk '{str="";c=0;split($0,arr,","); for (v in arr) c++; for (m=c;m >= 1;m--) for (n=1; n<m;n++) if (arr[m] == arr[n]) delete arr[m]; for (k=1;k<=c;k++) {if (k ==1 ) {s=arr[k] } else if (arr[k] != "") str=str" "arr[k] } print str}' sample.dat

The awk command is formatted and shown below:

awk '{
   str="";
   c=0;
   split($0,arr,",");
   for (v in arr) 
      c++;
   for (m=c;m >= 1;m--) 
     for (n=1; n<m;n++) 
        if (arr[m] == arr[n])
           delete arr[m];
   for (k=1;k<=c;k++) 
   {
      if (k ==1 ) 
      {
         s=arr[k]
      } 
      else if (arr[k] != "") 
          str=str""arr[k]"," 
    } 
    print substr(str,1,length(str)-1)
   }' sample.dat

Explanation of the Awk Command:

The split function splits the each line from the file into tokens and stores the tokens as elements in the array (arr).
The for loop is used to find the number of elements in the array. The value c contains the total number of elements in the array.
Next, two for loops are used. The outer for loop is used to read the array elements from the last. The inner for loop reads the elements from the starting of the array. In the if condition, the last element in the array is compared with the remaining elements in the array. If a match is found, then the element is removed from the array. This causes the duplicate words to be removed from the line. This comparison is done for each element in the array and the duplicate elements removed.
The final for loop creates string with all the elements in the array.

TOP COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

Top command in unix or linux operating system is one of the useful commands to know about the system information. The top command provides real-time view of the running system and also the list of tasks currently managed by the kernel. Top is a non-interactive command and provides limited interactive options to the users.

The syntax of top command is

top [options]

The options are:

-b : Starts top command in batch mode. Useful for sending top output to other programs or file.
-d : specify the delay time between the screen updates.
-n : Number of iterations, the top should produce before ending.
-u : Monitor only the specified user processes.
-p : Monitor only the specified processes. Specify the process ID

Top Command Examples:

1. Monitor system information

The basic functionality of the top command is to monitor the system information. Just run the top command on the terminal to print the system information.

$ top
top - 19:05:50 up 21 min,  4 users,  load average: 0.02, 0.07, 0.15
Tasks: 174 total,   2 running, 172 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.3%us,  2.6%sy,  0.1%ni, 88.0%id,  3.8%wa,  0.2%hi,  0.1%si,  0.0%st
Mem:   1990204k total,   756084k used,  1234120k free,    74648k buffers
Swap:  3984080k total,        0k used,  3984080k free,   391680k cached

PID  USER  PR  NI VIRT   RES   SHR  S  %CPU %MEM  TIME+   COMMAND
1156 root  20  0  94144  35m   9m   S   10  1.8   1:14.27 Xorg
1908 user  20  0  38604  13m   9428 S   2   0.7   0:03.41 gnome-terminal                          
2315 user  20  0  2468   1092  784  R   2   0.1   0:00.01 top

Top provides dynamic information of the system. As and when the system information changes, it keeps on updating the information on the terminal. The fields of top command are explained below:

PID: Tasks process id.
USER: User name of the owner who started the process.
PR: priority of the task.
NI : Nice value of the task. Negative value means highest priority. Positive value means lowest priority. Zero means priority cant be determined.
VIRT: virtual image. Total amount of memory used by the task.
RES: Resident size. Non-swapped memory used by the task.
SHR : Shared memory used by the task.
S : Status of the process.D - uninterruptible sleep;R - running; S - sleeping; T - traced or stopped; Z - zombie.
%CPU : Cpu usage.
%MEM: Usage of physical memory.
TIME: Cpu time. Time of the task since it started.
COMMAND: Program name or command name.

2. Redirect top command output to a file.

If you write the output of top command to a file, the data is written in binary format and is not readable by the user. Use the -b option to write the output of the top command in text format.

$ top -n 1 -b > top_output.dat

Here the -n 1 option specifies the top command to run for only one iteration.

CHMOD COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

Chmod (change mode) is one of the most frequently used commands in unix or linux operating system. The chmod command is used to change the file or directory access permissions. To know about the access permissions of a file or directory, use the ls -l command as shown below:

$ ls -l sample.sh
-rwx-rw-r-- 1 matt deploy 94 Oct  4 03:12 sample.sh

Here in the above example: Use matt has the read, write and execute permissions on the file. Group deploy has read and write permissions. Others have only the read permission.
File and Directory permissions:

There are three different permissions. They are:

Read (4): Permitted to read the contents of the file. In case of directory, you can view all the files and sub-directories in that directory.
Write (2): Permitted to write to the file. In case of directory, you can create files and sub-directories.
Execute (1): Execute the file as a program/shell script. In case of directory, You can enter into that directory.

Here in the above, the numbers in the brackets represents the numeric values for the corresponding permissions. If you want to have a combination of permissions add the required numbers. For example, for read and execute, it is 4+1=5.

The syntax of chmod command is

chmod [options] mode filename

THe important options are:

-R : recursively change the permissions of a directory.
-v : Verbose

Chmod Examples in Linux / Unix:

1. Give read, write and execute permissions to everyone.

Read, write and execute: 4+2+1=7

$ chmod 777 sample.sh

In the above example, you can see that the permissions are specified with a three digit number. The first digit is for user permissions, second is for group and third is for others permission. This type of representation is called octal representation. Alternatively, you can use the symbolic representation to give the permissions.

chmod ugo+rwx sample.sh

We will see more details about the symbolic representation later.

2. Give read permission to user, write permission to group and execute permission to others.

$ chmod 421 sample.sh

3. Recursive permissions to directory

To give read and write permissions to all the users to a directory (including files and subdirectories) use the recursive option -R.

chmod -R 666 /dir

Symbolic Representation of Permissions:

The following symbols are used to represent the users, groups and others:

u : User
g : Group
o : Others a : All (user, group and others)

The following symbols represent the permissions:

r : read
w : write
x : execute

The following symbols represent the permissions grant or revoke:

+ : Additional permissions. Selected permissions are added.
- : Revoke the permissions. Selected permissions are revoked.
= : Specific permissions. Only selected permissions are assigned.

Examples:

1. Remove write permission from group

$ chmod g-w sample.sh

This will only removes the write permission for the group.

2. Add new permission execute to others

$ chmod o+x sample.sh

In addition to the existing permissions, this will add execute permission to others.

3. Give only read permissions to the user

$ chmod u=w sample.sh

This will remove the existing permissions to the user and gives only write permission to the user.

TRANSLATE/ TR COMMAND EXAMPLES IN UNIX AND LINUX TUTORIALS

Tr stands for translate or transliterate. The tr utility in unix or linux system is used to translate, delete or squeeze characters. The syntax of tr command is

tr [options] set1 [set2]

The options of tr command are:

-c : complements the set of characters in string.
-d : deletes the characters in set1
-s : replaces repeated characters listed in the set1 with single occurrence
-t : truncates set1

Tr command Examples:

1. Convert lower case letters to upper case

The following tr command translates the lower case letters to capital letters in the give string:

> echo "linux dedicated server" | tr "[:lower:]" "[:upper:]"
LINUX DEDICATED SERVER
> echo "linux dedicated server" | tr "[a-z]" "[A-Z]"
LINUX DEDICATED SERVER

2. Transform upper case letters to lower case.

Similar to the above example, you can translate the uppercase letters to small letters.

> echo "UNIX DEDICATED SERVER" | tr "[:upper:]" "[:lower:]"
unix dedicated server
> echo "UNIX DEDICATED SERVER" | tr "[A-Z]" "[a-z]"
unix dedicated server

3. Replace non-matching characters.

The -c option is used to replace the non-matching characters with another set of characters.

> echo "unix" | tr -c "u" "a"
uaaa

In the above example, except the character "c" other characters are replaced with "a"

4. Delete non-printable characters

The -d option can be used to delete characters. The following example deletes all the non-printable characters from a file.

> tr -cd "[:print:]" < filename

5. Squeezing characters

You can squeeze more than one occurrence of continuous characters with single occurrence. The following example squeezes two or more successive blank spaces into a single space.

> echo "linux    server" | tr -s " "
linux server

Here you can replace the space character with any other character by specifying in set2.

> "linux    server" | tr -s " " ","
linux,server

6. Delete characters

The following example removes the word linux from the string.

> echo "linuxserver" | tr -d "linux"
server

UNIQ COMMAND EXAMPLES IN UNIX AND LINUX TUTORIALS

Uniq command in unix or linux system is used to suppress the duplicate lines from a file. It discards all the successive identical lines except one from the input and writes the output.

The syntax of uniq command is

uniq [option] filename

The options of uniq command are:

c : Count of occurrence of each line.
d : Prints only duplicate lines.
D : Print all duplicate lines
f : Avoid comparing first N fields.
i : Ignore case when comparing.
s : Avoid comparing first N characters.
u : Prints only unique lines.
w : Compare no more than N characters in lines

Uniq Command Examples:

First create the following example.txt file in your unix or linux operating system.

> cat example.txt
Unix operating system
unix operating system
unix dedicated server
linux dedicated server

1. Suppress duplicate lines

The default behavior of the uniq command is to suppress the duplicate line. Note that, you have to pass sorted input to the uniq, as it compares only successive lines.

> uniq example.txt
unix operating system
unix dedicated server
linux dedicated server

If the lines in the file are not in sorted order, then use the sort command and then pipe the output to the uniq command.

> sort example.txt | uniq

2. Count of lines.

The -c option is used to find how many times each line occurs in the file. It prefixes each line with the count.

> uniq -c example.txt
      2 unix operating system
      1 unix dedicated server
      1 linux dedicated server

3. Display only duplicate lines.

You can print only the lines that occur more than once in a file using the -d option.

> uniq -d example.txt
unix operating system

> uniq -D example.txt
unix operating system
unix operating system

The -D option prints all the duplicate lines.

4. Skip first N fields in comparison.

The -f option is used to skip the first N columns in comparison. Here the fields are delimited by the space character.

> uniq -f2 example.txt
unix operating system
unix dedicated server

In the above example the uniq command, just compares the last fields. For the first two lines, the last field contains the string "system". Uniq prints the first line and skips the second. Similarly it prints the third line and skips the fourth line.

5. Print only unique lines.

You can skip the duplicate lines and print only unique lines using the -u option

> uniq -u example.txt
unix dedicated server
linux dedicated server

HEAD COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

The head command in unix or linux system is used to print the first N lines from the file to the terminal. The syntax of head command is

head [options] [files]

The head command options are:

c : Prints the first N bytes of file; With leading -, prints all but the last N bytes of the file.
n : Prints first N lines; With leading - print all but the last N lines of each file.

Head Command Examples:
Create the following file in your linux or unix operating system for practicing the examples:

> cat example.txt
linux storage
ubuntu os
fedora

1. Display first 10 lines

By default, the head command prints the first 10 lines from a file.

> head example.txt

2. Display first N lines

Use the -n option to print the first n lines from a file. The following example prints the first 2 lines from the file:

> head -n2 example.txt
linux storage
ubuntu os

3. Skip last N lines

You can skip the last N lines from a file and print the remaining lines. The following example skips the last 2 lines and prints the remaining lines.

> head -n-2 example.txt
linux storage

4. Print the first n bytes.

use the -c option to print the first N bytes from the file. The following example prints the first 5 bytes from the file.

> head -c5 example.txt
linux

5. Skip printing last n bytes.

Use the leading "-", to skip printing last N bytes.

> head -c-7 example.txt
linux storage
ubuntu os

6. Print line between M and N lines.

You can combine the head command with tail command to print lines between the line numbers M and N. The following command prints the lines between numbers 5 and 10.

> head -n10 filename | tail -5

UNIX SED COMMAND TO DELETE LINES IN FILE - 15 EXAMPLES

In this article we will see how to delete or remove specific lines that contains a pattern or in a particular position in a file.

As an example, consider the below file with sample data:

> cat file
linux
unix
fedora
debian
ubuntu

1. Delete first line or header line

> sed '1d' file
unix
fedora
debian
ubuntu

The d command is to delete a line. 1d indicates remove the first line. Nd means delete Nth line.

Note that the above command prints the content of the file on the terminal by removing the first line. It actually wont remove the line from the file itself. To remove the line in the original file, use the -i option with sed command. The -i option works only with GNU sed.

> sed -i '1d' file

If you dont have a GNU sed installed, redirect the output of the command to another file.

> sed '1d' file > newfile

2. Delete last line or footer line or trailer line

> sed '$d' file
linux
unix
fedora
debian

Here $ in the above command indicates the last line.

3. Delete particular line. Removes second line our case.

> sed '2d' file
linux
fedora
debian
ubuntu

4. Delete range of lines.

Lets delete lines from 2 to 4. The range is indicated by a comma separated values.

> sed '2,4d' file
linux
ubuntu

5. Delete lines other than the first line or header line

> sed '1!d' file
linux

The ! operator is to negate the command.

6. Delete lines other than last line or footer line

> sed '$!d' file
ubuntu

7. Delete lines other than the specified range

> sed '2,4!d' file
unix
fedora
debian

Removes lines other than 2nd, 3rd and 4th.

8. Delete first and last line

> sed '1d;$d' file
unix
fedora
debian

The semicolon operator is to specify multiple conditions.

9. Delete empty lines or blank lines

> sed '/^$/d' file

The ^$ indicates sed command to delete empty lines. However, this sed wont remove the lines that contain spaces.

10. Delete lines that begin with particular character

> sed '/^u/d' file
linux
fedora
debian

^ is to specify the starting of the line. Above sed command removes all the lines that start with character 'u'.

11. Delete lines that end with particular character.

> sed '/x$/d' file
fedora
debian
ubuntu

$ is to indicate the end of the line. The above command deletes all the lines that end with character 'x'.

12. Delete lines which are in upper case or capital letters

> sed '/^[A-Z]*$/d' file

13. Delete lines that contain a pattern

> sed '/debian/d' file
linux
unix
fedora
ubuntu

14. Delete line starting from a pattern till the last line.

> sed '/fedora/,$d' file
linux
unix

15. Delete last line only if it contains the pattern

> sed '${/ubuntu/d;}' file
linux
unix
fedora
debian

Here $ indicates the last line. If you want to delete a particular line only if it contains a pattern, then put the number of the line in place of the $.

Pages

UNIX INTERVIEW QUESTIONS

UNIX INTERVIEW QUESTIONS ON AWK COMMAND

UNIX INTERVIEW QUESTIONS ON GREP COMMAND

UNIX INTERVIEW QUESTIONS ON SED COMMAND

UNIX INTERVIEW QUESTIONS ON CUT COMMAND

UNIX INTERVIEW QUESTIONS ON FIND COMMAND

TOP UNIX INTERVIEW QUESTIONS - PART 1

TOP UNIX INTERVIEW QUESTIONS - PART 2

TOP UNIX INTERVIEW QUESTIONS - PART 3

TOP UNIX INTERVIEW QUESTIONS - PART 4

TOP UNIX INTERVIEW QUESTIONS - PART 5

TOP UNIX INTERVIEW QUESTIONS - PART 6

TOP UNIX INTERVIEW QUESTIONS - PART 7

TOP UNIX INTERVIEW QUESTIONS - PART 8

TOP EXAMPLES OF AWK COMMAND IN UNIX

EXAMPLES OF AWK COMMAND IN UNIX - PART 2

SED COMMAND IN UNIX AND LINUX EXAMPLES

Sed Command Examples

GREP COMMAND IN UNIX AND LINUX EXAMPLES

FIND COMMAND IN UNIX AND LINUX EXAMPLES

REDIRECT OUTPUT TO MULTIPLE FILES

CONVERT LOWER CASE TO UPPER CASE

CONVERT STRING TO INITCAP IN UNIX

WHAT IS A UNIX OPERATING SYSTEM

UNIX CORE UNIT - KERNEL

UNIX OUTER UNIT - SHELL

FUNCTIONS OF UNIX SHELL

UNIX USER LOGIN PROGRAMS - GETTY AND LOGIN

UNIX FILE SYSTEM

DIFFERENT TYPES OF UNIX FILES

UNIX FILE STRUCTURE (FILE TREE)

SWAPPING AND PAGING IN UNIX

FIND LENGTH OF STRING IN UNIX / LINUX BASH SCRIPT

GREP / PRINT LINES BETWEEN TWO MATCHING PATTERNS/STRINGS - UNIX/LINUX

HOW TO GET HOSTNAME FROM IP ADDRESS - UNIX /LINUX

EXECUTE MYSQL COMMAND IN BASH / SHELL SCRIPT

PASTE COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

HOW TO DELETE SYMBOLIC LINK - UNIX / LINUX

HOSTNAME COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

MAIL COMMAND EXAMPLES IN UNIX / LINUX TUTORIAL

ZIP COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

SORT COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

HOW TO FIND PRIME NUMBER BASH (SHELL) SCRIPT IN UNIX / LINUX

FTP BATCH (BASH) SCRIPT EXAMPLE IN UNIX / LINUX TUTORIAL

GREP UNIQUE OCCURRENCES IN UNIX / LINUX TUTORIALS

TAR COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

GREP COMMAND IN UNIX SHELL SCRIPT

FTP (FILE TRANSFER PROTOCOL) COMMAND EXAMPLES

CVS COMMAND EXAMPLES - UNIX / LINUX TUTORIALS

KILL COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS

SSH COMMAND EXAMPLES - UNIX / LINUX TUTORIALS

WC COMMAND EXAMPLES - COUNT OF LINES, WORDS, CHARACTERS - UNIX / LINUX

SCP COMMAND EXAMPLES - LINUX / UNIX TUTORIALS

XARGS COMMAND EXAMPLES IN UNIX / LINUX TUTORIAL

BASH SHELL SCRIPT TO READ / PARSE COMMA SEPARATED (CSV) FILE - UNIX / LINUX

ADD JOB TO CRON (CRONTAB COMMAND EXAMPLES) - UNIX / LINUX TUTORIALS

YUM COMMAND EXAMPLES - INSTALL, UPDATE, UNINSTALL - LINUX TUTORIALS

FIND AND REMOVE FILES MODIFIED / ACCESSED N DAYS AGO - UNIX / LINUX

REPLACE STRING ON NTH LINE - UNIX / LINUX

DELETE RANGE OF LINES - UNIX / LINUX

REMOVE LAST LINE OR FOOTER LINE - UNIX / LINUX

DELETE FIRST LINE OR HEADER LINE - UNIX / LINUX

PRINT LINES ENDING WITH STRING - UNIX / LINUX

SEARCH (MATCH) FOR WHOLE WORDS IN FILE - UNIX / LINUX

PRINT NON MATCHING LINES (INVERSE OF GREP COMMAND) - UNIX / LINUX

PRINT LINES STARTING WITH STRING - UNIX / LINUX

FIND THE COUNT OF LINES MATCHING THE PATTERN- UNIX /LINUX

REMOVE EMPTY LINES FROM A FILE IN UNIX / LINUX

PRINT N LINES AFTER A PATTERN MATCH - UNIX / LINUX

PRINT LINES BEFORE A PATTERN MATCH - UNIX / LINUX

GREP STRING IN MULTIPLE FILES - UNIX / LINUX

CASE INSENSITIVE GREP COMMAND - UNIX / LINUX

HOW TO SEND MAIL FROM SHELL SCRIPT

CONNECT TO ORACLE DATABASE IN UNIX SHELL SCRIPT

DELETE ALL LINES IN VI / VIM EDITOR - UNIX / LINUX

HOW TO READ LINES USING LOOPS IN SHELL SCRIPTING

EXAMPLES OF ARRAYS IN AWK COMMAND - UNIX / LINUX

CREATE TUNNEL IN UNIX USING PUTTY

VIM EDITOR - SAVE AND QUIT OPTIONS