TOP EXAMPLES OF AWK COMMAND IN UNIX
Awk
is one of the most powerful tools in Unix used for processing the rows
and columns in a file. Awk has built in string functions and associative
arrays. Awk supports most of the operators, conditional blocks, and
loops available in C language.
One of the good things is that you can convert Awk scripts into Perl scripts using a2p utility.
The basic syntax of AWK:
Here the actions in the begin block are performed before processing the file and the actions in the end block are performed after processing the file. The rest of the actions are performed while processing the file.
Examples:
Create a file input_file with the following data. This file can be easily created using the output of ls -l.
From the data, you can observe that this file has rows and columns. The rows are separated by a new line character and the columns are separated by a space characters. We will use this file as the input for the examples discussed here.
1. awk '{print $1}' input_file
Here $1 has a meaning. $1, $2, $3... represents the first, second, third columns... in a row respectively. This awk command will print the first column in each row as shown below.
To print the 4th and 6th columns in a file use awk '{print $4,$5}' input_file
Here the Begin and End blocks are not used in awk. So, the print command will be executed for each row it reads from the file. In the next example we will see how to use the Begin and End blocks.
2. awk 'BEGIN {sum=0} {sum=sum+$5} END {print sum}' input_file
This will prints the sum of the value in the 5th column. In the Begin block the variable sum is assigned with value 0. In the next block the value of 5th column is added to the sum variable. This addition of the 5th column to the sum variable repeats for every row it processed. When all the rows are processed the sum variable will hold the sum of the values in the 5th column. This value is printed in the End block.
3. In this example we will see how to execute the awk script written in a file. Create a file sum_column and paste the below script in that file
Now execute the the script using awk command as
awk -f sum_column input_file.
This will run the script in sum_column file and displays the sum of the 5th column in the input_file.
4. awk '{ if($9 == "t4") print $0;}' input_file
This awk command checks for the string "t4" in the 9th column and if it finds a match then it will print the entire line. The output of this awk command is
5. awk 'BEGIN { for(i=1;i<=5;i++) print "square of", i, "is",i*i; }'
This will print the squares of first numbers from 1 to 5. The output of the command is
Notice that the syntax of “if” and “for” are similar to the C language.
Awk Built in Variables:
You have already seen $0, $1, $2... which prints the entire line, first column, second column... respectively. Now we will see other built in variables with examples.
FS - Input field separator variable:
So far, we have seen the fields separted by a space character. By default Awk assumes that fields in a file are separted by space characters. If the fields in the file are separted by any other character, we can use the FS variable to tell about the delimiter.
6. awk 'BEGIN {FS=":"} {print $2}' input_file
OR
awk -F: '{print $2} input_file
This will print the result as
OFS - Output field separator variable:
By default whenever we printed the fields using the print statement the fields are displayed with space character as delimiter. For example
7. awk '{print $4,$5}' input_file
The output of this command will be
We can change this default behavior using the OFS variable as
awk 'BEGIN {OFS=":"} {print $4,$5}' input_file
Note: print $4,$5 and print $4$5 will not work the same way. The first one displays the output with space as delimiter. The second one displays the output without any delimiter.
NF - Number of fileds variable:
The NF can be used to know the number of fields in line
8. awk '{print NF}' input_file
This will display the number of columns in each row.
NR - number of records variable:
The NR can be used to know the line number or count of lines in a file.
9. awk '{print NR}' input_file
This will display the line numbers from 1.
10. awk 'END {print NR}' input_file
This will display the total number of lines in the file.
String functions in Awk:
Some of the string functions in awk are:
index(string,search)
length(string)
split(string,array,separator)
substr(string,position)
substr(string,position,max)
tolower(string)
toupper(string)
Advanced Examples:
1. Filtering lines using Awk split function
The awk split function splits a string into an array using the delimiter.
The syntax of split function is
split(string, array, delimiter)
Now we will see how to filter the lines using the split function with an example.
The input "file.txt" contains the data in the following format
Required output: Now we have to print only the lines in which whose 2nd field has the string "UNIX" as the 3rd field( The 2nd filed in the line is separated by comma delimiter ).
The ouptut is:
The awk command for getting the output is:
One of the good things is that you can convert Awk scripts into Perl scripts using a2p utility.
The basic syntax of AWK:
awk 'BEGIN {start_action} {action} END {stop_action}' filename
Here the actions in the begin block are performed before processing the file and the actions in the end block are performed after processing the file. The rest of the actions are performed while processing the file.
Examples:
Create a file input_file with the following data. This file can be easily created using the output of ls -l.
-rw-r--r-- 1 center center 0 Dec 8 21:39 p1 -rw-r--r-- 1 center center 17 Dec 8 21:15 t1 -rw-r--r-- 1 center center 26 Dec 8 21:38 t2 -rw-r--r-- 1 center center 25 Dec 8 21:38 t3 -rw-r--r-- 1 center center 43 Dec 8 21:39 t4 -rw-r--r-- 1 center center 48 Dec 8 21:39 t5
From the data, you can observe that this file has rows and columns. The rows are separated by a new line character and the columns are separated by a space characters. We will use this file as the input for the examples discussed here.
1. awk '{print $1}' input_file
Here $1 has a meaning. $1, $2, $3... represents the first, second, third columns... in a row respectively. This awk command will print the first column in each row as shown below.
-rw-r--r-- -rw-r--r-- -rw-r--r-- -rw-r--r-- -rw-r--r-- -rw-r--r--
To print the 4th and 6th columns in a file use awk '{print $4,$5}' input_file
Here the Begin and End blocks are not used in awk. So, the print command will be executed for each row it reads from the file. In the next example we will see how to use the Begin and End blocks.
2. awk 'BEGIN {sum=0} {sum=sum+$5} END {print sum}' input_file
This will prints the sum of the value in the 5th column. In the Begin block the variable sum is assigned with value 0. In the next block the value of 5th column is added to the sum variable. This addition of the 5th column to the sum variable repeats for every row it processed. When all the rows are processed the sum variable will hold the sum of the values in the 5th column. This value is printed in the End block.
3. In this example we will see how to execute the awk script written in a file. Create a file sum_column and paste the below script in that file
#!/usr/bin/awk -f
BEGIN {sum=0}
{sum=sum+$5}
END {print sum}
Now execute the the script using awk command as
awk -f sum_column input_file.
This will run the script in sum_column file and displays the sum of the 5th column in the input_file.
4. awk '{ if($9 == "t4") print $0;}' input_file
This awk command checks for the string "t4" in the 9th column and if it finds a match then it will print the entire line. The output of this awk command is
-rw-r--r-- 1 pcenter pcenter 43 Dec 8 21:39 t4
5. awk 'BEGIN { for(i=1;i<=5;i++) print "square of", i, "is",i*i; }'
This will print the squares of first numbers from 1 to 5. The output of the command is
square of 1 is 1 square of 2 is 4 square of 3 is 9 square of 4 is 16 square of 5 is 25
Notice that the syntax of “if” and “for” are similar to the C language.
Awk Built in Variables:
You have already seen $0, $1, $2... which prints the entire line, first column, second column... respectively. Now we will see other built in variables with examples.
FS - Input field separator variable:
So far, we have seen the fields separted by a space character. By default Awk assumes that fields in a file are separted by space characters. If the fields in the file are separted by any other character, we can use the FS variable to tell about the delimiter.
6. awk 'BEGIN {FS=":"} {print $2}' input_file
OR
awk -F: '{print $2} input_file
This will print the result as
39 p1 15 t1 38 t2 38 t3 39 t4 39 t5
OFS - Output field separator variable:
By default whenever we printed the fields using the print statement the fields are displayed with space character as delimiter. For example
7. awk '{print $4,$5}' input_file
The output of this command will be
center 0 center 17 center 26 center 25 center 43 center 48
We can change this default behavior using the OFS variable as
awk 'BEGIN {OFS=":"} {print $4,$5}' input_file
center:0 center:17 center:26 center:25 center:43 center:48
Note: print $4,$5 and print $4$5 will not work the same way. The first one displays the output with space as delimiter. The second one displays the output without any delimiter.
NF - Number of fileds variable:
The NF can be used to know the number of fields in line
8. awk '{print NF}' input_file
This will display the number of columns in each row.
NR - number of records variable:
The NR can be used to know the line number or count of lines in a file.
9. awk '{print NR}' input_file
This will display the line numbers from 1.
10. awk 'END {print NR}' input_file
This will display the total number of lines in the file.
String functions in Awk:
Some of the string functions in awk are:
index(string,search)
length(string)
split(string,array,separator)
substr(string,position)
substr(string,position,max)
tolower(string)
toupper(string)
Advanced Examples:
1. Filtering lines using Awk split function
The awk split function splits a string into an array using the delimiter.
The syntax of split function is
split(string, array, delimiter)
Now we will see how to filter the lines using the split function with an example.
The input "file.txt" contains the data in the following format
1 U,N,UNIX,000 2 N,P,SHELL,111 3 I,M,UNIX,222 4 X,Y,BASH,333 5 P,R,SCRIPT,444
Required output: Now we have to print only the lines in which whose 2nd field has the string "UNIX" as the 3rd field( The 2nd filed in the line is separated by comma delimiter ).
The ouptut is:
1 U,N,UNIX,000 3 I,M,UNIX,222
The awk command for getting the output is:
awk '{
split($2,arr,",");
if(arr[3] == "UNIX")
print $0
} ' file.txt
EXAMPLES OF AWK COMMAND IN UNIX - PART 2
1. Inserting a new line after every 2 lines
We will see how to implement this using the awk command with an example.
The input "file.txt" contains the below data:
Let say, we want to insert the new line "9 Z" after every two lines in the input file. The required output data after inserting a new line looks as
The awk command for getting this output is
2. Replace the Nth occurrence of a pattern
The input file contains the data.
Now we want to replace the fourth occurrence of the first filed "AAA" with "ZZZ" in the file.
The required output is:
The awk command for getting this output is
3. Find the sum of even and odd lines separately
The input file data:
You have to get the second field and then find the sum the even and odd lines.
The required output is
The awk command for producing this output is
4. Fibonacci series using awk command
Now we will produce the Fibonacci series using the awk command.
The output is
5. Remove leading zeros from a file using the awk command. The input file contains the below data.
After removing the leading zeros, the output should contain the below data.
The awk command for this is.
We will see how to implement this using the awk command with an example.
The input "file.txt" contains the below data:
1 A 2 B 3 C 4 D 5 E 6 F
Let say, we want to insert the new line "9 Z" after every two lines in the input file. The required output data after inserting a new line looks as
1 A 2 B 9 Z 3 C 4 D 9 Z 5 E 6 F 9 Z
The awk command for getting this output is
awk '{
if(NR%2 == 0)
{
print $0"\n9 Z";
}
else
{
print $0
}
}' file.txt
2. Replace the Nth occurrence of a pattern
The input file contains the data.
AAA 1 BBB 2 CCC 3 AAA 4 AAA 5 BBB 6 CCC 7 AAA 8 BBB 9 AAA 0
Now we want to replace the fourth occurrence of the first filed "AAA" with "ZZZ" in the file.
The required output is:
AAA 1 BBB 2 CCC 3 AAA 4 AAA 5 BBB 6 CCC 7 ZZZ 8 BBB 9 AAA 0
The awk command for getting this output is
awk 'BEGIN {count=0}
{
if($1 == "AAA")
{
count++
}
if(count == 4)
{
sub("AAA","ZZZ",$1)
}
}
{
print $0
}' file.txt
3. Find the sum of even and odd lines separately
The input file data:
A 10 B 39 C 22 D 44 E 75 F 89 G 67
You have to get the second field and then find the sum the even and odd lines.
The required output is
174, 172
The awk command for producing this output is
awk '{
if(NR%2 == 1)
{
sum_e = sum_e + $2
}
else
{
sum_o = sum_o + $2
}
}
END { print sum_e,sum_o }' file.txt
4. Fibonacci series using awk command
Now we will produce the Fibonacci series using the awk command.
awk ' BEGIN{
for(i=0;i<=10;i++)
{
if (i <=1 )
{
x=0;
y=1;
print i;
}
else
{
z=x+y;
print z;
x=y;
y=z;
}
}
}'
The output is
0 1 1 2 3 5 8 13 21 34 55
5. Remove leading zeros from a file using the awk command. The input file contains the below data.
0012345 05678 01010 00001
After removing the leading zeros, the output should contain the below data.
12345 5678 1010 1
The awk command for this is.
awk '{print $1 + 0}' file.txt
awk '{printf "%d\n",$0}' file.txt
SED COMMAND IN UNIX AND LINUX EXAMPLES
Sed
is a Stream Editor used for modifying the files in unix (or linux).
Whenever you want to make changes to the file automatically, sed comes
in handy to do this. Most people never learn its power; they just simply
use sed to replace text. You can do many things apart from replacing
text with sed. Here I will describe the features of sed with examples.
Consider the below text file as an input.
1. Replacing or substituting string
Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.
Here the "s" specifies the substitution operation. The "/" are delimiters. The "unix" is the search pattern and the "linux" is the replacement string.
By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line.
2. Replacing the nth occurrence of a pattern in a line.
Use the /1, /2 etc flags to replace the first, second occurrence of a pattern in a line. The below command replaces the second occurrence of the word "unix" with "linux" in a line.
3. Replacing all the occurrence of the pattern in a line.
The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.
4. Replacing from nth occurrence to all occurrences in a line.
Use the combination of /1, /2 etc and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... "unix" word with "linux" word in a line.
5. Changing the slash (/) delimiter
You can use any delimiter other than the slash. As an example if you want to change the web url to another url as
In this case the url consists the delimiter character which we used. In that case you have to escape the slash with backslash character, otherwise the substitution won't work.
Using too many backslashes makes the sed command look awkward. In this case we can change the delimiter to another character as shown in the below example.
6. Using & as the matched string
There might be cases where you want to search for the pattern and replace that pattern by adding some extra characters to it. In such cases & comes in handy. The & represents the matched string.
7. Using \1,\2 and so on to \9
The first pair of parenthesis specified in the pattern represents the \1, the second represents the \2 and so on. The \1,\2 can be used in the replacement string to make changes to the source string. As an example, if you want to replace the word "unix" in a line with twice as the word like "unixunix" use the sed command as below.
The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is
Another example is switching the first three characters in a line
8. Duplicating the replaced line with /p flag
The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.
9. Printing only the replaced lines
Use the -n option along with the /p print flag to display only the replaced lines. Here the -n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.
If you use -n alone without /p, then the sed does not print anything.
10. Running multiple sed commands.
You can run multiple sed commands by piping the output of one sed command as input to another sed command.
Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.
11. Replacing string on a specific line number.
You can restrict the sed command to replace the string on a specific line number. An example is
The above sed command replaces the string only on the third line.
12. Replacing string on a range of lines.
You can specify a range of line numbers to the sed command for replacing a string.
Here the sed command replaces the lines with range from 1 to 3. Another example is
Here $ indicates the last line in the file. So the sed command replaces the text from second line to last line in the file.
13. Replace on a lines which matches a pattern.
You can specify a pattern to the sed command to match in a line. If the pattern match occurs, then only the sed command looks for the string to be replaced and if it finds, then the sed command replaces the string.
Here the sed command first looks for the lines which has the pattern "linux" and then replaces the word "unix" with "centos".
14. Deleting lines.
You can delete the lines a file by specifying the line number or a range or numbers.
15. Duplicating lines
You can make the sed command to print each line of a file two times.
16. Sed as grep command
You can make sed command to work as similar to grep command.
Here the sed command looks for the pattern "unix" in each line of a file and prints those lines that has the pattern.
You can also make the sed command to work as grep -v, just by using the reversing the sed with NOT (!).
The ! here inverts the pattern match.
17. Add a line after a match.
The sed command can add a new line after a pattern match is found. The "a" command to sed tells it to add a new line after a match is found.
18. Add a line before a match
The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.
19. Change a line
The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.
20. Transform like tr command
The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.
Here the sed command transforms the alphabets "ul" into their uppercase format "UL"
Consider the below text file as an input.
>cat file.txt unix is great os. unix is opensource. unix is free os. learn operating system. unixlinux which one you choose.
Sed Command Examples
1. Replacing or substituting string
Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.
>sed 's/unix/linux/' file.txt linux is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose.
Here the "s" specifies the substitution operation. The "/" are delimiters. The "unix" is the search pattern and the "linux" is the replacement string.
By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line.
2. Replacing the nth occurrence of a pattern in a line.
Use the /1, /2 etc flags to replace the first, second occurrence of a pattern in a line. The below command replaces the second occurrence of the word "unix" with "linux" in a line.
>sed 's/unix/linux/2' file.txt unix is great os. linux is opensource. unix is free os. learn operating system. unixlinux which one you choose.
3. Replacing all the occurrence of the pattern in a line.
The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.
>sed 's/unix/linux/g' file.txt linux is great os. linux is opensource. linux is free os. learn operating system. linuxlinux which one you choose.
4. Replacing from nth occurrence to all occurrences in a line.
Use the combination of /1, /2 etc and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... "unix" word with "linux" word in a line.
>sed 's/unix/linux/3g' file.txt unix is great os. unix is opensource. linux is free os. learn operating system. unixlinux which one you choose.
5. Changing the slash (/) delimiter
You can use any delimiter other than the slash. As an example if you want to change the web url to another url as
>sed 's/http:\/\//www/' file.txt
In this case the url consists the delimiter character which we used. In that case you have to escape the slash with backslash character, otherwise the substitution won't work.
Using too many backslashes makes the sed command look awkward. In this case we can change the delimiter to another character as shown in the below example.
>sed 's_http://_www_' file.txt >sed 's|http://|www|' file.txt
6. Using & as the matched string
There might be cases where you want to search for the pattern and replace that pattern by adding some extra characters to it. In such cases & comes in handy. The & represents the matched string.
>sed 's/unix/{&}/' file.txt
{unix} is great os. unix is opensource. unix is free os.
learn operating system.
{unix}linux which one you choose.
>sed 's/unix/{&&}/' file.txt
{unixunix} is great os. unix is opensource. unix is free os.
learn operating system.
{unixunix}linux which one you choose.
7. Using \1,\2 and so on to \9
The first pair of parenthesis specified in the pattern represents the \1, the second represents the \2 and so on. The \1,\2 can be used in the replacement string to make changes to the source string. As an example, if you want to replace the word "unix" in a line with twice as the word like "unixunix" use the sed command as below.
>sed 's/\(unix\)/\1\1/' file.txt unixunix is great os. unix is opensource. unix is free os. learn operating system. unixunixlinux which one you choose.
The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is
>sed 's/\(unix\)\(linux\)/\2\1/' file.txt unix is great os. unix is opensource. unix is free os. learn operating system. linuxunix which one you choose.
Another example is switching the first three characters in a line
>sed 's/^\(.\)\(.\)\(.\)/\3\2\1/' file.txt inux is great os. unix is opensource. unix is free os. aelrn operating system. inuxlinux which one you choose.
8. Duplicating the replaced line with /p flag
The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.
>sed 's/unix/linux/p' file.txt linux is great os. unix is opensource. unix is free os. linux is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose. linuxlinux which one you choose.
9. Printing only the replaced lines
Use the -n option along with the /p print flag to display only the replaced lines. Here the -n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.
>sed -n 's/unix/linux/p' file.txt linux is great os. unix is opensource. unix is free os. linuxlinux which one you choose.
If you use -n alone without /p, then the sed does not print anything.
10. Running multiple sed commands.
You can run multiple sed commands by piping the output of one sed command as input to another sed command.
>sed 's/unix/linux/' file.txt| sed 's/os/system/' linux is great system. unix is opensource. unix is free os. learn operating system. linuxlinux which one you chosysteme.
Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.
>sed -e 's/unix/linux/' -e 's/os/system/' file.txt linux is great system. unix is opensource. unix is free os. learn operating system. linuxlinux which one you chosysteme.
11. Replacing string on a specific line number.
You can restrict the sed command to replace the string on a specific line number. An example is
>sed '3 s/unix/linux/' file.txt unix is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose.
The above sed command replaces the string only on the third line.
12. Replacing string on a range of lines.
You can specify a range of line numbers to the sed command for replacing a string.
>sed '1,3 s/unix/linux/' file.txt linux is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose.
Here the sed command replaces the lines with range from 1 to 3. Another example is
>sed '2,$ s/unix/linux/' file.txt linux is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose.
Here $ indicates the last line in the file. So the sed command replaces the text from second line to last line in the file.
13. Replace on a lines which matches a pattern.
You can specify a pattern to the sed command to match in a line. If the pattern match occurs, then only the sed command looks for the string to be replaced and if it finds, then the sed command replaces the string.
>sed '/linux/ s/unix/centos/' file.txt unix is great os. unix is opensource. unix is free os. learn operating system. centoslinux which one you choose.
Here the sed command first looks for the lines which has the pattern "linux" and then replaces the word "unix" with "centos".
14. Deleting lines.
You can delete the lines a file by specifying the line number or a range or numbers.
>sed '2 d' file.txt >sed '5,$ d' file.txt
15. Duplicating lines
You can make the sed command to print each line of a file two times.
>sed 'p' file.txt
16. Sed as grep command
You can make sed command to work as similar to grep command.
>grep 'unix' file.txt >sed -n '/unix/ p' file.txt
Here the sed command looks for the pattern "unix" in each line of a file and prints those lines that has the pattern.
You can also make the sed command to work as grep -v, just by using the reversing the sed with NOT (!).
>grep -v 'unix' file.txt >sed -n '/unix/ !p' file.txt
The ! here inverts the pattern match.
17. Add a line after a match.
The sed command can add a new line after a pattern match is found. The "a" command to sed tells it to add a new line after a match is found.
>sed '/unix/ a "Add a new line"' file.txt unix is great os. unix is opensource. unix is free os. "Add a new line" learn operating system. unixlinux which one you choose. "Add a new line"
18. Add a line before a match
The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.
>sed '/unix/ i "Add a new line"' file.txt "Add a new line" unix is great os. unix is opensource. unix is free os. learn operating system. "Add a new line" unixlinux which one you choose.
19. Change a line
The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.
>sed '/unix/ c "Change line"' file.txt "Change line" learn operating system. "Change line"
20. Transform like tr command
The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.
>sed 'y/ul/UL/' file.txt Unix is great os. Unix is opensoUrce. Unix is free os. Learn operating system. UnixLinUx which one yoU choose.
Here the sed command transforms the alphabets "ul" into their uppercase format "UL"
GREP COMMAND IN UNIX AND LINUX EXAMPLES
Grep
is the frequently used command in Unix (or Linux). Most of us use grep
just for finding the words in a file. The power of grep comes with using
its options and regular expressions. You can analyze large sets of log
files with the help of grep command.
Grep stands for Global search for Regular Expressions and Print.
The basic syntax of grep command is
grep [options] pattern [list of files]
Let see some practical examples on grep command.
1. Running the last executed grep command
This saves a lot of time if you are executing the same command again and again.
2. Search for a string in a file
This is the basic usage of grep command. It searches for the given string in the specified file.
3. Searching for a string in multiple files.
4. Case insensitive search
The -i option enables to search for a string case insensitively in the give file. It matches the words like "UNIX", "Unix", "unix".
5. Specifying the search string as a regular expression pattern.
6. Checking for the whole words in a file.
By default, grep matches the given string/pattern even if it found as a substring in a file. The -w option to grep makes it match only the whole words.
7. Displaying the lines before the match.
Some times, if you are searching for an error in a log file; it is always good to know the lines around the error lines to know the cause of the error.
8. Displaying the lines after the match.
9. Displaying the lines around the match
10. Searching for a sting in all files recursively
You can search for a string in all the files under the current directory and sub-directories with the help -r option.
11. Inverting the pattern match
You can display the lines that are not matched with the specified search sting pattern using the -v option.
12. Displaying the non-empty lines
You can remove the blank lines using the grep command.
13. Displaying the count of number of matches.
We can find the number of lines that matches the given string/pattern
14. Display the file names that matches the pattern.
We can just display the files that contains the given string/pattern.
15. Display the file names that do not contain the pattern.
We can display the files which do not contain the matched string/pattern.
16. Displaying only the matched pattern.
By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.
17. Displaying the line numbers.
We can make the grep command to display the position of the line which contains the matched string in a file using the -n option
18. Displaying the position of the matched string in the line
The -b option allows the grep command to display the character position of the matched string in a file.
19. Matching the lines that start with a string
The ^ regular expression pattern specifies the start of a line. This can be used in grep to match the lines which start with the given string or pattern.
20. Matching the lines that end with a string
The $ regular expression pattern specifies the end of a line. This can be used in grep to match the lines which end with the given string or pattern.
Grep stands for Global search for Regular Expressions and Print.
The basic syntax of grep command is
grep [options] pattern [list of files]
Let see some practical examples on grep command.
1. Running the last executed grep command
This saves a lot of time if you are executing the same command again and again.
!grepThis displays the last executed grep command and also prints the result set of the command on the terminal.
2. Search for a string in a file
This is the basic usage of grep command. It searches for the given string in the specified file.
grep "Error" logfile.txtThis searches for the string "Error" in the log file and prints all the lines that has the word "Error".
3. Searching for a string in multiple files.
grep "string" file1 file2 grep "string" file_patternThis is also the basic usage of the grep command. You can manually specify the list of files you want to search or you can specify a file pattern (use regular expressions) to search for.
4. Case insensitive search
The -i option enables to search for a string case insensitively in the give file. It matches the words like "UNIX", "Unix", "unix".
grep -i "UNix" file.txt
5. Specifying the search string as a regular expression pattern.
grep "^[0-9].*" file.txtThis will search for the lines which starts with a number. Regular expressions is huge topic and I am not covering it here. This example is just for providing the usage of regular expressions.
6. Checking for the whole words in a file.
By default, grep matches the given string/pattern even if it found as a substring in a file. The -w option to grep makes it match only the whole words.
grep -w "world" file.txt
7. Displaying the lines before the match.
Some times, if you are searching for an error in a log file; it is always good to know the lines around the error lines to know the cause of the error.
grep -B 2 "Error" file.txtThis will prints the matched lines along with the two lines before the matched lines.
8. Displaying the lines after the match.
grep -A 3 "Error" file.txtThis will display the matched lines along with the three lines after the matched lines.
9. Displaying the lines around the match
grep -C 5 "Error" file.txtThis will display the matched lines and also five lines before and after the matched lines.
10. Searching for a sting in all files recursively
You can search for a string in all the files under the current directory and sub-directories with the help -r option.
grep -r "string" *
11. Inverting the pattern match
You can display the lines that are not matched with the specified search sting pattern using the -v option.
grep -v "string" file.txt
12. Displaying the non-empty lines
You can remove the blank lines using the grep command.
grep -v "^$" file.txt
13. Displaying the count of number of matches.
We can find the number of lines that matches the given string/pattern
grep -c "sting" file.txt
14. Display the file names that matches the pattern.
We can just display the files that contains the given string/pattern.
grep -l "string" file.txt
15. Display the file names that do not contain the pattern.
We can display the files which do not contain the matched string/pattern.
grep -l "string" file.txt
16. Displaying only the matched pattern.
By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.
grep -o "string" file.txt
17. Displaying the line numbers.
We can make the grep command to display the position of the line which contains the matched string in a file using the -n option
grep -n "string" file.txt
18. Displaying the position of the matched string in the line
The -b option allows the grep command to display the character position of the matched string in a file.
grep -o -b "string" file.txt
19. Matching the lines that start with a string
The ^ regular expression pattern specifies the start of a line. This can be used in grep to match the lines which start with the given string or pattern.
grep "^start" file.txt
20. Matching the lines that end with a string
The $ regular expression pattern specifies the end of a line. This can be used in grep to match the lines which end with the given string or pattern.
grep "end$" file.txt
FIND COMMAND IN UNIX AND LINUX EXAMPLES
Find
is one of the powerful utility of Unix (or Linux) used for searching
the files in a directory hierarchy. The syntax of find command is
Let see some practical exercises on using find command.
1. How to run the last executed find command?
This will execute the last find command. It also displays the last find command executed along with the result on the terminal.
2. How to find for a file using name?
This will find all the files with name "sum.java" in the current directory and sub-directories.
3. How to find for files using name and ignoring case?
This will find all the files with name "sum.java" while ignoring the case in the current directory and sub-directories.
4. How to find for a file in the current directory only?
This will find for the file "sum.java" in the current directory only
5. How to find for files containing a specific word in its name?
It displayed all the files which have the word "java" in the filename
6. How to find for files in a specific directory?
This will look for the files in the /etc directory with "java" in the filename
7. How to find the files whose name are not "sum.java"?
This is like inverting the match. It prints all the files except the given file "sum.java".
8. How to limit the file searches to specific directories?
You can see here the find command displayed all the files with name "sum.java" in the current directory and sub-directories.
a. How to print the files in the current directory and one level down to the current directory?
b. How to print the files in the current directory and two levels down to the current directory?
c. How to print the files in the subdirectories between level 1 and 4?
9. How to find the empty files in a directory?
10. How to find the largest file in the current directory and sub directories
The find command "find . -type f -exec ls -s {} \;" will list all the files along with the size of the file. Then the sort command will sort the files based on the size. The head command will pick only the first line from the output of sort.
11. How to find the smallest file in the current directory and sub directories
Another method using find is
12. How to find files based on the file type?
a. Finding socket files
b. Finding directories
c. Finding hidden directories
d. Finding regular files
e. Finding hidden files
13. How to find files based on the size?
a. Finding files whose size is exactly 10M
b. Finding files larger than 10M size
c. Finding files smaller than 10M size
14. How to find the files which are modified after the modification of a give file.
This will display all the files which are modified after the file "sum.java"
15. Display the files which are accessed after the modification of a give file.
16. Display the files which are changed after the modification of a give file.
17. How to find the files based on the file permissions?
This will display the files which have read, write, and execute permissions. To know the permissions of files and directories use the command "ls -l".
18. Find the files which are modified within 30 minutes.
19. Find the files which are modified within 1 day.
20. How to find the files which are modified 30 minutes back
21. How to find the files which are modified 1 day back.
22. Print the files which are accessed within 1 hour.
23. Print the files which are accessed within 1 day.
24. Display the files which are changed within 2 hours.
25. Display the files which are changed within 2 days.
26. How to find the files which are created between two files.
So far we have just find the files and displayed on the terminal. Now we will see how to perform some operations on the files.
1. How to find the permissions of the files which contain the name "java"?
Alternate method is
2. Find the files which have the name "java" in it and then display only the files which have "class" word in them?
3. How to remove files which contain the name "java".
This will delete all the files which have the word “java" in the file name in the current directory and sub-directories.
find [pathnames] [conditions]
Let see some practical exercises on using find command.
1. How to run the last executed find command?
!find
This will execute the last find command. It also displays the last find command executed along with the result on the terminal.
2. How to find for a file using name?
find -name "sum.java" ./bkp/sum.java ./sum.java
This will find all the files with name "sum.java" in the current directory and sub-directories.
3. How to find for files using name and ignoring case?
find -iname "sum.java" ./SUM.java ./bkp/sum.java ./sum.java
This will find all the files with name "sum.java" while ignoring the case in the current directory and sub-directories.
4. How to find for a file in the current directory only?
find -maxdepth 1 -name "sum.java" ./sum.java
This will find for the file "sum.java" in the current directory only
5. How to find for files containing a specific word in its name?
find -name "*java*" ./SUM.java ./bkp/sum.java ./sum.java ./multiply.java
It displayed all the files which have the word "java" in the filename
6. How to find for files in a specific directory?
find /etc -name "*java*"
This will look for the files in the /etc directory with "java" in the filename
7. How to find the files whose name are not "sum.java"?
find -not -name "sum.java" . ./SUM.java ./bkp ./multiply.java
This is like inverting the match. It prints all the files except the given file "sum.java".
8. How to limit the file searches to specific directories?
find -name "sum.java" ./tmp/sum.java ./bkp/var/tmp/files/sum.java ./bkp/var/tmp/sum.java ./bkp/var/sum.java ./bkp/sum.java ./sum.java
You can see here the find command displayed all the files with name "sum.java" in the current directory and sub-directories.
a. How to print the files in the current directory and one level down to the current directory?
find -maxdepth 2 -name "sum.java" ./tmp/sum.java ./bkp/sum.java ./sum.java
b. How to print the files in the current directory and two levels down to the current directory?
find -maxdepth 3 -name "sum.java" ./tmp/sum.java ./bkp/var/sum.java ./bkp/sum.java ./sum.java
c. How to print the files in the subdirectories between level 1 and 4?
find -mindepth 2 -maxdepth 5 -name "sum.java" ./tmp/sum.java ./bkp/var/tmp/files/sum.java ./bkp/var/tmp/sum.java ./bkp/var/sum.java ./bkp/sum.java
9. How to find the empty files in a directory?
find . -maxdepth 1 -empty ./empty_file
10. How to find the largest file in the current directory and sub directories
find . -type f -exec ls -s {} \; | sort -n -r | head -1
The find command "find . -type f -exec ls -s {} \;" will list all the files along with the size of the file. Then the sort command will sort the files based on the size. The head command will pick only the first line from the output of sort.
11. How to find the smallest file in the current directory and sub directories
find . -type f -exec ls -s {} \; | sort -n -r | tail -1
Another method using find is
find . -type f -exec ls -s {} \; | sort -n | head -1
12. How to find files based on the file type?
a. Finding socket files
find . -type s
b. Finding directories
find . -type d
c. Finding hidden directories
find -type d -name ".*"
d. Finding regular files
find . -type f
e. Finding hidden files
find . -type f -name ".*"
13. How to find files based on the size?
a. Finding files whose size is exactly 10M
find . -size 10M
b. Finding files larger than 10M size
find . -size +10M
c. Finding files smaller than 10M size
find . -size -10M
14. How to find the files which are modified after the modification of a give file.
find -newer "sum.java"
This will display all the files which are modified after the file "sum.java"
15. Display the files which are accessed after the modification of a give file.
find -anewer "sum.java"
16. Display the files which are changed after the modification of a give file.
find -cnewer "sum.java"
17. How to find the files based on the file permissions?
find . -perm 777
This will display the files which have read, write, and execute permissions. To know the permissions of files and directories use the command "ls -l".
18. Find the files which are modified within 30 minutes.
find . -mmin -30
19. Find the files which are modified within 1 day.
find . -mtime -1
20. How to find the files which are modified 30 minutes back
find . -not -mmin -30
21. How to find the files which are modified 1 day back.
find . -not -mtime -1
22. Print the files which are accessed within 1 hour.
find . -amin -60
23. Print the files which are accessed within 1 day.
find . -atime -1
24. Display the files which are changed within 2 hours.
find . -cmin -120
25. Display the files which are changed within 2 days.
find . -ctime -2
26. How to find the files which are created between two files.
find . -cnewer f1 -and ! -cnewer f2
So far we have just find the files and displayed on the terminal. Now we will see how to perform some operations on the files.
1. How to find the permissions of the files which contain the name "java"?
find -name "*java*"|xargs ls -l
Alternate method is
find -name "*java*" -exec ls -l {} \;
2. Find the files which have the name "java" in it and then display only the files which have "class" word in them?
find -name "*java*" -exec grep -H class {} \;
3. How to remove files which contain the name "java".
find -name "*java*" -exec rm -r {} \;
This will delete all the files which have the word “java" in the file name in the current directory and sub-directories.
REDIRECT OUTPUT TO MULTIPLE FILES
The tee command in unix writes the output to multiple files and also displays the output on terminal.
Example:
date | tee -a file1 file2 file3
For more details look at "man tee"
Example:
date | tee -a file1 file2 file3
For more details look at "man tee"
CONVERT LOWER CASE TO UPPER CASE
'tr'
command will convert one set of characters to another set. The
following command converts lower case alphabets in to upper case.
echo "apple" | tr [a-z] [A-Z]
Similarly to convert from upper case to lower case, use the following command
echo "APPLE" | tr [A-Z] [a-z]
For more details on tr look at "man tr".
CONVERT STRING TO INITCAP IN UNIX
The following unix command converts the first letter in a string to upper case and the remaining letters to lower case.
echo apple | awk '{print toupper(substr($1,1,1)) tolower(substr($1,2))}'
echo apple | awk '{print toupper(substr($1,1,1)) tolower(substr($1,2))}'
WHAT IS A UNIX OPERATING SYSTEM
Unix
is a multi-tasking, multi-user operating system. It is a layer between
the hardware and the applications that run on the computer. It has
functions which manage the hardware and applications.
The structure of Unix operating system can be divided into three parts.
- Kernel is the core part of Unix which interacts with the hardware for low level functions.
- Shell is the outer unit of Unix which interacts with the user to perform the functions.
- File System.
UNIX CORE UNIT - KERNEL
The
kernel is the heart of a UNIX system and manages the hardware,
executing processes etc. When the computer is booted, kernel is loaded
into the computer's main memory and it remains there until the computer
is shut down. The kernel performs many low-level and system-level
functions.
The tasks of kernel include
- Interpreting and sending basic instructions to the computer's processor.
- Running and scheduling the processes.
- Allocating the necessary hardware.
- Controlling the I/O operations.
UNIX OUTER UNIT - SHELL
The
instructions to the kernel are complex and highly technical. To protect
the kernel from the short comings of user, a shell is built around the
kernel. The Shell acts like a mediator between the user and the kernel.
Whenever a user run a command, the shell interprets the command and
passes the command to the kernel.
Three types of shell are standard in Unix
Three types of shell are standard in Unix
- Bourne shell is developed by Stephen Bourne. It is the most widely used shell and is a program with name sh. The bourne shell prompts with $ symbol
- Korn shell is developed by David Korn. The korn shell has additional features than bourne shell and is called by the name ksh.
- C shell is developed by Bill Joy and is called by the name csh.
FUNCTIONS OF UNIX SHELL
Some of the basic functions of shell are:
- Command line interpretation
- Program initiation
- Input-output redirection
- Pipeline connection
- Substitution of filenames
- Maintenance of variables
- Environment control
- Shell programming
UNIX USER LOGIN PROGRAMS - GETTY AND LOGIN
The
Kernel should know who the user is logging in and how to communicate
with the user. To do this the kernel invokes two programs, getty and
login.
The kernel invokes the getty program for every user terminal. When the getty program receives input from the user, it invokes the login program. The login program verifies the identity of the user by checking the password file. If the user fails to provide valid password, the login program returns the control back to the getty program. If the user enters a valid password, the login program takes the user to the shell prompt.
The kernel invokes the getty program for every user terminal. When the getty program receives input from the user, it invokes the login program. The login program verifies the identity of the user by checking the password file. If the user fails to provide valid password, the login program returns the control back to the getty program. If the user enters a valid password, the login program takes the user to the shell prompt.
UNIX FILE SYSTEM
The
strength of the Unix lies in treating the files in a consistent way.
For Unix a file is a file. This consistency makes it easy to work with
files and the user does not have to learn special commands for new
tasks. The user can write Unix programs easily without worrying about
whether he’s communicating to a terminal, a printer, or an ordinary file
on a disk drive.
For example a "cat" command can be used to display the contents of a file on terminal screen and can also send the file to a printer. As far as Unix is concerned the terminal and the printer are files just as other files.
For example a "cat" command can be used to display the contents of a file on terminal screen and can also send the file to a printer. As far as Unix is concerned the terminal and the printer are files just as other files.
DIFFERENT TYPES OF UNIX FILES
There are mainly three types of Unix files. They are
Regular files hold data and executable programs. Executable programs are the commands (ls) that you enter on the prompt. The data can be anything and there is no specific format enforced in the way the data is stored.
The regular files can be visualized as the leaves in the UNIX tree.
Directories
Directories are files that contain other files and sub-directories. Directories are used to organize the data by keeping closely related files in the same place. The directories are just like the folders in windows operating system.
The kernel alone can write the directory file. When a file is added to or deleted from this directory, the kernel makes an entry.
A directory file can be visualized as the branch of the UNIX tree.
Special Or Device Files
These files represent the physical devices. Files can also refer to computer hardware such as terminals and printers. These device files can also refer to tape and disk drives, CD-ROM players, modems, network interfaces, scanners, and any other piece of computer hardware. When a process writes to a special file, the data is sent to the physical device associated with it. Special files are not literally files, but are pointers that point to the device drivers located in the kernel. The protection applicable to files is also applicable to physical devices.
- Regular files
- Directories
- Special or Device files
Regular Files
Regular files hold data and executable programs. Executable programs are the commands (ls) that you enter on the prompt. The data can be anything and there is no specific format enforced in the way the data is stored.
The regular files can be visualized as the leaves in the UNIX tree.
Directories
Directories are files that contain other files and sub-directories. Directories are used to organize the data by keeping closely related files in the same place. The directories are just like the folders in windows operating system.
The kernel alone can write the directory file. When a file is added to or deleted from this directory, the kernel makes an entry.
A directory file can be visualized as the branch of the UNIX tree.
Special Or Device Files
These files represent the physical devices. Files can also refer to computer hardware such as terminals and printers. These device files can also refer to tape and disk drives, CD-ROM players, modems, network interfaces, scanners, and any other piece of computer hardware. When a process writes to a special file, the data is sent to the physical device associated with it. Special files are not literally files, but are pointers that point to the device drivers located in the kernel. The protection applicable to files is also applicable to physical devices.
UNIX FILE STRUCTURE (FILE TREE)
The
Unix file structure is organized in a reverse tree structure manner.
The following figure shows a typical organization of files in Unix
system.
The diagram looks like any upside-down tree. The slash (/) indicates the root directory. Names like etc, usr, local are directories and science.txt is a file. The regular files in Unix are the leaves in a tree structure.
The diagram looks like any upside-down tree. The slash (/) indicates the root directory. Names like etc, usr, local are directories and science.txt is a file. The regular files in Unix are the leaves in a tree structure.
SWAPPING AND PAGING IN UNIX
Swapping
The whole process in swapping is moved from the swap device to the main memory for execution. The process size must be less than or equal to the available main memory. It is easier to implementation and overhead to the system. Swapping systems does not handle the memory more flexibly as compared to the paging systems.
Paging
Only the required memory pages are moved to main memory from the swap device for execution. The process size does not matter. Paging gives the concept of the virtual memory. It provides greater flexibility in mapping the virtual address space into the physical memory of the machine. It allows more number of processes to fit in the main memory simultaneously and allows the greater process size than the available physical memory. Demand paging systems handle the memory more flexibly.
The whole process in swapping is moved from the swap device to the main memory for execution. The process size must be less than or equal to the available main memory. It is easier to implementation and overhead to the system. Swapping systems does not handle the memory more flexibly as compared to the paging systems.
Paging
Only the required memory pages are moved to main memory from the swap device for execution. The process size does not matter. Paging gives the concept of the virtual memory. It provides greater flexibility in mapping the virtual address space into the physical memory of the machine. It allows more number of processes to fit in the main memory simultaneously and allows the greater process size than the available physical memory. Demand paging systems handle the memory more flexibly.
===========================================================
FIND LENGTH OF STRING IN UNIX / LINUX BASH SCRIPT
Q) How to find the length of the given string in unix or linux operating system?
Length of the string is the number of characters in the strings content. There are many different ways to compute the length of the string. Here we will see some of the ways to find the length of the string using the shell script.
Length of the string is the number of characters in the strings content. There are many different ways to compute the length of the string. Here we will see some of the ways to find the length of the string using the shell script.
> cat length.sh
#!/bin/bash
str="Linux dedicated server"
#1. finding size of string using the wc command
len=`echo $str|wc -c`
echo "length of $str=$len"
#2.caluclating the length using the awk command
len=`echo $str|awk '{print length}'`
#3.length using the expr command
len=`expr length "$str"`
echo "length of $str=$len"
#4.length using the shell built in hash(#)
echo "length of $str=$len"
echo "length of $str=${#str}"
GREP / PRINT LINES BETWEEN TWO MATCHING PATTERNS/STRINGS - UNIX/LINUX
Q) How to print the lines between two matching strings using the linux or unix commands?
Create the following sample file in the unix operating system:
Let say, we want to grep the lines between the two matching strings linux and fedora. After grepping, the output should be:
Solution:
1. Using the sed command.
To the sed command, we can specify the starting pattern and ending pattern to print the lines. The syantax and the example is shown below:
2. Using the awk command
Similar to the sed command, you can specify the starting strind and ending string in the awk command. The syntax and the example is shown eblow:
Create the following sample file in the unix operating system:
> cat sample.dat unix operating system linux dedicated server debian virtual server fedora system windows gui system
Let say, we want to grep the lines between the two matching strings linux and fedora. After grepping, the output should be:
linux dedicated server debian virtual server fedora system
Solution:
1. Using the sed command.
To the sed command, we can specify the starting pattern and ending pattern to print the lines. The syantax and the example is shown below:
Syntax: sed -n '/start-pattern/,/end-pattern/p' filename Example: > sed -n '/linux/,/fedora/p' sample.dat
2. Using the awk command
Similar to the sed command, you can specify the starting strind and ending string in the awk command. The syntax and the example is shown eblow:
Syntax: awk '/start-string/,/end-string/' filename Example: > awk '/unix/,/debian/' sample.dat
HOW TO GET HOSTNAME FROM IP ADDRESS - UNIX /LINUX
The
host command in unix or linux operating system can be used to get the
hostname from an IP address. An example is shown below:
The host command gets the hostname listed in the /etc/resolv.conf file by querying the nameserver.
Another way to get the hostname is to user the nslookup command.
Here the hostname is highlighted in the green color. The alias name for the hostname is highlighted in the red color. If you are connected the remote host, you can get the hostname of the remote machine by using the arp command. It will list all the hostnames with the IP address. Another way is to simply type the hostname command on the remote server to know its host name.
> host 4.59.125.171 171.125.59.4.in-addr.arpa is an alias for 171.128-254.125.59.4.in-addr.arpa. 171.128-254.125.59.4.in-addr.arpa domain name pointer 4-59-125-171-hou.servercraft.co.
The host command gets the hostname listed in the /etc/resolv.conf file by querying the nameserver.
Another way to get the hostname is to user the nslookup command.
> nslookup 4.59.125.171 Server: 192.168.172.100 Address: 192.168.172.100#53 Non-authoritative answer: 171.125.59.4.in-addr.arpa canonical name = 171.128-254.125.59.4.in-addr.arpa. 171.128-254.125.59.4.in-addr.arpa name = 4-59-125-171-hou.servercraft.co. Authoritative answers can be found from: 128-254.125.59.4.in-addr.arpa nameserver = ns1.p23.dynect.net. 128-254.125.59.4.in-addr.arpa nameserver = ns5.p23.dynect.net. 128-254.125.59.4.in-addr.arpa nameserver = ns3.p23.dynect.net. 128-254.125.59.4.in-addr.arpa nameserver = ns2.p23.dynect.net. 128-254.125.59.4.in-addr.arpa nameserver = ns4.p23.dynect.net. ns4.p23.dynect.net internet address = 204.13.251.23 ns3.p23.dynect.net internet address = 208.78.71.23 ns1.p23.dynect.net internet address = 208.78.70.23 ns2.p23.dynect.net internet address = 204.13.250.23
Here the hostname is highlighted in the green color. The alias name for the hostname is highlighted in the red color. If you are connected the remote host, you can get the hostname of the remote machine by using the arp command. It will list all the hostnames with the IP address. Another way is to simply type the hostname command on the remote server to know its host name.
EXECUTE MYSQL COMMAND IN BASH / SHELL SCRIPT
Q) How to connect to mysql database from a bash script in unix or linux and run sql queries?
Bash scripting helps in automating things. We can automate running sql queries by connecting to the mysql database through a shell script in unix or linux system.
Here we will see how to run a small sql query in mysql database through a script. The bash script code is shown below:
Here in the above script, the first part declares the mysql db variables and assigns the DB details. The second part prepares sql query. And the final part executes the mysql command.
Bash scripting helps in automating things. We can automate running sql queries by connecting to the mysql database through a shell script in unix or linux system.
Here we will see how to run a small sql query in mysql database through a script. The bash script code is shown below:
#!/usr/bin/bash #Script to run automated sql queries #Declaring mysql DB connection MASTER_DB_USER='username' MASTER_DB_PASSWD='password' MASTER_DB_PORT=3160 MASTER_DB_HOST='mysql.hostname' MASTER_DB_NAME='mysqlDbName' #Prepare sql query SQL_Query='select * from tablename limit 10' #mysql command to connect to database MYSQL -u$MASTER_DB_USER -p$MASTER_DB_PASSWD -P$MASTER_DB_PORT -h$MASTER_DB_HOST -D$MASTER_DB_NAME <<EOF $SQL_Query EOF echo "End of script"
Here in the above script, the first part declares the mysql db variables and assigns the DB details. The second part prepares sql query. And the final part executes the mysql command.
PASTE COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS
Paste
command is one of the useful commands in unix or linux operating
system. The paste command merges the lines from multiple files. The
paste command sequentially writes the corresponding lines from each file
separated by a TAB delimiter on the unix terminal.
The syntax of the paste command is
The options of paste command are:
Paste Command Examples:
Create the following three files in your unix or linux servers to practice to practice the examples:
1. Merging files in parallel
By default, the paste command merges the files in parallel. The paste command writes corresponding lines from the files as a tab delimited on the terminal.
2. Specifying the delimiter
Paste command uses the tab delimiter by default for merging the files. You can change the delimiter to any other character by using the -d option.
In the above example, pipe delimiter is specified
3. Merging files in sequentially.
You can merge the files in sequentially using the -s option. The paste command reads each file in sequentially. It reads all the lines from a single file and merges all these lines into a single line.
The following example shows how to specify a delimiter for sequential merging of files:
4. Specifying multiple delimiters.
Multiple delimiters come in handy when you want to merge more than two files with different delimiters. For example I want to merge file1, file2 with pipe delimiter and file2, file3 with comma delimiter. In this case multiple delimiters will be helpful.
5. Combining N consecutive lines
The paste command can also be used to merge N consecutive lines from a file into a single line. The following example merges 2 consecutive lines into a single line
The syntax of the paste command is
paste [options] files-list
The options of paste command are:
-d : Specify of a list of delimiters. -s : Paste one file at a time instead of in parallel. --version : version information --help : Help about the paste command.
Paste Command Examples:
Create the following three files in your unix or linux servers to practice to practice the examples:
> cat file1 Unix Linux Windows > cat file2 Dedicated server Virtual server > cat file3 Hosting Machine Operating system
1. Merging files in parallel
By default, the paste command merges the files in parallel. The paste command writes corresponding lines from the files as a tab delimited on the terminal.
> paste file1 file2
Unix Dedicated server
Linux Virtual server
Windows
> paste file2 file1
Dedicated server Unix
Virtual server Linux
Windows
2. Specifying the delimiter
Paste command uses the tab delimiter by default for merging the files. You can change the delimiter to any other character by using the -d option.
> paste -d"|" file1 file2 Unix|Dedicated server Linux|Virtual server Windows|
In the above example, pipe delimiter is specified
3. Merging files in sequentially.
You can merge the files in sequentially using the -s option. The paste command reads each file in sequentially. It reads all the lines from a single file and merges all these lines into a single line.
> paste -s file1 file2 Unix Linux Windows Dedicated server Virtual server
The following example shows how to specify a delimiter for sequential merging of files:
> paste -s -d"," file1 file2 Unix,Linux,Windows Dedicated server,Virtual server
4. Specifying multiple delimiters.
Multiple delimiters come in handy when you want to merge more than two files with different delimiters. For example I want to merge file1, file2 with pipe delimiter and file2, file3 with comma delimiter. In this case multiple delimiters will be helpful.
> paste -d"|," file1 file2 file3 Unix|Dedicated server,Hosting Linux|Virtual server,Machine Windows|,Operating system
5. Combining N consecutive lines
The paste command can also be used to merge N consecutive lines from a file into a single line. The following example merges 2 consecutive lines into a single line
> cat file1 | paste - - Unix Linux Windows
HOW TO DELETE SYMBOLIC LINK - UNIX / LINUX
Q) I have created a new symbolic link to a directory. Now i want to remove it. How can i remove the symbolic link?
Before we actually see how to delete a symbolic link, let see creating the symbolic to a directory. Follow the below steps for creating and deleting symbolic link:
1. Create New Directory
I am creating a new directory, “linkdir”, in /var/tmp. Later I will create a symbolic link to this directory.
2. Creating symbolic link
The ln command in unix or linux is used to create a symbolic link.
You can observe from the output of the ls -l command, that a symbolic link has been created.
3. Deleting a symbolic link
In Unix or linux system everything is a file. Just as we remove the files using the rm command, in the same way remove the link.
Note that, the rm command just removes the link. However it will not delete the target directory /var/tmp/linkdir. Another way of removing the symbolic link is to use the unlink command.
Before we actually see how to delete a symbolic link, let see creating the symbolic to a directory. Follow the below steps for creating and deleting symbolic link:
1. Create New Directory
I am creating a new directory, “linkdir”, in /var/tmp. Later I will create a symbolic link to this directory.
> pwd /home/mark >mkdir /var/tmp/linkdir
2. Creating symbolic link
The ln command in unix or linux is used to create a symbolic link.
> pwd /home/mark > ln -s /var/tmp/linkdir linkname >ls -l total 16 -rw-rw-r-- 1 mark mark 0 Aug 29 23:45 doc lrwxrwxrwx 1 mark mark 16 Aug 30 00:43 linkname -> /var/tmp/linkdir -rw-rw-r-- 1 mark mark 14 Aug 29 22:31 sample
You can observe from the output of the ls -l command, that a symbolic link has been created.
3. Deleting a symbolic link
In Unix or linux system everything is a file. Just as we remove the files using the rm command, in the same way remove the link.
> rm linkname > ls -l total 12 -rw-rw-r-- 1 pcenter pcenter 0 Aug 29 23:45 doc -rw-rw-r-- 1 pcenter pcenter 14 Aug 29 22:31 sample
Note that, the rm command just removes the link. However it will not delete the target directory /var/tmp/linkdir. Another way of removing the symbolic link is to use the unlink command.
unlink linkname
HOSTNAME COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS
Hostname
is the name of the system or server you are logged into. The hostname
can also refer to the sitename or computer name. As an example, if an
organization domain name is "google.com" and a specific computer name in
that doman is "unix-box", then the hostname of the computer is
"unix-box.google.com".
The syntax of hostname command in unix or linux system is
The options of hostname command are:
Hostname Command Examples:
1. Print the hostname of the system The basic functionality of the hostname command is to display the name of the system on the terminal. Just type the hostname on the unix terminal and press enter to print the hostname.
2. Ip address of the computer You can find the ip address of the computer by using the -i option with hostname command.
3. Print the domain name To know the domain name where the computer resides, use the -d option with hostname command.
4. Short hostname By default the hostname command prints the complete name of the computer. You can print a short name by using the -s option. This prints the name upto the first dot in the full hostname.
5. Getting help To get help about the hostanme command either use the man command or the -h option with hostname command.
The syntax of hostname command in unix or linux system is
hostname [options] [file]
The options of hostname command are:
-a : Prints the alisa name of the host if created any. -d : prints the domain name -i : prints the ip address of the host -s : prints the shortname of the host. -v : verbose data -V : version information -h : help about hostname command
Hostname Command Examples:
1. Print the hostname of the system The basic functionality of the hostname command is to display the name of the system on the terminal. Just type the hostname on the unix terminal and press enter to print the hostname.
> hostname unix-box.google.com
2. Ip address of the computer You can find the ip address of the computer by using the -i option with hostname command.
> hostname -i 125.20.223.69
3. Print the domain name To know the domain name where the computer resides, use the -d option with hostname command.
> hostname -d google.com
4. Short hostname By default the hostname command prints the complete name of the computer. You can print a short name by using the -s option. This prints the name upto the first dot in the full hostname.
> hostname -s unix-box
5. Getting help To get help about the hostanme command either use the man command or the -h option with hostname command.
> man hostname > hostname -h
MAIL COMMAND EXAMPLES IN UNIX / LINUX TUTORIAL
The
Mail command in unix or linux system is used to send emails to the
users, to read the received emails, to delete the emails etc. Mail
command will come in handy especially when writing automated scripts.
For example, you have written an automated script for taking weekly
backup of oracle database. How to know the status of backup, whether it
is succeeded or not? In this case, sending an email from the automated
script at the end of the backup will be helpful in knowing the status.
The syntax of mail command is:
The options of mail command are listed below:
Mail Command Examples - Sending Emails:
1. Sending sample email to user
The basic functionality of the mail command in unix or linux system is to send an email to the user.
Here the echo statement is used for specifying the body of the email. The -s option is used for specifying the mail subject. The mail command sends the email to the user to@example.com
2. Specifying the body in a file
You want to compose a mail which contains 100 lines in the body. Specifying the body with the echo statement is a tedious process. So write the contents of the body in a file and send the mail using one of the following options:
Here the body.txt file contains the body of the email. 3. Send mail to more than one user You can send email to more than one user by specifying the users in comma separated list.
4. Using the cc and bcc option You can copy the emails to more number of users by using the -c and -b options. An example is shown below:
5. Specifying the from address So far the above examples send the emails with from address as the logged in user. You can explicitly specify the from-address using the -r option.
6. Attaching files. The mail command does not provide an option for attaching files. There is a workaround for attaching files using the uuencode command. Pipe the output of uuencode command for attaching files.
Mail Command Examples - Reading Emails:
1. Viewing all the received emails Simply type the mail and then press enter to view the received emails.
Another way of viewing the emails is using the -f option. This is shown below:
From the above output, you can see that, it displays the from-address, date and subject of the emails in the inbox. It also displays the ampersand (&) prompt at the end. To go back to the main prompt, type CTRL+z or CTRL+d depending on your operating system and press enter. The ampersand prompt allows you to read, reply, navigate and delete the emails.
2. Reading an email.
To read the Nth email, just enter the mail number at the ampersand prompt and press enter. This is shown below:
This displays the second email details.
3. Navigating through inbox emails. To go to the next email, enter the + symbol. To go back to the previous email, enter the - symbol at the ampersand prompt.
4. Replying email. Once you have read an email, you can give reply to the mail by typing "reply" and pressing enter.
5. Deleting emails. You can delete a read email by typing the d and pressing enter. You can also specify the email numbers to d option for deleting them.
The syntax of mail command is:
mail [options] to-address [-- sendmail-options]
The options of mail command are listed below:
-v : Verbose mode. Delivery details are displayed on the terminal. -s : Specify the subject of the mail -c : Send carbon copies of the mail to the list of users. This is like cc option in Microsoft outlook. -b : Send blind copies of the mail to the list of users. This is like bcc option in outlook. -f : Read the contents of the mailbox -r : Specify the from address in send mail options.
Mail Command Examples - Sending Emails:
1. Sending sample email to user
The basic functionality of the mail command in unix or linux system is to send an email to the user.
echo "Mail body" | mail -s "Mail subject" to@example.com
Here the echo statement is used for specifying the body of the email. The -s option is used for specifying the mail subject. The mail command sends the email to the user to@example.com
2. Specifying the body in a file
You want to compose a mail which contains 100 lines in the body. Specifying the body with the echo statement is a tedious process. So write the contents of the body in a file and send the mail using one of the following options:
Using cat statement: cat body.txt | mail -s "Mail subject" to@example.com Using input redirection operator mail -s "Mail subject" to@example.com < body.txt
Here the body.txt file contains the body of the email. 3. Send mail to more than one user You can send email to more than one user by specifying the users in comma separated list.
mail -s "Mail subject" "user1@example.com,user2@example.com" < body.txt
4. Using the cc and bcc option You can copy the emails to more number of users by using the -c and -b options. An example is shown below:
mail -s "Mail subject" -c "ccuser@gmail.com" -b "bccuser@yahoo.com" "user@example.com" < body.txt
5. Specifying the from address So far the above examples send the emails with from address as the logged in user. You can explicitly specify the from-address using the -r option.
cat body.txt | mail -s "Mail subject" "to-user@example.com" -- -r "from-user@example.com"
6. Attaching files. The mail command does not provide an option for attaching files. There is a workaround for attaching files using the uuencode command. Pipe the output of uuencode command for attaching files.
uuencode attachment-file | mail -s "Mail subject" "to-user@example.com" < body.txt
Mail Command Examples - Reading Emails:
1. Viewing all the received emails Simply type the mail and then press enter to view the received emails.
Another way of viewing the emails is using the -f option. This is shown below:
> mail -f /var/spool/mail/user Mail version 8.1 6/6/93. Type ? for help. "/var/spool/mail/user": 2 messages 2 new >N 1 root@hostname Tue May 17 00:00 21/1013 "Mail subject 1" N 2 root@hostname Wed May 18 00:00 21/1053 "Mail subject 2" &
From the above output, you can see that, it displays the from-address, date and subject of the emails in the inbox. It also displays the ampersand (&) prompt at the end. To go back to the main prompt, type CTRL+z or CTRL+d depending on your operating system and press enter. The ampersand prompt allows you to read, reply, navigate and delete the emails.
2. Reading an email.
To read the Nth email, just enter the mail number at the ampersand prompt and press enter. This is shown below:
> mail -f /var/spool/mail/user Mail version 8.1 6/6/93. Type ? for help. "/var/spool/mail/user": 2 messages 2 new >N 1 root@hostname Tue May 17 00:00 21/1013 "Mail subject 1" N 2 root@hostname Wed May 18 00:00 21/1053 "Mail subject 2" &2 Message 2: From root@hostname Wed May 18 00:00 21/1053 --------------- Subject: Mail subject 2 ------------
This displays the second email details.
3. Navigating through inbox emails. To go to the next email, enter the + symbol. To go back to the previous email, enter the - symbol at the ampersand prompt.
&- Message 1: From root@hostname Tue May 17 00:00 21/1013 --------------- Subject: Mail subject 1 ------------
4. Replying email. Once you have read an email, you can give reply to the mail by typing "reply" and pressing enter.
&reply
To: root@hostname
root@hostname
Subject: Re: Mail subject1
5. Deleting emails. You can delete a read email by typing the d and pressing enter. You can also specify the email numbers to d option for deleting them.
To delete read email &d To delete emails 1 and 2 &d 1 2 To delete range emails from 10 to 30 &d 10-30 To delete all emails in the mbox (mail box) &d *
ZIP COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS
zip
is used to compress the files to reduce file size and also used as file
package utility. zip is available in many operating systems like unix,
linux, windows etc.
If you have a limited bandwidth between two servers and want to transfer the files faster, then zip the files and transfer.
The syntax of zip command is
The options of zip command are:
Zip Command Examples:
The files in my current directory are listed below:
Here docs is a directory which contains the files linux.pdf, unix.pdf and oracle.pdf. We will see how to use zip command with examples.
1. Creating a zip file
The zip command in unix or linux system creates an archive with the specified files. This is shown below:
The above command creates the zip file with name archive.zip
2. Extracting files from zip
To extract files from the zip, use the unzip command in unix system. This is shown below:
3. Removing file from a zip file
After creating a zip file, you can remove a file from the archive using the -d option. To remove the file unix-server.dat from the archive, run the below zip command:
4. Update existing zip file
You can update the files in already created zip file. If any of the files are modified after zipping, you can fresh the zip file with only those modified files using the -f option.
Another way is using the -u option. This option can be used to update the specified list of files or add new files to the existing zip file.
5. Recursively zip files in directory.
To zip a directory recursively, use the -r option with the zip command. This example is shown below:
6. Excluding files in zipping
Let say you are zipping all the files in the current directory and want to exclude some unwanted files. You can exclude these unwanted files using the -x option.
The above command zips all the files in the current directory except the file linux-virtual-server.bat
7. Faster compressing
You can compress the files very fast using the -1 option with zip command. An example is shown below with and without using fast compression.
If you use fast compression, the archive file created will occupy more space (size) when compared to normal compression.
8. Better compression.
To reduce more amount of size the files occupied, you can use the -9 option with the zip command. This gives a better compression.
If you have a limited bandwidth between two servers and want to transfer the files faster, then zip the files and transfer.
The syntax of zip command is
zip [options] zipfile files_list
The options of zip command are:
-d : Removes the file from the zip archive -u : Updates the file in the zip archive -m : Deletes the original files after zipping. -r : Recursively zips the files in a directory -x : Exclude the files in creating the zip -v : verbose mode -1 : Compresses the files faster -9 : Compresses the files better -f : freshen only changed files. zipfile : creates the zip file with name as zipfile.zip files_list : list of files to be zipped.
Zip Command Examples:
The files in my current directory are listed below:
docs/linux.pdf docs/oracle.pdf docs/unix.pdf linux-virtual-server.bat unix-server.dat
Here docs is a directory which contains the files linux.pdf, unix.pdf and oracle.pdf. We will see how to use zip command with examples.
1. Creating a zip file
The zip command in unix or linux system creates an archive with the specified files. This is shown below:
> zip archive linux-virtual-server.bat unix-server.dat adding: linux-virtual-server.bat (deflated 80%) adding: unix-server.dat (deflated 80%) > ls archive.zip docs linux-virtual-server.bat unix-server.dat
The above command creates the zip file with name archive.zip
2. Extracting files from zip
To extract files from the zip, use the unzip command in unix system. This is shown below:
> unzip archive.zip Archive: archive.zip inflating: linux-virtual-server.bat inflating: unix-server.dat > ls archive.zip linux-virtual-server.bat unix-server.dat
3. Removing file from a zip file
After creating a zip file, you can remove a file from the archive using the -d option. To remove the file unix-server.dat from the archive, run the below zip command:
> zip -d archive.zip unix-server.dat deleting: unix-server.dat > unzip archive.zip Archive: archive.zip inflating: linux-virtual-server.bat
4. Update existing zip file
You can update the files in already created zip file. If any of the files are modified after zipping, you can fresh the zip file with only those modified files using the -f option.
> zip -f archive.zip freshening: linux-virtual-server.bat (stored 0%)
Another way is using the -u option. This option can be used to update the specified list of files or add new files to the existing zip file.
> zip -u archive.zip linux-virtual-server.bat temp updating: linux-virtual-server.bat (deflated 79%) adding: temp (stored 0%)
5. Recursively zip files in directory.
To zip a directory recursively, use the -r option with the zip command. This example is shown below:
> zip -r dir_archive docs adding: docs/ (stored 0%) adding: docs/unix.pdf (stored 0%) adding: docs/oracle.pdf (stored 0%) adding: docs/linux.pdf (stored 0%)
6. Excluding files in zipping
Let say you are zipping all the files in the current directory and want to exclude some unwanted files. You can exclude these unwanted files using the -x option.
zip exclude_archive * -x linux-virtual-server.bat
The above command zips all the files in the current directory except the file linux-virtual-server.bat
7. Faster compressing
You can compress the files very fast using the -1 option with zip command. An example is shown below with and without using fast compression.
> zip -1 fast_archive linux-virtual-server.bat adding: linux-virtual-server.bat (deflated 79%) >zip normal_archive linux-virtual-server.bat adding: linux-virtual-server.bat (deflated 80%)
If you use fast compression, the archive file created will occupy more space (size) when compared to normal compression.
8. Better compression.
To reduce more amount of size the files occupied, you can use the -9 option with the zip command. This gives a better compression.
> zip -9 better_archive linux-virtual-server.bat adding: linux-virtual-server.bat (deflated 81%)
SORT COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS
Sort
command in unix or linux system is used to order the elements or text.
Sort command has the capability of sorting numerical values and strings.
The sort command can order the lines in a text file.
The syntax of sort command is:
The options are:
Sort Command Examples:
Before practicing the examples create the below two files in your unix system:
1. Sorting lines of text
The default sort command uses alphabetical order (ASCII order) to sort the file. It treats each line as a string and then sorts the lines.
2. Sorting based on the field positions.
You can specify the field postions using the -k option of sort command. The sort command uses the space or tab as the default delimiter. To sort based on the data in the second field, run the below command:
You can also pecify more than field with k option as a comma separated list. The below command uses the second and fourth fields to sort the data.
3. Numeric sorting
Instead of the default alphabetical sorting order, you can make the sort command to sort in numeric order using the -n option. This is shown below:
4. Sort in reverse order
By default, the sort command sorts the data in ascending order. You can change this to descending order using the -r option.
5. Suppressing duplicates or Print only unique values
You can produce only unique values in the output using the - u option of the sort command.
Another way is piping the output of sort command to uniq command.
6. Delimited file input
In the second, third and fourth examples we have sorted the data based on the field positions. Here the fields are separted by space or tab character. What if the fields are specifed by any other character? In such cases, we have to specify the input delimiter with the -t option. An example is shown below:
7. Sorting on months.
We can sort the data in the monthwise using the -M option of the sort command. This is shown below:
Treats the first 3 characters in the string as month and then sorts in months order.
The syntax of sort command is:
sort [options] filename
The options are:
-b : Ignores leading spaces in each line -d : Uses dictionary sort order. Conisders only spaces and alphanumeric characters in sorting -f : Uses case insensitive sorting. -M : Sorts based on months. Considers only first 3 letters as month. Eg: JAN, FEB -n : Uses numeric sorting -R : Sorts the input file randomly. -r : Reverse order sorting -k : Sorts file based on the data in the specified field positions. -u : Suppresses duplicate lines -t : input field separator
Sort Command Examples:
Before practicing the examples create the below two files in your unix system:
> cat order.txt Unix distributed 05 server Linux virtual 3 server Unix distributed 05 server Distributed processing 6 system > cat delim_sort.txt Mayday|4 Janmon|1 Declast|12
1. Sorting lines of text
The default sort command uses alphabetical order (ASCII order) to sort the file. It treats each line as a string and then sorts the lines.
> sort order.txt Distributed processing 6 system Linux virtual 3 server Unix distributed 05 server Unix distributed 05 server
2. Sorting based on the field positions.
You can specify the field postions using the -k option of sort command. The sort command uses the space or tab as the default delimiter. To sort based on the data in the second field, run the below command:
> sort -k2 order.txt Unix distributed 05 server Unix distributed 05 server Distributed processing 6 system Linux virtual 3 server
You can also pecify more than field with k option as a comma separated list. The below command uses the second and fourth fields to sort the data.
> sort -k2,4 order.txt
3. Numeric sorting
Instead of the default alphabetical sorting order, you can make the sort command to sort in numeric order using the -n option. This is shown below:
> sort -nk3 order.txt Linux virtual 3 server Unix distributed 05 server Unix distributed 05 server Distributed processing 6 system
4. Sort in reverse order
By default, the sort command sorts the data in ascending order. You can change this to descending order using the -r option.
> sort -nrk3 order.txt Distributed processing 6 system Unix distributed 05 server Unix distributed 05 server Linux virtual 3 server
5. Suppressing duplicates or Print only unique values
You can produce only unique values in the output using the - u option of the sort command.
> sort -u order.txt Distributed processing 6 system Linux virtual 3 server Unix distributed 05 server
Another way is piping the output of sort command to uniq command.
> sort order.txt | uniq
6. Delimited file input
In the second, third and fourth examples we have sorted the data based on the field positions. Here the fields are separted by space or tab character. What if the fields are specifed by any other character? In such cases, we have to specify the input delimiter with the -t option. An example is shown below:
> sort -t'|' -nrk2 delim_sort.txt Declast|12 Mayday|4 Janmon|1
7. Sorting on months.
We can sort the data in the monthwise using the -M option of the sort command. This is shown below:
> sort -M delim_sort.txt Janmon|1 Mayday|4 Declast|12
Treats the first 3 characters in the string as month and then sorts in months order.
HOW TO FIND PRIME NUMBER BASH (SHELL) SCRIPT IN UNIX / LINUX
Q) How to find whether the give number is prime number or not using the bash script in unix or linux system?
A prime number is the one which is divisible by one and itself. Examples of prime numbers are 2,3,5 etc. Here we will see how to find whehter a number is prime or not using the bash script in unix or linux operating system. Before going to the script, let see the factor command in unix.
The factor command is used to find the number of factors a given number have. Examples of factor command are listed below:
From the above output of the factor command, you can observe that the output contains two words for prime numbers. We will use the factor and wc command to find whether the number is prime or not. The complete bash script for finding the prime number is shown below:
Save the script with name as prime.sh and run the bash script with the below inputs:
A prime number is the one which is divisible by one and itself. Examples of prime numbers are 2,3,5 etc. Here we will see how to find whehter a number is prime or not using the bash script in unix or linux operating system. Before going to the script, let see the factor command in unix.
The factor command is used to find the number of factors a given number have. Examples of factor command are listed below:
> factor 1 1: > factor 2 2: 2 > factor 3 3: 3 > factor 4 4: 2 2 > factor 20 20: 2 2 5
From the above output of the factor command, you can observe that the output contains two words for prime numbers. We will use the factor and wc command to find whether the number is prime or not. The complete bash script for finding the prime number is shown below:
#!/bin/bash #Assign the first argument from the command line to the variable Input_Number=$1 if [ $Input_Number -eq 1 ] then echo "$Input_Number is neither prime nor composite number" exit 0; fi Num_Factors=`factor $Input_Number|wc -w` echo $Num_Factors if [ $Num_Factors -eq 2 ] then echo "$Input_Number is a prime number" else echo "$Input_Number is not a prime number" fi
Save the script with name as prime.sh and run the bash script with the below inputs:
> bash prime.sh 1 1 is neither prime nor composite number > bash prime.sh 7 7 is a prime number > bash prime.sh 10 10 is not a prime number
FTP BATCH (BASH) SCRIPT EXAMPLE IN UNIX / LINUX TUTORIAL
The
FTP is a useful command in unix or linux system used for transfering
files between a local server and remote server. You can transfer files
between unix systems and also non-unix systems like windows operating
system using FTP.
We will see a sample bash script which transfers files from the local machine to the remote machine.
The following script reads the ftp instructions from the dat file and executes them on the remote machine.
The contents of the instructions.dat file is
This script copies the unix_server.rpm file to the remote host. Know more about the ftp command: Ftp command examples in unix and linux
We will see a sample bash script which transfers files from the local machine to the remote machine.
The following script reads the ftp instructions from the dat file and executes them on the remote machine.
#!/usr/bin/bash echo "A sample Ftp batch script to transfer files." echo "start ftp script" ftp -s:ftp_instructions.dat remote-host echo "End ftp script"
The contents of the instructions.dat file is
user password cd /var/tmp put unix_server.rpm quit
This script copies the unix_server.rpm file to the remote host. Know more about the ftp command: Ftp command examples in unix and linux
GREP UNIQUE OCCURRENCES IN UNIX / LINUX TUTORIALS
Q) How to grep for unique occurrences of a pattern in a file using the unix or linux commands?
First let me explain the problem. I have a file in my unix operating system. Sample content from the file is shown below:
When I grep for the word linux from the above file i will get three lines in the output. This is shown below:
However I want only one occurrence of the linux to be printed on my terminal. That is the output should contain only the first line. How to get this result using unix or linux commands?
Solution:
1. Using grep and head command.
Pipe the output of grep command to head command to get the first line.
2. Using m option of grep command.
The m option can be used to display the number of matching lines. To display the first matching line:
3. Using the sed command
We can also use the sed command to print unique occurrence of a pattern. This is shown below:
Here the n option duplicates each matched line. The p suppresses the duplicates. The q prints the first matched line.
4. Using awk command.
The awk command to display the first matched line is
Here the awk command searches for the pattern linux in each line. Once the pattern is found, it prints the line and exits searching in further lines.
First let me explain the problem. I have a file in my unix operating system. Sample content from the file is shown below:
> cat OS-Versions.dat linux version 1.1 linux version 1.2 unix distributed system version 2.1 linux virtual server version 2.1 debian server version 1.1
When I grep for the word linux from the above file i will get three lines in the output. This is shown below:
> grep "linux" OS-Versions.dat linux version 1.1 linux version 1.2 linux virtual server version 2.1
However I want only one occurrence of the linux to be printed on my terminal. That is the output should contain only the first line. How to get this result using unix or linux commands?
Solution:
1. Using grep and head command.
Pipe the output of grep command to head command to get the first line.
> grep "linux" OS-Versions.dat | head -1
2. Using m option of grep command.
The m option can be used to display the number of matching lines. To display the first matching line:
> grep -m 1 "linux" OS-Versions.dat
3. Using the sed command
We can also use the sed command to print unique occurrence of a pattern. This is shown below:
> sed -n "/linux/p;q" OS-Versions.dat
Here the n option duplicates each matched line. The p suppresses the duplicates. The q prints the first matched line.
4. Using awk command.
The awk command to display the first matched line is
> awk '/linux/ {print $0; exit }' OS-Versions.dat
Here the awk command searches for the pattern linux in each line. Once the pattern is found, it prints the line and exits searching in further lines.
TAR COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS
In
windows operating system, you might have used the winzip and winrar
softwares for extracting and archiving the files. Similarly in unix or
linux operating system, the tar command is used for creating archive
files and also extracting files from the archives.
With the tar command, you can also create compressed archive files. In unix or linux operating system, there are many other commands like gzip and gunzip for creating and extracting archive files. Here we will see the important tar command examples in unix and linux systems which are used frequently in our daily work.
The syntax of tar command is
The options of tar command are:
Tar Command Examples:
1. Creating a tar file
Let see a sample example by archiving all the files in my current directory. The ls -l command displays the files and directories in the current directory.
We see how to tar all these files using the -c option with the tar command. This is shown below:
Observe the output of ls command and see the archive.tar file is created.
2. Printing the contents of tar file
We have created the tar file and we dont know whether it contains the actual files or not. To view the contents of the tar file use the -t option as
3. Updating the tar file with new contents.
You can add new files to the existing archive (tar) file using the -r option.
Here the touch command creates a new file. The first tar command adds the new file to the existing archive file. The second command displays the contents of the tar file.
4. Extracting the contents of tar file
In the first example, we have created the archive file. Now we will see how to extract the set of files from the archive. To extract the contents of the tar file use the -x option.
5. Creating compressed tar file
So far we have created a uncompressed tar file in the above examples. We can create a compressed tar file using the gzip or bzip2.
To extract or to view the files in a compressed tar file use the appropriate compression option (z or j).
6. Creating tar file with specified list of files
You can specify a list of files to be included in the newly created tar file.
Here the tar command creates the unix_files.tar file which contains only the files unix_server.bat and unix_system.dat
7. Extracting specific files from the tar
You can extract a specific file or a set of files from the archived file.
8. Extracting files from multiple archive files.
To extract the files from multiple archive files use the -M option with each -f option. This is shown below:
With the tar command, you can also create compressed archive files. In unix or linux operating system, there are many other commands like gzip and gunzip for creating and extracting archive files. Here we will see the important tar command examples in unix and linux systems which are used frequently in our daily work.
The syntax of tar command is
tar [options] [Archive file] [files list]
The options of tar command are:
c : creates a tar file. v : verbose. Displays the files information. f : Specify the tar file name. r : updates the tar file with new files. x : Extracts files from the archive (tar file). t : view contents of tar file. z : Specify the tar command to create a tar file using gzip in unix. j : uses bzip2 to create the tar file.
Tar Command Examples:
1. Creating a tar file
Let see a sample example by archiving all the files in my current directory. The ls -l command displays the files and directories in the current directory.
> ls -l drwxr-xr-x 2 user group 4096 Aug 8 03:23 debian -rw-r--r-- 1 user group 174 Aug 2 23:39 file -rw-r--r-- 1 user group 0 Aug 8 03:22 linux_server.bat -rw-r--r-- 1 user group 76 Aug 2 02:21 test.sh -rw-r--r-- 1 user group 0 Aug 8 03:22 unix_distro
We see how to tar all these files using the -c option with the tar command. This is shown below:
> tar -cvf archive.tar * debian/ file linux_server.bat test.sh unix_distro > ls archive.tar debian file linux_server.bat test.sh unix_distro
Observe the output of ls command and see the archive.tar file is created.
2. Printing the contents of tar file
We have created the tar file and we dont know whether it contains the actual files or not. To view the contents of the tar file use the -t option as
> tar -tvf archive.tar drwxr-xr-x user/group 0 2012-08-08 03:23:07 debian/ -rw-r--r-- user/group 174 2012-08-02 23:39:51 file -rw-r--r-- user/group 0 2012-08-08 03:22:19 linux_server.bat -rw-r--r-- user/group 76 2012-08-02 02:21:32 test.sh -rw-r--r-- user/group 0 2012-08-08 03:22:09 unix_distro
3. Updating the tar file with new contents.
You can add new files to the existing archive (tar) file using the -r option.
>touch red-hat-linux.dat >tar -rvf archive.tar red-hat-linux.dat red-hat-linux.dat >tar -tvf archive.tar drwxr-xr-x pcenter/pcenter 0 2012-08-08 03:23:07 debian/ -rw-r--r-- pcenter/pcenter 174 2012-08-02 23:39:51 file -rw-r--r-- pcenter/pcenter 0 2012-08-08 03:22:19 linux_server.bat -rw-r--r-- pcenter/pcenter 76 2012-08-02 02:21:32 test.sh -rw-r--r-- pcenter/pcenter 0 2012-08-08 03:22:09 unix_distro -rw-r--r-- pcenter/pcenter 0 2012-08-08 04:00:00 red-hat-linux.dat
Here the touch command creates a new file. The first tar command adds the new file to the existing archive file. The second command displays the contents of the tar file.
4. Extracting the contents of tar file
In the first example, we have created the archive file. Now we will see how to extract the set of files from the archive. To extract the contents of the tar file use the -x option.
> tar -xvf archive.tar debian/ file linux_server.bat test.sh unix_distro
5. Creating compressed tar file
So far we have created a uncompressed tar file in the above examples. We can create a compressed tar file using the gzip or bzip2.
Compressing files using gzip > tar -zcvf new_tar_file.tar.gz * Compressing files using bzip2 > tar -jcvf new_tar_file.tar.bz2 *
To extract or to view the files in a compressed tar file use the appropriate compression option (z or j).
To view files in a gzip compressed tar file > tar -ztvf new_tar_file.tar.gz To extract files from a gip compressed tar file > tar -zxvf new_tar_file.tar.gz To view files in a bzip2 compressed tar file > tar -jtvf new_tar_file.tar.bz2 To extract files from a bzip2 compressed tar file > tar -jxvf new_tar_file.tar.bz2
6. Creating tar file with specified list of files
You can specify a list of files to be included in the newly created tar file.
> tar -cvf unix_files.tar unix_server.bat unix_system.dat
Here the tar command creates the unix_files.tar file which contains only the files unix_server.bat and unix_system.dat
7. Extracting specific files from the tar
You can extract a specific file or a set of files from the archived file.
To extract a specifi file > tar -xvf unix_files.tar unix_server.bat To extract all files that start with name unix > tar -xvf unix_files.tar --wildcards "unix*"
8. Extracting files from multiple archive files.
To extract the files from multiple archive files use the -M option with each -f option. This is shown below:
> tar -xv -Mf archive.tar -Mf unix_files.tar
GREP COMMAND IN UNIX SHELL SCRIPT
Q) How to use the grep command in unix or linux bash scripts to search for a pattern match?
You might have used the grep command to search for a string in a file on the unix command line. Here we will see how to use the grep command in a bash script.
Consider the below file with data
Let see the bash script that prints the lines which contain the word server in it. The bash script code is shown below:
Now we will execute this script and see what the output is
Here in the script we have hardcoded the filename and the search pattern. We will see how to pass these as arguments from the command line to the script. The bash script is:
Now run the script as shown below:
You might have used the grep command to search for a string in a file on the unix command line. Here we will see how to use the grep command in a bash script.
Consider the below file with data
> cat sample_file.txt linux storage unix distributed system linux file server debian server fedora backup server
Let see the bash script that prints the lines which contain the word server in it. The bash script code is shown below:
>vi grep_script.sh #!/bin/bash WORD=server INPUT_FILE=sample_file.txt grep "$WORD" $INPUT_FILE
Now we will execute this script and see what the output is
> bash grep_script.sh linux file server debian server fedora backup server
Here in the script we have hardcoded the filename and the search pattern. We will see how to pass these as arguments from the command line to the script. The bash script is:
>vi grep_script_command_line.sh #!/bin/bash INPUT_FILE=$1 WORD=$2 grep "$WORD" $INPUT_FILE
Now run the script as shown below:
> bash grep_script_command_line.sh sample_file.txt server
FTP (FILE TRANSFER PROTOCOL) COMMAND EXAMPLES
The
FTP (file transfer program) utility is used to transfer files between a
local machine and remote network machine Using the File Transfer
protocol. In simple terms it transfers / copies files between two
computers. You can transfer files between unix systems and also non-unix
systems like windows operating system using FTP.
The FTP command is simple to use and easy to learn. Let see useful examples of FTP command in detail.
FTP Command Examples:
If you are using windows operating system, open the command prompt and practice the below FTP commands. If you are using unix or linux operating systems, just simply type the ftp command on the terminal.
1. Connecting to Remote Host
First you need to connect to a remote host before doing any operations. You can use any one of the following methods to connect to a remote host. First method is
Once the ftp connects to the remote server name, it will prompt you to enter the user name and password. After successful login, your terminal or prompt changes to "ftp>".
Another method is to use the open option with ftp command. This is shown below:
If the ftp command fails to connect to the remote server, then you will get the below error:
2. Copy file from remote machine to local machine.
The get option is used to download or transfer a file from the remote system to the local system.
This will download the specified file (windows-cleveland.bat) from the remote systems current directory.
3. Copying multiple files from remote machine to local machine.
You can use the mget to transfer multiple files from the remote host to local host.
This will download all the png images to the local machine.
4. Transferring file from local server to remote server
The put option is used to copy the file from the local host to the remote host.
This command puts the rpm file into the remote machine.
5. Transferring multiple files to the remote server.
You can use the mput option to transfer more than one file from local system to the remote system.
6. Executing commands in remote machine.
After connecting to the remote network machine using the ftp, you can run commands like ls to list the files, cd to change directory and many more.
This will list the files and directories in the remote machines current directory.
7. Executing commands in local machine.
Once you have connected to the remote host, to run the commands on local machine you need to exit from the ftp connection. Instead of this, there is a way to run commands on local host without exiting from the ftp connection. Use the ! symbol before the command you want to run.
Now this will list the files in the local machines current directory.
8. Changing the file transferring mode.
You can change the file transfer modes to ascii and binary modes. Use the below commands to change the mode.
9. Deleting files on remote machine
You can use the delete or mdelete to remove a single file or multiple files in the remote machine.
10. Disconnecting from ftp connection.
Use the quit command to close the ftp connection.
11. Using FTP command in batch scripts
The following script reads the instructions from the dat file and executes them on the remote machine.
The contents of the instructions.dat file is
12. Getting the help about ftp command.
To know more about the ftp command, just type the help on the prompt. It will display the options/commands that you can use with ftp command.
The FTP command is simple to use and easy to learn. Let see useful examples of FTP command in detail.
FTP Command Examples:
If you are using windows operating system, open the command prompt and practice the below FTP commands. If you are using unix or linux operating systems, just simply type the ftp command on the terminal.
1. Connecting to Remote Host
First you need to connect to a remote host before doing any operations. You can use any one of the following methods to connect to a remote host. First method is
> ftp remote-server-name connected to remote-server-name User-Name: Password: ftp>
Once the ftp connects to the remote server name, it will prompt you to enter the user name and password. After successful login, your terminal or prompt changes to "ftp>".
Another method is to use the open option with ftp command. This is shown below:
>ftp ftp>open remote-server-name connected to remote-server-name User-Name: Password: ftp>
If the ftp command fails to connect to the remote server, then you will get the below error:
ftp: connect: Connection refused
2. Copy file from remote machine to local machine.
The get option is used to download or transfer a file from the remote system to the local system.
ftp> get windows-cleveland.bat
This will download the specified file (windows-cleveland.bat) from the remote systems current directory.
3. Copying multiple files from remote machine to local machine.
You can use the mget to transfer multiple files from the remote host to local host.
ftp>mget *.png
This will download all the png images to the local machine.
4. Transferring file from local server to remote server
The put option is used to copy the file from the local host to the remote host.
ftp>put linux-virtual-server.rpm
This command puts the rpm file into the remote machine.
5. Transferring multiple files to the remote server.
You can use the mput option to transfer more than one file from local system to the remote system.
ftp>put *.rpm
6. Executing commands in remote machine.
After connecting to the remote network machine using the ftp, you can run commands like ls to list the files, cd to change directory and many more.
ftp> ls
This will list the files and directories in the remote machines current directory.
7. Executing commands in local machine.
Once you have connected to the remote host, to run the commands on local machine you need to exit from the ftp connection. Instead of this, there is a way to run commands on local host without exiting from the ftp connection. Use the ! symbol before the command you want to run.
ftp> !ls
Now this will list the files in the local machines current directory.
8. Changing the file transferring mode.
You can change the file transfer modes to ascii and binary modes. Use the below commands to change the mode.
ftp>ascii ftp>binary
9. Deleting files on remote machine
You can use the delete or mdelete to remove a single file or multiple files in the remote machine.
ftp>delete linux-dedicated-server.dat ftp>mdelete *.dat
10. Disconnecting from ftp connection.
Use the quit command to close the ftp connection.
ftp>quit
11. Using FTP command in batch scripts
The following script reads the instructions from the dat file and executes them on the remote machine.
echo "Ftp command batch script" echo "start" ftp -s:instructions.dat remote-host echo "End"
The contents of the instructions.dat file is
user password cd /var/tmp put oracle_storage.exe quit
12. Getting the help about ftp command.
To know more about the ftp command, just type the help on the prompt. It will display the options/commands that you can use with ftp command.
ftp>help Commands may be abbreviated. Commands are: ! disconnect mdelete preserve runique $ edit mdir progress send account exit mget prompt sendport append form mkdir proxy site ascii ftp mls put size bell get mode pwd sndbuf binary gate modtime quit status bye glob more quote struct case hash mput rcvbuf sunique cd help msend recv system cdup idle newer reget tenex chmod image nlist rename trace close lcd nmap reset type cr less ntrans restart umask debug lpwd open rhelp user delete ls page rmdir verbose dir macdef passive rstatus ?
CVS COMMAND EXAMPLES - UNIX / LINUX TUTORIALS
CVS
(concurrent Version Control system) is a version controlling system
used to record the history of the files. Whenever a code is changed in
software, there might be chance of bugs creeping into that. With CVS,
you can easily get the old version of the code and see what part of the
code exactly created the bug.
In CVS, we can save every version of the file. The CVS only stores the differences between the files. This saves a huge amount of disk space.
The general syntax of CVS command is
CVS Command Examples:
1. Checking out file
You can check out a file from the CVS repository with the checkout (co) option. This is shown below:
2. Adding a file to the repository
Use the add option to add a new file to the cvs repository.
This will not commit the file to the cvs. It just simply adds the file.
3. Committing the file.
Once you have added a file to the CVS repository, you have to commit the file. Use the commit option with cvs command for committing a file.
This will open an editor. Enter the comments and save by using the :wq.
4. Difference between files
You can find the differences between the local file with the latest version of the file in the cvs repository using the diff option.
5. Update the file
You can update the local file with the latest version of the file from CVS repository using the update option.
6. Update to particular version
You can get a particular version of the file from the cvs. Specify the version number of the file with -j option.
7. Adding binary files
You can add binary or image files to the CVS repository. Use the -kb option to add binary files.
8. Removing file from CVS
You can remove unwanted files permanently from the CVS repository using the remove option.
After issuing this command you have to do a cvs commit. Otherwise the file will not be removed from the repository.
In CVS, we can save every version of the file. The CVS only stores the differences between the files. This saves a huge amount of disk space.
The general syntax of CVS command is
cvs [option] filename
CVS Command Examples:
1. Checking out file
You can check out a file from the CVS repository with the checkout (co) option. This is shown below:
cvs co oracle_storage.dat
2. Adding a file to the repository
Use the add option to add a new file to the cvs repository.
cvs add mysql.bat
This will not commit the file to the cvs. It just simply adds the file.
3. Committing the file.
Once you have added a file to the CVS repository, you have to commit the file. Use the commit option with cvs command for committing a file.
cvs commit msql.bat
This will open an editor. Enter the comments and save by using the :wq.
4. Difference between files
You can find the differences between the local file with the latest version of the file in the cvs repository using the diff option.
cvs diff wireless.php
5. Update the file
You can update the local file with the latest version of the file from CVS repository using the update option.
cvs update -A network.dat
6. Update to particular version
You can get a particular version of the file from the cvs. Specify the version number of the file with -j option.
cvs update -j version-number network.bat
7. Adding binary files
You can add binary or image files to the CVS repository. Use the -kb option to add binary files.
cvs add -kb unix.png
8. Removing file from CVS
You can remove unwanted files permanently from the CVS repository using the remove option.
cvs remove linux_system.dat
After issuing this command you have to do a cvs commit. Otherwise the file will not be removed from the repository.
KILL COMMAND EXAMPLES IN UNIX / LINUX TUTORIALS
The
Kill command in unix or linux operating system is used to send a signal
to the specified process or group. If we dont specify any signal, then
the kill command passes the SIGTERM signal. We mostly use the kill
command for terminating or killing a process. However we can also use
the kill command for running a stopped process.
The syntax of kill command is
The options to the kill command are:
Let see some of the useful kill command examples in unix or linux system.
Kill Command Examples:
1. Listing all the signal names.
Run the kill command with -l option to list all the available signal names.
Some the important signals which we use in our daily usage ar listed below:
To know more about a signal, check in man pages. To know about the signal 9, run the below man command:
2. Getting the process id
To know the process id of a process running in the unix system, use the ps command as
The second field in the output is the process Id. Here the /usr/local/sbin/sshd -R is running with the process id 4529.
3. Killing a process.
To kill processes simply pass the process id to the kill command. This is shown below:
4. Forcefully killing a process.
Use the -9 option with the kill command to kill a process force fully. The following kill command terminates the process forcefully:
Avoid using the kill -9 for terminating a process. This will cause memory leaks in the operating system and leads to many other issues.
The syntax of kill command is
kill [-s signal] pid kill -l
The options to the kill command are:
- pid : list of process that kill command should send a signal
- -s signal : send the specified signal to the process
- -l : list all the available signals.
Let see some of the useful kill command examples in unix or linux system.
Kill Command Examples:
1. Listing all the signal names.
Run the kill command with -l option to list all the available signal names.
> kill -l HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM USR1 USR2 CLD PWR WINCH URG POLL STOP TSTP CONT TTIN TTOU VTALRM PROF XCPU XFSZ WAITING LWP FREEZE THAW CANCEL LOST RTMIN RTMIN+1 RTMIN+2 RTMIN+3 RTMAX-3 RTMAX-2 RTMAX-1 RTMAX
Some the important signals which we use in our daily usage ar listed below:
| Number | Signal Name | Description |
|---|---|---|
| 0 | SIGNULL | Used to check access to the process id |
| 1 | SIGHUP | Hup signal. Terminates the process. |
| 2 | SIGNINT | Interrupt signal. Terminating the process |
| 3 | SIGQUIT | Quit signal. Terminate process with core dump |
| 9 | SIGKILL | Forcibly killing a process |
| 24 | SIGSTOP | Pausing the process |
| 26 | SIGCONT | Runs a stopped process |
To know more about a signal, check in man pages. To know about the signal 9, run the below man command:
man 5 signal
2. Getting the process id
To know the process id of a process running in the unix system, use the ps command as
ps -aef root 4529 657 0 Jul 20 ? 0:06 /usr/local/sbin/sshd -R .... ....
The second field in the output is the process Id. Here the /usr/local/sbin/sshd -R is running with the process id 4529.
3. Killing a process.
To kill processes simply pass the process id to the kill command. This is shown below:
kill 4529
4. Forcefully killing a process.
Use the -9 option with the kill command to kill a process force fully. The following kill command terminates the process forcefully:
kill -9 1567 kill -SIGKILL 1567 kill -KILL 1567 kill -s SIGKILL 1567 kill -s KILL 1567
Avoid using the kill -9 for terminating a process. This will cause memory leaks in the operating system and leads to many other issues.
SSH COMMAND EXAMPLES - UNIX / LINUX TUTORIALS
SSH
client utility in unix or linux server is used to logging into a remote
host and execute commands on the remote machine. The rlogin and rsh
commands can also be used to login into the remote machine. However
these are not secure. The ssh command provides a secure connection
between two hosts over a insecure network.
The syntax ssh command is
Let see the examples of ssh command.
SSH Command Examples:
1. Logging to a remote server
You can login to a remote server from the local host as shown below:
Alternatively you can use the below ssh command for connecting to remote host:
Note: If you are logging for the first time, then it will prints a message that host key not found and you can give yes to continue. The host key of the remote server will be cached and added to the .ssh2/hostkeys directory in your home directory. From second time onwards you just need to enter the password.
2. Logging out from remote server
Simply enter the exit command on the terminal to close the connection. This is shown below:
3. Running remote commands from local host
Sometimes it is necessary to run the unix commands on the remote server from the local host. An example is shown below:
The ssh command connects to the remote host, runs the ls command, prints the output on the local host terminal and exits the connection from remote host.
Let see whether the ls command actually displayed the correct result or not by connecting to the remote host.
4. Version of the SSH command
We can find the version of SSH installed on the unix system using the -V option to the ssh. This is shown below:
5. Debugging the SSH Client
When we are not able to connect to the remote host, it is good to debug and find the exact error messages that causing the issue. Use the -v option for debugging the ssh client.
6. Copying files between remote host and local host.
We can use the scp command to copy the files securely between the local host and remote host using the ssh authentication.
To copy the file from local host to remote hosts /var/tmp/ directory, run the below scp command.
To copy the file from remote hosts /usr/local/bin/ directory to local hosts current directory, run the below scp command.
The syntax ssh command is
ssh [-l username] hostname | user@remote-hostname [command]
Let see the examples of ssh command.
SSH Command Examples:
1. Logging to a remote server
You can login to a remote server from the local host as shown below:
localhost:[~]> ssh -l username remote-server username@remote-server password: remote-server:[~]>
Alternatively you can use the below ssh command for connecting to remote host:
localhost:[~]> ssh username@remote-server username@remote-server password: remote-server:[~]>
Note: If you are logging for the first time, then it will prints a message that host key not found and you can give yes to continue. The host key of the remote server will be cached and added to the .ssh2/hostkeys directory in your home directory. From second time onwards you just need to enter the password.
2. Logging out from remote server
Simply enter the exit command on the terminal to close the connection. This is shown below:
remote-server:[~]>exit logout Connection to remote-server closed. localhost:[~]>
3. Running remote commands from local host
Sometimes it is necessary to run the unix commands on the remote server from the local host. An example is shown below:
localhost:[~]> ssh user@remote-host "ls test" online-backup.dat oracle-storage.bat unix-dedicated-server.txt
The ssh command connects to the remote host, runs the ls command, prints the output on the local host terminal and exits the connection from remote host.
Let see whether the ls command actually displayed the correct result or not by connecting to the remote host.
localhost:[~]> ssh user@remote-host user@remotehost password: remotehost:[~]> cd test remotehost:[~/test]> ls online-backup.dat oracle-storage.bat unix-dedicated-server.txt
4. Version of the SSH command
We can find the version of SSH installed on the unix system using the -V option to the ssh. This is shown below:
> ssh -V OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
5. Debugging the SSH Client
When we are not able to connect to the remote host, it is good to debug and find the exact error messages that causing the issue. Use the -v option for debugging the ssh client.
ssh -v user@remote-host OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to remote-host [172.22.200.140] port 22. debug1: Connection established. debug1: identity file /home/user/.ssh/identity type -1 debug1: identity file /home/user/.ssh/id_rsa type -1 debug1: identity file /home/user/.ssh/id_dsa type 2 debug1: loaded 3 keys .......... ..........
6. Copying files between remote host and local host.
We can use the scp command to copy the files securely between the local host and remote host using the ssh authentication.
To copy the file from local host to remote hosts /var/tmp/ directory, run the below scp command.
scp filename user@remote-host:/var/tmp/
To copy the file from remote hosts /usr/local/bin/ directory to local hosts current directory, run the below scp command.
scp user@remote-host:/usr/local/bin/add.sh .
WC COMMAND EXAMPLES - COUNT OF LINES, WORDS, CHARACTERS - UNIX / LINUX
WC
command in unix or linux is used to find the number of lines, words and
characters in a file. The syntax of wc command is shown below:
You can use the following options with the wc command.
Let see how to use the wc command with few examples. Create the following file in your unix or linux operating system.
WC Command Examples:
1. Printing count of lines
This is the most commonly used operation to find the number of lines from a file. Run the below command to display the number of lines:
Here in the output, the first field indicates count and second field is the filename
2. Displaying the number of words.
Just use the -w option to find the count of words in a file. This is shown below:
3. Print count of bytes, count of characters from a file
We can use the -c and -m options to find the number of bytes and characters respectively in a file.
4. Print the length of longest line
The -L option is used to print the number of characters in the longest line from a file.
In this example, the second line is the longest line with 23 characters.
5. Print count of lines, words and characters.
If you dont specify any option to the wc command, by default it prints the count of lines, words and characters. This is shown below:
6. Wc help
For any help on the wc command, just run the wc --help on the unix terminal.
wc [options] filenames
You can use the following options with the wc command.
-l : Prints the number of lines in a file. -w : prints the number of words in a file. -c : Displays the count of bytes in a file. -m : prints the count of characters from a file. -L : prints only the length of the longest line in a file.
Let see how to use the wc command with few examples. Create the following file in your unix or linux operating system.
> cat unix_wc.bat Oracle Storage unix distributed system linux file server debian server Oracle backup server
WC Command Examples:
1. Printing count of lines
This is the most commonly used operation to find the number of lines from a file. Run the below command to display the number of lines:
wc -l unix_wc.bat 5 unix_wc.bat
Here in the output, the first field indicates count and second field is the filename
2. Displaying the number of words.
Just use the -w option to find the count of words in a file. This is shown below:
wc -w unix_wc.bat 13 unix_wc.bat
3. Print count of bytes, count of characters from a file
We can use the -c and -m options to find the number of bytes and characters respectively in a file.
> wc -c unix_wc.bat 92 unix_wc.bat > wc -m unix_wc.bat 92 unix_wc.bat
4. Print the length of longest line
The -L option is used to print the number of characters in the longest line from a file.
wc -L unix_wc.bat 23 unix_wc.bat
In this example, the second line is the longest line with 23 characters.
5. Print count of lines, words and characters.
If you dont specify any option to the wc command, by default it prints the count of lines, words and characters. This is shown below:
wc unix_wc.bat 5 13 92 unix_wc.bat
6. Wc help
For any help on the wc command, just run the wc --help on the unix terminal.
SCP COMMAND EXAMPLES - LINUX / UNIX TUTORIALS
SCP
stands for secure copy is used to copy data (files or directories) from
one unix or linux system to another unix or linux server. SCP uses
secured shell (ssh) to transfer the data between the remote hosts. The
features of SCP are:
SCP Command Syntax:
The syntax of SCP command is
Each element of the scp command is explained in detail below:
SCP Command Options:
The important SCP command options are listed below:
SCP Command Examples:
Let see the examples of scp command in unix or linux system.
1. Copying with in the same system
You can use the scp command just like the cp command to copy files from one directory to another directory.
This command copies the file unix-storage.dat from current directory to the /var/tmp directory.
2. Copy file from local host to remote server
This is most frequently used operation to transfer files in unix system.
This command connects to the remote host and copies the specified file to the /remote/directory/.
3. Copy files from remote host to local server.
This operation is used when taking backup of the files in remote server.
This command copies the oracle backup file in the remote host to the current directory.
4. Copying files between two remote servers
The scp command can also be used to copy files between two remote hosts.
The above command copies the mysql bakup shell script from the source remote host the /var/tmp directory of target remote host.
5. Copying a directory.
To copy all the files in a directory, use the -r option with the scp command. This makes the scp command to copy the directory recursively.
The above command copies the directory from local server to the remote host.
6. Improving performance of scp command
By default the scp command uses the Triple-DES cipher/AES-128 to encrypt the data. Using the blowfish or arcfour encryption will improve the performance of the scp command.
7. Limit bandwidth
You can limit the bandwidth used by the scp command using the -l option.
8. Specifying the port number
We can make the scp command to copy the files over a specified port number using the -P option.
- Copies files within in the same machine
- Copies files from local machine to remote machine.
- Copies files from remote machine to local machine.
- Copies files between two different remote servers.
SCP Command Syntax:
The syntax of SCP command is
scp [Options] [[User@]From_Host:]Source_File [[User@]To_Host:][Destination_File]
Each element of the scp command is explained in detail below:
- User is the one who have the permissions to access the files and directories. User should have read permissions if it is a source and write permissions if it is the destination.
- From_Host: hostname or Ip address where the source file or directory resides. This is optional if the from host is the host where you are running the scp command.
- Source_File: Files or directories to be copied to the destination.
- To_Host: Destination host where you want to copy the files. You can omit this when you want to copy the files to the host where you are issuing the scp command.
- Destination_File: Name of the file or directory in the target host.
SCP Command Options:
The important SCP command options are listed below:
- -r : Recursively copies the contents of source files or directories.
- -p : Preserves the access time, modification time, permissions of the source files in the destination.
- -q : Progress bar in not displayed
- -v : verbose mode. Displays debugging messages.
- -P : copy files using the specified port number.
SCP Command Examples:
Let see the examples of scp command in unix or linux system.
1. Copying with in the same system
You can use the scp command just like the cp command to copy files from one directory to another directory.
scp Unix-storage.dat /var/tmp/
This command copies the file unix-storage.dat from current directory to the /var/tmp directory.
2. Copy file from local host to remote server
This is most frequently used operation to transfer files in unix system.
scp filename user@remotehost:/remote/directory/
This command connects to the remote host and copies the specified file to the /remote/directory/.
3. Copy files from remote host to local server.
This operation is used when taking backup of the files in remote server.
scp user@remotehost:/usr/backup/oracle_backup.dat .
This command copies the oracle backup file in the remote host to the current directory.
4. Copying files between two remote servers
The scp command can also be used to copy files between two remote hosts.
scp source_user@source_remote_host:/usr/bin/mysql_backup.sh target_user@target_remote_host:/var/tmp/
The above command copies the mysql bakup shell script from the source remote host the /var/tmp directory of target remote host.
5. Copying a directory.
To copy all the files in a directory, use the -r option with the scp command. This makes the scp command to copy the directory recursively.
scp -r directory user@remotehost:/var/tmp/
The above command copies the directory from local server to the remote host.
6. Improving performance of scp command
By default the scp command uses the Triple-DES cipher/AES-128 to encrypt the data. Using the blowfish or arcfour encryption will improve the performance of the scp command.
scp -c blowfish filename user@remoteserver:/var/ scp -c arcfour localfile user@remoteserver:/var/
7. Limit bandwidth
You can limit the bandwidth used by the scp command using the -l option.
scp -l bandwidth_limit filename user@hostname:/usr/backup/ Here bandwidth_limit is numeric to be specified in kilobits per second.
8. Specifying the port number
We can make the scp command to copy the files over a specified port number using the -P option.
scp -P 6001 storage_backup.bat username@hostname:/tmp/
XARGS COMMAND EXAMPLES IN UNIX / LINUX TUTORIAL
Xargs
command in unix or linux operating system is used to pass the output of
one command as an argument to another command. Some of the unix or
linux commands like ls and find produces a long list of filenames. We
want to do some operation on this list of file names like searching for a
pattern, removing and renaming files etc. The xargs command provide
this capability by taking the huge list of arguments as input , divides
the list into small chunks and then passes them as arguments to other
unix commands.
Unix Xargs Command Examples:
1. Renaming files with xargs
We have to first list the files to be renamed either by using the ls or find command and then pipe the output to xargs command to rename the files. First list the files which end with ".log" using the ls command.
You can see how the log files are renamed with backup (bkp) suffix. Here the option "i" tells the xargs command to replace the {} with the each file returned by the ls command.
2. Searching for a pattern
We can combine the grep command with xargs to search for a pattern in a list of files returned by another unix command (ls or find). Let’s list out all the bash files in the current directory with the find command in unix.
Now we grep for the "echo" statements from the list of files returned by the find command with the help of xargs. The command is shown below:
If you don’t use xargs and piped the output of find command to grep command directly, then the grep command treats each file returned by the find command as a line of string and searches for the word "echo" in that line rather in that file.
3. Removing files using xargs
We can remove the temporary files in a directory using the rm command along with the xargs command. This is shown below:
This removes all the files with ".tmp" suffix.
4. Converting Multi-line output to Single line output.
If you run the ls -1 command, it will list each file on a separate line. This is shown below:
We can convert this multi-line output to single line output using the xargs command. This is shown below:
5. Handling spaces in file names
By default the xargs command treats the space as a delimiter and sends each item as an argument to the unix command. If the file name contains a space (example: "oracle storage"), then each item will be treated as a separate file and will be passed as an argument. This will cause an issue. Let see how to handle the spaces in file names with an example.
You can see that grep command is treating oracle as separate file and storage as separate file. This is because of xargs treats space as delimiter. To avoid this kind of errors use the -i option with braces as shown in below:
If you want to know what command the xargs is executing use the -t option with xargs. This will print the command on the terminal before executing it.
6. Passing subset of arguments
We can pass only a subset of arguments from a long list of arguments using the -n option with xargs command. This is shown in below.
You can see from the above output that 3 arguments are passed at a time to the echo statement.
Important Notes on Xargs Command:
1. Xargs directly cannot handle files which contain new lines or spaces in their names. To handle this kind of files use the -i option with xargs command. Another way to handle these characters is to treat the new line or spaces as null characters using th -0 option with xargs. However this requires that the input to xargs should also use the null as separator. An example is shown below
The print0 in find command makes the newline or space characters as null separator.
2. By default the xargs uses the end of the file string as "_". If this string appears in input string, then xargs command stops reading the input and rest of the input is ignored. You can change the end of file string by using the "-eof" option.
3. To know more about xargs command, run the xargs --help on the unix or linux terminal.
Unix Xargs Command Examples:
1. Renaming files with xargs
We have to first list the files to be renamed either by using the ls or find command and then pipe the output to xargs command to rename the files. First list the files which end with ".log" using the ls command.
ls *.log oracle.log storage.log
> ls *.log | xargs -i mv {} {}_bkp
> ls *_bkp
oracle.log_bkp storage.log_bkp
You can see how the log files are renamed with backup (bkp) suffix. Here the option "i" tells the xargs command to replace the {} with the each file returned by the ls command.
2. Searching for a pattern
We can combine the grep command with xargs to search for a pattern in a list of files returned by another unix command (ls or find). Let’s list out all the bash files in the current directory with the find command in unix.
find . -name "*.bash" ./sql_server.bash ./mysql_backup.bash ./oracle_backup.bash
Now we grep for the "echo" statements from the list of files returned by the find command with the help of xargs. The command is shown below:
find . -name "*.bash" |xargs grep "echo"
If you don’t use xargs and piped the output of find command to grep command directly, then the grep command treats each file returned by the find command as a line of string and searches for the word "echo" in that line rather in that file.
3. Removing files using xargs
We can remove the temporary files in a directory using the rm command along with the xargs command. This is shown below:
ls "*.tmp" | xargs rm
This removes all the files with ".tmp" suffix.
4. Converting Multi-line output to Single line output.
If you run the ls -1 command, it will list each file on a separate line. This is shown below:
ls -1 oracle.txt online_backup.dat mysql_storage.bat
We can convert this multi-line output to single line output using the xargs command. This is shown below:
ls -1 | xargs oracle.txt online_backup.dat mysql_storage.bat
5. Handling spaces in file names
By default the xargs command treats the space as a delimiter and sends each item as an argument to the unix command. If the file name contains a space (example: "oracle storage"), then each item will be treated as a separate file and will be passed as an argument. This will cause an issue. Let see how to handle the spaces in file names with an example.
Creating a file which contains space in the name > touch "oracle storage" > ls oracle\ storage | xargs grep "log" grep: oracle: No such file or directory grep: storage: No such file or directory
You can see that grep command is treating oracle as separate file and storage as separate file. This is because of xargs treats space as delimiter. To avoid this kind of errors use the -i option with braces as shown in below:
> ls oracle\ storage | xargs -i grep "log" {}
If you want to know what command the xargs is executing use the -t option with xargs. This will print the command on the terminal before executing it.
6. Passing subset of arguments
We can pass only a subset of arguments from a long list of arguments using the -n option with xargs command. This is shown in below.
> ls -1 backup mysql network online oracle storage wireless > ls -1 | xargs -n 3 echo backup mysql network online oracle storage wireless
You can see from the above output that 3 arguments are passed at a time to the echo statement.
Important Notes on Xargs Command:
1. Xargs directly cannot handle files which contain new lines or spaces in their names. To handle this kind of files use the -i option with xargs command. Another way to handle these characters is to treat the new line or spaces as null characters using th -0 option with xargs. However this requires that the input to xargs should also use the null as separator. An example is shown below
find . -print0 | xargs -0 rm
The print0 in find command makes the newline or space characters as null separator.
2. By default the xargs uses the end of the file string as "_". If this string appears in input string, then xargs command stops reading the input and rest of the input is ignored. You can change the end of file string by using the "-eof" option.
3. To know more about xargs command, run the xargs --help on the unix or linux terminal.
BASH SHELL SCRIPT TO READ / PARSE COMMA SEPARATED (CSV) FILE - UNIX / LINUX
Q) How to parse CVS files and print the contents on the terminal using the bash shell script in Unix or Linux system?
It is the most common operation in Unix system to read the data from a delimited file and applying some operations on the data. Here we see how to read the Comma separated value (CSV) file using the while loop in shell script and print these values on the Unix terminal.
Consider the below CSV file as an example:
This file contains two fields. First field is operating system and the second field contains the hosting server type. Let see how to parse this CVS file with simple bash script shown below:
Here IFS is the input field separator. As the file is comma delimited, the IFS variable is set with comma. The output of the above script is
Here in the code, the fourth line (IFS=',') and sixth line (while) can be merged into a single statement as shown below:
It is the most common operation in Unix system to read the data from a delimited file and applying some operations on the data. Here we see how to read the Comma separated value (CSV) file using the while loop in shell script and print these values on the Unix terminal.
Consider the below CSV file as an example:
> cat os_server.csv Unix, dedicated server Linux, virtual server
This file contains two fields. First field is operating system and the second field contains the hosting server type. Let see how to parse this CVS file with simple bash script shown below:
#!/usr/bin/bash INPUT_FILE='unix_file.csv' IFS=',' while read OS HS do echo "Operating system - $OS" echo "Hosting server type - $HS" done < $INPUT_FILE
Here IFS is the input field separator. As the file is comma delimited, the IFS variable is set with comma. The output of the above script is
Operating system - Unix Hosting server type - dedicated server Operating system - Linux Hosting server type - virtual server
Here in the code, the fourth line (IFS=',') and sixth line (while) can be merged into a single statement as shown below:
while IFS=',' read OS HS
ADD JOB TO CRON (CRONTAB COMMAND EXAMPLES) - UNIX / LINUX TUTORIALS
Unix
or Linux operating system provides a feature for scheduling the jobs.
You can setup command or scripts which will run periodically at the
specified time. The Crontab is command used to add or remove jobs from
the cron. The cron service is a daemon runs in the background and checks
for /etc/crontab file, /etc/con.*/ directories and /var/spool/cron/
directory for any scheduled jobs.
Each user has a separate /var/spool/cron/crontab file. Users are not allowed directly to modify the files. The crontab command is used for setting up the jobs in the cron.
The format of crontab command is
You can easily remember this command in the below format
The field descriptions of the crontab are explained below:
Let see the usage of crontab command with examples.
1. List crontab entries
You can list out all the jobs which are already scheduled in cron. Use "crontab -l" for listing the jobs.
The above contab command displays the cron entries. Here the shell script for listing the unix versions (list_unix_version.sh) is scheduled to run daily at midnight.
2. List crontab entries of other users
To list the corntab entries of other user in the unix, use the -u option with crontab. The syntax is shown below:
3. Removing all crontab entries
You can un-schedule all the jobs by removing them from the crontab. The syntax for removing all the crontab entries is
4. Editing the crontab
You can edit the crontab and add a new job to it. You can also remove an existing job from the crontab. Use the -e option for editing the crontab.
This will open a file in VI editor. Now use the VI commands for adding, removing the jobs and for saving the crontab entries.
5. Schedule a job to take oracle backup on every Sunday at midnight
Edit crontab using "crontab -e" and append the following entry in the file.
6. Schedule a job to run every six hours in a day
You can schedule a job to run more than once in a day. As an example the following crontab entry takes the mysql backup more than once in a day.
Here the list 0,6,12,18 indicates midnight, 6am, 12pm and 6pm respectively.
7. Schedule job to run for the first 15 days of the month.
You can schedule a job by specifying the range of values for a field. The following example takes the sql server backup daily at midnight for the first 15 days in a month.
8. Schedule job to run every minute.
The following crontab command runs the command to send emails to group of users for every minute.
9. Taking backup of cron entries
Before editing the cron entries, it is good to take backup of the cron entries. So that even if you do mistake you can get back those entries from the backup.
10. Restoring the cron entries
You can restore the cron entries from the backup as
Understanding the Operators:
There are three operators allowed for specifying the scheduling times. They are:
Disabling Emails:
By default the crontab sends emails to the local user if the commands or scripts produce any output. To disable sending of emails redirect the output of commands to /dev/null 2>&1.
Note: you cannot schedule a job to run at second’s level as the minimum allowed scheduling is at minute level.
Each user has a separate /var/spool/cron/crontab file. Users are not allowed directly to modify the files. The crontab command is used for setting up the jobs in the cron.
The format of crontab command is
* * * * * command to be executed
You can easily remember this command in the below format
MI HH DOM MON DOW command
The field descriptions of the crontab are explained below:
MI : Minutes from 0 to 59 HH : Hours from 0 to 23 DOM : Day of month from 0 to 31 MON : Months from 1 to 12 DOW : Day of week from 0 to 7 (0 or 7 represents Sunday) Command: Any command or script to be scheduled
Let see the usage of crontab command with examples.
1. List crontab entries
You can list out all the jobs which are already scheduled in cron. Use "crontab -l" for listing the jobs.
crontab -l 0 0 * * * /usr/local/bin/list_unix_versions.sh
The above contab command displays the cron entries. Here the shell script for listing the unix versions (list_unix_version.sh) is scheduled to run daily at midnight.
2. List crontab entries of other users
To list the corntab entries of other user in the unix, use the -u option with crontab. The syntax is shown below:
crontab -u username -l
3. Removing all crontab entries
You can un-schedule all the jobs by removing them from the crontab. The syntax for removing all the crontab entries is
crontab -r For removing other user’s crontab entries: crontab -u username -r
4. Editing the crontab
You can edit the crontab and add a new job to it. You can also remove an existing job from the crontab. Use the -e option for editing the crontab.
crontab -e For editing other user’s crontab entries: crontab -u username -e
This will open a file in VI editor. Now use the VI commands for adding, removing the jobs and for saving the crontab entries.
5. Schedule a job to take oracle backup on every Sunday at midnight
Edit crontab using "crontab -e" and append the following entry in the file.
0 0 * * 0 /usr/local/bin/oracle_backup.sh
6. Schedule a job to run every six hours in a day
You can schedule a job to run more than once in a day. As an example the following crontab entry takes the mysql backup more than once in a day.
0 0,6,12,18 * * * /usr/bin/mysql_backup.sh
Here the list 0,6,12,18 indicates midnight, 6am, 12pm and 6pm respectively.
7. Schedule job to run for the first 15 days of the month.
You can schedule a job by specifying the range of values for a field. The following example takes the sql server backup daily at midnight for the first 15 days in a month.
0 0 * 1-15 * /usr/bin/sql_server_backup.sh
8. Schedule job to run every minute.
The following crontab command runs the command to send emails to group of users for every minute.
* * * * * /bin/batch_email_send.sh
9. Taking backup of cron entries
Before editing the cron entries, it is good to take backup of the cron entries. So that even if you do mistake you can get back those entries from the backup.
crontab -l > /var/tmp/cron_backup.dat
10. Restoring the cron entries
You can restore the cron entries from the backup as
crontab cron_backup.dat
Understanding the Operators:
There are three operators allowed for specifying the scheduling times. They are:
- Asterisk (*) : Indicates all possible values for a field. An asterisk in the month field indicates all possible months (January to December).
- Comma (,) : Indicates list of values. See example 6 above.
- Hyphen (-): Indicates range of values. See example 7 above.
Disabling Emails:
By default the crontab sends emails to the local user if the commands or scripts produce any output. To disable sending of emails redirect the output of commands to /dev/null 2>&1.
0 0 * 20 * /usr/bin/online_backup.sh > /dev/null 2>&1
Note: you cannot schedule a job to run at second’s level as the minimum allowed scheduling is at minute level.
YUM COMMAND EXAMPLES - INSTALL, UPDATE, UNINSTALL - LINUX TUTORIALS
Yum
(Yellowdog Updater Modified) is one of the package manager utility in
Linux operating system. Yum command is used for installing, updating and
removing packages on Linux environment. Some other package manger
utilities in linux system are apt-get, dpkg, rpm etc.
By default yum is installed on some of the linux distributions like CentOS, Fedora, Redhat. Let see some of the mostly used yum commands with examples.
1. Listing available packages
You can list all the available packages in the yum repository using the list.
2. View installed packages
To print all the packages which are installed on your linux system, execute the following command.
3. Search for package
Searching for a package to be installed helps you when the exact package name is not known in advance. The syntax for searching a package is
If you want to search for mysql package, then execute the following yum command.
However this yum command matches only in the name and summary. Use "search all" for everything.
4. How to install package using yum
"Yum install package_name" will install the specified package name in the linux operating system. The yum command will automatically finds the dependencies and also installs them in the linux machine.
Yum will prompt for the user to accept or decline before installing the package. If you want yum to avoid prompting the user, then use -y option with yum command.
5. Check package is installed or not.
After installing, you don’t know whether the package is installed or not. To check whether a package is installed or not, run the below yum command.
6. Updating existing package using yum
You can upgrade older version of package to the newer version by using the yum command. To check for the updates and for upgrading a package to the current version, run the following yum command.
7. Uninstalling a package.
You can remove (uninstall) a package with all dependencies using the "yum remove package_name". This is shown below:
The yum remove prompts the user to accept or decline the uninstalling of package.
8. Information about the package.
You can print and check for the information about the package before installing it. Execute the following yum command to get info about the package.
9. Print what package provides the file.
You can know the name of the package that file belongs to. For example, if you want to know the file '/etc/passwd' belongs to which file, run the below yum command.
10. Print List of group software
Basically, the related softwares are grouped together. You can install all the packages belong to a single group at one shot. This will help you in saving time in installing each individual package. To print the list of available package groups, run the below yum command.
11. Installing a software group.
You can install the group software, by running the following command.
12. Update a software group.
You can update the installed software group from a older version to latest version. The yum command for this is
13. Removing a software group
You can uninstall (delete) existing software group using 'yum groupremove'. This is shown below:
14. Print yum repository
You can display all the packages in the yum repository using the below command.
Here "all" is optional. If you provide "all", then it displays enabled and disabled repositories. Otherwise it displays only enabled repositories.
15. More info about yum command
If you want to know more information about the yum command, then run the man on yum as
By default yum is installed on some of the linux distributions like CentOS, Fedora, Redhat. Let see some of the mostly used yum commands with examples.
1. Listing available packages
You can list all the available packages in the yum repository using the list.
yum list yum list httpd Display list of updated softwares yum list updates
2. View installed packages
To print all the packages which are installed on your linux system, execute the following command.
yum list installed
3. Search for package
Searching for a package to be installed helps you when the exact package name is not known in advance. The syntax for searching a package is
yum search package_to_be_searched
If you want to search for mysql package, then execute the following yum command.
yum search mysql
However this yum command matches only in the name and summary. Use "search all" for everything.
4. How to install package using yum
"Yum install package_name" will install the specified package name in the linux operating system. The yum command will automatically finds the dependencies and also installs them in the linux machine.
yum install firefox.x86_64
Yum will prompt for the user to accept or decline before installing the package. If you want yum to avoid prompting the user, then use -y option with yum command.
yum -y install package_name
5. Check package is installed or not.
After installing, you don’t know whether the package is installed or not. To check whether a package is installed or not, run the below yum command.
yum list installed package_name yum list installed firefox.x86_64
6. Updating existing package using yum
You can upgrade older version of package to the newer version by using the yum command. To check for the updates and for upgrading a package to the current version, run the following yum command.
yum update firefox.x86_64
7. Uninstalling a package.
You can remove (uninstall) a package with all dependencies using the "yum remove package_name". This is shown below:
yum remove firefox.x86_64
The yum remove prompts the user to accept or decline the uninstalling of package.
8. Information about the package.
You can print and check for the information about the package before installing it. Execute the following yum command to get info about the package.
yum info firefox.x86_64
9. Print what package provides the file.
You can know the name of the package that file belongs to. For example, if you want to know the file '/etc/passwd' belongs to which file, run the below yum command.
yum whatprovides /etc/passwd
10. Print List of group software
Basically, the related softwares are grouped together. You can install all the packages belong to a single group at one shot. This will help you in saving time in installing each individual package. To print the list of available package groups, run the below yum command.
yum grouplist
11. Installing a software group.
You can install the group software, by running the following command.
yum groupinstall 'software_group_name'
12. Update a software group.
You can update the installed software group from a older version to latest version. The yum command for this is
yum groupupdate 'software_group_name'
13. Removing a software group
You can uninstall (delete) existing software group using 'yum groupremove'. This is shown below:
yum groupremove 'software_group_name'
14. Print yum repository
You can display all the packages in the yum repository using the below command.
yum repolist [all]
Here "all" is optional. If you provide "all", then it displays enabled and disabled repositories. Otherwise it displays only enabled repositories.
15. More info about yum command
If you want to know more information about the yum command, then run the man on yum as
man yum
REPLACE STRING ON NTH LINE - UNIX / LINUX
Q)
How to search for a string (or pattern) in a file and replace that
matched string with another string only on the specified line number in
the file?
Let see this with the help of an example. Consider the sample file with the following content:
First we will see how to replace a pattern with another patter using sed command. The sed command for replacing the string "Fedora" with "BSD" is:
The above sed command will replace the string on all the matched lines. In this example, it replaces the string on first, second, third and fifth lines.
If we want to replace the pattern on a specific line number, then we have to specify the line number to the sed command. The sed command syntax for replacing the pattern on Nth line is:
To replace "Fedora" with "BSD" on second line, run the below sed command on the unix terminal:
Let see this with the help of an example. Consider the sample file with the following content:
> cat fedora_overview.dat Fedora is an operating system based on linux kernal. Fedora applications include LibreOffice, Empathy and GIMP. Security features of Fedora is Security-Enhanced Linux. Implements variety of security policies, access controls. SELinux operating system was introduces in Fedora Core 2.
First we will see how to replace a pattern with another patter using sed command. The sed command for replacing the string "Fedora" with "BSD" is:
sed 's/Fedora/BSD/' fedora_oveview.dat
The above sed command will replace the string on all the matched lines. In this example, it replaces the string on first, second, third and fifth lines.
If we want to replace the pattern on a specific line number, then we have to specify the line number to the sed command. The sed command syntax for replacing the pattern on Nth line is:
sed 'n s/search_pattern/replace_pattern/' filename
To replace "Fedora" with "BSD" on second line, run the below sed command on the unix terminal:
sed '2 s/Fedora/BSD/' fedora_oveview.dat
DELETE RANGE OF LINES - UNIX / LINUX
Q) How to delete range of lines from a file using unix or inux command?
Unix provides simple way to delete lines whose number are line numbers are between m and n. This feature is not directly available in windows operating system.
We can use the sed command for removing the lines. The syntax for removing range of lines (between m and n) is:
Here the number n should be greater than m. Let see this with an example. Consider the sample file with the following contents:
From the above file if we want to delete the lines from number 2 to 4, then run the below sed command in unix:
However this command just prints the lines on the terminal and did not remove from the file. To delete the lines from the source file itself use the -i option to the sed command.
You can negate this operation and can delete lines that are not in the specified range.. This is shown in the following sed command:
Unix provides simple way to delete lines whose number are line numbers are between m and n. This feature is not directly available in windows operating system.
We can use the sed command for removing the lines. The syntax for removing range of lines (between m and n) is:
sed 'm,nd' filename
Here the number n should be greater than m. Let see this with an example. Consider the sample file with the following contents:
> cat linux_range.dat Linux is just like unix operating system. Linux is leading os on servers like mainframes and super computers. Linux also runs on embeded systems like network routers, mobiles etc. Android is built on using linux kernal system. Variants of linux are debian, fedora and open SUSE.
From the above file if we want to delete the lines from number 2 to 4, then run the below sed command in unix:
sed '2,4d' linux_range.dat
However this command just prints the lines on the terminal and did not remove from the file. To delete the lines from the source file itself use the -i option to the sed command.
sed -i '2,4d' linux_range.dat
You can negate this operation and can delete lines that are not in the specified range.. This is shown in the following sed command:
sed -i '2,4!d' linux_range.dat
REMOVE LAST LINE OR FOOTER LINE - UNIX / LINUX
Q) How to delete the trailer line (first line) from a file using the unix or inux command?
Let see how to remove the last line from a file with an example. Consider the file with sample content as shown below:
Unix Sed command is popularly used for searching a pattern and then replacing the matched pattern with another string. However we can also use the sed command for deleting the lines from a file.
To remove the last line from a file, run the below sed command:
Here $ represents the last line in a file. d is for deleting the line. The above command will display the contents of the file on the unix terminal excluding the footer line. However it does not delete the line from the source file. If you want the line to be removed from the source file itself, then use the -i option with sed command. This command is shown below:
If you want only the footer line to be present in the file and remove other lines from the line, then you have to negate the delete operation. For this use the exclamation (!) before the d. This is shown in the following sed command:
Let see how to remove the last line from a file with an example. Consider the file with sample content as shown below:
> cat unix_file.txt Unix is a multitasking, multi-user operating system. Unix operating system was first developed in assembly language. In later periods unix is developed with C programming. In academics BSD variant of unix is used. Mostly used unix flavors are Solaris, HP-UX and AIX.
Unix Sed command is popularly used for searching a pattern and then replacing the matched pattern with another string. However we can also use the sed command for deleting the lines from a file.
To remove the last line from a file, run the below sed command:
sed '$d' unix_file.txt
Here $ represents the last line in a file. d is for deleting the line. The above command will display the contents of the file on the unix terminal excluding the footer line. However it does not delete the line from the source file. If you want the line to be removed from the source file itself, then use the -i option with sed command. This command is shown below:
sed -i '$d' unix_file.txt
If you want only the footer line to be present in the file and remove other lines from the line, then you have to negate the delete operation. For this use the exclamation (!) before the d. This is shown in the following sed command:
sed -i '$!d' unix_file.txt
DELETE FIRST LINE OR HEADER LINE - UNIX / LINUX
Q) How to remove the header line (first line) from a file using the unix or inux command?
Let see how to delete the first line from a file with an example. Consider the file with sample content as shown below:
Mostly we see the sed command for replacing the strings in a file. We can also use the sed command for removing the lines in a file. To delete the first line from a file, run the following sed command:
Here 1 represents the first line. d is for deleting the line. The above command will print the contents of the file on the unix terminal by removing the first line from the file. However it does not remove the line from the source file. If you want to changes in the source file itself, then use the -i option with sed command. this command is shown below:
You can keep only the first line and remove the remaining lines from the file by negating the above sed command. You have to use the exclamation (!) before the d command. The following sed command keeps only the frist line in the file and removes the other lines:
Let see how to delete the first line from a file with an example. Consider the file with sample content as shown below:
> cat linux_file.txt First line a file is called header line. Remaining lines are generally called as detail or data lines. Another detail line which is in the third row. The last line in a unix file is called footer line.
Mostly we see the sed command for replacing the strings in a file. We can also use the sed command for removing the lines in a file. To delete the first line from a file, run the following sed command:
sed '1d' linux_file.txt
Here 1 represents the first line. d is for deleting the line. The above command will print the contents of the file on the unix terminal by removing the first line from the file. However it does not remove the line from the source file. If you want to changes in the source file itself, then use the -i option with sed command. this command is shown below:
sed -i '1d' linux_file.txt
You can keep only the first line and remove the remaining lines from the file by negating the above sed command. You have to use the exclamation (!) before the d command. The following sed command keeps only the frist line in the file and removes the other lines:
sed -i '1!d' unix_file.txt
PRINT LINES ENDING WITH STRING - UNIX / LINUX
Q) How to display the lines from a file that ends with specified string ( or pattern) using unix or linux commands?
Printing the lines that ends with specified pattern on the terminal is most commonly used operation in unix environment. Grep is the frequently used command in unix for searching a pattern in a file and printing the lines that contains the specified pattern.
We will see how to print the lines that end with the specified pattern with an example. Consider the Sample log file data as an example, which is shown below:
Now if we want to print the lines that end with the string "vmware", then use the grep command with dollar ($) in the pattern. The complete unix grep command is shown below:
Here $ is used to indicate the end of the line in a file. This grep command prints the lines that end with the word "vmware". In this example, the third and fourth lines are printed on the unix terminal.
If you want to display the lines that end with the word "system", then run the following grep command on unix command prompt:
Printing the lines that ends with specified pattern on the terminal is most commonly used operation in unix environment. Grep is the frequently used command in unix for searching a pattern in a file and printing the lines that contains the specified pattern.
We will see how to print the lines that end with the specified pattern with an example. Consider the Sample log file data as an example, which is shown below:
> cat unix_os_install.dat How to install unix virtual machine on windows operating system First download virtual box or vmware and install it on windows Next place the ubuntu CD and follow the instructions in vmware For running ubuntu allocate at least 512MB RAM in vmware After installing, start the unix operating system
Now if we want to print the lines that end with the string "vmware", then use the grep command with dollar ($) in the pattern. The complete unix grep command is shown below:
grep "vmware$" unix_os_install.dat
Here $ is used to indicate the end of the line in a file. This grep command prints the lines that end with the word "vmware". In this example, the third and fourth lines are printed on the unix terminal.
If you want to display the lines that end with the word "system", then run the following grep command on unix command prompt:
grep "windows$" unix_os_install.dat
SEARCH (MATCH) FOR WHOLE WORDS IN FILE - UNIX / LINUX
Q) How to print the lines from a file that contain the specified whole word using unix or linux command.
Whole words are complete words which are not part of another string. As an example consider the sentence, "How to initialize shell". Here the words "how, to, shell, initialize" are whole words. However the word "initial" is not a whole word as it is part of another string (initialize).
Let see this in detail with the help of an example. Consider the following sample data in a file:
Now we have the sample file. First we will see how to search for a word and print the lines with the help of grep command in unix. The following grep command prints the lines that have the word "match" in the line:
The above command displays the first two lines on the unix terminal. Even though the first line does not contain the whole word "match", the grep command displays the line as it matches for the word in the string "matching". This is the default behavior of grep command.
To print only the lines that contain the whole words, you have to use the -w option to the grep command. The grep command for this is:
Now the above command only displays the second line on the unix terminal. Another example for matching the whole word “boy” is shown below:
Whole words are complete words which are not part of another string. As an example consider the sentence, "How to initialize shell". Here the words "how, to, shell, initialize" are whole words. However the word "initial" is not a whole word as it is part of another string (initialize).
Let see this in detail with the help of an example. Consider the following sample data in a file:
> cat unix_word.txt matching for whole words in file is easy with unix use the unix grep command match for a pattern. Another example of whole word is: boy's Here boy is a whole word.
Now we have the sample file. First we will see how to search for a word and print the lines with the help of grep command in unix. The following grep command prints the lines that have the word "match" in the line:
grep "match" unix_word.txt
The above command displays the first two lines on the unix terminal. Even though the first line does not contain the whole word "match", the grep command displays the line as it matches for the word in the string "matching". This is the default behavior of grep command.
To print only the lines that contain the whole words, you have to use the -w option to the grep command. The grep command for this is:
grep -w "match" unix_word.txt
Now the above command only displays the second line on the unix terminal. Another example for matching the whole word “boy” is shown below:
Grep -w "boy" unix_word.txt
PRINT NON MATCHING LINES (INVERSE OF GREP COMMAND) - UNIX / LINUX
Q) How to print the lines from a file that does not contain the specified pattern using unix / linux command?
The grep command in unix by default prints the lines from a file that contain the specified pattern. We can use the same grep command to display the lines that do not contain the specified pattern. Let see this with the help of an example.
Consider the following sample file as an example:
First we will see how to display the lines that match a specified pattern. To print the lines that contain the word "ubuntu", run the below grep command on unix terminal:
The above command displays the third and fourth lines from the above sample file.
Now we will see how to print non matching lines which means the lines that do not contain the specified pattern. Use the -v option to the grep command for inverse matching. This is shown below:
This command prints the first, second and fifth lines from the example file.
The grep command in unix by default prints the lines from a file that contain the specified pattern. We can use the same grep command to display the lines that do not contain the specified pattern. Let see this with the help of an example.
Consider the following sample file as an example:
> cat unix_practice.txt You can practice unix commands by installing the unix operating system. There so many unix flavors available in the market. ubuntu is one of the used operating system. it is available for free. Go to the ubuntu website and download the OS image. Alternatively you can order for a free CD. The question is how to start learning unix. First know about the unix operating system in detail. Then start slowly learning unix commands one by one. Practice these unix command daily to have a grip.
First we will see how to display the lines that match a specified pattern. To print the lines that contain the word "ubuntu", run the below grep command on unix terminal:
grep "ubuntu" unix_practice.txt
The above command displays the third and fourth lines from the above sample file.
Now we will see how to print non matching lines which means the lines that do not contain the specified pattern. Use the -v option to the grep command for inverse matching. This is shown below:
grep -v "ubuntu" unix_practice.txt
This command prints the first, second and fifth lines from the example file.
PRINT LINES STARTING WITH STRING - UNIX / LINUX
Q) How to print the lines from a file that starts with specified string (pattern) using unix or linux commands?
Displaying lines that starts with specified pattern is most commonly used when processing log files in unix environment. Log files are used to store the messages of shell scripts (echo statements). We can search for errors in the log file using grep command. Generally, the error keyword will appear at the start of the line.
Sample log file data is shown below:
Now if we want to get the lines that start with the string "Error", then use the grep command with anchor (^) in the pattern. The complete unix grep command is shown below:
Here ^ is used to specify the start of the line in a file. This grep command will displays the lines that start with the word "Error". In this example, the third line starts with the specified pattern (Error).
Displaying lines that starts with specified pattern is most commonly used when processing log files in unix environment. Log files are used to store the messages of shell scripts (echo statements). We can search for errors in the log file using grep command. Generally, the error keyword will appear at the start of the line.
Sample log file data is shown below:
> cat linux_log_file.dat Success: Shell script execution started. Success: Exported the environment variables in shell environment. Success: Able to connect to oracle DB from bash script. Success: Run the SQL statement and inserted rows into table. Error: Unable to process the stored procedure. Message: Processing the statements in shell scripts stopped due to Error. Message: Script failed. Aborting the bash script to avoid further errors.
Now if we want to get the lines that start with the string "Error", then use the grep command with anchor (^) in the pattern. The complete unix grep command is shown below:
grep "^Error" linux_log_file.dat
Here ^ is used to specify the start of the line in a file. This grep command will displays the lines that start with the word "Error". In this example, the third line starts with the specified pattern (Error).
FIND THE COUNT OF LINES MATCHING THE PATTERN- UNIX /LINUX
Q) How to print the count of number of lines from a file that match the specified pattern in unix or linux operating system?
Let say you are looking for the word "unix" in a file and want to display the count of lines that contain the word "unix". We will see how to find the count with an example.
Assume that i have a file (unix_sample.dat) in my unix operating system. Sample data from the file is shown below:
In the sample data, the word "unix" appears in two lines. Now we will print this count on unix terminal using the commands in unix.
1. Using wc command.
We can pipe the output of grep command to wc command to find the number of lines that match a pattern. The unix command is
2. Using grep -c option
The grep command has the -c option to find the count of lines that match a pattern. The grep command for this is
Let say you are looking for the word "unix" in a file and want to display the count of lines that contain the word "unix". We will see how to find the count with an example.
Assume that i have a file (unix_sample.dat) in my unix operating system. Sample data from the file is shown below:
> cat unix_sample.dat Monitoring your hosting on unix server is very important. Otherwise you don’t know whether the unix server is running fine or not. Use monitoring tools or email alerts to get the status of the unix server. Send the status, logs in the email to your email Id.
In the sample data, the word "unix" appears in two lines. Now we will print this count on unix terminal using the commands in unix.
1. Using wc command.
We can pipe the output of grep command to wc command to find the number of lines that match a pattern. The unix command is
grep "unix" unix_sample.dat | wc -l
2. Using grep -c option
The grep command has the -c option to find the count of lines that match a pattern. The grep command for this is
grep -c "unix" unix_sample.dat
REMOVE EMPTY LINES FROM A FILE IN UNIX / LINUX
Unix commands can be used to remove the empty (blank) lines from a file. Let see this with the help of an example.
Consider the following data file as an example:
The above sample file contains three empty lines. We will see how to remove these blank lines with the help of unix / linux commands.
1. Remove empty lines with Grep command.
The grep command can be used to delete the blank lines from a file. The command is shown below:
Here ^ specifies the start of the line and $ specifies the end of the line. The -v option inverses the match of the grep command.
2. Delete blank lines with Sed command
The sed command can also be used to remove the empty lines from the file. This command is shown below:
The difference between the sed and grep here is: Sed command removes the empty lines from the file and prints on the unix terminal. Where as grep command matches the non-empty lines and displays on the terminal.
Consider the following data file as an example:
> cat linux_hosting.dat Hosting a website on linux operating system helps your site guarding from viruses. Host website on a dedicated linux server to reduce the load on the server. This improves the performance and provides good uptime of the site.
The above sample file contains three empty lines. We will see how to remove these blank lines with the help of unix / linux commands.
1. Remove empty lines with Grep command.
The grep command can be used to delete the blank lines from a file. The command is shown below:
grep -v "^$" linux_hosting.dat
Here ^ specifies the start of the line and $ specifies the end of the line. The -v option inverses the match of the grep command.
2. Delete blank lines with Sed command
The sed command can also be used to remove the empty lines from the file. This command is shown below:
sed '/^$/d' linux_hosting.dat
The difference between the sed and grep here is: Sed command removes the empty lines from the file and prints on the unix terminal. Where as grep command matches the non-empty lines and displays on the terminal.
PRINT N LINES AFTER A PATTERN MATCH - UNIX / LINUX
The
unix grep command can be used to print the lines from a file that match
specified pattern. The grep command has an option for printing the
lines around the line that match the pattern. Here we will see how to
display N line after a matching line with the help of an example.
Consider the below data file:
First of all we will see the general syntax for displaying N lines after the matched line. The syntax of grep command is:
This grep command will display N lines after the matched line and also prints the matched line. Now we will try to display the 2 lines after the line that contains the word "flexibility". The grep command for this is
This grep command will print first, second and third lines from the above file.
Consider the below data file:
> cat linux_enterpise.dat The advantage of linux operating system is its flexibility. You can use linux in various systems from mobile phones to space crafts. You have to choose a best linux operating system. Things involved in choosing are hardware, scale of software program etc. Some Os are ubuntu, fedora, linux mint, puppy linux etc.
First of all we will see the general syntax for displaying N lines after the matched line. The syntax of grep command is:
grep -A N "pattern" filename
This grep command will display N lines after the matched line and also prints the matched line. Now we will try to display the 2 lines after the line that contains the word "flexibility". The grep command for this is
grep -A 2 "flexibility" linux_enterpise.dat
This grep command will print first, second and third lines from the above file.
PRINT LINES BEFORE A PATTERN MATCH - UNIX / LINUX
We
know how to use the unix grep command to display the lines from a file
that match a pattern. However we can use the grep command to display the
lines around the line that match the pattern. We will see this with the
help of an example.
Consider the below data file which talks about importance of online backup:
First, we will see the general syntax for displaying N lines before the matched line. The syntax of grep command is:
This grep command will display N lines before the matched line and also prints the matched line. Now we will try to print the 2 lines before the line that contains the word "important". The grep command for this is
This will display second, third and fourth lines from the above file.
Consider the below data file which talks about importance of online backup:
> cat online_backup.dat The most important concern is to keep your documents safe and secure in a protected place. There are so many companies which offer online backup services. However selecting a good online backup service is important. The companies should offer a free trial of backup service. Use a backup software to take backup and restore of your data.
First, we will see the general syntax for displaying N lines before the matched line. The syntax of grep command is:
grep -B N "pattern" filename
This grep command will display N lines before the matched line and also prints the matched line. Now we will try to print the 2 lines before the line that contains the word "important". The grep command for this is
grep -B 2 "important" online_backup.dat
This will display second, third and fourth lines from the above file.
GREP STRING IN MULTIPLE FILES - UNIX / LINUX
The
Unix Grep command is used to search for a pattern in a line from a file
and if it founds the pattern displays the line on the terminal. We can
also use the grep command to match for a pattern in multiple files.
We will see this with the help of an example. Let’s consider two files shown below:
To grep for the word "hosting" from these two files specify both the file names as space separated list in grep command. The complete command is
The output of the above grep command is
This will display the filename along with the matching line. Instead of specifying each file name, you can specify a pattern (regular expressions) for the filename. Let say you want to grep for the word "company" in all the files whose name starts with "webhost_", you can use the below grep command:
We will see this with the help of an example. Let’s consider two files shown below:
> cat webhost_online.dat There are so many web hosting companies which provides services for hosting a website > cat webhost_trail.dat You can go for trail before choosing a web hosting company. Once you are happy with the free web hosting trail then you can host your website there.
To grep for the word "hosting" from these two files specify both the file names as space separated list in grep command. The complete command is
grep hosting webhost_online.dat webhost_trail.dat
The output of the above grep command is
webhost_online.dat:There are so many web hosting companies webhost_online.dat:which provides services for hosting a website webhost_trail.dat:You can go for trail before choosing a web hosting company. webhost_trail.dat:Once you are happy with the free web hosting trail
This will display the filename along with the matching line. Instead of specifying each file name, you can specify a pattern (regular expressions) for the filename. Let say you want to grep for the word "company" in all the files whose name starts with "webhost_", you can use the below grep command:
grep hosting webhost_*
CASE INSENSITIVE GREP COMMAND - UNIX / LINUX
Q) How to make the grep command case in-sensitive and search for a pattern in a file?
Let see how to do this with an example. Consider the below "Car insurance" data file:
In the above file, you can see the name "tom" appears in different cases (upper case, lower case and mixed case).
If I want to display the lines that contain the pattern tom with ordinary grep command, it will display only the third line. The grep command is shown below:
To make this grep command case insensitive use the -i option to the command. Now it will display the first, third and fifth lines from the file. The case in sensitive grep command is
Let see how to do this with an example. Consider the below "Car insurance" data file:
> cat car_insurance.dat Tom bought a new car and confused about car insurance quotes. He is worrying which car insurance policy he should take. So tom went to an insurance company and asks for clear explanation of the policies. The insurance guy then explains about various policies in detail. Now TOM gets an idea and chooses the right insurance for him.
In the above file, you can see the name "tom" appears in different cases (upper case, lower case and mixed case).
If I want to display the lines that contain the pattern tom with ordinary grep command, it will display only the third line. The grep command is shown below:
grep tom car_insurance.dat
To make this grep command case insensitive use the -i option to the command. Now it will display the first, third and fifth lines from the file. The case in sensitive grep command is
grep -i tom car_insurance.dat
HOW TO SEND MAIL FROM SHELL SCRIPT
We
write automated scripts to perform scheduled tasks and put them in
crontab. These automated scripts run at their scheduled times. However
we don’t know whether the scripts are succeeded or not. So sending an
email from automated bash scripts in unix host helps us to know whether
the script is succeeded or not.
Here we will see simple bash script to send emails using the mail command in linux operating system.
From the name of the variables you can easily understand the significance of each. In the mail command -s represents the subject. Here for the address by default the logged in unix / linux hostname is used as the sent address. For example if you have logged into unix host which is "example.com" and specified the from address as "test". Then your complete from address will be "test@example.com".
In the above bash script we specified the body from a file and did not specified any attachments. We will enhance the above script to attach files, to read body from a file and specifying a list of users in CC. The enhanced mail script is shown below:
The uuencode is used to attach files using the mail command. Here -c option in mail command is used to specify the list of users in cc list.
Here we will see simple bash script to send emails using the mail command in linux operating system.
#!/bin/bash
TO_ADDRESS="recipient@domain.com"
FROM_ADDRESS="sender"
SUBJECT="Mail Server Hosting Demo"
BODY="This is a linux mail system. Linux is one of the email operating systems which can be used to send and receive emails."
echo ${BODY}| mail -s ${SUBJECT} ${TO_ADDRESS} -- -r ${FROM_ADDRESS}
From the name of the variables you can easily understand the significance of each. In the mail command -s represents the subject. Here for the address by default the logged in unix / linux hostname is used as the sent address. For example if you have logged into unix host which is "example.com" and specified the from address as "test". Then your complete from address will be "test@example.com".
In the above bash script we specified the body from a file and did not specified any attachments. We will enhance the above script to attach files, to read body from a file and specifying a list of users in CC. The enhanced mail script is shown below:
#!/bin/bash
TO_ADDRESS="recipient@domain.com"
FROM_ADDRESS="sender"
SUBJECT="linux mail send attachment example"
BODY_FILE="script.dat"
ATTACHMENT_FILE="logfile.txt"
CC_LIST="user1@gmail.com;user2@yahoomail.com;user3@earthlink.com;user4@cheetahmail.com"
uuencode ${ATTACHMENT_FILE} | mail -s ${SUBJECT} -c ${CC_LIST} ${TO_ADDRESS} -- -r ${FROM_ADDRESS} < ${BODY_FILE}
The uuencode is used to attach files using the mail command. Here -c option in mail command is used to specify the list of users in cc list.
CONNECT TO ORACLE DATABASE IN UNIX SHELL SCRIPT
Q) How to connect to oracle database and run sql queries using a unix shell script?
The first thing you have to do to connect to oracle database in unix machine is to install oracle database drivers on the unix box. Once you installed, test whether you are able to connect to the database from command prompt or not. If you are able to connect to the database, then everything is going fine.
Here i am not going to discuss about how to install oracle database drivers. I am just providing the shell script which can be used to connect to the database and run sql statements.
The following Shell script connects to the scott schema of the oracle database and writes the database to the "query.log" file.
If the sql statements are failed to run, then the errors are written to the same "query.log" file. A better solution is to write the sql statements output to one file and the errors to another file. The below script uses the spooling concept in oracle to write to data to another file:
Here the output of the select statement is written to the "query_output.dat" file.
The first thing you have to do to connect to oracle database in unix machine is to install oracle database drivers on the unix box. Once you installed, test whether you are able to connect to the database from command prompt or not. If you are able to connect to the database, then everything is going fine.
Here i am not going to discuss about how to install oracle database drivers. I am just providing the shell script which can be used to connect to the database and run sql statements.
The following Shell script connects to the scott schema of the oracle database and writes the database to the "query.log" file.
#!/bin/bash
LogDirectory='/var/tmp/logs'
DataDirectory='/var/tmp/data'
DBUSER='scott'
DBUSERPASSWORD='tiger'
DB='oracle'
sqlplus -s <<EOF > ${LogDirectory}/query.log
${DBUSER}/${DBUSERPASSWORD}@${MYDB}
set linesize 32767
set feedback off
set heading off
select * from dual;
EOF
If the sql statements are failed to run, then the errors are written to the same "query.log" file. A better solution is to write the sql statements output to one file and the errors to another file. The below script uses the spooling concept in oracle to write to data to another file:
#!/bin/bash
LogDirectory='/var/tmp/logs'
DataDirectory='/var/tmp/data'
DBUSER='scott'
DBUSERPASSWORD='tiger'
DB='oracle'
sqlplus -s <<EOF > ${LogDirectory}/query.log
${DBUSER}/${DBUSERPASSWORD}@${MYDB}
set linesize 32767
set feedback off
set heading off
spool ${DataDirectory}/query_output.dat
SELECT * from dual
spool off
EOF
Here the output of the select statement is written to the "query_output.dat" file.
DELETE ALL LINES IN VI / VIM EDITOR - UNIX / LINUX
Q) How to delete all the lines in a file when opened in a VI editor or VIM editor?
Those who are new to unix will use the dd to delete each and every line to empty the file. There is an easy way to delete all the lines in a file when opened in an Editor.
Follow the below steps to empty a file:
See how simple it is to remove all the lines in a file. We will see how to empty the file when not opened in an editor. In unix /dev/null is any empty stream, you can use that to empty a file. The following commands shows how to empty a file
Those who are new to unix will use the dd to delete each and every line to empty the file. There is an easy way to delete all the lines in a file when opened in an Editor.
Follow the below steps to empty a file:
- Go to command mode in the editor by pressing ESC key on the keyboard.
- Press gg. It will take to the first line of the file.
- Then press dG. This will delete from the first line to the last line.
See how simple it is to remove all the lines in a file. We will see how to empty the file when not opened in an editor. In unix /dev/null is any empty stream, you can use that to empty a file. The following commands shows how to empty a file
cat /dev/null > file
HOW TO READ LINES USING LOOPS IN SHELL SCRIPTING
Q) How to read each line from a file using loops in bash scripting?
Reading lines from files and then processing on each line is a basic operation in shell scripting. We will see here how to read each line from a file using for and while loop in bash scripting.
Read Line using While Loop:
The below bash script reads line from the file, "logfile.txt", using while loop and prints the line on the terminal:
Here the variable i is just used to represent the line number.
Read Line using For Loop:
The following shell script reads line using for loop from the file errors.txt:
Reading lines from files and then processing on each line is a basic operation in shell scripting. We will see here how to read each line from a file using for and while loop in bash scripting.
Read Line using While Loop:
The below bash script reads line from the file, "logfile.txt", using while loop and prints the line on the terminal:
#!/bin/bash i=1 while read LINE do echo $i $LINE i=`expr $i+ 1` done < logfile.txt
Here the variable i is just used to represent the line number.
Read Line using For Loop:
The following shell script reads line using for loop from the file errors.txt:
#!/usr/bin/bash n=1 for y in `cat errors.txt` do echo $n $y n=`expr $n+ 1` done
EXAMPLES OF ARRAYS IN AWK COMMAND - UNIX / LINUX
Awk
command in unix has rich set of features. One of the feature is it can
store the elements in arrays and can process the data in the elements.
Here we will see how to use arrays in awk command with examples.
Examples of Arrays in Awk Command:
1. Finding the sum of values
I want to find the sum of values in the first column of all the lines and display it on the unix or linux terminal. Let say my file has the below data:
After summing up all the values, the output should be 60. The awk command to sum the values without using the arrays is shown below:
Here i have used a variable to store the sum of values. At the end after summing up all the values, the sum is printed on the terminal.
The awk command to find the sum of values by using arrays is shown below:
Here an array is used to store the sum of values. Basically this array will store the cumulative sum of values, at the end it contains the total and it is displayed on the terminal.
2. Ranking values in a file.
Let say I have a source file which contains the employees data. This file has three fields first field is department_id, second one is employee name and third one is salary. Sample data from the file is shown below:
Now i want to assign ranks to the employees in each department based on their salary. The output should look as
Here Employees AAA and CCC got same rank as their salaries are same.
To solve this problem first we have to sort the data and then pipe it to awk command. The complete command is shown below:
For readability purpose the above command is written in multiple lines. You have to write the above command in single line to make it work in unix.
Examples of Arrays in Awk Command:
1. Finding the sum of values
I want to find the sum of values in the first column of all the lines and display it on the unix or linux terminal. Let say my file has the below data:
> cat file.txt 10 20 30
After summing up all the values, the output should be 60. The awk command to sum the values without using the arrays is shown below:
awk 'BEGIN {sum=0} {sum=sum+$1} END {print sum}' text
Here i have used a variable to store the sum of values. At the end after summing up all the values, the sum is printed on the terminal.
The awk command to find the sum of values by using arrays is shown below:
awk '{arr[NR]=arr[NR-1]+$1} END {print arr[NR]}' text
Here an array is used to store the sum of values. Basically this array will store the cumulative sum of values, at the end it contains the total and it is displayed on the terminal.
2. Ranking values in a file.
Let say I have a source file which contains the employees data. This file has three fields first field is department_id, second one is employee name and third one is salary. Sample data from the file is shown below:
> cat emp.dat 10, AAA, 6000 10, BBB, 8000 10, CCC, 6000 20, DDD, 4000 20, EEE, 2000 20, FFF, 7000
Now i want to assign ranks to the employees in each department based on their salary. The output should look as
20, FFF, 7000, 1 20, DDD, 4000, 2 20, EEE, 2000, 3 10, BBB, 8000, 1 10, AAA, 6000, 2 10, CCC, 6000, 2
Here Employees AAA and CCC got same rank as their salaries are same.
To solve this problem first we have to sort the data and then pipe it to awk command. The complete command is shown below:
sort -nr -k1 -k3 text |
awk -F"," '{
department_array[NR]=$1;
salary_array[NR]=$3;
if (department_array[NR] != department_array[NR-1])
{
rank_array[NR]=1;
}
else if (salary_array[NR] == salary_array[NR-1] )
{
rank_array[NR] =rank_array[NR-1];
}
else
{
rank_array[NR] = rank_array[NR-1]+1;
}
print department_array[NR]","$2","salary_array[NR]","rank_array[NR];
}'
For readability purpose the above command is written in multiple lines. You have to write the above command in single line to make it work in unix.
CREATE TUNNEL IN UNIX USING PUTTY
Q) How to create tunnel to access network resources (Internet) in a remote unix machine using putty client?
You might have faced situations where you want to open a website from your browser and the website URL is blocked by your company. Especially this happens in software companies. Here i will show you how to open a blocked website by creating a tunnel.
The software’s required to create a tunnel are:
Creating Tunnle in Unix
Follow the below steps to create a tunnel
Now you can open any website with this approach provided your remote host has access.
You might have faced situations where you want to open a website from your browser and the website URL is blocked by your company. Especially this happens in software companies. Here i will show you how to open a blocked website by creating a tunnel.
The software’s required to create a tunnel are:
- Putty client tool
- Mozilla firefox browser
- Access to remote unix server. This server should be capable of opening any website.
Creating Tunnle in Unix
Follow the below steps to create a tunnel
- Open the putty client tool. Enter the remote unix hostname in the "Host Name (or IP address)". In this demo i have entered the hostname as "example.hostname.com".
- To save this hostname, enter a name like "Tunnel" in the "Saved Sessions" place. This is shown in the below image:
- On the left side of the client, you can see a navigation panel. Go to SSH-> Tunnels.
- Again enter the remote hostname (example.hostname.com) in "Destination" section.
- Enter the source port as 1100 (any value you prefer) and check the Dynamic option. This is shown below:
- Now click on Add. Go back to the previous window by clicking on the Session in the left side pan. Here clik on save. I will save your tunnel details.
- Open this tunnel and enter your remote machine login details. Do not close this unix session. If you close it, your tunneling won’t work.
- Open the Mozilla fire fox browser. Go to Tools->Options->Advanced->Network->Settings.
- In the settings, Check the manual proxy configurations, enter the Socket host as localhost and port as 1100 (Same port which is specified in tunnel configuration) and click on Ok. This is shown in below image.
Now you can open any website with this approach provided your remote host has access.
VIM EDITOR - SAVE AND QUIT OPTIONS
VIM
is a powerful editor in unix or linux. The VIM editor got so many
features. Here we will see the options for saving and quitting from the
vim editor.
The following options works in command mode of VIM editor. To go to the command mode press ESC key on the keyboard and then type the below commands:
The following options works in command mode of VIM editor. To go to the command mode press ESC key on the keyboard and then type the below commands:
- :w ->Saves the contents of the file without exiting from the VIM editor
- :wq ->Saves the text in the file and then exits from the editor
- :w filename -> Saves the contents of the opened file in the specified filename. However it won’t save the contents of the current file.
- :x -> Saves changes to the current file and then exits. Similar to the :wq
- :m,nw filename -> Here m and n are numbers. This option will write the lines from the specified numbers m and n to the mentioned filename.
- :q -> Exits from the current file only if you did not do any changes to the file.
- :q! -> Exits from the current file and ignores any changes that you made to the file.
DIRNAME COMMAND EXAMPLES IN UNIX / LINUX
The unix dirname command strips non-directory suffix from a file name.
The syntax of dirname command is
The dirname command removes the trailing / component from the NAME and prints the remaining portion. If the NAME does not contain / component then it prints '.' (means current directory).
Dirname command is useful when dealing with directory paths in unix or linux operating systems. Some examples on dirname command are shown below:
Dirname Command Examples:
1. Remove the file name from absolute path.
Let say my directory path is /usr/local/bin/add.sh. Now i want to remove /add.sh and display only /usr/local/bin, then we can use the dirname command.
2. dirname sum.pl
Here you can see that the NAME does not contain the / component. In this case the dirname produces '.' as the output.
Note: The directories and filename which i have passed as arguments to dirname command in the above examples are just strings. There is no need of these directories or files to exist in the unix machine.
The syntax of dirname command is
dirname NAME
The dirname command removes the trailing / component from the NAME and prints the remaining portion. If the NAME does not contain / component then it prints '.' (means current directory).
Dirname command is useful when dealing with directory paths in unix or linux operating systems. Some examples on dirname command are shown below:
Dirname Command Examples:
1. Remove the file name from absolute path.
Let say my directory path is /usr/local/bin/add.sh. Now i want to remove /add.sh and display only /usr/local/bin, then we can use the dirname command.
> dirname /usr/local/bin/add.sh /usr/local/bin
2. dirname sum.pl
Here you can see that the NAME does not contain the / component. In this case the dirname produces '.' as the output.
> dirname sum.pl .
Note: The directories and filename which i have passed as arguments to dirname command in the above examples are just strings. There is no need of these directories or files to exist in the unix machine.
SPLIT COMMAND EXAMPLES IN UNIX / LINUX
The
Split command in unix or linux operating system splits a file into many
pieces (multiple files). We can split a file based on the number of
lines or bytes. We will see how to use the split command with an
example.
As an example, let’s take the below text file as the source file which we want to split:
There are three lines in that file and the size of the file is 47 bytes.
Split Command Examples:
1. Splitting file on number of lines.
The Split command has an option -l to split the file based on the number of lines. Let say i want to split the text file with number of lines in each file as 2. The split command for this is
The new files created are xaa and xab. Always the newly created (partitioned) file names start with x. We will see the contents of these files by doing a cat operation.
As there only three lines in the source file we got only one line in the last created file.
2. Splitting file on the number of bytes
We can use the -b option to specify the number of bytes that each partitioned file should contains. As an example we will split the source files on 10 bytes as
The files created are xaa, xab, xac, xad, xae. The first four files contain 10 bytes and the last file contains 7 bytes as the source file size is 47 bytes.
3. Changing the newly created file names from character sequences to numeric sequences.
So far we have seen that the newly created file names are created in character sequences like xaa, Xab and so on. We can change this to numeric sequence by using the -d option as
The names of the new files created are x00 and x01.
4. Changing the number of digits in the sequence of filenames.
In the above example, you can observe that the sequences have two digits (00 and 01) in the file names. You can change the number of digits in the sequence by using the -a option as
Now the files created are x000 and x001
As an example, let’s take the below text file as the source file which we want to split:
> cat textfile unix linux os windows mac os linux environment
There are three lines in that file and the size of the file is 47 bytes.
Split Command Examples:
1. Splitting file on number of lines.
The Split command has an option -l to split the file based on the number of lines. Let say i want to split the text file with number of lines in each file as 2. The split command for this is
split -l2 textfile
The new files created are xaa and xab. Always the newly created (partitioned) file names start with x. We will see the contents of these files by doing a cat operation.
> cat xaa unix linux os windows mac os > cat xab linux environment
As there only three lines in the source file we got only one line in the last created file.
2. Splitting file on the number of bytes
We can use the -b option to specify the number of bytes that each partitioned file should contains. As an example we will split the source files on 10 bytes as
split -b10 textfile
The files created are xaa, xab, xac, xad, xae. The first four files contain 10 bytes and the last file contains 7 bytes as the source file size is 47 bytes.
3. Changing the newly created file names from character sequences to numeric sequences.
So far we have seen that the newly created file names are created in character sequences like xaa, Xab and so on. We can change this to numeric sequence by using the -d option as
split -l2 -d textfile
The names of the new files created are x00 and x01.
4. Changing the number of digits in the sequence of filenames.
In the above example, you can observe that the sequences have two digits (00 and 01) in the file names. You can change the number of digits in the sequence by using the -a option as
split -l2 -d -a3 textfile
Now the files created are x000 and x001
SWAP FIELDS (STRINGS) IN A FILE - UNIX / LINUX
First we will see how to swap two strings in a line and then we will see how to swap two columns in a file.
As an example, consider the text file with below data:
Swap Strings using Sed command:
Let see how to swap the words unix and linux using sed command in unix or linux environment. The sed command to swap the strings is shown below:
The parentheses are used to remember the pattern. \1 indicates first pattern and \2 indicates second pattern.
Swap Fields using Awk command:
From the above file structure, we can observe that the file is in format of rows and columns where the columns are delimited by space.
Awk command can be used to process delimited files. Awk command to swap the first two fields in a file is
Another way using awk is
As an example, consider the text file with below data:
unix linux os windows mac os
Swap Strings using Sed command:
Let see how to swap the words unix and linux using sed command in unix or linux environment. The sed command to swap the strings is shown below:
> sed 's/\(unix\) \(linux\)/\2 \1/' textfile linux unix os windows mac os
The parentheses are used to remember the pattern. \1 indicates first pattern and \2 indicates second pattern.
Swap Fields using Awk command:
From the above file structure, we can observe that the file is in format of rows and columns where the columns are delimited by space.
Awk command can be used to process delimited files. Awk command to swap the first two fields in a file is
> awk '{$0=$2." "$1" "$3; print $0}' textfile
linux unix os
mac windows os
Another way using awk is
awk '{print $2" "$1" "$3}' textfile
HOW TO REPLACE BRACES SYMBOLS IN UNIX / LINUX
Q)
My log file contains the braces symbols '(' and ')'. I would like to
replace the braces with empty string. Sample data in the log file is
shown below:
The output should not contain the braces and the data should look as
How can i achieve this using unix or linux commands?
Solution:
1. Replacing using tr command
We can use the tr command to delete characters in a file. The deleting of strings using tr command is shown below:
2. Replacing using sed command Sed command is popularly used for replacing the text in a file with another text. The sed command is
Another way of replacing is using sed with pipes in unix:
> cat logfile Error - (unix script failed)
The output should not contain the braces and the data should look as
Error - unix script failed
How can i achieve this using unix or linux commands?
Solution:
1. Replacing using tr command
We can use the tr command to delete characters in a file. The deleting of strings using tr command is shown below:
tr -d '()' < logfile
2. Replacing using sed command Sed command is popularly used for replacing the text in a file with another text. The sed command is
sed 's/[()]//g' logfile
Another way of replacing is using sed with pipes in unix:
sed 's/(//' logfile| sed 's/)//'
SPLIT FILE DATA INTO MULTIPLE FILES - UNIX / LINUX
Q)
I have file with 10000 lines in unix or linux operating system. I want
to split this file and create 10 files such that each file has 1000
lines. What I mean is the first 100 lines should go into one file; next
100 lines should go into another file and so on. How to do this using
unix commands.
Solution:
Unix has the split command which can be used to partition the data in a file into multiple files. The command to split a file based on the number of lines is shown below:
The above split command splits the file such that each file has 1000 lines. Here the option l indicates the number of lines. You can split the file based on number of bytes using the -b option.
By default, the partitioned filenames starts with x like xab, xac, xad and so on. Instead of alphabetical sequences, you can use numeric sequences in filenames like x01, x02 using the -d option.
You can specify the number of digits to be used in the numeric sequences with the help of -a option.
Examples: Let say i have a text file with 4 lines. The data in the file is shown below:
We will run the split command for each of the points discussed above and see what files will be created.
Solution:
Unix has the split command which can be used to partition the data in a file into multiple files. The command to split a file based on the number of lines is shown below:
split -l 1000 filename
The above split command splits the file such that each file has 1000 lines. Here the option l indicates the number of lines. You can split the file based on number of bytes using the -b option.
split -b 1024 filename
By default, the partitioned filenames starts with x like xab, xac, xad and so on. Instead of alphabetical sequences, you can use numeric sequences in filenames like x01, x02 using the -d option.
split -l 1000 -d filename
You can specify the number of digits to be used in the numeric sequences with the help of -a option.
split -l 1000 -d -a 3 filename
Examples: Let say i have a text file with 4 lines. The data in the file is shown below:
> cat textfile unix is os linux environment centos red hat linux
We will run the split command for each of the points discussed above and see what files will be created.
> split -l 2 textfile Files: xaa, xab > split -b 10 textfile Files: xaa, xab, xac, xad, xae > split -l 2 -d textfile Files: x00, x01 > split -l 2 -d -a 3 textfile Files: x000, x001
REMOVE LAST CHARACTER IN STRING - UNIX / LINUX
Q)
I have a file with bunch of lines. I want to remove the last character
in each line from that file. How can i achieve this in unix or linux
environment.
Solution:
1. SED command to remove last character
You can use the sed command to delete the last character from a text. The sed command is
2. Bash script
The below bash script can be used to remove the last character in a file.
3. Using Awk command We can use the built-in functions length and substr of awk command to delete the last character in a text.
4. Using rev and cut command We can use the combination of reverse and cut command to remove the last character. The command is shown below:
Solution:
1. SED command to remove last character
You can use the sed command to delete the last character from a text. The sed command is
sed s/.$// filename
2. Bash script
The below bash script can be used to remove the last character in a file.
#! /bin/bash
while read LINE
do
echo ${LINE%?}
done < filename
3. Using Awk command We can use the built-in functions length and substr of awk command to delete the last character in a text.
awk '{$0=substr($0,1,length($0)-1); print $0}' filename
4. Using rev and cut command We can use the combination of reverse and cut command to remove the last character. The command is shown below:
rev products.txt | cut -c2- |rev
CONVERT MULTIPLE ROWS INTO SINGLE ROW - UNIX/LINUX
Q) I have a products data in the text file. The data in the file look as shown below:
Now my requirement is to group each 3 consecutive rows into a single row and produce a comma separated list of products. The output should look as
I want this to be implemented using unix or linux commands in different ways?
Solution:
1. One way we can implement this is using the awk command. The complete awk command is shown below:
2. Another way is using the paste command. The solution using the paste command is shown below.
> cat product.txt iphone samsung nokia yahoo google aol amazon ebay walmart
Now my requirement is to group each 3 consecutive rows into a single row and produce a comma separated list of products. The output should look as
iphone,samsung,nokia yahoo,google,aol amazon,ebay,walmart
I want this to be implemented using unix or linux commands in different ways?
Solution:
1. One way we can implement this is using the awk command. The complete awk command is shown below:
awk '{printf("%s%s",$0,NR%3?",":"\n")}' products.txt
2. Another way is using the paste command. The solution using the paste command is shown below.
paste -d, - - - < products.txt
AWK COMMAND TO SPLIT COLUMN INTO ROW - UNIX/LINUX
Awk command to split list data in a column into multiple rows - Unix/Linux
Q) I have a flat file in the unix or linux environment. The data in the flat file looks as below
The flat file contains the list of subjects that were taken by the students in their curriculum. I want the subjects list in each column to be splitted into multiple rows. After splitting the data in the target should look as:
Write a command in UNIX or LINUx operating system to produce the result?
Solution:
We can use the AWK command which can process the files with table like structures. The solution to the problem using Awk command is
Q) I have a flat file in the unix or linux environment. The data in the flat file looks as below
> cat students.txt Mark Maths,Physics,Chemistry Chris Social Henry Biology, Science
The flat file contains the list of subjects that were taken by the students in their curriculum. I want the subjects list in each column to be splitted into multiple rows. After splitting the data in the target should look as:
Mark Maths Mark Physics Mark Chemistry Chris Social Henry Science Henry Science
Write a command in UNIX or LINUx operating system to produce the result?
Solution:
We can use the AWK command which can process the files with table like structures. The solution to the problem using Awk command is
awk '{n=split($2,s,",");for (i=1;i<=n;i++) {$2=s[i];print}}' students.txt
SEARCH AND GREP FOR TEXT IN UNIX/LINUX
We
will see how to search for files and then grep for a string of text in
those files. First i will use the find command in unix or linux to
search for the regular files in the current directory. The grep command
to search for the normal files in the current directory is shown below:
Now we will grep for a particular word in these files and display only the filenames that has the matching word. The unix command is shown below:
The above command just displays the filenames that has the specified word. Now we will try to display the lines from the files that have the matching word. The unix command for this is:
If you want to put space between the results of the above command, display the line using echo. The complete unix command is
The above example shows how to use multiple grep’s with the find command in unix or linux.
> find . -type f ./docs/sum.pl ./add.sh ./sample
Now we will grep for a particular word in these files and display only the filenames that has the matching word. The unix command is shown below:
> find . -type f -exec grep -l word {} \;
The above command just displays the filenames that has the specified word. Now we will try to display the lines from the files that have the matching word. The unix command for this is:
> find . -type f -exec grep -l word {} \; -exec grep word {} \;
If you want to put space between the results of the above command, display the line using echo. The complete unix command is
> find . -type f -exec grep -l word {} \; -exec grep word {} \; -exec echo \;
The above example shows how to use multiple grep’s with the find command in unix or linux.
JOIN COMMAND IN UNIX/LINUX EXAMPLES
Join
command is one of the text processing utility in Unix/Linux. Join
command is used to combine two files based on a matching fields in the
files. If you know SQL, the join command is similar to joining two
tables in a database.
The syntax of join command is
The join command options are
Unix Join Command Examples
1. Write a join command to join two files on the first field?
The basic usage of join command is to join two files on the first field. By default the join command matches the files on the first fields when we do not specify the field numbers explicitly. Let's say we have two files emp.txt and dept.txt
Here we will join on the first field and see the output. By default, the join command treats the field delimiter as space or tab.
Important Note: Before joining the files, make sure to sort the fields on the joining fields. Otherwise you will get incorrect result.
2. Write a join command to join the two files? Here use the second field from the first file and the first field from the second file to join.
In this example, we will see how to join two files on different fields rather than the first field. For this consider the below two files as an example
From the above, you can see the join fields are the second field from the emp.txt and the first field from the dept.txt. The join command to match these two files is
You can also see that the two files can also be joined on the third filed. As the both the files have the matching join field, you can use the j option in the join command.
Here -1 2 specifies the second field from the first file (emp.txt) and -2 1 specifies the first field from the second file (dept.txt)
3. Write a join command to select the required fields from the input files in the output? Select first filed from first file and second field from second file in the output.
By default, the join command prints all the fields from both the files (except the join field is printed once). We can choose what fields to be printed on the terminal with the -o option. We will use the same files from the above example.
Here 1.1 means in the first file select the first field. Similarly, 2.2 means in the second file select the second field
4. Write a command to join two delimited files? Here the delimiter is colon (:)
So far we have joined files with space delimiter. Here we will see how to join files with a colon as delimiter. Consider the below two files.
The -t option is used to specify the delimiter. The join command for joining the files is
5. Write a command to ignore case when joining the files?
If the join fields are in different cases, then the join will not be performed properly. To ignore the case in join use the -i option.
6. Write a join command to print the lines which do not match the values in joining fields?
By default the join command prints only the matched lines from both the files which means prints the matched lines that passed the join condition. We can use the -a option to print the non-matched lines.
The syntax of join command is
join [options] file1 file2
The join command options are
-1 field number : Join on the specified field number in the first file -2 field number : Join on the specified field number in the second file -j field number : Equivalent to -1 fieldnumber and -2 fieldnumber -o list : displays only the specified fields from both the files -t char : input and output field delimiter -a filenumber : Prints non matched lines in a file -i : ignore case while joining
Unix Join Command Examples
1. Write a join command to join two files on the first field?
The basic usage of join command is to join two files on the first field. By default the join command matches the files on the first fields when we do not specify the field numbers explicitly. Let's say we have two files emp.txt and dept.txt
> cat emp.txt 10 mark 10 steve 20 scott 30 chris > cat dept.txt 10 hr 20 finance 30 db
Here we will join on the first field and see the output. By default, the join command treats the field delimiter as space or tab.
> join emp.txt dept.txt 10 mark hr 10 steve hr 20 scott finance 30 chris db
Important Note: Before joining the files, make sure to sort the fields on the joining fields. Otherwise you will get incorrect result.
2. Write a join command to join the two files? Here use the second field from the first file and the first field from the second file to join.
In this example, we will see how to join two files on different fields rather than the first field. For this consider the below two files as an example
> cat emp.txt mark 10 1 steve 10 1 scott 20 2 chris 30 3 > cat dept.txt 10 hr 1 20 finance 2 30 db 3
From the above, you can see the join fields are the second field from the emp.txt and the first field from the dept.txt. The join command to match these two files is
> join -1 2 -2 1 emp.txt dept.txt 10 mark 1 hr 1 10 steve 1 hr 1 20 scott 2 finance 2 30 chris 3 db 3
You can also see that the two files can also be joined on the third filed. As the both the files have the matching join field, you can use the j option in the join command.
Here -1 2 specifies the second field from the first file (emp.txt) and -2 1 specifies the first field from the second file (dept.txt)
> join -j 3 emp.txt dept.txt 1 mark 10 10 hr 1 steve 10 10 hr 2 scott 20 20 finance 3 chris 30 30 db
3. Write a join command to select the required fields from the input files in the output? Select first filed from first file and second field from second file in the output.
By default, the join command prints all the fields from both the files (except the join field is printed once). We can choose what fields to be printed on the terminal with the -o option. We will use the same files from the above example.
> join -o 1.1 2.2 -1 2 -2 1 emp.txt dept.txt mark hr steve hr scott finance chris db
Here 1.1 means in the first file select the first field. Similarly, 2.2 means in the second file select the second field
4. Write a command to join two delimited files? Here the delimiter is colon (:)
So far we have joined files with space delimiter. Here we will see how to join files with a colon as delimiter. Consider the below two files.
> cat emp.txt mark:10 steve:10 scott:20 chris:30 > cat dept.txt 10:hr 20:finance 30:db
The -t option is used to specify the delimiter. The join command for joining the files is
> join -t: -1 2 -2 1 emp.txt dept.txt 10:mark:hr 10:steve:hr 20:scott:finance 30:chris:db
5. Write a command to ignore case when joining the files?
If the join fields are in different cases, then the join will not be performed properly. To ignore the case in join use the -i option.
> cat emp.txt mark,A steve,a scott,b chris,C > cat dept.txt a,hr B,finance c,db > join -t, -i -1 2 -2 1 emp.txt dept.txt A,mark,hr a,steve,hr b,scott,finance C,chris,db
6. Write a join command to print the lines which do not match the values in joining fields?
By default the join command prints only the matched lines from both the files which means prints the matched lines that passed the join condition. We can use the -a option to print the non-matched lines.
> cat P.txt A 1 B 2 C 3 > cat Q.txt B 2 C 3 D 4 Print non pairable lines from first file. > join -a 1 P.txt Q.txt A 1 B 2 2 C 3 3 Print non pairable lines from second file. > join -a 2 P.txt Q.txt B 2 2 C 3 3 D 4 Print non pairable lines from both file. > join -a 1 -a 2 P.txt Q.txt A 1 B 2 2 C 3 3 D 4
MOVE / RENAME FILES, DIRECTORY - MV COMMAND IN UNIX / LINUX
Q.
How to rename a file or directory in unix (or linux) and how to move a
file or directory from the current directory to another directory?
Unix provides a simple mv (move) command which can be used to rename or move files and directories. The syntax of mv command is
The options of mv command are
If the newname already exists, then the mv command overwrites that file. Let see some examples on how to use mv command.
Unix mv command examples
1. Write a unix/linux command to rename a file?
Renaming a file is one of the basic features of the mv command. To rename a file from "log.dat" to "bad.dat", use the below mv command
Note that if the "bad.dat" file already exists, then its contents will be overwritten by "log.dat". To avoid this use the -i option, which prompts you before overwriting the file.
2. Write a unix/linux command to rename a directory?
Just as renaming a file, you can use the mv command to rename a directory. To rename the directory from docs to documents, run the below command
If the documents directory already exists, then the docs directory will be moved in to the documents directory.
3. Write a unix/linux command to move a file into another directory?
The mv command can also be used to move the file from one directory to another directory. The below command moves the sum.pl file in the current directory to /var/tmp directory.
If the sum.pl file already exists in the /var/tmp directory, then the contents of that file will be overwritten.
4. Write a unix/linux command to move a directory in to another directory?
Just as moving a file, you can move a directory into another directory. The below mv command moves the documents directory into the tmp directory
5. Write a unix/linux command to move all the files in the current directory to another directory?
You can use the regular expression pattern * to move all the files from one directory to another directory.
The above command moves all the files and directories in the current directory to the /var/tmp/ directory.
6. mv *
What happens if you simply type mv * and then press enter?
It depends on the files you have in the directory. The * expands to all the files and directories. Three scenarios are possible.
Some Tips:
Unix provides a simple mv (move) command which can be used to rename or move files and directories. The syntax of mv command is
mv [options] oldname newname
The options of mv command are
f : Do not prompt before overwriting a file. i : Prompts for the user input before overwriting a file.
If the newname already exists, then the mv command overwrites that file. Let see some examples on how to use mv command.
Unix mv command examples
1. Write a unix/linux command to rename a file?
Renaming a file is one of the basic features of the mv command. To rename a file from "log.dat" to "bad.dat", use the below mv command
> mv log.dat bad.dat
Note that if the "bad.dat" file already exists, then its contents will be overwritten by "log.dat". To avoid this use the -i option, which prompts you before overwriting the file.
mv -i log.dat bad.dat mv: overwrite `bad.dat'?
2. Write a unix/linux command to rename a directory?
Just as renaming a file, you can use the mv command to rename a directory. To rename the directory from docs to documents, run the below command
mv docs/ documents/
If the documents directory already exists, then the docs directory will be moved in to the documents directory.
3. Write a unix/linux command to move a file into another directory?
The mv command can also be used to move the file from one directory to another directory. The below command moves the sum.pl file in the current directory to /var/tmp directory.
mv sum.pl /var/tmp/
If the sum.pl file already exists in the /var/tmp directory, then the contents of that file will be overwritten.
4. Write a unix/linux command to move a directory in to another directory?
Just as moving a file, you can move a directory into another directory. The below mv command moves the documents directory into the tmp directory
mv documents /tmp/
5. Write a unix/linux command to move all the files in the current directory to another directory?
You can use the regular expression pattern * to move all the files from one directory to another directory.
mv * /var/tmp/
The above command moves all the files and directories in the current directory to the /var/tmp/ directory.
6. mv *
What happens if you simply type mv * and then press enter?
It depends on the files you have in the directory. The * expands to all the files and directories. Three scenarios are possible.
- If the current directory has only files, then the contents of all the files (except one file) will be written in to the one file. The one file is the last file which depends on the pattern *.
- If the current directory contains only directories, then all the directories (except one directory) will be moved to another directory.
- If the current directory contains both files and directories, then it depends on the expansion of the *. If the pattern * gives the last one as directory then all the files will be moved to that directory. Otherwise the mv command will fail.
Some Tips:
- Try to avoid mv *
- Avoid moving large number of files.
SED -I COMMAND EXAMPLES IN UNIX AND LINUX
Sed
is great tool for replacing the text in a file. sed is a stream editor
which means edit the file as a stream of characters. To replace a text
using the unix sed command, you have to pass the search string and
replacement string. By default the sed command does not edit the file
and displays the output on the terminal.
We will see the usage of -i command with an example
Consider the below text file with data
We will replace the word "tutorial" with "example" in the file using the sed command.
The sed command replaced the text in the file and displayed the result on the terminal. However it did not changed the contents of the file. You can redirect the output of sed command and save it in a file as
The -i option comes in handy to edit the original file itself. If you use the -i option the sed command replaces the text in the original file itself rather than displaying it on the terminal.
Be careful in using the -i option. Once you changed the contents of the file, you cannot revert back to the original file. It is good to take the backup of the original file. You can provide a suffix to the -i option for taking the backup of the file. Now we will replace the "example" with "tutorial" and at the same time will take the backup of the file.
See the backup file created with the contents of the original file.
We will see the usage of -i command with an example
Consider the below text file with data
> cat file.txt linux sed command tutorial
We will replace the word "tutorial" with "example" in the file using the sed command.
> sed 's/tutorial/example/' file.txt linux sed command example > cat file.txt linux sed command tutorial
The sed command replaced the text in the file and displayed the result on the terminal. However it did not changed the contents of the file. You can redirect the output of sed command and save it in a file as
> sed 's/tutorial/example/' file.txt > new_file.txt
The -i option comes in handy to edit the original file itself. If you use the -i option the sed command replaces the text in the original file itself rather than displaying it on the terminal.
> sed -i 's/tutorial/example/' file.txt > cat file.txt linux sed command example
Be careful in using the -i option. Once you changed the contents of the file, you cannot revert back to the original file. It is good to take the backup of the original file. You can provide a suffix to the -i option for taking the backup of the file. Now we will replace the "example" with "tutorial" and at the same time will take the backup of the file.
> sed -i_bkp 's/example/tutorial/' file.txt > ls file.txt* file.txt file.txt_bkp > cat file.txt_bkp linux sed command example > cat file.txt linux sed command tutorial
See the backup file created with the contents of the original file.
DELETE DIRECTORY, FILES - RM, RMDIR COMMAND IN UNIX / LINUX
Q. How to delete directories and files in unix/linux
Unix provides rmdir and rm commands to remove the directories and files. Let see each command in detail.
Unix rmdir command syntax
The syntax of rmdir command is
The rmdir command options are
Unix rmdir command examples
1. Write a unix/linux command to remove a directory?
The rmdir command deletes only the empty directories. If a directory contains files or sub directories, then the rmdir command fails.
Here the docs directory is not empty, that is why the rmdir command failed to remove the directory. To remove the docs directory first we have to make the directory empty and then delete the directory.
We will see later how to remove non-empty directories with a single command.
2. Write a unix/linux command to remove the directory and its parent directories?
As mentioned earlier the -p option allows the rmdir command to delete the directory and also its parent directories.
This rmdir command removes the docs directory completely. If you don’t use the -p option, then it only deletes the movies directory.
3. Write a unix/linux command to remove directories using pattern matching?
You can specify the directory names using the regular expressions and can delete them.
This rm command deletes the directories like doc, documents, doc_1 etc.
Now we will see the rm command in unix.
Unix rm command syntax
The syntax of rm command is
The rm command options are
The rm command can be used to delete both the files and directories. The rm command also deletes the non-empty directories.
Unix rm command examples
1. Write a unix/linux command to remove a file?
This is the basic feature of rm command. To remove a file, logfile.dat, in the current directory use the below rm command
2. Write a unix/linux command to remove all the files in a directory?
use the * regular pattern as the file list in rm command for deleting all the files in the current directory.
3. Write a unix/linux command to delete empty directory?
The rm command can also be used to delete the empty directory. The command for this is
If the directory is non-empty, then the above command fails to remove the directories.
4. Write a unix/linux command to delete directories recursively (delete non empty directories)?
As mentioned earlier, the -r option can be used to remove the directories and sub directories.
This removes the docs directory even if it is non-empty.
Unix provides rmdir and rm commands to remove the directories and files. Let see each command in detail.
Unix rmdir command syntax
The syntax of rmdir command is
rmdir [options] directories
The rmdir command options are
-p : Removes directory and its parent directories -v : Provides the diagnostic information of the directory processed
Unix rmdir command examples
1. Write a unix/linux command to remove a directory?
The rmdir command deletes only the empty directories. If a directory contains files or sub directories, then the rmdir command fails.
rmdir docs/ rmdir: docs/: Directory not empty
Here the docs directory is not empty, that is why the rmdir command failed to remove the directory. To remove the docs directory first we have to make the directory empty and then delete the directory.
rm doc/* rmdir docs/
We will see later how to remove non-empty directories with a single command.
2. Write a unix/linux command to remove the directory and its parent directories?
As mentioned earlier the -p option allows the rmdir command to delete the directory and also its parent directories.
rmdir -p docs/entertainment/movies/
This rmdir command removes the docs directory completely. If you don’t use the -p option, then it only deletes the movies directory.
3. Write a unix/linux command to remove directories using pattern matching?
You can specify the directory names using the regular expressions and can delete them.
rm doc*
This rm command deletes the directories like doc, documents, doc_1 etc.
Now we will see the rm command in unix.
Unix rm command syntax
The syntax of rm command is
rm [options] [directory|file]
The rm command options are
f : Removes all files in a directory without prompting the user. i : Interactive: prompts the user for confirmation before deleting a file. R or r : Recursively remove directories and sub directories.
The rm command can be used to delete both the files and directories. The rm command also deletes the non-empty directories.
Unix rm command examples
1. Write a unix/linux command to remove a file?
This is the basic feature of rm command. To remove a file, logfile.dat, in the current directory use the below rm command
rm logfile.dat
2. Write a unix/linux command to remove all the files in a directory?
use the * regular pattern as the file list in rm command for deleting all the files in the current directory.
rm *
3. Write a unix/linux command to delete empty directory?
The rm command can also be used to delete the empty directory. The command for this is
rm docs/
If the directory is non-empty, then the above command fails to remove the directories.
4. Write a unix/linux command to delete directories recursively (delete non empty directories)?
As mentioned earlier, the -r option can be used to remove the directories and sub directories.
rm -r docs
This removes the docs directory even if it is non-empty.
CUT COMMAND IN UNIX ( LINUX) EXAMPLES
Cut
command in unix (or linux) is used to select sections of text from each
line of files. You can use the cut command to select fields or columns
from a line by specifying a delimiter or you can select a portion of
text by specifying the range or characters. Basically the cut command
slices a line and extracts the text.
Unix Cut Command Example
We will see the usage of cut command by considering the below text file as an example
1. Write a unix/linux cut command to print characters by position?
The cut command can be used to print characters in a line by specifying the position of the characters. To print the characters in a line, use the -c option in cut command
The above cut command prints the fourth character in each line of the file. You can print more than one character at a time by specifying the character positions in a comma separated list as shown in the below example
This command prints the fourth and sixth character in each line.
2.Write a unix/linux cut command to print characters by range?
You can print a range of characters in a line by specifying the start and end position of the characters.
The above cut command prints the characters from fourth position to the seventh position in each line. To print the first six characters in a line, omit the start position and specify only the end position.
To print the characters from tenth position to the end, specify only the start position and omit the end position.
If you omit the start and end positions, then the cut command prints the entire line.
3.Write a unix/linux cut command to print the fields using the delimiter?
You can use the cut command just as awk command to extract the fields in a file using a delimiter. The -d option in cut command can be used to specify the delimiter and -f option is used to specify the field position.
This command prints the second field in each line by treating the space as delimiter. You can print more than one field by specifying the position of the fields in a comma delimited list.
The above command prints the second and third field in each line.
Note: If the delimiter you specified is not exists in the line, then the cut command prints the entire line. To suppress these lines use the -s option in cut command.
4. Write a unix/linux cut command to display range of fields?
You can print a range of fields by specifying the start and end position.
The above command prints the first, second and third fields. To print the first three fields, you can ignore the start position and specify only the end position.
To print the fields from second fields to last field, you can omit the last field position.
5. Write a unix/linux cut command to display the first field from /etc/passwd file?
The /etc/passwd is a delimited file and the delimiter is a colon (:). The cut command to display the first field in /etc/passwd file is
6. The input file contains the below text
Using the cut command extract the portion after the dot.
First reverse the text in each line and then apply the command on it.
Unix Cut Command Example
We will see the usage of cut command by considering the below text file as an example
> cat file.txt unix or linux os is unix good os is linux good os
1. Write a unix/linux cut command to print characters by position?
The cut command can be used to print characters in a line by specifying the position of the characters. To print the characters in a line, use the -c option in cut command
cut -c4 file.txt x u l
The above cut command prints the fourth character in each line of the file. You can print more than one character at a time by specifying the character positions in a comma separated list as shown in the below example
cut -c4,6 file.txt xo ui ln
This command prints the fourth and sixth character in each line.
2.Write a unix/linux cut command to print characters by range?
You can print a range of characters in a line by specifying the start and end position of the characters.
cut -c4-7 file.txt x or unix linu
The above cut command prints the characters from fourth position to the seventh position in each line. To print the first six characters in a line, omit the start position and specify only the end position.
cut -c-6 file.txt unix o is uni is lin
To print the characters from tenth position to the end, specify only the start position and omit the end position.
cut -c10- file.txt inux os ood os good os
If you omit the start and end positions, then the cut command prints the entire line.
cut -c- file.txt
3.Write a unix/linux cut command to print the fields using the delimiter?
You can use the cut command just as awk command to extract the fields in a file using a delimiter. The -d option in cut command can be used to specify the delimiter and -f option is used to specify the field position.
cut -d' ' -f2 file.txt or unix linux
This command prints the second field in each line by treating the space as delimiter. You can print more than one field by specifying the position of the fields in a comma delimited list.
cut -d' ' -f2,3 file.txt or linux unix good linux good
The above command prints the second and third field in each line.
Note: If the delimiter you specified is not exists in the line, then the cut command prints the entire line. To suppress these lines use the -s option in cut command.
4. Write a unix/linux cut command to display range of fields?
You can print a range of fields by specifying the start and end position.
cut -d' ' -f1-3 file.txt
The above command prints the first, second and third fields. To print the first three fields, you can ignore the start position and specify only the end position.
cut -d' ' -f-3 file.txt
To print the fields from second fields to last field, you can omit the last field position.
cut -d' ' -f2- file.txt
5. Write a unix/linux cut command to display the first field from /etc/passwd file?
The /etc/passwd is a delimited file and the delimiter is a colon (:). The cut command to display the first field in /etc/passwd file is
cut -d':' -f1 /etc/passwd
6. The input file contains the below text
> cat filenames.txt logfile.dat sum.pl add_int.sh
Using the cut command extract the portion after the dot.
First reverse the text in each line and then apply the command on it.
rev filenames.txt | cut -d'.' -f1
DELETE EMPTY LINES USING SED / GREP COMMAND IN UNIX (OR LINUX)
In
Unix / Linux you can use the Sed / Grep command to remove empty lines
from a file. For example, Consider the below text file as input
Now we will see how to remove the lines from the above file in unix / linux
1. Remove lines using unix sed command
The d command in sed can be used to delete the empty lines in a file.
Here the ^ specifies the start of the line and $ specifies the end of the line. You can redirect the output of above command and write it into a new file.
2. Delete lines using unix grep command
First we will see how to search for empty lines using grep command.
Now we will use the -v option to the grep command to reverse the pattern matching
The output of both sed and grep commands after deleting the empty lines from the file is
> cat file.txt Remove line using unix grep command Delete lines using unix sed command How it works
Now we will see how to remove the lines from the above file in unix / linux
1. Remove lines using unix sed command
The d command in sed can be used to delete the empty lines in a file.
sed '/^$/d' file.txt
Here the ^ specifies the start of the line and $ specifies the end of the line. You can redirect the output of above command and write it into a new file.
sed '/^$/d' file.txt > no_empty_lines.txt
2. Delete lines using unix grep command
First we will see how to search for empty lines using grep command.
grep '^$' file.txt
Now we will use the -v option to the grep command to reverse the pattern matching
grep -v '^$' file.txt
The output of both sed and grep commands after deleting the empty lines from the file is
Remove line using unix grep command Delete lines using unix sed command How it works
CHANGE DIRECTORY (CD) EXAMPLES | UNIX AND LINUX COMMAND
The
Change directory (cd) command is one of the simple commands in Unix (or
Linux) and it is very easy to use. The cd command is used to change
from the current directory to another directory. The syntax of cd
command is
CD Command Examples
1. Write a unix/linux cd command to change to home directory?
Just simply type cd command on the unix terminal and then press the enter key. This will change your directory to home directory.
Now i am in the /usr/local/bin directory. After typing the cd command and unix window, you will go to your home directory.
Here pwd command displays the present working directory.
2. Write a unix/linux cd command to go back to one directory?
The cd .. changes the directory to its parent directory by going back one level. The space between the cd and .. is must.
3. Write a unix/linux cd command to go back to two directories?
The cd ../../ takes you back to two directories. You can extend this cd command to go back to n number of directories.
4. Write a unix/linux cd command to change the directory using the absolute path?
In case of changing directory using absolute path you have to specify the full directory path. Absolute path directories always start with a slash (/). An example is changing your directory to /usr/bin from your home directory.
5. Write a unix/linux cd command to change the directory using the relative path?
In relative path, you have to specify the directory path relative to your current directory. For example, you are in /var/tmp directory and you want to go to /var/lib directory, then you can use the relative path.
Here the cd ../lib, first takes you to the parent directory which is /var and then changes the directory to the lib.
6. Write a unix/linux cd command to change back to previous directory.
As an example, i am in the directory /home/matt/documents and i changed to a new directory /home/matt/backup. Now i want to go back to my previous directory /home/matt/documents. In this case, you can use the cd - command to go back to the previous directory.
cd [directory] Here directory is the name of the directory where you wish to go.
CD Command Examples
1. Write a unix/linux cd command to change to home directory?
Just simply type cd command on the unix terminal and then press the enter key. This will change your directory to home directory.
> pwd /usr/local/bin
Now i am in the /usr/local/bin directory. After typing the cd command and unix window, you will go to your home directory.
> cd > pwd /home/matt
Here pwd command displays the present working directory.
2. Write a unix/linux cd command to go back to one directory?
The cd .. changes the directory to its parent directory by going back one level. The space between the cd and .. is must.
> pwd /var/tmp > cd .. > pwd /var
3. Write a unix/linux cd command to go back to two directories?
The cd ../../ takes you back to two directories. You can extend this cd command to go back to n number of directories.
> pwd /usr/local/bin > cd ../../ > pwd /usr
4. Write a unix/linux cd command to change the directory using the absolute path?
In case of changing directory using absolute path you have to specify the full directory path. Absolute path directories always start with a slash (/). An example is changing your directory to /usr/bin from your home directory.
> cd /usr/bin
5. Write a unix/linux cd command to change the directory using the relative path?
In relative path, you have to specify the directory path relative to your current directory. For example, you are in /var/tmp directory and you want to go to /var/lib directory, then you can use the relative path.
> pwd /var/tmp > cd ../lib > pwd /var/lib
Here the cd ../lib, first takes you to the parent directory which is /var and then changes the directory to the lib.
6. Write a unix/linux cd command to change back to previous directory.
As an example, i am in the directory /home/matt/documents and i changed to a new directory /home/matt/backup. Now i want to go back to my previous directory /home/matt/documents. In this case, you can use the cd - command to go back to the previous directory.
> pwd /home/matt/documents > cd /home/matt/backup >pwd /home/matt/backup > cd - > pwd /home/matt/documents
UNIX TIMESTAMP COMMAND
What is Unix Timestamp
Unix timestamp is the representation of time as the running total of number of seconds since the unix epoch time on January 1st, 1970. Simply the Unix timestamp is the number of seconds between the particular date and the Unix Epoch.
The unix timestamp become standard in computer systems for tracking the information especially in distributed processing system like hadoop, cloud computing etc.
Here we will see how to convert the unix date to timestamp and unix timestamp to date. We will also see how to generate the unix current timestamp. Let see each one:
1. Unix Current Timestamp
To find the unix current timestamp use the %s option in the date command. The %s option calculates unix timestamp by finding the number of seconds between the current date and unix epoch.
You will get a different output if you run the above date command.
2. Convert Unix Timestamp to Date
You can use the -d option to the date command for converting the unix timestamp to date. Here you have to specify the unix epoch and the timestamp in seconds.
3. Convert Unix Date to Timestamp
You have to combine the -d option and the %s option for converting the unix date to timestamp.
Unix timestamp is the representation of time as the running total of number of seconds since the unix epoch time on January 1st, 1970. Simply the Unix timestamp is the number of seconds between the particular date and the Unix Epoch.
The unix timestamp become standard in computer systems for tracking the information especially in distributed processing system like hadoop, cloud computing etc.
Here we will see how to convert the unix date to timestamp and unix timestamp to date. We will also see how to generate the unix current timestamp. Let see each one:
1. Unix Current Timestamp
To find the unix current timestamp use the %s option in the date command. The %s option calculates unix timestamp by finding the number of seconds between the current date and unix epoch.
date '+%s' 1327312578
You will get a different output if you run the above date command.
2. Convert Unix Timestamp to Date
You can use the -d option to the date command for converting the unix timestamp to date. Here you have to specify the unix epoch and the timestamp in seconds.
date -d "1970-01-01 956684800 sec GMT" Tue Apr 25 10:46:40 PDT 2000
3. Convert Unix Date to Timestamp
You have to combine the -d option and the %s option for converting the unix date to timestamp.
date -d "2000-01-01 GMT" '+%s' 946684800
COPY (CP) FILE AND DIRECTORY EXAMPLES | UNIX AND LINUX COMMAND
Copy
(cp) is the frequently used command in Unix (or Linux). The cp Command
is used to copy the files from one directory to another directory. The
cp command can also be used to copy the directories also. The syntax of
cp command is
Examples of cp Command
1. Write a unix/linux cp command to copy file in to a directory?
The basic usage of cp command is to copy a file from the current directory to another directory.
The cp command copies the file sum.pl into the tmp directory. The cp command does not remove the source file. It just copies the file into a new location. If a file with the same name as the source exists in the destination location, then by default the cp command overwrites that new file
2. Write a unix/linux cp to prompt for user before overwriting a file ( Interactive cp command)?
The -i option to the cp command provides the ability to prompt for a user input whether to overwrite the destination file or not.
If you enter y, then the cp command overwrites the destination file, otherwise the cp command does not copy the file.
3. Write a unix/linux cp command to copy multiple files in to a new directory?
You can specify multiple files as the source and can copy to the new location.
The cp command copies the log.dat, bad.dat files in the current directory to the tmp directory.
4. Write a unix/linux cp command to do a Regular expression copy?
You can copy a set of files by specifying a regular expression pattern.
Here the cp command copies all the files which has "dat" as suffix to the destination directory.
5. Write a unix/linux cp command to copy a file in to the current directory?
You can copy a file from a different directory to the current directory.
Here the cp command copies the multiply.sh file in the /usr/local/bin directory the current directory. The dot (.) indicates the current directory.
6. Write a unix/linux cp command to copy all the files in a directory?
The cp command can be used to copy all the files in directory to another directory.
This command copies all the files in the docs directory to the tmp directory.
7. Write a unix/linux cp command to copy files from multiple directories?
You can copy the files from different directories into a new location.
The command copies the files from docs and script directories to the destination directory tmp.
8. Write a unix/linux cp command to Copy a directory.
You can recursively copy a complete directory and its sub directory to another location using the cp command
This copies the complete directory docs into the new directory tmp
9. Write a unix/linux cp command to Forcibly copy a file with -f option?
You can force the cp command to copy an existing destination file even it cannot be opened.
cp [options] source destination
Examples of cp Command
1. Write a unix/linux cp command to copy file in to a directory?
The basic usage of cp command is to copy a file from the current directory to another directory.
cp sum.pl tmp/
The cp command copies the file sum.pl into the tmp directory. The cp command does not remove the source file. It just copies the file into a new location. If a file with the same name as the source exists in the destination location, then by default the cp command overwrites that new file
2. Write a unix/linux cp to prompt for user before overwriting a file ( Interactive cp command)?
The -i option to the cp command provides the ability to prompt for a user input whether to overwrite the destination file or not.
> cp sum.pl tmp/ cp: overwrite `tmp/sum.pl'?
If you enter y, then the cp command overwrites the destination file, otherwise the cp command does not copy the file.
3. Write a unix/linux cp command to copy multiple files in to a new directory?
You can specify multiple files as the source and can copy to the new location.
cp log.dat bad.dat tmp/
The cp command copies the log.dat, bad.dat files in the current directory to the tmp directory.
4. Write a unix/linux cp command to do a Regular expression copy?
You can copy a set of files by specifying a regular expression pattern.
cp *.dat tmp/
Here the cp command copies all the files which has "dat" as suffix to the destination directory.
5. Write a unix/linux cp command to copy a file in to the current directory?
You can copy a file from a different directory to the current directory.
cp /usr/local/bin/multiply.sh .
Here the cp command copies the multiply.sh file in the /usr/local/bin directory the current directory. The dot (.) indicates the current directory.
6. Write a unix/linux cp command to copy all the files in a directory?
The cp command can be used to copy all the files in directory to another directory.
cp docs/* tmp/
This command copies all the files in the docs directory to the tmp directory.
7. Write a unix/linux cp command to copy files from multiple directories?
You can copy the files from different directories into a new location.
cp docs/* scripts/* tmp/
The command copies the files from docs and script directories to the destination directory tmp.
8. Write a unix/linux cp command to Copy a directory.
You can recursively copy a complete directory and its sub directory to another location using the cp command
cp -r docs tmp/
This copies the complete directory docs into the new directory tmp
9. Write a unix/linux cp command to Forcibly copy a file with -f option?
You can force the cp command to copy an existing destination file even it cannot be opened.
cp -f force_file.txt /var/tmp/
LS COMMAND IN UNIX AND LINUX EXAMPLES
ls
is the most widely used command in unix or linux. ls command is used to
list the contents of a directory. Learn the power of ls command to make
your life easy. The syntax of ls command is
1. Write a unix/linux ls command to display the hidden files and directories?
To display the hidden files and directories in the current directory use the -a option of the ls command.
Hidden files are the one whose name starts with dot (.). The las -a displays the current directory (.) and parent directory (..) also. If you want to exclude the current directory, parent directory, then use -A option.
2. Write a unix/linux ls command to classify the files with special characters
The -F option to ls command classifies the files. It marks the
3. Write a unix/linux ls command to print each file in a separate line?
The -1 option to the ls command specifies that each file should be displayed on a separate line
4. Write a unix/linux ls command to display the inode number of file?
In some cases, you want to know the inode number of a file. Use -i option to the ls command to print the inode number of a file.
5. Write a unix/linux ls command to display complete information about the files?
The -l option provides lots of information about the file type, owner, group, permissions, file size, last modification date.
6. Write a unix/linux ls command to sort the files by their modification time?
The -t option allows the ls command to sort the files in descending order based on the modification time.
7. Write a unix/linux ls command to sort the files in ascending order of modification time?
The -r option reverses the order of the files displayed. Combine the -t and -r options to sort the files in ascending order.
8. Write a unix/linux ls command to print the files recursively?
So far the ls command prints the files in the current directory. Use the -R option to recursively print the files in the sub-directories also.
9. Write a unix/linux ls command to print the files in a specific directory?
You can pass a directory to the ls command as an argument to print for the files in it.
10. Write a unix/linux ls command to display files in columns?
The -x option specifies the ls command to display the files in columns.
ls [options] [pathnames]
1. Write a unix/linux ls command to display the hidden files and directories?
To display the hidden files and directories in the current directory use the -a option of the ls command.
> ls -a . .. documents .hidden_file sum.pl
Hidden files are the one whose name starts with dot (.). The las -a displays the current directory (.) and parent directory (..) also. If you want to exclude the current directory, parent directory, then use -A option.
> ls -A documents .hidden_file sum.pl
2. Write a unix/linux ls command to classify the files with special characters
The -F option to ls command classifies the files. It marks the
- Directories with trailing slash (/)
- Executable files with trailing asterisk (*)
- FIFOs with trailing vertical bar (|)
- Symbolic links with trailing at the rate sign (@)
- Regular files with nothing
> ls -F documents/ sum.pl link@
3. Write a unix/linux ls command to print each file in a separate line?
The -1 option to the ls command specifies that each file should be displayed on a separate line
> ls -1 documents sum.pl
4. Write a unix/linux ls command to display the inode number of file?
In some cases, you want to know the inode number of a file. Use -i option to the ls command to print the inode number of a file.
> ls -i1 10584066 documents 3482450 sum.pl
5. Write a unix/linux ls command to display complete information about the files?
The -l option provides lots of information about the file type, owner, group, permissions, file size, last modification date.
> ls -l total 16 drwxr-xr-x 2 matt db 4096 Jan 30 23:08 documents -rw-r--r-- 1 matt db 49 Jan 31 01:17 sum.pl
- The first character indicates the type of the file. - for normal file, d for directory, l for link file and s for socket file
- The next 9 characters in the first field represent the permissions. Each 3 characters refers the read (r), write (w), execute (x) permissions on owner, group and others. - means no permission.
- The second field indicates the number of links to that file.
- The third field indicates the owner name.
- The fourth field indicates the group name.
- The fifth field represents the file size in bytes.
- The sixth field represents the last modification date and time of the file.
- And finally the seventh field is the name of the file.
6. Write a unix/linux ls command to sort the files by their modification time?
The -t option allows the ls command to sort the files in descending order based on the modification time.
> ls -t1 sum.pl documents
7. Write a unix/linux ls command to sort the files in ascending order of modification time?
The -r option reverses the order of the files displayed. Combine the -t and -r options to sort the files in ascending order.
> ls -rt1 documents sum.pl
8. Write a unix/linux ls command to print the files recursively?
So far the ls command prints the files in the current directory. Use the -R option to recursively print the files in the sub-directories also.
> ls -R .: documents sum.pl ./documents: file.txt
9. Write a unix/linux ls command to print the files in a specific directory?
You can pass a directory to the ls command as an argument to print for the files in it.
> ls /usr/local/bin
10. Write a unix/linux ls command to display files in columns?
The -x option specifies the ls command to display the files in columns.
> ls -x
SED COMMAND IN UNIX AND LINUX EXAMPLES
Sed
is a Stream Editor used for modifying the files in unix (or linux).
Whenever you want to make changes to the file automatically, sed comes
in handy to do this. Most people never learn its power; they just simply
use sed to replace text. You can do many things apart from replacing
text with sed. Here I will describe the features of sed with examples.
Consider the below text file as an input.
1. Replacing or substituting string
Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.
Here the "s" specifies the substitution operation. The "/" are delimiters. The "unix" is the search pattern and the "linux" is the replacement string.
By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line.
2. Replacing the nth occurrence of a pattern in a line.
Use the /1, /2 etc flags to replace the first, second occurrence of a pattern in a line. The below command replaces the second occurrence of the word "unix" with "linux" in a line.
3. Replacing all the occurrence of the pattern in a line.
The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.
4. Replacing from nth occurrence to all occurrences in a line.
Use the combination of /1, /2 etc and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... "unix" word with "linux" word in a line.
5. Changing the slash (/) delimiter
You can use any delimiter other than the slash. As an example if you want to change the web url to another url as
In this case the url consists the delimiter character which we used. In that case you have to escape the slash with backslash character, otherwise the substitution won't work.
Using too many backslashes makes the sed command look awkward. In this case we can change the delimiter to another character as shown in the below example.
6. Using & as the matched string
There might be cases where you want to search for the pattern and replace that pattern by adding some extra characters to it. In such cases & comes in handy. The & represents the matched string.
7. Using \1,\2 and so on to \9
The first pair of parenthesis specified in the pattern represents the \1, the second represents the \2 and so on. The \1,\2 can be used in the replacement string to make changes to the source string. As an example, if you want to replace the word "unix" in a line with twice as the word like "unixunix" use the sed command as below.
The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is
Another example is switching the first three characters in a line
8. Duplicating the replaced line with /p flag
The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.
9. Printing only the replaced lines
Use the -n option along with the /p print flag to display only the replaced lines. Here the -n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.
If you use -n alone without /p, then the sed does not print anything.
10. Running multiple sed commands.
You can run multiple sed commands by piping the output of one sed command as input to another sed command.
Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.
11. Replacing string on a specific line number.
You can restrict the sed command to replace the string on a specific line number. An example is
The above sed command replaces the string only on the third line.
12. Replacing string on a range of lines.
You can specify a range of line numbers to the sed command for replacing a string.
Here the sed command replaces the lines with range from 1 to 3. Another example is
Here $ indicates the last line in the file. So the sed command replaces the text from second line to last line in the file.
13. Replace on a lines which matches a pattern.
You can specify a pattern to the sed command to match in a line. If the pattern match occurs, then only the sed command looks for the string to be replaced and if it finds, then the sed command replaces the string.
Here the sed command first looks for the lines which has the pattern "linux" and then replaces the word "unix" with "centos".
14. Deleting lines.
You can delete the lines a file by specifying the line number or a range or numbers.
15. Duplicating lines
You can make the sed command to print each line of a file two times.
16. Sed as grep command
You can make sed command to work as similar to grep command.
Here the sed command looks for the pattern "unix" in each line of a file and prints those lines that has the pattern.
You can also make the sed command to work as grep -v, just by using the reversing the sed with NOT (!).
The ! here inverts the pattern match.
17. Add a line after a match.
The sed command can add a new line after a pattern match is found. The "a" command to sed tells it to add a new line after a match is found.
18. Add a line before a match
The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.
19. Change a line
The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.
20. Transform like tr command
The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.
Here the sed command transforms the alphabets "ul" into their uppercase format "UL"
Consider the below text file as an input.
>cat file.txt unix is great os. unix is opensource. unix is free os. learn operating system. unixlinux which one you choose.
Sed Command Examples
1. Replacing or substituting string
Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.
>sed 's/unix/linux/' file.txt linux is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose.
Here the "s" specifies the substitution operation. The "/" are delimiters. The "unix" is the search pattern and the "linux" is the replacement string.
By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line.
2. Replacing the nth occurrence of a pattern in a line.
Use the /1, /2 etc flags to replace the first, second occurrence of a pattern in a line. The below command replaces the second occurrence of the word "unix" with "linux" in a line.
>sed 's/unix/linux/2' file.txt unix is great os. linux is opensource. unix is free os. learn operating system. unixlinux which one you choose.
3. Replacing all the occurrence of the pattern in a line.
The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.
>sed 's/unix/linux/g' file.txt linux is great os. linux is opensource. linux is free os. learn operating system. linuxlinux which one you choose.
4. Replacing from nth occurrence to all occurrences in a line.
Use the combination of /1, /2 etc and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... "unix" word with "linux" word in a line.
>sed 's/unix/linux/3g' file.txt unix is great os. unix is opensource. linux is free os. learn operating system. unixlinux which one you choose.
5. Changing the slash (/) delimiter
You can use any delimiter other than the slash. As an example if you want to change the web url to another url as
>sed 's/http:\/\//www/' file.txt
In this case the url consists the delimiter character which we used. In that case you have to escape the slash with backslash character, otherwise the substitution won't work.
Using too many backslashes makes the sed command look awkward. In this case we can change the delimiter to another character as shown in the below example.
>sed 's_http://_www_' file.txt >sed 's|http://|www|' file.txt
6. Using & as the matched string
There might be cases where you want to search for the pattern and replace that pattern by adding some extra characters to it. In such cases & comes in handy. The & represents the matched string.
>sed 's/unix/{&}/' file.txt
{unix} is great os. unix is opensource. unix is free os.
learn operating system.
{unix}linux which one you choose.
>sed 's/unix/{&&}/' file.txt
{unixunix} is great os. unix is opensource. unix is free os.
learn operating system.
{unixunix}linux which one you choose.
7. Using \1,\2 and so on to \9
The first pair of parenthesis specified in the pattern represents the \1, the second represents the \2 and so on. The \1,\2 can be used in the replacement string to make changes to the source string. As an example, if you want to replace the word "unix" in a line with twice as the word like "unixunix" use the sed command as below.
>sed 's/\(unix\)/\1\1/' file.txt unixunix is great os. unix is opensource. unix is free os. learn operating system. unixunixlinux which one you choose.
The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is
>sed 's/\(unix\)\(linux\)/\2\1/' file.txt unix is great os. unix is opensource. unix is free os. learn operating system. linuxunix which one you choose.
Another example is switching the first three characters in a line
>sed 's/^\(.\)\(.\)\(.\)/\3\2\1/' file.txt inux is great os. unix is opensource. unix is free os. aelrn operating system. inuxlinux which one you choose.
8. Duplicating the replaced line with /p flag
The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.
>sed 's/unix/linux/p' file.txt linux is great os. unix is opensource. unix is free os. linux is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose. linuxlinux which one you choose.
9. Printing only the replaced lines
Use the -n option along with the /p print flag to display only the replaced lines. Here the -n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.
>sed -n 's/unix/linux/p' file.txt linux is great os. unix is opensource. unix is free os. linuxlinux which one you choose.
If you use -n alone without /p, then the sed does not print anything.
10. Running multiple sed commands.
You can run multiple sed commands by piping the output of one sed command as input to another sed command.
>sed 's/unix/linux/' file.txt| sed 's/os/system/' linux is great system. unix is opensource. unix is free os. learn operating system. linuxlinux which one you chosysteme.
Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.
>sed -e 's/unix/linux/' -e 's/os/system/' file.txt linux is great system. unix is opensource. unix is free os. learn operating system. linuxlinux which one you chosysteme.
11. Replacing string on a specific line number.
You can restrict the sed command to replace the string on a specific line number. An example is
>sed '3 s/unix/linux/' file.txt unix is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose.
The above sed command replaces the string only on the third line.
12. Replacing string on a range of lines.
You can specify a range of line numbers to the sed command for replacing a string.
>sed '1,3 s/unix/linux/' file.txt linux is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose.
Here the sed command replaces the lines with range from 1 to 3. Another example is
>sed '2,$ s/unix/linux/' file.txt linux is great os. unix is opensource. unix is free os. learn operating system. linuxlinux which one you choose.
Here $ indicates the last line in the file. So the sed command replaces the text from second line to last line in the file.
13. Replace on a lines which matches a pattern.
You can specify a pattern to the sed command to match in a line. If the pattern match occurs, then only the sed command looks for the string to be replaced and if it finds, then the sed command replaces the string.
>sed '/linux/ s/unix/centos/' file.txt unix is great os. unix is opensource. unix is free os. learn operating system. centoslinux which one you choose.
Here the sed command first looks for the lines which has the pattern "linux" and then replaces the word "unix" with "centos".
14. Deleting lines.
You can delete the lines a file by specifying the line number or a range or numbers.
>sed '2 d' file.txt >sed '5,$ d' file.txt
15. Duplicating lines
You can make the sed command to print each line of a file two times.
>sed 'p' file.txt
16. Sed as grep command
You can make sed command to work as similar to grep command.
>grep 'unix' file.txt >sed -n '/unix/ p' file.txt
Here the sed command looks for the pattern "unix" in each line of a file and prints those lines that has the pattern.
You can also make the sed command to work as grep -v, just by using the reversing the sed with NOT (!).
>grep -v 'unix' file.txt >sed -n '/unix/ !p' file.txt
The ! here inverts the pattern match.
17. Add a line after a match.
The sed command can add a new line after a pattern match is found. The "a" command to sed tells it to add a new line after a match is found.
>sed '/unix/ a "Add a new line"' file.txt unix is great os. unix is opensource. unix is free os. "Add a new line" learn operating system. unixlinux which one you choose. "Add a new line"
18. Add a line before a match
The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.
>sed '/unix/ i "Add a new line"' file.txt "Add a new line" unix is great os. unix is opensource. unix is free os. learn operating system. "Add a new line" unixlinux which one you choose.
19. Change a line
The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.
>sed '/unix/ c "Change line"' file.txt "Change line" learn operating system. "Change line"
20. Transform like tr command
The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.
>sed 'y/ul/UL/' file.txt Unix is great os. Unix is opensoUrce. Unix is free os. Learn operating system. UnixLinUx which one yoU choose.
Here the sed command transforms the alphabets "ul" into their uppercase format "UL"
BASIC UNIX AND LINUX COMMANDS WITH EXAMPLES
Learning
unix operating system is very easy. It is just that you need to
understand the unix server concepts and familiar with the unix commands.
Here I am providing some important unix commands which will be used in
daily work.
Unix Commands With Examples:
1. Listing files
The first thing after logging into the unix system, everyone does is listing the files in a directory. The ls command is used to list the files in a directory.
If you simply execute ls on the command prompt, then it will display the files and directories in the current directory.
You can pass a directory as an argument to ls command. In this case, the ls command prints all the files and directories in the specific directory you have passed.
2. Displaying the contents of a file.
The next thing is to display the contents of a file. The cat command is used to display the contents in a file.
3. Displaying first few lines from a file.
The head command can be used to print the specified number of lines from the starting of a file. The below head command displays the first five lines of file.
4. Displaying last few lines from a file.
The tail command can be used to print the specified number of lines from the ending of a file. The below tail command displays the last three lines of file.
5. Changing the directories
The cd command can be used to change from one directory to another directory. You need to specify the target directory where you want to go.
After typing this cd command you will be in /var/tmp directory.
6. Creating a file.
The touch command simply creates an empty file. The below touch command creates a new file in the current directory.
7. copying the contents of one file into another.
The cp command is used to copy the content of source file into the target file. If the target file already have data, then it will be overwritten.
8. Creating a directory.
Directories are a way of organizing your files. The mkdir command is used to create the specified directory.
This will create the backup directory in the current directory.
9. Renaming and moving the files.
The mv command is used to rename the files and it also used for moving the files from one directory into another directory.
10. Finding the number of lines in a file
The wc command can be used to find the number of line, words and characters in a file.
To know about the unix command, it is always good to see the man pages. To see the man pages simply pass the command as an argument to the man.
Unix Commands With Examples:
1. Listing files
The first thing after logging into the unix system, everyone does is listing the files in a directory. The ls command is used to list the files in a directory.
>ls add.sh logfile.txt prime.pl
If you simply execute ls on the command prompt, then it will display the files and directories in the current directory.
>ls /usr/local/bin
You can pass a directory as an argument to ls command. In this case, the ls command prints all the files and directories in the specific directory you have passed.
2. Displaying the contents of a file.
The next thing is to display the contents of a file. The cat command is used to display the contents in a file.
>cat file.txt This is a sample unix file Learning about unix server is awesome
3. Displaying first few lines from a file.
The head command can be used to print the specified number of lines from the starting of a file. The below head command displays the first five lines of file.
>head -5 logfile.dat
4. Displaying last few lines from a file.
The tail command can be used to print the specified number of lines from the ending of a file. The below tail command displays the last three lines of file.
>tail -3 logfile.dat
5. Changing the directories
The cd command can be used to change from one directory to another directory. You need to specify the target directory where you want to go.
>cd /var/tmp
After typing this cd command you will be in /var/tmp directory.
6. Creating a file.
The touch command simply creates an empty file. The below touch command creates a new file in the current directory.
touch new_file.txt
7. copying the contents of one file into another.
The cp command is used to copy the content of source file into the target file. If the target file already have data, then it will be overwritten.
>cp source_file target_file
8. Creating a directory.
Directories are a way of organizing your files. The mkdir command is used to create the specified directory.
>mkdir backup
This will create the backup directory in the current directory.
9. Renaming and moving the files.
The mv command is used to rename the files and it also used for moving the files from one directory into another directory.
Renaming the file. >mv file.txt new_file.txt Moving the file to another directory. >mv new_file.txt tmp/
10. Finding the number of lines in a file
The wc command can be used to find the number of line, words and characters in a file.
>wc logfile.txt 21 26 198 logfile.txt
To know about the unix command, it is always good to see the man pages. To see the man pages simply pass the command as an argument to the man.
REPLACE STRING WITH AWK/SED COMMAND IN UNIX
Replace String With Awk/Sed Command In Unix:
You might have used the Sed Command often to replace the text in file. Awk can also be used to replace the strings in a file.
Here i will show you how to replace the string with awk command. To learn about replacing text with sed command go though the link, Replace String with Sed Command
Replace text with Awk command
1. First we will see a simple example of replacing the text. The source file contians the below data
We want to replace the word "unix" with "fedora". Here the word "unix" is in the second field. So, we need to check for the word "unix" in the second field and replace it with workd "fedora" by assigning the new value to the second field. The awk command to replace the text is
2. Now we will see a bit complex example.Consider the text file with the below data
Now replace the string, "top" in right section with the string "right". The output should look as
Here the delimiter in the text file is brace. We have to specify the delimiters in awk command with the record separators. The below awk command can be used to replace the string in a file
Here RS is the input record separator and ORS is the output record separator.
You might have used the Sed Command often to replace the text in file. Awk can also be used to replace the strings in a file.
Here i will show you how to replace the string with awk command. To learn about replacing text with sed command go though the link, Replace String with Sed Command
Replace text with Awk command
1. First we will see a simple example of replacing the text. The source file contians the below data
>cat file.txt Learn unix Learn linux
We want to replace the word "unix" with "fedora". Here the word "unix" is in the second field. So, we need to check for the word "unix" in the second field and replace it with workd "fedora" by assigning the new value to the second field. The awk command to replace the text is
awk '{if($2=="unix") {$2="fedora"} print $0}' file.txt
Learn fedora
Learn linux
2. Now we will see a bit complex example.Consider the text file with the below data
>cat file.txt left ( In left ) right ( In top ) top ( In top ) bottom ( In bottom )
Now replace the string, "top" in right section with the string "right". The output should look as
left ( In left ) right ( In right ) top ( In top ) bottom ( In bottom )
Here the delimiter in the text file is brace. We have to specify the delimiters in awk command with the record separators. The below awk command can be used to replace the string in a file
awk -vRS=")" '/right/{ gsub(/top/,"right"); }1' ORS=")" file.txt
Here RS is the input record separator and ORS is the output record separator.
DATE COMMAND IN UNIX AND LINUX EXAMPLES
Date
command is used to print the date and time in unix. By default the date
command displays the date in the time zone that the unix operating
system is configured.
Now let see the date command usage in unix
Date Command Examples:
1. Write a unix/linux date command to print the date on the terminal?
This is the default format in which the date command print the date and time. Here the unix server is configured in pacific standard time.
2. Write a unix/linux date command to print the date in GMT/UTC time zone?
The -u option to the date command tells it to display the time in Greenwich Mean Time.
3. Write a unix/linux date command to sett the date in unix?
You can change the date and time by using the -s option to the date command.
4. Write a unix/linux date command to display only the date part and ignore the time part?
You can format the output of date command by using the %. Here %m for month, %d for day and %Y for year.
5. Write a unix/linux date command to display only the time part and ignore the date part?
Here %H is for hours in 24 hour format, %M is for minutes and %S for seconds
6. Write a unix/linux date command to format both the date and time part.
7. Write a unix/linux date command to find the number of seconds from unix epoch.
Unix epoch is the date on January 1st, 1970. The %s option is used to find the number of seconds between the current date and unix epoch.
Now let see the date command usage in unix
Date Command Examples:
1. Write a unix/linux date command to print the date on the terminal?
>date Mon Jan 23 01:37:51 PST 2012
This is the default format in which the date command print the date and time. Here the unix server is configured in pacific standard time.
2. Write a unix/linux date command to print the date in GMT/UTC time zone?
>date -u Mon Jan 23 09:40:21 UTC 2012
The -u option to the date command tells it to display the time in Greenwich Mean Time.
3. Write a unix/linux date command to sett the date in unix?
You can change the date and time by using the -s option to the date command.
>date -s "01/01/2000 12:12:12"
4. Write a unix/linux date command to display only the date part and ignore the time part?
>date '+%m-%d-%Y' 01-23-2012
You can format the output of date command by using the %. Here %m for month, %d for day and %Y for year.
5. Write a unix/linux date command to display only the time part and ignore the date part?
>date '+%H-%M-%S' 01-48-45
Here %H is for hours in 24 hour format, %M is for minutes and %S for seconds
6. Write a unix/linux date command to format both the date and time part.
>date '+%m-%d-%Y %H-%M-%S' 01-23-2012 01-49-59
7. Write a unix/linux date command to find the number of seconds from unix epoch.
>date '+%s' 1327312228
Unix epoch is the date on January 1st, 1970. The %s option is used to find the number of seconds between the current date and unix epoch.
UNIX SEARCH FILE
One
of the basic feature of any operating system is to search for files.
Unix operating system also provides this feature for searching the
files. The Find Command in Unix is used for searching files and
directories in Unix, Linux and other Unix like operating systems.
You can specify search criteria for searching files and directories. If you do not specify any criteria, the find command searches for the files in the current directory.
Unix Search Command Examples:
1. Searching for the files in the current directory.
The dot(.) represents the current directory and -name option specifies the name of the file to be searched. This find command searches for all the files with ".sh" as the suffix.
2. Searching for the file in all the directories.
The / specifies the home directory of the user, which is at the highest level and the -type option specifies the type of file. This command searches for the regular file,"job.xml", in all the directories.
3. Searching for the file in a particular directory.
This find command searches for all the java files in the /usr/local/bin directory.
4. Searching for a directory.
The -type d indicates the directory. This find command searches for the tmp directory in the current directory.
5. Searching for a directory in another directory
You can specify search criteria for searching files and directories. If you do not specify any criteria, the find command searches for the files in the current directory.
Unix Search Command Examples:
1. Searching for the files in the current directory.
find . -name '*.sh'
The dot(.) represents the current directory and -name option specifies the name of the file to be searched. This find command searches for all the files with ".sh" as the suffix.
2. Searching for the file in all the directories.
find / -type f -name 'job.xml'
The / specifies the home directory of the user, which is at the highest level and the -type option specifies the type of file. This command searches for the regular file,"job.xml", in all the directories.
3. Searching for the file in a particular directory.
find /usr/local/bin/ -type f -name '*.java'
This find command searches for all the java files in the /usr/local/bin directory.
4. Searching for a directory.
find . -type d -name 'tmp'
The -type d indicates the directory. This find command searches for the tmp directory in the current directory.
5. Searching for a directory in another directory
find /var/tmp/ -typd d -name 'personal'This find command searches for the personal directory in the /var/tmp directory.
FIND COMMAND TO DELETE FILES AND DIRECTORIES
Most
of you might have used find command to search for files and
directories. The find command can also be used to delete the files and
directories. The find command has -delete option
which can be used to delete files, directories etc. Be careful while
using the -delete option of the find command especially when using
recursive find. Otherwise you will end up in deleting the important
files.
1. The basic find command to delete a file in the current directory is
2. The find command to remove empty files in the current directory is
3. The find command to delete empty directories is
1. The basic find command to delete a file in the current directory is
find . -name filename -delete
2. The find command to remove empty files in the current directory is
find . -type f -empty -delete
3. The find command to delete empty directories is
find . -type d -empty -delete
MAKE AWK COMMAND CASE INSENSITIVE
Awk
command is used to parse the files which have delimited data. By
default, awk command does a case-sensitive parsing. The awk command has a
IGNORECASE built-in variable to do a case insensitive parsing. We will
see about the IGNORECASE in detail here.
Consider a sample text file with the below data
The below awk command can be used to display the lines which have the word "iphone" in it.
awk '{if($2 == "iphone") print $0 }' file.txt
This awk command looks for the word "iphone" in the second column of each line and if it finds a match, then it displays that line. Here it just matches the word "iphone" and did not match the word "Iphone". The awk did a case sensitive match.
The output of the above command is
You can make the awk to do case insensitive and match even for the words like "Iphone" or "IPHONE" etc.
The IGNORECASE is a built in variable which can be used in awk command to make it either case sensitive or case insensitive.
If the IGNORECASE value is 0, then the awk does a case sensitive match. If the value is 1, then the awk does a case insensitive match.
The awk command for case insensitive match is
The ouput is
Consider a sample text file with the below data
>cat file.txt mark iphone jhon sony peter Iphone chrisy motorola
The below awk command can be used to display the lines which have the word "iphone" in it.
awk '{if($2 == "iphone") print $0 }' file.txt
This awk command looks for the word "iphone" in the second column of each line and if it finds a match, then it displays that line. Here it just matches the word "iphone" and did not match the word "Iphone". The awk did a case sensitive match.
The output of the above command is
mark iphone
You can make the awk to do case insensitive and match even for the words like "Iphone" or "IPHONE" etc.
The IGNORECASE is a built in variable which can be used in awk command to make it either case sensitive or case insensitive.
If the IGNORECASE value is 0, then the awk does a case sensitive match. If the value is 1, then the awk does a case insensitive match.
The awk command for case insensitive match is
awk 'BEGIN {IGNORECASE = 1} {if($2 == "iphone") print $0 }' file.txt
The ouput is
mark iphone peter Iphone
PARSING /ETC/PASSWD UNIX FILE WITH AWK COMMAND
Awk
command parses the files which have delimited structure. The
/etc/passwd file is a delimited file. Using the Awk command is a good
choice to parse the /etc/passwd file. Sample /etc/passwd file looks like
as below
The /etc/passwd file contains the data in the form of row and columns. The columns are delimited by a colon (:) character.
Now we will see how to write an Awk command which reads the /etc/passwd file and prints the names of the users who have the /bin/bash program as their defaualt shell command.
awk -F: '$7 == "/bin/bash" { print $1 }' /etc/passwd
The -F option is used to specify the filed delimiter.
The output of the above awk command is
root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync
The /etc/passwd file contains the data in the form of row and columns. The columns are delimited by a colon (:) character.
Now we will see how to write an Awk command which reads the /etc/passwd file and prints the names of the users who have the /bin/bash program as their defaualt shell command.
awk -F: '$7 == "/bin/bash" { print $1 }' /etc/passwd
The -F option is used to specify the filed delimiter.
The output of the above awk command is
root
GREP COMMAND IN UNIX AND LINUX EXAMPLES
Grep
is the frequently used command in Unix (or Linux). Most of us use grep
just for finding the words in a file. The power of grep comes with using
its options and regular expressions. You can analyze large sets of log
files with the help of grep command.
Grep stands for Global search for Regular Expressions and Print.
The basic syntax of grep command is
grep [options] pattern [list of files]
Let see some practical examples on grep command.
1. Running the last executed grep command
This saves a lot of time if you are executing the same command again and again.
2. Search for a string in a file
This is the basic usage of grep command. It searches for the given string in the specified file.
3. Searching for a string in multiple files.
4. Case insensitive search
The -i option enables to search for a string case insensitively in the give file. It matches the words like "UNIX", "Unix", "unix".
5. Specifying the search string as a regular expression pattern.
6. Checking for the whole words in a file.
By default, grep matches the given string/pattern even if it found as a substring in a file. The -w option to grep makes it match only the whole words.
7. Displaying the lines before the match.
Some times, if you are searching for an error in a log file; it is always good to know the lines around the error lines to know the cause of the error.
8. Displaying the lines after the match.
9. Displaying the lines around the match
10. Searching for a sting in all files recursively
You can search for a string in all the files under the current directory and sub-directories with the help -r option.
11. Inverting the pattern match
You can display the lines that are not matched with the specified search sting pattern using the -v option.
12. Displaying the non-empty lines
You can remove the blank lines using the grep command.
13. Displaying the count of number of matches.
We can find the number of lines that matches the given string/pattern
14. Display the file names that matches the pattern.
We can just display the files that contains the given string/pattern.
15. Display the file names that do not contain the pattern.
We can display the files which do not contain the matched string/pattern.
16. Displaying only the matched pattern.
By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.
17. Displaying the line numbers.
We can make the grep command to display the position of the line which contains the matched string in a file using the -n option
18. Displaying the position of the matched string in the line
The -b option allows the grep command to display the character position of the matched string in a file.
19. Matching the lines that start with a string
The ^ regular expression pattern specifies the start of a line. This can be used in grep to match the lines which start with the given string or pattern.
20. Matching the lines that end with a string
The $ regular expression pattern specifies the end of a line. This can be used in grep to match the lines which end with the given string or pattern.
Grep stands for Global search for Regular Expressions and Print.
The basic syntax of grep command is
grep [options] pattern [list of files]
Let see some practical examples on grep command.
1. Running the last executed grep command
This saves a lot of time if you are executing the same command again and again.
!grepThis displays the last executed grep command and also prints the result set of the command on the terminal.
2. Search for a string in a file
This is the basic usage of grep command. It searches for the given string in the specified file.
grep "Error" logfile.txtThis searches for the string "Error" in the log file and prints all the lines that has the word "Error".
3. Searching for a string in multiple files.
grep "string" file1 file2 grep "string" file_patternThis is also the basic usage of the grep command. You can manually specify the list of files you want to search or you can specify a file pattern (use regular expressions) to search for.
4. Case insensitive search
The -i option enables to search for a string case insensitively in the give file. It matches the words like "UNIX", "Unix", "unix".
grep -i "UNix" file.txt
5. Specifying the search string as a regular expression pattern.
grep "^[0-9].*" file.txtThis will search for the lines which starts with a number. Regular expressions is huge topic and I am not covering it here. This example is just for providing the usage of regular expressions.
6. Checking for the whole words in a file.
By default, grep matches the given string/pattern even if it found as a substring in a file. The -w option to grep makes it match only the whole words.
grep -w "world" file.txt
7. Displaying the lines before the match.
Some times, if you are searching for an error in a log file; it is always good to know the lines around the error lines to know the cause of the error.
grep -B 2 "Error" file.txtThis will prints the matched lines along with the two lines before the matched lines.
8. Displaying the lines after the match.
grep -A 3 "Error" file.txtThis will display the matched lines along with the three lines after the matched lines.
9. Displaying the lines around the match
grep -C 5 "Error" file.txtThis will display the matched lines and also five lines before and after the matched lines.
10. Searching for a sting in all files recursively
You can search for a string in all the files under the current directory and sub-directories with the help -r option.
grep -r "string" *
11. Inverting the pattern match
You can display the lines that are not matched with the specified search sting pattern using the -v option.
grep -v "string" file.txt
12. Displaying the non-empty lines
You can remove the blank lines using the grep command.
grep -v "^$" file.txt
13. Displaying the count of number of matches.
We can find the number of lines that matches the given string/pattern
grep -c "sting" file.txt
14. Display the file names that matches the pattern.
We can just display the files that contains the given string/pattern.
grep -l "string" file.txt
15. Display the file names that do not contain the pattern.
We can display the files which do not contain the matched string/pattern.
grep -l "string" file.txt
16. Displaying only the matched pattern.
By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.
grep -o "string" file.txt
17. Displaying the line numbers.
We can make the grep command to display the position of the line which contains the matched string in a file using the -n option
grep -n "string" file.txt
18. Displaying the position of the matched string in the line
The -b option allows the grep command to display the character position of the matched string in a file.
grep -o -b "string" file.txt
19. Matching the lines that start with a string
The ^ regular expression pattern specifies the start of a line. This can be used in grep to match the lines which start with the given string or pattern.
grep "^start" file.txt
20. Matching the lines that end with a string
The $ regular expression pattern specifies the end of a line. This can be used in grep to match the lines which end with the given string or pattern.
grep "end$" file.txt
METHODS TO FIND SUM OF NUMBERS IN A FILE - UNIX
In Unix, there are different ways to find the sum of numbers in a file. I will explain this with help of an example.
I have an input file (num.txt) which contains the below data
I have to sum the values in each line from the file and print the result on the terminal. The required output is
Method1: Finding the sum using the bash script. Save the below bash script in a file and execute the file.
Method2: Another way of implementing in bash is
Method3: You can use “Awk” command to find the sum of numbers in a file.
Method4: The "bc" command can be used to do math operations. We will use the "tr" command along with the "bc" to find the sum of numbers.
Here the "tr" command converts the new line characters into "+" character. The output of tr '\n' '+' < num.txt is 1+2+3+4+5+. The "sed" command removes the addition "+" sign at the end. Finally the "bc" command finds the sum.
Method5: Using "bc" with "paste" command.
The "paste" command works same as the "tr" command except it does not produce the additional "+" at the end.
Method6: Using "bc" with "sed" command.
Here the “sed” command adds the "+" sign at the end of each line.
I have an input file (num.txt) which contains the below data
>cat num.txt 1 2 3 4 5
I have to sum the values in each line from the file and print the result on the terminal. The required output is
15
Method1: Finding the sum using the bash script. Save the below bash script in a file and execute the file.
#!/bin/bash SUM=0 while read LINE do SUM=`expr $SUM + $LINE` done < num.txt echo $SUM
Method2: Another way of implementing in bash is
SUM=0
for num in $(cat num.txt)
do
((SUM+=num))
done
echo $SUM
Method3: You can use “Awk” command to find the sum of numbers in a file.
awk 'BEGIN{sum=0} {sum=sum+$1} END {print sum}' num.txt
Method4: The "bc" command can be used to do math operations. We will use the "tr" command along with the "bc" to find the sum of numbers.
tr '\n' '+' < num.txt | echo $(sed 's/+$//') | bc
Here the "tr" command converts the new line characters into "+" character. The output of tr '\n' '+' < num.txt is 1+2+3+4+5+. The "sed" command removes the addition "+" sign at the end. Finally the "bc" command finds the sum.
Method5: Using "bc" with "paste" command.
paste -sd+ num.txt | bc
The "paste" command works same as the "tr" command except it does not produce the additional "+" at the end.
Method6: Using "bc" with "sed" command.
echo $(sed 's/$/+/' num.txt) 0 | bc
Here the “sed” command adds the "+" sign at the end of each line.
GROUPING A SET OF LINES AS A PARAGRAPH - UNIX AWK COMMAND
The
awk command can be used to group a set of lines into a paragraph. We
will also use a bash shell script to group the lines into a paragraph.
As an example, consider the file, group.txt, with the below data
We want to group 3 lines in the file as a paragraph. The required output is
The bash script for achieving this is
Now we will see how to achieve this using the Awk command in Unix. The awk command for this is
>cat group.txt A one B two C three D four E five F six G seven
We want to group 3 lines in the file as a paragraph. The required output is
A one B two C three D four E five F six G seven
The bash script for achieving this is
#!/bin/bash line_count=1 while read line do S=`expr $line_count % 3` if [ "$S" -eq 0 ] then echo -e $line"\n" else echo $line fi line_count=`expr $line_count + 1` done < group.txt
Now we will see how to achieve this using the Awk command in Unix. The awk command for this is
awk '!( NR % 3 ) {$0 = $0"\n"} 1' group.txt
METHODS TO CONVERT HEXADECIMAL TO DECIMAL IN UNIX
We will see how to convert hexadecimal numbers into decimal numbers using Unix command bash scripting.
The bc command can be used to convert hexadecimal number into decimal number
Example:
The hexadecimal number "ABCD" is converted into decimal number 43981.
The bc command can also be used convert the decimal numbers back to hexadecimal numbers.
Exmple:
Converting hexadecimal to decimal using bash script:
Consider the below file with hexadecimal numbers.
The bash script shown below converts the hexadecimal numbers into decimal numbers
The output after running this script is
Converting decimal to hexadecimal using bash script:
The input file contains the decimal numbers with below data.
Now we will do the reverse process. The bash script for converting the decimal numbers to hexadecimal numbers is
After running this script the output will be
The bc command can be used to convert hexadecimal number into decimal number
Example:
>echo "ibase=16;ABCD"|bc 43981
The hexadecimal number "ABCD" is converted into decimal number 43981.
The bc command can also be used convert the decimal numbers back to hexadecimal numbers.
Exmple:
>echo "obase=16;43981"|bc ABCD
Converting hexadecimal to decimal using bash script:
Consider the below file with hexadecimal numbers.
>cat hexa.txt ABCD 125A F36C E962
The bash script shown below converts the hexadecimal numbers into decimal numbers
#!/bin/bash while read line do printf "%d\n" "0x"$line done < hexa.txt
The output after running this script is
43981 4698 62316 59746
Converting decimal to hexadecimal using bash script:
The input file contains the decimal numbers with below data.
>cat deci.txt 43981 4698 62316 59746
Now we will do the reverse process. The bash script for converting the decimal numbers to hexadecimal numbers is
#!/bin/bash while read line do printf "%0X\n" $line done < deci.txt
After running this script the output will be
ABCD 125A F36C E962
PRINT FILE IN REVERSE USING UNIX COMMAND
Q)
How to print the lines in a file in reverse order? Which means we have
to print the data of file from last line to the first line.
We will see different methods to reverse the data in a file. As an example, consider the file with the below data.
We need to display the lines in a file in reverse order. The output data is
1.The tac command in unix can be used to print the file in reverse. The tac command is
2. The sed command for reversing the lines in a file is.
3. Another usage of sed command is
We will see different methods to reverse the data in a file. As an example, consider the file with the below data.
>cat file.txt Header line2 line3 line4 Footer
We need to display the lines in a file in reverse order. The output data is
Footer line4 line3 line2 Header
1.The tac command in unix can be used to print the file in reverse. The tac command is
tac file.txt
2. The sed command for reversing the lines in a file is.
sed '1!G;h;$!d' file.txt
3. Another usage of sed command is
sed -n '1!G;h;$p' file.txt
METHODS TO REVERSE A STRING USING UNIX COMMANDS
This topic will cover different methods to reverse each character in a string and reversing the tokens in a string.
Reversing a string:
1. The sed command can be used to reverse a string. The sed command for this is
2. Aother usage of sed command with tac and tr is
3. The awk command for reversing the string is
The output of all the above four methods is the reverse of the string "hello world", which is
5. Using the rev command
We can use the rev command to reverse the string which is shown below:
Reversing the tokens in a string:
1. The awk command can be used to reverse the tokens in a string. The awk command for this is
2. Using the tac and tr command we can reverse the tokens in a string. The unix command is
3. The bash script for reversing the tokens in a string is.
The output of the above two methods is
Reversing a string:
1. The sed command can be used to reverse a string. The sed command for this is
echo "hello world" | sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
2. Aother usage of sed command with tac and tr is
echo "hello world" |sed 's/./&\n/g' |tac |tr -d '\n'
3. The awk command for reversing the string is
echo "hello world" | awk '{
n=split($0,arr,"");
for(i=1;i<=n;i++)
s=arr[i] s
}
END
{
print s
}'
4. In this method, a bash script will be written to reverse the string. The bash script is#!/bin/bash str="hello world" len=`echo $str | wc -c` len=`expr $len - 1` rev="" while test $len -gt 0 do rev1=`echo $str | cut -c$len` rev=$rev$rev1 len=`expr $len - 1` done echo $rev
The output of all the above four methods is the reverse of the string "hello world", which is
dlrow olleh
5. Using the rev command
We can use the rev command to reverse the string which is shown below:
echo "hello world"|rev
Reversing the tokens in a string:
1. The awk command can be used to reverse the tokens in a string. The awk command for this is
echo "hello world" | awk '{
n=split($0,A);
S=A[n];
for(i=n-1;i>0;i--)
S=S" "A[i]
}
END
{
print S
}'
2. Using the tac and tr command we can reverse the tokens in a string. The unix command is
echo "hello world"|tac -s " "| tr "\n" " "
3. The bash script for reversing the tokens in a string is.
#!/bin/bash TOKENS="hello world" for i in $TOKENS do STR="$i $STR" done echo $STR
The output of the above two methods is
world hello
REMOVE TRAILING ZEROS USING UNIX COMMAND
Q) How to remove the trailing zeros from each line of a file?
Trailing zeros are the zero which appear only at the end of the line. There are many ways to remove the trailing zeros. Let us assume that the source file contains the below data.
The required output should not contain the trailing zeros. The output should be
The unix command for producing this result is
Here the rev command will reverse the string in each line. Now the trailing zeros will become leading zeros. In the awk command the string is converted into a number and the leading zeros will be removed. At the end, the rev command again reverses the string.
Trailing zeros are the zero which appear only at the end of the line. There are many ways to remove the trailing zeros. Let us assume that the source file contains the below data.
>cat file.txt 12345 67890 10100 10000
The required output should not contain the trailing zeros. The output should be
12345 6789 101 1
The unix command for producing this result is
rev file.txt | awk '{print $1*1}'|rev
Here the rev command will reverse the string in each line. Now the trailing zeros will become leading zeros. In the awk command the string is converted into a number and the leading zeros will be removed. At the end, the rev command again reverses the string.
REMOVE THE LINES FROM A FILE WHICH ARE SAME AS THE FIRST LINE - UNIX AWK
Q) How to remove the lines which are same as the first line.
Awk command can be used to remove the lines which are same as the first line in a file. I will also show you another method of removing the file. As an example, consider the file with the below data.
The first line contains the text "Header". We need to remove the lines which has the same text as the first line.
The required output data is
The awk command can be used to achieve this. The awk command for this is
The other way to get the output is using bash script.
Awk command can be used to remove the lines which are same as the first line in a file. I will also show you another method of removing the file. As an example, consider the file with the below data.
Header line2 line3 Header line5 line6 line7 Header
The first line contains the text "Header". We need to remove the lines which has the same text as the first line.
The required output data is
Header line2 line3 line5 line6 line7
The awk command can be used to achieve this. The awk command for this is
awk '{
if(NR==1)
{
x=$0;
print $0
}
else if(x!=$0)
print $0
}' file.txt
The other way to get the output is using bash script.
x=`head -1 file.txt`; echo $x; cat file.txt | grep -v \^"${x}"\$
EXAMPLES OF BASENAME COMMAND IN UNIX
The basename utility is used to
basename [string] [suffix]
Here 'string' is the input string and suffix is the string which needs to removed from the input string.
Examples:
1. basename /usr/bin/perlscript
This will remove the prefix, /usr/bin/, and prints only the string 'perlscript'
2. basename perlscript script
This will remove the suffix 'script' from 'perlscript' and prints only 'perl'
3. basename /usr/bin/perlscript script
This will remove both the prefix and suffix and prints only 'perl'
basename command is mostly used in shell scripts to get the name of the shell script file you are running. Sample shell script code is shown below
- Remove any prefix ending in /.
- Remove the suffix from a string.
basename [string] [suffix]
Here 'string' is the input string and suffix is the string which needs to removed from the input string.
Examples:
1. basename /usr/bin/perlscript
This will remove the prefix, /usr/bin/, and prints only the string 'perlscript'
2. basename perlscript script
This will remove the suffix 'script' from 'perlscript' and prints only 'perl'
3. basename /usr/bin/perlscript script
This will remove both the prefix and suffix and prints only 'perl'
basename command is mostly used in shell scripts to get the name of the shell script file you are running. Sample shell script code is shown below
#!/usr/bin/sh filename=`basename $0` echo $filename
EXAMPLES OF ALIAS COMMAND IN UNIX
Alias
command is an alternative name used for long strings that are
frequently used. It is mostly used for creating a simple name for a long
command.
Syntax of alias command:
alias [alias_name=['command']]
For more information on alias utility see the man pages. Type 'man alias' on the command prompt.
Examples:
1. alias
If you simply type alias on the command prompt and then enter, it will list all the aliases that were created.
2. alias pg='ps -aef'
The ps -aef command will list all the running processes. After creating the alias pg for ps -aef, then by using the pg on command prompt will display the running processes. The pg will work same as the ps -aef.
By creating an alias for a command on the command prompt will be present only for that session. Once you exit from the session, then the aliases won’t take effect. To make the aliases to remain permanent, place the alias command in the ".profile" of the user. Open the user ".profile" and place the command alias pg="ps -aef", save the file and then source the ".profile" file. Now the alias pg will remain forever.
To remove an alias use the unalias command
Example: unalias pg
Syntax of alias command:
alias [alias_name=['command']]
For more information on alias utility see the man pages. Type 'man alias' on the command prompt.
Examples:
1. alias
If you simply type alias on the command prompt and then enter, it will list all the aliases that were created.
2. alias pg='ps -aef'
The ps -aef command will list all the running processes. After creating the alias pg for ps -aef, then by using the pg on command prompt will display the running processes. The pg will work same as the ps -aef.
By creating an alias for a command on the command prompt will be present only for that session. Once you exit from the session, then the aliases won’t take effect. To make the aliases to remain permanent, place the alias command in the ".profile" of the user. Open the user ".profile" and place the command alias pg="ps -aef", save the file and then source the ".profile" file. Now the alias pg will remain forever.
To remove an alias use the unalias command
Example: unalias pg
CONVERTING AWK SCRIPT TO PERL SCRIPT - EXAMPLES OF A2P UNIX COMMAND
Unix
provides the a2p (awk to perl) utility for converting the awk script to
perl script. The a2p command takes an awk script and produces a
comparable perl script.
Syntax of a2p:
a2p [options] [awk_script_filename]
Some of the useful options that you can pass to a2p are:
-D<number> Sets debugging flags.
-F<character> This will tell a2p that awk script is always invoked with -F option.
-<number> This makes a2p to assume that input will always have the specified number of fields.
For more options see the man pages; man a2p
Example1:
The awk script which prints the squares of numbers up to 10 is shown below. Call the below script as awk_squares.
Now we will convert this script using the a2p as
a2p awk_squares > perl_squares
The content of converted perl script, perl_squares, is shown below:
Example2:
We will see an awk script which prints the first field from a file. The awk script for this is shown below. Call this script at awk_first_field.
We will convert this awk script into per script using the a2p command as
a2p awk_first_field > perl_first_field
The content of converted perl script, perl_first_field, is shown below:
Syntax of a2p:
a2p [options] [awk_script_filename]
Some of the useful options that you can pass to a2p are:
-D<number> Sets debugging flags.
-F<character> This will tell a2p that awk script is always invoked with -F option.
-<number> This makes a2p to assume that input will always have the specified number of fields.
For more options see the man pages; man a2p
Example1:
The awk script which prints the squares of numbers up to 10 is shown below. Call the below script as awk_squares.
#!/bin/awk -f
BEGIN
{
for (i=1; i <= 10; i++)
{
print "The square of ", i, " is ", i*i;
}
exit;
}
Run this script using awk command; awk -f awk_squares. This will produce squares of numbers up to 10.Now we will convert this script using the a2p as
a2p awk_squares > perl_squares
The content of converted perl script, perl_squares, is shown below:
#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"mailto:$@%22%7D'
if $running_under_some_shell;
# this emulates #! processing on NIH machines.
# (remove #! line above if indigestible)
eval '$'.$1.'$2;' while $ARGV[0] =~ /^([A-Za-z_0-9]+=)(.*)/ && shift;
# process any FOO=bar switches
$, = ' '; # set output field separator
$\ = "\n"; # set output record separator
for ($i = 1; $i <= 10; $i++) {
print 'The square of ', $i, ' is ', $i * $i;
}
last line;
Run the perl script as: perl perl_squares. This will produce the same result as the awk.Example2:
We will see an awk script which prints the first field from a file. The awk script for this is shown below. Call this script at awk_first_field.
#!/bin/awk -f
{
print $1;
}
Run this script using awk command by passing a file as input: awk
-f awk_first_field file_name. This will prints the first field of each
line from the file_name.We will convert this awk script into per script using the a2p command as
a2p awk_first_field > perl_first_field
The content of converted perl script, perl_first_field, is shown below:
#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"mailto:$@%22%7D'
if $running_under_some_shell;
# this emulates #! processing on NIH machines.
# (remove #! line above if indigestible)
eval '$'.$1.'$2;' while $ARGV[0] =~ /^([A-Za-z_0-9]+=)(.*)/ && shift;
# process any FOO=bar switches
$, = ' '; # set output field separator
$\ = "\n"; # set output record separator
while (<>) {
($Fld1) = split(' ', $_, -1);
print $Fld1;
}
Now run the perl script as: perl perl_first_field file_name. This will produce the same result as awk command.
No comments:
Post a Comment