Showing posts with label Unix. Show all posts
Showing posts with label Unix. Show all posts

Friday 9 October 2015

How to create a large file using shell script, To test with Hadoop Map Reduce processing



1. Open new Terminal.


2. Create 'myscript.sh'  file using below command:

touch myscipt.sh


3. Create 'smallfile.txt' file using below command:

touch smallfile.txt


4. Open the file using below command:

gedit smallfile.txt


5. Enter some sample data into the file.

"
I am going to hyderabad
I am learning hadoop course
"

6. Save the file using command: Ctrl + S


7. Now prepare a large file using below steps.


8. Open  the file using below command:

gedit myscript.sh


9.  Add the below script into the file.

#!/bin/bash
for i in {1..1000}
do
   cat smallfile.txt >> largefile.txt
done


10. Save the file using command: Ctrl + S


11. Provide execute permission to script file using below command:

chmod +x myscript.sh


12. Run the script file to generate a large file using below command:

./myscript.sh


13. Verify the 'largefile.txt' in current folder.


14. Run the above command multiple times to get more large file.


15. Try "hadoop fs -put " commad to place into HDFS.


16. Run Mapreduce programs using above file in step15.





Wednesday 9 October 2013

UNIX INTERVIEW QUESTIONS

UNIX INTERVIEW QUESTIONS ON AWK COMMAND

Awk is powerful tool in Unix. Awk is an excellent tool for processing the files which have data arranged in rows and columns format. It is a good filter and report writer. 
1. How to run awk command specified in a file?
awk -f filename

2. Write a command to print the squares of numbers from 1 to 10 using awk command
awk 'BEGIN { for(i=1;i<=10;i++) {print "square of",i,"is",i*i;}}'

3. Write a command to find the sum of bytes (size of file) of all files in a directory.
ls -l | awk 'BEGIN {sum=0} {sum = sum + $5} END {print sum}'

4. In the text file, some lines are delimited by colon and some are delimited by space. Write a command to print the third field of each line.

awk '{ if( $0 ~ /:/ ) { FS=":"; } else { FS =" "; } print $3 }' filename

5. Write a command to print the line number before each line?
awk '{print NR, $0}' filename

6. Write a command to print the second and third line of a file without using NR.
awk 'BEGIN {RS="";FS="\n"} {print $2,$3}' filename

7. Write a command to print zero byte size files?
ls -l | awk '/^-/ {if ($5 !=0 ) print $9 }'

8. Write a command to rename the files in a directory with "_new" as postfix?
ls -F | awk '{print "mv "$1" "$1".new"}' | sh

9. Write a command to print the fields in a text file in reverse order?
awk 'BEGIN {ORS=""} { for(i=NF;i>0;i--) print $i," "; print "\n"}' filename

10. Write a command to find the total number of lines in a file without using NR
awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' filename

Another way to print the number of lines is by using the NR. The command is
awk 'END{print NR}' filename


UNIX INTERVIEW QUESTIONS ON GREP COMMAND

The grep is one of the powerful tools in unix. Grep stands for "global search for regular expressions and print". The power of grep lies in using regular expressions mostly.

The general syntax of grep command is
grep [options] pattern [files]

1. Write a command to print the lines that has the the pattern "july" in all the files in a particular directory?

grep july *
This will print all the lines in all files that contain the word “july” along with the file name. If any of the files contain words like "JULY" or "July", the above command would not print those lines.

2. Write a command to print the lines that has the word "july" in all the files in a directory and also suppress the filename in the output.

grep -h july *

3. Write a command to print the lines that has the word "july" while ignoring the case.

grep -i july *
The option i make the grep command to treat the pattern as case insensitive.

4. When you use a single file as input to the grep command to search for a pattern, it won't print the filename in the output. Now write a grep command to print the filename in the output without using the '-H' option.
grep pattern filename /dev/null
The /dev/null or null device is special file that discards the data written to it. So, the /dev/null is always an empty file.

Another way to print the filename is using the '-H' option. The grep command for this is
grep -H pattern filename

5. Write a Unix command to display the lines in a file that do not contain the word "july"?
grep -v july filename
The '-v' option tells the grep to print the lines that do not contain the specified pattern.

6. Write a command to print the file names in a directory that has the word "july"?
grep -l july *
The '-l' option make the grep command to print only the filename without printing the content of the file. As soon as the grep command finds the pattern in a file, it prints the pattern and stops searching other lines in the file.

7. Write a command to print the file names in a directory that does not contain the word "july"?
grep -L july *
The '-L' option makes the grep command to print the filenames that do not contain the specified pattern.

8. Write a command to print the line numbers along with the line that has the word "july"?
grep -n july filename
The '-n' option is used to print the line numbers in a file. The line numbers start from 1

9. Write a command to print the lines that starts with the word "start"?
grep '^start' filename
The '^' symbol specifies the grep command to search for the pattern at the start of the line.

10. Write a command to print the lines which end with the word "end"?
grep 'end$' filename
The '$' symbol specifies the grep command to search for the pattern at the end of the line.

11. Write a command to select only those lines containing "july" as a whole word?
grep -w july filename
The '-w' option makes the grep command to search for exact whole words. If the specified pattern is found in a string, then it is not considered as a whole word. For example: In the string "mikejulymak", the pattern "july" is found. However "july" is not a whole word in that string.


UNIX INTERVIEW QUESTIONS ON SED COMMAND

SED is a special editor used for modifying files automatically.

1. Write a command to replace the word "bad" with "good" in file?
sed s/bad/good/ < filename

2. Write a command to replace the word "bad" with "good" globally in a file?
sed s/bad/good/g < filename

3. Write a command to replace the character '/' with ',' in a file?
sed 's/\//,/' < filename
sed 's|/|,|' < filename

4. Write a command to replace the word "apple" with "(apple)" in a file?
sed s/apple/(&)/ < filename

5. Write a command to switch the two consecutive words "apple" and "mango" in a file?
sed 's/\(apple\) \(mango\)/\2 \1/' < filename

6. Write a command to replace the second occurrence of the word "bat" with "ball" in a file?
sed 's/bat/ball/2' < filename

7. Write a command to remove all the occurrences of the word "jhon" except the first one in a line with in the entire file?
sed 's/jhon//2g' < filename

8. Write a command to remove the first number on line 5 in file?
sed '5 s/[0-9][0-9]*//' < filename

9. Write a command to remove the first number on all lines that start with "@"?
sed '\,^@, s/[0-9][0-9]*//' < filename

10. Write a command to replace the word "gum" with "drum" in the first 100 lines of a file?
sed '1,00 s/gum/drum/' < filename

11. write a command to replace the word "lite" with "light" from 100th line to last line in a file?
sed '100,$ s/lite/light/' < filename

12. Write a command to remove the first 10 lines from a file?
sed '1,10 d' < filename

13. Write a command to duplicate each line in a file?
sed 'p' < filename

14. Write a command to duplicate empty lines in a file?
sed '/^$/ p' < filename

15. Write a sed command to print the lines that do not contain the word "run"?
sed -n '/run/!p' < filename


UNIX INTERVIEW QUESTIONS ON CUT COMMAND

The cut command is used to used to display selected columns or fields from each line of a file. Cut command works in two modes:
  • Delimited selection: The fields in the line are delimited by a single character like blank,comma etc.
  • Range selection: Each field starts with certain fixed offset defined as range.
1. Write a command to display the third and fourth character from each line of a file?
cut -c 3,4 filename

2. Write a command to display the characters from 10 to 20 from each line of a file?
cut -c 10-20 filename

3. Write a command to display the first 10 characters from each line of a file?
cut -c -10 filename

4. Write a comamnd to display from the 10th character to the end of the line?
cut -c 10- filename

5. The fields in each line are delimited by comma. Write a command to display third field from each line of a file?
cut -d',' -f2 filename

6. Write a command to print the fields from 10 to 20 from each line of a file?
cut -d',' -f10-20 filename

7. Write a command to print the first 5 fields from each line?
cut -d',' -f-5 filename

8. Write a command to print the fields from 10th to the end of the line?
cut -d',' -f10- filename

9. By default the cut command displays the entire line if there is no delimiter in it. Which cut option is used to supress these kind of lines?
The -s option is used to supress the lines that do not contain the delimiter.

10. Write a cut command to extract the username from 'who am i' comamnd?
who am i | cut -f1 -d' '


UNIX INTERVIEW QUESTIONS ON FIND COMMAND

Find utility is used for searching files using the directory information.

1. Write a command to search for the file 'test' in the current directory?
find -name test -type f

2. Write a command to search for the file 'temp' in '/usr' directory?
find /usr -name temp -type f

3. Write a command to search for zero byte size files in the current directory?
find -size 0 -type f

4. Write a command to list the files that are accessed 5 days ago in the current directory?
find -atime 5 -type f

5. Write a command to list the files that were modified 5 days ago in the current directory?
find -mtime 5 -type f

6. Write a command to search for the files in the current directory which are not owned by any user in the /etc/passwd file?
find . -nouser -type f

7. Write a command to search for the files in '/usr' directory that start with 'te'?
find /usr -name 'te*' -type f

8. Write a command to search for the files that start with 'te' in the current directory and then display the contents of the file?
find . -name 'te*' -type f -exec cat {} \;

9. Write a command to list the files whose status is changed 5 days ago in the current directory?
find -ctime 5 -type f

10. Write a command to list the files in '/usr' directory that start with 'ch' and then display the number of lines in each file?
find /usr -name 'ch*' -type f -exec wc -l {} \;


TOP UNIX INTERVIEW QUESTIONS - PART 1

1. How to display the 10th line of a file?
head -10 filename | tail -1

2. How to remove the header from a file?
sed -i '1 d' filename

3. How to remove the footer from a file?
sed -i '$ d' filename

4. Write a command to find the length of a line in a file?

The below command can be used to get a line from a file.
sed –n '<n> p' filename

We will see how to find the length of 10th line in a file
sed -n '10 p' filename|wc -c

5. How to get the nth word of a line in Unix?
cut –f<n> -d' '

6. How to reverse a string in unix?
echo "java" | rev

7. How to get the last word from a line in Unix file?
echo "unix is good" | rev | cut -f1 -d' ' | rev

8. How to replace the n-th line in a file with a new line in Unix?
sed -i'' '10 d' filename      # d stands for delete
sed -i'' '10 i new inserted line' filename    # i stands for insert

9. How to check if the last command was successful in Unix?
echo $?

10. Write command to list all the links from a directory?
ls -lrt | grep "^l"

11. How will you find which operating system your system is running on in UNIX?
uname -a

12. Create a read-only file in your home directory?
touch file; chmod 400 file

13. How do you see command line history in UNIX?

The 'history' command can be used to get the list of commands that we are executed.

14. How to display the first 20 lines of a file?

By default, the head command displays the first 10 lines from a file. If we change the option of head, then we can display as many lines as we want.
head -20 filename

An alternative solution is using the sed command
sed '21,$ d' filename

The d option here deletes the lines from 21 to the end of the file

15. Write a command to print the last line of a file?

The tail command can be used to display the last lines from a file.
tail -1 filename

Alternative solutions are:
sed -n '$ p' filename
awk 'END{print $0}' filename


TOP UNIX INTERVIEW QUESTIONS - PART 2

1. How do you rename the files in a directory with _new as suffix?
ls -lrt|grep '^-'| awk '{print "mv "$9" "$9".new"}' | sh

2. Write a command to convert a string from lower case to upper case?
echo "apple" | tr [a-z] [A-Z]

3. Write a command to convert a string to Initcap.
echo apple | awk '{print toupper(substr($1,1,1)) tolower(substr($1,2))}'

4. Write a command to redirect the output of date command to multiple files?

The tee command writes the output to multiple files and also displays the output on the terminal.
date | tee -a file1 file2 file3

5. How do you list the hidden files in current directory?
ls -a | grep '^\.'

6. List out some of the Hot Keys available in bash shell? 
  • Ctrl+l - Clears the Screen.
  • Ctrl+r - Does a search in previously given commands in shell.
  • Ctrl+u - Clears the typing before the hotkey.
  • Ctrl+a - Places cursor at the beginning of the command at shell.
  • Ctrl+e - Places cursor at the end of the command at shell.
  • Ctrl+d - Kills the shell.
  • Ctrl+z - Places the currently running process into background.

7. How do you make an existing file empty?
cat /dev/null >  filename

8. How do you remove the first number on 10th line in file?
sed '10 s/[0-9][0-9]*//' < filename

9. What is the difference between join -v and join -a?
join -v : outputs only matched lines between two files.
join -a : In addition to the matched lines, this will output unmatched lines also.

10. How do you display from the 5th character to the end of the line from a file?
cut -c 5- filename


TOP UNIX INTERVIEW QUESTIONS - PART 3

1. Display all the files in current directory sorted by size?
ls -l | grep '^-' | awk '{print $5,$9}' |sort -n|awk '{print $2}'

2. Write a command to search for the file 'map' in the current directory?
find -name map -type f

3. How to display the first 10 characters from each line of a file?
cut -c -10 filename

4. Write a command to remove the first number on all lines that start with "@"?
sed '\,^@, s/[0-9][0-9]*//' < filename

5. How to print the file names in a directory that has the word "term"?
grep -l term *

The '-l' option make the grep command to print only the filename without printing the content of the file. As soon as the grep command finds the pattern in a file, it prints the pattern and stops searching other lines in the file.

6. How to run awk command specified in a file?
awk -f filename

7. How do you display the calendar for the month march in the year 1985?

The cal command can be used to display the current month calendar. You can pass the month and year as arguments to display the required year, month combination calendar.
cal 03 1985

This will display the calendar for the March month and year 1985.

8. Write a command to find the total number of lines in a file?
wc -l filename

Other ways to print the total number of lines are
awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' filename
awk 'END{print NR}' filename

9. How to duplicate empty lines in a file?
sed '/^$/ p' < filename

10. Explain iostat, vmstat and netstat?
  • Iostat: reports on terminal, disk and tape I/O activity.
  • Vmstat: reports on virtual memory statistics for processes, disk, tape and CPU activity.
  • Netstat: reports on the contents of network data structures.



TOP UNIX INTERVIEW QUESTIONS - PART 4

1. How do you write the contents of 3 files into a single file?
cat file1 file2 file3 > file

2. How to display the fields in a text file in reverse order?
awk 'BEGIN {ORS=""} { for(i=NF;i>0;i--) print $i," "; print "\n"}' filename

3. Write a command to find the sum of bytes (size of file) of all files in a directory.
ls -l | grep '^-'| awk 'BEGIN {sum=0} {sum = sum + $5} END {print sum}'

4. Write a command to print the lines which end with the word "end"?
grep 'end$' filename

The '$' symbol specifies the grep command to search for the pattern at the end of the line.

5. Write a command to select only those lines containing "july" as a whole word?
grep -w july filename

The '-w' option makes the grep command to search for exact whole words. If the specified pattern is found in a string, then it is not considered as a whole word. For example: In the string "mikejulymak", the pattern "july" is found. However "july" is not a whole word in that string.

6. How to remove the first 10 lines from a file?
sed '1,10 d' < filename

7. Write a command to duplicate each line in a file?
sed 'p' < filename

8. How to extract the username from 'who am i' comamnd?
who am i | cut -f1 -d' '

9. Write a command to list the files in '/usr' directory that start with 'ch' and then display the number of lines in each file?
wc -l /usr/ch*

Another way is 
find /usr -name 'ch*' -type f -exec wc -l {} \;

10. How to remove blank lines in a file ?
grep -v ‘^$’ filename > new_filename

TOP UNIX INTERVIEW QUESTIONS - PART 5

1. How to display the processes that were run by your user name ?
ps -aef | grep <user_name>

2. Write a command to display all the files recursively with path under current directory?
find . -depth -print

3. Display zero byte size files in the current directory?
find -size 0 -type f

4. Write a command to display the third and fifth character from each line of a file?
cut -c 3,5 filename

5. Write a command to print the fields from 10th to the end of the line. The fields in the line are delimited by a comma?
cut -d',' -f10- filename

6. How to replace the word "Gun" with "Pen" in the first 100 lines of a file?
sed '1,00 s/Gun/Pen/' < filename

7. Write a Unix command to display the lines in a file that do not contain the word "RAM"?
grep -v RAM filename

The '-v' option tells the grep to print the lines that do not contain the specified pattern.

8. How to print the squares of numbers from 1 to 10 using awk command
awk 'BEGIN { for(i=1;i<=10;i++) {print "square of",i,"is",i*i;}}'

9. Write a command to display the files in the directory by file size?
ls -l | grep '^-' |sort -nr -k 5

10. How to find out the usage of the CPU by the processes?

The top utility can be used to display the CPU usage by the processes.

TOP UNIX INTERVIEW QUESTIONS - PART 6

1. Write a command to remove the prefix of the string ending with '/'.

The basename utility deletes any prefix ending in /. The usage is mentioned below:
basename /usr/local/bin/file 

This will display only file

2. How to display zero byte size files?
ls -l | grep '^-' | awk '/^-/ {if ($5 !=0 ) print $9 }'

3. How to replace the second occurrence of the word "bat" with "ball" in a file?
sed 's/bat/ball/2' < filename

4. How to remove all the occurrences of the word "jhon" except the first one in a line with in the entire file?
sed 's/jhon//2g' < filename

5. How to replace the word "lite" with "light" from 100th line to last line in a file?
sed '100,$ s/lite/light/' < filename

6. How to list the files that are accessed 5 days ago in the current directory?
find -atime 5 -type f

7. How to list the files that were modified 5 days ago in the current directory?
find -mtime 5 -type f

8. How to list the files whose status is changed 5 days ago in the current directory?
find -ctime 5 -type f

9. How to replace the character '/' with ',' in a file?
sed 's/\//,/' < filename
sed 's|/|,|' < filename

10. Write a command to find the number of files in a directory.
ls -l|grep '^-'|wc -l

TOP UNIX INTERVIEW QUESTIONS - PART 7

1. Write a command to display your name 100 times.
The Yes utility can be used to repeatedly output a line with the specified string or 'y'.
yes <your_name> | head -100

2. Write a command to display the first 10 characters from each line of a file?
cut -c -10 filename

3. The fields in each line are delimited by comma. Write a command to display third field from each line of a file?
cut -d',' -f2 filename

4. Write a command to print the fields from 10 to 20 from each line of a file?
cut -d',' -f10-20 filename

5. Write a command to print the first 5 fields from each line?
cut -d',' -f-5 filename

6. By default the cut command displays the entire line if there is no delimiter in it. Which cut option is used to suppress these kind of lines?

The -s option is used to suppress the lines that do not contain the delimiter.

7. Write a command to replace the word "bad" with "good" in file?
sed s/bad/good/ < filename

8. Write a command to replace the word "bad" with "good" globally in a file?
sed s/bad/good/g < filename

9. Write a command to replace the word "apple" with "(apple)" in a file?
sed s/apple/(&)/ < filename

10. Write a command to switch the two consecutive words "apple" and "mango" in a file?
sed 's/\(apple\) \(mango\)/\2 \1/' < filename

11. Write a command to display the characters from 10 to 20 from each line of a file?
cut -c 10-20 filename
 

TOP UNIX INTERVIEW QUESTIONS - PART 8

1. Write a command to print the lines that has the the pattern "july" in all the files in a particular directory?
grep july *

This will print all the lines in all files that contain the word “july” along with the file name. If any of the files contain words like "JULY" or "July", the above command would not print those lines.

2. Write a command to print the lines that has the word "july" in all the files in a directory and also suppress the file name in the output.
grep -h july *

3. Write a command to print the lines that has the word "july" while ignoring the case.
grep -i july *

The option i make the grep command to treat the pattern as case insensitive.

4. When you use a single file as input to the grep command to search for a pattern, it won't print the filename in the output. Now write a grep command to print the file name in the output without using the '-H' option.
grep pattern file name /dev/null

The /dev/null or null device is special file that discards the data written to it. So, the /dev/null is always an empty file.
Another way to print the file name is using the '-H' option. The grep command for this is
grep -H pattern filename

5. Write a command to print the file names in a directory that does not contain the word "july"?
grep -L july *

The '-L' option makes the grep command to print the file names that do not contain the specified pattern.

6. Write a command to print the line numbers along with the line that has the word "july"?
grep -n july filename

The '-n' option is used to print the line numbers in a file. The line numbers start from 1

7. Write a command to print the lines that starts with the word "start"?
grep '^start' filename

The '^' symbol specifies the grep command to search for the pattern at the start of the line.

8. In the text file, some lines are delimited by colon and some are delimited by space. Write a command to print the third field of each line.
awk '{ if( $0 ~ /:/ ) { FS=":"; } else { FS =" "; } print $3 }' filename

9. Write a command to print the line number before each line?
awk '{print NR, $0}' filename

10. Write a command to print the second and third line of a file without using NR.
awk 'BEGIN {RS="";FS="\n"} {print $2,$3}' filename

11. How to create an alias for the complex command and remove the alias?

The alias utility is used to create the alias for a command. The below command creates alias for ps -aef command.
alias pg='ps -aef'

If you use pg, it will work the same way as ps -aef.

To remove the alias simply use the unalias command as
unalias pg

12. Write a command to display today's date in the format of 'yyyy-mm-dd'?

The date command can be used to display today's date with time
date '+%Y-%m-%d'
Related Posts Plugin for WordPress, Blogger...