How to create a large file using shell script, To test with Hadoop Map Reduce processing

1. Open new Terminal.


2. Create 'myscript.sh'  file using below command:

touch myscipt.sh


3. Create 'smallfile.txt' file using below command:

touch smallfile.txt


4. Open the file using below command:

gedit smallfile.txt


5. Enter some sample data into the file.

"
I am going to hyderabad
I am learning hadoop course
"

6. Save the file using command: Ctrl + S


7. Now prepare a large file using below steps.


8. Open  the file using below command:

gedit myscript.sh


9.  Add the below script into the file.

#!/bin/bash
for i in {1..1000}
do
   cat smallfile.txt >> largefile.txt
done


10. Save the file using command: Ctrl + S


11. Provide execute permission to script file using below command:

chmod +x myscript.sh


12. Run the script file to generate a large file using below command:

./myscript.sh


13. Verify the 'largefile.txt' in current folder.


14. Run the above command multiple times to get more large file.


15. Try "hadoop fs -put " commad to place into HDFS.


16. Run Mapreduce programs using above file in step15.