Basic Unix Commands Developer Should Know

1. Java Installation and setting Java Environment variables.
sudo apt-get update
sudo apt-get install oracle-java8-installer
sudo apt install oracle-java8-set-default
sudo -H gedit /etc/environment
add below lines:
JAVA_HOME="/usr/lib/jvm/java-8-oracle"
JRE_HOME="/usr/lib/jvm/java-8-oracle/jre"

2. Maven Installation
cd /home/phani/Softwares/
wget http://www-eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
sudo tar -xvzf apache-maven-3.3.9-bin.tar.gz
sudo mv apache-maven-3.3.9 maven  =>rename
cd /etc/profile.d/
sudo gedit maven-env.sh
export MAVEN_HOME=/home/phani/Softwares/maven
export M2_HOME=/home/phani/Softwares/maven
export PATH=$PATH:$M2_HOME/bin

Commands:
$ cat filename
$ cat > file1.txt => creat file and append text and Ctrl+d
$ cat sample1.txt sample2.txt > sample3.txt => concatenate these two files and can save to another file named sample3.txt 
$ cat -n file1.txt   =>line number
$ cat file2.txt> file1.txt => To copy the content of one file to another file, you can use the greater than ‘>’ symbol with the cat command.
$ cat sample1.txt >> sample2.txt =>To append the contents of one file to another, you can use the double greater than ‘>>’ symbol with the cat command.
$gedit filename =>opens in text pad

cp /tmp/mozilla_phani0/eclipse-jee-helios-SR1-linux-gtk-x86_64.tar.gz .
tar xvf file.tar
x = eXtract, this indicated an extraction
v= verbose (optional) the files with relative locations will be displayed. 
f = from/to file ... (what is next after the f is the archive file) 

Remove files/folders
rm -rf /path/to/directory/*
-f - stands for "force" which is helpful when you don't want to be asked/prompted if you want to remove an archive, for example.
-r - stands for "recursive" which means that you want to go recursively down every folder and remove everything.

Duplicate copy folder
cp eclipse eclipse_bk

Creating Link (shortcut desktop)
ln -s source_file destination_path


Open Shell:  xterm

echo $BASH
command,argument , options
ls -a list all files incuding hiden
ls -l list files in long format
ls -l-a combine both a and l options
ls -la combine both a and l options
ls -la /bin list all in /bin in long format

Order of options does not matter
The file name has to be at the end

Getting Help
Manual Pages => command: man
man ls
man cd
man man

Using the manual pages
use space to move down a page
move back a page with 'b'
search with '/'
exit with 'q'

drwxr
d -directory

History
ctrl+p previous
ctrl+n next

File Management
less FileName
reset =>used for reset the terminal
file fileName =>used to get the file information
file * => getting all files and folder information
mv =>move
rm -r images  =>remove folder and its content recrusively
touch =>creating empty file

cat 
 best for small files -no paging
 may mess up your terminal
 use reset to fix terminal
less 
  pager with lots of features
  see 'man less'
  use space to move down a page
  move back a page with b
  search with '/'
  exit with 'q'

Call the correct program directly
firefox index.html
opera index.html
you have to know what is installed on your system

File names
 filenames can contain just about anything
 except /
 hidden files start with a dot

Case sensitivity

Extensions(.exe,.zip) are optional
file =>command show types

Absolute paths
 start with /
 relative to the root
 /
 /bin/bash

Relative paths
 don't start with a/
 resolved relative to current working direcctory
 /Users/reinder/library

ls -R
cp source target
cp a dir/b
cp a b c dir
cp dir/a .
cp * dir

Copying directories
 Use cp with the -R switch
  Copies everything in the directory recursively
  cp -R  source_dir target_dir
  cp -R dir1 dir2 dir3 target_dir
  cp -R dir1 file1 dir2 file2 target_dir

ls -R folder =>show the nested files in the foler 

moving files
 mv a b =>rename
 mv a dir/b
 use it to move directories as well
 mv a b c dir
 mv * dir

Deleting Files
 rm =>delete files perminently
 rm a
 rm a b c
 rm dir/a dir/b
 rm *

rmdir 
 will remove empty directories only
rm -r
 will recursively remove all the directories in the folder

safety first: the -i switch
 prompt before overwriting or deleting files
 use the -i switch
 cp -i and mv -i =>will ask you before overwriting files
 rm -i will ask before deletion

Combine it with other options
 cp -Ri
 rm -ri

Wildcards
 ls a*
 ls *at*
 ls -d m*n =>only directory
 ? => 1 character match
 [acd7_]
  Matches one of the characters in the list
  Above would match a,c,d,7 or _
  [^ax2] matches anything but a,x,2
  range:[a-z],[0-9],[A-C3-5] 

rm important_document-v[2-4].doc
rm important_document-v1[^789].docs
rm important_document-v?.docs

Brace Expansion
 Generate Strings
  Does not have to match existing filenames
  Syntax: pre{list,of,strings}post
  touch {a,b,c}.txt => touch a.txt b.txt c.txt
  mv file.{txt,jpg} dir/
  touch {a..c}{1..3}.txt =>touch a1.txt a2.txt...c3.txt
  touch file{ab,b,c}{1,2,3}.{jpg,txt,zip}
  
Brace expansion comes before wildcard expression
 mv *{txt,jpg} Documents => mv *txt *jpg Documents
 mv filea?.{jpg,txt}a  => mv filea?.jpg filea?.txt a

Output Redirection
 Redirecting standard output stream
 > 
  saves the output of a command to a file
  ls > listing.txt
  cat> story.txt =>ctrl+D to save
  This will overwrite existing files
 >>
  Appends the output to the end of a file
  echo "buy milk" >> shopping.txt =>append by mil to shopping.txt end of the file

PipeLine
  cmd1 | cmd2 => output of cmd1 pass to input to cmd2 


grep 1978 oscars.tsv
grep 1978 oscars.tsv | sort
grep 1978 oscars.tsv | sort > 1978_films.txt
cut -f 3 oscars.tsv  => cut is used to sect specific column
cut -f 3 oscars.tsv | grep 4
cut -f 3 oscars.tsv | grep 4 | wc -l  => wc-word count -l lines

Command substitution
 Replace a command with its output
  Ouput will become a part of the command line
  Put command between $()
  echo "hello,$(whoami)"
  echo "Buy milk">"notes$(date).txt"

 Note the use of double quotes
  Keep command substitution intact
 Older form uses backticks
  echo "you are currently on 'hostname'"
  echo "you are currently on $(hostname)"
  
Terminal and the Command Line
 Bash: text in, text out
 Terminal
    Handles key pressed
    Draws text
    Fonts
    Colors
    Scrolling
    Copy/past

copy/past => select the text click on the middle mouse it will past the selected text

Movement Keys:
ctrl-a start of line
ctrl-e end of the line
ctrl-f forward 1 char ->right arrow
ctrl-b back 1 char ->left arrow
alt-f forward 1 word -> command-Left
alt-b back 1 word ->command right

Deletion
Ctrl-D delete a char de
ctrl-H deelete a char backword
alt-D delete a word
Ctrl-w deelete a world backword ->Alt Backspace
ctrl-K delete reset of line
Ctrl-U delete from start of line

Ctrl-C break => end a running program
Ctrl-D end of transmission cat > x
ctrl-R search back in history

Editors
 Nano
  Tiny,simple,fast
 vi (vim)
  vi fileName => Press I to insert Esc for command mode=> :wq! to save and exit
  :w save
  :q to exit
  :q! exit without saving  
  pre-installed on linux

 Emacs
  Very full featured
  Pre installed on some linux distributors


Sort:
  sort -k2 math_grades   => -k is key column name ex: k2 second comumn
  sort -nk2 match_grades
  sort -rnk2 match_grades =>r reverse, n number sort
  sort studentlist | uniq -c
  sort studentlist | uniq -c |sort -nr
  -r reverse sort
  -n sorts numerically
  -k sorts by fields
    sort -k2
    space-seperated fields by default

  filter out repeated lines: uniq
     Sort attendance | uniq
     -c counts lines

Head & Tail:
  Head:
   Show first 10 lines of input by default
   -n gives number of lines
   head -n 1
  Tail:
   Show last 10 lines of input by default
   -n gives number of lines
   tail -n 1
   tail -f follows newly appended data

 ls -lS | head    => first 10 lines
 ls -lS | head -n 1  => first line
 ls -lrS | tail -n 2 =>end of the input use tail , S-size of file
 ls -lrS | tail -n 2| tail -n 1
 tail auth.log =>last couple of entries on a file
 tail -f auth.log => f-follow option

wc story.txt =>word count , 1st result no.of lines, no.of words, no.of bites
wc -l story.txt => -l lines
ls | wc -l
ls -a | wc -l
grep hello story.txt
grep steve *grades
grep -i steve *grades => -i ignore case
grep -v lecture math_attendance | sort |uniq => -v rule
grep -Ev "^$" math_attaendance | sort | uniq => -Ev extended regular expression
ls -l | grep  -v "^d"

Grep:
 grep searchs for a text in a file
 Or multiple files. Ex: grep string*
 -i makes search case-insensitive
 -c counts occurences
 -I shows line number of occurences
 -v inverts the search (filter)

Can use regular expressions

find /usr
find /usr -name emacs
find . -name '*.txt'
find . -name '*.txt' -exec grep -l curious {} \; => -l file name having curious

Find:
 powerful tool for searching files
 find dir - lists all files in dir
 find dir -name f
  Lists all files named f in dir
  Match Expression: find dir -name '*txt'

for More: http://goo.gl/Rid5

Search and Replace text (tr only reads from standard input)
tr S s physics_grades
cat physics_grades | tr S s 
tr S s < physics_grades
Seperating date with semicolon in text file
 grep \; oscars.tsv 
 tr \\t  \; < oscars.tsv > oscars.csv

Advanced tools
 sed
  Stream editor
  Transform text
  Replace words
  Most common use: to replace "old" with "new"  
  send 's/old/new/g'

 awk
  complete programming language
  very useful for column-oriented files
 perl
 python

sed 's/curiouser/stranger/g' demo/alice > alice2
awk '{ total += s2 } END {print total/NR}' math_grades

Processing data:
 sort -nk2 -t\; oscars.cst
 sort -nk2 -t\; oscars.cst | head
 cut -f  2 -d\; oscars.cst  => -d delimeter ,-f field
 cut -f  2,3 -d\; oscars.cst  => -d delimeter ,-f field 
 paste *grades 
 sort physics_grades > phys_sorted
 tr S s < physics_grades > physics_grades2
 paste math_sorted phy_sorted
 join math_sorted phys_sorted

Column-based date
 sort -k to sort on a specific column
   use -t to specify delimiter
 cut to select a column from the input
   cut -f 2 shows only second column
   -d for delimiter
 paste
  puts lines of input files next to each other in the output
 join
   joins input files based on matching keys



Editing text files => Nano, vi, emacs
Sorting => sort, uniq
Head and tail
Searching and filtering => grep
Replacing text => tr,sd
Finding files => find
Advanced tools => awk,sd,perl,python
Column-based files => cut,paste,join
Input redirection with <
Multiple commands on one line => ;



Jobs and Process
 cp /volumes/data-2T/Movie.avi .
 fg
  cp /volumes/data-2T/Movie.avi .
 bg
  cp /volumes/data-2T/Movie.avi .
  
 ./delayed_ls &  => & means run backround

 ./delayed_ls > output_file & 

 rm -i oscars.tsv & 
 fg
  rm -i oscar.tsv

Job Controls: 
 ^Z suspend a running job
 fg => send job to foreground
 bg => send job to background
 Background Jobs
  Use & at end of line
  Cannot read input from the user
  If the program tries, bash will suspend it and let you know
  Tip: only use wiht programs that don't need user interaction
  Output willl mess up your screen
  Tip: redirect output to file
  find . > all_files &

./delayed_ls &
[1] 23837 => first one is jobid and second is process id

Killing:
 Foreground program: ^C
 End any program with kill
  Can only end processes you own
 Kill by job id:
   kill %2
   kill %cp
   (fg and bg work with job id's as well)
 Kill by process id
  kill 6543
  hard kill with -KILL:
   kill -KILL 6543
  pkill => match process by part of name
  

  jobs => shows list of jobs
  kill %job_id
  ps -e => gives all list of running processses
  kill process_no
  pkill =>dangerious 

Inspecting Processes:
 Jobs => shows bash jobs for current shell
 ps
  display processes running under current shell
  
 Displaying all processes
  ps -e
 Include owner
  ps -ef

Process memory and CPU usage:
 top => show all top running processsor
 press ut  then phani
 kki 


Background jobs => &, ^Z , bg,fg
Inspect processes => jobs,ps,top
End processes => kill, xkill, pkill

Bash Startup Files
 .profile => loaded for login shell
 .bashrc => loaded for non-login shell
 Tip: read .bashrc from .profile => source ~/.bashrc

  echo $PATH
  PATH = "$PATH:~/bin" => always append at end
  export EDITOR="nano" => Export variables that are needed in subprocess