Unix shell intro

README.md

Intro

• shell takes commands and passes to the operating system
• most popular shell is bash (Bourne Again Shell): a replacement of sh, the origianl Unix shell, which was written by Steve Bourne
• motivating examples:

# Copies all HTML files to `destination`,
# but only those that do not exist or are newer
cp -u *.html destination
# Finds the ten most common words in a text file
curl http://composingprograms.com/shakespeare.txt | \
tr -d '[[:punct:]]' | tr ' ' '\n' | sort | uniq -c | sort -nr | head
# Finds all the function definitions in a Python file 
find . -name '*.py' -exec grep -H '^def ' {} \;

• explain shell
• appreciate geeky jokes: make me a sandwich

who | grep -i blonde | date; cd ~; unzip; touch; strip; finger; mount; gasp; yes; uptime; umount; sleep

• powerful suite of commands; examples: man cal date free exit

Navigation and system exploration

Navigation.

• cd change directory

• ls lists directory contents

• pwd prints working directory

• files are organized in a tree

• / root directory

• . current directory

• .. parrent directory

• ~ home directory

• ~username username's home directory

Some important folders:

• / root
• /bin programs, common binaries
• /etc system configurations
• /home users' home
• /tmp, /var temporary files
• /usr user-installed programs

Exploring the system.

• ls lists direcotry contents

• file determines file type

• less view file contents

• Many of these commands accept arguments or parameters

• Options can be combined: ls -l -a -r can be written as ls -lar

• Help --help or -help or -h

Manipulating files and directories

• cp copies files and directories
• mv moves files and directories
• mkdir creates directories
• rm removes files and directories
• touch create a new empty file or updates its last-modified time stamp
• ln create hard and symbolic links

Warning! The above command do not prompt for confirmation It makes it easy to overwrite/delete a file; use the -i flag to overwrite this setting.

Flags:

• -i, --interative
• -r, --recursive
• -u, --update

Wildcards:

Wildcard	Matches
*	any character
?	any single character
[characters]	any
[!characters]	any
[[:class:]]	any character that is a member of the specified class

Classes are:

• [:alnum:]
• [:alpha:]
• [:digit:]
• [:lower:]
• [:upper:]

Redirection

Get images from here

Standard input, output, error.

• Redirecting standard output > and >>

ls -l /usr/bin
ls -l /usr/bin > ls-output.txt
ls -l ls-output.txt
less ls-output.txt
ls -l /bin/usr > ls-output.txt
> ls-output.txt
ls -l /usr/bin >> ls-output.txt
ls -l /usr/bin >> ls-output.txt

• Redirecting standard error 1> stdout and 2> stderr:

ls -l /bin/usr 2> ls-error.txt

• Redirecting both standard output and standard error to one file &>

ls -l /bin/usr &> ls-output.txt

• cat concatenate files

• head outputs the first part of a file

• tail outputs the last part of a file

• sort sort lines of text

• uniq report or omit repeated lines

• wc counts newlines, words and bytes for each file

• tee read from standard input and write to standard output and files

• grep prints lines matching a pattern

• sort sorts lines of text

• find finds files in a given directory

• regex101: Online regex tester and debugger

• regexone: Interactive tutorial for regex

Pipelines.

command1 | command2

• Dissecting the motivating example:

curl http://composingprograms.com/shakespeare.txt \
| tr -d '[[:punct:]]' | tr ' ' '\n' | sort \
| uniq -c | sort -nr | head

Unix philosophy:

• Write programs that do one thing and do it well.
• Write programs to work together.
• Write programs to handle text streams, because that is a universal interface.

Keyboard shortcuts and miscellany

• clear clears the screen

• history display the contents of the history list

• My favorite: ALT+. -- place the argument of the most recent command on the shell

mv a{,.bk}
!!  # execute last command
sudo !!

Access Rights and Intro to Processes

File system access rights

• In your directory, type ls -l (for long listing)
• You will see that you now get lots of details about the contents of your directory, similar to the example below:

-rwxrw-r-- 1 marius speed   5538367 Apr 21 11:35 'LectureX.pdf'

Each file (and directory) has associated access rights: in the left-hand column is a 10 symbol string consisting of the symbols d, r, w, x, -, and, occasionally, s or S:

• If d is present, it will be at the left hand end of the string, and indicates a directory: otherwise - will be the starting symbol of the string. The 9 remaining symbols indicate the permissions, or access rights, and are taken as three groups of 3.
• The left group of 3 gives the file permissions for the user that owns the file (or directory) (marius in the above example);
• The middle group gives the permissions for the group of people to whom the file (or directory) belongs (speed in the above example);
• The rightmost group gives the permissions for all others.

The symbols r, w, etc., have slightly different meanings depending on whether they refer to a simple file or to a directory.

Access rights on files:

• r (or -), indicates read permission (or otherwise), that is, the presence or absence of permission to read and copy the file;
• w (or -), indicates write permission (or otherwise), that is, the permission (or otherwise) to change a file;
• x (or -), indicates execution permission (or otherwise), that is, the permission to execute a file, where appropriate

Access rights on folders

• r allows users to list files in the directory;
• w means that users may delete files from the directory or move files into it;
• x means the right to access files in the directory. This implies that you may read files in the directory provided you have read permission on the individual files.

Example:

-rw-------
#  means a file that only the owner can read and write, no-one else can read or write and no-one has execution rights (e.g. a mailbox file).

Changing access rights

chmod: changing a file mode. Only the owner of a file can use chmod to change the permissions of a file.

• Symbols: u user, g group, o other, a all, r read, w write (and delete), x execute (and access directory), + add permission, - take away permission.

Example:

chmod go-rwx some_file
# removes read write and execute permissions on the file for the group and others. It will leave the other permissions unaffected.
chmod a+rw some_file
# gives read and write permissions on the file to all.

Processes and Jobs

A process is an executing program identified by a unique PID (process identifier). To see information about your processes, with their associated PID and status, type ps.

A process may be

• in the foreground,
• in the background, or
• be suspended.

In general the shell does not return the UNIX prompt until the current process has finished executing. Some processes take a long time to run and hold up the terminal. Backgrounding a long process has the effect that the UNIX prompt is returned immediately, and other tasks can be carried out while the original process continues executing.

Running background processes

To background a process, type an & at the end of the command line. For example, the command sleep waits a given number of seconds before continuing.

sleep 10
# will wait 10 seconds before returning the command prompt. Until the command prompt is returned, you can do nothing except wait.

sleep 10 & 
[1] 27349
# will run sleep in the background.
# the sleep command is typed in by the user; the next line, indicating job number and PID, is returned by the machine.  

The & runs the job in the background and returns the prompt straight away, allowing you do run other programs while waiting for that one to finish.

The user is returned a job number (numbered from 1) enclosed in square brackets, together with a PID and is notified when a background process is finished. Backgrounding is useful for jobs which will take a long time to complete (e.g., training a ML system).

Backgrounding a current foreground process

• Type sleep 1000
• You can suspend the process running in the foreground by typing ^Z, i.e., hold down the [Ctrl] key and type [z]. Then to put it in the background, type bg.

Listing suspended and background processes

When a process is running, backgrounded or suspended, it will be entered onto a list along with a job number. To examine this list, type jobs. An example output could be

[1] Suspended sleep 1000
[2] Running firefox
[3] Running matlab

To restart (foreground) a suspended processes:

# fg %jobnumber

fg 1 
# will bring sleep 1000 back
# typing fg w/o a job number foregrounds the last suspended process

Killing a process

kill: terminate or signal a process.

It is sometimes necessary to kill a process (for example, when an executing program is in an infinite loop). To kill a job running in the foreground, type ^C.

kill jobnumber
# kills a suspended or background process 

sleep 100 &
jobs
# say it is job number 2, for example
kill 2 

ps: process status

Alternatively, processes can be killed by finding their process numbers (PIDs) and using kill PID_number.

sleep 100 & 
ps 

PID TT S TIME COMMAND
20077 pts/5 S 0:05 sleep 100
21563 pts/5 T 0:00 firefox
21873 pts/5 S 0:25 gedit

# let's kill the sleep 100 process 
kill 20077 
# and check if it worked
ps 

If a process refuses to be killed, use the -9 option: kill -9 20077.

Optional

Play with htop: https://codeahoy.com/2017/01/20/hhtop-explained-visually/

Further resources

• CSE391: System and Software Tools
• Commandline Fu: sorted by votes
• The Linux Command Line: A Complete Introduction
• CS61a – Unix Tutorial