Basic shell concept and shell script skills.

1. Linux Shell
&
Shell Script
netman<netman@study-area.org>

2. Part I
Command Line

3. System Layer
O/S
hardware
software
user

4. Interface
kernel
hardware
shell

5. Shell Definition
● Command interpreter:
● Issue commands from user to system
● Display command results from system to user

6. Shell Type
● sh:
● Burne Shell (sh)
● Burne Again Shell (bash)
● csh:
● C Shell (csh)
● TC Shell (tcsh)
● Korn Shell (ksh)
● /etc/shells

7. Shell Prompt
● Function:
Telling the user:
You can type command now!
● Type:
● Super User: #
● Regular User: $ or >

8. Carriage Return (CR)
● Function:
Telling the system:
You can run command now!
● Generated by:
<Enter>

9. Command Line
Everything typed between Shell
Prompt and Carriage Return.

10. Command Line Components
● A Command (must present):
What to do?
● Options (zero or more):
How to do?
●Arguments (zero or more):
Which to do with?

11. Internal Field Separator (IFS)
● Function:
To separate command line components.
● Speak in general:
To cut a command line into words(fields).
● Generated by:
● <Space>
● <Tab>
● <Enter> (*note: CR also)

12. A Command Line Format
Command<IFS>[Options...]<IFS>[Arguments...]

13. Option Format
● Preceding Character:
­
+
●Full Format:
Starting with ­­
● Short Format:
Starting with ­
Combinable

14. Option Example
● Find the difference:
ls ­a ­l l
ls ­al l
ls ­all
ls ­­all
ls ­­ ­­all

15. A Simple Command: echo
● Function:
To display all arguments to STDOUT(screen),
plus an ending <newline> character.

16. A Simple Command: echo
● Major options:
­n : disable the trailing <newline>
­e : enable interpretation of escapes ()

17. Escaped Characters in echo
● Most Frequently Used:
backslash
b backspace
c produce no further output
n new line
r carriage return
t horizontal tab
v vertical tab
0NNN byte with octal value
xHH byte with hexadecimal value

18. Examples of echo
● Using ­n option:
$ echo first line
first line
$ echo ­n first line
first line $

19. Examples of echo
● Using escape character:
$ echo ­e "atbtcndtetf"
a b c
d e f

20. Examples of echo
● Using escape with octal value:
$ echo ­e
"141011142011143012144011145
011146"
a b c
d e f

21. Examples of echo
● Using escape with hex value:
$ echo ­e
"x61x09x62x09x63x0ax64x09x65
x09x66"
a b c
d e f

22. Part II
Quoting

23. Character Type in Command Line
● Literal Character:
Plain text, no function
123456 abcdefg …
● Meta Character:
Reserved with functions

24. Frequently Used Meta Characters
= ： set variable value
$ ： variable substitution
> ： redirect to STDOUT
< ： redirect from STDIN
| ： pipe line
& ： background running
() ： run commands in nested sub-shell
{} ： command grouping
; ： run commands in frequency
&& ： run command while TRUE
|| ： run command while FALSE
! ： re-run command in history

25. Quoting Usage
● Purpose:
Disable the functions of Meta Characters.

26. Quoting Method
● Escaping ( ):
Disable meta character following backward
slash by each.
● Example:
$
(
<newline>

27. Quoting Method
● Hard Quoting ( '' ):
Disable all meta characters within single
quotes.
● Example:
'...$...(...)...'

28. Quoting Method
● Soft Quoting ( ”” ):
Disable some meta characters within double
quotes.
● Example:
“...$...(...)...“

29. Exception in Soft Quoting
● Reserved functions:
$ : substitute
: escape
` : command substitute
! : history

30. Quoting Example
● Disable <IFS>:
$ A=B C # white space
$ C: command not found.
$ echo $A
$ A="B C"
$ echo $A
B C

31. Quoting Example
●Disable <CR>:
$ A='B
> C
> '
$ echo "$A"
B
C

32. Quoting Example
●Think about:
$ echo "$A"
B
C
$ echo $A
B C

33. Quoting Example
●Think about:
$ A=B C
$ echo '”$A”'
”$A”
$ echo “'$A'“
'B C'

34. Quoting Example
● Identical:
$ awk '{print $0}' 1.txt
$ awk "{print $0}" 1.txt
$ awk {print $0} 1.txt
$ A=0
$ awk "{print $$A}" 1.txt

35. Part III
Variable

36. Variable Setting
● Setting format:
name=value
● Setting rules:
● No space
● No number beginning
● No $ in name
● Case sensitive

37. Variable Setting
● Common errors:
A= B
1A=B
$A=B
a=B

38. Variable Substitution
● Substitute then re-construct:
$ A=ls
$ B=la
$ C=/tmp
$ $A ­$B $C
● Result:
$ ls ­la /tmp

39. Viewing a Variable Value
● Using the echo command:
$ echo $A ­$B $C
ls ­la /tmp

40. Value Expansion
● Using separator:
$ A=B:C:D
$ A=$A:E
● Using {}:
$ A=BCD
$ A=${A}E

41. The export command
● Purpose:
To set Environment Variable, inheritable in
sub shells.

42. Variable Exporting Method
● Using export command:
$ A=B
$ export A
Or
$ export A=B
● Using declare command:
$ declare ­x A
$ A=B

43. Variable Exportation Effect
A=B export A
Before exporting
After exporting

44. Viewing Environment Variable
● Using env command:
$ env
●Using export command:
$ export

45. Variable Revocation
● Using unset command:
$ A=B
$ B=C
$ unset $A
● Different to null value:
$ A=
$ unset A

46. Part IV
Sub Shell

47. Process Hierarchy
● Every command issues a process
when running.
● A command process is a child, and
the shell is the parent.
● Child process returns a value($?) to
parent when exists.
shell
command
Return
Value

48. Process Environment
● Child process inherit it's environment
from parent.
● However, any changing of
environment in a child will NEVER
effect the parent!

49. Shell Script
● Definition:
A serial of command lines defined in a text
file before we running them.

50. Script Running
● Using a sub shell to run commands:
Environment changing only effects the sub
shell.
● Interpreter is defined at the first line:
#!/path/to/shell
shell
sub shell
commands

51. Source Running
● Using source command to run script:
● There is no sub shell, interpreter is ignored.
● Environment changing effects the current
shell.
shell
commands

52. Exec Running
● Using exec command to run script:
● The current shell is terminated when script
starting.
● The process is hanged over to interpreter.
shell
commands
interpreter

53. Practice and Understand
● Write a shell script (my.sh):
pwd
cd /tmp
pwd
sleep 3
● Make it executable:
$ chmod +x my.sh

54. Practice and Understand
● Run the script in different ways:
$ ./my.sh
$ pwd
$ . ./my.sh
$ pwd
$ cd ­
$ exec ./my.sh
Identical to:
source ./my.sh

55. Part V
Command Grouping

56. Sequence Running
● Using the ; symbol:
$ cmd1 ; cmd2; cmd3
● Equivalent to:
$ cmd1
$ cmd2
$ cmd3

57. Command Grouping Method
● Using {} to run command group in
current shell:
$ { cmd1 ; cmd2; cmd3; }
● Using () to run command group in a
nested sub shell:
$ ( cmd1 ; cmd2; cmd3 )

58. Command Grouping Effect
● Using {} :
Environment changing effects the
current shell.
● Using ():
Environment changing does NOT
effects the current shell.

59. Named Command Group
● Also known as function
Command group is run when calling the
function name.
$ my_function () {
cmd1
cmd2
cmd3
}
$ my_function

60. Part VI
Shell Features

61. Command Substitution
● Using $() :
$ cmd1 … $(cmd2 …) …
● Using `` (don't be confused with ''):
$ cmd1 … `cmd2 …` …

62. Multiple Substitution
● Using $() :
$ cmd1 … $(cmd2 … $(cmd3 … ) … ) …
● Using `` :
$ cmd1 … `cmd2 … `cmd3 … ` … ` …

63. Advanced Variable Substitution
● ${#var} :
the length of value

64. Advanced Variable Substitution
● ${var#pattern} :
remove shortest pattern at beginning
● ${var##pattern} :
remove longest pattern at beginning
● ${var%pattern} :
remove shortest pattern at end
● ${var%%pattern} :
remove longest pattern at end

65. Advanced Variable Substitution
● ${var:n:m} :
since position n to have m characters,
(position starting from 0)

66. Advanced Variable Substitution
● ${var/pattern/str} :
substitute the first pattern to str
● ${var//pattern/str} :
substitute all pattern to str

67. Array Setting
● Using ():
array=(value1 value2 value3)
● Position assigning:
array[0]=value1
array[1]=value2
array[2]=value3

68. Array Substitution
● All values:
${array[@]}
${array[*]}
● Position values:
${array[0]}
${array[1]}
${array[2]}

69. Array Substitution
● Length of value:
${#array[0]}
● Number of value:
${#array[@]}

70. Arithmetic Expansion
● 32Bit integer:
-2,147,483,648 to 2,147,483,647
● No floating point ！
● Using external commands instead:
● bc
● awk
● perl

71. Arithmetic Operation
● Operators:
+ : add
­ : subtract
* : multiply
/ : divide
& : AND
| : OR
^ : XOR
! : NOT

72. Arithmetic Operation
●Using $(()) or $[]:
$ a=5;b=7;c=2
$ echo $((a+b*c))
19
$ echo $[(a+b)/c]
6

73. Arithmetic Operation
●Using declare with variable:
$ declare ­i d
$ d=(a+b)/c
$ echo $d
6

74. Arithmetic Operation
●Using let with variable:
$ let f=(a+b)/c
$ echo $f
6

75. Arithmetic Operation
●Using (()) with variable:
$ a=1
$ ((a++))
$ echo $a
2
$((a­=5))
$ echo $a
­3

76. Part VII
Positional Parameter

77. Script Parameter
● Assigned by command line:
script_name par1 par2 par3 ...
● Reset by set command:
set par1 par2 par3 ...
● Separated by <IFS> :
set “par1 par2” par3 ...

78. Parameter Gathering
● Substituted by $n (n=position) :
$0 $1 $2 $3 ...
● Position:
$0 : script_name itself
$1 : the 1st parameter
$2 : the 2nd parameter
and so on...

79. Parameter Substitution
● Reserved variables:
${nn} : position greater than 9
$# : the number of parameters
$@ or $* : All parameters individually
“$*” : All parameters in one
“$@” : All parameters with position reserved

80. Parameter Shifting
● Using shift [n]
to erase the first n parameters.

81. Function Parameter
● Function has own position except $0

82. Part VIII
Data Stream

83. File Descriptor (FD)
● Processes use File Descriptors to
input or output (I/O) data with system
● Each process has 256 FD
●The first 3 FD are standard
0 : Standard Input (STDIN)
1 : Standard Output (STDOUT)
2 : Standard Error Output (STDERR)

84. Standard FD
● By default, each standard FD is
connected to an I/O device:
STDIN : Keyboard
STDOUT : Screen
STDERR : Screen
processFD0
FD2
FD1
keyboard screen

85. IO Redirection
● Standard FD can be changed in
command line:
STDIN : < file
STDOUT : > file
STDERR : 2> file
processFD0
FD2
FD1
keyboard screen

86. IO Redirection Example
● Command default:
$ mail -s test root
this is a test mail.
please skip.
^d

87. IO Redirection Example
● Change the STDIN by using < :
$ mail -s test root < /etc/passwd

88. IO Redirection Example
● Change the STDOUT by using > :
$ cat /etc/passwd > std.out

89. IO Redirection Example
● Change the STDERR by using 2> :
$ cat /etc/password 2> std.err

90. Device /dev/null
● Keep the STDERR only:
$ find /etc > /dev/null

91. Output Appending
● Keep the existing content in target
file by using >> :
$ find /etc >/dev/null 2>> std.out

92. HERE Document
● Multiple line input by using << TAG :
$ mail -s test root << .
this is a test mail.
please skip.
.

93. Output Combination
● Save STDOUT and STDERR to a same
file by using 2>&1 :
$ cat std.out std.none > std.both 2>&1
Or
$ cat std.out std.none &> std.both
● Note:
Order is important! cmd1
FD2
FD1
FD0

94. Pipe Line
● Connect STDOUT of left command to
STDIN of right command:
$ cmd1 | cmd2
cmd1
FD2
FD1
FD0
screen
cmd2
FD2
FD1
FD0

95. Pipe Line
● Combine STDERR into STDOUT in
pipe line:
$ cmd1 2>&1 | cmd2
cmd1
FD2
FD1
FD0
screen
cmd2
FD2
FD1
FD0

96. Output Splitting
● Tap an output copy to a file by using
command tee :
$ cmd1 | tee file | cmd2
$ cmd1 | tee -a file | cmd2
cmd1
FD2
FD1
FD0
tee file

97. Xargs
● Change the STDIN to be as argument
by using xargs :
$ cmd1 | xargs cmd2
cmd1
FD2
FD1
FD0
cmd2 --opt args...
FD0

98. Part IX
Conditional Execution

99. Return Value (RV)
● Every command has a Return Value
when exists, also called Exist Status.
● Specified by exit in script:
exit [n]
Or, inherited from last command
● Using $? to have RV of last command

100. Return Value Range
● Range:
0-255
● Type:
TRUE: 0
FALSE: 1-255

101. Conditional Running
● Using && to run only while TRUE:
cmd1 && cmd2
●Using || to run only while FALSE:
cmd1 || cmd2

102. Test Condition
● Using test command to return
TRUE or FALSE accordingly:
test expression
[ expression ]

103. Test Expression
( EXPRESSION )
EXPRESSION is true
! EXPRESSION
EXPRESSION is false
EXPRESSION1 ­a EXPRESSION2
both EXPRESSION1 and EXPRESSION2 are true
EXPRESSION1 ­o EXPRESSION2
either EXPRESSION1 or EXPRESSION2 is true

104. String Expression
­n STRING
the length of STRING is nonzero
STRING equivalent to ­n STRING
­z STRING
the length of STRING is zero
STRING1 = STRING2
the strings are equal
STRING1 != STRING2
the strings are not equal

105. Integer Expression
INTEGER1 ­eq INTEGER2
INTEGER1 is equal to INTEGER2
INTEGER1 ­ge INTEGER2
INTEGER1 is greater than or equal to INTEGER2
INTEGER1 ­gt INTEGER2
INTEGER1 is greater than INTEGER2
INTEGER1 ­le INTEGER2
INTEGER1 is less than or equal to INTEGER2
INTEGER1 ­lt INTEGER2
INTEGER1 is less than INTEGER2
INTEGER1 ­ne INTEGER2
INTEGER1 is not equal to INTEGER2

106. File Expression
FILE1 ­nt FILE2
FILE1 is newer (mtime) than FILE2
FILE1 ­ot FILE2
FILE1 is older than FILE2

107. File Expression
­e FILE
FILE exists
­s FILE
FILE exists and has a size greater than
zero

108. File Expression
­d FILE
FILE exists and is a directory
­f FILE
FILE exists and is a regular file
­L FILE
FILE exists and is a symbolic link (same
as ­h)

109. File Expression
­u FILE
FILE exists and its set­user­ID bit is set
­g FILE
FILE exists and is set­group­ID
­k FILE
FILE exists and has its sticky bit set

110. File Expression
­G FILE
FILE exists and is owned by the effective
group ID
­O FILE
FILE exists and is owned by the effective
user ID

111. File Expression
­G FILE
FILE exists and is owned by the effective
group ID
­O FILE
FILE exists and is owned by the effective
user ID

112. File Expression
­r FILE
FILE exists and read permission is granted
­w FILE
FILE exists and write permission is granted
­x FILE
FILE exists and execute (or search)
permission is granted

113. Test Example
● Think about:
$ unset A
$ test ­n $A
$ echo $?
0
$ test ­n “$A”
$ echo $?
1

114. Test Example
● Tips:
test str
test ­n str
test ­n
test ­n ­n
test ­n $A
test ­n
test ­n ­n

115. Test Example
● Tips:
test ­n “$A“
test ­n <NULL>

116. The if­then Statement
● Syntax:
if cmd1
then
cmd2...
fi

117. The if­then­else Statement
● Syntax:
if cmd1
then
cmd2 ...
else
cmd3 ...
fi

118. Using Command Group
● Syntax:
cmd1 && {
cmd2 ...
:
} || {
cmd3 ...
}

119. Multiple Conditions (elif)
● Syntax:
if cmd1
then
cmd2 ...
elif cmd3
then
cmd4 ...
fi
repeatable

120. Run By case
● Syntax:
case “$VAR” in
pattern1)
cmd1 ...
;;
pattern2)
cmd2 ...
;;
esac
repeatable

121. Part X
Looping

122. Loop Statement
● Loop flow:
loop
condition
do
cmd ...
done

123. The for Loop
● Syntax:
for VAR in values ...
do
commands ...
done

124. The for Loop
● Tips:
● A new variable is set when running a for loop
● The value sources are vary.
● The times of loop depends on the number of
value.
● Each value is used once in the do­done
statement in order.

125. The while Loop
● Syntax:
while cmd1
do
cmd2 ...
done

126. The while Loop
● Tips:
● The cmd1 is run at each loop cycle.
● The do­done statement only be run while cmd1
returns TRUE value.
● The whole loop is terminated once cmd1 returns
FALSE value.
● Infinity loop may be designed in purpose, or
accidentally.

127. The until Loop
● Syntax:
until cmd1
do
cmd2 ...
done

128. The until Loop
● Tips:
● The cmd1 is run at each loop cycle.
● The do­done statement only be run while cmd1
returns FALSE value.
● The whole loop is terminated once cmd1 returns
TRUE value.

129. Using break In Loop
● Loop flow:
loop
condition
do
cmd ...
break
cmd ...
done

130. Using break In Loop
● Tips:
● The loop is terminated once break runs.
● A number can be specified to break the Nth
outbound loop.
1
2
3

131. Using continue In Loop
● Loop flow:
loop
condition
do
cmd ...
continue
cmd ...
done

132. Using continue In Loop
● Tips:
● The remained lines in current loop cycle are
omitted once continue runs, script goes straight
to continue next cycle.
● A number can be specified to continue the Nth
outbound loop.

133. Using sleep In Loop
● Tips:
● The script is temporally stopped when the
sleep runs.
● A number of second can be specified to stop the
script in how long.
● Useful for periodical jobs with infinity loop.

134. Plus:
Regular Expression

135. Regular Expression (RE)
● Tips:
● Processing in line base.
● Meta characters may conflict with shell, mu be
quoted.
● Commonly used in text filtering:
grep, sed, awk, perl, php, etc...

136. Regular Expression
● Character set:
abc : individual character
(abc) : a set of character
(abc|xyz) : one set of
[abc] : one character of list
[^abc] : one character of non-list

137. Regular Expression
● Anchor Characters:
^ : the beginning of line
$ : the end of line
< : the beginning of word
> : the end of word

138. Regular Expression
● Modifier
To modify the preceding character or set:
* : zero or more times
? : zero or one times
+ : one or more times
{n} : n times
{n,m} : n to m times

139. Regular Expression
● Boundary
IMPORTANT: anything outside the boundary is
ignored.
● Think about:
Which of following match abc{3,5}?
abcc
abcccc
abcccccc

140. Regular Expression
● Tips:
The abc{3,5} doesn't care about what
character following the 5th c :
abcccccc

141. Regular Expression
● Extended and traditional RE:
Extended Traditional
+ +
? ?
() ()
{} {}
| (none)

142. Show Time:
Live Coding

143. The End