Regular Expression

1. Mahzad.Zahedi@rcisp.com
May 2014
REGULAR EXPRESSIONS)REGEX)
1

2. REFERENCE
The Complete Tutorial (Temp folder)
2

3. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
3

4. INTRODUCTION
 A regular expression (regex or regexp) is a special text
string for describing a search pattern, validate data.
 A regular expression “engine” is a piece of software that
can process regular expressions.
 There are many software applications and programming
languages that support regular expressions.
4

5. .NET 1.0–4.5
Java 4–8
Perl 5.8–5.18
PCRE C library = Perl Compatible Regular Expressions
PHP
Delphi
R
JavaScript
VBScript
XRegExp
Python
Ruby
Tcl ARE
POSIX BRE
• similar to the one used by the traditional UNIX grep command
• most metacharacters require a backslash to give the
metacharacter a{1,2} matches a or aa
POSIX ERE
• similar to the one used by the UNIX egrep command
• Quantifiers ?, +, {n}, {n,m} and {n,}
• Backreferences
• Alternation
GNU BRE
GNU ERE
Oracle
XML
XPath
5
EditPrO

6. INTRODUCTION(CONT.)
 Advantage:
 Reducing development time for Programmer
 Fast executing
 Ex : reg(ular expressions?|ex(p|es)?)
 regular expressions
 regular
 regexp
 regexp
 regexes
6

7. INTRODUCTION(CONT.)
 Pattern matching an esentional problem
 Many applications need to "parse" a input
1) URLs
2) Log Files:
3) XML
http://first.dk/index.php?id=141&view=details
13/02/2010 66.249.65.107 get /support.html
20/02/2010 42.116.32.64 post /search.html
protocol host path query-string
(list of key-value pairs)
<article>
<title>Three Models for
the...</title>
<author>Noam Chomsky</author>
<year>1956</year>
</article>
7

8. LITERAL CHARACTERS & SPECIAL
CHARACTERS
 Literals
 A single literal character, ex : «a»
 “Jack is a boy”
 Some literal characters
 Apply «cat» to “He captured a catfish for his cat.”
 Non-Printable Characters
 «t » tab character (ASCII 0x09)
 «r» for carriage return (0x0D)
 «n» for line feed (0x0A)
 …
1-‫انطباق‬‫ای‬‫ر‬‫ب‬‫تالش‬‫ج‬‫ر‬‫توکن‬ ‫اولین‬‫کس‬
‫شته‬‫ر‬ ‫با‬:C‫با‬H‫شکست‬ ‫و‬!!
2-‫موجود‬ ‫جکس‬‫ر‬‫از‬ ‫ی‬‫دیگر‬‫جایگشت‬
‫شته‬‫ر‬ ‫در‬‫بعدی‬‫اکتر‬‫ر‬‫کا‬‫لذا‬‫نیست‬
3-‫تو‬‫اغ‬‫ر‬‫س‬ ‫موفق‬‫انطباق‬ ‫مین‬‫ر‬‫چها‬ ‫در‬‫کن‬
‫بعدی‬
4-‫میخورد‬‫شکست‬‫انطباق‬ ‫ششمین‬‫در‬‫و‬
‫مین‬‫ر‬‫چها‬ ‫در‬‫میشود‬‫متوجه‬‫موف‬‫ی‬ ‫بررس‬‫ق‬
‫م‬ ‫ادامه‬ ‫اکتر‬‫ر‬‫کا‬ ‫پنجمین‬‫از‬ ‫و‬ ‫نبودده‬‫ی‬
‫دهد‬
∕∕
8

9. LITERAL CHARACTERS & SPECIAL
CHARACTERS (CONT.)
 Reserve certain characters for special use
Meaning char
Beginning of string ^ caret
End of string $ doller sign
Any character except newline . dot
Match 0 or more * star
Match 1 or more + plus
Match 0 or 1 ? Question mark
alternative | pipe symbol
Grouping; ”store” ( ) parenthesis
Special backslash
opening square bracket [
9

10. LITERAL CHARACTERS & SPECIAL
CHARACTERS (CONT.)
 use any of these characters as a literal in a regex
If you forget to escape a special character
NOTE !
 Most regular expression flavors treat the brace «{» as a literal
character, unless it is part of a repetition operator like « M{1,3}».
 An exception to this rule is the java.util.regex
1+1=2 literal
123+111=234 other meaning
+1=2 ERROR
1+1=2
10

11. LITERAL CHARACTERS & SPECIAL
CHARACTERS (CONT.)
 Q...E escape sequence
 E.g. «Q*d+*E» matches the literal text „*d+*”.
o Special Characters and Programming Languages
 Compiler will turn the escaped backslash in the source
code into a single backslash in the string that is passed
on to the regex library
 The regex «1+1=2» as “1+1=2”
 The regex «c:temp» as “c:temp”
compiler Regex lib
11

12. FIRST LOOK AT HOW A REGEX ENGINE
WORKS INTERNALL
 Two kinds of regular
expression engines:
 text-directed- DFA
 regex-directed- NFA
 awk, egrep, flex, lex,
MySQL are text-directed
 A few of versions are regex-
directed
 The regex directed is
more powerful.
12

13. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
13

14. CHARACTER CLASSES OR CHARACTER
SETS
 To match only one out of several characters
 «gr[ae]y» = > „ gray” or „grey” (for both American or British English )
 Using a hyphen inside a character class to specify a
range of characters.
 [0-9a-fA-F]
 Useful Application
 Find a word, even if it is misspelled => «sep[ae]r[ae]te»
 Find an identifier => «[A-Za-z_][A-Za-z_0-9]*»
 Find a C-style hexadecimal number => «0[xX][A-Fa-f0-9]+»
14

15. NEGATED CHARACTER CLASSES
 [^ …]
 Match any character that is not in the character
class.
 «q[^u]» : “a q followed by a character that is not a
u”. => Iraq is a country
 Negated character class still must match a
character. [^abc] => fdgha
 Unlike the dot, negated character classes also
match line break characters. 15

16. METACHARACTERS INSIDE CHARACTER
CLASSES
 Metacharacters inside a character class are:
 The closing bracket ( ] )=>(])
 The backslash ()=> []
 The caret (^) => [^]
 The hyphen (-)=> [-]
 [+*]=[+*] -- > reducing readability
 Other Solutions:
 Placing them in a position where they do not take on their
special meaning.
 Closing bracket right after the opening bracket []x]
 Caret anywhere except right after the opening bracket [x^]
 Hyphen any where except middle [-x] 16

17. METACHARACTERS INSIDE CHARACTER
CLASSES
 All non-printable characters in character classes
just like outside of character classes.
 E.g. [$u20AC] : dollar or euro sign
 Perl and PCRE also support the Q...E sequence
inside character classes
 E.g. «[Q[-]E]» matches „[”, „-” or „]”.
 POSIX regular expressions treat the backslash as a
literal character inside character classes.
 Can’t use backslashes to escape
 So just use in correct position
17

18. SHORTHAND CHARACTER CLASSES
 Both inside and outside the square brackets are
used
 Ex: 1 + 2 = 3
 sd=whitespace followed by a digit “ 2”
 [sd]=whitespace or digit “1 2 3”
Class Meaninig
w Word character, [a-zA-z0-9_].
d Digit character, [0-9].
s Whitespace character, [ nrt ].
W Non-word character, [^a-zA-z0-9_] =[^w]
D Non-digit character, [^0-9]=[^d]
S Non-whitespace character, [^ nrft ]=[^s]
18

19. NEGATED SHORTHAND CHARACTER
CLASSES
 «[DS]» is not the same as «[^ds]».
 [^ds]= any char that is not a digit or whitespace.
 123 5] ⌐(a U b )
 [DS] =any char that is either not a digit, or is not
whitespace.
 123 5] ⌐ a U ⌐b
19

20. REPEATING CHARACTER CLASSES
 By using the «?», «*» or «+» operators
 «[0-9]+» “833337” „222” …
 For repeating the matched character, rather than
the class we need “backreferences”
 «([0-9])1+»
 will match „222”
 will match „3333” for “833337 “
20

21. LOOKING INSIDE THE REGEX ENGINE
 The order inside a character class does not matter
 Ex : «gr[ae]y» “Is his hair grey or gray?”
1. Failing to match “g” every 12 steps
2. „g” is matched in 13th step
3. Matching “r” token in the regex with “r” in text
4. Failing to match “a“ token with “e”
5. Try to match other permutations of the regex
pattern
6. Matching the last regex token with “y” in text
7. the leftmost match was returned : grey
21

22. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
22

23. THE DOT MATCHES (ALMOST) ANY
CHARACTER
 The most commonly misused metacharacter.
 The dot will not match a newline character by
default (Why)?
 «[^n]» (UNIX regex flavors)
 «[^rn]» (Widows regex flavors)
 In Perl, the mode where the dot also matches
newlines is called "single-line mode“
 In .NET framework “Regex.Match("string", "regex",
RegexOptions.Singleline)”
 JavaScript and VBScript do not have an option to
make the dot match line break characters : «[sS]»
23

24. USE THE DOT SPARINGLY
 The dot is a very powerful regex metacharacter
 It allows you to be lazy ,Ex: mm/dd/yy format
 Solutions:
• dd.dd.dd 02512703
• dd[- /.]dd[- /.]dd 99/99/99
• (0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01]) [- .](19|20)dd
09/31/2079
24

25. USE NEGATED CHARACTER SETS INSTEAD
OF THE DOT
 star is greedy
 Ex: we have a problem with "string one" and "string
two "
 Regexp : ".*"
 "string one" and "string two“
 Regexp : "[^"rn]*"
 "string one" "string two"
25

26. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
26

27. START OF STRING AND END OF STRING
ANCHORS
 Literals and class characters match a character
 Anchors do not match any character at all. Instead,
they match a position
 Caret «^» : «^a» to “abc”
 Dolor sign «$» : «c$» to “abc”
 Useful Application
 For validating user input, using anchors is very important
 if ($input =~ m/d+/) qsdf4ghjk => «^d+$» qsdf4ghjk 44467
27

28. USING ^ AND $ AS START OF LINE AND END
OF LINE ANCHORS
 If you have a string consisting of multiple lines,Ex:
 “ first linen second line”
 In tools as EditPad Pro (work with entire files)
 In Programming Languages
 Perl : "multi-line mode“
 m/^regex$/m
28

29. PERMANENT START OF STRING AND END OF
STRING ANCHORS
 «A» : only ever matches at the start of the file
 «Z» : only ever matches at the end of the file
 Anchors match at a position, rather than matching a
character
 Anchors can result in a zero-length match.
 Since the match does not include any characters,
nothing is deleted in replcament
 In VB.NET
 Dim Quoted as String = Regex.Replace(Original, "^", "> ",
RegexOptions.Multiline)
29

30. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
30

31. WORD BOUNDARIES
 The metacharacter «b» is an anchor like ^“ ” & “$”
 This match is zero-length.
 Simply put: «b» allows you to perform a “whole
words only”
 «b4b» matches a “4” 44 a4
 2 positions :
 Before the first & last word character
 Between a word character and a non-word character
31

32. LOOKING INSIDE THE REGEX ENGINE
 Ex: «bisb» string : “This island is beautiful”.
 “b” matches position before “T”
 Matching the next token: the literal «i»
 The engine does not advance to the next character in the
string, because the previous regex token was zero-lenght, «i»
does not match “T”.
 «b» can not match at the position between the “T” and
the “h”.
 ….
 POSIX does not support word boundaries at all.
32

33. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
33

34. ALTERNATION WITH THE VERTICAL BAR OR
PIPE SYMBOL
 Similar to character classes to match a single
character
 Remember That The Regex Engine Is Eager
 It will stop searching as soon as it finds a valid match.
 RE: Get|GetValue|Set|SetValue
 Str : SetValue
What are solutions?
1-‫توکن‬ ‫اولین‬G‫و‬ ‫داده‬‫انطباق‬ ‫ل‬‫او‬ ‫اکتر‬‫ر‬‫کا‬‫با‬
‫شکست‬!!
2-‫بعدی‬ ‫های‬‫گزینه‬«‫یا‬»‫شکست‬ ‫و‬!!
3-‫بعدی‬‫توکن‬S‫با‬S‫و‬ ‫داده‬ ‫ق‬‫انطبا‬‫شته‬‫ر‬‫در‬
‫موفق‬!‫تا‬‫ادامه‬ ‫و‬«t»
4-،‫جیکس‬‫ر‬‫بودن‬ ‫مشتاق‬ ‫خاطر‬‫به‬SET
‫میشود‬‫برگردونده‬ 34

35. ALTERNATION WITH THE VERTICAL BAR OR
PIPE SYMBOL (CONT.)
 Solutions are:
 Changing the order of options
 GetValue|Get|SetValue|Set
 Using greedy feature of question mark ”?”
 Get(Value)?|Set(Value)?
 Using b
 b(Get|GetValue|Set|SetValue)b
 The POSIX standard mandates that the longest match be
returned, regardless if the regex an NFA or DFA algorithm.
35

36. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
36

37. OPTIONAL ITEMS
 “?” makes the preceding token in the regular
expression optional.
 You can make several tokens optional by grouping
them together using round brackets
 Feb(ruary)? 23(rd)?
 „February 23rd”, „February 23”, „Feb 23rd” , „Feb 23”.
 Important Regex Concept: Greediness
 The engine will always try to match that part. Only if this
causes the entire regular expression to fail, will try
ignoring the part the question mark applies to. 37

38. LOOKING INSIDE THE REGEX ENGINE
 EX: «colou?r» , Str: “The colonel likes the color
green”.
1. 5th char matches successfully from “c” to “o”
2. Checking wheather “u” matches “n” and fail
 Question mark : failing is accesptable.
3. Next token , fails to match “n”.
4. starts again trying to match «c» to the first o in
“colonel”.
5. ….
38

39. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
39

40. REPETITION WITH STAR AND PLUS
 Valid HTML tag
 «<[A-Za-z][A-Za-z0-9]*>»
 «<[A-Za-z0-9]+>» „<1>”
Class Meaninig
* 0 or more
+ 1 or more
? 0 or 1
{3} Exactly 3
{3,} 3 or more
{3,5} 3, 4 or 5
Add a ? to a quantifier to make it ungreedy. 40

41. WATCH OUT FOR THE GREEDINESS!
 EX: Matching HTML tag
 This is a <EM>first</EM> test
 <.+>
1. The first token in the regex is «<».
2. The next token is the dot, which matches any character except
newlines
3. The dot is repeated by the plus. The plus is greedy
4. The dot fails when the engine has reached the void after the
end of the string.
5. Engine continue with the next token «>» &can not match
6. The engine remembers that the plus has repeated the dot more
often than is required so it backtrack
7. It is reduced to „EM>first</EM> tes” and next token in the regex
is still «>»
8. It will continue for the first valid match (eager)
41

42. LAZINESS INSTEAD OF GREEDINESS
 Lazy quantifiers are sometimes also called “ungreedy”
 This is a <EM>first</EM> test
 “<.+?>”
1. «<» matches the first „<” in the string
2. The next token is the dot, this time repeated by a lazy
plus
 This tells the regex engine to repeat the dot as few times
as possible (MIN=1)
 Matches “.” With “E”
3. Matches “>” with “M” and fails
• But this time, the backtracking will force the lazy plus to
expand
4. Return <EM> </EM>
42

43. LOOKING INSIDE THE REGEX ENGINE
 Ex: <([A-Z][A-Z0-9]*)[^>]*>.*?</1>
 Str: “Testing <B><I>bold italic</I></B> text”
1. Matching at the first „<”
2. «[A-Z]» matches „B” & advances to «[A-Z0-9]» and “>”
3. This match fails. However, because of the star, that’s
perfectly fine
4. Storing what was matched inside them, „B” is stored
5. The regex is advanced to «[^>]» & string remains at “>” &
go to 3
6. Matching “>” with “>”
7. The next token is a dot, repeated by a lazy star
43

44. AN ALTERNATIVE TO LAZINESS
 An option for making the plus lazy instead of
backtracking
 Greedy plus and a negated character class
 <EM>first</EM>
 «<[^>]+>»
 Backtracking slows down the regex engine
 you will save plenty of CPU cycles when using such a
regex
44

45. REPEATING Q...E ESCAPE SEQUENCE
 «Q*d+*E+»
 In Perl : “*d+**d+*”
 In java : “*d+**d+*”
 If you want Java to return the same match as Perl
 «Q*d+E*+»
 If you want Perl to repeat the whole sequence like
Java does
 «(Q*d+*E)+»
45

46. TABLE OF CONTENT
 1. Introduction
 2. Literal Characters
 3. First Look at How a Regex Engine Works
 4. Character Classes or Character Sets
 5. The Dot Matches (Almost) Any Character .
 6. Start of String and End of String Anchors
 7. Word Boundaries
 8. Alternation with The Vertical Bar or Pipe Symbol
 9. Optional Items
 10. Repetition with Star and Plus
 11. Use Round Brackets for Grouping
46

47. USE ROUND BRACKETS FOR GROUPING
 Grouping the part of the regular expression
together for applying a regex operator
 Creating a Backreference
 reuses part of the regex match
 slows down the regex engine
 Optimize this regular expression into «Set(?:Value)?
 How to Use Backreferences
 abc5abc
 )[abc])+51$ 1=> a b c - > abbc5c
 <div>hello</div>
 <([a-z]*)>.*</1> 47

48. REPETITION AND BACKREFERENCES
 ([abc]+)» & «([abc])+» to “cab” string
 ([abc]+)» : “cab” to be referenced
 «([abc])+» : “b” to be referenced
48

49. USE ROUND BRACKETS FOR GROUPING
(CONT.)
 Reusing the same backreference more than once.
 ([a-c])x1x1» „axaxa” „bxbxb” „cxcxc”
 Backreferences Cannot be used inside itself.
 ([abc]1)
 Round brackets Cannot be used inside character
classes, as metacharacters.
 (a)[1b]
 Useful Example: Checking for Doubled Words
 «b(w+)s+1b»
49

50. POSIX CLASS
Posix ASCII ‫توضیح‬
[:alnum:] [A-Za-z0-9] ‫همه‬‫وعددی‬ ‫حرفی‬ ‫های‬‫اکتر‬‫ر‬‫کا‬
[:alpha:] [A-Za-z] ‫و‬‫بزرگ‬ ‫حروف‬‫کوچک‬
[:blank:] [t] ‫فاصله‬‫تب‬ ‫و‬
[:digit:] [0-9] ‫اعداد‬
[:punct:] [?<=>;:/.,+*()'
&%$#”!-]
@[]^_`{|}~]
‫عالیم‬‫ی‬ ‫ش‬‫ر‬‫نگا‬
50

51. ٍEXAMPLE 1
 Example 1. Beginning of line ( ^ )
 grep "^Nov 10" messages.1
 Example 2. End of the line ( $)
 grep "terminating.$" messages
.
Nov 10 01:12:55 gs123 ntpd[2241]: time reset +0.177479 s
Nov 10 01:17:17 gs123 ntpd[2241]: synchronized to OCAL(0)
Nov 10 01:18:49 gs123 ntpd[2241]: synchronized to 15.1.13.13
Jul 12 17:01:09 cloneme kernel: Kernel log daemon terminating.
Oct 28 06:29:54 cloneme kernel: Kernel log daemon terminating
51

52. EXAMPLE 2
 Example 3. quantifier (*)(+)(?)
 [hc]*at =cchat,hcat,hhhat,at 0 or more
 [hc]+at= ccchat, hcat, No at 1 or more
 [hc]?at= hat, cat, at 0 or 1
 Example 4.Escaping the special character ()
 grep "127.0.0.1" /var/log/messages.4
Oct 28 06:31:10 btovm871 ntpd[2241]: Listening on interface lo,
127.0.0.1#123 Enabled
52

53. ٍEXAMPLE 3
 Example 5.Excluding specific characters
‫الف‬:
 Match text hog
 Match text dog ---- > [^b]og
 Skip Text bog
‫ب‬:‫غیر‬ ‫ی‬‫اکتر‬‫ر‬‫کا‬a‫یا‬b‫یا‬c
[^abc]
abccc
adb
gh
53

54. EXAMPLE 4
 Example 6. Composite syntax
‫است‬ ‫یر‬‫ز‬ ‫شرح‬ ‫به‬ ‫الگ‬‫اطالعات‬:
 Sun Jun 4 22:08:39 2006 [pid 21611] [dcid] OK
LOGIN: Client “192.168.1.1”
 ^w+sw+sd+ S+ d+ [pid d+]s [(w+)] OK LOGIN:
Client “(d+.d+.d+.d+)”$
54