What Is a Regular Expression

A regular expression (regex) is a string used to match and manipulate text. Its two core capabilities are search and replace:

  • Search — essentially matching a pattern against a string.
  • Replace — once a pattern is matched, you can replace it with something else, which is often more useful in practice.

A great place to learn and experiment: https://regexr.com/

Matching Any Single Character

. matches any single character — letters, digits, and even . itself — except for the newline character.

\. matches a literal . instead of “any character”. The backslash \ is a metacharacter: it gives the following character a special meaning, or escapes it to remove its special meaning.

Matching a Set of Characters

[abc] matches any one character among a, b, or c. Square brackets [ ] define a character class: every character listed inside is a member of that set, and the engine must match exactly one of them.

Negated Character Classes

[^abc] matches any single character except a, b, or c. The caret ^, when it is the first character inside [ ], negates the set — forcing a match on anything outside the listed characters.

Matching a Range of Characters

[a-z] matches any single lowercase letter from a to z. [0-9] matches any digit from 0 to 9. Because character ranges (like 0-9, A-Z) are so common, regex provides a shorthand metacharacter - to define a range inside a character class.

Digit metacharacters:

  • \d matches any digit, equivalent to [0-9].
  • \D matches any non-digit, equivalent to [^0-9].

Alphanumeric metacharacters:

  • \w matches any word character — a letter, a digit, or an underscore — equivalent to [a-zA-Z0-9_].
  • \W matches any non-word character, equivalent to [^a-zA-Z0-9_].

Whitespace metacharacters:

  • \s matches any whitespace character (space, tab, form feed, etc.), equivalent to [ \f\n\r\t\v].
  • \S matches any non-whitespace character, equivalent to [^ \f\n\r\t\v].

Repetition

  • * matches the preceding element zero or more times.
  • + matches the preceding element one or more times. For example, a+ matches one or more consecutive as.
  • ? matches the preceding element zero or one time (i.e., optional).
  • {n} matches exactly n repetitions. For example, [A-Z]{3} matches exactly three uppercase letters.
  • {n,m} matches between n and m repetitions.
  • {n,} matches at least n repetitions.

Position Matching (Anchors)

  • \b matches a word boundary. For example, \bfoo\b matches the word foo but not foobar.
  • ^ matches the start of a line/string. For example, ^foo matches a line that begins with foo.
  • $ matches the end of a line/string. For example, foo$ matches a line that ends with foo.

How Regex Works Under the Hood

The theoretical foundation of regular expressions is the finite automaton (finite state machine, FSM). In computer science, any regular expression can be converted into an equivalent finite automaton. An FSM is an abstract mathematical model made of a set of states and the transitions (conditions) that connect them.

The compilation process: when you write a regex, the regex engine (such as PCRE or RE2) first “compiles” it into an internal data structure — usually an NFA or a DFA. This is analogous to compiling a high-level language into machine code. In practice, regex engines are built on two kinds of automata:

  • DFA (Deterministic Finite Automaton): parallel and stateless. At any input position there is exactly one determined transition. DFAs are fast and predictably bounded in time, but they cannot capture groups easily.
  • NFA (Non-deterministic Finite Automaton): backtracking and stateful. It explores possible paths and uses backtracking to find a match. NFAs support richer features (capturing groups, look-around), at the cost of potentially exponential worst-case time.

Modern engines often blend the two: for example, RE2 uses a DFA-style approach for guaranteed linear-time matching, while PCRE is a backtracking NFA engine offering more expressive power.