What is "token"

A token in computation is a text segment or symbol that can be manipulated by a parser, which provides meaning to the text; in other words, it is a character set (from an alphabet, for example) with a collective meaning.

Tokens are the patterns that occur in a string, for example: on a date, 03/29/1991, you could use two tokens to split the string into three parts, using the / bar as default. Accordingly, any date entered may be divided and analysed separately in day, month and year. The same can be done with regular expressions, but the token method uses much less processing, and is therefore faster, although it is not as robust.