What is a lexical analysis?

Lexical Analysis is the process of using the character string - or simpler text - and converting into meaningful groups called tokens. This methodology has a wide range of applications, from interpretation of computer languages ​​to book analysis. Lexical analysis is not synonymous with analysis; Rather, it is the first step from the overall process of the analysis and creates a raw material for later use.

The building blocks of chips, also called lexemes, can be generated in many ways, depending on the grammar needed for lexical analysis. A common example is the division of words in words; This is often done by distributing sentences around the premises. Each continuous string of characters generated without spaces is Lexeme. Text strings can be divided into one or many types of characters and create multiple versions of lexem with different complexity. The tokens are generated after the evaluation of each Lexeme and the joint with its corresponding value; Sedefinice, tokens refer to this pairing, not just Lexem. Its purpose is only to generate building blocks for the nextStudy, not to determine whether these pieces are valid or invalid. In the case of computer language interpretation, verification is performed by analyzing syntax and text verification can be performed in terms of context or content. If the input string is completely divided into the appropriate lexemes and each of these lexems has a suitable value, the analysis is considered successful.

Without context or ability to verify verification, lexical analysis cannot be used reliably to find input errors. Lexical grammar may have errors assigned to specific lexemas, and such analysis can also detect illegal or malformated chips. Although finding an illegal or malformed token signs invalid input, it has no effect on whether other tokens are valid and therefore it is not a strictly type of verification.

Although lexical analysis is an integral part of many algorithms, it must often be used in conjunctionwith other methodologies to create meaningful results. For example, the division of the text string into words to determine frequencies uses Lexeme formation, but Lexeme's creation itself cannot see how many times a specific Lexeme appears in the input. Lexical analysis may be useful in itself if lexem itself is, but a large amount of input could make it difficult to analyze raw lexema due to data volume.