Package scanner provides a scanner and tokenizer for UTF-8-encoded text. It takes an io.Reader providing the source, which then can be tokenized through repeated calls to the Scan function. For compatibility with existing tools, the NUL character is not allowed. If the first character in the source is a UTF-8 encoded byte order mark (BOM), it is discarded.
By default, a Scanner skips white space and Go comments and recognizes all literals as defined by the Go language specification. It may be customized to recognize only a subset of those literals and to recognize different identifier and white space characters.
package main
import (
"fmt"
"strings"
"text/scanner"
)
func main() {
const src = `
// This is scanned code.
if a > 10 {
someParsable = text
}`
var s scanner.Scanner
s.Init(strings.NewReader(src))
s.Filename = "example"
for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() {
fmt.Printf("%s: %s\n", s.Position, s.TokenText())
}
}
Output:
example:3:1: if
example:3:4: a
example:3:6: >
example:3:8: 10
example:3:11: {
example:4:2: someParsable
example:4:15: =
example:4:17: text
example:5:1: }
package main
import (
"fmt"
"strings"
"text/scanner"
"unicode"
)
func main() {
const src = "%var1 var2%"
var s scanner.Scanner
s.Init(strings.NewReader(src))
s.Filename = "default"
for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() {
fmt.Printf("%s: %s\n", s.Position, s.TokenText())
}
fmt.Println()
s.Init(strings.NewReader(src))
s.Filename = "percent"
// treat leading '%' as part of an identifier
s.IsIdentRune = func(ch rune, i int) bool {
return ch == '%' && i == 0 || unicode.IsLetter(ch) || unicode.IsDigit(ch) && i > 0
}
for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() {
fmt.Printf("%s: %s\n", s.Position, s.TokenText())
}
}
Output: default:1:1: % default:1:2: var1 default:1:7: var2 default:1:11: % percent:1:1: %var1 percent:1:7: var2 percent:1:11: %
package main
import (
"fmt"
"strings"
"text/scanner"
)
func main() {
const src = `
// Comment begins at column 5.
This line should not be included in the output.
/*
This multiline comment
should be extracted in
its entirety.
*/
`
var s scanner.Scanner
s.Init(strings.NewReader(src))
s.Filename = "comments"
s.Mode ^= scanner.SkipComments // don't skip comments
for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() {
txt := s.TokenText()
if strings.HasPrefix(txt, "//") || strings.HasPrefix(txt, "/*") {
fmt.Printf("%s: %s\n", s.Position, txt)
}
}
}
Output: comments:2:5: // Comment begins at column 5. comments:6:1: /* This multiline comment should be extracted in its entirety. */
package main
import (
"fmt"
"strings"
"text/scanner"
)
func main() {
// tab-separated values
const src = `aa ab ac ad
ba bb bc bd
ca cb cc cd
da db dc dd`
var (
col, row int
s scanner.Scanner
tsv [4][4]string // large enough for example above
)
s.Init(strings.NewReader(src))
s.Whitespace ^= 1<<'\t' | 1<<'\n' // don't skip tabs and new lines
for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() {
switch tok {
case '\n':
row++
col = 0
case '\t':
col++
default:
tsv[row][col] = s.TokenText()
}
}
fmt.Print(tsv)
}
Output: [[aa ab ac ad] [ba bb bc bd] [ca cb cc cd] [da db dc dd]]
Predefined mode bits to control recognition of tokens. For instance, to configure a Scanner such that it only recognizes (Go) identifiers, integers, and skips comments, set the Scanner's Mode field to:
ScanIdents | ScanInts | SkipComments
With the exceptions of comments, which are skipped if SkipComments is set, unrecognized tokens are not ignored. Instead, the scanner simply returns the respective individual characters (or possibly sub-tokens). For instance, if the mode is ScanIdents (not ScanStrings), the string "foo" is scanned as the token sequence '"' Ident '"'.
Use GoTokens to configure the Scanner such that it accepts all Go literal tokens including Go identifiers. Comments will be skipped.
View Sourceconst ( EOF = -(iota + 1) Ident Int Float Char String RawString )
The result of Scan is one of these tokens or a Unicode character.
View Sourceconst GoWhitespace = 1<<'\t' | 1<<'\n' | 1<<'\r' | 1<<' '
GoWhitespace is the default value for the Scanner's Whitespace field. Its value selects Go's white space characters.
This section is empty.
TokenString returns a printable string for a token or Unicode character.
Position is a value that represents a source position. A position is valid if Line > 0.
IsValid reports whether the position is valid.
A Scanner implements reading of Unicode characters and tokens from an io.Reader.
Init initializes a Scanner with a new source and returns s. [Scanner.Error] is set to nil, [Scanner.ErrorCount] is set to 0, [Scanner.Mode] is set to GoTokens, and [Scanner.Whitespace] is set to GoWhitespace.
Next reads and returns the next Unicode character. It returns EOF at the end of the source. It reports a read error by calling s.Error, if not nil; otherwise it prints an error message to os.Stderr. Next does not update the [Scanner.Position] field; use Scanner.Pos() to get the current position.
Peek returns the next Unicode character in the source without advancing the scanner. It returns EOF if the scanner's position is at the last character of the source.
Pos returns the position of the character immediately after the character or token returned by the last call to Scanner.Next or Scanner.Scan. Use the [Scanner.Position] field for the start position of the most recently scanned token.
Scan reads the next token or Unicode character from source and returns it. It only recognizes tokens t for which the respective [Scanner.Mode] bit (1<<-t) is set. It returns EOF at the end of the source. It reports scanner errors (read and token errors) by calling s.Error, if not nil; otherwise it prints an error message to os.Stderr.
TokenText returns the string corresponding to the most recently scanned token. Valid after calling Scanner.Scan and in calls of [Scanner.Error].
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4