An efficient multi-threaded package for reading(writing) delimited files. It is designed as a file parser for InMemoryDatasets.jl.
DLMReader writes and reads AbstractDatasets types, i.e. other types must be converted to/from AbstractDatasets.
It works very well for huge files (long or/and wide).
DLMReaderdoes not guessdelimiterand if it is different from,, it must be passed via thedelimiterkeyword argument. By default, theDLMReaderpackage assumes Strings are not quoted, if they are quoted, user must pass the quote character via thequotecharkeyword argument.
DLMReader.jl has some interesting features which distinguish it from other packages for reading delimited files. In what follows, we list few of them;
Informats: The DLMReader package uses informats to call a class of functions on the raw text before parsing its value(s). This provides a flexible and extendable approach to parse values with special patterns. For instance, using the predefined informat COMMA! allows users to read a numeric column with "thousands separator" and/or the dollar sign, e.g. using this informat, the raw text like "$12,000.00" will be parsed as "12000.00". Moreover, informats support function composing, e.g. COMMA! ∘ ACC! parses "$(12,000.00)" as "-12000.00", i.e. ACC! is first applied and then COMMA! is applied on its result.
informats can be applied on whole line before processing individual values.Fixed-width text: If users pass the columns locations via the fixed keyword argument, the package reads those columns as fixed-width format. For instance, passing fixed = Dict(1=>1:1, 2=>2:2) helps to parse "10" as "[1,0]". Mixing fixed-width format and delimited format is also allowed.
Multiple observations per line: The package allows reading more than one observation per line. This can be done by passing the multiple_obs = true keyword argument. The multithreading feature (plus some other features) will be switched off if this option is set.
Fast file writer: The DLMReader package exploits the byrow function from InMemoryDatasets.jl to write delimited files into disk. This enables DLMReader to convert values to string using multiple threads.
Alternative delimiters: User can pass a vector of delimiters to the function. In this case, filereader treats any of the passed delimiters as field delimiter.
Multiple Date formats: User can pass different date formats for different columns.
Different integer base: The DLMReader package allows users pass the integer base if it is different from 10 when parsing integers.
String as delimiter: User can pass a string as delimiter of values. This must be passed via the dlmstr keyword argument.
Informative warnings/info: If something goes wrong during the reading phase, the package will provide detailed warnings/info to help user investigate the issue.
See here for some benchmarks.
The following files will be used during the examples, it is assumed that the files are located in the current working directory
ex1.csv
a, b, c
1,2,NA
2,3,2001-1-2
2,4,2020-4-2
1,2,2000-12-1
ex2.csv
a::b::C::DD
12::1345::15::15
12::13::15::15
12::13::15::15
12::13::15::15
12::13::15::15
12::13::15::15
12::13::15::15
12::13::::15
12::13::15::15
12::13::15::157
ex3.csv
ex4.csv
ex5.csv
x1;x2:x3,x4
1;2;123;3
2;4,4,5
ex6.csv
id1 $2,000,000 3
id2 $34,000 4
id3 $200,000 1
And the code to read them into Julia
julia> using DLMReader
julia> filereader("ex1.csv", dtformat = Dict(3 => dateformat"y-m-d"))
julia> filereader("ex2.csv", dlmstr = "::")
julia> filereader("ex3.csv", types = [Int, Int, Int], header = false, linebreak = ';', delimiter = '\n')
julia> filereader("ex4.csv", fixed = Dict(1 => 1:4), header = false)
julia> filereader("ex5.csv", delimiter = [';', ':', ','])
julia> filereader("ex6.csv", delimiter = ' ', informat = Dict(2=>COMMA!), header = [:ID, :price, :quarter])
COMMA! is a built-in informat which removes the comma from numbers. If number contains dollar or sterling signs, it also removes them. The trimmed text is sent to the parser for converting to a number.
julia> filereader(IOBuffer("1,2,3,4,5\n6,7,8\n10\n"),
header = [:x1, :x2],
types = [Int, Int],
multiple_obs = true)
5×2 Dataset
Row │ x1 x2
│ identity identity
│ Int64? Int64?
─────┼────────────────────
1 │ 1 2
2 │ 3 4
3 │ 5 6
4 │ 7 8
5 │ 10 missing
julia> filereader(IOBuffer(""" name1 name2 avg1 avg2 y
0 A D 75 5 32
1 A D 75 5 32
2 D L 32 7 12
3 F C 99 8 42
4 F C 99 8 42
5 C A 43 6 39
6 C A 43 6 39
7 L R 53 3 11
8 R F 21 2 25
9 R F 21 2 25
"""), delimiter = ' ', ignorerepeated = true, emptycolname = true)
10×6 Dataset
Row │ NONAME1 name1 name2 avg1 avg2 y
│ identity identity identity identity identity identity
│ Int64? String? String? Int64? Int64? Int64?
─────┼────────────────────────────────────────────────────────────
1 │ 0 A D 75 5 32
2 │ 1 A D 75 5 32
3 │ 2 D L 32 7 12
4 │ 3 F C 99 8 42
5 │ 4 F C 99 8 42
6 │ 5 C A 43 6 39
7 │ 6 C A 43 6 39
8 │ 7 L R 53 3 11
9 │ 8 R F 21 2 25
10 │ 9 R F 21 2 25
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4