• About Miller • File formats • Miller features in the context of the Unix toolkit • Record-heterogeneity • Reference • Data examples • FAQ • Internationalization • Compiling, portability, dependencies, and testing • Performance • Why C? • Why call it Miller? • How original is Miller? • Things to do • Contact information • GitHub repo |
• On-line help • then-chaining • I/O options • Formats • Record/field/pair separators • Number formatting • Regular expressions • Data transformations • cat • check • count-distinct • cut • filter • group-by • group-like • having-fields • head • histogram • join • label • put • regularize • rename • reorder • sample • sort • stats1 • stats2 • step • tac • tail • top • uniq • Functions for filter and put • Data types • Null data Command overviewWhereas the Unix toolkit is made of the separate executables cat, tail, cut, sort, etc., Miller has subcommands, invoked as follows: mlr tac *.dat mlr cut --complement -f os_version *.dat mlr sort -f hostname,uptime *.dat
On-line helpExamples:$ mlr --help Usage: mlr [I/O options] {verb} [verb-dependent options ...] {file names} Verbs: cat check count-distinct cut filter group-by group-like having-fields head histogram join label put regularize rename reorder sample sort stats1 stats2 step tac tail top uniq Example: mlr --csv --rs lf --fs tab cut -f hostname,uptime file1.csv file2.csv Please use "mlr -h" or "mlr --help" to show this message. Please use "mlr --version" to show the software version. Please use "mlr {verb name} --help" for verb-specific help. Please use "mlr --list-all-verbs" to list only verb names. Please use "mlr --help-all-verbs" for help on all verbs. Functions for filter and put: abs acos acosh asin asinh atan atan2 atanh cbrt ceil cos cosh erf erfc exp expm1 floor invqnorm log log10 log1p logifit max min pow qnorm round roundm sin sinh sqrt tan tanh urand + - - * / % ** =~ !=~ == != > >= < <= && || ! strlen sub gsub tolower toupper . boolean float int string hexfmt fmtnum systime sec2gmt gmt2sec sec2hms sec2dhms hms2sec dhms2sec fsec2hms fsec2dhms hms2fsec dhms2fsec Please use "mlr --help-function {function name}" for function-specific help. Please use "mlr --help-all-functions" or "mlr -f" for help on all functions. Data-format options, for input, output, or both: --idkvp --odkvp --dkvp Delimited key-value pairs, e.g "a=1,b=2" (default) --inidx --onidx --nidx Implicitly-integer-indexed fields (Unix-toolkit style) --icsv --ocsv --csv Comma-separated value (or tab-separated with --fs tab, etc.) --ipprint --opprint --pprint --right Pretty-printed tabular (produces no output until all input is in) --ixtab --oxtab --xtab Pretty-printed vertical-tabular -p is a keystroke-saver for --nidx --fs space --repifs Examples: --csv for CSV-formatted input and output; --idkvp --opprint for DKVP-formatted input and pretty-printed output. Separator options, for input, output, or both: --rs --irs --ors Record separators, e.g. 'lf' or '\r\n' --fs --ifs --ofs --repifs Field separators, e.g. comma --ps --ips --ops Pair separators, e.g. equals sign Notes: * IPS/OPS are only used for DKVP and XTAB formats, since only in these formats do key-value pairs appear juxtaposed. * IRS/ORS are ignored for XTAB format. Nominally IFS and OFS are newlines; XTAB records are separated by two or more consecutive IFS/OFS -- i.e. a blank line. * OFS must be single-character for PPRINT format. This is because it is used with repetition for alignment; multi-character separators would make alignment impossible. * OPS may be multi-character for XTAB format, in which case alignment is disabled. * DKVP, NIDX, CSVLITE, PPRINT, and XTAB formats are intended to handle platform-native text data. In particular, this means LF line-terminators by default on Linux/OSX. You can use "--dkvp --rs crlf" for CRLF-terminated DKVP files, and so on. * CSV is intended to handle RFC-4180-compliant data. In particular, this means it uses CRLF line-terminators by default. You can use "--csv --rs lf" for Linux-native CSV files. * You can specify separators in any of the following ways, shown by example: - Type them out, quoting as necessary for shell escapes, e.g. "--fs '|' --ips :" - C-style escape sequences, e.g. "--rs '\r\n' --fs '\t'". - To avoid backslashing, you can use any of the following names: cr crcr newline lf lflf crlf crlfcrlf tab space comma pipe slash colon semicolon equals * Default separators by format: File format RS FS PS dkvp \n , = csv \r\n , (N/A) csvlite \n , (N/A) nidx \n space (N/A) xtab (N/A) \n space pprint \n space (N/A) Relevant to CSV/CSV-lite input only: --implicit-csv-header Use 1,2,3,... as field labels, rather than from line 1 of input files. Tip: combine with "label" to recreate missing headers. Double-quoting for CSV output: --quote-all Wrap all fields in double quotes --quote-none Do not wrap any fields in double quotes, even if they have OFS or ORS in them --quote-minimal Wrap fields in double quotes only if they have OFS or ORS in them (default) --quote-numeric Wrap fields in double quotes only if they have numbers in them Numerical formatting: --ofmt {format} E.g. %.18lf, %.0lf. Please use sprintf-style codes for double-precision. Applies to verbs which compute new values, e.g. put, stats1, stats2. See also the fmtnum function within mlr put (mlr --help-all-functions). Other options: --seed {n} with n of the form 12345678 or 0xcafefeed. For put/filter urand(). Output of one verb may be chained as input to another using "then", e.g. mlr stats1 -a min,mean,max -f flag,u,v -g color then sort -f color For more information please see http://johnkerl.org/miller/doc and/or http://github.com/johnkerl/miller. This is Miller version >= v2.3.2. $ mlr sort --help Usage: mlr sort {flags} Flags: -f {comma-separated field names} Lexical ascending -n {comma-separated field names} Numerical ascending; nulls sort last -nf {comma-separated field names} Numerical ascending; nulls sort last -r {comma-separated field names} Lexical descending -nr {comma-separated field names} Numerical descending; nulls sort first Sorts records primarily by the first specified field, secondarily by the second field, and so on. Any records not having all specified sort keys will appear at the end of the output, in the order they were encountered, regardless of the specified sort order. Example: mlr sort -f a,b -nr x,y,z which is the same as: mlr sort -f a -f b -nr x -nr y -nr z then-chainingIn accord with the Unix philosophy, you can pipe data into or out of Miller. For example:mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime mlr cut --complement -f os_version then sort -f hostname,uptime *.dat % cat piped.sh mlr cut -x -f i,y data/big | mlr sort -n y > /dev/null % time sh piped.sh real 0m2.828s user 0m3.183s sys 0m0.137s % cat chained.sh mlr cut -x -f i,y then sort -n y data/big > /dev/null % time sh chained.sh real 0m2.082s user 0m1.933s sys 0m0.137s I/O optionsFormatsOptions:--dkvp --idkvp --odkvp --nidx --inidx --onidx --csv --icsv --ocsv --csvlite --icsvlite --ocsvlite --pprint --ipprint --ppprint --right --xtab --ixtab --oxtabThese are as discussed in File formats, with the exception of --right which makes pretty-printed output right-aligned:
Record/field/pair separatorsMiller has record separators IRS and ORS, field separators IFS and OFS, and pair separators IPS and OPS. For example, in the DKVP line a=1,b=2,c=3, the record separator is newline, field separator is comma, and pair separator is the equals sign. These are the default values. Options:--rs --irs --ors --fs --ifs --ofs --repifs --ps --ips --ops
Number formattingThe command-line option --ofmt {format string} is the global number format for commands which generate numeric output, e.g. stats1, stats2, histogram, and step, as well as mlr put. Examples:--ofmt %.9le --ofmt %.6lf --ofmt %.0lf $ echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")' x=3.1,y=4.3,z=13.330000 $ echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")' x=0xffff,y=0xff,z=00feff01 $ echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)' x=0xffff,y=0xff,z=0xfeff01 Regular expressionsMiller lets you use regular expressions (of type POSIX.2) in the following contexts:
$ cat data/regex-in-data.dat name=jane,regex=^j.*e$ name=bill,regex=^b[ou]ll$ name=bull,regex=^b[ou]ll$ $ mlr filter '$name =~ $regex' data/regex-in-data.dat name=jane,regex=^j.*e$ name=bull,regex=^b[ou]ll$ Data transformationscatMost useful for format conversions (see File formats), and concatenating multiple same-schema CSV files to have the same header:
check$ mlr check --help Usage: mlr check Consumes records without printing any output. Useful for doing a well-formatted check on input data. count-distinct$ mlr count-distinct --help Usage: mlr count-distinct [options] -f {a,b,c} Field names for distinct count. Prints number of records having distinct values for specified field names. Same as uniq -c. $ mlr count-distinct -f a,b then sort -nr count data/medium a=zee,b=wye,count=455 a=pan,b=eks,count=429 a=pan,b=pan,count=427 a=wye,b=hat,count=426 a=hat,b=wye,count=423 a=pan,b=hat,count=417 a=eks,b=hat,count=417 a=eks,b=eks,count=413 a=pan,b=zee,count=413 a=zee,b=hat,count=409 a=eks,b=wye,count=407 a=zee,b=zee,count=403 a=pan,b=wye,count=395 a=wye,b=pan,count=392 a=zee,b=eks,count=391 a=zee,b=pan,count=389 a=hat,b=eks,count=389 a=wye,b=eks,count=386 a=hat,b=zee,count=385 a=wye,b=zee,count=385 a=hat,b=hat,count=381 a=wye,b=wye,count=377 a=eks,b=pan,count=371 a=hat,b=pan,count=363 a=eks,b=zee,count=357 cut$ mlr cut --help Usage: mlr cut [options] Passes through input records with specified fields included/excluded. -f {a,b,c} Field names to include for cut. -o Retain fields in the order specified here in the argument list. Default is to retain them in the order found in the input data. -x|--complement Exclude, rather that include, field names specified by -f. -r Treat field names as regular expressions. "ab", "a.*b" will match any field name containing the substring "ab" or matching "a.*b", respectively; anchors of the form "^ab$", "^a.*b$" may be used. The -o flag is ignored when -r is present. Examples: mlr cut -f hostname,status mlr cut -x -f hostname,status mlr cut -r -f '^status$,sda[0-9]' mlr cut -r -f '^status$,"sda[0-9]"' mlr cut -r -f '^status$,"sda[0-9]"i' (this is case-insensitive)
filter$ mlr filter --help Usage: mlr filter [-v] [-x] {expression} Prints records for which {expression} evaluates to true. With -x, prints records for which {expression} evaluates to false. With -v, first prints the AST (abstract syntax tree) for the expression, which gives full transparency on the precedence and associativity rules of Miller's. grammar. Please use a dollar sign for field names and double-quotes for string literals. Miller built-in variables are NF NR FNR FILENUM FILENAME PI E. Examples: mlr filter 'log10($count) > 4.0' mlr filter 'FNR == 2 (second record in each file)' mlr filter 'urand() < 0.001' (subsampling) mlr filter '$color != "blue" && $value > 4.2' mlr filter '($x<.5 && $y<.5) || ($x>.5 && $y>.5)' mlr filter '($name =~ "^sys.*east$") || ($name =~ "^dev.[0-9]+"i)' Please see http://johnkerl.org/miller/doc/reference.html for more information including function list. $ mlr filter 'FNR == 2' data/small* a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 1=pan,2=pan,3=1,4=0.3467901443380824,5=0.7268028627434533 a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134 $ mlr --opprint filter '$a == "pan" || $b == "wye"' data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 wye wye 3 0.20460330576630303 0.33831852551664776 eks wye 4 0.38139939387114097 0.13418874328430463
mlr --opprint filter ' ($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' \ then stats2 -a corr -f x,y data/medium group-by$ mlr group-by --help Usage: mlr group-by {comma-separated field names} Outputs records in batches having identical values at specified field names.
group-like$ mlr group-like --help Usage: mlr group-like Outputs records in batches having identical field names.
having-fields$ mlr having-fields --help Usage: mlr having-fields [options] Conditionally passes through records depending on each record's field names. Options: --at-least {comma-separated names} --which-are {comma-separated names} --at-most {comma-separated names} --all-matching {regular expression} --any-matching {regular expression} --none-matching {regular expression} Examples: mlr having-fields --which-are amount,status,owner mlr having-fields --any-matching 'sda[0-9]' mlr having-fields --any-matching '"sda[0-9]"' mlr having-fields --any-matching '"sda[0-9]"i' (this is case-insensitive)
head$ mlr head --help Usage: mlr head [options] -n {count} Head count to print; default 10 -g {a,b,c} Optional group-by-field names for head counts Passes through the first n records, optionally by category.
histogram$ mlr histogram --help Usage: mlr histogram [options] -f {a,b,c} Value-field names for histogram counts --lo {lo} Histogram low value --hi {hi} Histogram high value --nbins {n} Number of histogram bins Just a histogram. Input values < lo or > hi are not counted. $ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 data/medium bin_lo bin_hi x_count x2_count x3_count 0.000000 0.100000 1072 3231 4661 0.100000 0.200000 938 1254 1184 0.200000 0.300000 1037 988 845 0.300000 0.400000 988 832 676 0.400000 0.500000 950 774 576 0.500000 0.600000 1002 692 476 0.600000 0.700000 1007 591 438 0.700000 0.800000 1007 560 420 0.800000 0.900000 986 571 383 0.900000 1.000000 1013 507 341 join$ mlr join --help Usage: mlr join [options] Joins records from specified left file name with records from all file names at the end of the Miller argument list. Functionality is essentially the same as the system "join" command, but for record streams. Options: -f {left file name} -j {a,b,c} Comma-separated join-field names for output -l {a,b,c} Comma-separated join-field names for left input file; defaults to -j values if omitted. -r {a,b,c} Comma-separated join-field names for right input file(s); defaults to -j values if omitted. --lp {text} Additional prefix for non-join output field names from the left file --rp {text} Additional prefix for non-join output field names from the right file(s) --np Do not emit paired records --ul Emit unpaired records from the left file --ur Emit unpaired records from the right file(s) -u Enable unsorted input. In this case, the entire left file will be loaded into memory. Without -u, records must be sorted lexically by their join-field names, else not all records will be paired. File-format options default to those for the right file names on the Miller argument list, but may be overridden for the left file as follows. Please see the main "mlr --help" for more information on syntax for these arguments. -i {one of csv,dkvp,nidx,pprint,xtab} --irs {record-separator character} --ifs {field-separator character} --ips {pair-separator character} --repifs --repips --use-mmap --no-mmap Please see http://johnkerl.org/miller/doc/reference.html for more information including examples.
label$ mlr label --help Usage: mlr label {new1,new2,new3,...} Given n comma-separated names, renames the first n fields of each record to have the respective name. (Fields past the nth are left with their original names.) Particularly useful with --inidx, to give useful names to otherwise integer-indexed fields. Examples: "echo 'a b c d' | mlr --inidx --odkvp cat" gives "1=a,2=b,3=c,4=d" "echo 'a b c d' | mlr --inidx --odkvp label s,t" gives "s=a,t=b,3=c,4=d" % grep -v '^#' /etc/passwd | mlr --nidx --fs : --opprint label name,password,uid,gid,gecos,home_dir,shell | head name password uid gid gecos home_dir shell nobody * -2 -2 Unprivileged User /var/empty /usr/bin/false root * 0 0 System Administrator /var/root /bin/sh daemon * 1 1 System Services /var/root /usr/bin/false _uucp * 4 4 Unix to Unix Copy Protocol /var/spool/uucp /usr/sbin/uucico _taskgated * 13 13 Task Gate Daemon /var/empty /usr/bin/false _networkd * 24 24 Network Services /var/networkd /usr/bin/false _installassistant * 25 25 Install Assistant /var/empty /usr/bin/false _lp * 26 26 Printing Services /var/spool/cups /usr/bin/false _postfix * 27 27 Postfix Mail Server /var/spool/postfix /usr/bin/false $ cat data/headerless.csv John,23,present Fred,34,present Alice,56,missing Carol,45,present $ mlr --csv --rs lf --implicit-csv-header cat data/headerless.csv 1,2,3 John,23,present Fred,34,present Alice,56,missing Carol,45,present $ mlr --csv --rs lf --implicit-csv-header label name,age,status data/headerless.csv name,age,status John,23,present Fred,34,present Alice,56,missing Carol,45,present $ mlr --icsv --rs lf --implicit-csv-header --opprint label name,age,status data/headerless.csv name age status John 23 present Fred 34 present Alice 56 missing Carol 45 present put$ mlr put --help Usage: mlr put [-v] {expression} Adds/updates specified field(s). With -v, first prints the AST (abstract syntax tree) for the expression, which gives full transparency on the precedence and associativity rules of Miller's grammar. Please use a dollar sign for field names and double-quotes for string literals. Miller built-in variables are NF NR FNR FILENUM FILENAME PI E. Multiple assignments may be separated with a semicolon. Examples: mlr put '$y = log10($x); $z = sqrt($y)' mlr put '$filename = FILENAME' mlr put '$colored_shape = $color . "_" . $shape' mlr put '$y = cos($theta); $z = atan2($y, $x)' mlr put '$name = sub($name, "http.*com"i, "")' Please see http://johnkerl.org/miller/doc/reference.html for more information including function list. $ ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j=$i+1;$k=$i+$j' i j k 0 1.000000 1.000000 1 2.000000 3.000000 2 3.000000 5.000000 3 4.000000 7.000000 4 5.000000 9.000000 5 6.000000 11.000000 6 7.000000 13.000000 7 8.000000 15.000000 8 9.000000 17.000000 9 10.000000 19.000000 $ mlr --opprint put '$nf=NF; $nr=NR; $fnr=FNR; $filenum=FILENUM; $filename=FILENAME' data/small data/small2 a b i x y nf nr fnr filenum filename pan pan 1 0.3467901443380824 0.7268028627434533 5 1 1 1 data/small eks pan 2 0.7586799647899636 0.5221511083334797 5 2 2 1 data/small wye wye 3 0.20460330576630303 0.33831852551664776 5 3 3 1 data/small eks wye 4 0.38139939387114097 0.13418874328430463 5 4 4 1 data/small wye pan 5 0.5732889198020006 0.8636244699032729 5 5 5 1 data/small pan eks 9999 0.267481232652199086 0.557077185510228001 5 6 1 2 data/small2 wye eks 10000 0.734806020620654365 0.884788571337605134 5 7 2 2 data/small2 pan wye 10001 0.870530722602517626 0.009854780514656930 5 8 3 2 data/small2 hat wye 10002 0.321507044286237609 0.568893318795083758 5 9 4 2 data/small2 pan zee 10003 0.272054845593895200 0.425789896597056627 5 10 5 2 data/small2 mlr --opprint put ' $nf = NF; $nr = NR; $fnr = FNR; $filenum = FILENUM; $filename = FILENAME' \ data/small data/small2 regularize$ mlr regularize --help Usage: mlr regularize For records seen earlier in the data stream with same field names in a different order, outputs them with field names in the previously encountered order. Example: input records a=1,c=2,b=3, then e=4,d=5, then c=7,a=6,b=8 output as a=1,c=2,b=3, then e=4,d=5, then a=6,c=7,b=8 rename$ mlr rename --help Usage: mlr rename [options] {old1,new1,old2,new2,...} Renames specified fields. Options: -r Treat old field names as regular expressions. "ab", "a.*b" will match any field name containing the substring "ab" or matching "a.*b", respectively; anchors of the form "^ab$", "^a.*b$" may be used. New field names may be plain strings, or may contain capture groups of the form "\1" through "\9". Wrapping the regex in double quotes is optional, but is required if you wish to follow it with 'i' to indicate case-insensitivity. -g Do global replacement within each field name rather than first-match replacement. Examples: mlr rename -r 'Date_[0-9]+,Date,' Rename all such fields to be "Date" mlr rename -r '"Date_[0-9]+",Date' Same mlr rename -r 'Date_([0-9]+).*,\1' Rename all such fields to be of the form 20151015 mlr rename -r '"name"i,Name' Rename "name", "Name", "NAME", etc. to "Name"
reorder$ mlr reorder --help Usage: mlr reorder [options] -f {a,b,c} Field names to reorder. -e Put specified field names at record end: default is to put them at record start. Examples: mlr reorder -f a,b sends input record "d=4,b=2,a=1,c=3" to "a=1,b=2,d=4,c=3". mlr reorder -e -f a,b sends input record "d=4,b=2,a=1,c=3" to "d=4,c=3,a=1,b=2".
sample$ mlr sample --help Usage: mlr sample [options] Reservoir sampling, optionally by category. -k {count} Required: number of records to output, total, or by group if using -g. -g {a,b,c} Optional: group-by-field names for samples. $ mlr --opprint sample -k 4 data/colored-shapes.dkvp color shape flag i u v w x purple triangle 0 90122 0.9986871176198068 0.3037738877233719 0.5154934457238382 5.365962021016529 red circle 0 3139 0.04835898233323954 -0.03964684310055758 0.5263660881848111 5.3758779366493625 orange triangle 0 67847 0.36746306902109926 0.5161574810505635 0.5176199566173642 3.1748088656576567 yellow square 1 33576 0.3098376725521097 0.8525628505287842 0.49774122460981685 4.494754378604669 $ mlr --opprint sample -k 4 data/colored-shapes.dkvp color shape flag i u v w x blue square 1 16783 0.09974385090654347 0.7243899920872646 0.5353718443278438 4.431057737383438 orange square 1 93291 0.5944176543007182 0.17744449786454086 0.49262281749172077 3.1548117990710653 yellow square 1 54436 0.5268161165014636 0.8785588662666121 0.5058773791931063 7.019185838783636 yellow square 1 55491 0.0025440267883102274 0.05474106287787284 0.5102729153751984 3.526301273728043 $ mlr --opprint sample -k 2 -g color data/colored-shapes.dkvp color shape flag i u v w x yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565 yellow square 1 917 0.8547010348386344 0.7356782810796262 0.4531511689924275 5.774541777078352 red circle 1 4000 0.05490416175132373 0.07392337815122155 0.49416101516594396 5.355725080701707 red square 0 87506 0.6357719216821314 0.6970867759393995 0.4940826462055272 6.351579417310387 purple triangle 0 14898 0.7800986870203719 0.23998073813992293 0.5014775988383656 3.141006771777843 purple triangle 0 151 0.032614487569017414 0.7346633365041219 0.7812143304483805 2.6831992610568047 green triangle 1 126 0.1513010528347546 0.40346767294704544 0.051213231883952326 5.955109300797182 green circle 0 17635 0.029856606049114442 0.4724542934246524 0.49529606749929744 5.239153910272168 blue circle 1 1020 0.414263129226617 0.8304946402876182 0.13151094520189244 4.397873687920433 blue triangle 0 220 0.441773289968473 0.44597731903759075 0.6329360666849821 4.3064608776550894 orange square 0 1885 0.8079311983747106 0.8685956833908394 0.3116410800256374 4.390864584500387 orange triangle 0 1533 0.32904497195507487 0.23168161807490417 0.8722623057355134 5.164071635714438 $ mlr --opprint sample -k 2 -g color then sort -f color data/colored-shapes.dkvp color shape flag i u v w x blue circle 0 215 0.7803586969333292 0.33146680638888126 0.04289047852629113 5.725365736377487 blue circle 1 3616 0.8548431579124808 0.4989623130006362 0.3339426415875795 3.696785877560498 green square 0 356 0.7674272008085286 0.341578843118008 0.4570224877870851 4.830320062215299 green square 0 152 0.6684429446914862 0.016056003736548696 0.4656148241291592 5.434588759225423 orange triangle 0 587 0.5175826237797857 0.08989091493635304 0.9011709461770973 4.265854207755811 orange triangle 0 1533 0.32904497195507487 0.23168161807490417 0.8722623057355134 5.164071635714438 purple triangle 0 14192 0.5196327866973567 0.7860928603468063 0.4964368415453642 4.899167143824484 purple triangle 0 65 0.6842806710360729 0.5823723856331258 0.8014053396013747 5.805148213865135 red square 1 2431 0.38378504852300466 0.11445015005595527 0.49355539228753786 5.146756570128739 red triangle 0 57097 0.43763430414406546 0.3355450325004481 0.5322349637512487 4.144267240289442 yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565 yellow square 1 158 0.41527900739142165 0.7118027080775757 0.4200799665161291 5.33279067554884 sort$ mlr sort --help Usage: mlr sort {flags} Flags: -f {comma-separated field names} Lexical ascending -n {comma-separated field names} Numerical ascending; nulls sort last -nf {comma-separated field names} Numerical ascending; nulls sort last -r {comma-separated field names} Lexical descending -nr {comma-separated field names} Numerical descending; nulls sort first Sorts records primarily by the first specified field, secondarily by the second field, and so on. Any records not having all specified sort keys will appear at the end of the output, in the order they were encountered, regardless of the specified sort order. Example: mlr sort -f a,b -nr x,y,z which is the same as: mlr sort -f a -f b -nr x -nr y -nr z $ mlr --opprint sort -f a -nr x data/small a b i x y eks pan 2 0.7586799647899636 0.5221511083334797 eks wye 4 0.38139939387114097 0.13418874328430463 pan pan 1 0.3467901443380824 0.7268028627434533 wye pan 5 0.5732889198020006 0.8636244699032729 wye wye 3 0.20460330576630303 0.33831852551664776 $ head -n 10 data/multicountdown.dat upsec=0.002,color=green,count=1203 upsec=0.083,color=red,count=3817 upsec=0.188,color=red,count=3801 upsec=0.395,color=blue,count=2697 upsec=0.526,color=purple,count=953 upsec=0.671,color=blue,count=2684 upsec=0.899,color=purple,count=926 upsec=0.912,color=red,count=3798 upsec=1.093,color=blue,count=2662 upsec=1.327,color=purple,count=917 $ head -n 20 data/multicountdown.dat | mlr --opprint sort -f color upsec color count 0.395 blue 2697 0.671 blue 2684 1.093 blue 2662 2.064 blue 2659 2.2880000000000003 blue 2647 0.002 green 1203 1.407 green 1187 1.448 green 1177 2.313 green 1161 0.526 purple 953 0.899 purple 926 1.327 purple 917 1.703 purple 908 0.083 red 3817 0.188 red 3801 0.912 red 3798 1.416 red 3788 1.587 red 3782 1.601 red 3755 1.832 red 3717 $ mlr sort -n x data/sort-missing.dkvp x=1 x=2 x=4 a=3 $ mlr sort -nr x data/sort-missing.dkvp x=4 x=2 x=1 a=3 stats1$ mlr stats1 --help Usage: mlr stats1 [options] Options: -a {sum,count,...} Names of accumulators: p10 p25.2 p50 p98 p100 etc. and/or one or more of: count Count instances of fields mode Find most-frequently-occurring values for fields; first-found wins tie sum Compute sums of specified fields mean Compute averages (sample means) of specified fields stddev Compute sample standard deviation of specified fields var Compute sample variance of specified fields meaneb Estimate error bars for averages (assuming no sample autocorrelation) min Compute minimum values of specified fields max Compute maximum values of specified fields -f {a,b,c} Value-field names on which to compute statistics -g {d,e,f} Optional group-by-field names -s Print iterative stats. Useful in tail -f contexts (in which case please avoid pprint-format output since end of input stream will never be seen). Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape Example: mlr stats1 -a count,mode -f size Example: mlr stats1 -a count,mode -f size -g shape Notes: * p50 is a synonym for median. * min and max output the same results as p0 and p100, respectively, but use less memory. * count and mode allow text input; the rest require numeric input. In particular, 1 and 1.0 are distinct text for count and mode. * When there are mode ties, the first-encountered datum wins.
stats2$ mlr stats2 --help Usage: mlr stats2 [options] -a {linreg-ols,corr,...} Names of accumulators: one or more of: linreg-pca Linear regression using principal component analysis linreg-ols Linear regression using ordinary least squares r2 Quality metric for linreg-ols (linreg-pca emits its own) logireg Logistic regression corr Sample correlation cov Sample covariance covx Sample-covariance matrix -f {a,b,c,d} Value-field name-pairs on which to compute statistics. There must be an even number of names. -g {e,f,g} Optional group-by-field names. -v Print additional output for linreg-pca. -s Print iterative stats. Useful in tail -f contexts (in which case please avoid pprint-format output since end of input stream will never be seen). --fit Rather than printing regression parameters, applies them to the input data to compute new fit fields. All input records are held in memory until end of input stream. Has effect only for linreg-ols, linreg-pca, and logireg. Only one of -s or --fit may be used. Example: mlr stats2 -a linreg-pca -f x,y Example: mlr stats2 -a linreg-ols,r2 -f x,y -g size,shape Example: mlr stats2 -a corr -f x,y
# Prepare input data: mlr filter '($x<.5 && $y<.5) || ($x>.5 && $y>.5)' data/medium > data/medium-squares # Do a linear regression and examine coefficients: mlr --ofs newline stats2 -a linreg-pca -f x,y data/medium-squares x_y_pca_m=1.014419 x_y_pca_b=0.000308 x_y_pca_quality=0.861354 # Option 1 to apply the regression coefficients and produce a linear fit: # Set x_y_pca_m and x_y_pca_b as shell variables: eval $(mlr --ofs newline stats2 -a linreg-pca -f x,y data/medium-squares) # In addition to x and y, make a new yfit which is the line fit, then plot # using your favorite tool: mlr --onidx put '$yfit='$x_y_pca_m'*$x+'$x_y_pca_b then cut -x -f a,b,i data/medium-squares \ | pgr -p -title 'linreg-pca example' -xmin 0 -xmax 1 -ymin 0 -ymax 1 # Option 2 to apply the regression coefficients and produce a linear fit: use --fit option mlr --onidx stats2 -a linreg-pca --fit -f x,y then cut -f a,b,i data/medium-squares \ | pgr -p -title 'linreg-pca example' -xmin 0 -xmax 1 -ymin 0 -ymax 1 ![]() $ head -n 10 data/multicountdown.dat upsec=0.002,color=green,count=1203 upsec=0.083,color=red,count=3817 upsec=0.188,color=red,count=3801 upsec=0.395,color=blue,count=2697 upsec=0.526,color=purple,count=953 upsec=0.671,color=blue,count=2684 upsec=0.899,color=purple,count=926 upsec=0.912,color=red,count=3798 upsec=1.093,color=blue,count=2662 upsec=1.327,color=purple,count=917 $ mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color then put '$donesec = -$upsec_count_pca_b/$upsec_count_pca_m' data/multicountdown.dat color green upsec_count_pca_m -32.756917 upsec_count_pca_b 1213.722730 upsec_count_pca_n 24 upsec_count_pca_quality 0.999984 donesec 37.052410 color red upsec_count_pca_m -37.367646 upsec_count_pca_b 3810.133400 upsec_count_pca_n 30 upsec_count_pca_quality 0.999989 donesec 101.963431 color blue upsec_count_pca_m -29.231212 upsec_count_pca_b 2698.932820 upsec_count_pca_n 25 upsec_count_pca_quality 0.999959 donesec 92.330514 color purple upsec_count_pca_m -39.030097 upsec_count_pca_b 979.988341 upsec_count_pca_n 21 upsec_count_pca_quality 0.999991 donesec 25.108529 step$ mlr step --help Usage: mlr step [options] -a {delta,rsum,...} Names of steppers: comma-separated, one or more of: delta Compute differences in field(s) between successive records ratio Compute ratios in field(s) between successive records rsum Compute running sums of field(s) between successive records counter Count instances of field(s) between successive records -f {a,b,c} Value-field names on which to compute statistics -g {d,e,f} Optional group-by-field names Computes values dependent on the previous record, optionally grouped by category.
$ each 10 uptime | mlr -p step -a delta -f 11 ... 20:08 up 36 days, 10:38, 5 users, load averages: 1.42 1.62 1.73 0.000000 20:08 up 36 days, 10:38, 5 users, load averages: 1.55 1.64 1.74 0.020000 20:08 up 36 days, 10:38, 7 users, load averages: 1.58 1.65 1.74 0.010000 20:08 up 36 days, 10:38, 9 users, load averages: 1.78 1.69 1.76 0.040000 20:08 up 36 days, 10:39, 9 users, load averages: 2.12 1.76 1.78 0.070000 20:08 up 36 days, 10:39, 9 users, load averages: 2.51 1.85 1.81 0.090000 20:08 up 36 days, 10:39, 8 users, load averages: 2.79 1.92 1.83 0.070000 20:08 up 36 days, 10:39, 4 users, load averages: 2.64 1.90 1.83 -0.020000 tac$ mlr tac --help Usage: mlr tac Prints records in reverse order from the order in which they were encountered.
tail$ mlr tail --help Usage: mlr tail [options] -n {count} Tail count to print; default 10 -g {a,b,c} Optional group-by-field names for tail counts Passes through the last n records, optionally by category.
top$ mlr top --help Usage: mlr top [options] -f {a,b,c} Value-field names for top counts. -g {d,e,f} Optional group-by-field names for top counts. -n {count} How many records to print per category; default 1. -a Print all fields for top-value records; default is to print only value and group-by fields. Requires a single value-field name only. --min Print top smallest values; default is top largest values. Prints the n records with smallest/largest values at specified fields, optionally by category.
uniq$ mlr uniq --help Usage: mlr uniq [options] -g {d,e,f} Group-by-field names for uniq counts -c Show repeat counts in addition to unique values Prints distinct values for specified field names. With -c, same as count-distinct.
Functions for filter and put$ mlr --help-all-functions abs (math: #args=1): Absolute value. acos (math: #args=1): Inverse trigonometric cosine. acosh (math: #args=1): Inverse hyperbolic cosine. asin (math: #args=1): Inverse trigonometric sine. asinh (math: #args=1): Inverse hyperbolic sine. atan (math: #args=1): One-argument arctangent. atan2 (math: #args=2): Two-argument arctangent. atanh (math: #args=1): Inverse hyperbolic tangent. cbrt (math: #args=1): Cube root. ceil (math: #args=1): Ceiling: nearest integer at or above. cos (math: #args=1): Trigonometric cosine. cosh (math: #args=1): Hyperbolic cosine. erf (math: #args=1): Error function. erfc (math: #args=1): Complementary error function. exp (math: #args=1): Exponential function e**x. expm1 (math: #args=1): e**x - 1. floor (math: #args=1): Floor: nearest integer at or below. invqnorm (math: #args=1): Inverse of normal cumulative distribution function. Note that invqorm(urand()) is normally distributed. log (math: #args=1): Natural (base-e) logarithm. log10 (math: #args=1): Base-10 logarithm. log1p (math: #args=1): log(1-x). logifit (math: #args=3): Given m and b from logistic regression, compute fit: $yhat=logifit($x,$m,$b). max (math: #args=2): max of two numbers; null loses min (math: #args=2): min of two numbers; null loses pow (math: #args=2): Exponentiation; same as **. qnorm (math: #args=1): Normal cumulative distribution function. round (math: #args=1): Round to nearest integer. roundm (math: #args=2): Round to nearest multiple of m: roundm($x,$m) is the same as round($x/$m)*$m sin (math: #args=1): Trigonometric sine. sinh (math: #args=1): Hyperbolic sine. sqrt (math: #args=1): Square root. tan (math: #args=1): Trigonometric tangent. tanh (math: #args=1): Hyperbolic tangent. urand (math: #args=0): Floating-point numbers on the unit interval. Int-valued example: '$n=floor(20+urand()*11)'. + (math: #args=2): Addition. - (math: #args=1): Unary minus. - (math: #args=2): Subtraction. * (math: #args=2): Multiplication. / (math: #args=2): Division. % (math: #args=2): Remainder; never negative-valued. ** (math: #args=2): Exponentiation; same as pow. =~ (boolean: #args=2): String (left-hand side) matches regex (right-hand side), e.g. '$name =~ "^a.*b$"'. !=~ (boolean: #args=2): String (left-hand side) does not match regex (right-hand side), e.g. '$name !=~ "^a.*b$"'. == (boolean: #args=2): String/numeric equality. Mixing number and string results in string compare. != (boolean: #args=2): String/numeric inequality. Mixing number and string results in string compare. > (boolean: #args=2): String/numeric greater-than. Mixing number and string results in string compare. >= (boolean: #args=2): String/numeric greater-than-or-equals. Mixing number and string results in string compare. < (boolean: #args=2): String/numeric less-than. Mixing number and string results in string compare. <= (boolean: #args=2): String/numeric less-than-or-equals. Mixing number and string results in string compare. && (boolean: #args=2): Logical AND. || (boolean: #args=2): Logical OR. ! (boolean: #args=1): Logical negation. strlen (string: #args=1): String length. sub (string: #args=3): Example: '$name=sub($name, "old", "new")' (replace once). gsub (string: #args=3): Example: '$name=gsub($name, "old", "new")' (replace all). tolower (string: #args=1): Convert string to lowercase. toupper (string: #args=1): Convert string to uppercase. . (string: #args=2): String concatenation. boolean (conversion: #args=1): Convert int/float/bool/string to boolean. float (conversion: #args=1): Convert int/float/bool/string to float. int (conversion: #args=1): Convert int/float/bool/string to int. string (conversion: #args=1): Convert int/float/bool/string to string. hexfmt (conversion: #args=1): Convert int to string, e.g. 255 to "0xff". fmtnum (conversion: #args=2): Convert int/float/bool to string using printf-style format string, e.g. "%06lld". systime (time: #args=0): Floating-point seconds since the epoch, e.g. 1440768801.748936. sec2gmt (time: #args=1): Formats seconds since epoch (integer part only) as GMT timestamp, e.g. sec2gmt(1440768801.7) = "2015-08-28T13:33:21Z". gmt2sec (time: #args=1): Parses GMT timestamp as integer seconds since epoch. sec2hms (time: #args=1): Formats integer seconds as in sec2hms(5000) = "01:23:20" sec2dhms (time: #args=1): Formats integer seconds as in sec2dhms(500000) = "5d18h53m20s" hms2sec (time: #args=1): Recovers integer seconds as in hms2sec("01:23:20") = 5000 dhms2sec (time: #args=1): Recovers integer seconds as in dhms2sec("5d18h53m20s") = 500000 fsec2hms (time: #args=1): Formats floating-point seconds as in fsec2hms(5000.25) = "01:23:20.250000" fsec2dhms (time: #args=1): Formats floating-point seconds as in fsec2dhms(500000.25) = "5d18h53m20.250000s" hms2fsec (time: #args=1): Recovers floating-point seconds as in hms2fsec("01:23:20.250000") = 5000.250000 dhms2fsec (time: #args=1): Recovers floating-point seconds as in dhms2fsec("5d18h53m20.250000s") = 500000.250000 To set the seed for urand, you may specify decimal or hexadecimal 32-bit numbers of the form "mlr --seed 123456789" or "mlr --seed 0xcafefeed". Miller's built-in variables are NF, NR, FNR, FILENUM, and FILENAME (awk-like) along with the mathematical constants PI and E. Data typesMiller’s input and output are all string-oriented: there is (as of August 2015 anyway) no support for binary record packing. In this sense, everything is a string in and out of Miller. During processing, field names are always strings, even if they have names like "3"; field values are usually strings. Field values’ ability to be interpreted as a non-string type only has meaning when comparison or function operations are done on them. And it is an error condition if Miller encounters non-numeric (or otherwise mistyped) data in a field in which it has been asked to do numeric (or otherwise type-specific) operations. Field values are treated as numeric for the following:
Null dataOne of Miller’s key features is its support for heterogeneous data. Accordingly, if you try to sort on field hostname when not all records in the data stream have a field named hostname, it is not an error (although you could pre-filter the data stream using mlr having-fields --at-least hostname then sort ...). Rather, records lacking one or more sort keys are simply output contiguously by mlr sort. Field values may also be null by being specified with present key but empty value: e.g. sending x=,y=2 to mlr put '$z=$x+$y'. Rules for null-handling:
|