# Unix utils are so awesome



## YZMSQ (Dec 8, 2012)

Hi,
Just come across this awesome article, and cannot help to practise and share them:e
P.S.: Has anyone here done some data mining work on FreeBSD?:e


----------



## UNIXgod (Dec 8, 2012)

Nice info. Lots of fun filtering examples.

What sort of data mining are you specifically interested in?


----------



## jrm@ (Dec 8, 2012)

In the second example, he uses column() to pretty print the pasted groups so the columns align properly.  This only works if you supply the $'\t' for the -s switch.  Why doesn't this work with just '\t'?  In other words what does the $ mean in front of '\t'?


----------



## wblock@ (Dec 8, 2012)

Interesting:
`% man sh | less -p Dollar-Single`


----------



## UNIXgod (Dec 8, 2012)

wblock@ said:
			
		

> Interesting:
> `% man sh | less -p Dollar-Single`





			
				jrm said:
			
		

> In the second example, he uses column() to pretty print the pasted groups so the columns align properly.  This only works if you supply the $'\t' for the -s switch.  Why doesn't this work with just '\t'?  In other words what does the $ mean in front of '\t'?



So from what I gather $'\t' is expanding it when it's parsed initially before it's streamed into column.

example with echo:

```
%  echo before$'\t\t'after 
before		after
```

of course echo works without the currency sigil. Column probably has the limitation so the issue is delegated to the shell( and the user) to deal with the actual parsing than the program. Interesting workaround. Learned something new today.


----------



## wblock@ (Dec 9, 2012)

But echo(1) does not do backslash expansions...  printf(1) does, though.


----------



## UNIXgod (Dec 9, 2012)

wblock@ said:
			
		

> But echo(1) does not do backslash expansions...  printf(1) does, though.



I should get used to using printf more often. I guess a simpler way to explain it is that when usings $'\t' the shell basically converts it to the double quote "\t"

so instead of this:


```
column -s $'\t'
```

it treats it like this:

```
column -s [B]"[/B]\t[B]"[/B]
```


----------



## wblock@ (Dec 9, 2012)

I think the second will not work as expected because the "\t" does not get translated by the shell to a real tab.  Unless column(1) does those translations.  Probably the backslash will be ignored and it will use a t as a column delimiter.

The first example could be entered with a literal tab:

```
column -s "[color="Red"]{ctrl-v}{tab}[/color]"   (actual ctrl-v and tab keystrokes)
```

But the dollar-single notation lets a tab be visible.


----------



## YZMSQ (Dec 9, 2012)

UNIXgod said:
			
		

> Nice info. Lots of fun filtering examples.
> 
> What sort of data mining are you specifically interested in?


I did do some data mining work when I was busying with my graduate thesis on "Data Mining based Internet Application Classification", at that time I learned and used awk and plain shell script to manipulate raw data before I threw them into Weka to build some perfect mathematic models. It's really interesting.


----------



## jrm@ (Dec 9, 2012)

wblock@ said:
			
		

> But echo(1) does not do backslash expansions...  printf(1) does, though.



Some interesting points from the POSIX specification of echo are:



> If the first operand is -n, or if any of the operands contain a backslash ( '\' ) character, the results are implementation-defined.
> On XSI-conformant systems escape characters such as '\t' will print a TAB.
> The printf utility can be used portably to emulate any of the traditional behaviors of the echo utility as follows...
> New applications are encouraged to use printf instead of echo.




```
% /bin/echo before "\t\t" after
before \t\t after

tcsh % echo before "\t\t" after
before \t\t after

sh % echo before "\t\t" after
before \t\t after

eshell % echo before "\t\t" after
("before" "\\t\\t" "after")

eshell % echo "before \t\t after"
before \t\t after

zsh % echo before "\t\t" after
before           after
```


----------



## UNIXgod (Dec 9, 2012)

YZMSQ said:
			
		

> I did do some data mining work when I was busying with my graduate thesis on "Data Mining based Internet Application Classification", at that time I learned and used awk and plain shell script to manipulate raw data before I threw them into Weka to build some perfect mathematic models. It's really interesting.



Nice. I've also done some with awk. Nothing statistical but scraped some data recursively from page to page. Been toying with mechanize. Not as fun as awk but integrates well inside ruby applications.

I wouldn't mind learning more about post requests. I know we can painfully do them through telnet or a third party application like curl but wouldn't mind knowing if there is other ways without third-party applications or libraries.


----------



## UNIXgod (Dec 9, 2012)

jrm said:
			
		

> Some interesting points from the POSIX specification of echo are:
> 
> 
> 
> ...



I was in zsh. I see what happened now. 

Here is with ash using printf then echo:


```
$ printf before"\t"after"\n"
before	after
$ printf before$'\t'after$'\n'
before	after
$ echo before$'\t'after$'\n'
before	after

$ echo before"\t"after"\n"
before\tafter\n
```


----------

