# No newline in stream redirection



## cootue (Feb 21, 2010)

Hi everybody,
First of all I'm sorry for probably really trivial issue.
I have problem with curl not adding newlines to it's log file which makes it quite hard to read...
code looks like this:


```
curl -o file.dat http://damain.com/file.dat 2> logfile.log &
```

I'm not sure where the problem lies, as on debian each message gets own line...

Could anybody advise me how to fix this?


----------



## sixtydoses (Feb 22, 2010)

I have no problems with the command. You might need to use the `-w` option (write-out) combine with `n` (newline).

http://curl.haxx.se/docs/manpage.html


----------



## cootue (Feb 22, 2010)

_-w_ is for displaying one message after download has been completed and I'd like to be logging progress in between as it's "infinite" file which download I'm stopping with _kill_.


----------



## DutchDaemon (Feb 22, 2010)

Not sure if it helps, but try piping it through logger(1).


----------



## J65nko (Feb 22, 2010)

If you are using a sh compatible shell try
	
	



```
curl -o file.dat http://damain.com/file.dat [color=blue]> logfile.log 2>&1[/color] &
```


----------



## cootue (Feb 22, 2010)

J65nko,
nope, result is same.

I noticed that in fact it's adding something at end of each line, it shows as *^M* in editors.
vi, less, and mcedit don't recognize it as new line.
But nano does!
It's kind of workaround, but it'd be cool to know how to add "proper" newline there...


----------



## J65nko (Feb 22, 2010)

Actually the redirection command that I suggested redirects stderr (fd 2) to stdout (fd 1).
If it adds a CNTRL-M or "\r", you can use tr(1) to replace it with a Unix newline, a "\n'


```
curl -o file.dat http://damain.com/file.dat 2>&1 | tr '\r' '\n' >logfile &
```

BTW are you aware that in FreeBSD fetch(1) and ftp(1) can do auto fetching too?


----------



## SirDice (Feb 22, 2010)

cootue said:
			
		

> I noticed that in fact it's adding something at end of each line, it shows as *^M* in editors.


That's a "carriage return" (ascii 0x0D). The carriage return just sets the cursor back to the beginning of the line, linefeed actually advances one line. Similar to an old typewriter. Most *nix however only use "Line Feed" (ascii 0x0a) for a new line. Windows uses CR+LF.

As you've noticed, this pops up with 'progress' type output. What this actually does is print a line, do a carriage return, print next line, carriage return etc. This has the effect of the second print overwriting the first.


----------



## cootue (Feb 22, 2010)

curl without '-o' argument outputs file to stdout, so "progress info" is in stderr stream, right?

Trick with tr leaves logfile empty, probably cause it lost data somewhere between streams or something, ehh.
I like java because you don't have to use magic to make things work.

At first I had wget in my script, but I didn't like using external variables (or how 'http_proxy' is called) for proxy support. I changed it to curl for its '-x' argument, but when it comes to logging this program seriously lacks key features. I think I'll try to find yet another alternative.

Thanks guys for helping me.


----------



## DutchDaemon (Feb 22, 2010)

Have you tried logger yet?


----------



## cootue (Feb 23, 2010)

I've tried it, but curl outputting progress to stderr makes it too much for me.

I went back searching for right tool, checked lftp, aria2, fetch, pavuk, but none of those got what I need. Then I found that there's 'http_proxy' option in .wgetrc, and with '-e' it's all I'll ever need.

Over and out from contented wget user!


----------

