# FreeBSD and UTF-8 Problems



## npgm (Sep 17, 2013)

Hello.

I've added 

```
:charset=UTF-8:\
:lang=en_US.UTF-8:
```
  to my login.conf file and now my locale is all set to "en_US.UTF-8".

However, I've noticed that the man pages are no longer displaying correctly and now that I started working with more files, I've also noticed that non-ASCII characters don't display correctly. What could be causing this?

When I do `man ls`, I see this.

```
LS(1)			FreeBSD General Commands Manual 		 LS(1)

NAME
     ls âˆÃ‡Ã¶ list directory contents
```

In my text files, vi is displaying letters in hex, I think. For example the word Â«bÃºsquedasÂ» is displayed as Â«b\xc3\xbasquedasÂ» instead.

It seems like there is a character encoding conflict. I know I save my stuff using UTF-8 encoding without BOM outside of FreeBSD. My locale in FreeBSD is set to UTF-8. So I'm not sure what's going on here. Has anyone else experienced this?


----------



## SirDice (Sep 17, 2013)

As far as I know syscons(4) still has problems with UTF-8. That's one of the reasons why it's in dire need of an update. 

Not sure what the current status is but there was some work being done for a replacement: https://wiki.freebsd.org/Newcons


----------



## xtaz (Sep 17, 2013)

I experienced this myself in a normal PuTTY SSH terminal, so it has nothing to do with syscons. I "fixed" it by aliasing `man` to `man -o` which forces the use of non localized man pages.


----------



## npgm (Nov 9, 2013)

Is there another level of abstraction that I can add to get Unicode characters to display? I've heard something about ROXTerm and it being able to support Unicode, but honestly, I'm not even sure what that is or how (or if) it differs from SH.


----------



## zodias (Nov 11, 2013)

xtaz said:
			
		

> I experienced this myself in a normal PuTTY SSH terminal, so it has nothing to do with syscons. I "fixed" it by aliasing `man` to `man -o` which forces the use of non localized man pages.



I do it through PuTTY settings: Window -> Translation and choose UTF-8.


----------



## kpa (Nov 11, 2013)

A couple of different things have to happen to have proper UTF-8 support on a terminal. First the application should know that it should output UTF-8 encoded strings, this is done with the locale(1) settings. Secondly the terminal program (the console on the machine or for example PuTTY running on remote machine) must know to interpret the input it gets from the remote end as UTF-8 and also have the proper fonts in Unicode encoding to be able to display the UTF-8 encoded Unicode characters.


----------



## xtaz (Nov 11, 2013)

It's got nothing to do with the PuTTY settings etc. The locale is properly set to UTF-8, PuTTY is also properly set to UTF-8. Everything else like typing Euro/Chinese characters etc. in text editors or my IRC client works perfectly fine. It's literally just that the man pages don't get formatted properly. Which as I said I "fixed" by stopping it trying to use localised man pages with the `man -o` command. Although this is something that I did several years ago. It may be fixed these days. I haven't actually tested it recently. In fact I've just unaliased the `man -o` alias and taken a look at several pages, and they all look fine. So it probably was fixed at some point. I'm currently running 10.0-BETA3.


----------

