# msdosfs + utf8: cannot read/write files with utf8 in names



## nekoexmachina (Mar 6, 2010)

Hello again, forums!
Got a flash-player, which i've used lots of time and wrote files from linux with mount command:
`mount -o iocharset=utf8 /dev/disk /mount/path`
In bsd, i've tried to just mount it raw, to mount it with -D UTF-8 -L en_US.UTF-8, tried some msdos-encodings - none of them seem to work.
How do i mount it properly?


----------



## graudeejs (Mar 6, 2010)

I believe Fat doesn't support UTF-8


----------



## nekoexmachina (Mar 7, 2010)

>I believe Fat doesn't support UTF-8 
Well, err..
My player & linux doesn't believe this info.
I mean, linux writes/reads all range of utf8 stuff (some russian songs, some japanese songs, etc, with song titles in filenames), and so does my player.


----------



## graudeejs (Mar 7, 2010)

In that case I think it was utf-16
http://en.wikipedia.org/wiki/File_Allocation_Table


----------



## nekoexmachina (Mar 7, 2010)

Ooops. 
Thanks, but:
[CMD=]iconv -l|grep UTF-16[/cmd]

```
UTF-16
UTF-16BE
UTF-16LE
```

[CMD=]sudo mount_msdosfs -L en_US.UTF-8 -D UTF-16 /dev/da1s1 ~/flash[/cmd]

```
mount_msdosfs: msdosfs_iconv: Illegal byte sequence
```

Trouble is with msdosfs_iconv bmm, not with flash, because i've tried a completely empty one with the same result. What could i do?


----------



## bschmidt (Mar 7, 2010)

I tried with `# -L en_US.UTF-8 -D UTF-8` which works for me. Might I ask for your `$ locale`? Mine looks like:

```
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=en_US.UTF-8
```


----------



## nekoexmachina (Mar 7, 2010)

```
LANG=C
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=en_US.UTF-8
```


----------



## nekoexmachina (Mar 9, 2010)

well, what i've got right now, just tested it again: old file names are '???', and i can't read/modify any of them
new files are being written OK, but the player itself shows me, err, same '???' for filenames with new files, but it can read files, it can read tags - so probably it's ok - if could not be fixed by an average user with almost no programming skills.


----------



## graudeejs (Mar 9, 2010)

Does player shows wrong filenames or IDv3 tags?
perhaps file IDv3 tags are in different encoding


----------



## bschmidt (Mar 9, 2010)

Another question, where do you set the LC_* variables? Just for the shell your using (.profile, ..) or through login.conf? I ask that, because if you use xdm, kdm, .. your applications still use a different locale.


----------



## nekoexmachina (Mar 9, 2010)

*killasmurf86*
Yes, player shows _tags_, which in utf8 properly, but not filenames. Dunno why, because in linux i've mounted just same as here (with utf8 encoding).
*bschmidt*
I do not use x/k/g/whateverdm, LC_ALL & LANG are set in ~/.zshrc.


----------



## bschmidt (Mar 9, 2010)

Please try with LC* set via /etc/login.conf and 
	
	



```
options         TEKEN_UTF8
```
 in your kernel configuration. I'm pretty sure that this will make a difference.


----------



## nekoexmachina (Mar 9, 2010)

TEKEN_UTF8 is for tty unicode support, as i know. How could that help? I'll try it some time later, but i think that there should be some other way, and i'm sure that this doesn't have anything to do with mounting.
And, the last fix for the thread - message #11 is little bit wrong:

```
#sudo mount_msdosfs -D UTF-8 -L en_US.UTF-8 /dev/da0s1 ./flash
$cd flash/Music
$touch Ñ„Ñ‹Ð°
$ls Ñ„Ñ‹Ð°
Ñ„Ñ‹Ð°
$touch åŒäººã‚½ãƒ•ãƒˆ
touch: åŒäººã‚½ãƒ•ãƒˆ: Invalid argument
```
and that goes for, err, everything else than cyrillic and english.


----------



## alexandre (Aug 20, 2010)

Hello,

I have a similar problem with a hard disk formatted in FAT32, which contains folders and files with japanese titles. Here is a screenshot of the problem :




I don't have problem with french accents, nor with japanese input or anything else on my computer, just with this FAT32 disk. I also have followed the handbook (http://www.freebsd.org/doc/handbook/using-localization.html), but nothing works. And I have finally tried with "options         TEKEN_UTF8" in the kernel, but it didn't work for me.

Here are some files which may be useful if someone can help me :

~/.login_conf

```
# $FreeBSD$
#
# see login.conf(5)
#
me:\
    :lang=ja_JP.UTF-8:\
    :setenv=LC_ALL=ja_JP.UTF-8:\
    :setenv=LC_COLLATE=ja_JP.UTF-8:\ 
    :setenv=LC_CTYPE=ja_JP.UTF-8:\
    :setenv=LC_MESSAGES=ja_JP.UTF-8:\
    :setenv=LC_MONETARY=ja_JP.UTF-8:\
    :setenv=LC_NUMERIC=ja_JP.UTF-8:\
    :setenv=LC_TIME=fr_FR.UTF-8:\
    :charset=UTF-8:\
    :xmodifiers="@im=ibus": #Set ibus as the XIM Input Server
```

/etc/login.conf

```
# login.conf - login class capabilities database.
#
# Remember to rebuild the database after each change to this file:
#
#	cap_mkdb /etc/login.conf
#
# This file controls resource limits, accounting limits and
# default user environment settings.
#
# $FreeBSD$
#

# Default settings effectively disable resource limits, see the
# examples below for a starting point to enable them.

# defaults
# These settings are used by login(1) by default for classless users
# Note that entries like "cputime" set both "cputime-cur" and "cputime-max"
#
# Note that since a colon ':' is used to separate capability entries,
# a \c escape sequence must be used to embed a literal colon in the
# value or name of a capability (see the ``CGETNUM AND CGETSTR SYNTAX
# AND SEMANTICS'' section of getcap(3) for more escape sequences).

language_name|Account Type Description:\
    :tc=default:\
    :lang=ja_JP.UTF-8:\
    :setenv=LC_ALL=ja_JP.UTF-8:\
#    :setenv=LC_COLLATE=ja_JP.UTF-8:\ 
#    :setenv=LC_CTYPE=ja_JP.UTF-8:\
#    :setenv=LC_MESSAGES=ja_JP.UTF-8:\
#    :setenv=LC_MONETARY=ja_JP.UTF-8:\
#    :setenv=LC_NUMERIC=ja_JP.UTF-8:\
#    :setenv=LC_TIME=fr_FR.UTF-8:\
    :charset=UTF-8:\
#    :xmodifiers="@im=ibus": #Set ibus as the XIM Input Server

default:\
	:passwd_format=md5:\
	:copyright=/etc/COPYRIGHT:\
	:welcome=/etc/motd:\
	:setenv=MAIL=/var/mail/$,BLOCKSIZE=K,FTP_PASSIVE_MODE=YES:\
	:path=/sbin /bin /usr/sbin /usr/bin /usr/games /usr/local/sbin /usr/local/bin ~/bin:\
	:nologin=/var/run/nologin:\
	:cputime=unlimited:\
	:datasize=unlimited:\
	:stacksize=unlimited:\
	:memorylocked=unlimited:\
	:memoryuse=unlimited:\
	:filesize=unlimited:\
	:coredumpsize=unlimited:\
	:openfiles=unlimited:\
	:maxproc=unlimited:\
	:sbsize=unlimited:\
	:vmemoryuse=unlimited:\
	:swapuse=unlimited:\
	:pseudoterminals=unlimited:\
	:priority=0:\
	:ignoretime@:\
	:umask=022:


#
# A collection of common class names - forward them all to 'default'
# (login would normally do this anyway, but having a class name
#  here suppresses the diagnostic)
#
standard:\
	:tc=default:
xuser:\
	:tc=default:
staff:\
	:tc=default:
daemon:\
	:tc=default:
news:\
	:tc=default:
dialer:\
	:tc=default:

#
# Root can always login
#
# N.B.  login_getpwclass(3) will use this entry for the root account,
#       in preference to 'default'.
root:\
	:ignorenologin:\
	:tc=default:

#
# Russian Users Accounts. Setup proper environment variables.
#
russian|Russian Users Accounts:\
	:charset=KOI8-R:\
	:lang=ru_RU.KOI8-R:\
	:tc=default:


######################################################################
######################################################################
##
## Example entries
##
######################################################################
######################################################################

## Example defaults
## These settings are used by login(1) by default for classless users
## Note that entries like "cputime" set both "cputime-cur" and "cputime-max"
#
#default:\
#	:cputime=infinity:\
#	:datasize-cur=22M:\
#	:stacksize-cur=8M:\
#	:memorylocked-cur=10M:\
#	:memoryuse-cur=30M:\
#	:filesize=infinity:\
#	:coredumpsize=infinity:\
#	:maxproc-cur=64:\
#	:openfiles-cur=64:\
#	:priority=0:\
#	:requirehome@:\
#	:umask=022:\
#	:tc=auth-defaults:
#
#
##
## standard - standard user defaults
##
#standard:\
#	:copyright=/etc/COPYRIGHT:\
#	:welcome=/etc/motd:\
#	:setenv=MAIL=/var/mail/$,BLOCKSIZE=K:\
#	:path=~/bin /bin /usr/bin /usr/local/bin:\
#	:manpath=/usr/share/man /usr/local/man:\
#	:nologin=/var/run/nologin:\
#	:cputime=1h30m:\
#	:datasize=8M:\
#	:vmemoryuse=100M:\
#	:stacksize=2M:\
#	:memorylocked=4M:\
#	:memoryuse=8M:\
#	:filesize=8M:\
#	:coredumpsize=8M:\
#	:openfiles=24:\
#	:maxproc=32:\
#	:priority=0:\
#	:requirehome:\
#	:passwordtime=90d:\
#	:umask=002:\
#	:ignoretime@:\
#	:tc=default:
#
#
##
## users of X (needs more resources!)
##
#xuser:\
#	:manpath=/usr/share/man /usr/local/man:\
#	:cputime=4h:\
#	:datasize=12M:\
#	:vmemoryuse=infinity:\
#	:stacksize=4M:\
#	:filesize=8M:\
#	:memoryuse=16M:\
#	:openfiles=32:\
#	:maxproc=48:\
#	:tc=standard:
#
#
##
## Staff users - few restrictions and allow login anytime
##
#staff:\
#	:ignorenologin:\
#	:ignoretime:\
#	:requirehome@:\
#	:accounted@:\
#	:path=~/bin /bin /sbin /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin:\
#	:umask=022:\
#	:tc=standard:
#
#
##
## root - fallback for root logins
##
#root:\
#	:path=~/bin /bin /sbin /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin:\
#	:cputime=infinity:\
#	:datasize=infinity:\
#	:stacksize=infinity:\
#	:memorylocked=infinity:\
#	:memoryuse=infinity:\
#	:filesize=infinity:\
#	:coredumpsize=infinity:\
#	:openfiles=infinity:\
#	:maxproc=infinity:\
#	:memoryuse-cur=32M:\
#	:maxproc-cur=64:\
#	:openfiles-cur=1024:\
#	:priority=0:\
#	:requirehome@:\
#	:umask=022:\
#	:tc=auth-root-defaults:
#
#
##
## Settings used by /etc/rc
##
#daemon:\
#	:coredumpsize@:\
#	:coredumpsize-cur=0:\
#	:datasize=infinity:\
#	:datasize-cur@:\
#	:maxproc=512:\
#	:maxproc-cur@:\
#	:memoryuse-cur=64M:\
#	:memorylocked-cur=64M:\
#	:openfiles=1024:\
#	:openfiles-cur@:\
#	:stacksize=16M:\
#	:stacksize-cur@:\
#	:tc=default:
#
#
##
## Settings used by news subsystem
##
#news:\
#	:path=/usr/local/news/bin /bin /sbin /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin:\
#	:cputime=infinity:\
#	:filesize=128M:\
#	:datasize-cur=64M:\
#	:stacksize-cur=32M:\
#	:coredumpsize-cur=0:\
#	:maxmemorysize-cur=128M:\
#	:memorylocked=32M:\
#	:maxproc=128:\
#	:openfiles=256:\
#	:tc=default:
#
#
##
## The dialer class should be used for a dialup PPP/SLIP accounts
## Welcome messages/news suppressed
##
#dialer:\
#	:hushlogin:\
#	:requirehome@:\
#	:cputime=unlimited:\
#	:filesize=2M:\
#	:datasize=2M:\
#	:stacksize=4M:\
#	:coredumpsize=0:\
#	:memoryuse=4M:\
#	:memorylocked=1M:\
#	:maxproc=16:\
#	:openfiles=32:\
#	:tc=standard:
#
#
##
## Site full-time 24/7 PPP/SLIP connections
## - no time accounting, restricted to access via dialin lines
##
#site:\
#	:ignoretime:\
#	:passwordtime@:\
#	:refreshtime@:\
#	:refreshperiod@:\
#	:sessionlimit@:\
#	:autodelete@:\
#	:expireperiod@:\
#	:graceexpire@:\
#	:gracetime@:\
#	:warnexpire@:\
#	:warnpassword@:\
#	:idletime@:\
#	:sessiontime@:\
#	:daytime@:\
#	:weektime@:\
#	:monthtime@:\
#	:warntime@:\
#	:accounted@:\
#	:tc=dialer:\
#	:tc=staff:
#
#
##
## Example standard accounting entries for subscriber levels
##
#
#subscriber|Subscribers:\
#	:accounted:\
#	:refreshtime=180d:\
#	:refreshperiod@:\
#	:sessionlimit@:\
#	:autodelete=30d:\
#	:expireperiod=180d:\
#	:graceexpire=7d:\
#	:gracetime=10m:\
#	:warnexpire=7d:\
#	:warnpassword=7d:\
#	:idletime=30m:\
#	:sessiontime=4h:\
#	:daytime=6h:\
#	:weektime=40h:\
#	:monthtime=120h:\
#	:warntime=4h:\
#	:tc=standard:
#
#
##
## Subscriber accounts. These accounts have their login times
## accounted and have access limits applied.
##
#subppp|PPP Subscriber Accounts:\
#	:tc=dialer:\
#	:tc=subscriber:
#
#
#subslip|SLIP Subscriber Accounts:\
#	:tc=dialer:\
#	:tc=subscriber:
#
#
#subshell|Shell Subscriber Accounts:\
#	:tc=subscriber:
#
##
## If you want some of the accounts to use traditional UNIX DES based
## password hashes.
##
#des_users:\
#	:passwd_format=des:\
#	:tc=default:
```

I have followed the first method of the handbook, so I suppose showing you /etc/profile,  /etc/csh.login, and ~/.xinitrc might not be relevant. But I have skipped "Changing Login Classes with vipw(8)" and "Changing Login Classes with pw(8)" and set defaultclass with /etc/adduser.conf :

/etc/adduser.conf

```
defaultclass = ja_JP.UTF-8
```


----------



## alexandre (Aug 20, 2010)

I have forgotten one last thing, sorry ! Here is what I have in /system/storage/default_options/vfat, in gconf-editor (I use gnome with Freebsd 8.1) :


----------

