# Help with Missing Charsets and Tor Timming Out



## Puffles_the_Dragon (Nov 9, 2018)

Hello, I am new to FreeBSD and BSD in general. I am loving FreeBSD a lot so far. I installed my system and have been tinkering with it a little bit, and have a few things I haven't been able to reslove myself. First thing, my system langauge is english as that is my native language however I have a lot of music in Chinese, japanese etc. the files and metadata for said files are written in thei respective lanaguges. These files don't show their characters however, but rather question marks (??? those). I am confident this is due to missing charsets as I had the same issue on Arch Linux. I followed this: https://cooltrainer.org/a-freebsd-desktop-howto/#install-fonts, however after rebooting after installing all those fonts the issue wasn't fixed. How would I reslove this issue? Second issue, I use tor for some things and have it also running on my other laptop so I know this issue isn't due to it not working on my network. I have installed tor and set it to run at boot as well as did the following:

then vi /etc/sysctl.conf:

        # prevent traffic analysis that exploits sequential IP IDs
        net.inet.ip.random_id=1

After that was all done I setup my many applications to run through it, they all seem to time out when they try to connect to the network through tor. How do I fix this? Thank you for your time.


----------



## SirDice (Nov 9, 2018)

The Howto section is not for asking how to do things.

Select the right forum for new threads

And I suggest splitting your question, it's usually not very effective if you ask for help on two completely different subjects, the thread will get confusing with different answers intermingled.


----------



## Puffles_the_Dragon (Nov 9, 2018)

SirDice said:


> The Howto section is not for asking how to do things.
> 
> Select the right forum for new threads
> 
> And I suggest splitting your question, it's usually not very effective if you ask for help on two completely different subjects, the thread will get confusing with different answers intermingled.



I'm so sorry about that. Thank you for moving it for me. I will stick to one topic per post in the feature.


----------



## Phishfry (Nov 10, 2018)

The first place you need to start is with localization for your language.
https://www.freebsd.org/doc/handbook/using-localization.html


----------



## Puffles_the_Dragon (Nov 14, 2018)

Phishfry said:


> The first place you need to start is with localization for your language.
> https://www.freebsd.org/doc/handbook/using-localization.html


I have been going through that and apologize, but I am having a hard time understanding it. So for example, say I want my main langauge to be US english, but I have MP3s that are in Chinese. These MP3s show ? marks where the Chinese should be, how do I have it show the Chinese? In case it helps, here is everything I have done to my system. Every command and every package installed: https://pastebin.com/eRuDeGU2.


----------



## yuripv (Nov 14, 2018)

The tags showing "?" doesn't really have anything to do with *language* or using correct fonts (if it was fonts problem, you would see boxes with numbers inside showing the character code instead), those are (most likely) in different *codeset* (encoding), so you need a tool translating those tags from say Big5 or GB18030 to UTF-8 (that's what you use, right?).  One obvious example is iconv(1), but I can't give any more advice here as I don't know what or how you use those exactly.

One thing you *could* try is run the program that is showing "?" with an encoding frequently used with Chinese (you need to guess here), e.g. `LC_CTYPE=zh_CN.GB18030 program` or `LC_CTYPE=zh_CN.Big5 program` (there's no guarantee it will help, or if it's useful).


----------



## Puffles_the_Dragon (Nov 14, 2018)

yuripv said:


> The tags showing "?" doesn't really have anything to do with *language* or using correct fonts (if it was fonts problem, you would see boxes with numbers inside showing the character code instead), those are (most likely) in different *codeset* (encoding), so you need a tool translating those tags from say Big5 or GB18030 to UTF-8 (that's what you use, right?).  One obvious example is iconv(1), but I can't give any more advice here as I don't know what or how you use those exactly.
> 
> One thing you *could* try is run the program that is showing "?" with an encoding frequently used with Chinese (you need to guess here), e.g. `LC_CTYPE=zh_CN.GB18030 program` or `LC_CTYPE=zh_CN.Big5 program` (there's no guarantee it will help, or if it's useful).


Is there a way to transfer the encoding to utf8 on those files?


----------



## yuripv (Nov 14, 2018)

Puffles_the_Dragon said:


> Is there a way to transfer the encoding to utf8 on those files?


To answer this question you need to specify where exactly you are seeing the "?" stuff -- is it filenames, or mp3 tags, or...?


----------



## Deleted member 30996 (Nov 14, 2018)

At least you can see yours... I have 60 .mp3's with Russian character set that I cannot see in the folder, cannot delete the folder on the USB stick as root, or move it to the trashcan as root. I do all that kind of work by invoking x11-fm/xfe as root through the terminal and should be done with it.

I have installed russian/p5-Lingua-RU-Charset and still cannot even see the files in the folder on the USB stick. I do have x11-fonts/font-screen-cyrillic installed, the prior is supposed to detect and convert it.  When I initially transferred the folder to the HDD it transferred all the English characters worded files but gave an error on these.





I will eventually remove everything else I have stored on the stick and `dd` it to solve the issue, as I'm more interested in that than the mp3's but have not been able to fix it myself to this point.


----------



## Puffles_the_Dragon (Nov 14, 2018)

yuripv said:


> To answer this question you need to specify where exactly you are seeing the "?" stuff -- is it filenames, or mp3 tags, or...?


I posted this comment already, but think I posted it to the wrong person. If this now shows twice I apologize, I am not use to how the FreeBSD forum works just yet. When I get home, I'll upload an image of my issue. It is hard to describe at the moment and I feel a visual will help


----------



## Puffles_the_Dragon (Nov 14, 2018)

Trihexagonal said:


> At least you can see yours... I have 60 .mp3's with Russian character set that I cannot see in the folder, cannot delete the folder on the USB stick as root, or move it to the trashcan as root. I do all that kind of work by invoking x11-fm/xfe as root through the terminal and should be done with it.
> 
> I have installed russian/p5-Lingua-RU-Charset and still cannot even see the files in the folder on the USB stick. I do have x11-fonts/font-screen-cyrillic installed, the prior is supposed to detect and convert it.  When I initially transferred the folder to the HDD it transferred all the English characters worded files but gave an error on these.
> 
> ...


I haven't actually been able to move my music over to my machine from the USB it's on either. It keeps giving an error about the ? marks. I will upload an image of the issue in hopes it helps. Definitely feel your pain though.


----------



## yuripv (Nov 14, 2018)

Meanwhile, just to show what I mean, here's a simple example:

1. Create some files with cyrillic in names (my current locale is C.UTF-8):
thor:yuripv:~/tst$ touch тест1
thor:yuripv:~/tst$ touch файл2
thor:yuripv:~/tst$ touch хлам3
thor:yuripv:~/tst$ ll
total 2
13619 -rw-r--r--  1 yuripv  staff  uarch 0 Nov 14 20:49 тест1
13620 -rw-r--r--  1 yuripv  staff  uarch 0 Nov 14 20:49 файл2
13621 -rw-r--r--  1 yuripv  staff  uarch 0 Nov 14 20:49 хлам3

They show up correctly in the listing, good.

2. Now convert the filenames to KOI8-R (was widely used for Russian previously):
thor:yuripv:~/tst$ for i in *; do mv $i $(echo $i | iconv -f utf8 -t koi8-r); done
thor:yuripv:~/tst$ ll
total 2
13620 -rw-r--r--  1 yuripv  staff  uarch 0 Nov 14 20:49 ????2
13621 -rw-r--r--  1 yuripv  staff  uarch 0 Nov 14 20:49 ????3
13619 -rw-r--r--  1 yuripv  staff  uarch 0 Nov 14 20:49 ????1

Not so good, byte sequences that are perfectly legal in KOI8-R are illegal in UTF-8, so ls can NOT display the filenames now.

You guys could use the same method, but in reverse direction, translating from the encodings the filenames are now in to UTF-8, just need to know which one that is  i.e.:

$ for i in *; do mv $i $(echo $i | iconv -f <CURRENT-ENCODING> -t utf8); done


----------



## yuripv (Nov 14, 2018)

Trihexagonal that russian/p5-Lingua-RU-Charset port is just a perl module, and won't help on its own.  I had hoped that textproc/uchardet would help in detecting current encoding your files are in, but it's being silly and can't even detect UTF-8, sadly.


----------



## yuripv (Nov 14, 2018)

OK, looks like we have a winner: textproc/py-chardet.  Try feeding the output from ls to `chardetect` command that the port installs.


----------



## Puffles_the_Dragon (Nov 14, 2018)

yuripv said:


> Meanwhile, just to show what I mean, here's a simple example:
> 
> 1. Create some files with cyrillic in names (my current locale is C.UTF-8):
> thor:yuripv:~/tst$ touch тест1
> ...


I am also gonna post a file I have been making with every command and edit I have made on my system in it, I think there are some things I possibly missed. I am starting to feel like this is fixable though


----------



## Deleted member 30996 (Nov 14, 2018)

yuripv I got a readout using russian/rux but it is a mess. They're all The Sensational Alex Harvey Band songs:



I don't know how I would go about entering any of that output to `chardetect`. security/wipe won't wipe a directory, I'm sure `dd` will.


----------



## yuripv (Nov 14, 2018)

Trihexagonal I didn't say anything about ports in russian/, just do a `echo /media/da0s1/Alex/SAHB/* | chardetect`.


----------



## Deleted member 30996 (Nov 14, 2018)

Sorry, I misunderstood you. That's a more readable format:



But it's history now. I moved the 6GB of data backup I had back onto the HDD and `dd` made short work of it. I'm in the process of transferring it back onto the clean USB stick.

I do appreciate your help but am glad to be done with it now.


----------



## Puffles_the_Dragon (Nov 15, 2018)

yuripv said:


> To answer this question you need to specify where exactly you are seeing the "?" stuff -- is it filenames, or mp3 tags, or...?


Here is a screenshot and a list of ei


yuripv said:


> OK, looks like we have a winner: textproc/py-chardet.  Try feeding the output from ls to `chardetect` command that the port installs.


I guess I can't upload an image. Everytime I got to do so via imgur FireFox quits. So I guess I can't on my FreeBSD laptop.


----------



## Deleted member 30996 (Nov 15, 2018)

Puffles_the_Dragon said:


> Here is a screenshot and a list of ei
> 
> I guess I can't upload an image. Everytime I got to do so via imgur FireFox quits. So I guess I can't on my FreeBSD laptop.



If you've already got a screenshot use the Attach files option to your lower left and upload it directly.


----------



## Puffles_the_Dragon (Nov 15, 2018)

Trihexagonal said:


> If you've already got a screenshot use the Attach files option to your lower left and upload it directly.


I tried, whenver I do that firefox quits instead of opening Thunar.


----------



## Puffles_the_Dragon (Nov 15, 2018)

yuripv said:


> Trihexagonal I didn't say anything about ports in russian/, just do a `echo /media/da0s1/Alex/SAHB/* | chardetect`.


I haven't been able to fix the charset issue, I wanted to upload an image to show the issue I am having. Whenever I am trying to upload an image on here, imgur etc. FireFox justs quits instead of opening Thunar. How do I fix this? I am so sorry for all the questions, I have been trying my best and seem to keep hitting problems


----------



## Deleted member 30996 (Nov 15, 2018)

This version of www/firefox has crashed on me more than any I've used.

If you're using pkg and not ports (don't mix them) I would try www/palemoon. It will only take a minute to install and may be more stable for you. It's what I'm using now.

I'd build it anyway. That way you're not stuck on one if a vulnerability is found. I have www/seamonkey installed too but it's vulnerable and has been for a while.


----------



## Puffles_the_Dragon (Nov 15, 2018)

Trihexagonal said:


> This version of www/firefox has crashed on me more than any I've used.
> 
> If you're using pkg and not ports (don't mix them) I would try www/palemoon. It will only take a minute to install and may be more stable for you. It's what I'm using now.
> 
> I'd build it anyway. That way you're not stuck on one if a vulnerability is found. I have www/seamonkey installed too but it's vulnerable and has been for a while.


They haven't been fixing the vulnerabilities and other issues?


----------



## Deleted member 30996 (Nov 15, 2018)

Puffles_the_Dragon said:


> They haven't been fixing the vulnerabilities and other issues?



Patches for 3rd party programs like www/seamonkey have to come from upstream, Mozilla in this case, and is not a FreeBSD shortcoming.

FreeBSD is the base system and the terminal you are presented with after your first reboot. Everything else is 3rd party programs. No Xorg, DE or WM, multimedia players, etc. and where most vulnerabilities come into play.

You stay current with FreeBSD vulnerabilities by using `# freebsd-update fetch` and `# freebsd-update install` if there is a return. Run `# pkg audit -F` to check your 3rd party programs and go from there.


----------

