# Really? Is FreeBSD wandering into the land of Linux Voodoo?



## jakedp (Jun 19, 2019)

Joking aside, very fustrated, how do I delete these files? Neither rm or unlink work and bsdinstall exits because it cannot unlink them.


```
zsh: sure you want to delete all the files in /usr/jail/www/* [yn]? y
rm: lib/libc.so.7: Operation not permitted
rm: lib/libcrypt.so.5: Operation not permitted
rm: lib/libthr.so.3: Operation not permitted
rm: libexec/ld-elf.so.1: Operation not permitted
rm: sbin/init: Operation not permitted
rm: usr/bin/crontab: Operation not permitted
rm: usr/bin/login: Operation not permitted
rm: usr/bin/su: Operation not permitted
rm: usr/bin/passwd: Operation not permitted
rm: usr/bin/opiepasswd: Operation not permitted
rm: usr/bin/opieinfo: Operation not permitted
rm: usr/bin/chpass: Operation not permitted
rm: usr/bin: Directory not empty
rm: usr/lib/librt.so.1: Operation not permitted
rm: usr/lib: Directory not empty
rm: var/empty: Operation not permitted
```

The FreeBSD Handbook and other documentation is either plain wrong or absurd, or creates broken garbage, on creating jails. 
Running 11.2-RELEASE.


----------



## ralphbsz (Jun 19, 2019)

What is your process's id? Who owns these files? What are their permissions? Do they have ACLs or chflags?

I'm quite sure that the handbook and documentation are correct, and I'm quite sure that FreeBSD does not create broken garbage, not even in jails.


----------



## hukadan (Jun 19, 2019)

You need to clear the immutable flag with `chflags -R noschg /usr/jail/www/` before removing those files.


----------



## Eric A. Borisch (Jun 19, 2019)

Likely the system immutable flag:  https://forums.freebsd.org/threads/cant-delete-jail-directory.57624/


----------



## jakedp (Jun 19, 2019)

ralphbsz said:


> What is your process's id? Who owns these files? What are their permissions? Do they have ACLs or chflags?
> 
> I'm quite sure that the handbook and documentation are correct, and I'm quite sure that FreeBSD does not create broken garbage, not even in jails.



5 days, 4 hours a day at least and have not gotten one jail working with networking. 18 years of building Linux servers so not new or foreign to the concepts. Read every relevant man page and been through the Handbook many times. It is not between the screen and the chair...


```
# ls -al /lib/libc.so.7 
-r--r--r--  1 root  wheel  1779544 Jun 22  2018 /lib/libc.so.7
```

Done from /:

```
# ls -lo
total 89
-rw-r--r--   2 root  wheel     -  957 Jun 22  2018 .cshrc
-rw-r--r--   2 root  wheel     -  474 Jun 22  2018 .profile
-rw-------   1 root  wheel     - 1024 Jun 18 22:28 .rnd
drwxrwxr-x   2 root  operator  -  512 Jun 22  2018 .snap
-r--r--r--   1 root  wheel     - 6197 Jun 22  2018 COPYRIGHT
drwxr-xr-x   2 root  wheel     - 1024 Jun 19 14:16 bin
drwxr-xr-x   9 root  wheel     - 1536 Jun 19 14:16 boot
dr-xr-xr-x  10 root  wheel     -  512 Jun 19 14:21 dev
-rw-------   1 root  wheel     - 4096 Jun 19 14:21 entropy
drwxr-xr-x  27 root  wheel     - 2560 Jun 19 14:40 etc
lrwxr-xr-x   1 root  wheel     -    8 Jun 18 22:51 home -> usr/home
drwxr-xr-x   4 root  wheel     - 1536 Jun 19 14:16 lib
drwxr-xr-x   3 root  wheel     -  512 Jun 22  2018 libexec
drwxr-xr-x   2 root  wheel     -  512 Jun 22  2018 media
drwxr-xr-x   2 root  wheel     -  512 Jun 22  2018 mnt
drwxr-xr-x   2 root  wheel     -  512 Jun 22  2018 net
dr-xr-xr-x   2 root  wheel     -  512 Jun 22  2018 proc
drwxr-xr-x   2 root  wheel     - 2560 Jun 19 14:16 rescue
drwxr-xr-x   4 root  wheel     -  512 Jun 19 15:04 root
drwxr-xr-x   2 root  wheel     - 2560 Jun 22  2018 sbin
lrwxr-xr-x   1 root  wheel     -   11 Jun 22  2018 sys -> usr/src/sys
drwxrwxrwt   9 root  wheel     -  512 Jun 19 15:01 tmp
drwxr-xr-x  17 root  wheel     -  512 Jun 19 14:50 usr
drwxr-xr-x  24 root  wheel     -  512 Jun 19 14:21 var
```

Done from /usr/jail/www:

```
# ls -lo
total 28
-rw-r--r--  1 root  wheel  -  957 Jun 22  2018 .cshrc
-rw-r--r--  1 root  wheel  -  474 Jun 22  2018 .profile
drwxr-xr-x  2 root  wheel  - 1536 Jun 22  2018 lib
drwxr-xr-x  2 root  wheel  -  512 Jun 19 15:02 libexec
drwxr-xr-x  2 root  wheel  - 2560 Jun 22  2018 sbin
drwxr-xr-x  4 root  wheel  -  512 Jun 22  2018 usr
drwxr-xr-x  3 root  wheel  -  512 Jun 22  2018 var
```

So it seems chflags, I have not touched any ACLs. Only thing done from a base install is freebsd-update and freebsd-update -b /usr/jail/www/ and install zsh, screen, git, emacs.

Well, figure out to remove the flags recursively and delete the dirs to start over.


----------



## jakedp (Jun 19, 2019)

I did not see the replies, the chflags was the problem and did not think of it until after posting the fustration.

I' am not sure how to get around this (did a 'bsdinstall jail /usr/jail/www/')?:
`# service jail start  
Starting jails:jail: www: ip4.addr: not an IPv4 address: inherit
jail: www: ip6.addr: not an IPv6 address: inherit`

_/etc/jail.conf_

```
www {
    host.hostname = www.example.com;
    allow.nomount;
    interface = "vtnet1";
    ip4.addr = "inherit";
    ip6.addr = "inherit";
    path = "/usr/jail/www";
    mount.devfs;
    exec.start = "/bin/sh /etc/rc.conf";
    exec.stop = "/bin/sh /etc/c.shutdown";
}
```


----------



## Eric A. Borisch (Jun 19, 2019)

You missed the -o when you listed one of the files of interest:

```
$ ls -al /lib/libc.so.7
-r--r--r--  1 root  wheel  1978344 Dec  3  2018 /lib/libc.so.7
$ ls -alo /lib/libc.so.7
-r--r--r--  1 root  wheel  schg 1978344 Dec  3  2018 /lib/libc.so.7
```

Hukadan already posted it, but you want -R for recursive chflags(1).


----------



## hukadan (Jun 19, 2019)

For the networking, was it no network or no DNS ? In the latter case, did you copy the /etc/resolv.conf from the host to the jail ? You need to provide more information.


----------



## jakedp (Jun 19, 2019)

hukadan said:


> For the networking, was it no network or no DNS ? In the latter case, did you copy the /etc/resolv.conf from the host to the jail ? You need to provide more information.



Networking on the host (VPS; static IP through DHCP) but have tried VLAN, bridging, ipconfig aliasing through various iterations. Other than the solved chflags scenario the only hiccup has been networking through the jail. bsdinstall is not mentioned in the Handbook, and although the src method works for binary it is the only binary method I found that works properly.

Only networking is left. I did the /etc/resolve.conf copy and no go.

I' am thinking to ifconfig alias vtnet0 (host) to vnet1 (jail)?


----------



## PMc (Jun 19, 2019)

jakedp said:


> Joking aside, very fustrated, how do I delete these files? Neither rm or unlink work and bsdinstall exits because it cannot unlink them.
> 
> 
> ```
> ...



Thats fully normal. To remove all files of a subordinate installation (DESTDIR or jail):


```
cd <your-logical-rootdir> && chflags -R noschg . && rm -rf * .[a-z]?*
```

But indeed, I do not know where this might be documented.


----------



## ralphbsz (Jun 20, 2019)

PMc said:


> But indeed, I do not know where this might be documented.


man chflags.  man rm has a link to chflags.  man security discusses flags, but not explicitly, only their handling and effect on security.
Handbook chapter 3.4 explicitly discusses flags in detail. This is in the section on "FreeBSD basics" of the handbook. One might assume that someone would at least read the basic section of the handbook, before claiming "wrong", "absurd" and "garbage".


----------



## scottro (Jun 20, 2019)

Specific things can be hard to find. Have you read everything in the basics section?    
Missing what you need isn't difficult, and FreeBSD documentation has fallen behind, with parts of the handbook, especially on third party programs, being incomplete or out of date.  If the OP is reading about ZFS, and it doesn't mention an essential thing, such as chflags, they are right. The chapter was incomplete, and chflags should have been mentioned.  If you write a program, and it's missing shared library X because you expect that the library was provided by something else that some people don't install, the program is buggy.


----------



## Spartrekus (Jun 20, 2019)

jakedp said:


> Joking aside, very fustrated, how do I delete these files? Neither rm or unlink work and bsdinstall exits because it cannot unlink them.
> 
> 
> ```
> ...



That's law of computer sciences 
> Linux will go to look like MS Windows, because it is made by various companies, users, and then brought alltogether.
> FreeBSD will go to look like Linux, because FreeBSD takes some source codes, which are made for both Linux and BSD.

Ideally, clang + src code should work only for BSD, to be sure to have a code that runs only on BSD and that you cannot compile it on Linux. Then, you have really to rewrite almost everything by hand.


----------



## hukadan (Jun 20, 2019)

Spartrekus said:


> FreeBSD will go to look like Linux, because FreeBSD takes some source codes, which are made for both Linux and BSD.


Two questions here :

What is the link between the problem described by the OP and your sentence (hint : a joke is not a technical problem) ?
Could you stop trolling ?


----------



## ralphbsz (Jun 20, 2019)

scottro said:


> Specific things can be hard to find. Have you read everything in the basics section?


True. In order to use a Unix system, you need to understand the basics, for example file permissions, authentication, identity, and access control. Some of those basics are standardized (for example through POSIX) and the same between most Unix-style systems. Others are not standardized. The thing we are discussing here is not standardized, but both Linux and FreeBSD do implement it mostly the same, just with different names: what FreeBSD calls flags is called attr in Linux; the command corresponding to chflags would be chattr in Linux.



> ... and FreeBSD documentation has fallen behind, with parts of the handbook, ... being incomplete or out of date.  If the OP is reading about ZFS, and it doesn't mention an essential thing, such as chflags, they are right. The chapter was incomplete, and chflags should have been mentioned.


No, I disagree.  File access control (permissions, flags, and ACLs) is part of the basic OS definition of how user processes access files, and that is well documented, in particular in the handbook, but also in man pages. I don't think there is a need for the documentation for full-function native file systems (UFS and ZFS) to repeat that information. Where I agree with you: It would be nice if the file system documentation would link to the basic; in man pages, that can be done using the "see also" section.



> ... especially on third party programs ...
> If you write a program, and it's missing shared library X because you expect that the library was provided by something else that some people don't install, the program is buggy.


Correct: If you want a certain library, you need to install it. It will not magically appear just because you want it.
The documentation situation on third-party programs (often ports) is tricky. By definition they come from third parties, who can make great or horrible documentation, and usually something in between. In some cases, the volunteers who maintain the port trees add FreeBSD-specific documentation. But the discussion in this thread is about a base OS feature (flags).


----------



## jakedp (Jun 20, 2019)

Thanks for the help. The only thing I cannot figure out is networking within the jail. Although there is a particular way I want to do it. How do I mark this as solved?


----------



## jakedp (Jun 20, 2019)

ralphbsz said:


> True. In order to use a Unix system, you need to understand the basics, for example file permissions, authentication, identity, and access control. Some of those basics are standardized (for example through POSIX) and the same between most Unix-style systems. Others are not standardized. The thing we are discussing here is not standardized, but both Linux and FreeBSD do implement it mostly the same, just with different names: what FreeBSD calls flags is called attr in Linux; the command corresponding to chflags would be chattr in Linux.
> 
> 
> No, I disagree.  File access control (permissions, flags, and ACLs) is part of the basic OS definition of how user processes access files, and that is well documented, in particular in the handbook, but also in man pages. I don't think there is a need for the documentation for full-function native file systems (UFS and ZFS) to repeat that information. Where I agree with you: It would be nice if the file system documentation would link to the basic; in man pages, that can be done using the "see also" section.
> ...



My fustration is from networking within a jail without forwarding, subnetting, etc. The OP was the effect and not the cause. In my fustration I also over exaggerated. The documentation is not so bad except how to network within a jail without adding latency and complexity through various schemes. After much experimentation the binary way to install a jail is best done through bsdinstall which is not mentioned.

There is assumptions that I do not know of is the main point. I could use qjail, exjail, but then I would not understand. Once I understand the process then I can script it like I do with chroots for Slackware and Arch.

The basics between Winux and BSD are quite different these days once one goes into the details. At first glance and usuage one can recognize the similarities but in the last few years the ethos of Linux (Winux) has diverged from unix quite a bit and it was always more 'progressive' than BSDs. systemd proves who was right about premises.

Have only had to deal with attr on CentOS once... I degress and go off-topic.


----------



## ralphbsz (Jun 20, 2019)

jakedp said:


> My fustration is from networking within a jail without forwarding, subnetting, etc.


And there I have no idea how to help, never set up anything in a jail other than a local daemon for playing (and took that one out of a jail when the playing turned out to not be fun).

Let me ask you a question though. Why are you using a jail? Can you run the same thing without a jail, at similar security but with much less hassle? If you really need a jail, but find setting them up to be too hard, maybe you could instead run a full-fledged VM instead. Uses more resources, but might be easier for you to set up.  What I'm really saying is this: Try making a 3-way tradeoff between security, resource usage, and convenience, and perhaps the jail setup problem goes away.


----------



## Eric A. Borisch (Jun 20, 2019)

I'm not sure where you found the "inherit" option for IP addresses, but that's not supported in jail.conf(5); you need to provide an IP address that is assigned to some interface running on the host (perhaps an alias.)

The IP address will be one that you've already assigned to an 'actual' network adapter, perhaps as an alias. There are a number of tools out there to make jails easier to manage; iocage and ezjail come to mind.


----------



## jakedp (Jun 20, 2019)

I wanted OS/process separation and learn jails; the best thing about BSD along with ports. A VPS running as a QEMU VM with a VM inside it seems a good way to waste electricity. Too heavy. The network thing comes up again with a VM anyways: subnetting, VLAN, NAT, etc. A VM brings more problems and complexity than it solves.


----------



## jakedp (Jun 20, 2019)

Eric A. Borisch said:


> I'm not sure where you found the "inherit" option for IP addresses, but that's not supported in jail.conf(5); you need to provide an IP address that is assigned to some interface running on the host (perhaps an alias.)
> 
> The IP address will be one that you've already assigned to an 'actual' network adapter, perhaps as an alias. There are a number of tools out there to make jails easier to manage; iocage and ezjail come to mind.



Easier means not learning.


----------



## Eric A. Borisch (Jun 20, 2019)

jakedp said:


> Easier means not learning



Agreed, but it can be useful to see what commands / options they eventually set up. sysutils/DTraceToolkit's execsnoop can be very helpful to that end... or `dwatch -X proc-create` with dwatch(1) in 11.2 or 12.


----------



## RedPhoenix (Jun 20, 2019)

Spartrekus said:


> That's law of computer sciences
> > Linux will go to look like MS Windows, because it is made by various companies, users, and then brought alltogether.
> > FreeBSD will go to look like Linux, because FreeBSD takes some source codes, which are made for both Linux and BSD.
> 
> Ideally, clang + src code should work only for BSD, to be sure to have a code that runs only on BSD and that you cannot compile it on Linux. Then, you have really to rewrite almost everything by hand.


Yeah, everyone is trying to imitate each other... Worst of all is macOS, which is Single UNIX Specification compliant, yet their OS was made for, quite frankly, what Apple perceives to be morons.


----------



## scottro (Jun 21, 2019)

ralphbsz, I suspect we may agree more than disagree.  I'm not really familiar with the OP's situation, but what I mean is that in, for example, the old instructions to buildworld--around early double oughts--you had to use chflags on /usr/obj, and it was specifically mentioned in the steps.   So, if the zfs page is giving instructions and a chflags is necessary,I feel it should be mentioned.
I do find that sometimes, the most useful instructions are those by the less experienced, as the knowledgeable tends to use mental shorthand.  Does that make sense?  In other words, a newcomer might right download the patch and patch the file, and this would be foreign to them. Whereas, a less experienced writer might say, run fetch http://patchfile.patch. Now run the command patch file < patchfile.patch.
As a character in a medieval mystery I was reading says, Everyone's a beginner at someone else's craft. 

I don't know what the bar should be--that is, how much should we expect anyone to know before doing something. As one gets more experienced, they get a better idea of what they should do on their own, and that's one thing I like about these forums.  Almost all of us (including you with your many helpful posts), realize that it's not always easy.


----------



## xtremae (Jun 21, 2019)

scottro said:


> but what I mean is that in, for example, the old instructions to buildworld--around early double oughts--you had to use chflags on /usr/obj, and it was specifically mentioned in the steps. So, if the zfs page is giving instructions and a chflags is necessary,I feel it should be mentioned.


I agree with your point. The handbook serves the purpose of a guide rather than that of an out of context exhaustive reference. If the flags need to change during the process, the handbook needs to be explicit.


----------



## driesm (Jun 21, 2019)

Eric A. Borisch said:


> I'm not sure where you found the "inherit" option for IP addresses, but that's not supported in jail.conf(5); you need to provide an IP address that is assigned to some interface running on the host (perhaps an alias.)
> 
> The IP address will be one that you've already assigned to an 'actual' network adapter, perhaps as an alias. There are a number of tools out there to make jails easier to manage; iocage and ezjail come to mind.


Inherit is in fact an option though. It just reveals everything from the host network stack to the jail. So a daemon can bind on any IP address from the host. EG; in jail.conf; ip4 = inherit; ip6 = inherit;


----------



## Eric A. Borisch (Jun 21, 2019)

Mea culpa; I didn't dig far enough: jail(8)


----------



## rigoletto@ (Jun 21, 2019)

jakedp said:


> The only thing I cannot figure out is networking within the jail.



By default you can't, you should manage from the host. If you want to have a "virtual" network stack into the jail you need to enable vnet(9). Thread 70356


----------



## jakedp (Jun 22, 2019)

Eric A. Borisch said:


> Agreed, but it can be useful to see what commands / options they eventually set up. sysutils/DTraceToolkit's execsnoop can be very helpful to that end... or `dwatch -X proc-create` with dwatch(1) in 11.2 or 12.



I will be delving into DTRACE as I adjust more!


----------



## jakedp (Jun 22, 2019)

Eric A. Borisch said:


> Mea culpa; I didn't dig far enough: jail(8)



_host    Set the origin of hostname    and related information.  Possible
         values are    ``inherit'' to use the system information and ``new''
         for the jail to use the information from the above    fields.     Set-
         ting any of the above fields implies a value of ``new''. _- from jail(8)

Ah. Should have thought of that. If I understand that correctly that will inherit all required net information from the host(?).


----------



## Spartrekus (Jun 22, 2019)

The most important with the jail is permissions.
Since it is a Linux/ BSD project, you have chances that the config files are in different directories than the regular BSD ones. Since the jail has already a strange directory pattern, you will end to look at the source code. If you can look at it, maybe there are chances to bring this up.



jakedp said:


> The basics between Winux and BSD are quite different these days once one goes into the details. At first glance and usuage one can recognize the similarities but in the last few years the ethos of Linux (Winux) has diverged from unix quite a bit and it was always more 'progressive' than BSDs. systemd proves who was right about premises.
> 
> Have only had to deal with attr on CentOS once... I degress and go off-topic.



I guess that it started at the age of pulseaudio, once gnome changed... then systemd.
Maybe, influence of RedHat or Ubuntu. By the end, FreeBSD will change as well. You cannot keep using a same technology, because the world changes *fast*. Therefore users will need new software's to be implemented. The more amazing, new, revolutionary Wayland-like things, the increased chances that it brings changes in the base of FreeBSD. People could then move from FreeBSD to OpenBSD to get their servers running.


----------



## hukadan (Jun 22, 2019)

Spartrekus said:


> Since it is a Linux/ BSD project, you have chances that the config files are in different directories than the regular BSD ones.


jail(8) is definitely *not* a Linux project.


Spartrekus said:


> jail has already a strange directory pattern


The directory pattern of a jail(8) is not different that the one of a regular FreeBSD install.


----------



## driesm (Jun 22, 2019)

jakedp said:


> _host    Set the origin of hostname    and related information.  Possible
> values are    ``inherit'' to use the system information and ``new''
> for the jail to use the information from the above    fields.     Set-
> ting any of the above fields implies a value of ``new''. _- from jail(8)
> ...


No ... that is just the hostname that will be inherited from the host .
If you want the network stack to be the same in the jail you set this in jail.conf:

```
www {
    ip4 = inherit;
    ip6 = inherit;
}
```


----------



## jakedp (Jun 22, 2019)

Duffyx said:


> No ... that is just the hostname that will be inherited from the host .
> If you want the network stack to be the same in the jail you set this in jail.conf:
> 
> ```
> ...



Doesn' t work for me...


----------



## jakedp (Jun 22, 2019)

Spartrekus

Wayland is amazing? Let us remove most of the functionality of 30 years of software so gamers and eye candy enthusiasts have slightly looking better graphics. Never mind that is performs worse than X for most games, performs worse than most compositors. It is only good for Fedora and the GNOME virus, maybe. It also generates more heat for little benefit and only formally mentioned users.

Like everything out of Red Hat/Fedora it is Alpha software being pushed to the masses as Alpha Testers. Sounds familiar: Windows. At least Red Hat is not Oracle, Microsoft, Apple.


----------



## driesm (Jun 22, 2019)

jakedp said:


> Doesn' t work for me...


I was almost gonna post: "that's too bad for you". You really have to give me something.
Do you get the same error as at the start? Can you (re)share your current config that doesnt work?

A few things I noticed in the jail.conf you shared is that you specify "interface". I think that it must not be used together with inherit. You either bind the jail to a host IP address OR give it all addresses (inherit) OR use VIMAGE aka VNET. Also two little mistakes/typos: at exec.stop it should be /etc/rc.shutdown, and exec.start should be /etc/rc not /etc/rc.conf.


----------



## jakedp (Jun 23, 2019)

Thanks for pointing out the typos. I will try later in the future when I have the time again.


----------



## Spartrekus (Jun 24, 2019)

jakedp said:


> Spartrekus
> 
> Wayland is amazing? Let us remove most of the functionality of 30 years of software so gamers and eye candy enthusiasts have slightly looking better graphics. Never mind that is performs worse than X for most games, performs worse than most compositors. It is only good for Fedora and the GNOME virus, maybe. It also generates more heat for little benefit and only formally mentioned users.
> 
> Like everything out of Red Hat/Fedora it is Alpha software being pushed to the masses as Alpha Testers. Sounds familiar: Windows. At least Red Hat is not Oracle, Microsoft, Apple.


Well, Linux goes for Wayland. They do might remove those 30 years, as you wrote.
I believe in terminal and X11 if opensource.


----------

