# Massive problems with FreeBSD on AWS



## gschadow (Nov 1, 2020)

UPDATE: I was a little negative here. As I filed the bug report and went over all the details, I think I found the solution. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250770 

-----------------
Hi, I've been with FreeBSD since 386BSD 0.0new. Been around. And I always want to use FreeBSD on any servers, large and tiny, that I build, disk diskless, heady and headless, booting from disk, network, compact flash, whatever. I have done it. But with AWS there is a real bad problem and I am wondering why it doesn't get resolved for so many years? AWS after all is the most important hardware platform in today's world of servers. I have maintained a cage with HP DL585s and FibreChannel SAN, etc. but scrapped it all to move to "cloud" servers.

This here is a follow-up of Amazon AWS EC2 super light configuration a thread I started right here (with not a lot of feedback -- is this forum not the place to find real serious people? Are the serious people still preferring the email lists?)

Anyway on AWS there are severe stability problems. just a few quirks that make it scary, such as:


```
intsmb0: <Intel PIIX4 SMBUS Interface> irq 20 at device 1.3 on pci0
intsmb0: Could not allocate I/O space
device_attach: intsmb0 attach returned 6
```

appearing in the dmesg when booting a t3 instances.

When I boot a c5 or m5a it does not show up.

But worse is Even more scary, the system hung within hours, every time. That is, of course when we do more than what the average Linux guy can do with his Linux.  In my case it is that I have reduced the system disk to only 1 GB by mounting the entire /usr filesystem via EFS (NFS). And lo and behold, after a few hours, say, 2-3 hours, the system freezes up, but there is no information at all that I can see. No kernel warning on the console screen shot, nothing in the log, no /var/log/messages, nothing. I have to force stop the instance and then restart it and all runs fine again, there is no sign in any logs of any trouble.

I have seen some stuff on the bugs reports and email lists, from 2018 https://lists.freebsd.org/pipermail/freebsd-current/2018-April/069140.html and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235856 but this seems to be unresolved. There is a weakness with HVM vs. paravirtualization, and with ENA network drivers. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225791.

My trouble is, I have settled on this architecture with FreeBSD, and I just upgraded from 12.0 to 12.2 and nothing helped.


```
# Custom /etc/fstab for FreeBSD VM images
/dev/gpt/rootfs  /        ufs     rw      1       1
/dev/gpt/varfs   /var     ufs     rw      1       1
fdesc            /dev/fd  fdescfs rw      0       0
proc             /proc    procfs  rw      0       0
/usr             /.usr    nullfs  rw      0       0
fs-xxxxxx.efs.us-xxxxx-x.amazonaws.com:/ /usr nfs rw,nfsv4,minorversion=1,oneopenown,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,late,bg 0 0
```

It is freezing up within around 4 hours.

The problem was I had not specified minorversion=1, it seemed to me that this was implied in oneopenown but it is necessary. And with that the hanging was gone.


----------



## gschadow (Nov 1, 2020)

Bug report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250770


----------



## mark_j (Nov 1, 2020)

gschadow said:


> But with AWS there is a real bad problem and I am wondering why it doesn't get resolved for so many years?


So you reported it years ago & it's not been followed up? All I see is your post on your setup.



gschadow said:


> AWS after all is the most important hardware platform in today's world of servers.


Oh, ok.



gschadow said:


> I have maintained a cage with HP DL585s and FibreChannel SAN, etc. but scrapped it all to move to "cloud" servers.
> 
> This here is a follow-up of Amazon AWS EC2 super light configuration a thread I started right here (with not a lot of feedback -- is this forum not the place to find real serious people? Are the serious people still preferring the email lists?)


I note people replied to your post back 'years ago' but they didn't address this issue? Shame on them. Where does all your support fees go to? Lazy forum readers.

I'm a real serious person, sometimes, btw.


----------



## rootbert (Nov 1, 2020)

one question: Do you experience any problems with a standard setup on AWS? If not it might be your custom build/configuration ...


----------



## drhowarddrfine (Nov 1, 2020)

I've paid scant attention to this but noticed an announcement about AWS supporting FreeBSD just within the last couple of days. If anything, to me it indicates some things may have changed from two years ago so that could be the source of issues.


----------



## ralphbsz (Nov 1, 2020)

gschadow said:


> is this forum not the place to find real serious people? Are the serious people still preferring the email lists?


A lot of the serious technical discussions happen on e-mail lists, indeed. This is more a user forum, where users help each other, not many developers here.

The question is: Did you find an AWS-specific problem, or a generic FreeBSD problem? Will the same thing happen on a real physical server, or on a Google Cloud or Microsoft Azure server? This matters for determining where to look for solutions.

Are there any AWS-specific discussion resources? For example, I have a single off-site machine (hosted at Google cloud), and I know there are some discussion groups there. I took a look once when setting up my machine, there was some chatter about FreeBSD, but nothing highly specific.  I haven't had any problems with my hosted machine that needed solving (other than my lack of time of using it for all the things I would like to get done); but then, I use the stock OS install on the machine.

Have you tried contacting AWS support (real trouble ticket)? Most likely, they'll say "you have modified our provided setup so much, we won't be able to help debug the problem", and they're right. But you might get lucky and they might take an interest, or point you at known problems and solutions.


----------



## getopt (Nov 1, 2020)

Related but off-topic:








						Massive Problems with FreeBSD ... (forums?)
					

A basic question was raised and an answer was given:    is this forum not the place to find real serious people? Are the serious people still preferring the email lists?  A lot of the serious technical discussions happen on e-mail lists, indeed. This is more a user forum, where users help each...




					forums.FreeBSD.org


----------



## gschadow (Nov 1, 2020)

I am making some edits to my initial post. I have actually resolved the issue once I filed the bug report  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250770 and did more leg work to get to the bottom of this. I was a little negative up there for which I apologize.


----------

