# valgrind with linux ABI emulation? Trying to debug memory issue w/ Lightworks under linuxulator



## neogeo (Feb 21, 2022)

In a continuation of a project introduced in another thread ... 
	

	







						Help porting for Lightworks under linux-c7 - Multimedia Production on FreeBSD?
					

While studying under a Bachelors Degree program in communication at a US university, with a specialization in multimedia topics, I've had an opportunity to gain some initial experience with software tools for multimedia production.  In developing the multimedia content for some course projects...




					forums.freebsd.org
				




... testing out a Lightworks (LWKS) for Linux installation under the FreeBSD linuxulator, in a FreeBSD 12.3-STABLE build, I noticed the following in the output from the `ntcardvt` binary from the Lightworks installation. This is when run under the FreeBSD linuxulator, a section of the kdump output of an earlier ktrace call:


```
40672 ntcardvt PSIG  SIGSEGV caught handler=0x808ce4ab0 mask=0x0 code=SEGV_MAPERR                                                            
 40672 ntcardvt CALL  linux_sys_futex(0x8055caf68,0x81,0x7fffffff,0,0,0x8028b51a8)
 40672 ntcardvt RET   linux_sys_futex 0                   
 40672 ntcardvt CALL  linux_sys_futex(0x8028cd1e0,0x81,0x7fffffff,0,0xca,0x1)
 40672 ntcardvt RET   linux_sys_futex 0                   
 40672 ntcardvt CALL  linux_sys_futex(0x8055c9f60,0x81,0x7fffffff,0,0x802ffa360,0x1)
 40672 ntcardvt RET   linux_sys_futex 0                                                                                                       
 40672 ntcardvt CALL  write(0x1,0x189b980,0x761)          
 40672 ntcardvt GIO   fd 1 wrote 1889 bytes
                                                                                            
       "Lightworks 64-bit (Release, build Lw : 132926 Common : 132108 Dependencies : 132677 dated Feb  9 2022)                              
                                                                                                                                            
        OS Version            : Linux version 3.2.0 (des@freebsd.org) (gcc version Clang 11.0.1) #4 Sun Dec 18 04:30:00 CET 1977            
        Available memory      : 11.4 GB                                                                                                     
        Total memory          : 15.8 GB                                                                                                     
        Virtual Address Range : 16777216.0 TB                                                                                               
        Number of CPUs        : 4                                                                                                           
        User has admin rights : No                                                                                                          
        Graphics driver       : Intel Open Source Technology Center
        OpenGL version        : 3.0 Mesa 20.3.5
                                                        
        19:55:22.326: Setting project base directory /usr/home/gimbal/Lightworks/Projects/
        19:55:22.327: Starting window manager
        19:55:22.328: FreeImage version: 3.18.0
        This program uses FreeImage, a free, open source image library supporting all common bitmap formats. See http://freeimage.sourceforge.net for details
        19:55:22.479: License initialisation error <License initialisation error>
        19:55:22.485: Checking library versions..
        ASSERTION : Lightworks, No .fx files found in /usr/share/lightworks/Shaders/Shaders
        19:55:22.490: Preferred device System Sound is not available, loading software only mode instead.
        19:55:22.490: WARNING!! There are no audio resources present
                                                        
        ---------------- Segmentation fault :( ----------------
              
        LwOS::exceptionHandler(int)
        linux_vdso.so.1() [0x7ffffffff514] 
        /opt/linux/debian-11/usr/lib/lightworks/ntcardvt(+0xc2550) [0x10e3550]
        /opt/linux/debian-11/usr/lib/lightworks/ntcardvt(+0xc63cd) [0x10e73cd]
        BackgroundTaskBase::execute()                   
        BackgroundTaskQueueBase::despatch(Lw::Ptr<iBackgroundTask, Lw::DtorTraits, Lw::InternalRefCountTraits>&)
        DecouplingQueue<iBackgroundTask>::decoupledThread(void*)
        Thread::protectedExecute(iThread::eExecRC (*)(void*), void*, void (*)(iThread::Exception const&))
        Thread::execute(void*)                          
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x802fe5ea7]
        /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x805505def]
                                                        

        --------------------------------------
```

I'm noticing this section in particular, such that appears to be written to some kind of a log file by ntcardvt and is visible under ktrace/kdump for the ntcardvt binary:


```
Virtual Address Range : 16777216.0 TB
```

Peculiar, isn't it? The machine has 16 GB RAM and a whole 32GB of swap space.

When I run this ntcardvt command under `limits -v 10G` the error is slightly different:


```
$ limits -v 10G   env -u LC_ALL ktrace -dif /tmp/ntcardvt.out /opt/linux/debian-11/usr/lib/lightworks/ntcardvt
LOH unaligned


---------------- Segmentation fault :( ----------------

LwOS::exceptionHandler(int)
linux_vdso.so.1() [0x7ffffffff514]
Lw::MTHeap::alloc(unsigned long)
TextFile::load(LightweightString<wchar_t>, char, bool)
TextFile::TextFile(LightweightString<wchar_t> const&, bool)
Lw::Localisation::StringTable::load(LightweightString<wchar_t> const&)
Lw::Localisation::getStringTableInternal()
resourceStrW(ResId)
/opt/linux/debian-11/usr/lib/lightworks/libedit.so(+0x10ece5) [0x805f0ece5]
/lib64/ld-linux-x86-64.so.2(+0xffe2) [0x8013c7fe2]
/lib64/ld-linux-x86-64.so.2(+0x100e9) [0x8013c80e9]
/lib64/ld-linux-x86-64.so.2(+0x10ca) [0x8013b90ca]


---------------------------------------

Abort trap (core dumped)
```

I believe the "LOH unaligned" message is from  Lightworks' LwOS library.

An excerpt of the kdump output, up to the point of the segfault:

```
70304 ntcardvt CALL  linux_openat(0xffffff9c,0x818860508,0,0)               
 70304 ntcardvt NAMI  "/opt/linux/debian-11/usr/share/lightworks/strings.txt"
 70304 ntcardvt NAMI  "/opt/linux/debian-11"       
 70304 ntcardvt NAMI  "/opt/linux/debian-11/usr/share/lightworks/strings.txt"
 70304 ntcardvt RET   linux_openat 3                                         
 70304 ntcardvt CALL  linux_newfstat(0x3,0x7fffffffcd70)                     
 70304 ntcardvt STRU  struct stat {dev=17641383549324237605, ino=37115, mode=0100644, nlink=1, uid=0, gid=0, rdev=18446744073709551615, atime=1645068380, mtime=16444287
17, ctime=1645068391.590767000, birthtime=1644428717, size=181949, blksize=131072, blocks=209, flags=0x800 }                                                     
 70304 ntcardvt RET   linux_newfstat 0                                       
 70304 ntcardvt CALL  linux_mmap2(0,0x10000000000,0,0x22,0xffffffff,0)
 70304 ntcardvt RET   linux_mmap2 -1 errno -12 Cannot allocate memory        
 70304 ntcardvt CALL  linux_mprotect(0xfffffffffffff000,0x2d6ef,0x3)         
 70304 ntcardvt RET   linux_mprotect -1 errno -22 Invalid argument
 70304 ntcardvt CALL  linux_newfstat(0x1,0x7fffffffc600)
 70304 ntcardvt STRU  struct stat {dev=1895890688, ino=779, mode=020620, nlink=1, uid=1000, gid=4, rdev=779, atime=1645468138, mtime=1645468138, ctime=1645468138, birth
time=-1, size=0, blksize=4096, blocks=0, flags=0x0 }
 70304 ntcardvt RET   linux_newfstat 0
 70304 ntcardvt CALL  write(0x1,0x149e740,0xe)
 70304 ntcardvt GIO   fd 1 wrote 14 bytes
       "LOH unaligned
       "
 70304 ntcardvt RET   write 14/0xe
 70304 ntcardvt PSIG  SIGSEGV caught handler=0x808ce4ab0 mask=0x0 code=SEGV_MAPERR
```


This is with the Linux support enabled in a FreeBSD 12.3-STABLE build from the stable/12 branch, uname:

```
FreeBSD riparian.cloud.thinkum.space 12.3-STABLE FreeBSD 12.3-STABLE stable/12-n1855-ce99de0241e RIPARIAN  amd64
```

This ntcardvt command is installed from a Debian package in a filesystem created originally with the debootstrap port, then modified with brandelf. I've been able to install and run aptitude and mate-terminal under linuxulator wth this filesystem.

I'd like to test this binary from Lightworks for Linux with valgrind, to see if that may help to debug the issue of the segfault. However I'm seeing the following when trying to run valgrind with this code:


```
$ valgrind /opt/linux/debian-11/usr/lib/lightworks/ntcardvt
valgrind: m_ume.c: can't open interpreter
```

So, I believe valgrind is not able to run the ntcardvt command, perhaps due to something about its linkage to a Linux ld.so, in its Linux object file format?

Is there a way to run valgrind with this Linux object code? and what to do debug this memory issue? Candidly, I'm not a lot of adept with C and C++ toolchains.

I can try to install this Lightworks Debian package in a debootstrapped ubuntu release. I wonder if the bug is in the Linux release code though? This Lightworks Debian package runs just well under Debian 10 in a bare-metal install.

I'd be happy to provide more debug info about this configuration, it's not a lot of succinct text though.

Ideally, this Lightworks build would "Just work" lol once installed to some Linux filesystem and run under FreeBSD, but with the message "Virtual Address Range : 16777216.0 TB" it looks like it'll need some debugging


----------



## shkhln (Feb 21, 2022)

Programs don't tend to suddenly develop memory leaks just because they aren't being run in a familiar environment. It would be wiser to study that stacktrace. Use a decompiler if you need to (legality varies by jurisdiction, but nobody really cares as long as you keep it to yourself).


neogeo said:


> ```
> Virtual Address Range : 16777216.0 TB
> ```
> 
> Peculiar, isn't it?


Not at all.


----------



## neogeo (Feb 21, 2022)

shkhln said:


> Programs don't tend to suddenly develop memory leaks just because they aren't being run in a familiar environment. It would be wiser to study that stacktrace. Use a decompiler if you need to (legality varies by jurisdiction, but nobody really cares as long as you keep it to yourself).



Memory leak, oh.

I'll try to find what may available under ports for debugging this application - had tried taking a look with `objdump -D` which produced assembler code LoL. Was hoping that valgrind might at least show anything that could help with a bug report to the Lightworks forums etc. I'm not famliar with these tools.

There's this too:


```
linux_mmap2(0,0x10000000000,0,0x22,0xffffffff,0)
```


----------



## neogeo (Feb 21, 2022)

There may be a syscall where it tries to compute how much "virutal memory" is available?


----------



## shkhln (Feb 21, 2022)

It's perfectly fine for the app to have as many (or as few) bugs as the devs want. That's not the point. Remember that you are debugging the Linux emulation, not the app. Make a trace on Linux (with thread ids, it's 2022) for comparison. Look for the differences in syscall arguments and return values. Check calls to the libc from the crash site, that might give you some idea what that code was doing.



neogeo said:


> There may be a syscall where it tries to compute how much "virutal memory" is available?


Why is this even a question? 2^64 bytes.


----------



## neogeo (Feb 21, 2022)

fwiw running with DISPLAY unset, the debug output is different still. This time, GTK shows up in its internal backtrace:


```
70730 ntcardvt GIO   fd 1 wrote 2655 bytes                                                                                                                      
       "Lightworks 64-bit (Release, build Lw : 132926 Common : 132108 Dependencies : 132677 dated Feb  9 2022)            
                                                                                                                                                                 
        11:13:26.018: GLX not available                                      
        OS Version            : Linux version 3.2.0 (des@freebsd.org) (gcc version Clang 11.0.1) #4 Sun Dec 18 04:30:00 CET 1977
        Available memory      : 10.9 GB
        Total memory          : 15.8 GB                                      
        Virtual Address Range : 16777216.0 TB
        Number of CPUs        : 4
        User has admin rights : No

        11:13:26.019: Setting project base directory /opt/linux/debian-11/tmp/tmphome/Lightworks/Projects/
        11:13:26.019: Starting window manager
        11:13:26.020: FreeImage version: 3.18.0
        This program uses FreeImage, a free, open source image library supporting all common bitmap formats. See http://freeimage.sourceforge.net for details
        OpenGL not available - denied!


        ---------------- Segmentation fault :( ----------------

        LwOS::exceptionHandler(int)
        linux_vdso.so.1() [0x7ffffffff514]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(+0x2e16f0) [0x8142e16f0]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(+0x1822c4) [0x8141822c4]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(+0x1a3ae8) [0x8141a3ae8]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(+0x18e21a) [0x81418e21a]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(+0x1a39b8) [0x8141a39b8]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(+0x18eb62) [0x81418eb62]
        /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0(g_type_create_instance+0x207) [0x80331c357]
        /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0(+0x19615) [0x803302615]
        /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0(g_object_new_with_properties+0x29d) [0x803303b1d]
        /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0(g_object_new+0xc1) [0x8033045f1]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(+0x1ac48b) [0x8141ac48b]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(+0x395879) [0x814395879]
        /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0(g_type_create_instance+0x207) [0x80331c357]
        /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0(+0x19615) [0x803302615]
        /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0(g_object_new_valist+0x414) [0x803304264]
        /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0(g_object_new+0x99) [0x8033045c9]
        /usr/lib/x86_64-linux-gnu/libgtk-3.so.0(gtk_message_dialog_new+0x195) [0x814265675]
        Shell::messageBox(LightweightString<wchar_t> const&, LightweightString<wchar_t> const&, iShell::eMessageButtons, iShell::eMessageIcon) const
        canvas_init(iRootWindow::InitArgs const&, Lw::Ptr<iWallpaperProvider, Lw::DtorTraits, Lw::InternalRefCountTraits> const&)
        glib_init(iRootWindow::InitArgs const&, Lw::Ptr<iWallpaperProvider, Lw::DtorTraits, Lw::InternalRefCountTraits> const&)
        /opt/linux/debian-11/usr/lib/lightworks/ntcardvt(+0xe639d) [0x110739d]

        ---------------------------------------

       "
```

this was under the following shell command:


```
env -u LC_ALL -u DISPLAY env HOME=/opt/linux/debian-11/tmp/tmphome ktrace -dif ntcardvt_tmphome_nodisp.out /opt/linux/debian-11/usr/lib/lightworks/ntcardvt
```



shkhln said:


> Why is this even a question? 2^64 bytes.



Trying to figure out what results in "16777216.0 TB" by the calculation used in ntcardvt

Will take a look at what an strace cmd might show, in a Lightworks installation on Debian 10.

Perhaps it could be useful to update the linux_base-c7 port and related to CentOS 8-stream, have begun working on this already with some tooling in Ruby lol. They've stopped producing single releases now, it's fairly like Debian 'sid' in the 8-stream series. That would at least result in it running with a different set of GTK libraries, from the CentOS 8-stream builds


----------



## neogeo (Feb 21, 2022)

After setting  the compat.linux.osrelease sysctl mib to something that may be closer to the Linux kernel version the Debian 11 build was produced for (`sudo sysctl compat.linux.osrelease=5.10.0`) the debug output is different then


```
70785 ntcardvt GIO   fd 1 wrote 1906 bytes
       "Lightworks 64-bit (Release, build Lw : 132926 Common : 132108 Dependencies : 132677 dated Feb  9 2022)
                                                                                 
        OS Version            : Linux version 5.10.0 (des@freebsd.org) (gcc version Clang 11.0.1) #4 Sun Dec 18 04:30:00 CET 1977
        Available memory      : 10.9 GB                                          
        Total memory          : 15.8 GB                                          
        Virtual Address Range : 16777216.0 TB                 
        Number of CPUs        : 4                                                
        User has admin rights : No                                               
        Graphics driver       : Intel Open Source Technology Center
        OpenGL version        : 3.0 Mesa 20.3.5
                                                                                 
        11:32:44.386: Setting project base directory /opt/linux/debian-11/tmp/tmphome/Lightworks/Projects/
        11:32:44.386: Starting window manager 
        11:32:44.387: FreeImage version: 3.18.0     
        This program uses FreeImage, a free, open source image library supporting all common bitmap formats. See http://freeimage.sourceforge.net for details
        11:32:44.588: License initialisation error <License initialisation error>
        11:32:44.593: Checking library versions..
        ASSERTION : Lightworks, No .fx files found in /usr/share/lightworks/Shaders/Shaders
        11:32:44.599: Preferred device System Sound is not available, loading software only mode instead.
        11:32:44.600: WARNING!! There are no audio resources present        
                                                                                 
        ---------------- Segmentation fault :( ----------------
                                       
        LwOS::exceptionHandler(int)
        linux_vdso.so.1() [0x7ffffffff514]                                
        /opt/linux/debian-11/usr/lib/lightworks/ntcardvt(+0xc2550) [0x10e3550]
        /opt/linux/debian-11/usr/lib/lightworks/ntcardvt(+0xc63cd) [0x10e73cd]
        BackgroundTaskBase::execute()                                            
        BackgroundTaskQueueBase::despatch(Lw::Ptr<iBackgroundTask, Lw::DtorTraits, Lw::InternalRefCountTraits>&)
        DecouplingQueue<iBackgroundTask>::decoupledThread(void*)                                                                                                     
        Thread::protectedExecute(iThread::eExecRC (*)(void*), void*, void (*)(iThread::Exception const&))
        Thread::execute(void*)
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x8037acea7]
        /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x805505def]


        ---------------------------------------

       "
```

Plenty to debug, here. It seems to get further along in its runtime with this configuration, though It's not picking up any audio support, and still the segfault from somewhere in this bytecode


----------



## _martin (Feb 21, 2022)

neogeo said:


> linux_vdso.so.1() [0x7ffffffff514]


It never returned from the vdso, which at first glance could be incomplete implementation of vdso part in ABI on FreeBSD side.

With the limits commands I see this:


```
70304 ntcardvt CALL  linux_mmap2(0,0x10000000000,0,0x22,0xffffffff,0)
 70304 ntcardvt RET   linux_mmap2 -1 errno -12 Cannot allocate memory      
 70304 ntcardvt CALL  linux_mprotect(0xfffffffffffff000,0x2d6ef,0x3)       
 70304 ntcardvt RET   linux_mprotect -1 errno -22 Invalid argument
```
Which seems it can't handle that limit but execution still continues.

When I looked at the "parent" thread you mentioned I see this:

```
* thread #4, name = 'ntcardvt', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x00000000010e3550
->  0x10e3550: movq   (%rax), %rdx
    0x10e3553: movq   0x48(%rdx), %rdx
    0x10e3557: cmpq   0x30780a(%rip), %rdx
    0x10e355e: jne    0x10e3f88
```
Well ok, error is obvious - attempt to read the 0 addr. I don't see how you got here though as your trace above doesn't reflect this dump.

I don't have much experience with lldb, I prefer gdb. I'd check the equivalent command of `info threads` to see what was each thread doing and check its stack trace. I'd leave the limits and valgrind alone and rather focus on stracing (truss-ing) the command to see where it fails.

You could check the code at `0x7ffffffff514`. It's not an aligned address (which is a must in higher versions of Linux, part of the exploit mitigations). It's fair to assume then syscall was called and error occured in it (i.e. in the vdso code). You could check which syscall it is and check why it fails.


----------



## neogeo (Feb 21, 2022)

_martin said:


> It never returned from the vdso, which at first glance could be incomplete implementation of vdso part in ABI on FreeBSD side.
> 
> With the limits commands I see this:
> 
> ...



Thank you for the help in interpreting the output here. The address [0x7ffffffff514] with the local linux_vdso.so.1 appears to be consistent across at least two of the segfaulted processes, although the internal backtrace by the lightworks ntcardvt sometimes differs up to that point, after any changes in the calling environment.

Notwithstanding the differences in the backtrace printed to log by ntcardvt, the backtrace from lldb seems to be consistent across subsequent calls. Using some local tooling here, it appears to be an equivalent backtrace with the following:



> $ lscript ntcardvt_tmphome_lldb_2.log env -u LC_ALL env HOME=/opt/linux/debian-11/tmp/tmphome lldb /opt/linux/debian-11/usr/lib/lightworks/ntcardvt
> 
> Script started, output file is ntcardvt_tmphome_lldb_2.log
> #-- env -u LC_ALL env HOME=/opt/linux/debian-11/tmp/tmphome lldb /opt/linux/debian-11/usr/lib/lightworks/ntcardvt
> ...



This is after updating the environment with `sudo sysctl compat.linux.osrelease=5.10.0`

If it could be of any remote help towards a port for Lightworks on FreeBSD, I'd tried to attach the unabridged kdump output from this call to ntcardvt to this forum post. The file is 2.4MiB in size however, and cannot be attached as a *.gz file. The ktrace *.out file itself is 1.9 MiB in size.

I'll take a look at linux_vdso.so.1 with hexdump, and will try to read up more about this at Safari Books Online.

Though the machine does not have 10000G of any resource, it fails like previously if run with the following:

```
limits -v 10000G env -u LC_ALL ktrace -dif /tmp/ntcardvt.out /opt/linux/compat-c7/usr/lib/lightworks/ntcardvt
```

There, it seems to not run into any soft limits (??) but then it segfaults like described in the earlier thread. The initial GUI is displayed but it falls out from there.

As the following and with any narrower limit, it's back to the application behavior described here - no GUI, and the segfault


```
limits -v 1000G env -u LC_ALL ktrace -dif /tmp/ntcardvt.out /opt/linux/compat-c7/usr/lib/lightworks/ntcardvt
```

Apologies about the awkward nature of my own  guesswork here. I've actually tried bootstrapping a full pkgsrc system to install a different set of libraries for it, to see if that might improve the situation. There were a lot of unrelated issues with that cross-OS  build (linux OS tgt, same machine, clang from Debian) under pkgsrc.

I'm not sure if it could be improved with any different set of GTK libraries, as I can only guess where the segfault is arriving from and how to debug this. WIll try to read up more about it though


----------



## _martin (Feb 21, 2022)

Remotely debugging X application that is attempting to use Linux ABI - now that's some punishment. 

If you can show the output using gdb I can share my thoughts. All these are gdb commands. Launch gdb and execute `set disassembly-flavor intel`. Then share the output of  `info threads`, then `info files` so we get the rough memory picture what is where. Rough disassembly around crash is also helpful: `disass 0x10e3550-0x30, 0x10e3550+030`. First instructions in the disass output could be butchered but that's ok.

This way you get the idea where it fails and on what it fails. Quite possibly you could get the same amount of information about this crash with using truss.

I don't know how you got to the valgrind and even why you assumed memory leak. I don't see any indication of that here. Also even if there was a leak there's usually no reason for program to crash (until you ran out of memory).


----------



## neogeo (Feb 21, 2022)

_martin said:


> Remotely debugging X application that is attempting to use Linux ABI - now that's some punishment.
> 
> If you can show the output using gdb I can share my thoughts. All these are gdb commands. Launch gdb and execute `set disassembly-flavor intel`. Then share the output of  `info threads`, then `info files` so we get the rough memory picture what is where. Rough disassembly around crash is also helpful: `disass 0x10e3550-0x30, 0x10e3550+030`. First instructions in the disass output could be butchered but that's ok.



Thanks for the advice! I'll build gdb with poudriere and will try that out



_martin said:


> This way you get the idea where it fails and on what it fails. Quite possibly you could get the same amount of information about this crash with using truss.
> 
> I don't know how you got to the valgrind and even why you assumed memory leak. I don't see any indication of that here. Also even if there was a leak there's usually no reason for program to crash (until you ran out of memory).



Truss is definitely a lot more informative. It even serves to show some of the original project settings from the buiild, where it linked onto TLS and SSL libs.

If it could be of any interest, I've attached the unabridged log from the truss call, in excerpt - it might be in the GTK libraries installed under the debootstrap filesystem?



> 71255 102306: linux_sys_futex(0x80923cf38,0x81,0x7fffffff,0x0,0x0,0x0) = 0 (0x0)
> 71255 102306: linux_sys_futex(0x80923cf38,0x81,0x7fffffff,0x0,0x0,0x0) = 0 (0x0)
> 71255 102306: linux_sys_futex(0x80923cf38,0x81,0x7fffffff,0x0,0x0,0x0) = 0 (0x0)
> 71255 102306: linux_ioctl(0x2,0x5401,0x7fffffffbd10) ERR#-25 'Inappropriate ioctl for device'
> ...



After that odd number of TB from the backtrace and log in ntcardvt, I'd thought it might be a good time to try out valgrind - perhaps that was some failing guesswork, it was at least a new idea after trying to get pkgsrc to build for this over the weekend ...

thx all for the advice and clarifications, I'll go build gdb now w/ poudriere


----------



## _martin (Feb 21, 2022)

This is interesting:

```
71255 102306: linux_ioctl(0x2,0x5401,0x7fffffffbd30) ERR#-25 'Inappropriate ioctl for device'
71255 102306: linux_getpid()                     = 71255 (0x11657)
71255 102306: gettimeofday({ 1645476641.164176 },0x0) = 0 (0x0)

^-- ** this would be called from vdso as a fast syscall

71255 102306: write(2,"\n(ntcardvt:71255): Gtk-CRITICAL **: 12:50:41.164: _gtk_style_provider_private_get_settings: assertion 'GTK_IS_STYLE_PROVIDER_PRIVATE (provider)' failed\n",152) = 152 (0x98)
71255 102306: SIGNAL 11 (SIGSEGV) code=SEGV_MAPERR trapno=12 addr=0x18

^-- ** we do see bogus virtual address 0x18 which is a reason for SIGSEGV
```
For better understanding though we need to poke around the crash.

Also that error before regarding gtk stuff is probably not related.


----------



## shkhln (Feb 21, 2022)

_martin said:


> ```
> 71255 102306: gettimeofday({ 1645476641.164176 },0x0) = 0 (0x0)
> 
> ^-- ** this would be called from vdso as a fast syscall
> ```


Not when logged by truss, no.



_martin said:


> Also that error before regarding gtk stuff is probably not related.


Are those assertions non-fatal? Would be better to get rid of the error.


----------



## neogeo (Feb 21, 2022)

I've enabled a lot of GTK debug options in the environment, referring to documentation for GTK 4. I'm not certain if those will be informative as to the failure here, albeit. I'll elide those from the debug output here



_martin said:


> If you can show the output using gdb I can share my thoughts. All these are gdb commands. Launch gdb and execute `set disassembly-flavor intel`. Then share the output of  `info threads`, then `info files` so we get the rough memory picture what is where. Rough disassembly around crash is also helpful: `disass 0x10e3550-0x30, 0x10e3550+030`. First instructions in the disass output could be butchered but that's ok.



Up to the point of the segfault, and then the `info threads` command


```
Gdk-Message: 13:53:08.131: Detectable autorepeat supported.                                                                                                          
[New LWP 102504 of process 71691]                                                                                                                                    
[New LWP 102506 of process 71691]                                                                                                                                    
[New LWP 102509 of process 71691]                                                                                                                                    
[New LWP 102533 of process 71691]                                                                                                                                                                                                             
Thread 5 received signal SIGSEGV, Segmentation fault.   
[Switching to LWP 102533 of process 71691]          
0x00000000004abcc2 in ?? ()                                                      
(gdb) info threads                                                               
  Id   Target Id                   Frame                                         
  1    LWP 102103 of process 71691 0x00000008077a8124 in ?? ()
  2    LWP 102504 of process 71691 0x00000008084b8c61 in ?? ()
  3    LWP 102506 of process 71691 0x0000000806be57b2 in ?? ()
  4    LWP 102509 of process 71691 0x0000000806be57b2 in ?? ()
* 5    LWP 102533 of process 71691 0x00000000004abcc2 in ?? ()
```

then with `info files`


```
(gdb) info files            
Symbols from "/opt/linux/compat-c7/usr/lib/lightworks/ntcardvt".
Native process:             
        Using the running image of child LWP 102533 of process 71691.
        While running this, GDB does not access memory from...
Local exec file:
        `/opt/linux/compat-c7/usr/lib/lightworks/ntcardvt', file type elf64-x86-64.
        Entry point: 0x4a8a90
        0x00000000004002a8 - 0x00000000004002c4 is .interp
        0x00000000004002c4 - 0x00000000004002e4 is .note.ABI-tag
        0x00000000004002e4 - 0x0000000000400308 is .note.gnu.build-id
        0x0000000000400308 - 0x00000000004027ec is .gnu.hash
        0x00000000004027f0 - 0x0000000000410e30 is .dynsym
        0x0000000000410e30 - 0x000000000042b972 is .dynstr
        0x000000000042b972 - 0x000000000042cca2 is .gnu.version
        0x000000000042cca8 - 0x000000000042cd88 is .gnu.version_r
        0x000000000042cd88 - 0x0000000000471860 is .rela.dyn
        0x0000000000471860 - 0x0000000000477d40 is .rela.plt
        0x0000000000478000 - 0x000000000047801b is .init
        0x0000000000478020 - 0x000000000047c370 is .plt
        0x000000000047c370 - 0x00000000005532f2 is .text
        0x00000000005532f4 - 0x0000000000553301 is .fini
        0x0000000000554000 - 0x0000000000559708 is .rodata
        0x0000000000559708 - 0x000000000055dca4 is .eh_frame_hdr
        0x000000000055dca8 - 0x0000000000575cd8 is .eh_frame
        0x0000000000575cd8 - 0x0000000000580e65 is .gcc_except_table
        0x0000000000581f00 - 0x0000000000581fb8 is .init_array
        0x0000000000581fb8 - 0x0000000000581fc0 is .fini_array
        0x0000000000581fc0 - 0x00000000005b1748 is .data.rel.ro
        0x00000000005b1748 - 0x00000000005b1d68 is .dynamic
        0x00000000005b1d68 - 0x00000000005b2000 is .got
        0x00000000005b2000 - 0x00000000005b41b8 is .got.plt
        0x00000000005b41b8 - 0x00000000005b4218 is .data
        0x00000000005b4220 - 0x00000000005bef68 is .bss
```

backtrace in gdb:

```
(gdb) backtrace
#0  0x00000000004abcc2 in ?? ()
#1  0x00000000004b02dd in ?? ()
#2  0x00000008046b76fc in ?? ()
#3  0x0000000804611f58 in ?? ()
#4  0x4277f1e462882224 in ?? ()
#5  0x0000000000000005 in ?? ()
#6  0x0000000808106d98 in ?? ()
#7  0x0000010833a3b8c0 in ?? ()
#8  0x00000008005975ca in ?? ()
#9  0x0000010833a3b8e0 in ?? ()
#10 0x0000000000000000 in ?? ()
```

This was with the recommended disassembly-flavor set before run. The disassembler output:


```
(gdb) disass 0x10e3550-0x30, 0x10e3550+030
Dump of assembler code from 0x10e3520 to 0x10e3568:
   0x00000000010e3520:  loopne 0x10e34ce
   0x00000000010e3522:  fimul  DWORD PTR [rax]
   0x00000000010e3524:  or     BYTE PTR [rax],al
   0x00000000010e3526:  add    BYTE PTR [rax],al
   0x00000000010e3528:  shl    BYTE PTR [rdx+rbx*8+0x808],0x0
   0x00000000010e3530:  movabs al,ds:0xa63f
   0x00000000010e3539:  rex.X cmps BYTE PTR ds:[rsi],BYTE PTR es:[rdi]
   0x00000000010e353b:  add    BYTE PTR [rax],al
   0x00000000010e353d:  add    BYTE PTR [rax],al
   0x00000000010e353f:  add    BYTE PTR [rax],ah
   0x00000000010e3541:  rex.X cmps BYTE PTR ds:[rsi],BYTE PTR es:[rdi]
   0x00000000010e3543:  add    BYTE PTR [rax],al
   0x00000000010e3545:  add    BYTE PTR [rax],al
   0x00000000010e3547:  add    BYTE PTR [rax+0x42],al
   0x00000000010e354a:  cmps   BYTE PTR ds:[rsi],BYTE PTR es:[rdi]
   0x00000000010e354b:  add    BYTE PTR [rax],al
   0x00000000010e354d:  add    BYTE PTR [rax],al
   0x00000000010e354f:  add    BYTE PTR [rax-0x53],dh
   0x00000000010e3552:  fimul  DWORD PTR [rax]
   0x00000000010e3554:  or     BYTE PTR [rax],al
   0x00000000010e3556:  add    BYTE PTR [rax],al
   0x00000000010e3558:  pop    rax
   0x00000000010e3559:  movs   BYTE PTR es:[rdi],BYTE PTR ds:[rsi]
   0x00000000010e355a:  fimul  DWORD PTR [rax]
   0x00000000010e355c:  or     BYTE PTR [rax],al
   0x00000000010e355e:  add    BYTE PTR [rax],al
   0x00000000010e3560:  and    BYTE PTR [rax-0x4c],bl
   0x00000000010e3563:  add    BYTE PTR [rax],al
   0x00000000010e3565:  add    BYTE PTR [rax],al
   0x00000000010e3567:  add    BYTE PTR [rax+0x3e],al
End of assembler dump.
```



I've run ntcardvt under gdb with the following, it results in copious debug messages from GTK. I'm not certain if any of the prefix paths may affect how it runs under this desktop environment. The GDK_DEBUG=interactive command does result in a new debug window that pops up with the GUI when run under gdb, I'm afraid it's not very interesting past the segfault albeit.


```
env -u LC_ALL DESKTOP_STARTUP_ID=com.example.nop GDK_PIXBUF_MODULE_FILE=/opt/linux/debian11/usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders.cache GTK_EXE_PREFIX=/opt/linux/debian/usr GTK_PATH=/opt/linux/debian-11/usr GTK_MEDIA=gstreamer GSK_DEBUG=renderer,cairo,opengl,shaders,surfaces,vulkan,fallback,glyphcache GSK_RENDERER=cairo GTK_A11Y=none GTK_CSD=0 GDK_BACKEND=x11 GTK_DEBUG=interactive GDK_DEBUG=cursor,eventloop,misc,frames,settings,selection,clipboard,dnd,opengl,vulkan gdb /opt/linux/compat-c7/usr/lib/lightworks/ntcardvt
```

Debugging under truss with those verbose options enabled, the inappropriate ioctl messages seem to pop up after various things in GTK, broadly speaking, like gsettings and glade builder API calls. Perhaps it could have anything to do with dbus calls under the Debian gtk build? though unrelated to the segfault


----------



## _martin (Feb 21, 2022)

The disass function would make sense only if you did it on the crash dump you had before. Now it's invalid as you disassembled something different.
So for the crash you shared now please include (gdb commands)

```
info proc map
thread 5
bt
x/5i 0x4abcc2
x/20i  0x4abcc2-0x20
i r
```
I may edit my post later.

edit1: you could just run the command as you are used to and then analyze the core dump. Make sure you have `ulimit -c unlimited` set, `sysctl kern.coredump` should return 1 (by default it's on I think).
If you ran this last command within gdb session then you don't have the core dump. Idea with the disass is to see the instructions around the failed instruction. Here:

```
[Switching to LWP 102533 of process 71691]            
0x00000000004abcc2 in ?? ()
```
You see which thread of which pid failed on what $pc (address). Use the address you'll find in the next core dump you'll see in either gdb session or if you run the command and check the core.


----------



## neogeo (Feb 21, 2022)

_martin said:


> The disass function would make sense only if you did it on the crash dump you had before. Now it's invalid as you disassembled something different.
> So for the crash you shared now please include (gdb commands)
> 
> ```
> ...



Sure, and thanks!

After `info proc map`


```
(gdb) thread 5                                                                                                                                                     
[Switching to thread 5 (LWP 102533 of process 71691)]                                                                                                              
#0  0x00000000004abcc2 in ?? ()                                                                                                                                    
(gdb) bt                                                                                                                                                           
#0  0x00000000004abcc2 in ?? ()                                                                                                                                    
#1  0x00000000004b02dd in ?? ()                                                                                                                                    
#2  0x00000008046b76fc in ?? ()                                                                                                                                    
#3  0x0000000804611f58 in ?? ()                                                                                                                                    
#4  0x4277f1e462882224 in ?? ()                                                                                                                                    
#5  0x0000000000000005 in ?? ()                                                                                                                                    
#6  0x0000000808106d98 in ?? ()                                                                                                                                    
#7  0x0000010833a3b8c0 in ?? ()                                                                                                                                    
#8  0x00000008005975ca in ?? ()                                                                                                                                    
#9  0x0000010833a3b8e0 in ?? ()                                                                                                                                    
#10 0x0000000000000000 in ?? ()

(gdb) x/5i 0x4abcc2                                                                                                                                                
=> 0x4abcc2:    mov    rax,QWORD PTR [rax]                                                                                                                         
   0x4abcc5:    mov    rax,QWORD PTR [rax+0x48]                                                                                                                    
   0x4abcc9:    cmp    rax,0x4ad900                                                                                                                                
   0x4abcd0:    jne    0x4accf0                                                                                                                                    
   0x4abcd6:    mov    BYTE PTR [rdi+0x28],0x1

gdb) x/20i  0x4abcc2-0x20                                                                                                                                         
   0x4abca2:    or     eax,0x24bc8d48                                                                                                                              
   0x4abca7:    add    BYTE PTR [rax],0x0                                                                                                                          
   0x4abcaa:    add    al,ch                                                                                                                                       
   0x4abcac:    mov    al,0xd0                                                                                                                                     
   0x4abcae:    (bad)                                                                                                                                              
   0x4abcaf:    (bad)                                                                                                                                              
   0x4abcb0:    call   0x47b890 <_ZN21LwDeviceDriverManager17restorePrefOutputEv@plt>                                                                              
   0x4abcb5:    call   0x478b90 <_ZN15LwAudioResource8instanceEv@plt>                                                                                              
   0x4abcba:    call   0x47c2a0 <_ZN12LwAudioMixer8instanceEv@plt>                                                                                                 
   0x4abcbf:    mov    rdi,rax
=> 0x4abcc2:    mov    rax,QWORD PTR [rax]
   0x4abcc5:    mov    rax,QWORD PTR [rax+0x48]
   0x4abcc9:    cmp    rax,0x4ad900
   0x4abcd0:    jne    0x4accf0
   0x4abcd6:    mov    BYTE PTR [rdi+0x28],0x1
   0x4abcda:    call   0x47c2a0 <_ZN12LwAudioMixer8instanceEv@plt>
   0x4abcdf:    mov    rdi,rax
   0x4abce2:    mov    rax,QWORD PTR [rax]
   0x4abce5:    call   QWORD PTR [rax+0x68]
   0x4abce8:    call   0x4786a0 <_ZN17AudioMixerManager8instanceEv@plt>
(gdb) i r
rax            0x0                 0
rbx            0x10833a3b4d0       1134737732816
rcx            0x80c260310         34563556112
rdx            0x0                 0
rsi            0x80c260310         34563556112
rdi            0x0                 0
rbp            0x10833a3b3f0       0x10833a3b3f0
rsp            0x10833a3b380       0x10833a3b380
r8             0x0                 0
r9             0x3c                60
r10            0xfffffffffffffd54  -684
r11            0x8031c3930         34411919664
r12            0x10833a3b3c0       1134737732544
r13            0x10833a3b3b0       1134737732528
r14            0x10833a3b3e0       1134737732576
r15            0x80c200010         34563162128
rip            0x4abcc2            0x4abcc2
eflags         0x10206             [ PF IF RF ]
cs             0x43                67
ss             0x3b                59
ds             <unavailable>
es             <unavailable>
fs             <unavailable>
gs             <unavailable>
fs_base        0x10833a3c700       1134737737472
gs_base        0x0                 0
```

the `info proc map` output was too long to lncude inline. I'll attach as a text file

If guesswork could serve any use, I notice the audio-related symbols in the gdb output from `x/20i  0x4abcc2-0x20`.  I've installed the linux c7 alsa ports, but there was the matter that lightworks was not able to detect audio support, as it indicated in some log data. After ldd, I believe it's linked onto alsa libs in Linux.

At some point, when I was installing the RPM under the centos-c7 filesystem from ports, it showed a missing dependency of a libudev.so.1. That message of a missing dependency somehow disappeared since, under the /opt/linux/compat-c7 filesystem but I believe it's now picking up libraries from the Debian filesystem there, per sysctl `compat.linux.emul_path: /opt/linux/debian-11`. Could it be that any of this could be related to Linux udev?

I've not yet tested out this Debian configuration with any other audio support, or under the centos-c7 ports. I'll try to test that with the audacity audio app, and to reproduce this process with the lightworks install under the c7 filesystem. Certainly this is more productive than debugging build failures with libtool under some other port system .... pkgsrc does provide a wip/libudev port but I've not been able to bootstrap the pkgsrc system under this cross-os environment. It bootstraps normally with only the compilers under FreeBSD


----------



## _martin (Feb 21, 2022)

Ok, now this is better. The code is relying on `LwAudioMixer::instance()` returning something sane:

```
0x4abcba:    call   0x47c2a0 <_ZN12LwAudioMixer8instanceEv@plt>                                                                                      
      0x4abcbf:    mov    rdi,rax
   => 0x4abcc2:    mov    rax,QWORD PTR [rax]
```
But this function returned NULL (%rax is 0). You need to dig deeper in that function. So it seems it's audio related.
This function comes from the library `libAud.so`.

It would help to know if this is a proprietary library or if it's an opensource one. Then you could check what it does and what it wants.

You could set the breakpoint to 0x4abcba (`b *0x4abcba`) and step in with the `si` to the function. It will first jump of the plt to resolve the actual address but eventually it will land in libAud.so. There it's all about RE to see what it does. The code around the error suggests it returns some sort of structure.
Maybe it would be easier to see if you get any audio device in the emulated Linux environment.

Looking back at your initial post this seems to be very important then:

```
19:55:22.490: Preferred device System Sound is not available, loading software only mode instead.
 19:55:22.490: WARNING!! There are no audio resources present
```


----------



## neogeo (Feb 21, 2022)

_martin said:


> Ok, now this is better. The code is relying on `LwAudioMixer::instance()` returning something sane:
> 
> ```
> 0x4abcba:    call   0x47c2a0 <_ZN12LwAudioMixer8instanceEv@plt>
> ...



Thanks! I've been wholly unable to debug this.

This libAud.so is from the Lightworks Debian release, one of the Lightworks libraries provided under usr/lib/lightworks/.

It seems that they've also bundled some local builds of some foss tools in the Lightworks release. They have provided a  `libportaudio.so.2` for instance, as well as a number of other libraries that might normally be available on the host, under some build version. I wonder if it may be possible to work out any audio hacks with portaudio here?

While libAud.so is a release from LWKS, maybe there could be any help for this in the forums there. I'll try to read up more about gdb, then to see what shows up with the breakpoint and code stepping procedure you recommended, before opening any forum threads there.

I'll also try to configure my Debian laptop and this FreeBSD laptop both for portaudio. Albeit in some guesswork, if I can discover how to configure Lightworks and Alsa to work out with portaudio on the Debian machine, maybe it could be possible to carry over some of that configuration, if not to set up some kind of an audio bridge under the Debian environment here on FreeBSD.

Happily, the Lightworks installation on the Debian machine may be of some use here after all.


----------



## _martin (Feb 21, 2022)

While I'm using FreeBSD for long time I never had much luck (and patience) with Linux ABI before. I was able to use Skype on it before but that's about it. So I don't know what would work and what not when it comes to this ABI.

From some breadcrumbs in the library itself I see `/home/lwks/workspace/development/lightworks/branches/lwks-release-2022.1/ole/Aud/ContentAnalysis.cpp` which would suggest it's a closed project.

You could do the static analysis of that function too but you are derailing yourself a lot from the original idea - to run that program. 

```
$ readelf -Wa libAud.so |grep _ZN12LwAudioMixer8instanceE
00000000002de100  000006ad00000007 R_X86_64_JUMP_SLOT     00000000000a0e00 _ZN12LwAudioMixer8instanceEv + 0
  1709: 00000000000a0e00   420 FUNC    GLOBAL DEFAULT   12 _ZN12LwAudioMixer8instanceEv
$

$ gdb libAud.so
Reading symbols from libAud.so...
(No debugging symbols found in libAud.so)
(gdb) disass _ZN12LwAudioMixer8instanceEv
Dump of assembler code for function _ZN12LwAudioMixer8instanceEv:
   0x00000000000a0e00 <+0>:    push   rbp
   0x00000000000a0e01 <+1>:    push   rbx
...
...
   0x00000000000a0f9f <+415>:    call   0x43890 <_Unwind_Resume@plt>
```
I must say after heap pwning doing RE on C++  is just next on my "meh list". Maybe worth saying you don't need to demangle those names yourself, you can use demanger for that.

To understand that function you need to understand the data it's working with. That can get hard fast. So maybe it's worth googling around (and even here on the forums) what are your possibilities of using audio devices under this emulator.

edit: Out of my curiosity: I checked the crash from the other thread you referenced here too. Logic of the code seems to be the same though code is slightly different and in different .text offsets. Did you use different version of the ntcardvt ?


----------



## Paul Floyd (Aug 14, 2022)

neogeo said:


> I'd like to test this binary from Lightworks for Linux with valgrind, to see if that may help to debug the issue of the segfault. However I'm seeing the following when trying to run valgrind with this code:
> 
> 
> ```
> ...



No. Valgrind does the loading of the guest executable, and it is hard coded for each platform. That means that Valgrind on FreeBSD expects FreeBSD elf and FreeBSD ld.so.

If you want to run a Linux binary on FreeBSD under Valgrind you would need a Linux build of Valgrind.


----------

