How does ld find the path to libbfd?

[CPPAWhile]

I have the file libbfd-2.26.1-system.so in the right place and with the proper symbolic link. I believe I have my LD_LIBRARY_PATH and PATH set up ok (although I'm not absolutely certain of this). When I run

strace -v ./pathto/ld

I get a few missing stuff having to do with etc/ld.so.preload which I normally don't get, I usually get that etc/ld.so.nohwcap and etc/ld.so.preload cannot be found. But anyway, usually not finding those won't stop the linker and it proceeds to find the file libbfd-whatever and loads it fine.

The problem is that the open call to open this in the run of ld is looking in the wrong path. Pretty far off as it has a weird /tls directory referred to. Any ideas why the path is so weird. I would think I have the correct versions of everything but clearly I'll double-check this. Any ideas why I can't run a good link with ld appreciated and perhaps should I fix up all the preload and other ld.so stuff and then retry? Or can I set it - I'm almost tempted just to set up a symlink in the "wrong" directory to point it right. The path should just be /usr/lib/x86_64-linux-gnu/... but it has /lib64/tls/x86_64/... which is pretty far off.

Should I just experiment by helping the ld process with a symlink that points it in the right direction just to see what happens? I might segfault so I don't know what is wise to do here.

[kbw]

I have the file libbfd-2.26.1-system.so in the right place and with the proper symbolic link.

What makes it the right place? Where is it?

I believe I have my LD_LIBRARY_PATH and PATH set up ok

PATH isn't used to find shared libraries. What is LD_LIBRARY_PATH?

The problem is that the open call to open this in the run of ld is looking in the wrong path.

We don't know what you mean.

Can you please restate the problem without assuming we know about your program and where it is.

[CPPAWhile]

I have the file libbfd-2.26.1-system.so in the right place and with the proper symbolic link.

I am bootstrapping a gcc compiler and it compiles fine so far, but does not link. When I do this, I simply create a directory, then I expand all the binaries from about 30 or so packages and I proceed to test the compiler and linker and runtimes made from them. So I am still setting things up but I wanted to test the linker so I am running the command strace -v to do that.

/usr/lib/x86_64-linux-gnu is a typical place for libbfd-2.26.1-system.so to reside. On all of my linux distros that I'm concerned with, the file is in that directory.

If I run strace on a machine that does NOT have any problems with ld, I would get an strace -v that looks something like this in the beginning:

CORRECT ld, or at least good enough to link, let's say:

execve("/usr/bin/ld", .......) = 0
brk(Null)
access(/etc/ld.so.nohwcap)
access(/etc/ld.so.preload)
open(/etc/ld.so.cache ...
fstat
mmap
close(3)
access(/etc/ld.so.nohwcap...
open("/usr/lib/x86_64-linux-gnu/libbfd-2.26.1-system.so ...) = 3

etc..

I am being brief and cryptic above just to illustrate the basic flow.

The difference in the case of the one that does not work is that the path to the open command is incorrect - I just don't know why - is it package-related?

If you need more detail, I'll have to put together a more detailed analysis. But I'm wondering, is it cache-related and if so, should I clear my caches out or should I find the incorrect entry and edit them? I realize I might be completely in the dark - it might have little to do with the caches. So I have two compilers - an old one and a more recent one that I am bootstrapping with binary installs of all the packages.

The tls reference has something to do with threading - so perhaps I need to turn something on or off?

I know that the structure of my ld.so.conf script might have a lot to do with it. So I will play with this this evening. My gnu machine has a slightly different way of dealing with .conf than typical ubuntu distros so I need to get this configged better.

[kbw]

So, is LD_LIBRARY_PATH set to /usr/lib/x86_64-linux-gnu? And is that where libbfd-2.26.1-system.so is?

And are you saying that using the same procedure, it works on some machines and not others?

[CPPAWhile]

So, is LD_LIBRARY_PATH set to /usr/lib/x86_64-linux-gnu? And is that where libbfd-2.26.1-system.so is?

And are you saying that using the same procedure, it works on some machines and not others?

I think I should pause here from posting and simply take a good look at all of the possible path environment variable issues, .conf file issues and simply make sure it all looks like it "should" work rather than take stabs with the compiler. The fact that it goes off on a thread local storage search is absolutely nuts - I need to just get rid of that by reinstalling stuff or re-configuring stuff and make real sure it doesn't happen again before I ask questions. I realize I don't know why yet, but I haven't been 100% sure that I have it all on this machine.

[mbozzi]

Both ld and libbfd are part of binutils.

Because of this, my first impression is that the new binutils and gcc have been configured to be incompatible, or there is no new binutils at all, or gcc doesn't know where to look for the new binutils.

GCC is supposed to let you bootstrap binutils and GCC (and MPFR, GMP, etc.) in parallel. Did you do this?

[CPPAWhile]

I had forgotten to install libc-bin. I have pretty much the files I need to compile and link now I would think - at least the base of them. I will keep checking. Here's an untar of what libc-bin adds to the mix. So I'm now running with a new ldconfig. So what I wonder is what should I do to get that configured. I realize that there is a lot of caching of stuff since the binary cache is faster than text. I don't seem to still have the libbfd error but I'm getting a segmentation fault. I will rebuild, then do an strace to see what it finds. It might actually be tripping up on libc.so.6 - strange but I'll make sure I have this, and in the right place. Then I need to recompile from scratch and see what's going on.

./
./sbin/
./sbin/ldconfig
./sbin/ldconfig.real
./etc/
./etc/ld.so.conf
./etc/default/
./etc/default/nss
./etc/ld.so.conf.d/
./etc/ld.so.conf.d/libc.conf
./etc/bindresvport.blacklist
./etc/gai.conf
./usr/
./usr/bin/
./usr/bin/catchsegv
./usr/bin/localedef
./usr/bin/zdump
./usr/bin/getconf
./usr/bin/locale
./usr/bin/tzselect
./usr/bin/pldd
./usr/bin/iconv
./usr/bin/ldd
./usr/bin/getent
./usr/sbin/
./usr/sbin/iconvconfig
./usr/sbin/zic
./usr/lib/
./usr/lib/locale/
./usr/lib/locale/C.UTF-8/
./usr/lib/locale/C.UTF-8/LC_TIME
./usr/lib/locale/C.UTF-8/LC_NUMERIC
./usr/lib/locale/C.UTF-8/LC_CTYPE
./usr/lib/locale/C.UTF-8/LC_MONETARY
./usr/lib/locale/C.UTF-8/LC_IDENTIFICATION
./usr/lib/locale/C.UTF-8/LC_ADDRESS
./usr/lib/locale/C.UTF-8/LC_COLLATE
./usr/lib/locale/C.UTF-8/LC_TELEPHONE
./usr/lib/locale/C.UTF-8/LC_NAME
./usr/lib/locale/C.UTF-8/LC_PAPER
./usr/lib/locale/C.UTF-8/LC_MEASUREMENT
./usr/lib/locale/C.UTF-8/LC_MESSAGES/
./usr/lib/locale/C.UTF-8/LC_MESSAGES/SYS_LC_MESSAGES
./usr/share/
./usr/share/libc-bin/
./usr/share/libc-bin/nsswitch.conf
./usr/share/lintian/
./usr/share/lintian/overrides/
./usr/share/lintian/overrides/libc-bin
./usr/share/doc/
./usr/share/doc/libc-bin/
./usr/share/doc/libc-bin/copyright
./usr/share/man/
./usr/share/man/man1/
./usr/share/man/man1/tzselect.1.gz
./usr/share/man/man1/getconf.1.gz
./usr/share/man/man1/catchsegv.1.gz
./usr/share/doc/libc-bin/changelog.Debian.gz

I can now explain why it searched so strangely for libbfd before, it probably didn't find it, then kept going further down my path variable and came across lib64 and kept searching or something? Anyway, I'm trying to check paths and things - paths DO matter in the absence of other variables. I need to make sure everything is picked up somehow.

[CPPAWhile]

I ran the command ./usr/bin/gcc-5 -v just to check some stuff. I will go through this and make sure it's ok and see if it helps me with my includes. I need to check that gcc-5 is still compiling - it was, but may not still be. I put the compiler in a directory called dev5.4.0-6. Everything is relative to that directory. I am just bootstrapping the binary compiler - version 5.4.0-6. This compiler works fine on other machines.


# ./usr/bin/gcc-5 -v
Using built-in specs.
COLLECT_GCC=./usr/bin/gcc-5
COLLECT_LTO_WRAPPER=/home/spot/dev5.4.0-6/usr/bin/../lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with:
../src/configure -v
--with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.10'
--with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs
--enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++
--prefix=/usr
--program-suffix=-5
--enable-shared
--enable-linker-build-id
--libexecdir=/usr/lib
--without-included-gettext
--enable-threads=posix
--libdir=/usr/lib
--enable-nls
--with-sysroot=/
--enable-clocale=gnu
--enable-libstdcxx-debug
--enable-libstdcxx-time=yes
--with-default-libstdcxx-abi=new
--enable-gnu-unique-object
--disable-vtable-verify
--enable-libmpx
--enable-plugin
--with-system-zlib
--disable-browser-plugin
--enable-java-awt=gtk
--enable-gtk-cairo
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre
--enable-java-home
--with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64
--with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64
--with-arch-directory=amd64
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar
--enable-objc-gc
--enable-multiarch
--disable-werror
--with-arch-32=i686
--with-abi=m64
--with-multilib-list=m32,m64,mx32
--enable-multilib
--with-tune=generic
--enable-checking=release
--build=x86_64-linux-gnu
--host=x86_64-linux-gnu
--target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)
#

I should mention that there are a few things perhaps still to install from build-essential list as the system may go out and pick up older versions although I should almost be there by now.

Update: I did a trace of gcc-5 and it is not finding libc.so.6 - I need to answer why. I think at one point it was finding that. Why did this status change on finding libc.so.6. It's just a basic file used by gcc. The file (symbolic link) is in it's normal location which is /lib/x86_64-linux-gnu, pointing to libc-2.23.so. Hmm? cc1 exe which does the work of gcc has a new conf file - I need to see that as that is what cannot find libc.so.6 I believe.

[CPPAWhile]

I just ran strace on ldconfig as follows:

strace ./sbin/ldconfig -v &> ldconfig5.out and it explains a lot of the troubles. The commands for the new compiler are not picking up the right libraries - they are picking up the old "lib64" stuff. So it would be almost impossible not to get a segmentation fault. I need to for now "bootstrap" the ldconfig command and other commands such as ld to find the right shared libraries. ldconfig is used to refresh the cache after a change in the ld configuration. If one does this correctly, one often does NOT have to specify a bunch of libraries or at least the command can ulitmately let it be know which -L or whatever will be needed - something like that. So first off, I need to get a working ldconfig command.

[keskiverto]

Red Hat did introduce Software Collections (SCL) for RHEL. RHEL release has lifespan of 10+ years and therefore tends to not look like cutting edge. Not on release and definitely not at end.

The SCL are "short-lived"; package lifetime of mere 3 years. Their idea is to be managed packages that can be installed "on the side" without disturbing anything in the "default locations". The files go under /opt on RHEL.

The default binaries depend on default ldconfig's config, which is in /etc/ld.so.conf.d/. Normal packages can add their special paths there. (Don't know whether other distros use that.)

The SCL packages have to be explicitly enabled. Say "start make with environment" or "start bash with environment".

Here is example of the environment set in order to enable devtoolset-7 (provides GCC 7.3.1):

# General environment variables
export PATH=/opt/rh/devtoolset-7/root/usr/bin${PATH:+:${PATH}}
export MANPATH=/opt/rh/devtoolset-7/root/usr/share/man:${MANPATH}
export INFOPATH=/opt/rh/devtoolset-7/root/usr/share/info${INFOPATH:+:${INFOPATH}}
export PCP_DIR=/opt/rh/devtoolset-7/root
# Some perl Ext::MakeMaker versions install things under /usr/lib/perl5
# even though the system otherwise would go to /usr/lib64/perl5.
export PERL5LIB=/opt/rh/devtoolset-7/root//usr/lib64/perl5/vendor_perl:/opt/rh/devtoolset-7/root/usr/lib/perl5:/opt/rh/devtoolset-7/root//usr/share/perl5/vendor_perl${PERL5LIB:+:${PERL5LIB}}
# bz847911 workaround:
# we need to evaluate rpm's installed run-time % { _libdir }, not rpmbuild time
# or else /etc/ld.so.conf.d files?
rpmlibdir=$(rpm --eval "%{_libdir}")
# bz1017604: On 64-bit hosts, we should include also the 32-bit library path.
if [ "$rpmlibdir" != "${rpmlibdir/lib64/}" ]; then
  rpmlibdir32=":/opt/rh/devtoolset-7/root${rpmlibdir/lib64/lib}"
fi
export LD_LIBRARY_PATH=/opt/rh/devtoolset-7/root$rpmlibdir$rpmlibdir32${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export LD_LIBRARY_PATH=/opt/rh/devtoolset-7/root$rpmlibdir$rpmlibdir32:/opt/rh/devtoolset-7/root$rpmlibdir/dyninst$rpmlibdir32/dyninst${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
# duplicate python site.py logic for sitepackages
pythonvers=2.7
export PYTHONPATH=/opt/rh/devtoolset-7/root/usr/lib64/python$pythonvers/site-packages:/opt/rh/devtoolset-7/root/usr/lib/python$pythonvers/site-packages${PYTHONPATH:+:${PYTHONPATH}}

The point is that "desired custom versions" could be found by prepending to PATH and LD_LIBRARY_PATH without messing with the ldconfig's config.

Furthermore, system's ldconfig is still used; the devtoolset does not provide its own. There are binutils, elfutils, gcc, make, gdb, strace, ld, but no ldconfig.


That said, the devtoolset-7 was not bootstrapped&build by me. I have no idea how the maintainers have to juggle.

[CPPAWhile]

Wow, keskiverto, there is a lot to know. As a result, I've decided that I will play a new game I invented - it's called RIGHT or WRONG. And it's played in my dev5.4.0-6 directory where I am bootstrapping a compiler so that I can then build one from source later.

But first I need to know some rules - after all, perhaps the rules will explain why my runs are trying to find a file that is in a path that I don't even care to have on my computer.

Rule 1. ldconfig creates the necessary links and cache (for use by the run-time linker, ld.so) to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories (/usr/lib and /lib). ... (This syntax also works on the command line).

Notice that in Rule 1, that they refer to /etc/ld.so.conf - which in turn has:
include /etc/ld.so.conf.d/*.conf

Aha - so that little ole file named libc.conf which is one of the *.conf's in my ld.so.conf.d directory just might be looked at and ldconfig might just decide to use it's information and act accordingly, but whoa - in that file is a line:
/usr/local/lib - ouch!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

So the first WRONG is that line. For now, I might change that to ./usr/local/lib just to get it out of the way, I don't yet know what I want to set it to - perhaps to the actual path just to see if it has any effect.

Rule 2. /etc/ld.so.preload - I will explain this rule later as my computer does not use this currently and it could be a sort of "security-hack" open-door type of thing so we'll skip that for now.

Rule 3. I guess I need to either 1) don't use ldconfig and get rid of all the caching or 2) use ldconfig but with the -n option to not check so-called "trusted" directories. I pretty much want ldconfig to skip those. I guess I don't understand what ldconfig really does. I had wanted a command that gives me two environments or three or whatever. I wonder if it can really do that. If I run ldconfig -v now - I'm not sure it will do what I want. And I am fearful that my old gcc compiler is going to stop working. The whole process of having two compilers needs to be explained but every single discussion I've seen assumes that you are building from source. The whole idea of having a binary file is so you can forget about the source code. This is crazy.

ONE THING I NEED TO KNOW BEFORE RUNNING ldconfig:
Can I set some option so that I'm absolutely sure which etc caches are updated. I want two sets of caches - my original I want untouched. I want only the ones "under" my dev5.4.0-6 changed in any way. I need to make sure this is true. I don't want to make the originals read only as my bootup process might hang or something. But if I look at the caches - which is hard since they are binary - perhaps I can see what is being written to?
So far, I can tell that no ls.so.cache has been written to and I think I can supply a flag to ldconfig to not do any writing - that would help me a huge amount so I guess Rule 3 should be re-worded as you need to run it by adding the safest flags you can find just to see what is happening first.
So I need to use -n (implies -N), -X, -C (this would allow me to supply my own cache file which would be nice). And there is an option -p to print the current cache. In my case, this SHOULD print nothing. I'll see what happens.
Update to Rule 3: My ldconfig run which I straced reads in a bunch of environment variables - like 59 of them. It probably uses those variables so I guess I need to override those. It simply looks in the wrong places perhaps due to those variables. I hope so because that's all I have to go on right now. So I need to turn off that kind of reading or override them some how or come up with a different set which I don't really want to do.
Rule 3.1: In life you have to do things that you "don't really want to do" so I propose the following as my next run of ldconfig: Definitely override at least two of those enviros, namely LD_LIBRARY_PATH and PATH because currently they are definitely pointing to the wrong paths. LOL. So the next time when I strace I should at least see something different - perhaps a disappearance of the tls (thread local storage) hunting. If so, I can at least say I'm making progress even if I'm a lousy linux engineer overall.

[CPPAWhile]

I have now given a new LD_LIBRARY_PATH and PATH for ldconfig when it runs and it reads those environment variables in or at least something does - like strace.

But it is still going to the tls directory. I have no idea how it gets this. I need to know what causes it to look for the correct names of things but in the wrong place (path).

[CPPAWhile]

I am just running tests of gcc under strace and I realize that running without a cache makes the gcc command and ldconfig or any command using shared libraries make a lot of calls to the open system call until it finds the library. So the /tls path is ok - it's not the problem - my fear shown in my above comments was due to misunderstanding that without a cache it looks all over. My gcc trace eventually finds the libc.so.6 and it's further down that I get a segmentation fault - but I might just be missing something.

So from this point on, I have three choices while I debug the run of gcc-5:
1) Use the old ldconfig on my machine to build up a cache and this will make things found faster and with less searching as the open command will jump right to the correct path by reading the cache file.

2) Defer the creation of a cache until later and simply continue debugging since it's ok to go without a cache as long as it hits the file eventually and can open it.

3) This is not a choice but a task - I need to find out what's missing that's causing the segmentation fault when I run gcc-5 -v.

But it's ok to run without a cache for now as long as I set PATH and LD_LIBRARY_PATH appropriately - just kind of slow.

I do know that my strace makes a successful call to arch_prctl, then should call mprotect after that but seg-faults. Hmm - need to find out why. Or is it barfing on something missing about locale. Not sure yet.

[CPPAWhile]

Plan for the day is to analyze the output of a "good" ldd -v verbose of gcc under strace. The -v flag will go further than just the top layer and list out all of the .so shared libraries, and if need be, later I'll use objdump but this should get me started. I'm hoping and praying that I'm missing a shared library or two.

[CPPAWhile]

Update to my compiler troubles. I am still working with a pretty minimal install of gcc-5, based mostly on examining dependency lists (this is binary only given by apt-depends, ldd -v, and objdump types of commands), and occasionally browsing the net to find a site that has the binaries. The apt-depends on Ubuntu systems is useful and also occasionally look at dpkg logs such as dpkg.log.1 eg.

In going over these systems, I discerned that I lack a build-locale-archive compiler. I simply don't have it and I have NOT installed glibc which might have it. I have also not installed libclib2 or whatever it's called - I'll have to update this, simply because I think I can run without these - just my feeling. So lacking the complete install of locale even though I DID install most or all of the locale files, I decided to add -LANG=en_US.UTF-8, this is with the hope of guiding gcc-5 through it's compile and more or less telling it to skip looking for a locale-compiled file in /usr/lib/locale where I am missing that file.

Well, to make a long story short, for the first time in many days, gcc-5 compiled a file without any segmentation faults, and exited 0 = success.

So now I have to try to link this thing and get my Hello World to run. I guess it won't be easy, but at least turning off the search for the localization file is helping things.

Progress as it's nice to NOT have a segmentation fault.

So the following command is what I used to compile successfully, for only the 2nd or 3rd time in a week (I decided to put the strace on it right away):
./usr/bin/strace -v gcc-5 -I./usr/include -I./usr/include/x86_64-linux-gnu -v -LANG=en_US.UTF-8 -c test.c

[CPPAWhile]

I've decided to start working with ldconfig and using all the options it has. I can create a new cache using the -C option and ldconfig puts it where I want. But I strongly suspect that it isn't putting the right paths in. I will need to struggle with all the different paths and shared library paths.

But one thing kind of bothers me - the use of "current cache" in the man page for the -p (print) option. What does it mean by current cache? I know I can always specify a particular cache with the -C option but I don't think that the -C option is really meant to be used with -p. But I will need to experiment an awful lot because I keep expecting to see the directory /root/spot/dev5.4.0-6/ prepended to everything but so far I never see it. Somehow I have to come up with a set of parameters that forces all of my command that have to do with ldconfig, ldd, ld, and gcc-5 to have the right paths but it never does. It always uses the wrong directories even though I keep setting the correct ones in every place I can think of. Very frustrating command they are.

It's nice that people publish links to how to configure gcc and white papers on how to do it for multiple gcc installation but they all start off by assuming that the person is compiling gcc-5. None simply assume that someone is installing the binaries, ie. how to set up a system to allow all the binaries to be found.

[CPPAWhile]

I had to run the ldconfig command a number of times to try to get the library location hints. I've managed to finally get it to start looking in the right directories. The problem is that if I specify ldconfig -n -X -C cachefile libdir1 libdir2 etc.., it will not build a new cache. I need to leave off the -n to get it to build the new cache. The problem with that is that it will still pick up some libraries from the old system - do I want this? I don't know yet. Having different versions of the same-named file is a bear. It always wants to go to the trusted directories unless I specify -n with ldconfig. So I run it now without -n and hope for the best. So my library list is kind of mixed. At least I'm starting to get them in there - ie. the libraries in the correct locations (not just any one the ldconfig finds in it's search paths).

ldconfig -X -C /mycustomgcc-5path/etc/cachefile libdir1 libdir2 etc..

So I now have a 2nd cache. Now I want to see if any of the new gcc-5 apps can run off that cache-hint set.

Update:
I compiled a Hello World program, but cannot link it yet until I solve the link path problem - probably easy to kludge once I get to it, but I took the .o file to another machine and linked it - it linked and ran successfully. So at least my current toolchain can take a .c file and make a reasonably ok .o file - enough to link into a program - and it DOES use the version of binutils that I wanted.