[mongodb-dev] Including new library into mongodb using scons

[SH SHin]

Hi,

I have already included my new library libtto.so in /usr/lib and libtto.h
in /usr/include.

After modifying some part of codes to utilize the functions included in
libtto, I tried scons -j16 all, and what I saw was this.

src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to
'tto_get_bg_task()'
src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to
'tto_begin(unsigned long)'
src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to
'tto_end(unsigned long)'
src/mongo/db/instance.cpp:237: error: undefined reference to
'tto_begin(unsigned long)'
src/mongo/db/instance.cpp:256: error: undefined reference to
'tto_end(unsigned long)'
src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:106: error:
undefined reference to 'taio_begin(unsigned long)'
src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:111: error:
undefined reference to 'taio_begin(unsigned long)'
collect2: error: ld returned 1 exit status
scons: *** [build/opt/mongo/db/exec/sort_test] Error 1


What I have tried so far is, (library hwloc is required for libtto)

in src/mongo/SConscript,
.
.
.

env.Append(LIBPATH=['tto'])
env.Append(LIBPATH=['hwloc'])
env.Append(LIBPATH=['/usr/lib'])
env.Append(CPPPATH=['tto'])
env.Append(CPPPATH=['hwloc'])
env.Append(CPPPATH=['/usr/include'])
.
.
.

mongod = env.Program(
target="mongod",
source=[
"db/db.cpp",
"db/mongod_options_init.cpp",
],
LIBDEPS=mongodLibDeps,
SYSLIBDEPS=[
"tto",
"hwloc",
],
)
.
.
.

env.Library(
target='base',
source=baseSource,
LIBDEPS=baseLibDeps,
SYSLIBDEPS=[
"tto",
"hwloc"
],
)
.
.

in src/third_party/wiredtiger/SConstruct
.
.

conf = Configure(env)
conf.env.Append(LIBPATH=['tto'])
conf.env.Append(LIBPATH=['hwloc'])
conf.env.Append(CPPPATH=['tto'])
conf.env.Append(CPPPATH=['hwloc'])
.
.


in src/third_party/wiredtiger/SConscript
.
.
env.Append(LIBPATH=['tat'])
env.Append(LIBPATH=['hwloc'])
env.Append(LIBPATH=['/usr/lib'])
env.Append(CPPPATH=['tto'])
env.Append(CPPPATH=['hwloc'])
env.Append(CPPPATH=['/usr/include'])

baseLibDeps=[
..
.

wtlib = env.Library(
target='wiredtiger,
source=wtsources,
LIBDEPS=[
'$BUILD_DIR/third_party/shim_snappy',
'$BUILD_DIR/third_party/shim_zlib',
],
SYSLIBDEPS=[
"tto",
"hwloc"
],
)
.
.
.


And also tried similar modification on the top SConscript but always same
result as shown above.

Is there any more modification that I should put in to let scons recognize
my new library??

I've been stuck with this problem for several days and no progress....

['Andrew Morrow' via mongodb-dev]

The part where you are adding tto and hwloc to the SYSLIBDEPS of various
things seems fine, but the other changes are not correct. First, you
shouldn't be hardcoding changes to CPPPATH and LIBPATH into the SConscript
like this. Second, you seem to be mixing paths and library names when you
do so. And finally, you shouldn't need to add /usr/include and /usr/lib to
the include and library search paths, since they are automatically searched
by the toolchain unless you are doing something very unusual.

Instead you can pass additional paths by adding them to the SCons
invocation command line when you call it, and those directories will be
searched as needed. So if you have a package foo in /opt/foo/{include,lib} and
a package bar in /opt/bar/{include, lib} that you want to be found, then
add foo and bar to SYSLIBDEPS as needed, and then add CPPATH="/opt/foo/include
/opt/bar/include" LIBPATH="/opt/foo/lib /opt/bar/lib" to the command line
when you invoke SCons. Note that I wouldn't expect you to need to do this
for hwloc as I would expect it to appear in the system include and library
search paths. I don't know what tto is, so I can't speak as to the correct
way to locate it, but the general idea is the same.

Thanks,
Andrew

Post by SH SHin
Hi,
I have already included my new library libtto.so in /usr/lib and libtto.h
in /usr/include.
After modifying some part of codes to utilize the functions included in
libtto, I tried scons -j16 all, and what I saw was this.
src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to
'tto_get_bg_task()'
src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to
'tto_begin(unsigned long)'
src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to
'tto_end(unsigned long)'
src/mongo/db/instance.cpp:237: error: undefined reference to
'tto_begin(unsigned long)'
src/mongo/db/instance.cpp:256: error: undefined reference to
'tto_end(unsigned long)'
undefined reference to 'taio_begin(unsigned long)'
undefined reference to 'taio_begin(unsigned long)'
collect2: error: ld returned 1 exit status
scons: *** [build/opt/mongo/db/exec/sort_test] Error 1
What I have tried so far is, (library hwloc is required for libtto)
in src/mongo/SConscript,
.
.
.
env.Append(LIBPATH=['tto'])
env.Append(LIBPATH=['hwloc'])
env.Append(LIBPATH=['/usr/lib'])
env.Append(CPPPATH=['tto'])
env.Append(CPPPATH=['hwloc'])
env.Append(CPPPATH=['/usr/include'])
.
.
.
mongod = env.Program(
target="mongod",
source=[
"db/db.cpp",
"db/mongod_options_init.cpp",
],
LIBDEPS=mongodLibDeps,
SYSLIBDEPS=[
"tto",
"hwloc",
],
)
.
.
.
env.Library(
target='base',
source=baseSource,
LIBDEPS=baseLibDeps,
SYSLIBDEPS=[
"tto",
"hwloc"
],
)
.
.
in src/third_party/wiredtiger/SConstruct
.
.
conf = Configure(env)
conf.env.Append(LIBPATH=['tto'])
conf.env.Append(LIBPATH=['hwloc'])
conf.env.Append(CPPPATH=['tto'])
conf.env.Append(CPPPATH=['hwloc'])
.
.
in src/third_party/wiredtiger/SConscript
.
.
env.Append(LIBPATH=['tat'])
env.Append(LIBPATH=['hwloc'])
env.Append(LIBPATH=['/usr/lib'])
env.Append(CPPPATH=['tto'])
env.Append(CPPPATH=['hwloc'])
env.Append(CPPPATH=['/usr/include'])
baseLibDeps=[
..
.
wtlib = env.Library(
target='wiredtiger,
source=wtsources,
LIBDEPS=[
'$BUILD_DIR/third_party/shim_snappy',
'$BUILD_DIR/third_party/shim_zlib',
],
SYSLIBDEPS=[
"tto",
"hwloc"
],
)
.
.
.
And also tried similar modification on the top SConscript but always same
result as shown above.
Is there any more modification that I should put in to let scons recognize
my new library??
I've been stuck with this problem for several days and no progress....
--
You received this message because you are subscribed to the Google Groups
"mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/
msgid/mongodb-dev/16c9deee-6df9-4033-8c41-a966260453c5%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-dev/16c9deee-6df9-4033-8c41-a966260453c5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[SH SHin]

Thanks for your reply. Here's what I've done.

libtto is implemented by ourselves, and it is successfully installed in our
/usr/lib and /usr/include as libtto, libtto.so, libtto.so.1, and
libtto.so.1.0.1 in /usr/lib and libtaio.h in /usr/include.

I got rid of all lines manually adding CPPPATH and LIBPATH in all
SConstruct and SConscript in mongodb, /src/mongo, and
/src/third_pary/wiredtiger. Didn't modify SYSLIBDEPS part since you
mentioned it is okay.



Since the recommendation was there is nothing to do for /usr/lib, and
/usr/include, and the target library libtto is installed in those folders,
then I have nothing to modify in my cmd-line. instead I tried to include
/usr/include/x86_64-linux-gnu and /usr/lib/x86_64-linux-gnu for CPPPATH and
LIBPATH repectively, where libhwloc is located at. And same results come
out.

*src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to
'tto_get_bg_task_id()'*
*src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to
'tto_task_begin(unsigned long)'*
*src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to
'tto_task_end(unsigned long)'*
*src/mongo/db/instance.cpp:237: error: undefined reference to
'tto_task_begin(unsigned long)'*
*src/mongo/db/instance.cpp:256: error: undefined reference to
'tto_task_end(unsigned long)'*
*src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:106: error:
undefined reference to 'tto_task_begin(unsigned long)'*
*src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:111: error:
undefined reference to 'tto_task_begin(unsigned long)'*


cmd-line was
scons CPPPATH="/usr/include/x86_64-linux-gnu"
LIBPATH="/usr/lib/x86_86-linux-gnu" -j16 all

If it is true that scons automatically collect the library information from
CPPPATH and LIBPATH(I think this is right because we checked through
CheckLib that the program acknowledges hwloc, but not tto, our
self-implemented library), does that mean that there is a problem on our
self-implemented library libtto? then, do I need to put the source code for
libtto into mongodb manually?

Post by 'Andrew Morrow' via mongodb-dev
The part where you are adding tto and hwloc to the SYSLIBDEPS of various
things seems fine, but the other changes are not correct. First, you
shouldn't be hardcoding changes to CPPPATH and LIBPATH into the SConscript
like this. Second, you seem to be mixing paths and library names when you
do so. And finally, you shouldn't need to add /usr/include and /usr/lib to
the include and library search paths, since they are automatically searched
by the toolchain unless you are doing something very unusual.
Instead you can pass additional paths by adding them to the SCons
invocation command line when you call it, and those directories will be
searched as needed. So if you have a package foo in /opt/foo/{include,lib} and
a package bar in /opt/bar/{include, lib} that you want to be found, then
add foo and bar to SYSLIBDEPS as needed, and then add CPPATH="/opt/foo/include
/opt/bar/include" LIBPATH="/opt/foo/lib /opt/bar/lib" to the command line
when you invoke SCons. Note that I wouldn't expect you to need to do this
for hwloc as I would expect it to appear in the system include and
library search paths. I don't know what tto is, so I can't speak as to
the correct way to locate it, but the general idea is the same.
Thanks,
Andrew

Post by SH SHin
Hi,
I have already included my new library libtto.so in /usr/lib and libtto.h
in /usr/include.
After modifying some part of codes to utilize the functions included in
libtto, I tried scons -j16 all, and what I saw was this.
src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to
'tto_get_bg_task()'
src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to
'tto_begin(unsigned long)'
src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to
'tto_end(unsigned long)'
src/mongo/db/instance.cpp:237: error: undefined reference to
'tto_begin(unsigned long)'
src/mongo/db/instance.cpp:256: error: undefined reference to
'tto_end(unsigned long)'
undefined reference to 'taio_begin(unsigned long)'
undefined reference to 'taio_begin(unsigned long)'
collect2: error: ld returned 1 exit status
scons: *** [build/opt/mongo/db/exec/sort_test] Error 1
What I have tried so far is, (library hwloc is required for libtto)
in src/mongo/SConscript,
.
.
.
env.Append(LIBPATH=['tto'])
env.Append(LIBPATH=['hwloc'])
env.Append(LIBPATH=['/usr/lib'])
env.Append(CPPPATH=['tto'])
env.Append(CPPPATH=['hwloc'])
env.Append(CPPPATH=['/usr/include'])
.
.
.
mongod = env.Program(
target="mongod",
source=[
"db/db.cpp",
"db/mongod_options_init.cpp",
],
LIBDEPS=mongodLibDeps,
SYSLIBDEPS=[
"tto",
"hwloc",
],
)
.
.
.
env.Library(
target='base',
source=baseSource,
LIBDEPS=baseLibDeps,
SYSLIBDEPS=[
"tto",
"hwloc"
],
)
.
.
in src/third_party/wiredtiger/SConstruct
.
.
conf = Configure(env)
conf.env.Append(LIBPATH=['tto'])
conf.env.Append(LIBPATH=['hwloc'])
conf.env.Append(CPPPATH=['tto'])
conf.env.Append(CPPPATH=['hwloc'])
.
.
in src/third_party/wiredtiger/SConscript
.
.
env.Append(LIBPATH=['tat'])
env.Append(LIBPATH=['hwloc'])
env.Append(LIBPATH=['/usr/lib'])
env.Append(CPPPATH=['tto'])
env.Append(CPPPATH=['hwloc'])
env.Append(CPPPATH=['/usr/include'])
baseLibDeps=[
..
.
wtlib = env.Library(
target='wiredtiger,
source=wtsources,
LIBDEPS=[
'$BUILD_DIR/third_party/shim_snappy',
'$BUILD_DIR/third_party/shim_zlib',
],
SYSLIBDEPS=[
"tto",
"hwloc"
],
)
.
.
.
And also tried similar modification on the top SConscript but always same
result as shown above.
Is there any more modification that I should put in to let scons
recognize my new library??
I've been stuck with this problem for several days and no progress....
--
You received this message because you are subscribed to the Google Groups
"mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-dev/16c9deee-6df9-4033-8c41-a966260453c5%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-dev/16c9deee-6df9-4033-8c41-a966260453c5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

['Andrew Morrow' via mongodb-dev]

Several follow-up questions, please see below

Post by SH SHin
Thanks for your reply. Here's what I've done.
libtto is implemented by ourselves, and it is successfully installed in
our /usr/lib and /usr/include as libtto, libtto.so, libtto.so.1, and
libtto.so.1.0.1 in /usr/lib and libtaio.h in /usr/include.

OK.

Post by SH SHin
I got rid of all lines manually adding CPPPATH and LIBPATH in all
SConstruct and SConscript in mongodb, /src/mongo, and
/src/third_pary/wiredtiger. Didn't modify SYSLIBDEPS part since you
mentioned it is okay.
Since the recommendation was there is nothing to do for /usr/lib, and
/usr/include, and the target library libtto is installed in those folders,
then I have nothing to modify in my cmd-line. instead I tried to include
/usr/include/x86_64-linux-gnu and /usr/lib/x86_64-linux-gnu for CPPPATH and
LIBPATH repectively, where libhwloc is located at.

I'd be surprised if you need those either. Unless there is something funky
with your toolchain I'd expect the architecture specific include and
library search paths to be automatically searched. Are you using the system
toolchain, or one you rolled yourself? Can it still locate the hwloc.h
header if you don't add those?

Post by SH SHin
And same results come out.
*src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to
'tto_get_bg_task_id()'*
*src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to
'tto_task_begin(unsigned long)'*
*src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to
'tto_task_end(unsigned long)'*
*src/mongo/db/instance.cpp:237: error: undefined reference to
'tto_task_begin(unsigned long)'*
*src/mongo/db/instance.cpp:256: error: undefined reference to
'tto_task_end(unsigned long)'*
undefined reference to 'tto_task_begin(unsigned long)'*
undefined reference to 'tto_task_begin(unsigned long)'*

- Have you added 'tto' as a SYSLIBDEP to each of the libraries that contain
controller.cpp, instance.cpp, and wiredtiger_kv_engine.cpp? You should be.
- Have you verified that libtto.so.1 actually exports the symbols that are
not found here? You should be able to check that with nm.
- Can you post here the modified declarations for the libraries?

Post by SH SHin
cmd-line was
scons CPPPATH="/usr/include/x86_64-linux-gnu" LIBPATH="/usr/lib/x86_86-linux-gnu"
-j16 all

I'll re-iterate that I find it odd that you would need to specify the arch
specific directories unless something odd with your toolchain.

Post by SH SHin
If it is true that scons automatically collect the library information
from CPPPATH and LIBPATH(I think this is right because we checked through
CheckLib that the program acknowledges hwloc, but not tto, our
self-implemented library), does that mean that there is a problem on our
self-implemented library libtto? then, do I need to put the source code for
libtto into mongodb manually?

- Can you show the call to CheckLib that you are doing?
- You should try a little more to get it to work the way you are now.
Vendoring the code comes with its own set of problems. I'm pretty sure we
can get it to work.

Post by SH SHin

Post by 'Andrew Morrow' via mongodb-dev
The part where you are adding tto and hwloc to the SYSLIBDEPS of various
things seems fine, but the other changes are not correct. First, you
shouldn't be hardcoding changes to CPPPATH and LIBPATH into the
SConscript like this. Second, you seem to be mixing paths and library
names when you do so. And finally, you shouldn't need to add /usr/include
and /usr/lib to the include and library search paths, since they are
automatically searched by the toolchain unless you are doing something very
unusual.
Instead you can pass additional paths by adding them to the SCons
invocation command line when you call it, and those directories will be
searched as needed. So if you have a package foo in /opt/foo/{inclu
de,lib} and a package bar in /opt/bar/{include, lib} that you want to be
found, then add foo and bar to SYSLIBDEPS as needed, and then add CPPATH="/opt/foo/include
/opt/bar/include" LIBPATH="/opt/foo/lib /opt/bar/lib" to the command
line when you invoke SCons. Note that I wouldn't expect you to need to do
this for hwloc as I would expect it to appear in the system include and
library search paths. I don't know what tto is, so I can't speak as to
the correct way to locate it, but the general idea is the same.
Thanks,
Andrew

Post by SH SHin
Hi,
I have already included my new library libtto.so in /usr/lib and
libtto.h in /usr/include.
After modifying some part of codes to utilize the functions included in
libtto, I tried scons -j16 all, and what I saw was this.
src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to
'tto_get_bg_task()'
src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to
'tto_begin(unsigned long)'
src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to
'tto_end(unsigned long)'
src/mongo/db/instance.cpp:237: error: undefined reference to
'tto_begin(unsigned long)'
src/mongo/db/instance.cpp:256: error: undefined reference to
'tto_end(unsigned long)'
undefined reference to 'taio_begin(unsigned long)'
undefined reference to 'taio_begin(unsigned long)'
collect2: error: ld returned 1 exit status
scons: *** [build/opt/mongo/db/exec/sort_test] Error 1
What I have tried so far is, (library hwloc is required for libtto)
in src/mongo/SConscript,
.
.
.
env.Append(LIBPATH=['tto'])
env.Append(LIBPATH=['hwloc'])
env.Append(LIBPATH=['/usr/lib'])
env.Append(CPPPATH=['tto'])
env.Append(CPPPATH=['hwloc'])
env.Append(CPPPATH=['/usr/include'])
.
.
.
mongod = env.Program(
target="mongod",
source=[
"db/db.cpp",
"db/mongod_options_init.cpp",
],
LIBDEPS=mongodLibDeps,
SYSLIBDEPS=[
"tto",
"hwloc",
],
)
.
.
.
env.Library(
target='base',
source=baseSource,
LIBDEPS=baseLibDeps,
SYSLIBDEPS=[
"tto",
"hwloc"
],
)
.
.
in src/third_party/wiredtiger/SConstruct
.
.
conf = Configure(env)
conf.env.Append(LIBPATH=['tto'])
conf.env.Append(LIBPATH=['hwloc'])
conf.env.Append(CPPPATH=['tto'])
conf.env.Append(CPPPATH=['hwloc'])
.
.
in src/third_party/wiredtiger/SConscript
.
.
env.Append(LIBPATH=['tat'])
env.Append(LIBPATH=['hwloc'])
env.Append(LIBPATH=['/usr/lib'])
env.Append(CPPPATH=['tto'])
env.Append(CPPPATH=['hwloc'])
env.Append(CPPPATH=['/usr/include'])
baseLibDeps=[
..
.
wtlib = env.Library(
target='wiredtiger,
source=wtsources,
LIBDEPS=[
'$BUILD_DIR/third_party/shim_snappy',
'$BUILD_DIR/third_party/shim_zlib',
],
SYSLIBDEPS=[
"tto",
"hwloc"
],
)
.
.
.
And also tried similar modification on the top SConscript but always
same result as shown above.
Is there any more modification that I should put in to let scons
recognize my new library??
I've been stuck with this problem for several days and no progress....
--
You received this message because you are subscribed to the Google
Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/ms
gid/mongodb-dev/16c9deee-6df9-4033-8c41-a966260453c5%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-dev/16c9deee-6df9-4033-8c41-a966260453c5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups
"mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/
msgid/mongodb-dev/f0e36a4c-38bc-4152-949d-a3076adc7ed6%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-dev/f0e36a4c-38bc-4152-949d-a3076adc7ed6%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.