I was trying to recompile sysutils/bacula-client on a FreeBSD 8.4 jail when I encountered some perplexing link errors. It took me a while to figure it out, but with the help of some IRC conversation, we figured it out.
I started off by deleting the old (quite old) port: bacula-client-5.0.3
Then I recompiled: cd /usr/ports/sysutils/bacula-client && make install clean
The errors in question are:
Linking bacula-fd ... /var/ports/basejail/usr/ports/sysutils/bacula-client/work/bacula-5.2.12/libtool --silent --tag=CXX --mode=link /usr/bin/c++ -L/usr/local/lib -L../lib -L../findlib -o bacula-fd filed.o authenticate.o acl.o backup.o estimate.o fd_plugins.o accurate.o filed_conf.o heartbeat.o job.o pythonfd.o restore.o status.o verify.o verify_vol.o xattr.o -lz -lbacfind -lbacpy -lbaccfg -lbac -lm -lpthread -lintl -lwrap filed.o(.text+0x56): In function `terminate_filed(int)': : undefined reference to `flush_mntent_cache()' filed.o(.text+0x11a): In function `terminate_filed(int)': : undefined reference to `lmgr_cleanup_main()' filed.o(.text+0x790): In function `main': : undefined reference to `lmgr_init_thread()' authenticate.o(.text+0xeb): In function `authenticate_director(JCR*)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' authenticate.o(.text+0x10b): In function `authenticate_director(JCR*)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' authenticate.o(.text+0x694): In function `authenticate_storagedaemon(JCR*)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' authenticate.o(.text+0x6b4): In function `authenticate_storagedaemon(JCR*)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' backup.o(.text+0x3e3): In function `encode_and_send_attributes(JCR*, FF_PKT*, int&)': : undefined reference to `get_hangup()' backup.o(.text+0x47d): In function `encode_and_send_attributes(JCR*, FF_PKT*, int&)': : undefined reference to `encode_stat(char*, stat*, int, int, int)' backup.o(.text+0x507): In function `encode_and_send_attributes(JCR*, FF_PKT*, int&)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' backup.o(.text+0x543): In function `encode_and_send_attributes(JCR*, FF_PKT*, int&)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' backup.o(.text+0x76b): In function `encode_and_send_attributes(JCR*, FF_PKT*, int&)': : undefined reference to `set_hangup(int)' backup.o(.text+0x171e): In function `save_file(JCR*, FF_PKT*, bool)': : undefined reference to `ff_pkt_set_link_digest(FF_PKT*, int, char const*, unsigned int)' fd_plugins.o(.text+0x150): In function `free_plugins(JCR*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x19c): In function `free_plugins(JCR*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x217): In function `plugin_check_file(JCR*, char*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x2cb): In function `plugin_check_file(JCR*, char*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x4ed): In function `baculaAcceptFile(bpContext*, save_pkt*)': : undefined reference to `accept_file(FF_PKT*)' fd_plugins.o(.text+0x72d): In function `baculaCheckChanges(bpContext*, save_pkt*)': : undefined reference to `check_changes(JCR*, FF_PKT*)' fd_plugins.o(.text+0xecb): In function `load_fd_plugins(char const*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0xf1d): In function `load_fd_plugins(char const*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0xf7b): In function `load_fd_plugins(char const*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0xfb6): In function `load_fd_plugins(char const*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x1f1e): In function `plugin_name_stream(JCR*, char*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x2167): more undefined references to `bplugin_list' follow fd_plugins.o(.text+0x298c): In function `baculaGetValue(bpContext*, bVariable, void*)': : undefined reference to `beef' fd_plugins.o(.text+0x2c78): In function `new_plugins(JCR*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x2dd4): In function `new_plugins(JCR*)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x33cb): In function `plugin_estimate(JCR*, FF_PKT*, bool)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x34dc): In function `plugin_estimate(JCR*, FF_PKT*, bool)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x3a5c): In function `plugin_save(JCR*, FF_PKT*, bool)': : undefined reference to `bplugin_list' fd_plugins.o(.text+0x3bfc): more undefined references to `bplugin_list' follow accurate.o(.text+0xf4): In function `accurate_cmd(JCR*)': : undefined reference to `htable::init(void*, void*, int, int)' accurate.o(.text+0x525): In function `accurate_finish(JCR*)': : undefined reference to `decode_stat(char*, stat*, int, int*)' accurate.o(.text+0x755): In function `accurate_finish(JCR*)': : undefined reference to `decode_stat(char*, stat*, int, int*)' accurate.o(.text+0x900): In function `accurate_check_file(JCR*, FF_PKT*)': : undefined reference to `decode_stat(char*, stat*, int, int*)' heartbeat.o(.text+0x33): In function `start_dir_heartbeat(JCR*)': : undefined reference to `lmgr_thread_create(pthread**, pthread_attr* const*, void* (*)(void*), void*)' heartbeat.o(.text+0x7f): In function `start_heartbeat_monitor(JCR*)': : undefined reference to `lmgr_thread_create(pthread**, pthread_attr* const*, void* (*)(void*), void*)' job.o(.text+0x1eb1): In function `setdebug_cmd(JCR*)': : undefined reference to `set_hangup(int)' job.o(.text+0x1ec3): In function `setdebug_cmd(JCR*)': : undefined reference to `get_hangup()' job.o(.text+0x1f55): In function `setdebug_cmd(JCR*)': : undefined reference to `get_hangup()' job.o(.text+0x21c0): In function `filed_free_jcr(JCR*)': : undefined reference to `free_path_list(JCR*)' job.o(.text+0x4680): In function `bootstrap_cmd(JCR*)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' job.o(.text+0x46d1): In function `bootstrap_cmd(JCR*)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' restore.o(.text+0x1f2d): In function `do_restore(JCR*)': : undefined reference to `unpack_attributes_record(JCR*, int, char*, int, ATTR*)' restore.o(.text+0x1f6a): In function `do_restore(JCR*)': : undefined reference to `decode_stat(char*, stat*, int, int*)' restore.o(.text+0x2942): In function `do_restore(JCR*)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' restore.o(.text+0x297b): In function `do_restore(JCR*)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' status.o(.text+0x5e9): In function `list_status_header(STATUS_PKT*)': : undefined reference to `bplugin_list' status.o(.text+0x609): In function `list_status_header(STATUS_PKT*)': : undefined reference to `bplugin_list' status.o(.text+0x624): In function `list_status_header(STATUS_PKT*)': : undefined reference to `bplugin_list' status.o(.text+0x8e4): In function `list_running_jobs(STATUS_PKT*)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' status.o(.text+0x90e): In function `list_running_jobs(STATUS_PKT*)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' status.o(.text+0xc0e): In function `list_running_jobs(STATUS_PKT*)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' status.o(.text+0xc38): In function `list_running_jobs(STATUS_PKT*)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' verify.o(.text+0x591): In function `verify_file(JCR*, FF_PKT*, bool)': : undefined reference to `encode_stat(char*, stat*, int, int, int)' verify.o(.text+0x5b3): In function `verify_file(JCR*, FF_PKT*, bool)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' verify.o(.text+0x5dd): In function `verify_file(JCR*, FF_PKT*, bool)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' verify_vol.o(.text+0x4d9): In function `do_verify_volume(JCR*)': : undefined reference to `bthread_mutex_lock_p(pthread_mutex**, char const*, int)' verify_vol.o(.text+0x506): In function `do_verify_volume(JCR*)': : undefined reference to `bthread_mutex_unlock_p(pthread_mutex**, char const*, int)' /usr/local/lib/libbac.so: undefined reference to `RAND_load_file' /usr/local/lib/libbac.so: undefined reference to `CRYPTO_num_locks' /usr/local/lib/libbac.so: undefined reference to `X509V3_EXT_get' /usr/local/lib/libbac.so: undefined reference to `EVP_CIPHER_CTX_init' /usr/local/lib/libbac.so: undefined reference to `EVP_VerifyFinal' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_free' /usr/local/lib/libbac.so: undefined reference to `EVP_DigestFinal' /usr/local/lib/libbac.so: undefined reference to `X509_free' /usr/local/lib/libbac.so: undefined reference to `SSL_shutdown' /usr/local/lib/libbac.so: undefined reference to `PEM_read_bio_DHparams' /usr/local/lib/libbac.so: undefined reference to `X509_NAME_oneline' /usr/local/lib/libbac.so: undefined reference to `ERR_error_string_n' /usr/local/lib/libbac.so: undefined reference to `X509_get_ext' /usr/local/lib/libbac.so: undefined reference to `X509_STORE_CTX_get_error_depth' /usr/local/lib/libbac.so: undefined reference to `SSL_get_error' /usr/local/lib/libbac.so: undefined reference to `EVP_aes_128_cbc' /usr/local/lib/libbac.so: undefined reference to `BIO_s_socket' /usr/local/lib/libbac.so: undefined reference to `EVP_PKEY_decrypt' /usr/local/lib/libbac.so: undefined reference to `sk_num' /usr/local/lib/libbac.so: undefined reference to `CRYPTO_set_id_callback' /usr/local/lib/libbac.so: undefined reference to `OBJ_nid2sn' /usr/local/lib/libbac.so: undefined reference to `PEM_read_bio' /usr/local/lib/libbac.so: undefined reference to `SSL_read' /usr/local/lib/libbac.so: undefined reference to `EVP_md5' /usr/local/lib/libbac.so: undefined reference to `EVP_CipherUpdate' /usr/local/lib/libbac.so: undefined reference to `EVP_get_cipherbyname' /usr/local/lib/libbac.so: undefined reference to `SSL_new' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_new' /usr/local/lib/libbac.so: undefined reference to `ASN1_item_d2i' /usr/local/lib/libbac.so: undefined reference to `SSL_library_init' /usr/local/lib/libbac.so: undefined reference to `EVP_CIPHER_iv_length' /usr/local/lib/libbac.so: undefined reference to `OPENSSL_add_all_algorithms_noconf' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_set_default_passwd_cb' /usr/local/lib/libbac.so: undefined reference to `CRYPTO_set_dynlock_destroy_callback' /usr/local/lib/libbac.so: undefined reference to `PEM_read_bio_PrivateKey' /usr/local/lib/libbac.so: undefined reference to `X509_get_pubkey' /usr/local/lib/libbac.so: undefined reference to `EVP_CipherFinal_ex' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_load_verify_locations' /usr/local/lib/libbac.so: undefined reference to `X509_NAME_get_text_by_NID' /usr/local/lib/libbac.so: undefined reference to `EVP_sha512' /usr/local/lib/libbac.so: undefined reference to `ASN1_STRING_dup' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_set_verify' /usr/local/lib/libbac.so: undefined reference to `EVP_aes_192_cbc' /usr/local/lib/libbac.so: undefined reference to `EVP_MD_CTX_cleanup' /usr/local/lib/libbac.so: undefined reference to `EVP_aes_256_cbc' /usr/local/lib/libbac.so: undefined reference to `X509_verify_cert_error_string' /usr/local/lib/libbac.so: undefined reference to `ASN1_OBJECT_it' /usr/local/lib/libbac.so: undefined reference to `ASN1_item_new' /usr/local/lib/libbac.so: undefined reference to `ASN1_STRING_set' /usr/local/lib/libbac.so: undefined reference to `BIO_free' /usr/local/lib/libbac.so: undefined reference to `ASN1_STRING_free' /usr/local/lib/libbac.so: undefined reference to `CRYPTO_add_lock' /usr/local/lib/libbac.so: undefined reference to `SSL_set_bio' /usr/local/lib/libbac.so: undefined reference to `sk_push' /usr/local/lib/libbac.so: undefined reference to `EVP_sha1' /usr/local/lib/libbac.so: undefined reference to `CRYPTO_free' /usr/local/lib/libbac.so: undefined reference to `BIO_new_file' /usr/local/lib/libbac.so: undefined reference to `SSL_load_error_strings' /usr/local/lib/libbac.so: undefined reference to `ASN1_INTEGER_it' /usr/local/lib/libbac.so: undefined reference to `ASN1_item_i2d' /usr/local/lib/libbac.so: undefined reference to `EVP_PKEY_type' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_set_cipher_list' /usr/local/lib/libbac.so: undefined reference to `BIO_new' /usr/local/lib/libbac.so: undefined reference to `ASN1_STRING_cmp' /usr/local/lib/libbac.so: undefined reference to `OBJ_nid2obj' /usr/local/lib/libbac.so: undefined reference to `EVP_CIPHER_CTX_cleanup' /usr/local/lib/libbac.so: undefined reference to `sk_value' /usr/local/lib/libbac.so: undefined reference to `X509_NAME_get_index_by_NID' /usr/local/lib/libbac.so: undefined reference to `EVP_DigestInit_ex' /usr/local/lib/libbac.so: undefined reference to `EVP_CIPHER_CTX_block_size' /usr/local/lib/libbac.so: undefined reference to `X509_get_subject_name' /usr/local/lib/libbac.so: undefined reference to `TLSv1_method' /usr/local/lib/libbac.so: undefined reference to `PEM_read_bio_X509' /usr/local/lib/libbac.so: undefined reference to `X509_EXTENSION_get_object' /usr/local/lib/libbac.so: undefined reference to `EVP_MD_CTX_init' /usr/local/lib/libbac.so: undefined reference to `EVP_DigestUpdate' /usr/local/lib/libbac.so: undefined reference to `OBJ_obj2nid' /usr/local/lib/libbac.so: undefined reference to `X509_NAME_ENTRY_get_data' /usr/local/lib/libbac.so: undefined reference to `RAND_bytes' /usr/local/lib/libbac.so: undefined reference to `SSL_accept' /usr/local/lib/libbac.so: undefined reference to `X509_STORE_CTX_get_current_cert' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_ctrl' /usr/local/lib/libbac.so: undefined reference to `CRYPTO_set_locking_callback' /usr/local/lib/libbac.so: undefined reference to `EVP_cleanup' /usr/local/lib/libbac.so: undefined reference to `DH_free' /usr/local/lib/libbac.so: undefined reference to `EVP_SignFinal' /usr/local/lib/libbac.so: undefined reference to `SSL_ctrl' /usr/local/lib/libbac.so: undefined reference to `BIO_int_ctrl' /usr/local/lib/libbac.so: undefined reference to `X509_NAME_get_entry' /usr/local/lib/libbac.so: undefined reference to `ERR_get_error' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_use_certificate_chain_file' /usr/local/lib/libbac.so: undefined reference to `RAND_cleanup' /usr/local/lib/libbac.so: undefined reference to `ASN1_INTEGER_set' /usr/local/lib/libbac.so: undefined reference to `EVP_sha256' /usr/local/lib/libbac.so: undefined reference to `X509_get_ext_by_NID' /usr/local/lib/libbac.so: undefined reference to `EVP_CIPHER_key_length' /usr/local/lib/libbac.so: undefined reference to `EVP_CipherInit_ex' /usr/local/lib/libbac.so: undefined reference to `CRYPTO_set_dynlock_lock_callback' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_use_PrivateKey_file' /usr/local/lib/libbac.so: undefined reference to `EVP_PKEY_size' /usr/local/lib/libbac.so: undefined reference to `ERR_free_strings' /usr/local/lib/libbac.so: undefined reference to `X509_STORE_CTX_get_error' /usr/local/lib/libbac.so: undefined reference to `SSL_connect' /usr/local/lib/libbac.so: undefined reference to `X509_get_issuer_name' /usr/local/lib/libbac.so: undefined reference to `SSL_free' /usr/local/lib/libbac.so: undefined reference to `EVP_CIPHER_CTX_set_key_length' /usr/local/lib/libbac.so: undefined reference to `CRYPTO_set_dynlock_create_callback' /usr/local/lib/libbac.so: undefined reference to `SSL_CTX_set_default_passwd_cb_userdata' /usr/local/lib/libbac.so: undefined reference to `X509_get_ext_count' /usr/local/lib/libbac.so: undefined reference to `SSL_get_peer_certificate' /usr/local/lib/libbac.so: undefined reference to `ASN1_item_free' /usr/local/lib/libbac.so: undefined reference to `ASN1_OCTET_STRING_it' /usr/local/lib/libbac.so: undefined reference to `EVP_PKEY_free' /usr/local/lib/libbac.so: undefined reference to `SSL_write' /usr/local/lib/libbac.so: undefined reference to `EVP_bf_cbc' /usr/local/lib/libbac.so: undefined reference to `EVP_PKEY_encrypt' *** Error code 1 Stop in /var/ports/basejail/usr/ports/sysutils/bacula-client/work/bacula-5.2.12/src/filed. ====== Error in /var/ports/basejail/usr/ports/sysutils/bacula-client/work/bacula-5.2.12/src/filed ====== *** Error code 1 Stop in /var/ports/basejail/usr/ports/sysutils/bacula-client/work/bacula-5.2.12. *** Error code 1 Stop in /basejail/usr/ports/sysutils/bacula-client. *** Error code 1 Stop in /basejail/usr/ports/sysutils/bacula-client. [dan@dbclone:/usr/ports/sysutils/bacula-client] $
I tried recompiling a few ports, such as gettext. But that was quite futile.
The port compiled just fine in another jail on the same host, but not in this jail. I started looking at /usr/local/lib/libbac.so on the various systems:
$ cd /usr/jail $ ls -l */usr/local/lib/libbac.so lrwxr-xr-x 1 root wheel 11 Jul 10 00:40 building.example.org/usr/local/lib/libbac.so -> libbac.so.5 -rwxr-xr-x 1 root wheel 383321 Sep 9 2010 dbclone.example.org/usr/local/lib/libbac.so lrwxr-xr-x 1 root wheel 11 Jul 9 22:29 ducky.example.org/usr/local/lib/libbac.so -> libbac.so.5 lrwxr-xr-x 1 root wheel 11 Jul 11 12:24 pg92.example.org/usr/local/lib/libbac.so -> libbac.so.5
OK, that’s pretty clear. The one problem system, dbclone, has a file, not a symlink.
bacula-client changed quite a bit between 5.0.3 and 5.2.12, specifically in the shared libraries.
mat, on IRC, mentioned crypt, and suggested I check the ldd output. Here it is on the faulty system:
$ ldd /usr/local/lib/libbac.so
/usr/local/lib/libbac.so:
libwrap.so.6 => /usr/lib/libwrap.so.6 (0x800c00000)
libz.so.5 => /lib/libz.so.5 (0x800d09000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x800e1e000)
libm.so.5 => /lib/libm.so.5 (0x80102e000)
libc.so.7 => /lib/libc.so.7 (0x800648000)
And on the successful system:
$ ldd /usr/local/lib/libbac.so
/usr/local/lib/libbac.so:
libwrap.so.6 => /usr/lib/libwrap.so.6 (0x800c00000)
libz.so.5 => /lib/libz.so.5 (0x800d09000)
libssl.so.6 => /usr/lib/libssl.so.6 (0x800e1e000)
libcrypto.so.6 => /lib/libcrypto.so.6 (0x800f73000)
libthr.so.3 => /lib/libthr.so.3 (0x801214000)
libintl.so.9 => /usr/local/lib/libintl.so.9 (0x80132d000)
libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x801436000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x801633000)
libm.so.5 => /lib/libm.so.5 (0x801843000)
libc.so.7 => /lib/libc.so.7 (0x800648000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801963000)
Clearly, the successful build has ssl, the failed build does not.
My solution: remove the libbac libraies from the failed system. To figure out what to remove, I looked at an existing system:
$ grep libb /var/db/pkg/bacula-client-5.2.12/+CONTENTS lib/libbac.la lib/libbac.so lib/libbac.so.5 lib/libbaccfg.la lib/libbaccfg.so lib/libbaccfg.so.5 lib/libbacfind.la lib/libbacfind.so lib/libbacfind.so.5 lib/libbacpy.la lib/libbacpy.so lib/libbacpy.so.5
On the failed system, I stopped the jail, took a snapshot (I’m running ZFS), and restarted the jail. This way, I have a fall-back position should I totally mess things up:
$ sudo ezjail-admin stop dbclone.example.org Password: Stopping jails: dbclone.example.org. $ sudo zfs snapshot storage/compressed/jails/dbclone.example.org@DeletingLIBBAC $ sudo ezjail-admin start dbclone.example.org Configuring jails:. Starting jails: dbclone.example.org.
Then I did this on the failed system:
[root@dbclone /home/dan]# cd /usr/local [root@dbclone /usr/local]# rm lib/libbac.la rm: lib/libbac.la: No such file or directory [root@dbclone /usr/local]# rm lib/libbac.so [root@dbclone /usr/local]# rm lib/libbac.so.5 rm: lib/libbac.so.5: No such file or directory [root@dbclone /usr/local]# rm lib/libbaccfg.la rm: lib/libbaccfg.la: No such file or directory [root@dbclone /usr/local]# rm lib/libbaccfg.so [root@dbclone /usr/local]# rm lib/libbaccfg.so.5 rm: lib/libbaccfg.so.5: No such file or directory [root@dbclone /usr/local]# rm lib/libbacfind.la rm: lib/libbacfind.la: No such file or directory [root@dbclone /usr/local]# rm lib/libbacfind.so [root@dbclone /usr/local]# rm lib/libbacfind.so.5 rm: lib/libbacfind.so.5: No such file or directory [root@dbclone /usr/local]# rm lib/libbacpy.la rm: lib/libbacpy.la: No such file or directory [root@dbclone /usr/local]# rm lib/libbacpy.so [root@dbclone /usr/local]# rm lib/libbacpy.so.5 rm: lib/libbacpy.so.5: No such file or directory [root@dbclone /usr/local]#
Then I did a make clean and a make install clean, and all was well again in the Bacula world.
