Detected Disorderly Shutdown last time Directory Server was running, recovering database.
Hello,
we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has been running smooth until today when we realized the DS was down and wouldn't go up.
After checking all the logs, we got this error message:
Detected Disorderly Shutdown last time Directory Server was running, recovering database.
We have even tried to reboot the entire server but the problem persists. Any suggestions on how to fix this will be really appreciated :D
Thanks!
On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
Hello,
we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has been running smooth until today when we realized the DS was down and wouldn't go up.
After checking all the logs, we got this error message:
Detected Disorderly Shutdown last time Directory Server wasrunning, recovering database.
Could you please try this command line? ps -ef | grep ns-slapd
If you see the ns-slapd process, start up (actually, database recovery) is still running. Depending upon the size of the transaction log, it'd take a very long time.
If you don't see the process in the ps output, please start the server from gdb: # gdb /usr/sbin/ns-slapd (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
Once it crashes, get the stacktraces. (gdb) thread apply all bt
And post the stacktraces?
We have even tried to reboot the entire server but the problem persists. Any suggestions on how to fix this will be really appreciated :D
Thanks!
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
Hello,
following your advice I ran the debugger and this is what I got:
(gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [15/Jul/2016:05:03:13 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 starting up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-OCSP: entry cache size 512000B is less than db size 516096B; We recommend to increase the entry cache size nsslapd-cachememsize. [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-CA: entry cache size 512000B is less than db size 1589248B; We recommend to increase the entry cache size nsslapd-cachememsize. [15/Jul/2016:05:03:13 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
Program received signal SIGBUS, Bus error. 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
and
(gdb) thread apply all bt
Thread 1 (Thread 0x7ffff7fd3800 (LWP 6597)): #0 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so #1 0x00007fffeef910ac in __env_detach () from /lib64/libdb-5.3.so #2 0x00007fffeef915da in __env_remove_env () from /lib64/libdb-5.3.so #3 0x00007fffeef8afdd in __env_open () from /lib64/libdb-5.3.so #4 0x00007fffed3014d5 in dblayer_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #5 0x00007fffed356aa1 in ldbm_back_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007ffff7b49283 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #7 0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00005555555653fb in main ()
Thanks!
On 07/14/2016 06:22 PM, Noriko Hosoi wrote:
On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
Hello,
we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has been running smooth until today when we realized the DS was down and wouldn't go up.
After checking all the logs, we got this error message:
Detected Disorderly Shutdown last time Directory Server wasrunning, recovering database.
Could you please try this command line? ps -ef | grep ns-slapd
If you see the ns-slapd process, start up (actually, database recovery) is still running. Depending upon the size of the transaction log, it'd take a very long time.
If you don't see the process in the ps output, please start the server from gdb: # gdb /usr/sbin/ns-slapd (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
Once it crashes, get the stacktraces. (gdb) thread apply all bt
And post the stacktraces?
We have even tried to reboot the entire server but the problem persists. Any suggestions on how to fix this will be really appreciated :D
Thanks!
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
Thank you, Carlos.
Could you repeat running gdb once more after installing debuginfo packages as follows? # yum install 389-ds-base-debuginfo libdb-debuginfo
Note: you may need to enable debuginfo in you /etc/yum.repo.d/*.repo enabled=1
Could you also share this output? # egrep nsslapd-db /etc/dirsrv/slapd-YOUR_SERVER_INSTANCE/dse.ldif
Thanks.
On 07/15/2016 02:22 AM, Carlos Barrabes wrote:
Hello,
following your advice I ran the debugger and this is what I got:
(gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [15/Jul/2016:05:03:13 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 starting up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-OCSP: entry cache size 512000B is less than db size 516096B; We recommend to increase the entry cache size nsslapd-cachememsize. [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-CA: entry cache size 512000B is less than db size 1589248B; We recommend to increase the entry cache size nsslapd-cachememsize. [15/Jul/2016:05:03:13 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
Program received signal SIGBUS, Bus error. 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
and
(gdb) thread apply all bt
Thread 1 (Thread 0x7ffff7fd3800 (LWP 6597)): #0 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so #1 0x00007fffeef910ac in __env_detach () from /lib64/libdb-5.3.so #2 0x00007fffeef915da in __env_remove_env () from /lib64/libdb-5.3.so #3 0x00007fffeef8afdd in __env_open () from /lib64/libdb-5.3.so #4 0x00007fffed3014d5 in dblayer_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #5 0x00007fffed356aa1 in ldbm_back_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007ffff7b49283 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #7 0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00005555555653fb in main ()
Thanks!
On 07/14/2016 06:22 PM, Noriko Hosoi wrote:
On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
Hello,
we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has been running smooth until today when we realized the DS was down and wouldn't go up.
After checking all the logs, we got this error message:
Detected Disorderly Shutdown last time Directory Server wasrunning, recovering database.
Could you please try this command line? ps -ef | grep ns-slapd
If you see the ns-slapd process, start up (actually, database recovery) is still running. Depending upon the size of the transaction log, it'd take a very long time.
If you don't see the process in the ps output, please start the server from gdb: # gdb /usr/sbin/ns-slapd (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
Once it crashes, get the stacktraces. (gdb) thread apply all bt
And post the stacktraces?
We have even tried to reboot the entire server but the problem persists. Any suggestions on how to fix this will be really appreciated :D
Thanks!
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
Hello,
this is the output after installing the packages you asked:
(gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpbe-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpbe-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does not match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does not match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcontentsync-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libwhoami-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libwhoami-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libwhoami-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libwhoami-plugin.so" (CRC mismatch).
[18/Jul/2016:04:55:43 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 starting up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING: pki-matrix-gabon-OCSP: entry cache size 512000B is less than db size 516096B; We recommend to increase the entry cache size nsslapd-cachememsize. [18/Jul/2016:04:55:43 +0100] - WARNING: pki-matrix-gabon-CA: entry cache size 512000B is less than db size 1589248B; We recommend to increase the entry cache size nsslapd-cachememsize. [18/Jul/2016:04:55:43 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
Program received signal SIGBUS, Bus error. __env_size_insert (head=<optimized out>, elp=0x7fffe9e0d1a8) at ../../src/env/env_alloc.c:599 599 if (elp->len >= elp_tmp->len)
and:
(gdb) thread apply all bt
Thread 1 (Thread 0x7ffff7fd3800 (LWP 13818)): #0 __env_size_insert (head=<optimized out>, elp=0x7fffe9e0d1a8) at ../../src/env/env_alloc.c:599 #1 0x00007fffeef910ac in __env_detach (env=env@entry=0x555555b37ad0, destroy=destroy@entry=1) at ../../src/env/env_region.c:823 #2 0x00007fffeef915da in __env_remove_env (env=env@entry=0x555555b37ad0) at ../../src/env/env_region.c:934 #3 0x00007fffeef8afdd in __env_open (dbenv=0x555555b52230, db_home=<optimized out>, flags=9987, mode=<optimized out>) at ../../src/env/env_open.c:201 #4 0x00007fffed3014d5 in dblayer_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #5 0x00007fffed356aa1 in ldbm_back_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007ffff7b49283 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #7 0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00005555555653fb in main ()
and finally:
# egrep nsslapd-db /etc/dirsrv/slapd-pki/dse.ldif nsslapd-dbcachesize: 500000 nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-pki/db nsslapd-db-durable-transaction: on nsslapd-db-checkpoint-interval: 60 nsslapd-db-compactdb-interval: 2592000 nsslapd-db-transaction-batch-val: 0 nsslapd-db-transaction-batch-min-wait: 50 nsslapd-db-transaction-batch-max-wait: 50 nsslapd-db-logbuf-size: 0 nsslapd-db-locks: 10000 nsslapd-db-private-import-mem: on nsslapd-db-deadlock-policy: 9
Thanks! Carlos.
On 07/15/2016 07:30 PM, Noriko Hosoi wrote:
Thank you, Carlos.
Could you repeat running gdb once more after installing debuginfo packages as follows? # yum install 389-ds-base-debuginfo libdb-debuginfo
Note: you may need to enable debuginfo in you /etc/yum.repo.d/*.repo enabled=1
Could you also share this output? # egrep nsslapd-db /etc/dirsrv/slapd-YOUR_SERVER_INSTANCE/dse.ldif
Thanks.
On 07/15/2016 02:22 AM, Carlos Barrabes wrote:
Hello,
following your advice I ran the debugger and this is what I got:
(gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [15/Jul/2016:05:03:13 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 starting up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-OCSP: entry cache size 512000B is less than db size 516096B; We recommend to increase the entry cache size nsslapd-cachememsize. [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-CA: entry cache size 512000B is less than db size 1589248B; We recommend to increase the entry cache size nsslapd-cachememsize. [15/Jul/2016:05:03:13 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
Program received signal SIGBUS, Bus error. 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
and
(gdb) thread apply all bt
Thread 1 (Thread 0x7ffff7fd3800 (LWP 6597)): #0 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so #1 0x00007fffeef910ac in __env_detach () from /lib64/libdb-5.3.so #2 0x00007fffeef915da in __env_remove_env () from /lib64/libdb-5.3.so #3 0x00007fffeef8afdd in __env_open () from /lib64/libdb-5.3.so #4 0x00007fffed3014d5 in dblayer_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #5 0x00007fffed356aa1 in ldbm_back_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007ffff7b49283 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #7 0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00005555555653fb in main ()
Thanks!
On 07/14/2016 06:22 PM, Noriko Hosoi wrote:
On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
Hello,
we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has been running smooth until today when we realized the DS was down and wouldn't go up.
After checking all the logs, we got this error message:
Detected Disorderly Shutdown last time Directory Server wasrunning, recovering database.
Could you please try this command line? ps -ef | grep ns-slapd
If you see the ns-slapd process, start up (actually, database recovery) is still running. Depending upon the size of the transaction log, it'd take a very long time.
If you don't see the process in the ps output, please start the server from gdb: # gdb /usr/sbin/ns-slapd (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
Once it crashes, get the stacktraces. (gdb) thread apply all bt
And post the stacktraces?
We have even tried to reboot the entire server but the problem persists. Any suggestions on how to fix this will be really appreciated :D
Thanks!
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
Hello,
any update on this issue?
Thanks! Carlos
On 07/18/2016 11:15 AM, Carlos Barrabes wrote:
Hello,
this is the output after installing the packages you asked:
(gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpbe-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpbe-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does not match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does not match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcontentsync-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug//usr/lib64/dirsrv/plugins/libwhoami-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libwhoami-plugin.so" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/usr/lib64/dirsrv/plugins/libwhoami-plugin.so.debug" does not match "/usr/lib64/dirsrv/plugins/libwhoami-plugin.so" (CRC mismatch).
[18/Jul/2016:04:55:43 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 starting up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [18/Jul/2016:04:55:43 +0100] - WARNING: pki-matrix-gabon-OCSP: entry cache size 512000B is less than db size 516096B; We recommend to increase the entry cache size nsslapd-cachememsize. [18/Jul/2016:04:55:43 +0100] - WARNING: pki-matrix-gabon-CA: entry cache size 512000B is less than db size 1589248B; We recommend to increase the entry cache size nsslapd-cachememsize. [18/Jul/2016:04:55:43 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
Program received signal SIGBUS, Bus error. __env_size_insert (head=<optimized out>, elp=0x7fffe9e0d1a8) at ../../src/env/env_alloc.c:599 599 if (elp->len >= elp_tmp->len)
and:
(gdb) thread apply all bt
Thread 1 (Thread 0x7ffff7fd3800 (LWP 13818)): #0 __env_size_insert (head=<optimized out>, elp=0x7fffe9e0d1a8) at ../../src/env/env_alloc.c:599 #1 0x00007fffeef910ac in __env_detach (env=env@entry=0x555555b37ad0, destroy=destroy@entry=1) at ../../src/env/env_region.c:823 #2 0x00007fffeef915da in __env_remove_env (env=env@entry=0x555555b37ad0) at ../../src/env/env_region.c:934 #3 0x00007fffeef8afdd in __env_open (dbenv=0x555555b52230, db_home=<optimized out>, flags=9987, mode=<optimized out>) at ../../src/env/env_open.c:201 #4 0x00007fffed3014d5 in dblayer_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #5 0x00007fffed356aa1 in ldbm_back_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007ffff7b49283 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #7 0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00005555555653fb in main ()
and finally:
# egrep nsslapd-db /etc/dirsrv/slapd-pki/dse.ldif nsslapd-dbcachesize: 500000 nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-pki/db nsslapd-db-durable-transaction: on nsslapd-db-checkpoint-interval: 60 nsslapd-db-compactdb-interval: 2592000 nsslapd-db-transaction-batch-val: 0 nsslapd-db-transaction-batch-min-wait: 50 nsslapd-db-transaction-batch-max-wait: 50 nsslapd-db-logbuf-size: 0 nsslapd-db-locks: 10000 nsslapd-db-private-import-mem: on nsslapd-db-deadlock-policy: 9
Thanks! Carlos.
On 07/15/2016 07:30 PM, Noriko Hosoi wrote:
Thank you, Carlos.
Could you repeat running gdb once more after installing debuginfo packages as follows? # yum install 389-ds-base-debuginfo libdb-debuginfo
Note: you may need to enable debuginfo in you /etc/yum.repo.d/*.repo enabled=1
Could you also share this output? # egrep nsslapd-db /etc/dirsrv/slapd-YOUR_SERVER_INSTANCE/dse.ldif
Thanks.
On 07/15/2016 02:22 AM, Carlos Barrabes wrote:
Hello,
following your advice I ran the debugger and this is what I got:
(gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [15/Jul/2016:05:03:13 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 starting up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 -- rounding up [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-OCSP: entry cache size 512000B is less than db size 516096B; We recommend to increase the entry cache size nsslapd-cachememsize. [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-CA: entry cache size 512000B is less than db size 1589248B; We recommend to increase the entry cache size nsslapd-cachememsize. [15/Jul/2016:05:03:13 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
Program received signal SIGBUS, Bus error. 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
and
(gdb) thread apply all bt
Thread 1 (Thread 0x7ffff7fd3800 (LWP 6597)): #0 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so #1 0x00007fffeef910ac in __env_detach () from /lib64/libdb-5.3.so #2 0x00007fffeef915da in __env_remove_env () from /lib64/libdb-5.3.so #3 0x00007fffeef8afdd in __env_open () from /lib64/libdb-5.3.so #4 0x00007fffed3014d5 in dblayer_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #5 0x00007fffed356aa1 in ldbm_back_start () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007ffff7b49283 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #7 0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00005555555653fb in main ()
Thanks!
On 07/14/2016 06:22 PM, Noriko Hosoi wrote:
On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
Hello,
we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has been running smooth until today when we realized the DS was down and wouldn't go up.
After checking all the logs, we got this error message:
Detected Disorderly Shutdown last time Directory Server wasrunning, recovering database.
Could you please try this command line? ps -ef | grep ns-slapd
If you see the ns-slapd process, start up (actually, database recovery) is still running. Depending upon the size of the transaction log, it'd take a very long time.
If you don't see the process in the ps output, please start the server from gdb: # gdb /usr/sbin/ns-slapd (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
Once it crashes, get the stacktraces. (gdb) thread apply all bt
And post the stacktraces?
We have even tried to reboot the entire server but the problem persists. Any suggestions on how to fix this will be really appreciated :D
Thanks!
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
-- 389-users mailing list 389-users@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.or...
389-users@lists.fedoraproject.org
-
Carlos Barrabes -
Noriko Hosoi