Hi guys,
Ubuntu 14.04 x86_64 SSSD 1.11.5 Active Directory backend
I've been experiencing this problem for a few weeks with no luck in troubleshooting it. During boot, the sssd_be process will continually spawn and crash due to a segfault. When it gets stuck in this loop, domain account login is almost impossible (local accounts still work).
I've attached sanitized logs and my sssd.conf file. Please have a look and let me know what you think. Thanks!
-Chris
On 09 Jul 2014, at 17:09, Chris Hartman qrstuv@gmail.com wrote:
Hi guys,
Ubuntu 14.04 x86_64 SSSD 1.11.5 Active Directory backend
Hi Chris,
thanks for reporting this issue. I think we have a bug in the dynamic DNS update code. Here are the last couple of lines before SSSD crashed: (Wed Jul 9 10:10:39 2014) [sssd[be[sanedomain]]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (Wed Jul 9 10:10:39 2014) [sssd[be[sanedomain]]] [be_nsupdate_done] (0x0200): nsupdate child status: 0 (Wed Jul 9 10:10:39 2014) [sssd[be[sanedomain]]] [nsupdate_msg_create_common] (0x0200): Creating update message for realm [SANEDOMAIN.LOCAL]. (Wed Jul 9 10:10:42 2014) [sssd[be[sanedomain]]] [server_setup] (0x0400): CONFDB: /var/lib/sss/db/config.ldb
“server_setup” is the first function that gets executed after SSSD restarts, so the lines right before that indicate where SSSD was before the crash.
Is it possible to generate a core file? That would help debugging quite a lot. I realise the core file can contain private information, so feel free to send it to me directly. If that not acceptable either, generate a backtrace with “bt full” in a gdb session, then you can send a sanitised backtrace.
I think that the following option (in the domain section) should help you work around the problem: dyndns_update = False
Please let us know if disabling the DNS update helped and if you can generate the core file.
Thanks again for the problem report!
I've been experiencing this problem for a few weeks with no luck in troubleshooting it. During boot, the sssd_be process will continually spawn and crash due to a segfault. When it gets stuck in this loop, domain account login is almost impossible (local accounts still work).
I've attached sanitized logs and my sssd.conf file. Please have a look and let me know what you think. Thanks!
-Chris <logs.tar.gz>_______________________________________________ sssd-users mailing list sssd-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users
Hi Jakub,
Is it possible to generate a core file? That would help debugging quite a
lot.
Thanks for your response. I'd be happy to generate a core file for you but I'm afraid I'm going to need some guidance.
My google-fu has gotten me a little familiar with what commands are involved, but I'm not exactly sure how I'd go about generating or retrieving a core dump, especially because the sssd_be process is the one that's crashing, not the main sssd process.
Would you be able provide assistance?
I think that the following option (in the domain section) should help you
work around the problem: dyndns_update = False
This actually didn't work for me, but this did: dyndns_update_ptr = False
So, apparently, the bug is related the ptr update procedure?
Thanks again!
-Chris
On Wed, Jul 9, 2014 at 5:18 PM, Jakub Hrozek jhrozek@redhat.com wrote:
On 09 Jul 2014, at 17:09, Chris Hartman qrstuv@gmail.com wrote:
Hi guys,
Ubuntu 14.04 x86_64 SSSD 1.11.5 Active Directory backend
Hi Chris,
thanks for reporting this issue. I think we have a bug in the dynamic DNS update code. Here are the last couple of lines before SSSD crashed: (Wed Jul 9 10:10:39 2014) [sssd[be[sanedomain]]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (Wed Jul 9 10:10:39 2014) [sssd[be[sanedomain]]] [be_nsupdate_done] (0x0200): nsupdate child status: 0 (Wed Jul 9 10:10:39 2014) [sssd[be[sanedomain]]] [nsupdate_msg_create_common] (0x0200): Creating update message for realm [SANEDOMAIN.LOCAL]. (Wed Jul 9 10:10:42 2014) [sssd[be[sanedomain]]] [server_setup] (0x0400): CONFDB: /var/lib/sss/db/config.ldb
“server_setup” is the first function that gets executed after SSSD restarts, so the lines right before that indicate where SSSD was before the crash.
Is it possible to generate a core file? That would help debugging quite a lot. I realise the core file can contain private information, so feel free to send it to me directly. If that not acceptable either, generate a backtrace with “bt full” in a gdb session, then you can send a sanitised backtrace.
I think that the following option (in the domain section) should help you work around the problem: dyndns_update = False
Please let us know if disabling the DNS update helped and if you can generate the core file.
Thanks again for the problem report!
I've been experiencing this problem for a few weeks with no luck in
troubleshooting it. During boot, the sssd_be process will continually spawn and crash due to a segfault. When it gets stuck in this loop, domain account login is almost impossible (local accounts still work).
I've attached sanitized logs and my sssd.conf file. Please have a look
and let me know what you think. Thanks!
-Chris <logs.tar.gz>_______________________________________________ sssd-users mailing list sssd-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users
On (10/07/14 14:30), Chris Hartman wrote:
Hi Jakub,
Is it possible to generate a core file? That would help debugging quite a
lot.
Thanks for your response. I'd be happy to generate a core file for you but I'm afraid I'm going to need some guidance.
My google-fu has gotten me a little familiar with what commands are involved, but I'm not exactly sure how I'd go about generating or retrieving a core dump, especially because the sssd_be process is the one that's crashing, not the main sssd process.
Would you be able provide assistance?
At first, you will need to install package with debug symbols. Unfortunately, I was not able to find debug symbols for sssd in ubuntu 14.04.
I found a howto on debian wiki, which can help you, https://wiki.debian.org/HowToGetABacktrace#Rebuilding_the_package_you.2BIBk-...
If it is too complicated for you. You can try to file a bug to ubuntu and they should help you with obtaining coredump(backtrace).
LS
On Thu, Jul 10, 2014 at 11:52:18PM +0200, Lukas Slebodnik wrote:
On (10/07/14 14:30), Chris Hartman wrote:
Hi Jakub,
Is it possible to generate a core file? That would help debugging quite a
lot.
Thanks for your response. I'd be happy to generate a core file for you but I'm afraid I'm going to need some guidance.
My google-fu has gotten me a little familiar with what commands are involved, but I'm not exactly sure how I'd go about generating or retrieving a core dump, especially because the sssd_be process is the one that's crashing, not the main sssd process.
Would you be able provide assistance?
At first, you will need to install package with debug symbols. Unfortunately, I was not able to find debug symbols for sssd in ubuntu 14.04.
Maybe Timo (CC) has some tips here?
I found a howto on debian wiki, which can help you, https://wiki.debian.org/HowToGetABacktrace#Rebuilding_the_package_you.2BIBk-...
If it is too complicated for you. You can try to file a bug to ubuntu and they should help you with obtaining coredump(backtrace).
11.07.2014 11:42, Jakub Hrozek kirjoitti:
On Thu, Jul 10, 2014 at 11:52:18PM +0200, Lukas Slebodnik wrote:
On (10/07/14 14:30), Chris Hartman wrote:
Hi Jakub,
Is it possible to generate a core file? That would help debugging quite a
lot.
Thanks for your response. I'd be happy to generate a core file for you but I'm afraid I'm going to need some guidance.
My google-fu has gotten me a little familiar with what commands are involved, but I'm not exactly sure how I'd go about generating or retrieving a core dump, especially because the sssd_be process is the one that's crashing, not the main sssd process.
Would you be able provide assistance?
At first, you will need to install package with debug symbols. Unfortunately, I was not able to find debug symbols for sssd in ubuntu 14.04.
Maybe Timo (CC) has some tips here?
Ubuntu provides ddebs created on package build that you can install, see:
https://wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages
it'll get in Debian too eventually:
https://wiki.debian.org/AutomaticDebugPackages
Sorry it's been so long since my original email. Life happens sometimes. At any rate, I'm able to continue with debugging now.
First off, thanks to Timo and Lukas for their help! I was able to generate the backtrace and core dump file as Jakub suggested. Since it may contain sensitive data, I will email him the files directly.
Jakub, look for an email from my momentarily. Thanks for taking the time to help me out!
-Chris
On Mon, Jul 14, 2014 at 4:07 PM, Timo Aaltonen tjaalton@ubuntu.com wrote:
11.07.2014 11:42, Jakub Hrozek kirjoitti:
On Thu, Jul 10, 2014 at 11:52:18PM +0200, Lukas Slebodnik wrote:
On (10/07/14 14:30), Chris Hartman wrote:
Hi Jakub,
Is it possible to generate a core file? That would help debugging
quite a
lot.
Thanks for your response. I'd be happy to generate a core file for you
but
I'm afraid I'm going to need some guidance.
My google-fu has gotten me a little familiar with what commands are involved, but I'm not exactly sure how I'd go about generating or retrieving a core dump, especially because the sssd_be process is the
one
that's crashing, not the main sssd process.
Would you be able provide assistance?
At first, you will need to install package with debug symbols. Unfortunately, I was not able to find debug symbols for sssd in ubuntu
14.04.
Maybe Timo (CC) has some tips here?
Ubuntu provides ddebs created on package build that you can install, see:
https://wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages
it'll get in Debian too eventually:
https://wiki.debian.org/AutomaticDebugPackages
-- t _______________________________________________ sssd-users mailing list sssd-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users
On (07/08/14 14:57), Chris Hartman wrote:
Sorry it's been so long since my original email. Life happens sometimes. At any rate, I'm able to continue with debugging now.
First off, thanks to Timo and Lukas for their help! I was able to generate the backtrace and core dump file as Jakub suggested. Since it may contain sensitive data, I will email him the files directly.
Jakub, look for an email from my momentarily. Thanks for taking the time to help me out!
[New LWP 32083] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Core was generated by `/usr/lib/i386-linux-gnu/sssd/sssd_be'. #0 0xb775d424 in __kernel_vsyscall () (gdb) bt #0 0xb775d424 in __kernel_vsyscall () #1 0xb7520d7c in __fdatasync_nocancel () at ../sysdeps/unix/syscall-template.S:81 #2 0xb6af2be5 in transaction_sync (tdb=0x8b2a138, offset=53268, length=4) at ../common/transaction.c:548 #3 0xb6af2d9e in _tdb_transaction_cancel (tdb=0x8b2a138) at ../common/transaction.c:603 #4 0xb6af3c95 in tdb_transaction_commit (tdb=0x8b2a138) at ../common/transaction.c:1161 #5 0xb6b172d5 in ltdb_end_trans (module=0x8b2b9b0) at ../ldb_tdb/ldb_tdb.c:1141 #6 0xb770aedf in ldb_transaction_commit (ldb=ldb@entry=0x8b20a78) at ../common/ldb.c:467 #7 0xb770be49 in ldb_autotransaction_request (ldb=0x8b20a78, req=0x8b38500) at ../common/ldb.c:563 #8 0xb770ca43 in ldb_modify (ldb=0x8b20a78, message=message@entry=0x8b49380) at ../common/ldb.c:1641
I am not really sure why but crash is in syscall fdatasync. As you can see, top 9 frames are from libraries libldb and libtdb. If it is a easy to reproduce could you try to mount tmpfs to the directory /var/lib/sss/db/. It can be filesystem issue.
tmpfs /var/lib/sss/db/ tmpfs size=300M,mode=0700,noauto 0 0
LS
On Fri, Aug 08, 2014 at 06:49:56PM +0200, Lukas Slebodnik wrote:
On (07/08/14 14:57), Chris Hartman wrote:
Sorry it's been so long since my original email. Life happens sometimes. At any rate, I'm able to continue with debugging now.
First off, thanks to Timo and Lukas for their help! I was able to generate the backtrace and core dump file as Jakub suggested. Since it may contain sensitive data, I will email him the files directly.
Jakub, look for an email from my momentarily. Thanks for taking the time to help me out!
[New LWP 32083] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Core was generated by `/usr/lib/i386-linux-gnu/sssd/sssd_be'. #0 0xb775d424 in __kernel_vsyscall () (gdb) bt #0 0xb775d424 in __kernel_vsyscall () #1 0xb7520d7c in __fdatasync_nocancel () at ../sysdeps/unix/syscall-template.S:81 #2 0xb6af2be5 in transaction_sync (tdb=0x8b2a138, offset=53268, length=4) at ../common/transaction.c:548 #3 0xb6af2d9e in _tdb_transaction_cancel (tdb=0x8b2a138) at ../common/transaction.c:603 #4 0xb6af3c95 in tdb_transaction_commit (tdb=0x8b2a138) at ../common/transaction.c:1161 #5 0xb6b172d5 in ltdb_end_trans (module=0x8b2b9b0) at ../ldb_tdb/ldb_tdb.c:1141 #6 0xb770aedf in ldb_transaction_commit (ldb=ldb@entry=0x8b20a78) at ../common/ldb.c:467 #7 0xb770be49 in ldb_autotransaction_request (ldb=0x8b20a78, req=0x8b38500) at ../common/ldb.c:563 #8 0xb770ca43 in ldb_modify (ldb=0x8b20a78, message=message@entry=0x8b49380) at ../common/ldb.c:1641
I am not really sure why but crash is in syscall fdatasync. As you can see, top 9 frames are from libraries libldb and libtdb. If it is a easy to reproduce could you try to mount tmpfs to the directory /var/lib/sss/db/. It can be filesystem issue.
tmpfs /var/lib/sss/db/ tmpfs size=300M,mode=0700,noauto 0 0
Thanks for looking into this!
Did you notice anything strange about the ldb_message in frame 8? That's the only thing I can think of that we might influence.
Do you know if Ubuntu uses the latest tdb and ldb versions?
Hi guys,
If it is a easy to reproduce could you try to mount tmpfs to the directory
/var/lib/sss/db/. It can be filesystem issue.
Mounting with a tmpfs has no effect on the crash.
Do you know if Ubuntu uses the latest tdb and ldb versions?
libtdb1 version in Ubuntu 14.04: 1.2.12-1 libldb1 version in Ubuntu 14.04: 1.1.16-1
According to http://www.samba.org/ftp/tdb/ the latest tbd version is 1.3.0 and http://www.samba.org/ftp/ldb/ says the latest ldb version is 1.1.17
For full disclosure, I already told this to Jakub in case it matters:
I ran into a little trouble because the sssd_be was the binary that was
actually crashing, but it was spawned automatically by the root sssd process. In order to generate the coredump and backtrace, I came up with this inelegant solution:
service sssd start; sleep 3; gdb -p $(ps ax|grep sssd_be|grep -v grep|cut -f1 -d" ") -ex generate-core-file
Hope this helps.
-Chris
On Fri, Aug 8, 2014 at 1:16 PM, Jakub Hrozek jhrozek@redhat.com wrote:
On Fri, Aug 08, 2014 at 06:49:56PM +0200, Lukas Slebodnik wrote:
On (07/08/14 14:57), Chris Hartman wrote:
Sorry it's been so long since my original email. Life happens
sometimes. At
any rate, I'm able to continue with debugging now.
First off, thanks to Timo and Lukas for their help! I was able to
generate
the backtrace and core dump file as Jakub suggested. Since it may
contain
sensitive data, I will email him the files directly.
Jakub, look for an email from my momentarily. Thanks for taking the
time to
help me out!
[New LWP 32083] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Core was generated by `/usr/lib/i386-linux-gnu/sssd/sssd_be'. #0 0xb775d424 in __kernel_vsyscall () (gdb) bt #0 0xb775d424 in __kernel_vsyscall () #1 0xb7520d7c in __fdatasync_nocancel () at
../sysdeps/unix/syscall-template.S:81
#2 0xb6af2be5 in transaction_sync (tdb=0x8b2a138, offset=53268,
length=4) at ../common/transaction.c:548
#3 0xb6af2d9e in _tdb_transaction_cancel (tdb=0x8b2a138) at
../common/transaction.c:603
#4 0xb6af3c95 in tdb_transaction_commit (tdb=0x8b2a138) at
../common/transaction.c:1161
#5 0xb6b172d5 in ltdb_end_trans (module=0x8b2b9b0) at
../ldb_tdb/ldb_tdb.c:1141
#6 0xb770aedf in ldb_transaction_commit (ldb=ldb@entry=0x8b20a78) at
../common/ldb.c:467
#7 0xb770be49 in ldb_autotransaction_request (ldb=0x8b20a78,
req=0x8b38500) at ../common/ldb.c:563
#8 0xb770ca43 in ldb_modify (ldb=0x8b20a78, message=message@entry=0x8b49380)
at ../common/ldb.c:1641
I am not really sure why but crash is in syscall fdatasync. As you can see, top 9 frames are from libraries libldb and libtdb. If it is a easy to reproduce could you try to mount tmpfs to the
directory
/var/lib/sss/db/. It can be filesystem issue.
tmpfs /var/lib/sss/db/ tmpfs
size=300M,mode=0700,noauto 0 0
Thanks for looking into this!
Did you notice anything strange about the ldb_message in frame 8? That's the only thing I can think of that we might influence.
Do you know if Ubuntu uses the latest tdb and ldb versions? _______________________________________________ sssd-users mailing list sssd-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users
On (08/08/14 14:48), Chris Hartman wrote:
Hi guys,
If it is a easy to reproduce could you try to mount tmpfs to the directory
/var/lib/sss/db/. It can be filesystem issue.
Mounting with a tmpfs has no effect on the crash.
Do you know if Ubuntu uses the latest tdb and ldb versions?
libtdb1 version in Ubuntu 14.04: 1.2.12-1 libldb1 version in Ubuntu 14.04: 1.1.16-1
Fedora has almost the same versions. [user@host ~]$ rpm -q libtdb libldb libtdb-1.2.12-2.fc20.x86_64 libldb-1.1.16-4.fc20.x86_64
According to http://www.samba.org/ftp/tdb/ the latest tbd version is 1.3.0 and http://www.samba.org/ftp/ldb/ says the latest ldb version is 1.1.17
For full disclosure, I already told this to Jakub in case it matters:
I ran into a little trouble because the sssd_be was the binary that was
actually crashing, but it was spawned automatically by the root sssd process. In order to generate the coredump and backtrace, I came up with this inelegant solution:
service sssd start; sleep 3; gdb -p $(ps ax|grep sssd_be|grep -v grep|cut -f1 -d" ") -ex generate-core-file
That's good way how to obtain coredump. I can be simplified if you have just one domain in sssd.conf
service sssd start; sleep 3; gdb -p $(pgrep sssd_be) -ex generate-core-file
Hope this helps.
Could you send me more coredumps privately with fuly updated ubuntu 14.04? Because it really strange that sssd crashed in syscal fdatasync.
LS
On (08/08/14 14:48), Chris Hartman wrote:
Hi guys,
If it is a easy to reproduce could you try to mount tmpfs to the directory
/var/lib/sss/db/. It can be filesystem issue.
Mounting with a tmpfs has no effect on the crash.
Do you know if Ubuntu uses the latest tdb and ldb versions?
libtdb1 version in Ubuntu 14.04: 1.2.12-1 libldb1 version in Ubuntu 14.04: 1.1.16-1
According to http://www.samba.org/ftp/tdb/ the latest tbd version is 1.3.0 and http://www.samba.org/ftp/ldb/ says the latest ldb version is 1.1.17
For full disclosure, I already told this to Jakub in case it matters:
I ran into a little trouble because the sssd_be was the binary that was
actually crashing, but it was spawned automatically by the root sssd process. In order to generate the coredump and backtrace, I came up with this inelegant solution:
service sssd start; sleep 3; gdb -p $(ps ax|grep sssd_be|grep -v grep|cut -f1 -d" ") -ex generate-core-file
Hope this helps.
I have another idea which could help us to to find a problem. We can try to run sssd_be with valgrind.
1) we need to find out arguments for process sssd_be sh-4.2$ pgrep -af sssd_be 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this output will be used in sssd.conf
2) add new option to domain section in sssd.conf command = valgrind -v --show-reachable=yes --log-file=/var/log/sssd/valgrind_idm.example.com.log /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
# do not forget to replace output from previous step.
3) It would be good to install debug symbols for libraries libtdb1 and libldb1
4) reproduce problem some interesting output can be in log file from valgrind.
LS
Greetings,
Could you send me more coredumps privately with fuly updated ubuntu 14.04?
Because it really strange that sssd crashed in syscal fdatasync.
Lukas, not a problem. Stand by for an email with a 7zip archive of a core dump and back trace. The debugging symbols for lib{tdb,ldb}1 were installed for this.
I have another idea which could help us to to find a problem.
We can try to run sssd_be with valgrind.
- we need to find out arguments for process sssd_be
sh-4.2$ pgrep -af sssd_be 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this output will be used in sssd.conf 2) add new option to domain section in sssd.conf command = valgrind -v --show-reachable=yes --log-file=/var/log/sssd/valgrind_idm.example.com.log /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files # do not forget to replace output from previous step.
- It would be good to install debug symbols for libraries libtdb1 and
libldb1 4) reproduce problem some interesting output can be in log file from valgrind.
I did this with interesting(?) results. The sssd_be process never actually crashed but there is some erroneous output in the valgrind log. Removing the special command option results in a crash as expected. I've sanitized and attached the log to this email. Let me know if that reveals anything.
-Chris
On Tue, Aug 12, 2014 at 9:29 AM, Lukas Slebodnik lslebodn@redhat.com wrote:
On (08/08/14 14:48), Chris Hartman wrote:
Hi guys,
If it is a easy to reproduce could you try to mount tmpfs to the directory
/var/lib/sss/db/. It can be filesystem issue.
Mounting with a tmpfs has no effect on the crash.
Do you know if Ubuntu uses the latest tdb and ldb versions?
libtdb1 version in Ubuntu 14.04: 1.2.12-1 libldb1 version in Ubuntu 14.04: 1.1.16-1
According to http://www.samba.org/ftp/tdb/ the latest tbd version is
1.3.0
and http://www.samba.org/ftp/ldb/ says the latest ldb version is 1.1.17
For full disclosure, I already told this to Jakub in case it matters:
I ran into a little trouble because the sssd_be was the binary that was
actually crashing, but it was spawned automatically by the root sssd process. In order to generate the coredump and backtrace, I came up with this inelegant solution:
service sssd start; sleep 3; gdb -p $(ps ax|grep sssd_be|grep -v
grep|cut
-f1 -d" ") -ex generate-core-file
Hope this helps.
I have another idea which could help us to to find a problem. We can try to run sssd_be with valgrind.
- we need to find out arguments for process sssd_be
sh-4.2$ pgrep -af sssd_be 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this output will be used in sssd.conf
- add new option to domain section in sssd.conf
command = valgrind -v --show-reachable=yes --log-file=/var/log/sssd/valgrind_idm.example.com.log /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
# do not forget to replace output from previous step.
- It would be good to install debug symbols for libraries libtdb1 and
libldb1
- reproduce problem some interesting output can be in log file from valgrind.
LS _______________________________________________ sssd-users mailing list sssd-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users
On (13/08/14 11:12), Chris Hartman wrote:
Greetings,
Could you send me more coredumps privately with fuly updated ubuntu 14.04?
Because it really strange that sssd crashed in syscal fdatasync.
Lukas, not a problem. Stand by for an email with a 7zip archive of a core dump and back trace. The debugging symbols for lib{tdb,ldb}1 were installed for this.
I have another idea which could help us to to find a problem.
We can try to run sssd_be with valgrind.
- we need to find out arguments for process sssd_be
sh-4.2$ pgrep -af sssd_be 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this output will be used in sssd.conf 2) add new option to domain section in sssd.conf command = valgrind -v --show-reachable=yes --log-file=/var/log/sssd/valgrind_idm.example.com.log /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files # do not forget to replace output from previous step.
- It would be good to install debug symbols for libraries libtdb1 and
libldb1 4) reproduce problem some interesting output can be in log file from valgrind.
I did this with interesting(?) results. The sssd_be process never actually crashed but there is some erroneous output in the valgrind log. Removing the special command option results in a crash as expected. I've sanitized and attached the log to this email. Let me know if that reveals anything.
Thank you very much for log files from valgrind. There is use after free problem in dynamic dns updates code.
Could you try to disable dynamic dns updates in domain section. dyndns_update = false
LS
Lukas,
Could you try to disable dynamic dns updates in domain section.
dyndns_update = false
I can do that. Would you like valgrind logs? Another coredump? And just to review, the crash happens only when 'dyndns_update_ptr' is true; the crash doesn't seem to be affected by the 'dyndns_update' option.
-Chris
On Wed, Aug 13, 2014 at 1:08 PM, Lukas Slebodnik lslebodn@redhat.com wrote:
On (13/08/14 11:12), Chris Hartman wrote:
Greetings,
Could you send me more coredumps privately with fuly updated ubuntu 14.04?
Because it really strange that sssd crashed in syscal fdatasync.
Lukas, not a problem. Stand by for an email with a 7zip archive of a core dump and back trace. The debugging symbols for lib{tdb,ldb}1 were
installed
for this.
I have another idea which could help us to to find a problem.
We can try to run sssd_be with valgrind.
- we need to find out arguments for process sssd_be
sh-4.2$ pgrep -af sssd_be 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com
--debug-to-files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this output will be used in sssd.conf
- add new option to domain section in sssd.conf
command = valgrind -v --show-reachable=yes
--log-file=/var/log/sssd/valgrind_idm.example.com.log
/usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files # do not forget to replace output from previous step.
- It would be good to install debug symbols for libraries libtdb1 and
libldb1 4) reproduce problem some interesting output can be in log file from valgrind.
I did this with interesting(?) results. The sssd_be process never
actually
crashed but there is some erroneous output in the valgrind log. Removing the special command option results in a crash as expected. I've sanitized and attached the log to this email. Let me know if that reveals anything.
Thank you very much for log files from valgrind. There is use after free problem in dynamic dns updates code.
Could you try to disable dynamic dns updates in domain section. dyndns_update = false
LS _______________________________________________ sssd-users mailing list sssd-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users
On Wed, Aug 13, 2014 at 07:08:33PM +0200, Lukas Slebodnik wrote:
On (13/08/14 11:12), Chris Hartman wrote:
Greetings,
Could you send me more coredumps privately with fuly updated ubuntu 14.04?
Because it really strange that sssd crashed in syscal fdatasync.
Lukas, not a problem. Stand by for an email with a 7zip archive of a core dump and back trace. The debugging symbols for lib{tdb,ldb}1 were installed for this.
I have another idea which could help us to to find a problem.
We can try to run sssd_be with valgrind.
- we need to find out arguments for process sssd_be
sh-4.2$ pgrep -af sssd_be 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this output will be used in sssd.conf 2) add new option to domain section in sssd.conf command = valgrind -v --show-reachable=yes --log-file=/var/log/sssd/valgrind_idm.example.com.log /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files # do not forget to replace output from previous step.
- It would be good to install debug symbols for libraries libtdb1 and
libldb1 4) reproduce problem some interesting output can be in log file from valgrind.
I did this with interesting(?) results. The sssd_be process never actually crashed but there is some erroneous output in the valgrind log. Removing the special command option results in a crash as expected. I've sanitized and attached the log to this email. Let me know if that reveals anything.
Thank you very much for log files from valgrind. There is use after free problem in dynamic dns updates code.
Could you try to disable dynamic dns updates in domain section. dyndns_update = false
LS
Thank you very much for debugging, Chris!
I filed https://fedorahosted.org/sssd/ticket/2405 to track this problem.
On (13/08/14 13:53), Chris Hartman wrote:
Lukas,
Could you try to disable dynamic dns updates in domain section.
dyndns_update = false
I can do that. Would you like valgrind logs? Another coredump? And just to review, the crash happens only when 'dyndns_update_ptr' is true; the crash doesn't seem to be affected by the 'dyndns_update' option.
Thank you very much for your time and all log files.
Finally, I am able to reproduce bug locally.
LS
On (09/07/14 11:09), Chris Hartman wrote:
Hi guys,
Ubuntu 14.04 x86_64 SSSD 1.11.5 Active Directory backend
I've been experiencing this problem for a few weeks with no luck in troubleshooting it. During boot, the sssd_be process will continually spawn and crash due to a segfault. When it gets stuck in this loop, domain account login is almost impossible (local accounts still work).
I've attached sanitized logs and my sssd.conf file. Please have a look and let me know what you think. Thanks!
Patch was accepted in the sssd upstream https://git.fedorahosted.org/cgit/sssd.git/commit/?id=0060992d68ba843d4d90b4...
Thank you very much for report and help with troubleshooting this bug.
LS
sssd-users@lists.fedorahosted.org