Hey there,
We need to see the pstacks from all threads to really determine the cause here. Can you send us a complete read out?
gdb -p "pid"
thread apply all bt
It'd be great if you can install debug info too to help.
Thanks,
On 31 May 2020, at 05:09, Crocker, Deborah crock@ua.edu wrote:
Some more information from a coworker:
Yeah, this sounds like an LDAP server bug. I haven't figured out what to look at to pin it down, but when it's slow to connect, I can see with strace that the primary thread hasn't called accept() yet for the connection I'm trying to open. Once it does, the whole thing goes very quickly, and I usually see a burst of other connections accepted and handled at the same time.
Deborah Crocker, PhD Systems Engineer III Office of Information Technology The University of Alabama Box 870346 Tuscaloosa, AL 36587 Office 205-348-3758 | Fax 205-348-9393 deborah.crocker@ua.edu
-----Original Message----- From: Crocker, Deborah crock@ua.edu Sent: Saturday, May 30, 2020 2:08 PM To: General discussion list for the 389 Directory server project. 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] new server setup hanging
We're trying to move into our new server setup. We have one that seems to be fine under a load but when we bring the next we're having trouble with it hanging. The second does have more clients (and different) so there could be something about what a client is doing. Here is the server: 389-Directory/1.3.10.1 B2020.133.1625 Installed from EPEL, running on CentOS Linux release 7.8.2003
And here is the pstack output listing the only thread that is not idle. Can anyone tell me what is going on?
Thread 44 (Thread 0x7f858e9b3700 (LWP 2515)): #0 0x00007f860a90fe02 in slapi_atomic_load_32 () at /usr/lib64/dirsrv/libslapd.so.0 #1 0x00007f860a8d4e8e in slapi_get_mapping_tree_node_by_dn () at /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f860a8d5179 in slapi_be_select () at /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f860a9296a0 in vattr_test_filter () at /usr/lib64/dirsrv/libslapd.so.0 #4 0x00007f860a8b6ec4 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007f8600a99e02 in acl__resource_match_aci () at /usr/lib64/dirsrv/plugins/libacl-plugin.so #7 0x00007f8600a9b280 in acl_access_allowed () at /usr/lib64/dirsrv/plugins/libacl-plugin.so #8 0x00007f8600aae9f7 in acl_access_allowed_main () at /usr/lib64/dirsrv/plugins/libacl-plugin.so #9 0x00007f860a8f0cbc in plugin_call_acl_plugin () at /usr/lib64/dirsrv/libslapd.so.0 #10 0x00007f860a8b638d in test_filter_access () at /usr/lib64/dirsrv/libslapd.so.0 #11 0x00007f860a8b6fb5 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 #12 0x00007f860a8b6d31 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 #13 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0 #14 0x00007f85ff7c0df1 in ldbm_back_next_search_entry_ext () at /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f860a8deca6 in send_results_ext.constprop.5 () at /usr/lib64/dirsrv/libslapd.so.0 #16 0x00007f860a8e0e09 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0 #17 0x0000557410dd3c0e in do_search () #18 0x0000557410dc198a in connection_threadmain () #19 0x00007f86086a0c5b in _pt_root () at /lib64/libnspr4.so #20 0x00007f8608040ea5 in start_thread () at /lib64/libpthread.so.0 #21 0x00007f86076ec8dd in clone () at /lib64/libc.so.6
Deborah Crocker, PhD Systems Engineer III Office of Information Technology The University of Alabama Box 870346 Tuscaloosa, AL 36587 Office 205-348-3758 | Fax 205-348-9393 deborah.crocker@ua.edu
-----Original Message----- From: William Brown wbrown@suse.de Sent: Wednesday, May 27, 2020 5:43 PM To: 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] Re: Re: Re: Advice to bring new servers into production
On 27 May 2020, at 23:20, Crocker, Deborah crock@ua.edu wrote:
Thanks - I think we have enough ideas in here to get this going. One last question: If replication is set up through the host name - how often does the directory server do a DNS look up, or does it do it once on startup (or creation of the rep agreement)?
I "think" it's every time it initiates the new connection - but remember, for replication, that *is* quite different to a client doing a search, so I'd be pretty careful about this. IMO you should be standing up your replacement servers in parallel, joining them all, moving the IP's then decomission the old servers. Alternately, you'll need an outage window to shutdown your old servers, export the ldif, and then import and bring up the new ones.
I think having "IP's are a limited resource" really does make this whole process much much harder than it needs to be for you ... :(
-----Original Message----- From: William Brown wbrown@suse.de Sent: Tuesday, May 26, 2020 10:48 PM To: 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] Re: Re: Advice to bring new servers into production
There are a few options. The best would be a load balancer which has the ip's so that it's transparent to your LDAP servers where they are.
But also as mentioned, the virtual IP's honestly is the best way. Linux can have multiple IP's on an interface so you can just have two IP's on one interface, andthat's the best way to do this.
Alternately, don't rely on the IP, lower your DNS ttl's to a very short time, change the DNS A/AAAA records, and then do it that way.
On 27 May 2020, at 06:17, Crocker, Deborah crock@ua.edu wrote:
I’d like not to take up two ip addresses per host indefinitely. We have re-IP’d our hosts before so I know we can to do this but it was during a downtime when everything was restarted. Just trying to get away with not restarting the masters.
Deborah Crocker, PhD Systems Engineer III Office of Information Technology The University of Alabama Box 870346 Tuscaloosa, AL 36587 Office 205-348-3758 | Fax 205-348-9393 deborah.crocker@ua.edu
From: Leo Pleiman lpleiman@salsalabs.com Sent: Tuesday, May 26, 2020 3:08 PM To: General discussion list for the 389 Directory server project. 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] Re: Advice to bring new servers into production
My experience has been that the replicas and consumers have a unique id, more than just an IP address which creates the trust relationship with the master. If your goal is to simply maintain an IP so your clients don't have to be repointed, I would build each new LDAP host and replication agreement, and then as you decommission the old hosts use their IP address as a virtual IP address on the replacement host. It would take a quick restart od the LDAP service to start a listener on the virtual Ip address.
Leo Pleiman Senior System Engineer Direct 202-787-3622 Cell 410-688-3873
On Tue, May 26, 2020 at 3:57 PM Crocker, Deborah crock@ua.edu wrote: We have a setup with 2 multi-masters and 3 consumers. We are now building new host and want to put them in place ultimately at the same IP address as the original ones. I need some advice on how to do this quickly and cleanly.
To add a new consumer the idea now is to set it up and set up replications agreements from each master using consumer DNS name (don't start continuous replication yet). After initializing new consumer from one master - turn off old consumer, remove old consumer agreement from each master, and re-IP new consumer. Do we need to restart masters to re-read DNS or will it pick that up when it starts the next replication? Is this the best way to do this?
Thanks
Deborah Crocker, PhD Systems Engineer III Office of Information Technology The University of Alabama Box 870346 Tuscaloosa, AL 36587 Office 205-348-3758 | Fax 205-348-9393 deborah.crocker@ua.edu
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap r oject.org _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap r oject.org
— Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr oject.org _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr oject.org
— Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....
— Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server SUSE Labs
We had to roll it back. There is one host running with it but the load is so light we never saw a problem. We think it was a known bug, maybe this:
https://pagure.io/389-ds-base/issue/50329
Do you want any info off the running host?
We'll now probably move into the 1.4.x trees. Any advice on which is the most stable?
-----Original Message----- From: William Brown wbrown@suse.de Sent: Sunday, May 31, 2020 6:25 PM To: 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] Re: new server setup hanging
Hey there,
We need to see the pstacks from all threads to really determine the cause here. Can you send us a complete read out?
gdb -p "pid"
thread apply all bt
It'd be great if you can install debug info too to help.
Thanks,
On 31 May 2020, at 05:09, Crocker, Deborah crock@ua.edu wrote:
Some more information from a coworker:
Yeah, this sounds like an LDAP server bug. I haven't figured out what to look at to pin it down, but when it's slow to connect, I can see with strace that the primary thread hasn't called accept() yet for the connection I'm trying to open. Once it does, the whole thing goes very quickly, and I usually see a burst of other connections accepted and handled at the same time.
Deborah Crocker, PhD Systems Engineer III Office of Information Technology The University of Alabama Box 870346 Tuscaloosa, AL 36587 Office 205-348-3758 | Fax 205-348-9393 deborah.crocker@ua.edu
-----Original Message----- From: Crocker, Deborah crock@ua.edu Sent: Saturday, May 30, 2020 2:08 PM To: General discussion list for the 389 Directory server project. 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] new server setup hanging
We're trying to move into our new server setup. We have one that seems to be fine under a load but when we bring the next we're having trouble with it hanging. The second does have more clients (and different) so there could be something about what a client is doing. Here is the server: 389-Directory/1.3.10.1 B2020.133.1625 Installed from EPEL, running on CentOS Linux release 7.8.2003
And here is the pstack output listing the only thread that is not idle. Can anyone tell me what is going on?
Thread 44 (Thread 0x7f858e9b3700 (LWP 2515)): #0 0x00007f860a90fe02 in slapi_atomic_load_32 () at /usr/lib64/dirsrv/libslapd.so.0 #1 0x00007f860a8d4e8e in slapi_get_mapping_tree_node_by_dn () at /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f860a8d5179 in slapi_be_select () at /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f860a9296a0 in vattr_test_filter () at /usr/lib64/dirsrv/libslapd.so.0 #4 0x00007f860a8b6ec4 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007f8600a99e02 in acl__resource_match_aci () at /usr/lib64/dirsrv/plugins/libacl-plugin.so #7 0x00007f8600a9b280 in acl_access_allowed () at /usr/lib64/dirsrv/plugins/libacl-plugin.so #8 0x00007f8600aae9f7 in acl_access_allowed_main () at /usr/lib64/dirsrv/plugins/libacl-plugin.so #9 0x00007f860a8f0cbc in plugin_call_acl_plugin () at /usr/lib64/dirsrv/libslapd.so.0 #10 0x00007f860a8b638d in test_filter_access () at /usr/lib64/dirsrv/libslapd.so.0 #11 0x00007f860a8b6fb5 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 #12 0x00007f860a8b6d31 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 #13 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0 #14 0x00007f85ff7c0df1 in ldbm_back_next_search_entry_ext () at /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f860a8deca6 in send_results_ext.constprop.5 () at /usr/lib64/dirsrv/libslapd.so.0 #16 0x00007f860a8e0e09 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0 #17 0x0000557410dd3c0e in do_search () #18 0x0000557410dc198a in connection_threadmain () #19 0x00007f86086a0c5b in _pt_root () at /lib64/libnspr4.so #20 0x00007f8608040ea5 in start_thread () at /lib64/libpthread.so.0 #21 0x00007f86076ec8dd in clone () at /lib64/libc.so.6
Deborah Crocker, PhD Systems Engineer III Office of Information Technology The University of Alabama Box 870346 Tuscaloosa, AL 36587 Office 205-348-3758 | Fax 205-348-9393 deborah.crocker@ua.edu
-----Original Message----- From: William Brown wbrown@suse.de Sent: Wednesday, May 27, 2020 5:43 PM To: 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] Re: Re: Re: Advice to bring new servers into production
On 27 May 2020, at 23:20, Crocker, Deborah crock@ua.edu wrote:
Thanks - I think we have enough ideas in here to get this going. One last question: If replication is set up through the host name - how often does the directory server do a DNS look up, or does it do it once on startup (or creation of the rep agreement)?
I "think" it's every time it initiates the new connection - but remember, for replication, that *is* quite different to a client doing a search, so I'd be pretty careful about this. IMO you should be standing up your replacement servers in parallel, joining them all, moving the IP's then decomission the old servers. Alternately, you'll need an outage window to shutdown your old servers, export the ldif, and then import and bring up the new ones.
I think having "IP's are a limited resource" really does make this whole process much much harder than it needs to be for you ... :(
-----Original Message----- From: William Brown wbrown@suse.de Sent: Tuesday, May 26, 2020 10:48 PM To: 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] Re: Re: Advice to bring new servers into production
There are a few options. The best would be a load balancer which has the ip's so that it's transparent to your LDAP servers where they are.
But also as mentioned, the virtual IP's honestly is the best way. Linux can have multiple IP's on an interface so you can just have two IP's on one interface, andthat's the best way to do this.
Alternately, don't rely on the IP, lower your DNS ttl's to a very short time, change the DNS A/AAAA records, and then do it that way.
On 27 May 2020, at 06:17, Crocker, Deborah crock@ua.edu wrote:
I’d like not to take up two ip addresses per host indefinitely. We have re-IP’d our hosts before so I know we can to do this but it was during a downtime when everything was restarted. Just trying to get away with not restarting the masters.
Deborah Crocker, PhD Systems Engineer III Office of Information Technology The University of Alabama Box 870346 Tuscaloosa, AL 36587 Office 205-348-3758 | Fax 205-348-9393 deborah.crocker@ua.edu
From: Leo Pleiman lpleiman@salsalabs.com Sent: Tuesday, May 26, 2020 3:08 PM To: General discussion list for the 389 Directory server project. 389-users@lists.fedoraproject.org Subject: [EXTERNAL] [389-users] Re: Advice to bring new servers into production
My experience has been that the replicas and consumers have a unique id, more than just an IP address which creates the trust relationship with the master. If your goal is to simply maintain an IP so your clients don't have to be repointed, I would build each new LDAP host and replication agreement, and then as you decommission the old hosts use their IP address as a virtual IP address on the replacement host. It would take a quick restart od the LDAP service to start a listener on the virtual Ip address.
Leo Pleiman Senior System Engineer Direct 202-787-3622 Cell 410-688-3873
On Tue, May 26, 2020 at 3:57 PM Crocker, Deborah crock@ua.edu wrote: We have a setup with 2 multi-masters and 3 consumers. We are now building new host and want to put them in place ultimately at the same IP address as the original ones. I need some advice on how to do this quickly and cleanly.
To add a new consumer the idea now is to set it up and set up replications agreements from each master using consumer DNS name (don't start continuous replication yet). After initializing new consumer from one master - turn off old consumer, remove old consumer agreement from each master, and re-IP new consumer. Do we need to restart masters to re-read DNS or will it pick that up when it starts the next replication? Is this the best way to do this?
Thanks
Deborah Crocker, PhD Systems Engineer III Office of Information Technology The University of Alabama Box 870346 Tuscaloosa, AL 36587 Office 205-348-3758 | Fax 205-348-9393 deborah.crocker@ua.edu
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedora p r oject.org _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedora p r oject.org
— Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap r oject.org _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap r oject.org
— Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr oject.org _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr oject.org _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr oject.org
— Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....
389-users@lists.fedoraproject.org
-
Crocker, Deborah -
William Brown