I’m in the process of installing 389 in CentOS 7 from epel (versions below) and find that the console becomes unresponsive after I install the 4th server. I can open the console and expand a few servers but within 30 seconds it consistently hangs. I am storing configuration data in one server.
Has anyone else seen this or do you have any insight?
Thanks,
-morgan
[morgan@devldap03 ~]$ rpm -qa|grep 389 pcp-pmda-ds389log-3.11.3-4.el7.x86_64 389-admin-console-1.1.12-1.el7.noarch 389-dsgw-1.1.11-5.el7.x86_64 389-adminutil-1.1.21-2.el7.x86_64 389-admin-1.1.46-1.el7.x86_64 389-ds-console-doc-1.2.16-1.el7.noarch 389-ds-1.2.2-6.el7.noarch 389-ds-base-libs-1.3.5.10-21.el7_3.x86_64 389-ds-base-1.3.5.10-21.el7_3.x86_64 389-admin-console-doc-1.1.12-1.el7.noarch 389-console-1.1.18-1.el7.noarch pcp-pmda-ds389-3.11.3-4.el7.x86_64 389-ds-console-1.2.16-1.el7.noarch [morgan@devldap03 ~]$
Hi Morgan,
We need more info. Try running the console in debug mode:
389-console -D 9
Also look at the configuration DS access log
Mark
On 08/16/2017 02:57 PM, Morgan Jones wrote:
I’m in the process of installing 389 in CentOS 7 from epel (versions below) and find that the console becomes unresponsive after I install the 4th server. I can open the console and expand a few servers but within 30 seconds it consistently hangs. I am storing configuration data in one server.
Has anyone else seen this or do you have any insight?
Thanks,
-morgan
[morgan@devldap03 ~]$ rpm -qa|grep 389 pcp-pmda-ds389log-3.11.3-4.el7.x86_64 389-admin-console-1.1.12-1.el7.noarch 389-dsgw-1.1.11-5.el7.x86_64 389-adminutil-1.1.21-2.el7.x86_64 389-admin-1.1.46-1.el7.x86_64 389-ds-console-doc-1.2.16-1.el7.noarch 389-ds-1.2.2-6.el7.noarch 389-ds-base-libs-1.3.5.10-21.el7_3.x86_64 389-ds-base-1.3.5.10-21.el7_3.x86_64 389-admin-console-doc-1.1.12-1.el7.noarch 389-console-1.1.18-1.el7.noarch pcp-pmda-ds389-3.11.3-4.el7.x86_64 389-ds-console-1.2.16-1.el7.noarch [morgan@devldap03 ~]$ _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
Hello Mark,
See attached, "AbstractServerObject.StatusThread: waiting for change listeners to register” repeats presumably forever after it hangs.
Thanks,
-morgan
On Aug 16, 2017, at 3:23 PM, Mark Reynolds mareynol@redhat.com wrote:
Hi Morgan,
We need more info. Try running the console in debug mode:
389-console -D 9
Also look at the configuration DS access log
Mark
On 08/16/2017 02:57 PM, Morgan Jones wrote:
I’m in the process of installing 389 in CentOS 7 from epel (versions below) and find that the console becomes unresponsive after I install the 4th server. I can open the console and expand a few servers but within 30 seconds it consistently hangs. I am storing configuration data in one server.
Has anyone else seen this or do you have any insight?
Thanks,
-morgan
[morgan@devldap03 ~]$ rpm -qa|grep 389 pcp-pmda-ds389log-3.11.3-4.el7.x86_64 389-admin-console-1.1.12-1.el7.noarch 389-dsgw-1.1.11-5.el7.x86_64 389-adminutil-1.1.21-2.el7.x86_64 389-admin-1.1.46-1.el7.x86_64 389-ds-console-doc-1.2.16-1.el7.noarch 389-ds-1.2.2-6.el7.noarch 389-ds-base-libs-1.3.5.10-21.el7_3.x86_64 389-ds-base-1.3.5.10-21.el7_3.x86_64 389-admin-console-doc-1.1.12-1.el7.noarch 389-console-1.1.18-1.el7.noarch pcp-pmda-ds389-3.11.3-4.el7.x86_64 389-ds-console-1.2.16-1.el7.noarch [morgan@devldap03 ~]$ _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
Sorry these logs look "normal", that message that keeps repeating is expected when the console is idle (it's waiting for you to do something).
Perhaps there is something in the admin logs:
/var/log/dirsrv/admin-serv
Regards, Mark
On 08/16/2017 03:39 PM, Morgan Jones wrote:
Hello Mark,
See attached, "AbstractServerObject.StatusThread: waiting for change listeners to register” repeats presumably forever after it hangs.
Thanks,
-morgan
On Aug 16, 2017, at 3:23 PM, Mark Reynolds mareynol@redhat.com wrote:
Hi Morgan,
We need more info. Try running the console in debug mode:
389-console -D 9
Also look at the configuration DS access log
Mark
On 08/16/2017 02:57 PM, Morgan Jones wrote:
I’m in the process of installing 389 in CentOS 7 from epel (versions below) and find that the console becomes unresponsive after I install the 4th server. I can open the console and expand a few servers but within 30 seconds it consistently hangs. I am storing configuration data in one server.
Has anyone else seen this or do you have any insight?
Thanks,
-morgan
[morgan@devldap03 ~]$ rpm -qa|grep 389 pcp-pmda-ds389log-3.11.3-4.el7.x86_64 389-admin-console-1.1.12-1.el7.noarch 389-dsgw-1.1.11-5.el7.x86_64 389-adminutil-1.1.21-2.el7.x86_64 389-admin-1.1.46-1.el7.x86_64 389-ds-console-doc-1.2.16-1.el7.noarch 389-ds-1.2.2-6.el7.noarch 389-ds-base-libs-1.3.5.10-21.el7_3.x86_64 389-ds-base-1.3.5.10-21.el7_3.x86_64 389-admin-console-doc-1.1.12-1.el7.noarch 389-console-1.1.18-1.el7.noarch pcp-pmda-ds389-3.11.3-4.el7.x86_64 389-ds-console-1.2.16-1.el7.noarch [morgan@devldap03 ~]$ _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
Thanks—is there a trick to turning on admin-serv logging? I don’t have one and at least on first blush don’t see a means of enabling it.
-morgan
On Aug 17, 2017, at 3:16 PM, Mark Reynolds mareynol@redhat.com wrote:
Sorry these logs look "normal", that message that keeps repeating is expected when the console is idle (it's waiting for you to do something).
Perhaps there is something in the admin logs:
/var/log/dirsrv/admin-serv
Regards, Mark
On 08/16/2017 03:39 PM, Morgan Jones wrote:
Hello Mark,
See attached, "AbstractServerObject.StatusThread: waiting for change listeners to register” repeats presumably forever after it hangs.
Thanks,
-morgan
On Aug 16, 2017, at 3:23 PM, Mark Reynolds mareynol@redhat.com wrote:
Hi Morgan,
We need more info. Try running the console in debug mode:
389-console -D 9
Also look at the configuration DS access log
Mark
On 08/16/2017 02:57 PM, Morgan Jones wrote:
I’m in the process of installing 389 in CentOS 7 from epel (versions below) and find that the console becomes unresponsive after I install the 4th server. I can open the console and expand a few servers but within 30 seconds it consistently hangs. I am storing configuration data in one server.
Has anyone else seen this or do you have any insight?
Thanks,
-morgan
[morgan@devldap03 ~]$ rpm -qa|grep 389 pcp-pmda-ds389log-3.11.3-4.el7.x86_64 389-admin-console-1.1.12-1.el7.noarch 389-dsgw-1.1.11-5.el7.x86_64 389-adminutil-1.1.21-2.el7.x86_64 389-admin-1.1.46-1.el7.x86_64 389-ds-console-doc-1.2.16-1.el7.noarch 389-ds-1.2.2-6.el7.noarch 389-ds-base-libs-1.3.5.10-21.el7_3.x86_64 389-ds-base-1.3.5.10-21.el7_3.x86_64 389-admin-console-doc-1.1.12-1.el7.noarch 389-console-1.1.18-1.el7.noarch pcp-pmda-ds389-3.11.3-4.el7.x86_64 389-ds-console-1.2.16-1.el7.noarch [morgan@devldap03 ~]$ _______________________________________________ 389-users mailing list --
389-users@lists.fedoraproject.org
To unsubscribe send an email to
389-users-leave@lists.fedoraproject.org
On 08/22/2017 01:36 PM, Morgan Jones wrote:
Thanks—is there a trick to turning on admin-serv logging? I don’t have one and at least on first blush don’t see a means of enabling it.
The Admin Server (389-admin/389-adminutil) logs by default. Are you saying there is no /var/log/dirsrv/admin-serv/ directory? Or that the directory is empty
Also, just to clarify, you are doing all of this on same system correct? And when you create 4 server instances the console starts to hang? Or have you registered 4 remote servers instead of creating local instances?
Regards, Mark
-morgan
On Aug 17, 2017, at 3:16 PM, Mark Reynolds mareynol@redhat.com wrote:
Sorry these logs look "normal", that message that keeps repeating is expected when the console is idle (it's waiting for you to do something).
Perhaps there is something in the admin logs:
/var/log/dirsrv/admin-serv
Regards, Mark
On 08/16/2017 03:39 PM, Morgan Jones wrote:
Hello Mark,
See attached, "AbstractServerObject.StatusThread: waiting for change listeners to register” repeats presumably forever after it hangs.
Thanks,
-morgan
On Aug 16, 2017, at 3:23 PM, Mark Reynolds mareynol@redhat.com wrote:
Hi Morgan,
We need more info. Try running the console in debug mode:
389-console -D 9
Also look at the configuration DS access log
Mark
On 08/16/2017 02:57 PM, Morgan Jones wrote:
I’m in the process of installing 389 in CentOS 7 from epel (versions below) and find that the console becomes unresponsive after I install the 4th server. I can open the console and expand a few servers but within 30 seconds it consistently hangs. I am storing configuration data in one server.
Has anyone else seen this or do you have any insight?
Thanks,
-morgan
[morgan@devldap03 ~]$ rpm -qa|grep 389 pcp-pmda-ds389log-3.11.3-4.el7.x86_64 389-admin-console-1.1.12-1.el7.noarch 389-dsgw-1.1.11-5.el7.x86_64 389-adminutil-1.1.21-2.el7.x86_64 389-admin-1.1.46-1.el7.x86_64 389-ds-console-doc-1.2.16-1.el7.noarch 389-ds-1.2.2-6.el7.noarch 389-ds-base-libs-1.3.5.10-21.el7_3.x86_64 389-ds-base-1.3.5.10-21.el7_3.x86_64 389-admin-console-doc-1.1.12-1.el7.noarch 389-console-1.1.18-1.el7.noarch pcp-pmda-ds389-3.11.3-4.el7.x86_64 389-ds-console-1.2.16-1.el7.noarch [morgan@devldap03 ~]$ _______________________________________________ 389-users mailing list --
389-users@lists.fedoraproject.org
To unsubscribe send an email to
389-users-leave@lists.fedoraproject.org
On Aug 22, 2017, at 2:15 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/22/2017 01:36 PM, Morgan Jones wrote:
Thanks—is there a trick to turning on admin-serv logging? I don’t have one and at least on first blush don’t see a means of enabling it.
The Admin Server (389-admin/389-adminutil) logs by default. Are you saying there is no /var/log/dirsrv/admin-serv/ directory? Or that the directory is empty
Oh, sorry, I do have an access and error log in /var/log/dirsrv/admin-serv. The logs are sparse between startup and hang: ==> error <== [Wed Aug 23 11:15:13.941399 2017] [:notice] [pid 15780:tid 139865400174336] [client 127.0.0.1:43150] admserv_check_authz(): passing [/admin-serv/authenticate] to the userauth handler
==> access <== 127.0.0.1 - cn=directory manager [23/Aug/2017:11:15:13 -0400] "GET /admin-serv/authenticate HTTP/1.0" 200 329
Also, just to clarify, you are doing all of this on same system correct? And when you create 4 server instances the console starts to hang? Or have you registered 4 remote servers instead of creating local instances?
I’m registering remote servers. We have a total of 5. It starts to hang after installing the 4th.
-morgan
On 08/23/2017 11:18 AM, Morgan Jones wrote:
On Aug 22, 2017, at 2:15 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/22/2017 01:36 PM, Morgan Jones wrote:
Thanks—is there a trick to turning on admin-serv logging? I don’t have one and at least on first blush don’t see a means of enabling it.
The Admin Server (389-admin/389-adminutil) logs by default. Are you saying there is no /var/log/dirsrv/admin-serv/ directory? Or that the directory is empty
Oh, sorry, I do have an access and error log in /var/log/dirsrv/admin-serv. The logs are sparse between startup and hang: ==> error <== [Wed Aug 23 11:15:13.941399 2017] [:notice] [pid 15780:tid 139865400174336] [client 127.0.0.1:43150] admserv_check_authz(): passing [/admin-serv/authenticate] to the userauth handler
==> access <== 127.0.0.1 - cn=directory manager [23/Aug/2017:11:15:13 -0400] "GET /admin-serv/authenticate HTTP/1.0" 200 329
Also, just to clarify, you are doing all of this on same system correct? And when you create 4 server instances the console starts to hang? Or have you registered 4 remote servers instead of creating local instances?
I’m registering remote servers. We have a total of 5. It starts to hang after installing the 4th.
Do you see a drop off in performance as you get close to the 4th? Or everything is fine, then you add more server and its grinds to halt?
If you register the servers in a different order, is it still the 4th that causes issues? Or is it one particular server?
Mark
-morgan
On Aug 23, 2017, at 11:32 AM, Mark Reynolds mareynol@redhat.com wrote:
On 08/23/2017 11:18 AM, Morgan Jones wrote:
I’m registering remote servers. We have a total of 5. It starts to hang after installing the 4th.
Do you see a drop off in performance as you get close to the 4th? Or everything is fine, then you add more server and its grinds to halt?
Everything seems fine with <=3. Once I add the 4th it hangs after 3-10 clicks in the interface (expanding/closing items, selecting items).
If you register the servers in a different order, is it still the 4th that causes issues? Or is it one particular server?
Good question, I’m not sure--I’ll switch up the order and report back shortly. These are brand new CentOS 7 systems so in theory have no cruft.
Thanks,
-morgan
I should add a few things:
I tried installing Java from Oracle, there are reports of problems with the openjdk.
Periodically the console does seem to return if I let it sit a few minutes. This is not consistent.
I tried an strace -f on the process. I don’t see anything particularly interesting though it’s not completely hung, the below repeats more or less over and over:
[pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 368271265}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 418417522}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 468582369}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 518770094}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 568927454}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 619067470}, ffffffff <unfinished ...> [pid 27466] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27466] futex(0x256b528, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27466] futex(0x256b554, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276467, 583608758}, ffffffff <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 669235628}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 719425518}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 769587606}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 819765324}, ffffffff <unfinished ...> [pid 27457] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27457] futex(0x7f389447ce28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27457] futex(0x7f389447ce54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276467, 792643143}, ffffffff <unfinished ...> [pid 27442] <... poll resumed> ) = 0 (Timeout) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0
-morgan
On Aug 23, 2017, at 11:18 AM, Morgan Jones morgan@morganjones.org wrote:
On Aug 22, 2017, at 2:15 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/22/2017 01:36 PM, Morgan Jones wrote:
Thanks—is there a trick to turning on admin-serv logging? I don’t have one and at least on first blush don’t see a means of enabling it.
The Admin Server (389-admin/389-adminutil) logs by default. Are you saying there is no /var/log/dirsrv/admin-serv/ directory? Or that the directory is empty
Oh, sorry, I do have an access and error log in /var/log/dirsrv/admin-serv. The logs are sparse between startup and hang: ==> error <== [Wed Aug 23 11:15:13.941399 2017] [:notice] [pid 15780:tid 139865400174336] [client 127.0.0.1:43150] admserv_check_authz(): passing [/admin-serv/authenticate] to the userauth handler
==> access <== 127.0.0.1 - cn=directory manager [23/Aug/2017:11:15:13 -0400] "GET /admin-serv/authenticate HTTP/1.0" 200 329
Also, just to clarify, you are doing all of this on same system correct? And when you create 4 server instances the console starts to hang? Or have you registered 4 remote servers instead of creating local instances?
I’m registering remote servers. We have a total of 5. It starts to hang after installing the 4th.
-morgan _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
On 08/23/2017 11:40 AM, Morgan Jones wrote:
I should add a few things:
I tried installing Java from Oracle, there are reports of problems with the openjdk.
What problems? You should be using: java-1.8.0-openjdk
Also, are you using SSL/TLS in your servers? Or are these just plain installs?
Thanks, Mark
Periodically the console does seem to return if I let it sit a few minutes. This is not consistent.
I tried an strace -f on the process. I don’t see anything particularly interesting though it’s not completely hung, the below repeats more or less over and over:
[pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 368271265}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 418417522}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 468582369}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 518770094}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 568927454}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 619067470}, ffffffff <unfinished ...> [pid 27466] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27466] futex(0x256b528, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27466] futex(0x256b554, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276467, 583608758}, ffffffff <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 669235628}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 719425518}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 769587606}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 819765324}, ffffffff <unfinished ...> [pid 27457] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27457] futex(0x7f389447ce28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27457] futex(0x7f389447ce54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276467, 792643143}, ffffffff <unfinished ...> [pid 27442] <... poll resumed> ) = 0 (Timeout) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0
-morgan
On Aug 23, 2017, at 11:18 AM, Morgan Jones morgan@morganjones.org wrote:
On Aug 22, 2017, at 2:15 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/22/2017 01:36 PM, Morgan Jones wrote:
Thanks—is there a trick to turning on admin-serv logging? I don’t have one and at least on first blush don’t see a means of enabling it.
The Admin Server (389-admin/389-adminutil) logs by default. Are you saying there is no /var/log/dirsrv/admin-serv/ directory? Or that the directory is empty
Oh, sorry, I do have an access and error log in /var/log/dirsrv/admin-serv. The logs are sparse between startup and hang: ==> error <== [Wed Aug 23 11:15:13.941399 2017] [:notice] [pid 15780:tid 139865400174336] [client 127.0.0.1:43150] admserv_check_authz(): passing [/admin-serv/authenticate] to the userauth handler
==> access <== 127.0.0.1 - cn=directory manager [23/Aug/2017:11:15:13 -0400] "GET /admin-serv/authenticate HTTP/1.0" 200 329
Also, just to clarify, you are doing all of this on same system correct? And when you create 4 server instances the console starts to hang? Or have you registered 4 remote servers instead of creating local instances?
I’m registering remote servers. We have a total of 5. It starts to hang after installing the 4th.
-morgan _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
On Aug 23, 2017, at 12:11 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/23/2017 11:40 AM, Morgan Jones wrote:
I should add a few things:
I tried installing Java from Oracle, there are reports of problems with the openjdk.
What problems? You should be using: java-1.8.0-openjdk
I started there, I’ll go back.
Also, are you using SSL/TLS in your servers? Or are these just plain installs?
They’re plain installs so far. SSL/TLS is the next step though.
-morgan
On 08/23/2017 11:40 AM, Morgan Jones wrote:
I should add a few things:
I tried installing Java from Oracle, there are reports of problems with the openjdk.
Periodically the console does seem to return if I let it sit a few minutes. This is not consistent.
I tried an strace -f on the process. I don’t see anything particularly interesting though it’s not completely hung, the below repeats more or less over and over:
[pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 368271265}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 418417522}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 468582369}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 518770094}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 568927454}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 619067470}, ffffffff <unfinished ...> [pid 27466] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27466] futex(0x256b528, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27466] futex(0x256b554, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276467, 583608758}, ffffffff <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 669235628}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 719425518}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 769587606}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27440] futex(0x7f38940cfd54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276466, 819765324}, ffffffff <unfinished ...> [pid 27457] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27457] futex(0x7f389447ce28, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 27457] futex(0x7f389447ce54, FUTEX_WAIT_BITSET_PRIVATE, 1, {1276467, 792643143}, ffffffff <unfinished ...> [pid 27442] <... poll resumed> ) = 0 (Timeout) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0
Sorry forgot to comment on this...
This explains the "hang" - connections to the remove server(s) are timing out.
Can you look at the DS access logs on a remote server during the hang (note there is a 30 sec log buffer with the access log). Perhaps just tail the access log, reproduce the hang (wait 30 seconds), and provide the complete tail output.
-morgan
On Aug 23, 2017, at 11:18 AM, Morgan Jones morgan@morganjones.org wrote:
On Aug 22, 2017, at 2:15 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/22/2017 01:36 PM, Morgan Jones wrote:
Thanks—is there a trick to turning on admin-serv logging? I don’t have one and at least on first blush don’t see a means of enabling it.
The Admin Server (389-admin/389-adminutil) logs by default. Are you saying there is no /var/log/dirsrv/admin-serv/ directory? Or that the directory is empty
Oh, sorry, I do have an access and error log in /var/log/dirsrv/admin-serv. The logs are sparse between startup and hang: ==> error <== [Wed Aug 23 11:15:13.941399 2017] [:notice] [pid 15780:tid 139865400174336] [client 127.0.0.1:43150] admserv_check_authz(): passing [/admin-serv/authenticate] to the userauth handler
==> access <== 127.0.0.1 - cn=directory manager [23/Aug/2017:11:15:13 -0400] "GET /admin-serv/authenticate HTTP/1.0" 200 329
Also, just to clarify, you are doing all of this on same system correct? And when you create 4 server instances the console starts to hang? Or have you registered 4 remote servers instead of creating local instances?
I’m registering remote servers. We have a total of 5. It starts to hang after installing the 4th.
-morgan _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
On Aug 23, 2017, at 12:17 PM, Mark Reynolds mareynol@redhat.com wrote:
[pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0
Sorry forgot to comment on this...
This explains the "hang" - connections to the remove server(s) are timing out.
Can you look at the DS access logs on a remote server during the hang (note there is a 30 sec log buffer with the access log). Perhaps just tail the access log, reproduce the hang (wait 30 seconds), and provide the complete tail output.
Aha, ok, thanks. Which access log do you want or do you want all of them?
-morgan
On 08/23/2017 12:31 PM, Morgan Jones wrote:
On Aug 23, 2017, at 12:17 PM, Mark Reynolds mareynol@redhat.com wrote:
[pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0
Sorry forgot to comment on this...
This explains the "hang" - connections to the remove server(s) are timing out.
Can you look at the DS access logs on a remote server during the hang (note there is a 30 sec log buffer with the access log). Perhaps just tail the access log, reproduce the hang (wait 30 seconds), and provide the complete tail output.
Aha, ok, thanks. Which access log do you want or do you want all of them?
Just the access log from the server that triggers the initial hang.
-morgan
Mark,
See attached. The break in the log is where it hung and then came back to life.
I’m also going to follow up with our networking and security folks to see if we can find anything there. These hosts are all on the same subnet for what it’s worth.
Thanks for the help.
-morgan
On Aug 23, 2017, at 12:35 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/23/2017 12:31 PM, Morgan Jones wrote:
On Aug 23, 2017, at 12:17 PM, Mark Reynolds mareynol@redhat.com wrote:
[pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0
Sorry forgot to comment on this...
This explains the "hang" - connections to the remove server(s) are timing out.
Can you look at the DS access logs on a remote server during the hang (note there is a 30 sec log buffer with the access log). Perhaps just tail the access log, reproduce the hang (wait 30 seconds), and provide the complete tail output.
Aha, ok, thanks. Which access log do you want or do you want all of them?
Just the access log from the server that triggers the initial hang.
-morgan
On 08/23/2017 03:09 PM, Morgan Jones wrote:
Mark,
See attached. The break in the log is where it hung and then came back to life.
Odd, I don't see the connection being closed. Looks like the client (the console) did not send any requests during that time, what about the other servers at that same time?
23/Aug/2017:12:56:13 <--> 23/Aug/2017:12:59:58
Also, did you try registering the servers in a different order?
Perhaps try and get a stack trace from the console during the hang using jstack (from Oracle java)
Thanks, Mark
I’m also going to follow up with our networking and security folks to see if we can find anything there. These hosts are all on the same subnet for what it’s worth.
Thanks for the help.
-morgan
On Aug 23, 2017, at 12:35 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/23/2017 12:31 PM, Morgan Jones wrote:
On Aug 23, 2017, at 12:17 PM, Mark Reynolds mareynol@redhat.com wrote:
[pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0
Sorry forgot to comment on this...
This explains the "hang" - connections to the remove server(s) are timing out.
Can you look at the DS access logs on a remote server during the hang (note there is a 30 sec log buffer with the access log). Perhaps just tail the access log, reproduce the hang (wait 30 seconds), and provide the complete tail output.
Aha, ok, thanks. Which access log do you want or do you want all of them?
Just the access log from the server that triggers the initial hang.
-morgan
Hello Mark et al,
This is a mea culpa—it turns out we’re running firewalld on our “new” centos 7 systems and it was blocking port 389. If I disable firewalld I can’t repeat the problem.
Thanks for your help on this.
-morgan
On Aug 23, 2017, at 4:24 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/23/2017 03:09 PM, Morgan Jones wrote:
Mark,
See attached. The break in the log is where it hung and then came back to life.
Odd, I don't see the connection being closed. Looks like the client (the console) did not send any requests during that time, what about the other servers at that same time?
23/Aug/2017:12:56:13 <--> 23/Aug/2017:12:59:58
Also, did you try registering the servers in a different order?
Perhaps try and get a stack trace from the console during the hang using jstack (from Oracle java)
Thanks, Mark
I’m also going to follow up with our networking and security folks to see if we can find anything there. These hosts are all on the same subnet for what it’s worth.
Thanks for the help.
-morgan
On Aug 23, 2017, at 12:35 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/23/2017 12:31 PM, Morgan Jones wrote:
On Aug 23, 2017, at 12:17 PM, Mark Reynolds mareynol@redhat.com wrote:
[pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0
Sorry forgot to comment on this...
This explains the "hang" - connections to the remove server(s) are timing out.
Can you look at the DS access logs on a remote server during the hang (note there is a 30 sec log buffer with the access log). Perhaps just tail the access log, reproduce the hang (wait 30 seconds), and provide the complete tail output.
Aha, ok, thanks. Which access log do you want or do you want all of them?
Just the access log from the server that triggers the initial hang.
-morgan
On 09/04/2017 10:20 PM, Morgan Jones wrote:
Hello Mark et al,
This is a mea culpa—it turns out we’re running firewalld on our “new” centos 7 systems and it was blocking port 389. If I disable firewalld I can’t repeat the problem.
Thanks for your help on this.
Thanks for the follow up, glad it got figured out.
Mark
-morgan
On Aug 23, 2017, at 4:24 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/23/2017 03:09 PM, Morgan Jones wrote:
Mark,
See attached. The break in the log is where it hung and then came back to life.
Odd, I don't see the connection being closed. Looks like the client (the console) did not send any requests during that time, what about the other servers at that same time?
23/Aug/2017:12:56:13 <--> 23/Aug/2017:12:59:58
Also, did you try registering the servers in a different order?
Perhaps try and get a stack trace from the console during the hang using jstack (from Oracle java)
Thanks, Mark
I’m also going to follow up with our networking and security folks to see if we can find anything there. These hosts are all on the same subnet for what it’s worth.
Thanks for the help.
-morgan
On Aug 23, 2017, at 12:35 PM, Mark Reynolds mareynol@redhat.com wrote:
On 08/23/2017 12:31 PM, Morgan Jones wrote:
On Aug 23, 2017, at 12:17 PM, Mark Reynolds mareynol@redhat.com wrote:
> [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) > [pid 27442] recvmsg(14, 0x7f3880ef74d0, 0) = -1 EAGAIN (Resource temporarily unavailable) > [pid 27442] poll([{fd=14, events=POLLRDNORM}, {fd=15, events=POLLRDNORM}], 2, 500 <unfinished ...> > [pid 27440] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) > [pid 27440] futex(0x7f38940cfd28, FUTEX_WAKE_PRIVATE, 1) = 0 Sorry forgot to comment on this...
This explains the "hang" - connections to the remove server(s) are timing out.
Can you look at the DS access logs on a remote server during the hang (note there is a 30 sec log buffer with the access log). Perhaps just tail the access log, reproduce the hang (wait 30 seconds), and provide the complete tail output.
Aha, ok, thanks. Which access log do you want or do you want all of them?
Just the access log from the server that triggers the initial hang.
-morgan
389-users@lists.fedoraproject.org