Hello everyone,
We were using Freeipa on Fedora 24. And we are in the process to upgrade to Fedora 28. We have a cluster of 2 nodes (freeipa-01 and freeipa-02).
I am trying to upgrade one server after the other, from one release to the next.
Basically:
freeipa-01 Fedora 24 -> Fedora 25
freeipa-02 Fedora 24 -> Fedora 25 freeipa-02 Fedora 25 -> Fedora 26
freeipa-01 Fedora 25 -> Fedora 26 freeipa-01 Fedora 26 -> Fedora 27
freeipa-02 Fedora 26 -> Fedora 27 freeipa-02 Fedora 27 -> Fedora 28
freeipa-01 Fedora 27 -> Fedora 28
Since Fedora doesn’t support to jump from one version to another, except one release at the time.
My idea is to check that once a server is upgraded, then everything is stable, before going to the next server, and try to be as near as possible from a version point of view between the 2 freeipa node cluster.
Today http://airmail.calendar/2018-06-12%2012:00:00%20CEST, I could upgrade without problems from Fedora 24 -> Fedora 25 on both nodes (freeipa-01 and freeipa-02).
In trying to upgrade to Fedora 26, I got some problems, the main problem is that the upgrade of ldap 389 is not successful, and the one from IPA either. After investigating a long moment, I have found that ns-slapd listen only to IPv6, on UDP, and NOT on IPv4 and TCP.
Here is what I have:
[root@freeipa-02 lib]# lsof -Pni |grep slap ns-slapd 21005 dirsrv 9u IPv6 1617283379 <//1617283379> 0t0 UDP *:389 ns-slapd 21005 dirsrv 77u IPv4 1617321218 <//1617321218> 0t0 TCP 10.100.0.102:60646->10.100.0.101:389 (ESTABLISHED) ns-slapd 21005 dirsrv 81u IPv4 1617317640 <//1617317640> 0t0 TCP 10.100.0.102:60648->10.100.0.101:389 (ESTABLISHED)
So, I decided to look at the file dse.ldif, and found that the entry "nsslapd-port” was set to “0” and no “nsslapd-listenhost” was not set at all. I have then added the line
nsslapd-listenhost: 0.0.0.0
and changed the nsslapd-port to look like:
nsslap-port: 389
And after doing a
systemctl stop dirsrv@DOM-LOCAL ; systemctl start dirsrv@DOM-LOCAL
No changes… all modification on my dse.ldif were gone.
I stopped again the dirsrv, did again my changes on dse.ldif, and run the following command:
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-DOM-LOCAL -i /var/run/dirsrv/slapd-DOM-LOCAL.pid
and now, I have the following:
[root@freeipa-02 updates]# lsof -Pni |grep 389 ns-slapd 78507 dirsrv 10u IPv6 1681165214 <//1681165214> 0t0 UDP *:389 ns-slapd 78507 dirsrv 11u IPv4 1681165216 <//1681165216> 0t0 TCP *:389 (LISTEN) ns-slapd 78507 dirsrv 114u IPv4 1684131928 <//1684131928> 0t0 TCP 10.100.0.102:389->10.100.0.110:36828 (ESTABLISHED)
So my questions are: - how to change the dse.ldif file? - Is there another way to ensure that the port that listen is TCP / 389 on IPv4? - Is there something that needs to be done between Fedora 25 and 26? Knowing that I will go to Fedora 28, is there something that I need to be aware of? - Anything that can help me generally with my upgrade path?
Best regards, Alessandro
On Tue, 2018-06-12 at 12:15 -0700, Alessandro Perucchi via FreeIPA- users wrote:
Hello everyone,
We were using Freeipa on Fedora 24. And we are in the process to upgrade to Fedora 28. We have a cluster of 2 nodes (freeipa-01 and freeipa-02).
I am trying to upgrade one server after the other, from one release to the next.
Basically:
freeipa-01 Fedora 24 -> Fedora 25
freeipa-02 Fedora 24 -> Fedora 25 freeipa-02 Fedora 25 -> Fedora 26
freeipa-01 Fedora 25 -> Fedora 26 freeipa-01 Fedora 26 -> Fedora 27
freeipa-02 Fedora 26 -> Fedora 27 freeipa-02 Fedora 27 -> Fedora 28
freeipa-01 Fedora 27 -> Fedora 28
Since Fedora doesn’t support to jump from one version to another, except one release at the time.
My idea is to check that once a server is upgraded, then everything is stable, before going to the next server, and try to be as near as possible from a version point of view between the 2 freeipa node cluster.
Today http://airmail.calendar/2018-06-12%2012:00:00%20CEST, I could upgrade without problems from Fedora 24 -> Fedora 25 on both nodes (freeipa-01 and freeipa-02).
In trying to upgrade to Fedora 26, I got some problems, the main problem is that the upgrade of ldap 389 is not successful, and the one from IPA either. After investigating a long moment, I have found that ns-slapd listen only to IPv6, on UDP, and NOT on IPv4 and TCP.
Here is what I have:
[root@freeipa-02 lib]# lsof -Pni |grep slap ns-slapd 21005 dirsrv 9u IPv6 1617283379 <//1617283379> 0t0 UDP *:389 ns-slapd 21005 dirsrv 77u IPv4 1617321218 <//1617321218> 0t0 TCP 10.100.0.102:60646->10.100.0.101:389 (ESTABLISHED) ns-slapd 21005 dirsrv 81u IPv4 1617317640 <//1617317640> 0t0 TCP 10.100.0.102:60648->10.100.0.101:389 (ESTABLISHED)
So, I decided to look at the file dse.ldif, and found that the entry "nsslapd-port” was set to “0” and no “nsslapd-listenhost” was not set at all. I have then added the line
nsslapd-listenhost: 0.0.0.0
and changed the nsslapd-port to look like:
nsslap-port: 389
And after doing a
systemctl stop dirsrv@DOM-LOCAL ; systemctl start dirsrv@DOM-LOCAL
No changes… all modification on my dse.ldif were gone.
I stopped again the dirsrv, did again my changes on dse.ldif, and run the following command:
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-DOM-LOCAL -i /var/run/dirsrv/slapd-DOM-LOCAL.pid
and now, I have the following:
[root@freeipa-02 updates]# lsof -Pni |grep 389 ns-slapd 78507 dirsrv 10u IPv6 1681165214 <//1681165214> 0t0 UDP *:389 ns-slapd 78507 dirsrv 11u IPv4 1681165216 <//1681165216> 0t0 TCP *:389 (LISTEN) ns-slapd 78507 dirsrv 114u IPv4 1684131928 <//1684131928> 0t0 TCP 10.100.0.102:389->10.100.0.110:36828 (ESTABLISHED)
So my questions are:
- how to change the dse.ldif file?
You have to stop ns-slapd before changing the file.
- Is there another way to ensure that the port that listen is TCP / 389 on
IPv4?
The port was disabled during some upgrade operations, your situation meant some upgrade failed and that old version failed to set back the port in dse.ldif This is a bug and shouldn't happen in recent versions.
- Is there something that needs to be done between Fedora 25 and 26?
Is this upgrade bug repeatable ? (keep in mind that F26 is practically EOL)
Knowing that I will go to Fedora 28, is there something that I need to be aware of?
Yes, read this list archives before you attempt F28 upgrades, you may have to use updates-testing as the GA bits where busted wrt replication for upgrades.
- Anything that can help me generally with my upgrade path?
In general your approach is ok, make backups :-)
Simo.
On 12 June 2018 at 21:22:56, Simo Sorce (simo@redhat.com) wrote:
On Tue, 2018-06-12 at 12:15 -0700, Alessandro Perucchi via FreeIPA- users wrote:
Hello everyone,
We were using Freeipa on Fedora 24. And we are in the process to upgrade
to
Fedora 28. We have a cluster of 2 nodes (freeipa-01 and freeipa-02).
I am trying to upgrade one server after the other, from one release to the
next.
Basically:
freeipa-01 Fedora 24 -> Fedora 25
freeipa-02 Fedora 24 -> Fedora 25 freeipa-02 Fedora 25 -> Fedora 26
freeipa-01 Fedora 25 -> Fedora 26 freeipa-01 Fedora 26 -> Fedora 27
freeipa-02 Fedora 26 -> Fedora 27 freeipa-02 Fedora 27 -> Fedora 28
freeipa-01 Fedora 27 -> Fedora 28
Since Fedora doesn’t support to jump from one version to another, except one release at the time.
My idea is to check that once a server is upgraded, then everything is stable, before going to the next server, and try to be as near as possible
from a version point of view between the 2 freeipa node cluster.
Today, I could upgrade without problems from Fedora 24 -> Fedora 25 on both nodes (freeipa-01 and freeipa-02).
In trying to upgrade to Fedora 26, I got some problems, the main problem
is
that the upgrade of ldap 389 is not successful, and the one from IPA
either.
After investigating a long moment, I have found that ns-slapd listen only to IPv6, on UDP, and NOT on IPv4 and TCP.
Here is what I have:
[root@freeipa-02 lib]# lsof -Pni |grep slap ns-slapd 21005 dirsrv 9u IPv6 1617283379 <//1617283379> 0t0 UDP *:389 ns-slapd 21005 dirsrv 77u IPv4 1617321218 <//1617321218> 0t0 TCP 10.100.0.102:60646->10.100.0.101:389 (ESTABLISHED) ns-slapd 21005 dirsrv 81u IPv4 1617317640 <//1617317640> 0t0 TCP 10.100.0.102:60648->10.100.0.101:389 (ESTABLISHED)
So, I decided to look at the file dse.ldif, and found that the entry "nsslapd-port” was set to “0” and no “nsslapd-listenhost” was not set at all. I have then added the line
nsslapd-listenhost: 0.0.0.0
and changed the nsslapd-port to look like:
nsslap-port: 389
And after doing a
systemctl stop dirsrv@DOM-LOCAL ; systemctl start dirsrv@DOM-LOCAL
No changes… all modification on my dse.ldif were gone.
I stopped again the dirsrv, did again my changes on dse.ldif, and run the following command:
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-DOM-LOCAL -i /var/run/dirsrv/slapd-DOM-LOCAL.pid
and now, I have the following:
[root@freeipa-02 updates]# lsof -Pni |grep 389 ns-slapd 78507 dirsrv 10u IPv6 1681165214 <//1681165214> 0t0 UDP *:389 ns-slapd 78507 dirsrv 11u IPv4 1681165216 <//1681165216> 0t0 TCP *:389 (LISTEN) ns-slapd 78507 dirsrv 114u IPv4 1684131928 <//1684131928> 0t0 TCP 10.100.0.102:389->10.100.0.110:36828 (ESTABLISHED)
So my questions are:
- how to change the dse.ldif file?
You have to stop ns-slapd before changing the file.
This is what I have done several times. or have I… maybe not…
I will try again.
- Is there another way to ensure that the port that listen is TCP / 389
on
IPv4?
The port was disabled during some upgrade operations, your situation meant some upgrade failed and that old version failed to set back the port in dse.ldif This is a bug and shouldn't happen in recent versions.
Does it means that I need to upgrade to Fedora 28, and then try to upgrade FreeIPA?
- Is there something that needs to be done between Fedora 25 and 26?
Is this upgrade bug repeatable ? (keep in mind that F26 is practically EOL)
Yes, it is repeatable, since I am trying to do it since this 24 hours, and it drives me crazy… and nothing by googling seems to help!
I know this is EOL, or nearly… That’s also why we wanted to upgrade to the latest.
Knowing that I will go to Fedora 28, is there something that I need to be aware of?
Yes, read this list archives before you attempt F28 upgrades, you may have to use updates-testing as the GA bits where busted wrt replication for upgrades.
Ok, guess I have some reading to do :-D
- Anything that can help me generally with my upgrade path?
In general your approach is ok, make backups :-)
Glad that I’m doing it right :-)
If you have any other approach, then I am also open to anything else.
Nevertheless thank you!
I think the problem might be related to that error I got within dirsrv:
[12/Jun/2018:21:28:41.780869662 +0000] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/freeipa-02.jahia.local@JAHIA.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
if I do a klist -kt /etc/dirsrv/ds.keytab
I get the following:
Keytab name: FILE:/etc/dirsrv/ds.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 2 12/20/16 09:39:50 ldap/freeipa-02.jahia.local@JAHIA.LOCAL 2 12/20/16 09:39:50 ldap/freeipa-02.jahia.local@JAHIA.LOCAL 2 12/20/16 09:39:50 ldap/freeipa-02.jahia.local@JAHIA.LOCAL 2 12/20/16 09:39:51 ldap/freeipa-02.jahia.local@JAHIA.LOCAL 2 12/20/16 09:39:51 ldap/freeipa-02.jahia.local@JAHIA.LOCAL 2 12/20/16 09:39:51 ldap/freeipa-02.jahia.local@JAHIA.LOCAL
And after I started the disabled service “krb5kdc”, everything was “””solved””” for that part of errors.
I also found that during the upgrade from F25 -> F26 the dse.ldif was changed to have
nsslapd-port: 0
instead of the port 389 as it was in F25.
Since `nsslapd-port: 0` means to use the ldaps.
and since everything behind is using that (ipa-server-upgrade) I cannot finish the upgrade correctly.
I could go past some steps of the upgrade, but not past the CA verification:
Upgrading IPA: [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server [6/10]: updating schema [7/10]: upgrading server [8/10]: stopping directory server [9/10]: restoring configuration [10/10]: starting directory server Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved /etc/dirsrv/slapd-JAHIA-LOCAL/certmap.conf is now managed by IPA. It will be overwritten. A backup of the original will be made. [Verifying that CA proxy configuration is correct] IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. CA did not start in 300.0s The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
And the error I get is basically:
0.localhost-startStop-1 - [12/Jun/2018:21:22:29 UTC] [8] [3] In Ldap (bound) connection pool to host freeipa-02.jahia.local port 636, Cannot connect to LDAP server. Error: netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)
which caused the following error:
12-Jun-2018 21:27:50.393 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase.backgroundProcess Exception processing realm com.netscape.cms.tomcat.ProxyRealm@67a 992d background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:130) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1154) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5715) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349) at java.lang.Thread.run(Thread.java:748)
and then the following error:
2018-06-12T21:27:21Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2018-06-12T21:27:21Z DEBUG Waiting for CA to start... 2018-06-12T21:27:22Z DEBUG request POST http://freeipa-02.jahia.local:8080/ca/admin/ca/getStatus 2018-06-12T21:27:22Z DEBUG request body '' 2018-06-12T21:27:22Z DEBUG response status 500 2018-06-12T21:27:22Z DEBUG response headers {'content-length': '2351', 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Tue, 12 Jun 2018 21:27:22 GMT', 'content-type': 'tex t/html;charset=utf-8'} 2018-06-12T21:27:22Z DEBUG response body '<!DOCTYPE html><html><head><title>Apache Tomcat/8.0.50 - Error report</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:# 525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white ;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}</style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><div class= "line"></div><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.< /u></p><p><b>exception</b></p><pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:138)\n\torg.apache.catalina.authenti cator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)\n\torg.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAcces sLogValve.java:620)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)\n\torg.apache.coyo te.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)\n\torg.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)\n\torg.apache.tomcat.util.net.NioEndpoint$So cketProcessor.run(NioEndpoint.java:1495)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tor g.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/8.0.50 logs.</u></p><hr class="line"><h3>Apache Tomcat/8.0.50</h3></body></html>'
So the question now would be… What am I missing to have ldaps with port 636?
Thank you in advance again!
Alessandro
On 12 June 2018 at 21:22:56, Simo Sorce (simo@redhat.com) wrote:
On Tue, 2018-06-12 at 12:15 -0700, Alessandro Perucchi via FreeIPA- users wrote:
Hello everyone,
We were using Freeipa on Fedora 24. And we are in the process to upgrade
to
Fedora 28. We have a cluster of 2 nodes (freeipa-01 and freeipa-02).
I am trying to upgrade one server after the other, from one release to
the
next.
Basically:
freeipa-01 Fedora 24 -> Fedora 25
freeipa-02 Fedora 24 -> Fedora 25 freeipa-02 Fedora 25 -> Fedora 26
freeipa-01 Fedora 25 -> Fedora 26 freeipa-01 Fedora 26 -> Fedora 27
freeipa-02 Fedora 26 -> Fedora 27 freeipa-02 Fedora 27 -> Fedora 28
freeipa-01 Fedora 27 -> Fedora 28
Since Fedora doesn’t support to jump from one version to another, except one release at the time.
My idea is to check that once a server is upgraded, then everything is stable, before going to the next server, and try to be as near as
possible
from a version point of view between the 2 freeipa node cluster.
Today http://airmail.calendar/2018-06-12%2012:00:00%20CEST, I could upgrade without problems from Fedora 24 -> Fedora 25 on both nodes (freeipa-01 and freeipa-02).
In trying to upgrade to Fedora 26, I got some problems, the main problem
is
that the upgrade of ldap 389 is not successful, and the one from IPA
either.
After investigating a long moment, I have found that ns-slapd listen only to IPv6, on UDP, and NOT on IPv4 and TCP.
Here is what I have:
[root@freeipa-02 lib]# lsof -Pni |grep slap ns-slapd 21005 dirsrv 9u IPv6 1617283379 <//1617283379> 0t0 UDP *:389 ns-slapd 21005 dirsrv 77u IPv4 1617321218 <//1617321218> 0t0 TCP 10.100.0.102:60646->10.100.0.101:389 (ESTABLISHED) ns-slapd 21005 dirsrv 81u IPv4 1617317640 <//1617317640> 0t0 TCP 10.100.0.102:60648->10.100.0.101:389 (ESTABLISHED)
So, I decided to look at the file dse.ldif, and found that the entry "nsslapd-port” was set to “0” and no “nsslapd-listenhost” was not set at all. I have then added the line
nsslapd-listenhost: 0.0.0.0
and changed the nsslapd-port to look like:
nsslap-port: 389
And after doing a
systemctl stop dirsrv@DOM-LOCAL ; systemctl start dirsrv@DOM-LOCAL
No changes… all modification on my dse.ldif were gone.
I stopped again the dirsrv, did again my changes on dse.ldif, and run the following command:
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-DOM-LOCAL -i /var/run/dirsrv/slapd-DOM-LOCAL.pid
and now, I have the following:
[root@freeipa-02 updates]# lsof -Pni |grep 389 ns-slapd 78507 dirsrv 10u IPv6 1681165214 <//1681165214> 0t0 UDP *:389 ns-slapd 78507 dirsrv 11u IPv4 1681165216 <//1681165216> 0t0 TCP *:389 (LISTEN) ns-slapd 78507 dirsrv 114u IPv4 1684131928 <//1684131928> 0t0 TCP 10.100.0.102:389->10.100.0.110:36828 (ESTABLISHED)
So my questions are:
- how to change the dse.ldif file?
You have to stop ns-slapd before changing the file.
- Is there another way to ensure that the port that listen is TCP / 389
on
IPv4?
The port was disabled during some upgrade operations, your situation meant some upgrade failed and that old version failed to set back the port in dse.ldif This is a bug and shouldn't happen in recent versions.
- Is there something that needs to be done between Fedora 25 and 26?
Is this upgrade bug repeatable ? (keep in mind that F26 is practically EOL)
Knowing that I will go to Fedora 28, is there something that I need to be aware of?
Yes, read this list archives before you attempt F28 upgrades, you may have to use updates-testing as the GA bits where busted wrt replication for upgrades.
- Anything that can help me generally with my upgrade path?
In general your approach is ok, make backups :-)
Simo.
freeipa-users@lists.fedorahosted.org