Dear all,
we are trying to upgrade our IPA 4.4 to current 4.6.
So we did a "yum update" and then a "ipa-server-upgrade" which miserably failes with:
ipaserver.install.server.upgrade: INFO: [Migrating certificate profiles to LDAP] ipalib.backend: DEBUG: Created connection context.ldap2_140564957536912 ipapython.ipaldap: DEBUG: flushing ldapi://%2fvar%2frun%2fslapd-PLEIADES-UNI-WUPPERTAL-DE.socket from SchemaCache ipapython.ipaldap: DEBUG: retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-PLEIADES-UNI-WUPPERTAL-DE.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fd7d5efe7a0> ipalib.backend: DEBUG: Destroyed connection context.ldap2_140564957536912 ipapython.dogtag: DEBUG: request GET https://ipa2.pleiades.uni-wuppertal.de:8443/ca/rest/account/login ipapython.dogtag: DEBUG: request body '' ipapython.dogtag: DEBUG: response status 401 ipapython.dogtag: DEBUG: response headers Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 01:00:00 CET WWW-Authenticate: Basic realm="Certificate Authority" Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 951 Date: Tue, 12 Feb 2019 10:48:51 GMT
ipapython.dogtag: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>This request requires HTTP authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' ipaserver.install.ipa_server_upgrade: ERROR: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. ipapython.admintool: DEBUG: File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2085, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1952, in upgrade_configuration ca_enable_ldap_profile_subsystem(ca) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_enable_ldap_profile_subsystem cainstance.migrate_profiles_to_ldap() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1814, in migrate_profiles_to_ldap _create_dogtag_profile(profile_id, profile_data, overwrite=False) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1820, in _create_dogtag_profile with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1302, in __enter__ raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
ipapython.admintool: DEBUG: The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API ipapython.admintool: ERROR: Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API ipapython.admintool: ERROR: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Unfortunately, we couldn't find anything useful in the pki-tomcat log. Of course you see the 404:
[root@ipa2 pki-tomcat]# tail catalina.2019-02-12.log Feb 12, 2019 11:48:16 AM com.netscape.cms.tomcat.PKIListener lifecycleEvent INFO: PKIListener: org.apache.catalina.core.StandardServer [after_start] Feb 12, 2019 11:48:16 AM com.netscape.cms.tomcat.PKIListener verifySubsystems INFO: PKIListener: Subsystem CA is running. Feb 12, 2019 11:48:16 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 10810 ms Feb 12, 2019 11:48:51 AM com.netscape.cms.tomcat.AbstractPKIAuthenticator doAuthenticate INFO: PKIAuthenticator: Authenticate with client certificate authentication Feb 12, 2019 11:48:51 AM com.netscape.cms.tomcat.AbstractPKIAuthenticator doAuthenticate INFO: PKIAuthenticator: Result: false
[root@ipa2 pki-tomcat]# tail localhost_access_log.2019-02-12.txt IP - - [12/Feb/2019:11:37:14 +0100] "GET /ca/rest/account/login HTTP/1.1" 401 951 IP - - [12/Feb/2019:11:39:48 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:39:48 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:40:20 +0100] "GET /ca/rest/account/login HTTP/1.1" 401 951 IP - - [12/Feb/2019:11:45:21 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:46:04 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:47:58 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:48:16 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:48:17 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:48:51 +0100] "GET /ca/rest/account/login HTTP/1.1" 401 951
Any ideas would be (again) much appreciated!
Thanks a lot
Torsten
Don't know if this is related, but I found somewhere that this should work, but it does not:
[root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]# ipa-pkinit-manage enable Configuring Kerberos KDC (krb5kdc) [1/1]: installing X509 Certificate for PKINIT PKINIT certificate request failed: Certificate issuance failed (CA_REJECTED: Server at "https://ipa2.pleiades.uni-wuppertal.de:8443/ca/ee/ca/profileSubmitSSLClient" replied: Profile KDCs_PKINIT_Certs Not Found) Failed to configure PKINIT Full PKINIT configuration did not succeed The setup will only install bits essential to the server functionality You can enable PKINIT after the setup completed using 'ipa-pkinit-manage' Done configuring Kerberos KDC (krb5kdc). The ipa-pkinit-manage command was successful [root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]# ipa-pkinit-manage status PKINIT is disabled The ipa-pkinit-manage command was successful [root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]#
[root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]# ipa config-show|grep -i pkinit IPA master capable of PKINIT: ipa2.pleiades.uni-wuppertal.de
but
[root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]# ipa pkinit-status ----------------- 2 servers matched ----------------- Servername: ipa.pleiades.uni-wuppertal.de PKINIT status: disabled
Servername: ipa2.pleiades.uni-wuppertal.de PKINIT status: enabled ------------------------------------- Anzahl der zurückgegebenen Einträge 2 ------------------------------------- [root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]#
(which is probably ok, since ipa is still on 4.4)
Hope someone has an idea.. I googled for hours already but couldn't find anything that helps.
Kind regards
Torsten
On 2/12/19 3:01 PM, Torsten Harenberg via FreeIPA-users wrote:
Don't know if this is related, but I found somewhere that this should work, but it does not:
[root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]# ipa-pkinit-manage enable Configuring Kerberos KDC (krb5kdc) [1/1]: installing X509 Certificate for PKINIT PKINIT certificate request failed: Certificate issuance failed (CA_REJECTED: Server at "https://ipa2.pleiades.uni-wuppertal.de:8443/ca/ee/ca/profileSubmitSSLClient" replied: Profile KDCs_PKINIT_Certs Not Found) Failed to configure PKINIT Full PKINIT configuration did not succeed The setup will only install bits essential to the server functionality You can enable PKINIT after the setup completed using 'ipa-pkinit-manage' Done configuring Kerberos KDC (krb5kdc). The ipa-pkinit-manage command was successful [root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]# ipa-pkinit-manage status PKINIT is disabled The ipa-pkinit-manage command was successful [root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]#
[root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]# ipa config-show|grep -i pkinit IPA master capable of PKINIT: ipa2.pleiades.uni-wuppertal.de
but
[root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]# ipa pkinit-status
2 servers matched
Servername: ipa.pleiades.uni-wuppertal.de PKINIT status: disabled
Servername: ipa2.pleiades.uni-wuppertal.de PKINIT status: enabled
Anzahl der zurückgegebenen Einträge 2
[root@ipa2 slapd-PLEIADES-UNI-WUPPERTAL-DE]#
(which is probably ok, since ipa is still on 4.4)
Hope someone has an idea.. I googled for hours already but couldn't find anything that helps.
Kind regards
Torsten
Hi,
there are a few things that you can check. The error "Failed to authenticate to CA REST API" points to authentication issue between IPA framework and Dogtag, which is done with a certificate.
Are there expired certificates on the server where the upgrade failed? I would check more specifically for the RA cert that is stored in /etc/httpd/alias (nickname=ipaCert) in IPA < 4.5 and in /var/lib/ipa/ra-agent.pem in IPA 4.5+.
You can check its content with $ certutil -L -d /etc/httpd/alias -n ipaCert or $ openssl x509 -noout -text -in /var/lib/ipa/ra-agent.pem
The same cert must be present in uid=ipara,ou=people,o=ipaca. $ ldapsearch -D "cn=directory manager" -W -b o=ipaca -LLL -o ldif-wrap=no "(uid=ipara)" usercertificate description Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca usercertificate:: MIIDvDC...jyi5w description: 2;7;CN=Certificate Authority,O=DOMAIN.COM;CN=IPA RA,O=DOMAIN.COM
The usercertificate field must contain the ipaCert certificate and the description field must contain 2;<serial>;<issuer>;<subject>.
Can you check if the cert is consistent in the NSS database/pem file and in LDAP?
HTH, flo
Hi Flo,
thanks (again!!!!) for your kind support. Really appreciate it. You found again the source of the problem and we were able to solve it (see buttom):
Am 12.02.19 um 18:22 schrieb Florence Blanc-Renaud:
Hi,
there are a few things that you can check. The error "Failed to authenticate to CA REST API" points to authentication issue between IPA framework and Dogtag, which is done with a certificate.
Are there expired certificates on the server where the upgrade failed?
[root@ipa2 ~]# getcert list | grep expires expires: 2021-01-14 08:53:15 UTC expires: 2021-01-14 08:51:44 UTC expires: 2021-01-14 08:55:41 UTC expires: 2039-01-25 08:52:13 UTC expires: 2021-01-14 08:51:08 UTC expires: 2020-06-28 03:05:27 UTC expires: 2020-07-20 03:05:47 UTC expires: 2020-07-20 03:05:30 UTC [root@ipa2 ~]#
--> Check!
I would check more specifically for the RA cert that is stored in /etc/httpd/alias (nickname=ipaCert) in IPA < 4.5 and in /var/lib/ipa/ra-agent.pem in IPA 4.5+.
Request ID '20170526083117': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PLEIADES.UNI-WUPPERTAL.DE subject: CN=IPA RA,O=PLEIADES.UNI-WUPPERTAL.DE expires: 2021-01-14 08:51:08 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
--> check!
You can check its content with $ certutil -L -d /etc/httpd/alias -n ipaCert or $ openssl x509 -noout -text -in /var/lib/ipa/ra-agent.pem
root@ipa2 ~]# certutil -L -d /etc/httpd/alias -n ipaCert Certificate: Data: Version: 3 (0x2) Serial Number: 268173514 (0xffc00ca) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=PLEIADES.UNI-WUPPERTAL.DE" Validity: Not Before: Fri Jan 25 08:51:08 2019 Not After : Thu Jan 14 08:51:08 2021 Subject: "CN=IPA RA,O=PLEIADES.UNI-WUPPERTAL.DE"
The same cert must be present in uid=ipara,ou=people,o=ipaca. $ ldapsearch -D "cn=directory manager" -W -b o=ipaca -LLL -o ldif-wrap=no "(uid=ipara)" usercertificate description Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca usercertificate:: MIIDvDC...jyi5w description: 2;7;CN=Certificate Authority,O=DOMAIN.COM;CN=IPA RA,O=DOMAIN.COM
The usercertificate field must contain the ipaCert certificate and the description field must contain 2;<serial>;<issuer>;<subject>.
[root@ipa2 ~]# ldapsearch -D "cn=directory manager" -W -b o=ipaca -LLL -o ldif-wrap=no "(uid=ipara)" usercertificate description Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca
[...]
description: 2;1342111780;CN=Certificate Authority,O=PLEIADES.UNI-WUPPERTAL.DE;CN=IPA RA,O=PLEIADES.UNI-WUPPERTAL.DE
And here we see a different serial compared to the one above.
We changed the certificates accordingly and voilà: update ran through like a charm. :-)
THanks again
Torsten
Torsten Harenberg via FreeIPA-users wrote:
Dear all,
we are trying to upgrade our IPA 4.4 to current 4.6.
So we did a "yum update" and then a "ipa-server-upgrade" which miserably failes with:
ipaserver.install.server.upgrade: INFO: [Migrating certificate profiles to LDAP] ipalib.backend: DEBUG: Created connection context.ldap2_140564957536912 ipapython.ipaldap: DEBUG: flushing ldapi://%2fvar%2frun%2fslapd-PLEIADES-UNI-WUPPERTAL-DE.socket from SchemaCache ipapython.ipaldap: DEBUG: retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-PLEIADES-UNI-WUPPERTAL-DE.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fd7d5efe7a0> ipalib.backend: DEBUG: Destroyed connection context.ldap2_140564957536912 ipapython.dogtag: DEBUG: request GET https://ipa2.pleiades.uni-wuppertal.de:8443/ca/rest/account/login ipapython.dogtag: DEBUG: request body '' ipapython.dogtag: DEBUG: response status 401 ipapython.dogtag: DEBUG: response headers Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 01:00:00 CET WWW-Authenticate: Basic realm="Certificate Authority" Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 951 Date: Tue, 12 Feb 2019 10:48:51 GMT
Makes me wonder if you have one or more expired certs.
Run getcert list | grep -i expires as root to see.
rob
ipapython.dogtag: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 401 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>This request requires HTTP authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' ipaserver.install.ipa_server_upgrade: ERROR: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. ipapython.admintool: DEBUG: File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2085, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1952, in upgrade_configuration ca_enable_ldap_profile_subsystem(ca) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_enable_ldap_profile_subsystem cainstance.migrate_profiles_to_ldap() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1814, in migrate_profiles_to_ldap _create_dogtag_profile(profile_id, profile_data, overwrite=False) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1820, in _create_dogtag_profile with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1302, in __enter__ raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
ipapython.admintool: DEBUG: The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API ipapython.admintool: ERROR: Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API ipapython.admintool: ERROR: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Unfortunately, we couldn't find anything useful in the pki-tomcat log. Of course you see the 404:
[root@ipa2 pki-tomcat]# tail catalina.2019-02-12.log Feb 12, 2019 11:48:16 AM com.netscape.cms.tomcat.PKIListener lifecycleEvent INFO: PKIListener: org.apache.catalina.core.StandardServer [after_start] Feb 12, 2019 11:48:16 AM com.netscape.cms.tomcat.PKIListener verifySubsystems INFO: PKIListener: Subsystem CA is running. Feb 12, 2019 11:48:16 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 10810 ms Feb 12, 2019 11:48:51 AM com.netscape.cms.tomcat.AbstractPKIAuthenticator doAuthenticate INFO: PKIAuthenticator: Authenticate with client certificate authentication Feb 12, 2019 11:48:51 AM com.netscape.cms.tomcat.AbstractPKIAuthenticator doAuthenticate INFO: PKIAuthenticator: Result: false
[root@ipa2 pki-tomcat]# tail localhost_access_log.2019-02-12.txt IP - - [12/Feb/2019:11:37:14 +0100] "GET /ca/rest/account/login HTTP/1.1" 401 951 IP - - [12/Feb/2019:11:39:48 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:39:48 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:40:20 +0100] "GET /ca/rest/account/login HTTP/1.1" 401 951 IP - - [12/Feb/2019:11:45:21 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:46:04 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:47:58 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:48:16 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:48:17 +0100] "POST /ca/admin/ca/getStatus HTTP/1.1" 200 167 IP - - [12/Feb/2019:11:48:51 +0100] "GET /ca/rest/account/login HTTP/1.1" 401 951
Any ideas would be (again) much appreciated!
Thanks a lot
Torsten
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Rob Crittenden -
Torsten Harenberg