On 6/30/20 10:24 AM, Mariusz Stolarczyk via FreeIPA-users wrote:
All,
I did a routine server updates last night on my IPA server. After the reboot I first noticed the DNS was not resolving and the ipa.service failed. The ipa.service failed to start so I ran the following:
# ipactl start IPA version error: data needs to be upgraded (expected version '4.6.6-11.el7.centos', current version '4.6.5-11.el7.centos.4') Automatically running upgrade, for details see /var/log/ipaupgrade.log Be patient, this may take a few minutes. Automatic upgrade failed: Update complete Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved [Verifying that CA proxy configuration is correct] [Verifying that KDC configuration is using ipa-kdb backend] [Fix DS schema file syntax] Syntax already fixed [Removing RA cert from DS NSS database] RA cert already removed [Enable sidgen and extdom plugins by default] [Updating HTTPD service IPA configuration] [Updating HTTPD service IPA WSGI configuration] Nothing to do for configure_httpd_wsgi_conf [Updating mod_nss protocol versions] Protocol versions already updated [Updating mod_nss cipher suite] [Updating mod_nss enabling OCSP] [Fixing trust flags in /etc/httpd/alias] Trust flags already processed [Moving HTTPD service keytab to gssproxy] [Removing self-signed CA] [Removing Dogtag 9 CA] [Checking for deprecated KDC configuration files] [Checking for deprecated backups of Samba configuration files] [Remove FILE: prefix from 'dedicated keytab file' in Samba configuration] [Update 'max smbd processes' in Samba configuration to prevent unlimited SMBLoris attack amplification] [Add missing CA DNS records] IPA CA DNS records already processed [Removing deprecated DNS configuration options] [Ensuring minimal number of connections] [Updating GSSAPI configuration in DNS] [Updating pid-file configuration in DNS] [Checking global forwarding policy in named.conf to avoid conflicts with automatic empty zones] Changes to named.conf have been made, restart named [Upgrading CA schema] CA schema update complete (no changes) [Verifying that CA audit signing cert has 2 year validity] [Update certmonger certificate renewal configuration] Certmonger certificate renewal configuration already up-to-date [Enable PKIX certificate path discovery and validation] PKIX already enabled [Authorizing RA Agent to modify profiles] [Authorizing RA Agent to manage lightweight CAs] [Ensuring Lightweight CAs container exists in Dogtag database] [Adding default OCSP URI configuration] [Ensuring CA is using LDAPProfileSubsystem] [Migrating certificate profiles to LDAP] IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. Unexpected error - see /var/log/ipaupgrade.log for details: NetworkError: cannot connect to 'https://FAKE-IPA-HOST.FAKE-IPA-DOMAIN.LAN:8443/ca/rest/account/login': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
See the upgrade log for more details and/or run /usr/sbin/ipa-server-upgrade again Aborting ipactl
The end of the /var/log/ipaupgrade.log file:
2020-06-29T22:43:38Z DEBUG stderr= 2020-06-29T22:43:38Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2020-06-29T22:43:38Z DEBUG Starting external process 2020-06-29T22:43:38Z DEBUG args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt 2020-06-29T22:43:38Z DEBUG Process finished, return code=0 2020-06-29T22:43:38Z DEBUG stdout= Certificate Nickname                     Trust Attributes
 SSL,S/MIME,JAR/XPI
caSigningCert cert-pki-ca                   CTu,Cu,Cu subsystemCert cert-pki-ca                   u,u,u Server-Cert cert-pki-ca                    u,u,u ocspSigningCert cert-pki-ca                  u,u,u auditSigningCert cert-pki-ca                 u,u,Pu
2020-06-29T22:43:38Z DEBUG stderr= 2020-06-29T22:43:38Z INFO Certmonger certificate renewal configuration already up-to-date 2020-06-29T22:43:38Z INFO [Enable PKIX certificate path discovery and validation] 2020-06-29T22:43:38Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2020-06-29T22:43:38Z INFO PKIX already enabled 2020-06-29T22:43:38Z INFO [Authorizing RA Agent to modify profiles] 2020-06-29T22:43:38Z INFO [Authorizing RA Agent to manage lightweight CAs] 2020-06-29T22:43:38Z INFO [Ensuring Lightweight CAs container exists in Dogtag database] 2020-06-29T22:43:38Z DEBUG Created connection context.ldap2_140346851657552 2020-06-29T22:43:38Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-SEQUOIARC-LAN.socket from SchemaCache 2020-06-29T22:43:38Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-SEQUOIARC-LAN.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fa50c3e8e60> 2020-06-29T22:43:39Z DEBUG Destroyed connection context.ldap2_140346851657552 2020-06-29T22:43:39Z INFO [Adding default OCSP URI configuration] 2020-06-29T22:43:39Z INFO [Ensuring CA is using LDAPProfileSubsystem] 2020-06-29T22:43:39Z INFO [Migrating certificate profiles to LDAP] 2020-06-29T22:43:39Z DEBUG Created connection context.ldap2_140346825804304 2020-06-29T22:43:39Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-SEQUOIARC-LAN.socket from SchemaCache 2020-06-29T22:43:39Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-SEQUOIARC-LAN.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fa50ac19b90> 2020-06-29T22:43:39Z DEBUG Destroyed connection context.ldap2_140346825804304 2020-06-29T22:43:39Z DEBUG request GET https://FAKE-IPA-HOST.FAKE-IPA-DOMAIN.LAN:8443/ca/rest/account/login 2020-06-29T22:43:39Z DEBUG request body '' 2020-06-29T22:43:39Z DEBUG httplib request failed: Traceback (most recent call last):  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 220, in _httplib_request   conn.request(method, path, body=request_body, headers=headers)  File "/usr/lib64/python2.7/httplib.py", line 1056, in request   self._send_request(method, url, body, headers)  File "/usr/lib64/python2.7/httplib.py", line 1090, in _send_request   self.endheaders(body)  File "/usr/lib64/python2.7/httplib.py", line 1052, in endheaders   self._send_output(message_body)  File "/usr/lib64/python2.7/httplib.py", line 890, in _send_output   self.send(msg)  File "/usr/lib64/python2.7/httplib.py", line 852, in send   self.connect()  File "/usr/lib64/python2.7/httplib.py", line 1275, in connect   server_hostname=sni_hostname)  File "/usr/lib64/python2.7/ssl.py", line 348, in wrap_socket   _context=self)  File "/usr/lib64/python2.7/ssl.py", line 609, in __init__   self.do_handshake()  File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake   self._sslobj.do_handshake() SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) 2020-06-29T22:43:39Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2020-06-29T22:43:39Z DEBUG  File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute   return_value = self.run()  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run   server.upgrade()  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2166, in upgrade   upgrade_configuration()  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2038, in upgrade_configuration   ca_enable_ldap_profile_subsystem(ca)  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 425, in ca_enable_ldap_profile_subsystem   cainstance.migrate_profiles_to_ldap()  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2027, in migrate_profiles_to_ldap   _create_dogtag_profile(profile_id, profile_data, overwrite=False)  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2033, in _create_dogtag_profile   with api.Backend.ra_certprofile as profile_api:  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1311, in __enter__   method='GET'  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 167, in https_request   method=method, headers=headers)  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 229, in _httplib_request   raise NetworkError(uri=uri, error=str(e))
2020-06-29T22:43:39Z DEBUG The ipa-server-upgrade command failed, exception: NetworkError: cannot connect to 'https://FAKE-IPA-HOST.FAKE-IPA-DOMAIN.LAN:8443/ca/rest/account/login': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) 2020-06-29T22:43:39Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: NetworkError: cannot connect to 'https://FAKE-IPA-HOST.FAKE-IPA-DOMAIN.LAN:8443/ca/rest/account/login': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) 2020-06-29T22:43:39Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
What should be my next debug steps?
Hi,
I would check whether any certificate expired: $ getcert list
Look specifically for the "status: " and "expires: " labels. If some certs have expired, you will need to find the CA renewal master and fix this host first. To find the CA renewal master: $ kinit admin $ ipa config-show | grep "CA renewal"
If you need help, please mention: - the output of "ipa server-role-find" - the output of "getcert list" on all the server nodes - are the httpd and ldap server certificates issued by IPA CA or by an external Certificate Authority?
HTH, flo
Thanks in advance, -ms
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...