yum update problem - FreeIPA-users - Fedora mailing-lists

20 Nov 2019


      hi,
after patching our centos 7 hosts to the latest version today, one of the
two replicas is having trouble.
[root@kdc2 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: STOPPED
kadmin Service: STOPPED
named Service: STOPPED
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: STOPPED
pki-tomcatd Service: RUNNING
smb Service: STOPPED
winbind Service: STOPPED
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful
and after digging in the logs I come across this in /var/log/ipaupgrade.log:
2019-11-20T18:18:29Z DEBUG stderr=
2019-11-20T18:18:31Z INFO Certmonger certificate renewal configuration
already up-to-date
2019-11-20T18:18:31Z INFO [Enable PKIX certificate path discovery and
validation]
2019-11-20T18:18:31Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-11-20T18:18:31Z INFO PKIX already enabled
2019-11-20T18:18:31Z INFO [Authorizing RA Agent to modify profiles]
2019-11-20T18:18:31Z INFO [Authorizing RA Agent to manage lightweight CAs]
2019-11-20T18:18:31Z INFO [Ensuring Lightweight CAs container exists in
Dogtag database]
2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_139740162547472
2019-11-20T18:18:31Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache
2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17cc24b638>
2019-11-20T18:18:31Z DEBUG Destroyed connection
context.ldap2_139740162547472
2019-11-20T18:18:31Z INFO [Adding default OCSP URI configuration]
2019-11-20T18:18:31Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2019-11-20T18:18:31Z INFO [Migrating certificate profiles to LDAP]
2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_139740160021648
2019-11-20T18:18:31Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache
2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17cc289b00>
2019-11-20T18:18:31Z DEBUG Destroyed connection
context.ldap2_139740160021648
2019-11-20T18:18:31Z DEBUG request GET
https://kdc2.l.domain.it:8443/ca/rest/account/login
2019-11-20T18:18:31Z DEBUG request body ''
2019-11-20T18:18:31Z DEBUG response status 401
2019-11-20T18:18:31Z DEBUG response headers Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 01:00:00 CET
WWW-Authenticate: Basic realm="Certificate Authority"
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 951
Date: Wed, 20 Nov 2019 18:18:31 GMT
2019-11-20T18:18:31Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 401 - </h1><HR size="1"
noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
<u></u></p><p><b>description</b> <u>This request requires HTTP
authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-11-20T18:18:31Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2019-11-20T18:18:31Z DEBUG   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
execute
    return_value = self.run()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run
    server.upgrade()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2146, in upgrade
    upgrade_configuration()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2018, in upgrade_configuration
    ca_enable_ldap_profile_subsystem(ca)
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 406, in ca_enable_ldap_profile_subsystem
    cainstance.migrate_profiles_to_ldap()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
line 2027, in migrate_profiles_to_ldap
    _create_dogtag_profile(profile_id, profile_data, overwrite=False)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
line 2033, in _create_dogtag_profile
    with api.Backend.ra_certprofile as profile_api:
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line
1315, in __enter__
    raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA
REST API'))
2019-11-20T18:18:31Z DEBUG The ipa-server-upgrade command failed,
exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2019-11-20T18:18:31Z ERROR Unexpected error - see /var/log/ipaupgrade.log
for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
In this kdc I see these errors in getcert list:
Request ID '20190220182014':
        status: MONITORING
        ca-error: Invalid cookie: u''
        stuck: no
        key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=L.DOMAIN.IT
        subject: CN=CA Audit,O=L.DOMAIN.IT
        expires: 2019-12-05 13:58:24 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190220182015':
        status: MONITORING
        ca-error: Invalid cookie: u''
        stuck: no
        key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=L.DOMAIN.IT
        subject: CN=OCSP Subsystem,O=L.DOMAIN.IT
        expires: 2019-12-05 13:58:24 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        eku: id-kp-OCSPSigning
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190220182016':
        status: MONITORING
        ca-error: Invalid cookie: u''
        stuck: no
        key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=L.DOMAIN.IT
        subject: CN=CA Subsystem,O=L.DOMAIN.IT
        expires: 2019-12-05 13:58:24 UTC
        key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190220182018':
        status: MONITORING
        ca-error: Invalid cookie: u''
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=L.DOMAIN.IT
        subject: CN=IPA RA,O=L.DOMAIN.IT
        expires: 2019-12-05 13:58:44 UTC
        key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20190220182019':
        status: MONITORING
        ca-error: Server at "
https://kdc2.l.domain.it:8443/ca/agent/ca/profileProcess" replied: 1:
Invalid Credential.
        stuck: no
        key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=L.DOMAIN.IT
        subject: CN=kdc2.l.domain.it,O=L.DOMAIN.IT
        expires: 2019-12-10 10:57:52 UTC
        key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes
I still have a working replica, so I could just reinstall and have a
working set in a couple of minutes, but I would like to find out what has
gone wrong.
The systems are running ipa-server-4.6.5-11.el7.centos.3.x86_64
Any help welcome ;-)
Thanks,
--
Groeten,
natxo