On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:

Hi Florence,

On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:

...

On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:

...

my IPA system consists of 2 masters with their own self-signed CAs, one of  them being the certificate renewal master (ipa1). The system has been  running for years and has been migrated from an IPA 3 system.

Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed  due to an unknown reason.").

Web UI logins on the other server (ipa2) work and everything else is  working fine, too, ipactl status reports all services running.

On login attempt:

--- httpd log  [...]  [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception  occurred processing WSGI script '/usr/share/ipa/wsgi.py'.  [...]  [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command  '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X  X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X  X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned  non-zero exit status 1  ---

--- krb5kdc.log  [...]  Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes  {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:  WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM,  Additional pre-authentication required  Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11  Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes  {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:  WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Failed  to verify own certificate (depth 0): certificate has expired  Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11  ---

--- ipa-checkcerts.py  IPA version 4.5.4-10.el7.centos.3  Check CA status  Check tracking  Check NSS trust  Check dates  Checking certificates in CS.cfg  Comparing certificates to requests in LDAP  Checking RA certificate  Checking authorities  Checking host keytab  Validating certificates  Checking renewal master  End-to-end cert API test  Checking permissions and ownership  Failures:  Unable to find request for serial 268304391  Unable to find request for serial 268304394  Unable to find request for serial 268304393  Unable to find request for serial 268304392  Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject  CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57  ---

--- ipa pkinit-status --all  -----------------  2 servers matched  -----------------     Server name: ipa2.example.com     PKINIT status: enabled

Server name: ipa1.example.com     PKINIT status: enabled  ----------------------------  Number of entries returned 2  ----------------------------

To my understanding, proper certificate exchange between my two servers  ceased working at some point. How do i track this down and fix it?

your issue looks similar to ticket #6792 [1]. Can you check the result of upgrade in /var/log/ipaupgrade.log? Also check the output of $ ipa-pkinit-manage status and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.

Regarding the certificates, does getcert list show expired certs? flo

[1] https://pagure.io/freeipa/issue/6792

---

$ ipa-pkinit-manage status PKINIT is enabled

---

There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem exist with proper permissions, but I found something in ipaupgrade.log:

---

2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= Certificate Nickname                                         Trust Attributes

SSL,S/MIME,JAR/XPI

Server-Cert                                                  u,u,u EXAMPLE.COM IPA CA                                           CT,C,C

2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----

2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=255 2018-09-12T13:37:19Z DEBUG stdout= 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert : PR_FILE_NOT_FOUND_ERROR: File not found [...]

---

Hi, this error can be ignored in most of the cases. The upgrade is trying to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the upgrade is run a second time, he won't find ipaCert in the NSS database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group, and must contain the same cert as the other masters.

What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and ca-bundle.pem? both must contain IPA CA certificate.

What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be readable by everyone. And what is the content of this cert? It should be issued by IPA CA.

About the errors spotted by ipa-checkcerts.py, what are the certificates with errors reported? You can find them with ipa cert-show <serial>.

flo

Mit freundlichen Gruessen/With best regards,

--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:

Hi Florence,

thank you very much for your help.

On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:

...

On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:

...

On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:

...

On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:

...

my IPA system consists of 2 masters with their own self-signed CAs,  one of   them being the certificate renewal master (ipa1). The system has been   running for years and has been migrated from an IPA 3 system.

Since a while, the Web UI logins on ipa1 don't work anymore ("Login  failed   due to an unknown reason.").

Web UI logins on the other server (ipa2) work and everything else is   working fine, too, ipactl status reports all services running.

On login attempt:

--- httpd log   [...]   [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):  Exception   occurred processing WSGI script '/usr/share/ipa/wsgi.py'.   [...]   [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command   '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X   X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X   X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned   non-zero exit status 1   ---

--- krb5kdc.log   [...]   Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes   {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:   WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM,   Additional pre-authentication required   Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd  11   Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes   {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:   WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM,  Failed   to verify own certificate (depth 0): certificate has expired   Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd  11   ---

--- ipa-checkcerts.py   IPA version 4.5.4-10.el7.centos.3   Check CA status   Check tracking   Check NSS trust   Check dates   Checking certificates in CS.cfg   Comparing certificates to requests in LDAP   Checking RA certificate   Checking authorities   Checking host keytab   Validating certificates   Checking renewal master   End-to-end cert API test   Checking permissions and ownership   Failures:   Unable to find request for serial 268304391   Unable to find request for serial 268304394   Unable to find request for serial 268304393   Unable to find request for serial 268304392   Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject   CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57   ---

--- ipa pkinit-status --all   -----------------   2 servers matched   -----------------      Server name: ipa2.example.com      PKINIT status: enabled

Server name: ipa1.example.com      PKINIT status: enabled   ----------------------------   Number of entries returned 2   ----------------------------

To my understanding, proper certificate exchange between my two servers   ceased working at some point. How do i track this down and fix it?

your issue looks similar to ticket #6792 [1]. Can you check the result of  upgrade in /var/log/ipaupgrade.log?  Also check the output of  $ ipa-pkinit-manage status  and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and  /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.

Regarding the certificates, does getcert list show expired certs?  flo

[1] https://pagure.io/freeipa/issue/6792

---  $ ipa-pkinit-manage status  PKINIT is enabled  ---

There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem  exist with proper permissions, but I found something in ipaupgrade.log:

---  2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L  -f /etc/httpd/alias/pwdfile.txt  2018-09-12T13:37:19Z DEBUG Process finished, return code=0  2018-09-12T13:37:19Z DEBUG stdout=  Certificate Nickname                                         Trust  Attributes

SSL,S/MIME,JAR/XPI

Server-Cert                                                  u,u,u  EXAMPLE.COM IPA CA                                           CT,C,C

2018-09-12T13:37:19Z DEBUG stderr=  2018-09-12T13:37:19Z DEBUG Starting external process  2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L  -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt  2018-09-12T13:37:19Z DEBUG Process finished, return code=0  2018-09-12T13:37:19Z DEBUG stdout=  -----BEGIN CERTIFICATE-----  [...]  -----END CERTIFICATE-----

2018-09-12T13:37:19Z DEBUG stderr=  2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store  2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store  2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')  2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')  2018-09-12T13:37:19Z DEBUG Starting external process  2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L  -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt  2018-09-12T13:37:19Z DEBUG Process finished, return code=255  2018-09-12T13:37:19Z DEBUG stdout=  2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert :  PR_FILE_NOT_FOUND_ERROR: File not found  [...]  ---

this error can be ignored in most of the cases. The upgrade is trying to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the upgrade is run a second time, he won't find ipaCert in the NSS database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group, and must contain the same cert as the other masters.

checked this, is ok.

...

What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and ca-bundle.pem? both must contain IPA CA certificate.

True on both servers.

...

What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be readable by everyone. And what is the content of this cert? It should be issued by IPA CA.

Permissions are ok, contents:

--- ipa1 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate:     Data:         Version: 3 (0x2)         Serial Number: 1 (0x1)     Signature Algorithm: sha256WithRSAEncryption         Issuer: O=EXAMPLE.COM, CN=ipa1.example.com         Validity             Not Before: Nov 28 12:43:05 2017 GMT             Not After : Nov 28 12:43:05 2018 GMT         Subject: O=EXAMPLE.COM, CN=ipa1.example.com         Subject Public Key Info:             Public Key Algorithm: rsaEncryption                 Public-Key: (2048 bit)                 Modulus:                     [...]                 Exponent: 65537 (0x10001)         X509v3 extensions:             X509v3 Subject Alternative Name:                 othername:<unsupported>, othername:<unsupported>             X509v3 Basic Constraints: critical                 CA:FALSE             X509v3 Subject Key Identifier:

86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE             1.3.6.1.4.1.311.20.2:                 .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s [...]

---

--- ipa2 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate:     Data:         Version: 3 (0x2)         Serial Number: 805240833 (0x2fff0001)     Signature Algorithm: sha256WithRSAEncryption         Issuer: O=EXAMPLE.COM, CN=Certificate Authority         Validity             Not Before: Jan 18 13:04:17 2018 GMT             Not After : Jan 19 13:04:17 2020 GMT         Subject: O=EXAMPLE.COM, CN=ipa2.example.com         Subject Public Key Info:             Public Key Algorithm: rsaEncryption                 Public-Key: (2048 bit)                 Modulus:                     [...]                 Exponent: 65537 (0x10001)         X509v3 extensions:             X509v3 Authority Key Identifier:

keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB

Authority Information Access:                 OCSP - URI:http://ipa-ca.example.com/ca/ocsp

X509v3 Key Usage: critical                 Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment             X509v3 Extended Key Usage:                 TLS Web Server Authentication, 1.3.6.1.5.2.3.5             X509v3 CRL Distribution Points:

Full Name:                   URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin                 CRL Issuer:                   DirName: O = ipaca, CN = Certificate Authority

X509v3 Subject Key Identifier:

96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3             X509v3 Subject Alternative Name:                 othername:<unsupported>, othername:<unsupported> [...]

---

Additional info: I have DNS separate from IPA, but i (hopefully) made proper records as IPA would have done it. In particular, i made an A record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the root cause of my problems, since DNS is not under my control.

Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by IPA CA, might this be the actual problem?

The cert is expired on ipa1, which is the real root cause. ipa-pkinit-manage status is reporting that PKINIT is enabled but this does not match the actual configuration. This is a known issue [1].

If the host ipa1 is running a CA instance, then you can delete /var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to re-generate the KDC cert. Note: if the host doesn't run a CA instance, then this won't work because of the issue [2].

To check which hosts are running a CA instance, you can use # ipa server-role-find --role 'CA server'

[1] https://pagure.io/freeipa/issue/7200 [2] https://pagure.io/freeipa/issue/7795

...

About the errors spotted by ipa-checkcerts.py, what are the certificates with errors reported? You can find them with ipa cert-show <serial>.

---

$ipa cert-show 57   Issuing CA: ipa   Certificate: [...]   Subject: CN=ipa1.example.com,O=EXAMPLE.COM   Issuer: CN=Certificate Authority,O=EXAMPLE.COM   Not Before: Tue Nov 28 12:42:07 2017 UTC   Not After: Mon Nov 18 12:42:07 2019 UTC   Serial number: 57   Serial number (hex): 0x39   Revoked: False

$ ipa cert-show 268304391   Issuing CA: ipa   Certificate: [...]   Subject: CN=IPA RA,O=EXAMPLE.COM   Issuer: CN=Certificate Authority,O=EXAMPLE.COM   Not Before: Wed Jul 11 14:26:35 2018 UTC   Not After: Tue Jun 30 14:26:35 2020 UTC   Serial number: 268304391   Serial number (hex): 0xFFE0007   Revoked: False

$ ipa cert-show 268304392   Issuing CA: ipa   Certificate: [...]   Subject: CN=CA Subsystem,O=EXAMPLE.COM   Issuer: CN=Certificate Authority,O=EXAMPLE.COM   Not Before: Wed Jul 11 14:26:45 2018 UTC   Not After: Tue Jun 30 14:26:45 2020 UTC   Serial number: 268304392   Serial number (hex): 0xFFE0008   Revoked: False

$ ipa cert-show 268304393   Issuing CA: ipa   Certificate: [...]   Subject: CN=OCSP Subsystem,O=EXAMPLE.COM   Issuer: CN=Certificate Authority,O=EXAMPLE.COM   Not Before: Wed Jul 11 14:27:05 2018 UTC   Not After: Tue Jun 30 14:27:05 2020 UTC   Serial number: 268304393   Serial number (hex): 0xFFE0009   Revoked: False

$ ipa cert-show 268304394   Issuing CA: ipa   Certificate: [...]   Subject: CN=CA Audit,O=EXAMPLE.COM   Issuer: CN=Certificate Authority,O=EXAMPLE.COM   Not Before: Wed Jul 11 14:27:25 2018 UTC   Not After: Tue Jun 30 14:27:25 2020 UTC   Serial number: 268304394   Serial number (hex): 0xFFE000A   Revoked: False

---

ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)" btw.

The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?

"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).

HTH, flo

Mit freundlichen Gruessen/With best regards,

--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

On 12/28/18 1:03 PM, dbischof--- via FreeIPA-users wrote:

Hi Florence,

On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:

...

On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:

...

thank you very much for your help.

On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:

...

On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:

...

On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:

...

On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote: >    my IPA system consists of 2 masters with their own self-signed > CAs, >   one of >    them being the certificate renewal master (ipa1). The system has >  been >    running for years and has been migrated from an IPA 3 system. > >    Since a while, the Web UI logins on ipa1 don't work anymore > ("Login >   failed >    due to an unknown reason."). > >    Web UI logins on the other server (ipa2) work and everything > else is >    working fine, too, ipactl status reports all services running. > >    On login attempt: > >    --- httpd log >    [...] >    [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): >   Exception >    occurred processing WSGI script '/usr/share/ipa/wsgi.py'. >    [...] >    [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: >  Command >    '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X >    X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X >    X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' >  returned >    non-zero exit status 1 >    --- > >    --- krb5kdc.log >    [...] >    Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 >  etypes >    {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: >    WELLKNOWN/ANONYMOUS@EXAMPLE.COM for > krbtgt/EXAMPLE.COM@EXAMPLE.COM, >    Additional pre-authentication required >    Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing > down >  fd >   11 >    Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 >  etypes >    {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: >    WELLKNOWN/ANONYMOUS@EXAMPLE.COM for > krbtgt/EXAMPLE.COM@EXAMPLE.COM, >   Failed >    to verify own certificate (depth 0): certificate has expired >    Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing > down >  fd >   11 >    --- > >    --- ipa-checkcerts.py >    IPA version 4.5.4-10.el7.centos.3 >    Check CA status >    Check tracking >    Check NSS trust >    Check dates >    Checking certificates in CS.cfg >    Comparing certificates to requests in LDAP >    Checking RA certificate >    Checking authorities >    Checking host keytab >    Validating certificates >    Checking renewal master >    End-to-end cert API test >    Checking permissions and ownership >    Failures: >    Unable to find request for serial 268304391 >    Unable to find request for serial 268304394 >    Unable to find request for serial 268304393 >    Unable to find request for serial 268304392 >    Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject >    CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57 >    --- > >    --- ipa pkinit-status --all >    ----------------- >    2 servers matched >    ----------------- >       Server name: ipa2.example.com >       PKINIT status: enabled > >       Server name: ipa1.example.com >       PKINIT status: enabled >    ---------------------------- >    Number of entries returned 2 >    ---------------------------- > >    To my understanding, proper certificate exchange between my two >  servers >    ceased working at some point. How do i track this down and fix > it? >   your issue looks similar to ticket #6792 [1]. Can you check the result  of   upgrade in /var/log/ipaupgrade.log?   Also check the output of   $ ipa-pkinit-manage status   and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and   /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r--  permissions.

Regarding the certificates, does getcert list show expired certs?   flo

[1] https://pagure.io/freeipa/issue/6792

---   $ ipa-pkinit-manage status   PKINIT is enabled   ---

There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem   exist with proper permissions, but I found something in ipaupgrade.log:

---   2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias  -L   -f /etc/httpd/alias/pwdfile.txt   2018-09-12T13:37:19Z DEBUG Process finished, return code=0   2018-09-12T13:37:19Z DEBUG stdout=   Certificate Nickname                                         Trust   Attributes

SSL,S/MIME,JAR/XPI

Server-Cert                                                  u,u,u   EXAMPLE.COM IPA CA                                           CT,C,C

2018-09-12T13:37:19Z DEBUG stderr=   2018-09-12T13:37:19Z DEBUG Starting external process   2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias  -L   -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt   2018-09-12T13:37:19Z DEBUG Process finished, return code=0   2018-09-12T13:37:19Z DEBUG stdout=   -----BEGIN CERTIFICATE-----   [...]   -----END CERTIFICATE-----

2018-09-12T13:37:19Z DEBUG stderr=   2018-09-12T13:37:19Z DEBUG Executing upgrade plugin:  update_ra_cert_store   2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store   2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')   2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')   2018-09-12T13:37:19Z DEBUG Starting external process   2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias  -L   -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt   2018-09-12T13:37:19Z DEBUG Process finished, return code=255   2018-09-12T13:37:19Z DEBUG stdout=   2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert:  ipaCert  :  PR_FILE_NOT_FOUND_ERROR: File not found   [...]   ---

this error can be ignored in most of the cases. The upgrade is trying to  move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the  files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the  upgrade is run a second time, he won't find ipaCert in the NSS database.  To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present  and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files  must be readable by root and ipaapi group, and must contain the same cert  as the other masters.

checked this, is ok.

...

What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and  ca-bundle.pem? both must contain IPA CA certificate.

True on both servers.

...

What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be  readable by everyone. And what is the content of this cert? It should be  issued by IPA CA.

Permissions are ok, contents:

--- ipa1  $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint  Certificate:       Data:           Version: 3 (0x2)           Serial Number: 1 (0x1)       Signature Algorithm: sha256WithRSAEncryption           Issuer: O=EXAMPLE.COM, CN=ipa1.example.com           Validity               Not Before: Nov 28 12:43:05 2017 GMT               Not After : Nov 28 12:43:05 2018 GMT           Subject: O=EXAMPLE.COM, CN=ipa1.example.com           Subject Public Key Info:               Public Key Algorithm: rsaEncryption                   Public-Key: (2048 bit)                   Modulus:                       [...]                   Exponent: 65537 (0x10001)           X509v3 extensions:               X509v3 Subject Alternative Name:                   othername:<unsupported>, othername:<unsupported>               X509v3 Basic Constraints: critical                   CA:FALSE               X509v3 Subject Key Identifier:

86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE               1.3.6.1.4.1.311.20.2:                   .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s  [...]  ---

--- ipa2  $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint  Certificate:       Data:           Version: 3 (0x2)           Serial Number: 805240833 (0x2fff0001)       Signature Algorithm: sha256WithRSAEncryption           Issuer: O=EXAMPLE.COM, CN=Certificate Authority           Validity               Not Before: Jan 18 13:04:17 2018 GMT               Not After : Jan 19 13:04:17 2020 GMT           Subject: O=EXAMPLE.COM, CN=ipa2.example.com           Subject Public Key Info:               Public Key Algorithm: rsaEncryption                   Public-Key: (2048 bit)                   Modulus:                       [...]                   Exponent: 65537 (0x10001)           X509v3 extensions:               X509v3 Authority Key Identifier:

keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB

Authority Information Access:                   OCSP - URI:http://ipa-ca.example.com/ca/ocsp

X509v3 Key Usage: critical                   Digital Signature, Non Repudiation, Key Encipherment,  Data Encipherment               X509v3 Extended Key Usage:                   TLS Web Server Authentication, 1.3.6.1.5.2.3.5               X509v3 CRL Distribution Points:

Full Name:                     URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin                   CRL Issuer:                     DirName: O = ipaca, CN = Certificate Authority

X509v3 Subject Key Identifier:

96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3               X509v3 Subject Alternative Name:                   othername:<unsupported>, othername:<unsupported>  [...]  ---

Additional info: I have DNS separate from IPA, but i (hopefully) made  proper records as IPA would have done it. In particular, i made an A  record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the  root cause of my problems, since DNS is not under my control.

Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by  IPA CA, might this be the actual problem?

The cert is expired on ipa1, which is the real root cause. ipa-pkinit-manage status is reporting that PKINIT is enabled but this does not match the actual configuration. This is a known issue [1].

If the host ipa1 is running a CA instance, then you can delete /var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to re-generate the KDC cert. Note: if the host doesn't run a CA instance, then this won't work because of the issue [2].

To check which hosts are running a CA instance, you can use # ipa server-role-find --role 'CA server'

[1] https://pagure.io/freeipa/issue/7200 [2] https://pagure.io/freeipa/issue/7795

removed the expired certificates, re-enabled PKINIT, certificates were regenerated, Web UI logins working again. Thank you very much.

...

...

...

About the errors spotted by ipa-checkcerts.py, what are the certificates  with errors reported? You can find them with ipa cert-show <serial>.

---  $ipa cert-show 57     Issuing CA: ipa     Certificate: [...]     Subject: CN=ipa1.example.com,O=EXAMPLE.COM     Issuer: CN=Certificate Authority,O=EXAMPLE.COM     Not Before: Tue Nov 28 12:42:07 2017 UTC     Not After: Mon Nov 18 12:42:07 2019 UTC     Serial number: 57     Serial number (hex): 0x39     Revoked: False

$ ipa cert-show 268304391     Issuing CA: ipa     Certificate: [...]     Subject: CN=IPA RA,O=EXAMPLE.COM     Issuer: CN=Certificate Authority,O=EXAMPLE.COM     Not Before: Wed Jul 11 14:26:35 2018 UTC     Not After: Tue Jun 30 14:26:35 2020 UTC     Serial number: 268304391     Serial number (hex): 0xFFE0007     Revoked: False

$ ipa cert-show 268304392     Issuing CA: ipa     Certificate: [...]     Subject: CN=CA Subsystem,O=EXAMPLE.COM     Issuer: CN=Certificate Authority,O=EXAMPLE.COM     Not Before: Wed Jul 11 14:26:45 2018 UTC     Not After: Tue Jun 30 14:26:45 2020 UTC     Serial number: 268304392     Serial number (hex): 0xFFE0008     Revoked: False

$ ipa cert-show 268304393     Issuing CA: ipa     Certificate: [...]     Subject: CN=OCSP Subsystem,O=EXAMPLE.COM     Issuer: CN=Certificate Authority,O=EXAMPLE.COM     Not Before: Wed Jul 11 14:27:05 2018 UTC     Not After: Tue Jun 30 14:27:05 2020 UTC     Serial number: 268304393     Serial number (hex): 0xFFE0009     Revoked: False

$ ipa cert-show 268304394     Issuing CA: ipa     Certificate: [...]     Subject: CN=CA Audit,O=EXAMPLE.COM     Issuer: CN=Certificate Authority,O=EXAMPLE.COM     Not Before: Wed Jul 11 14:27:25 2018 UTC     Not After: Tue Jun 30 14:27:25 2020 UTC     Serial number: 268304394     Serial number (hex): 0xFFE000A     Revoked: False  ---

ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation  cannot be completed: EXCEPTION (Invalid Credential.)" btw.

The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?

"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).

Replication appeared to be working, however, I did server maintenance today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does not come up properly:

--- ipa2 $ ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED

You need to have a look at krb5 logs to understand why kerberos failed to start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos for more information.

httpd Service: RUNNING ipa-custodia Service: STOPPED ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful

---

--- ipa1 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial         Serial Number: 268304391 (0xffe0007)

$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM

---

Looks good.

--- ipa2 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial         Serial Number: 268304391 (0xffe0007)

$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM

---

Looks bad.

So yes, the replication of the suffix o=ipaca seems broken. In a deployment with CA, the ldap server contains 2 different suffixes, one for the IdM data (your base DN as defined in /etc/ipa/default.conf) and another one for the CA, below o=ipaca.

You can have a look at https://www.freeipa.org/page/Troubleshooting/Directory_Server for replication troubleshooting, but the repl issue could be a consequence of kerberos server not starting.

--- /var/log/ipaupgrade.log [...] 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API

---

--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors [...] [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.example.com@EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)

---

--- ipa-checkcerts.py [...] ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ipa: INFO: Checking permissions and ownership Checking permissions and ownership ipa: INFO: Failures: Failures: ipa: INFO: Unable to find request for serial 268304391 Unable to find request for serial 268304391 ipa: INFO: Unable to find request for serial 268304394 Unable to find request for serial 268304394 ipa: INFO: Unable to find request for serial 268304393 Unable to find request for serial 268304393 ipa: INFO: Unable to find request for serial 268304392 Unable to find request for serial 268304392 ipa: INFO: Unable to find request for serial 268304390 Unable to find request for serial 268304390 ipa: INFO: RA agent description does not match 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP [...]

---

The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP, right? I guess this has to fixed by manually importing the proper certificate using ldapmodify, similar to the procedure described in [1]?

It is possible to manually modify the IPA RA cert in ipa2's ldap server but this won't solve the replication issue. It will however allow you to run ipa cert-show.

flo

[1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...

Mit freundlichen Gruessen/With best regards,

--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

On 1/8/19 3:51 PM, dbischof--- via FreeIPA-users wrote:

Hi Florence,

On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:

...

...

...

[...]

i shaved this thread a little, since it gets confusing. Hope i kept the interesting bits.

...

...

...

The errors related to "Unable to find request for serial xxx" mean  that the cert is tracked by certmonger, but there is no corresponding  LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.  Is the replication still working between your two masters?

"ipa cert-show" broken on ipa2 often points to an out-of-date  certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be  the same as on the renewal master, and also the same as in the entry  uid=ipara,ou=People,o=ipaca (which should be replicated and identical  on all the masters, except if you have replication issues).

Replication appeared to be working, however, I did server maintenance  today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2  does not come up properly:

--- ipa2  $ ipactl status  Directory Service: RUNNING  krb5kdc Service: STOPPED  kadmin Service: STOPPED

You need to have a look at krb5 logs to understand why kerberos failed to start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos for more information.

Turns out that kerberos starts when started manually with systemctl:

---

$ ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful

---

Kerberos log:

---

[...] : NEEDED_PREAUTH: admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11 Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11 Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for HTTP/ipa2.example.com@EXAMPLE.COM Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11

---

...

...

httpd Service: RUNNING  ipa-custodia Service: STOPPED  ntpd Service: RUNNING  pki-tomcatd Service: RUNNING  ipa-otpd Service: STOPPED  ipa: INFO: The ipactl command was successful  ---

--- ipa1  $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial           Serial Number: 268304391 (0xffe0007)

$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca  [...]  description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA  RA,O=EXAMPLE.COM  ---

Looks good.

--- ipa2  $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial           Serial Number: 268304391 (0xffe0007)

$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca  [...]  description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA  RA,O=EXAMPLE.COM  ---

Looks bad.

So yes, the replication of the suffix o=ipaca seems broken. In a deployment with CA, the ldap server contains 2 different suffixes, one for the IdM data (your base DN as defined in /etc/ipa/default.conf) and another one for the CA, below o=ipaca.

You can have a look at https://www.freeipa.org/page/Troubleshooting/Directory_Server for replication troubleshooting, but the repl issue could be a consequence of kerberos server not starting.

I checked the troubleshooting pages and carried out the steps described in there, but my system (ipa2) appears to be ok as far as the steps go.

...

...

--- /var/log/ipaupgrade.log  [...]  2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed,  exception: RemoteRetrieveError: Failed to authenticate to CA REST API  ---

--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors  [...]  [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not  get initial credentials for principal [ldap/ipa2.example.com@EXAMPLE.COM]  in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any  KDC for requested realm)  ---

--- ipa-checkcerts.py  [...]  ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)  ra.get_certificate(): EXCEPTION (Invalid Credential.)  ipa: INFO: Checking permissions and ownership  Checking permissions and ownership  ipa: INFO: Failures:  Failures:  ipa: INFO: Unable to find request for serial 268304391  Unable to find request for serial 268304391  ipa: INFO: Unable to find request for serial 268304394  Unable to find request for serial 268304394  ipa: INFO: Unable to find request for serial 268304393  Unable to find request for serial 268304393  ipa: INFO: Unable to find request for serial 268304392  Unable to find request for serial 268304392  ipa: INFO: Unable to find request for serial 268304390  Unable to find request for serial 268304390  ipa: INFO: RA agent description does not match 2;44;CN=Certificate  Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and  2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM  expected  ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not  found in LDAP  [...]  ---

The root cause appears to be the wrong IPA RA certificate in ipa2's  LDAP, right? I guess this has to fixed by manually importing the  proper certificate using ldapmodify, similar to the procedure  described in [1]?

It is possible to manually modify the IPA RA cert in ipa2's ldap server but this won't solve the replication issue. It will however allow you to run ipa cert-show.

Unfortunately, it doesn't (ipa2):

---

ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)

---

Kerberos log:

---

Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH: host/ipa2.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required

---

Works ok on ipa1.

I'm unsure how to proceed here. My obvious problem is ipa2 not getting upgraded properly:

---

[...] 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration] 2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf 2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Protocol versions already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z DEBUG Cipher suite already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Trust flags already processed 2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout=active

2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout= 2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Starting external process 2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout=active

2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Start of httpd.service complete 2019-01-08T09:46:09Z INFO [Removing self-signed CA] 2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed 2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA] 2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration files] 2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba configuration files] 2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8> 2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z INFO [Add missing CA DNS records] 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO IPA CA DNS records already processed 2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO [Upgrading CA schema] 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-certProfile.ldif 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-authority.ldif 2019-01-08T09:46:10Z DEBUG Not updating schema 2019-01-08T09:46:10Z INFO CA schema update complete (no changes) 2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2 year validity] 2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is 720 2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal configuration] 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/httpd/alias -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304389 (0xffe0005) [...] n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM= -----END CERTIFICATE-----

2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304388 (0xffe0004) [...] 7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds= -----END CERTIFICATE-----

2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout= Certificate Nickname                                         Trust Attributes

SSL,S/MIME,JAR/XPI

ocspSigningCert cert-pki-ca                                  u,u,u caSigningCert cert-pki-ca                                    CTu,Cu,Cu Server-Cert cert-pki-ca                                      u,u,u auditSigningCert cert-pki-ca                                 u,u,Pu subsystemCert cert-pki-ca                                    u,u,u

2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration already up-to-date 2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and validation] 2019-01-08T09:46:11Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:11Z INFO PKIX already enabled 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles] 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs] 2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in Dogtag database] 2019-01-08T09:46:11Z DEBUG Created connection context.ldap2_140245392902928 2019-01-08T09:46:11Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245392902928 2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration] 2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem] 2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP] 2019-01-08T09:46:12Z DEBUG Created connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG request GET https://ipa2.example.com:8443/ca/rest/account/login 2019-01-08T09:46:12Z DEBUG request body '' 2019-01-08T09:46:12Z DEBUG response status 401 2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 01:00:00 CET WWW-Authenticate: Basic realm="Certificate Authority" Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 951 Date: Tue, 08 Jan 2019 09:46:12 GMT

2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401 [...] message [...] This request requires HTTP authentication. [...]

</html>' 2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-01-08T09:46:12Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute     return_value = self.run()   File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run     server.upgrade()   File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2085, in upgrade     upgrade_configuration()   File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1952, in upgrade_configuration     ca_enable_ldap_profile_subsystem(ca)   File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_enable_ldap_profile_subsystem     cainstance.migrate_profiles_to_ldap()   File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1814, in migrate_profiles_to_ldap     _create_dogtag_profile(profile_id, profile_data, overwrite=False)   File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1820, in _create_dogtag_profile     with api.Backend.ra_certprofile as profile_api:   File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1302, in __enter__     raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))

2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

---

Hi, on ipa2, what is the content of /var/lib/ipa/ra-agent.pem? Is it the same as on ipa1? Can you check the file permissions?

Can you also compare with the ldap entry uid=ipara,ou=People,o=ipaca (description and usercertificate fields are important).

flo

Mit freundlichen Gruessen/With best regards,

--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote:

Hi Florence,

On Tue, 8 Jan 2019, Florence Blanc-Renaud wrote:

...

On 1/8/19 3:51 PM, dbischof--- via FreeIPA-users wrote:

...

Hi Florence,

On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:

...

...

...

[...]

i shaved this thread a little, since it gets confusing. Hope i kept the  interesting bits.

...

...

...

The errors related to "Unable to find request for serial xxx" mean   that the cert is tracked by certmonger, but there is no corresponding   LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.   Is the replication still working between your two masters?

"ipa cert-show" broken on ipa2 often points to an out-of-date   certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be   the same as on the renewal master, and also the same as in the entry   uid=ipara,ou=People,o=ipaca (which should be replicated and identical   on all the masters, except if you have replication issues).

Replication appeared to be working, however, I did server maintenance   today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2   does not come up properly:

--- ipa2   $ ipactl status   Directory Service: RUNNING   krb5kdc Service: STOPPED   kadmin Service: STOPPED

You need to have a look at krb5 logs to understand why kerberos failed to  start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos  for more information.

Turns out that kerberos starts when started manually with systemctl:

---  $ ipactl status  Directory Service: RUNNING  krb5kdc Service: RUNNING  kadmin Service: RUNNING  httpd Service: RUNNING  ipa-custodia Service: RUNNING  ntpd Service: RUNNING  pki-tomcatd Service: RUNNING  ipa-otpd Service: STOPPED  ipa: INFO: The ipactl command was successful  ---

Kerberos log:

---  [...] :  NEEDED_PREAUTH: admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM,  Additional pre-authentication required  Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11  Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes  {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes  {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for  krbtgt/EXAMPLE.COM@EXAMPLE.COM  Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11  Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes  {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes  {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for  HTTP/ipa2.example.com@EXAMPLE.COM  Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11  ---

...

...

httpd Service: RUNNING   ipa-custodia Service: STOPPED   ntpd Service: RUNNING   pki-tomcatd Service: RUNNING   ipa-otpd Service: STOPPED   ipa: INFO: The ipactl command was successful   ---

--- ipa1   $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial            Serial Number: 268304391 (0xffe0007)

$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca   [...]   description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA   RA,O=EXAMPLE.COM   ---

Looks good.

--- ipa2   $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial            Serial Number: 268304391 (0xffe0007)

$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca   [...]   description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA   RA,O=EXAMPLE.COM   ---

Looks bad.

So yes, the replication of the suffix o=ipaca seems broken. In a  deployment with CA, the ldap server contains 2 different suffixes, one  for the IdM data (your base DN as defined in /etc/ipa/default.conf) and  another one for the CA, below o=ipaca.

You can have a look at  https://www.freeipa.org/page/Troubleshooting/Directory_Server for  replication troubleshooting, but the repl issue could be a consequence of  kerberos server not starting.

I checked the troubleshooting pages and carried out the steps described in  there, but my system (ipa2) appears to be ok as far as the steps go.

...

...

--- /var/log/ipaupgrade.log   [...]   2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed,   exception: RemoteRetrieveError: Failed to authenticate to CA REST API   ---

--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors   [...]   [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could  not   get initial credentials for principal  [ldap/ipa2.example.com@EXAMPLE.COM]   in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any   KDC for requested realm)   ---

--- ipa-checkcerts.py   [...]   ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)   ra.get_certificate(): EXCEPTION (Invalid Credential.)   ipa: INFO: Checking permissions and ownership   Checking permissions and ownership   ipa: INFO: Failures:   Failures:   ipa: INFO: Unable to find request for serial 268304391   Unable to find request for serial 268304391   ipa: INFO: Unable to find request for serial 268304394   Unable to find request for serial 268304394   ipa: INFO: Unable to find request for serial 268304393   Unable to find request for serial 268304393   ipa: INFO: Unable to find request for serial 268304392   Unable to find request for serial 268304392   ipa: INFO: Unable to find request for serial 268304390   Unable to find request for serial 268304390   ipa: INFO: RA agent description does not match 2;44;CN=Certificate   Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and   2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA  RA,O=EXAMPLE.COM   expected   ipa: INFO: RA agent certificate not found in LDAP RA agent certificate  not   found in LDAP   [...]   ---

The root cause appears to be the wrong IPA RA certificate in ipa2's   LDAP, right? I guess this has to fixed by manually importing the   proper certificate using ldapmodify, similar to the procedure   described in [1]?

It is possible to manually modify the IPA RA cert in ipa2's ldap server  but this won't solve the replication issue. It will however allow you to  run ipa cert-show.

Unfortunately, it doesn't (ipa2):

---  ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid  Credential.)  ---

Kerberos log:

---  Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes  {18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH:  host/ipa2.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM,  Additional pre-authentication required  ---

Works ok on ipa1.

I'm unsure how to proceed here. My obvious problem is ipa2 not getting  upgraded properly:

---  [...]  2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration]  2019-01-08T09:46:07Z DEBUG Starting external process  2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled  2019-01-08T09:46:07Z DEBUG Process finished, return code=1  2019-01-08T09:46:07Z DEBUG stdout=  2019-01-08T09:46:07Z DEBUG stderr=  2019-01-08T09:46:07Z DEBUG Starting external process  2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload  2019-01-08T09:46:07Z DEBUG Process finished, return code=0  2019-01-08T09:46:07Z DEBUG stdout=  2019-01-08T09:46:07Z DEBUG stderr=  2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration]  2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf  2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions]  2019-01-08T09:46:07Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:07Z INFO Protocol versions already updated  2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite]  2019-01-08T09:46:07Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:07Z DEBUG Cipher suite already updated  2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP]  2019-01-08T09:46:07Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias]  2019-01-08T09:46:07Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:07Z INFO Trust flags already processed  2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy]  2019-01-08T09:46:07Z DEBUG Starting external process  2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled  2019-01-08T09:46:07Z DEBUG Process finished, return code=1  2019-01-08T09:46:07Z DEBUG stdout=  2019-01-08T09:46:07Z DEBUG stderr=  2019-01-08T09:46:07Z DEBUG Starting external process  2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service  2019-01-08T09:46:07Z DEBUG Process finished, return code=0  2019-01-08T09:46:07Z DEBUG stdout=  2019-01-08T09:46:07Z DEBUG stderr=  2019-01-08T09:46:07Z DEBUG Starting external process  2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service  2019-01-08T09:46:07Z DEBUG Process finished, return code=0  2019-01-08T09:46:07Z DEBUG stdout=active

2019-01-08T09:46:07Z DEBUG stderr=  2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete  2019-01-08T09:46:07Z DEBUG Starting external process  2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service  2019-01-08T09:46:09Z DEBUG Process finished, return code=0  2019-01-08T09:46:09Z DEBUG stdout=  2019-01-08T09:46:09Z DEBUG stderr=  2019-01-08T09:46:09Z DEBUG Starting external process  2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service  2019-01-08T09:46:09Z DEBUG Process finished, return code=0  2019-01-08T09:46:09Z DEBUG stdout=active

2019-01-08T09:46:09Z DEBUG stderr=  2019-01-08T09:46:09Z DEBUG Start of httpd.service complete  2019-01-08T09:46:09Z INFO [Removing self-signed CA]  2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed  2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA]  2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysrestore/sysrestore.state'  2019-01-08T09:46:09Z DEBUG Loading Index file from  '/var/lib/ipa/sysrestore/sysrestore.index'  2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration  files]  2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba  configuration files]  2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229')  2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229')  2019-01-08T09:46:09Z DEBUG flushing  ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache  2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache  url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket  conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8>  2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229')  2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229')  2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysrestore/sysrestore.state'  2019-01-08T09:46:09Z DEBUG Loading Index file from  '/var/lib/ipa/sysrestore/sysrestore.index'  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysrestore/sysrestore.state'  2019-01-08T09:46:09Z INFO [Add missing CA DNS records]  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:09Z INFO IPA CA DNS records already processed  2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options]  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections]  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS]  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS]  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysrestore/sysrestore.state'  2019-01-08T09:46:09Z DEBUG Loading Index file from  '/var/lib/ipa/sysrestore/sysrestore.index'  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysrestore/sysrestore.state'  2019-01-08T09:46:09Z DEBUG Loading Index file from  '/var/lib/ipa/sysrestore/sysrestore.index'  2019-01-08T09:46:09Z INFO DNS is not configured  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysrestore/sysrestore.state'  2019-01-08T09:46:09Z DEBUG Loading Index file from  '/var/lib/ipa/sysrestore/sysrestore.index'  2019-01-08T09:46:09Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:09Z INFO [Upgrading CA schema]  2019-01-08T09:46:10Z DEBUG Processing schema LDIF file  /usr/share/pki/server/conf/schema-certProfile.ldif  2019-01-08T09:46:10Z DEBUG Processing schema LDIF file  /usr/share/pki/server/conf/schema-authority.ldif  2019-01-08T09:46:10Z DEBUG Not updating schema  2019-01-08T09:46:10Z INFO CA schema update complete (no changes)  2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2 year  validity]  2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is  720  2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal  configuration]  2019-01-08T09:46:10Z DEBUG Loading Index file from  '/var/lib/ipa/sysrestore/sysrestore.index'  2019-01-08T09:46:10Z DEBUG Starting external process  2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/httpd/alias  -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt  2019-01-08T09:46:10Z DEBUG Process finished, return code=0  2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----  MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE  [...]  Serial Number: 268304389 (0xffe0005)  [...]  n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM=  -----END CERTIFICATE-----

2019-01-08T09:46:10Z DEBUG stderr=  2019-01-08T09:46:10Z DEBUG Loading Index file from  '/var/lib/ipa/sysrestore/sysrestore.index'  2019-01-08T09:46:10Z DEBUG Starting external process  2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d  dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f  /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt  2019-01-08T09:46:10Z DEBUG Process finished, return code=0  2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----  MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE  [...]  Serial Number: 268304388 (0xffe0004)  [...]  7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds=  -----END CERTIFICATE-----

2019-01-08T09:46:10Z DEBUG stderr=  2019-01-08T09:46:10Z DEBUG Loading Index file from  '/var/lib/ipa/sysrestore/sysrestore.index'  2019-01-08T09:46:10Z DEBUG Starting external process  2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d  dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt  2019-01-08T09:46:10Z DEBUG Process finished, return code=0  2019-01-08T09:46:10Z DEBUG stdout=  Certificate Nickname                                         Trust  Attributes

SSL,S/MIME,JAR/XPI

ocspSigningCert cert-pki-ca                                  u,u,u  caSigningCert cert-pki-ca                                    CTu,Cu,Cu  Server-Cert cert-pki-ca                                      u,u,u  auditSigningCert cert-pki-ca                                 u,u,Pu  subsystemCert cert-pki-ca                                    u,u,u

2019-01-08T09:46:10Z DEBUG stderr=  2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration  already up-to-date  2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and  validation]  2019-01-08T09:46:11Z DEBUG Loading StateFile from  '/var/lib/ipa/sysupgrade/sysupgrade.state'  2019-01-08T09:46:11Z INFO PKIX already enabled  2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles]  2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs]  2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in  Dogtag database]  2019-01-08T09:46:11Z DEBUG Created connection  context.ldap2_140245392902928  2019-01-08T09:46:11Z DEBUG flushing  ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache  2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache  url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket  conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8>  2019-01-08T09:46:12Z DEBUG Destroyed connection  context.ldap2_140245392902928  2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration]  2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem]  2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP]  2019-01-08T09:46:12Z DEBUG Created connection  context.ldap2_140245366754832  2019-01-08T09:46:12Z DEBUG flushing  ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache  2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache  url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket  conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128>  2019-01-08T09:46:12Z DEBUG Destroyed connection  context.ldap2_140245366754832  2019-01-08T09:46:12Z DEBUG request GET  https://ipa2.example.com:8443/ca/rest/account/login  2019-01-08T09:46:12Z DEBUG request body ''  2019-01-08T09:46:12Z DEBUG response status 401  2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1  Cache-Control: private  Expires: Thu, 01 Jan 1970 01:00:00 CET  WWW-Authenticate: Basic realm="Certificate Authority"  Content-Type: text/html;charset=utf-8  Content-Language: en  Content-Length: 951  Date: Tue, 08 Jan 2019 09:46:12 GMT

2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401  [...] message [...] This request requires HTTP authentication. [...]  </html>'  2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect  /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.  2019-01-08T09:46:12Z DEBUG   File  "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in  execute       return_value = self.run()     File  "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",

line 54, in run       server.upgrade()     File  "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",  line 2085, in upgrade       upgrade_configuration()     File  "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",  line 1952, in upgrade_configuration       ca_enable_ldap_profile_subsystem(ca)     File  "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",  line 396, in ca_enable_ldap_profile_subsystem       cainstance.migrate_profiles_to_ldap()     File  "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line  1814, in migrate_profiles_to_ldap       _create_dogtag_profile(profile_id, profile_data, overwrite=False)     File  "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line  1820, in _create_dogtag_profile       with api.Backend.ra_certprofile as profile_api:     File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py",  line 1302, in __enter__      raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to  CA REST API'))

2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed,  exception: RemoteRetrieveError: Failed to authenticate to CA REST API  2019-01-08T09:46:12Z ERROR Unexpected error - see /var/log/ipaupgrade.log  for details: RemoteRetrieveError: Failed to authenticate to CA REST API  2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See  /var/log/ipaupgrade.log for more information  ---

on ipa2, what is the content of /var/lib/ipa/ra-agent.pem? Is it the same as on ipa1? Can you check the file permissions?

Can you also compare with the ldap entry uid=ipara,ou=People,o=ipaca (description and usercertificate fields are important).

looked deeper this time, as you requested:

--- ipa1 $ ls -la /var/lib/ipa/ra-agent.* -r--r----- 1 root ipaapi 1854 Jul 11  2018 /var/lib/ipa/ra-agent.key -r--r----- 1 root ipaapi 1318 Jul 11  2018 /var/lib/ipa/ra-agent.pem

-- $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial         Serial Number: 268304391 (0xffe0007)

-- $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca # extended LDIF # # LDAPv3 # base <uid=ipara,ou=People,o=ipaca> with scope subtree # filter: (objectclass=*) # requesting: ALL #

# ipara, people, ipaca dn: uid=ipara,ou=people,o=ipaca userCertificate:: MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE  [...]  DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4= userCertificate:: MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE  [...]  DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ= userCertificate:: MIIDoTCCAomgAwIBAgIED/4ABzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQK  [...]  zon51BrP2rvAW9tpXAgPE1XqQuRzn userstate: 1 usertype: agentType cn: ipara sn: ipara uid: ipara objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM

# search result search: 2 result: 0 Success

# numResponses: 2 # numEntries: 1

-- $ head -n 5 ra-agent.key Bag Attributes     friendlyName: ipaCert     localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84 D7 11 Key Attributes: <No Attributes> -----BEGIN PRIVATE KEY-----

--- ipa2 $ ls -la /var/lib/ipa/ra-agent.* -r--r----- 1 root ipaapi 1828 Jul 11  2018 /var/lib/ipa/ra-agent.key -r--r----- 1 root ipaapi 1318 Jul 11  2018 /var/lib/ipa/ra-agent.pem

-- $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial         Serial Number: 268304391 (0xffe0007)

-- $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca # extended LDIF # # LDAPv3 # base <uid=ipara,ou=People,o=ipaca> with scope subtree # filter: (objectclass=*) # requesting: ALL #

# ipara, people, ipaca dn: uid=ipara,ou=people,o=ipaca description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser uid: ipara sn: ipara cn: ipara usertype: agentType userstate: 1 userCertificate:: MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE  [...]  DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4= userCertificate:: MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE  [...]  DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ=

# search result search: 2 result: 0 Success

# numResponses: 2 # numEntries: 1

-- $ head -n 5 ra-agent.key Bag Attributes     localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84 D7 11 Key Attributes: <No Attributes> -----BEGIN PRIVATE KEY----- [...]

---

Two differences:

1. The key file on ipa1 contains a "Bag Attribute" "friendlyName", that

the key key file on ipa2 misses - don't know if this is relevant, but probably not.

2. The LDAP entry on ipa1 contains 3 "userCertificate"s, but only the

first 2 of them are in ipa2's LDAP. This is probably the interesting part.

Am i right to assume, that adding the missing userCertificate to ipa2's LDAP will solve my problem? If yes, how would I achieve that?

Hi,

you can use ldapmodify to manually add the missing certificate:

1. transform the RA agent cert into der format $ openssl x509 -outform der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der

2. upload the cert in LDAP $ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca changetype: modify add: usercertificate usercertificate:< file:///tmp/ra-agent.der

modifying entry "uid=ipara,ou=people,o=ipaca"

<Ctrl-D> to exit

After that, you should be able to re-run ipa-server-upgrade. At this point, please make sure that replication could be re-established between the two nodes.

HTH, flo

Mit freundlichen Gruessen/With best regards,

--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

On 1/11/19 3:24 PM, dbischof--- via FreeIPA-users wrote:

Hi Florence,

On Thu, 10 Jan 2019, Florence Blanc-Renaud wrote:

...

On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote: [...] you can use ldapmodify to manually add the missing certificate:

1. transform the RA agent cert into der format $ openssl x509 -outform

der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der

2. upload the cert in LDAP

$ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca changetype: modify add: usercertificate usercertificate:< file:///tmp/ra-agent.der

modifying entry "uid=ipara,ou=people,o=ipaca"

<Ctrl-D> to exit

After that, you should be able to re-run ipa-server-upgrade. At this point, please make sure that replication could be re-established between the two nodes.

your help is greatly appreciated.

I had to change the cert serial in "description" additionally the same way via ldapmodify, but now ipa-server-upgrade goes smooth and IPA on ipa2 comes up properly after a reboot. Fine.

Regarding replication: Checking, whether replication works properly is achieved with "ipa-replica-manage -v list <host>", right? Has to work on both IPA servers and "last update ended" must be a reasonable recent timestamp?

Yes, ipa-replica-manage -v list <host> will display the status of the replication for the domain (user, hosts, ...). The value of "last update status" must be "Replica acquired successfully: Incremental update succeeded".

If the topology includes multiple CA instances, replication is also configured for the CA data, and the status can be found using ipa-csreplica-manage -v list <host>.

HTH, flo

Mit freundlichen Gruessen/With best regards,

--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...