All,
This is a strange one. When we exec this command under puppet control:
/usr/sbin/realm permit -R AMER.COMPANY.COM processehcprofiler@AMER.COMPANY.COM
Then sssd_be core dumps (segfault).
When we run that ‘realm permit’ command natively on the command line, it executes flawlessly. No core dump.
Naturally, our first thought was that it’s something different in the environment. So we dumped the environment under which puppet exec resource runs.
Yes, quite minimal.
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:
LD_LIBRARY_PATH=
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
HOSTNAME=austgcore25.us.company.com
S_COLORS=auto
PWD=/root
APT_LISTCHANGES_FRONTEND=none
DEBIAN_FRONTEND=noninteractive
APT_LISTBUGS_FRONTEND=none
MAIL=/var/spool/mail/root
SHELL=/bin/bash
TERM=xterm
SHLVL=2
MANPATH=:/opt/puppetlabs/puppet/share/man
PATH=/bin:/usr/bin:/sbin:/usr/sbin
HISTSIZE=1000
LESSOPEN=||/usr/bin/lesspipe.sh %s
_=/bin/env
But when we create a bash session with no environment, then add this puppet-supplied environment and run the above realm permit– all is still well.
We can reproduce this easily in puppet. Just delete our breadcrumb file (so that puppet re-executes this ‘realm permit’ command). And execute another puppet agent run.
Doing this, we obtained a core dump from sssd_be. And it points to some code in ad_id.c:
[root@austgcore25 tmp]# gdb -c core-sssd_be.15405.austgcore25.us.company.com.1563210863
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-5.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word".
[New LWP 15405]
Reading symbols from /usr/libexec/sssd/sssd_be...Reading symbols from /usr/lib/debug/usr/libexec/sssd/sssd_be-2.0.0-43.el8.x86_64.debug...done.
done.
warning: Ignoring non-absolute filename: <linux-vdso.so.1>
Missing separate debuginfo for linux-vdso.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/06/44254f9cbaa826db070a796046026adba58266
warning: .dynamic section for "/usr/lib64/libndr-nbt.so.0.0.1" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/sssd/sssd_be --domain AMER.COMPANY.COM --uid 0 --gid 0 --logger=files'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f79444f53c0 in ad_get_account_domain_search (req=req@entry=0x5557b6fd45b0) at src/providers/ad/ad_id.c:1276
1276 state->filter = sdap_combine_filters(state, state->base_filter,
(gdb)
This is in RHEL8.
What we suspect is that the shell in which puppet executes this ‘realm permit’ is not supplying something that this executable needs. If we know what it is, we can preface our puppet exec resource code snippet with the missing piece.
Spike
PS This ‘realm permit’ does seem to perform the correct actions eventually. It adds the expected user to the simple_allow_users line of the appropriate AD domain in /etc/sssd/sssd.conf file. But because it segfaults and then has to start up again, it takes a very long time to complete the puppet run. (There’s about 20 – 30 users + groups allowed; it has to segfault on each of them).