puppet.md

this weeks first "was it really that simple all along":

Somebody wrote a puppet module that installs an app (mokey). Running the puppet agent shows it creates the systemd unit file and restarts the app. Every time

root@jeemoi:~# puppet agent -t
Notice: .../Systemd::Unit_file[mokey.service]/File[/etc/systemd/system/mokey.service]/ensure: created (corrective)
...
Notice: .../Systemd::Unit_file[mokey.service]/Service[mokey.service]: Triggered 'refresh' from 2 events
Notice: Applied catalog in 3.26 seconds

root@jeemoi:~# puppet agent -t
Notice: .../Systemd::Unit_file[mokey.service]/File[/etc/systemd/system/mokey.service]/ensure: created (corrective)
...
Notice: .../Systemd::Unit_file[mokey.service]/Service[mokey.service]: Triggered 'refresh' from 2 events
Notice: Applied catalog in 3.26 seconds

the funny thing is that after this the file service unit is gone:

ls: cannot access '/etc/systemd/system/mokey.service': No such file or directory

after a lot of strace puppet agent -t --debug --trace 2>&1 | grep -C5 mokey.service and such, I could not find out how/why it disappears I ended up using eBPF to hook the unlink() syscall:

  Process: systemd (PID: 1)
  User: UID 0
  Parent PID: 0
  Syscall: tracepoint:syscalls:sys_enter_unlink
  File: /etc/systemd/system/mokey.service

[7:10 PM]varesa: so systemd deletes my unit file? wtf

turn on systemd debug logging and daemon-reexec

more clues:

Jan 20 18:52:38 jeemoi.idm.wappuradio.fi systemd[1]: Found unit mokey.service at /etc/systemd/system/mokey.service (mask)
Jan 20 18:52:38 jeemoi.idm.wappuradio.fi systemd[1]: Got message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=UnmaskUnitFiles cookie=2 reply_cookie=0 signature=asb error-name=n/a error-message=n/a

So I guess the unmask deletes the unit file, but why is it masked in the first place? That'd be like a symlink to /dev/null, right? Reading the source code of the puppet systemd::unit_file does not show any possible path how the file could end up as a symlink in this case

I end up just SSH'ing to the puppetmaster and patching all the systemd calls out of systemd::unit_file leaving only the file { ... } resource. It creates? An empty file

systemctl status?

● mokey.service
     Loaded: masked (Reason: Unit mokey.service is masked.)

well, turns out that an empty file is as good as symlinking to /dev/null for masking

also turns out that the error is in the puppet module:

  systemd::unit_file { 'mokey.service':
    ensure => present,
    enable => true,
    active => true,
    content => $_mokeyservice, <--- this damn variable is undefined and that's not even worth a warning??? LOL
  }

so puppet creates an empty file which after a daemon-reload ends up masking the unit. Then puppet enabling the service causes systemd to remove the mask (e.g. the file)