Bug#764225: slapd: fails to flush the cache if the system is shut down

[Wolfgang Schweer]

Source: slapd
Version: 2.4.39-1.1+b1
Severity: important
User: debian-***@lists.debian.org
Usertags: debian-edu

Hi,

while testing Debian Edu in jessie, I noticed that recently made LDAP
entries got lost if the system was rebootet or powered off.

Stopping and starting slapd before reboot kept the entries.

Maybe slapd should sort of inform other processes?

Feel free to reassign the bug to another package if it's not slapd's
fault.

Regards and thanks ahead,
Wolfgang

[Wolfgang Schweer]

This issue is a fatal problem for Debian Edu.

I guess the reason are missing systemd service files for the shutdown,
poweroff and reboot targets. So maybe slapd should ship those files
(like other packages do).

Wolfgang

[Wolfgang Schweer]

Do you have olcDbNoSync (or some equivalent setting in the bdb config
if you use that) set to true? If olcDbNoSync is set, then it could
result in dataloss, else the commited data should remain intact.
What backend do you use?

bdb, and yes: dbnosync is set

I haven't been able to reproduce this, with BDB/HDB/MDB backends and
olcDbNoSync: TRUE. I'm running 'ldapadd -f mods.ldif && reboot' and
mods.ldif contains 10000 small entries. Can you provide any more details
about the steps I should follow to observe the data loss?

Debian Edu uses GOsa² to manage user accounts, systems and groups.
Adding a system via GOsa² to the (BDB) LDAP database succeeds, using
ldapvi one can see the entries in LDAP. Rebooting the system just
afterwards results in loss of the added data. Same happens if a user
password is set via GOsa². The user password isn't valid after a server
reboot.

Stopping slapd manually before the reboot doesn't result in data loss.

BTW: As far as I can tell slapd is stopped properly on shutdown; after
rebooting I don't see any log messages about having to recover an unclean
database, or similar.

Same here with systemd.

Does Debian Edu usually set olcDbNoSync?

As far as I know: yes, since some time slapd.conf contains 'dbnosync'.
s/dbnosync/#dbnosync/ avoids the described data loss.

For testing I switched from systemd to sysvinit: same effect, same
workarounds succeed to keep newly added data. One difference, though:
under sysvinit there is no proper slapd stop during shutdown.

Wolfgang

[Ryan Tandy]

Post by Wolfgang Schweer
Debian Edu uses GOsa² to manage user accounts, systems and groups.
Adding a system via GOsa² to the (BDB) LDAP database succeeds, using
ldapvi one can see the entries in LDAP. Rebooting the system just
afterwards results in loss of the added data. Same happens if a user
password is set via GOsa². The user password isn't valid after a server
reboot.

Thanks for the info. I'll install Debian Edu and try it. I assume
"rebooting the system" means the server running slapd, and not the
system that was just added to the directory, right?

Post by Wolfgang Schweer
For testing I switched from systemd to sysvinit: same effect, same
under sysvinit there is no proper slapd stop during shutdown.

I don't experience this, either. Another thing to check once I have
Debian Edu installed.

[Wolfgang Schweer]

Post by Ryan Tandy

Post by Wolfgang Schweer
Debian Edu uses GOsa² to manage user accounts, systems and groups.
Adding a system via GOsa² to the (BDB) LDAP database succeeds, using
ldapvi one can see the entries in LDAP. Rebooting the system just
afterwards results in loss of the added data. Same happens if a user
password is set via GOsa². The user password isn't valid after a server
reboot.

Thanks for the info. I'll install Debian Edu and try it.

ATM, Debian Edu (jessie based) isn't installable, cause the ISO image is
broken. It will take about 10 days to have it fixed. See here for the
download link:

https://wiki.debian.org/DebianEdu/Status/Jessie

Post by Ryan Tandy
I assume "rebooting the system" means the server running slapd, and
not the system that was just added to the directory, right?

Yes.

Wolfgang

[Petter Reinholdtsen]

[Wolfgang Schweer]

Post by Wolfgang Schweer
ATM, Debian Edu (jessie based) isn't installable, cause the ISO image is
broken. It will take about 10 days to have it fixed. See here for the
https://wiki.debian.org/DebianEdu/Status/Jessie

I managed to get a workaround in place, while we wait. The ISO available from
<URL: ftp://ftp.skolelinux.org/pub/test/debian-edu-amd64-i386-NETINST-testing.iso >
(or http or rsync) was built 2014-10-02, when the ISO was still
working, and avoid the kernel version skew problem currentlyb blocking
testing.

If you boot it with this argument on the grub menu, it will fetch
fixed versions of the debian-edu packages, and avoid the problem with
the cfengine rules:

url=http://ftp.skolelinux.org/pub/test/jessie.dat

This would allow you to test a Debian Edu Jessie main server to
reproduce the problem Wolfgang discovered.

[Ryan Tandy]

Post by Petter Reinholdtsen
I managed to get a workaround in place, while we wait. The ISO available from
<URL: ftp://ftp.skolelinux.org/pub/test/debian-edu-amd64-i386-NETINST-testing.iso >
(or http or rsync) was built 2014-10-02, when the ISO was still
working, and avoid the kernel version skew problem currentlyb blocking
testing.
If you boot it with this argument on the grub menu, it will fetch
fixed versions of the debian-edu packages, and avoid the problem with
url=http://ftp.skolelinux.org/pub/test/jessie.dat

Thanks for providing that. I have a main-server installed now, however I
still have not encountered any instances of LDAP changes being lost. I
checked that slapd.conf still uses 'dbnosync', and I stopped squid to
avoid it delaying shutdown. I've added/modified users and groups in
gosa, I've changed users' passwords in gosa, and I've done bulk LDAP
modifications from the command line, but I still have not observed any
changes being lost after rebooting. Any idea what I'm missing? Any
configuration changes at your end?

(In case it matters: this particular tjener is a virtualbox VM.)

[Wolfgang Schweer]

I have a main-server installed now, however I still have not
encountered any instances of LDAP changes being lost.

Thanks for investigating. It's really weird: After installing a main
server with a method similar to the one you've used, I'm unable to
reproduce the data loss. And I can't figure out why it failed before
(main server installed sept 14th) and not now.

(In case it matters: this particular tjener is a virtualbox VM.)

Same here.

Wolfgang

[Wolfgang Schweer]

Post by Wolfgang Schweer

I have a main-server installed now, however I still have not
encountered any instances of LDAP changes being lost.

Thanks for investigating. It's really weird: After installing a main
server with a method similar to the one you've used, I'm unable to
reproduce the data loss. And I can't figure out why it failed before
(main server installed sept 14th) and not now.

(In case it matters: this particular tjener is a virtualbox VM.)

Same here.

After Debian Edu got a report from a user who wasn't able to log in with
another user account than 'first user', I investigated this issue again.

Main server installed as virtualbox VM, dbnosync set in
/etc/ldap/slapd.conf:

1) As first user: create a new account, set password.
2) Log out of the gui, log in as the new user. Works.
3) Poweroff the VM (aka main server power loss?).
4) Start VM again and log in as the new user. Fails.

Repeat these steps with dbnosync disabled. Login as new user works.

Using 'Shutdown via ACPI event' (or similar, translated from German)
instead of 'Poweroff' in 3) or rebooting the main server inside of the
VM: No data loss with dbnosync enabled. (For some reason data loss
occured in this case in the previous installation.)

So I guess that there's no slapd bug but Debian Edu is better off to go
with dbnosync disabled to avoid data loss in rare cases.

Wolfgang

[Ryan Tandy]

Hi,

Post by Wolfgang Schweer
Main server installed as virtualbox VM, dbnosync set in
1) As first user: create a new account, set password.
2) Log out of the gui, log in as the new user. Works.
3) Poweroff the VM (aka main server power loss?).
4) Start VM again and log in as the new user. Fails.

Yes, dbnosync is documented as risking data loss in the case of power
loss or a system crash:

http://manpages.debian.org/cgi-bin/man.cgi?query=slapd-bdb&apropos=0&sektion=0&manpath=Debian+7.0+wheezy&format=html&locale=en

http://docs.oracle.com/cd/E17076_04/html/api_reference/C/envset_flags.html#envset_flags_DB_TXN_NOSYNC

Post by Wolfgang Schweer
So I guess that there's no slapd bug but Debian Edu is better off to go
with dbnosync disabled to avoid data loss in rare cases.

OK. Closing the slapd bug based on that, but please do re-open it if
your original problem comes back again.

Thanks for making Debian Edu Jessie excellent!

Ryan

[Ryan Tandy]

You sure this isn’t just the process not stopping normally due to a
client holding a socket open or a long query thus the process eventually
gets kill -KILL ‘d. It certainly use to be the case that the stop script
was undesirable under some conditions.

Definitely possible, but in that case it should be clear from the
shutdown messages what's going on (cf. slow shutdown of squid in
debian-edu recently), or from reviewing syslog afterwards. None of the
messages on the bug so far indicated that sort of issue.

thanks,
Ryan