bottledaemon stop/start doesn't work if killed elsewhere

[Adam Funk]

Hi,

I'm using bottledaemon to run a little REST service on a Pi that takes
input from other machines on the LAN and stores stuff in a database.
I have a cron job to call 'stop' and 'start' on it daily, just in case
of problems.

Occasionally the oom-killer runs overnight and kills the process using
bottledaemon; when this happens (unlike properly stopping the daemon),
the pidfile and its lockfile are left on the filesystem, so the 'stop'
does nothing and the 'start' gets refusedq because the old pidfile and
lockfile are present. At the moment, I eventually notice something
wrong with the output data, ssh into the Pi, and rm the two files then
call 'start' on the daemon again.

Is there a recommended or good way to handle this situation
automatically?

Thanks,
Adam

[Robert Heller]

Post by Adam Funk
Hi,
I'm using bottledaemon to run a little REST service on a Pi that takes
input from other machines on the LAN and stores stuff in a database.
I have a cron job to call 'stop' and 'start' on it daily, just in case
of problems.
Occasionally the oom-killer runs overnight and kills the process using
bottledaemon; when this happens (unlike properly stopping the daemon),
the pidfile and its lockfile are left on the filesystem, so the 'stop'
does nothing and the 'start' gets refusedq because the old pidfile and
lockfile are present. At the moment, I eventually notice something
wrong with the output data, ssh into the Pi, and rm the two files then
call 'start' on the daemon again.
Is there a recommended or good way to handle this situation
automatically?

This is *definately* a bug in the systemd service scripts for bottledaemon.
The stop script should rm the pidfile and lockfile, even if the daemon happens
to be dead. This is what any *properly* written daemon stop script does. So,
you need to file a bug report. In fact the *start* script should check to see
if the pidfile and lockfile are still legit -- eg is the pid in the pidfile a
valid pid, etc. Also part of a *properly* written daemon start script.

Post by Adam Funk
Thanks,
Adam

--
Robert Heller -- 978-544-6933
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
***@deepsoft.com -- Webhosting Services

[Adam Funk]

Post by Robert Heller

Post by Adam Funk
Hi,
I'm using bottledaemon to run a little REST service on a Pi that takes
input from other machines on the LAN and stores stuff in a database.
I have a cron job to call 'stop' and 'start' on it daily, just in case
of problems.
Occasionally the oom-killer runs overnight and kills the process using
bottledaemon; when this happens (unlike properly stopping the daemon),
the pidfile and its lockfile are left on the filesystem, so the 'stop'
does nothing and the 'start' gets refusedq because the old pidfile and
lockfile are present. At the moment, I eventually notice something
wrong with the output data, ssh into the Pi, and rm the two files then
call 'start' on the daemon again.
Is there a recommended or good way to handle this situation
automatically?

This is *definately* a bug in the systemd service scripts for bottledaemon.
The stop script should rm the pidfile and lockfile, even if the daemon happens
to be dead. This is what any *properly* written daemon stop script does. So,
you need to file a bug report. In fact the *start* script should check to see
if the pidfile and lockfile are still legit -- eg is the pid in the pidfile a
valid pid, etc. Also part of a *properly* written daemon start script.

I've reported this is an issue on github. I guess for now I could
replace the crontab command

~/sensors/server-sensors.py stop ; ~/sensors/server-sensors.py start

with this?

~/sensors/server-sensors.py stop ; rm -f /tmp/sensors-server.pid /tmp/sensors-server.pid.lock ; ~/sensors/server-sensors.py start

[Liam Proven]

Post by Adam Funk
Occasionally the oom-killer runs overnight and kills the process using
bottledaemon

Ouch.

Does the machine have a GUI installed? It might be worth disabling
that for headless operation.

And just FWIW, on my Raspberry Pi 3 with Lubuntu, I run zram to
provide swap, and it seems to work well. Might be worth a try.
--
Liam Proven - Profile: https://about.me/liamproven
Email: ***@cix.co.uk - Google Mail/Hangouts/Plus: ***@gmail.com
Twitter/Facebook/Flickr: lproven - Skype/LinkedIn: liamproven
UK: [phone] - ČR (+ WhatsApp/Telegram/Signal): [phone]

[Adam Funk]

Post by Liam Proven

Post by Adam Funk
Occasionally the oom-killer runs overnight and kills the process using
bottledaemon

Ouch.
Does the machine have a GUI installed? It might be worth disabling
that for headless operation.

No, it's running Raspbian Lite.

Post by Liam Proven
And just FWIW, on my Raspberry Pi 3 with Lubuntu, I run zram to
provide swap, and it seems to work well. Might be worth a try.

It already has a 46 GB swap partition on the external USB drive (which
holds everything except /boot & has its own power supply).

[Liam Proven]

Post by Adam Funk
No, it's running Raspbian Lite.

Never heard of it.

*Googles*

Ah, OK. Whereas I shudder with horror and revulsion at a 350 MB
operating system that _dares_ to call itself "lite", well... as Cicero
said, "o tempora, o mores". (Oh the times, oh the morals.)

So, not any kind of 'buntu, then? :-)

I do have a Devuan box but nothing running vanilla Debian, so I don't know.

Post by Adam Funk
It already has a 46 GB swap partition on the external USB drive (which
holds everything except /boot & has its own power supply).

Swapping onto a USB device?! :-o Ew!

Oh well, not a lot of choice, I guess. You might want to enable zswap,
though, so at least the traffic over the USB bus will be compressed.
--
Liam Proven - Profile: https://about.me/liamproven
Email: ***@cix.co.uk - Google Mail/Hangouts/Plus: ***@gmail.com
Twitter/Facebook/Flickr: lproven - Skype/LinkedIn: liamproven
UK: [phone] - ČR (+ WhatsApp/Telegram/Signal): [phone]

[Colin Law]

Post by Liam Proven

Post by Adam Funk
No, it's running Raspbian Lite.

Never heard of it.
*Googles*
Ah, OK. Whereas I shudder with horror and revulsion at a 350 MB
operating system that _dares_ to call itself "lite", well... as Cicero
said, "o tempora, o mores". (Oh the times, oh the morals.)
So, not any kind of 'buntu, then? :-)

He did apologise earlier when he realised he had posted to the wrong list.

Lite is relative of course. It is a lot liter (I am glad to see that
google spellcheck complains about lite and liter) than raspbian with a
GUI. It should be called Raspbian Server of course.

Colin

[Liam Proven]

Post by Colin Law
He did apologise earlier when he realised he had posted to the wrong list.

I know. That's in part why the smiley.

But Raspian is a Debian, and so is Ubuntu, so there's a reasonable
chance it might be related, as are most Mint queries.

Post by Colin Law
Lite is relative of course. It is a lot liter (I am glad to see that
google spellcheck complains about lite and liter)

Agreed!

"Liter" is the American for "litre", of course, so that's acceptable
in international English.

Post by Colin Law
than raspbian with a
GUI. It should be called Raspbian Server of course.

Heh.

Sadly a lot of respectable server apps require a GUI now. This came up
on another list I'm on and attracted quite some debate. They're mostly
commercial proprietary stuff, was my impression, but legit business
tools for hardware setup/config and whatnot that do need a GUI.

I suppose one principle difference is that on a Unix box, the GUI
isn't part of the kernel and an unused GUI will just get swapped out.
On Windows, part of the GDI is inside the kernel and so it can't
readily be (according to my *very* limited understanding of Windows'
paging mechanisms).
--
Liam Proven - Profile: https://about.me/liamproven
Email: ***@cix.co.uk - Google Mail/Hangouts/Plus: ***@gmail.com
Twitter/Facebook/Flickr: lproven - Skype/LinkedIn: liamproven
UK: [phone] - ČR (+ WhatsApp/Telegram/Signal): [phone]

[Adam Funk]

Post by Liam Proven

Post by Adam Funk
No, it's running Raspbian Lite.

Never heard of it.
*Googles*
Ah, OK. Whereas I shudder with horror and revulsion at a 350 MB
operating system that _dares_ to call itself "lite", well... as Cicero
said, "o tempora, o mores". (Oh the times, oh the morals.)
So, not any kind of 'buntu, then? :-)

No, sorry, I posted to the wrong list!

Post by Liam Proven
I do have a Devuan box but nothing running vanilla Debian, so I don't know.

Post by Adam Funk
It already has a 46 GB swap partition on the external USB drive (which
holds everything except /boot & has its own power supply).

Swapping onto a USB device?! :-o Ew!
Oh well, not a lot of choice, I guess.

That's right: /boot has to be on the SD card, but everything else is
more reliable on the USB drive.

Post by Liam Proven
You might want to enable zswap,
though, so at least the traffic over the USB bus will be compressed.

I'll take a look, thanks.

[Liam Proven]

Post by Adam Funk

Post by Liam Proven
So, not any kind of 'buntu, then? :-)

No, sorry, I posted to the wrong list!

I know, I saw your comment. Don't worry about it. Firstly, it
happens, and secondly, as I said, I think it's broadly relevant.

But the OOM killer bringing a server down in this day and age is a
worrying thing...

Post by Adam Funk
That's right: /boot has to be on the SD card, but everything else is
more reliable on the USB drive.

OK. I don't run Linux a lot on my 'Pis (I'm more interesting in RISC
OS, Ultibo, Oberon and other OSes) and have no Raspbian so my Linux
card contains the whole distro, on ext2, with noatime, no swap and
just ZRAM. It's quite quick.

Post by Adam Funk
I'll take a look, thanks.

People are very sceptical about ZRAM, Zswap and their kin, but there's
no need to be.

I know swapping to RAM sounds crazy, but I think the proof that the
idea works is
that it was introduced as standard in Mac OS X as of version 10.9 "Mavericks":

https://www.lifewire.com/understanding-compressed-memory-os-x-2260327

The OS X implementation is slightly superior, inasmuch as it both
compresses into a swapfile in RAM, then writes the compressed info
into disk swap if required. In Linux terms it's a combination of ZRAM
+ ZSwap + ZCache.

The tech is also in Win10:

https://www.makeuseof.com/tag/ram-compression-improves-memory-responsiveness-windows-10/

And in ChromeOS, Android and IBM AIX.

So it's pretty mainstream stuff now. Multicore processors help a lot
-- a lot of code is still single-threaded, and with ZRAM, idle cores
can be used to do the compression, while other cores are busy.

So given the RasPi's limited CPU and disk bandwidth, but multiple
cores, I'd recommend it.
--
Liam Proven - Profile: https://about.me/liamproven
Email: ***@cix.co.uk - Google Mail/Hangouts/Plus: ***@gmail.com
Twitter/Facebook/Flickr: lproven - Skype/LinkedIn: liamproven
UK: [phone] - ČR (+ WhatsApp/Telegram/Signal): [phone]

[Adam Funk]

Post by Liam Proven

Post by Adam Funk

Post by Liam Proven
So, not any kind of 'buntu, then? :-)

No, sorry, I posted to the wrong list!

I know, I saw your comment. Don't worry about it. Firstly, it
happens, and secondly, as I said, I think it's broadly relevant.

That's generous.

Post by Liam Proven
But the OOM killer bringing a server down in this day and age is a
worrying thing...

It is a bit weird, but it only ever happens when texpire is running.

[Adam Funk]

Post by Adam Funk
Hi,
I'm using bottledaemon to run a little REST service on a Pi that takes
input from other machines on the LAN and stores stuff in a database.
I have a cron job to call 'stop' and 'start' on it daily, just in case
of problems.
Occasionally the oom-killer runs overnight and kills the process using
bottledaemon; when this happens (unlike properly stopping the daemon),
the pidfile and its lockfile are left on the filesystem, so the 'stop'
does nothing and the 'start' gets refusedq because the old pidfile and
lockfile are present. At the moment, I eventually notice something
wrong with the output data, ssh into the Pi, and rm the two files then
call 'start' on the daemon again.
Is there a recommended or good way to handle this situation
automatically?

Sorry, I posted this to the wrong group/list!