MONITRC run time updation causing MONIT to freeze.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

MONITRC run time updation causing MONIT to freeze.

Kannamraju P
Hi All,
  I am running into a race condition , when I try to dynamically update monitrc externally while MONIT is running , here is my use case

1. MONIT setup in daemon mode with 5 sec polling timeout . It  has a set of static services defined in the default MONITRC file.
2. Default  Monitrc includes one more custom config file ex: monitrc_custom file 
   ."include /etc/monitrc_custom"

    a) In this custom monitrc file , services get added and deleted dynamically by an external process
    b) Any updation to monitrc is followed by monit reload, so that MONIT picks up the updated service details

  With this setup I am observing that when both external service and monit try to access the custom monitrc file , MONIT is getting stuck . 
It becomes non responsive  and "Monit summary" results in all processes stuck in initializing state .

ex:

image.png


Is there a way I can apply some locking mechanism to avoid the contention of both external processes and monit accessing 
config files at same time ?


--
thanks & Regards,
Raju
Reply | Threaded
Open this post in threaded view
|

Re: MONITRC run time updation causing MONIT to freeze.

Jan-Henrik Haukeland
The problem might be related to a monit reload race condition [1] which was fixed in Monit 5.27.1. 



On 28 Oct 2020, at 01:17, Kannamraju P <[hidden email]> wrote:

Hi All,
  I am running into a race condition , when I try to dynamically update monitrc externally while MONIT is running , here is my use case

1. MONIT setup in daemon mode with 5 sec polling timeout . It  has a set of static services defined in the default MONITRC file.
2. Default  Monitrc includes one more custom config file ex: monitrc_custom file 
   ."include /etc/monitrc_custom"

    a) In this custom monitrc file , services get added and deleted dynamically by an external process
    b) Any updation to monitrc is followed by monit reload, so that MONIT picks up the updated service details

  With this setup I am observing that when both external service and monit try to access the custom monitrc file , MONIT is getting stuck . 
It becomes non responsive  and "Monit summary" results in all processes stuck in initializing state .

ex:

image.png


Is there a way I can apply some locking mechanism to avoid the contention of both external processes and monit accessing 
config files at same time ?


-- 
thanks & Regards,
Raju

Reply | Threaded
Open this post in threaded view
|

Re: MONITRC run time updation causing MONIT to freeze.

Lutz Mader
Hello Jan-Henrik,
the "Initializing" state persist for one monitor cycle only, with Monit
5.26.0 or Monit 5.27.0.

> The problem might be related to a monit reload race condition [1] which was fixed in Monit 5.27.1.
>
> 1. https://bitbucket.org/tildeslash/monit/issues/928

I do similar things but use a monitor cycle of 60 secound. A cycle of 5
secounds is very small.

>> 1. MONIT setup in daemon mode with 5 sec polling timeout . It  has a set of static services defined in the default MONITRC file.

After a configuration update with "monit reload" I see the
"Initializing" state for some Process checks and "Waiting" for Program
and Remote host checks. Especially for checks are using "every 5 cycles"
to reduce the number of monitor cycles for some checks.

Next week I'll switch to Monit 5.27.1 and do some tests.

With regards,
Lutz