Why are the services not started automatically on reboot?

Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Why are the services not started automatically on reboot?

Sonderegger, Markus

The monit manual says in “SYSTEM REBOOT AND SERVICE STARTUP”:

“In start mode, Monit will always start the service automatically on reboot, even if it was stopped before restart. This is the default mode and used if onreboot is not specified.”

 

I would expect that monit will call the START method when the system comes up but I experimented at least 2 days with PROCESS and PROGRAM with various combinations and did not manage to get the services started properly.

Do I misunderstand the saying of the manual that the services are started automatically?

 

Here some thoughts:

The manual says for a PROCESS service:

“If the pid-file does not exist or does not contain the PID number of a running process, Monit will call the entry's start method if defined.”

This no longer works when I add an EXEC action.

 

The manual also says for the validate argument:

“Check all services listed in the control file. This action is also the default behaviour when Monit runs in daemon mode.”

 

Our SysAdmin starts monit with the following systemd entry:

[Service]

Type=simple

PrivateTmp=true

UMask=0027

ExecStart=/opt/monit/bin/monit -I -vv

ExecStop=/opt/monit/bin/monit quit

ExecReload=/opt/monit/bin/monit reload

Restart=on-abnormal

StandardOutput=null

TimeoutStopSec=300

LimitCORE=infinity

LimitNPROC=16384

LimitNOFILE=65536

KillMode=none

 

Would we have to define ExecStart=/opt/monit/bin/monit -I –vv start all?

Doing so could we suppress starting all services by setting ONREBOOT NOSTART when we need to apply a patch before starting the services?

 

Many thanks for your help.

Regards

Markus

 



*****JuliusBaer Disclaimer***** This e-mail is for the intended recipient only and may contain confidential or privileged information. If you have received this e-mail by mistake, please contact us immediately and completely delete it (and any attachments) and do not forward it or inform any other person of its contents. If you send us messages by e-mail, we take this as your authorisation to correspond with you by e-mail, however, we reserve the right not to execute orders and instructions transmitted by e-mail at any time and without further explanation. If you do not wish to receive any further e-mail correspondence please let us know. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, amended, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Neither the Julius Baer Group nor the sender accept liability for any errors or omissions in the content of this message which arise as a result of its e-mail transmission. Please note that all e-mail communications to and from the Julius Baer Group may be monitored. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction.

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why are the services not started automatically on reboot?

Lutz Mader
Hello Markus,
let us know the used Monit version (the output from monit -V) and the
Linux version you used.

With regards, Lutz

p.s.
You are using RHEL I think?

Are you using Systemd to start your applications or handle Monit the
application startup only?

Any dependency defined for the services?

p.s.
There is no reason to do a "monit start all" after your system is up and
running. As long as you use the default setting ("onreboot start") Monit
start all defined applications.

To prevent the default behaviour you can use "onreboot laststate", but
Monit does not start stopped applications after a system restart then.
But you can stop applications and do some maintenance after a system
restart and start the aplications manually.
You can use "onreboot" with any service definition also.

I use something like this, for example (with Monit 5.25.2).
check process Postgres with pidfile /var/run/pgsql/data.pid
  onreboot laststate
  start program "/bin/launchctl load -F
/Library/LaunchDaemons/org.postgresql.postmaster.plist"
  stop program "/bin/launchctl unload
/Library/LaunchDaemons/org.postgresql.postmaster.plist"
  if failed unixsocket /var/run/pgsql/.s.PGSQL.5432 then restart
#  if failed host macmini.local port 5432 then restart
  if 3 restarts within 30 cycles then unmonitor

Reply | Threaded
Open this post in threaded view
|

RE: Why are the services not started automatically on reboot?

Sonderegger, Markus
Hello Lutz
Thanks for your comments.

> You are using RHEL I think?
Yes

> Are you using Systemd to start your applications or handle Monit the application startup only?
Monit

> Any dependency defined for the services?
Yes

> There is no reason to do a "monit start all" after your system is up and running. As long as you use the default setting ("onreboot start") Monit start all defined applications.
Good news, that is also my understanding.

If changed 2 service checks from PROGRAM to PROCESS and now the debug output in monit.log looks much more understandable.
I do not know why service check PROGRAM generated strange behavior like calling several time the START method on the same service  but it could be related to having the check program taking time to check if the applications run as they are heavy and take its time to start up. The check status is only collected after the next poll cycle.
I will investigate in more detail when I find time for it.

What still puzzles me that monit is not calling the START method for services which have no dependency and for the service which is at the base of the dependency tree (all other services depend from it) but calls the RESTART method.
Here the definition of the base service others depend on it
CHECK PROCESS mqm_TBCOM MATCHING "^/opt/mqm/bin/amqzxma0 -m TBCOM -u mqm$"
    START   = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /bin/strmqm TBCOM'" UID mqm GID mqm
    STOP    = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /bin/endmqm -i TBCOM'" UID mqm GID mqm
    RESTART = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /opt/mqm/bin/endmqm -p TBCOM; /opt/mqm/bin/strmqm TBCOM'" UID mqm GID mqm
    IF NOT EXIST THEN RESTART
    IF NOT EXIST THEN EXEC "/u01/app/bcom/monit/logger.sh CRITICAL"
    IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITOR
    IF 2 RESTARTS WITHIN 5 CYCLES THEN EXEC "/u01/app/bcom/monit/logger.sh FATAL"

The log shows that the RESTART method gets called.
[CEST Aug 14 22:07:14] error    : 'mqm_TBCOM' process is not running
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' exec: '/u01/app/bcom/monit/logger.sh CRITICAL'
[CEST Aug 14 22:07:14] error    : 'mqm_TBCOM' process is not running
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' trying to restart
[CEST Aug 14 22:07:14] debug    : 'appia_APPIA' process test failed [pid=22195] -- No such process
[CEST Aug 14 22:07:14] debug    : Cannot open proc file '/proc/2377/stat' -- No such file or directory
[CEST Aug 14 22:07:14] debug    : system statistic error -- cannot read /proc/2377/stat
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' restart: '/bin/bash -c . /u01/app/bcom/settings/bash_profile; /opt/mqm/bin/endmqm -p TBCOM; /opt/mqm/bin/strmqm TBCOM'
[CEST Aug 14 22:07:22] debug    : AMQ8146E: IBM MQ queue manager not available.
...

For all other services which are dependent of the base service monit correctly calls the START method.
Any ideas why this happens?
Could it be that the services do have an EXEC method defined. I use it to log RESTART of services into syslog calling the logger command.
It would be very nice if monit ALERT would be configurable in a sense that not only mailing is possible but another transport could be called e.g. logger command using syslog.

I also noticed that
CHECK DIRECTORY mqsi_mqsibackout_CS WITH PATH /u01/app/mqsi/filenodes/CcpcsfileToReconSwift940/CCPCSFILE.TO.RECON.SWIFT.940/mqsibackout
    IF CHANGED TIMESTAMP THEN EXEC "/u01/app/bcom/monit/logger.sh CRITICAL"
generates the following log entry I do not understand as the directory has not changed
[CEST Aug 14 22:07:27] debug    : 'mqsi_mqsibackout_CS' directory exists
[CEST Aug 14 22:07:27] debug    : 'mqsi_mqsibackout_CS' is directory
[CEST Aug 14 22:07:27] error    : 'mqsi_mqsibackout_CS' modify/change time for /u01/app/mqsi/filenodes/CcpcsfileToReconSwift940/CCPCSFILE.TO.RECON.SWIFT.940/mqsibackout changed from N/A to Fri, 21 Jun 2019 16:54:48
[CEST Aug 14 22:07:27] info     : 'mqsi_mqsibackout_CS' exec: '/u01/app/bcom/monit/logger.sh CRITICAL'

Many thanks for any help.
Regards
Markus

-----Original Message-----
From: monit-general <monit-general-bounces+markus.sonderegger=[hidden email]> On Behalf Of Lutz Mader
Sent: Sonntag, 11. August 2019 18:19
To: This is the general mailing list for monit <[hidden email]>
Subject: Re: Why are the services not started automatically on reboot?

Hello Markus,
let us know the used Monit version (the output from monit -V) and the Linux version you used.

With regards, Lutz

p.s.
You are using RHEL I think?

Are you using Systemd to start your applications or handle Monit the application startup only?

Any dependency defined for the services?

p.s.
There is no reason to do a "monit start all" after your system is up and running. As long as you use the default setting ("onreboot start") Monit start all defined applications.

To prevent the default behaviour you can use "onreboot laststate", but Monit does not start stopped applications after a system restart then.
But you can stop applications and do some maintenance after a system restart and start the aplications manually.
You can use "onreboot" with any service definition also.

I use something like this, for example (with Monit 5.25.2).
check process Postgres with pidfile /var/run/pgsql/data.pid
  onreboot laststate
  start program "/bin/launchctl load -F
/Library/LaunchDaemons/org.postgresql.postmaster.plist"
  stop program "/bin/launchctl unload
/Library/LaunchDaemons/org.postgresql.postmaster.plist"
  if failed unixsocket /var/run/pgsql/.s.PGSQL.5432 then restart #  if failed host macmini.local port 5432 then restart
  if 3 restarts within 30 cycles then unmonitor


*****JuliusBaer Disclaimer***** This e-mail is for the intended recipient only and may contain confidential or privileged information. If you have received this e-mail by mistake, please contact us immediately and completely delete it (and any attachments) and do not forward it or inform any other person of its contents. If you send us messages by e-mail, we take this as your authorisation to correspond with you by e-mail, however, we reserve the right not to execute orders and instructions transmitted by e-mail at any time and without further explanation. If you do not wish to receive any further e-mail correspondence please let us know. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, amended, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Neither the Julius Baer Group nor the sender accept liability for any errors or omissions in the content of this message which arise as a result of its e-mail transmission. Please note that all e-mail communications to and from the Julius Baer Group may be monitored. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction.

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why are the services not started automatically on reboot?

Lutz Mader
Hello Markus,
>
>> You are using RHEL I think?
> Yes
No, we/I use AIX (7.2), SLES (11.4, 12.3 and 12.4) and MacOS (10.6,
10.10). We/I use Monit 5.25.2, 5.26.0 and 5.19.0 (for the old MacOS 10.6
systems only).

>
>> Are you using Systemd to start your applications or handle Monit the application startup only?
> Monit
One monitor do the job only, nice to know.

>
>> Any dependency defined for the services?
> Yes
Take in mind, services are dependend to other services are starting only
if the parent services are in status "OK". Sometimes you find some
information in the log, a services can not start, because a parent
service is not "OK".

>
> If changed 2 service checks from PROGRAM to PROCESS and now the debug
> output in monit.log looks much more understandable.
I use "process" to handle several services, but I use "program" for some
specific applications as well, I will check my logs, but I thing I have
no problems with dependencies and process services.

I'll be back,
Lutz

Reply | Threaded
Open this post in threaded view
|

Re: Why are the services not started automatically on reboot?

Lutz Mader
In reply to this post by Sonderegger, Markus
Hello Markus,
the restart depends to your definition.

> Here the definition of the base service others depend on it
> CHECK PROCESS mqm_TBCOM MATCHING "^/opt/mqm/bin/amqzxma0 -m TBCOM -u mqm$"
>     START   = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /bin/strmqm TBCOM'" UID mqm GID mqm
>     STOP    = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /bin/endmqm -i TBCOM'" UID mqm GID mqm
>     RESTART = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /opt/mqm/bin/endmqm -p TBCOM; /opt/mqm> /bin/strmqm TBCOM'" UID mqm GID mqm
>     IF NOT EXIST THEN RESTART
>     IF NOT EXIST THEN EXEC "/u01/app/bcom/monit/logger.sh CRITICAL"
>     IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITOR
>     IF 2 RESTARTS WITHIN 5 CYCLES THEN EXEC "/u01/app/bcom/monit/logger.sh FATAL"
>
> The log shows that the RESTART method gets called.
> [CEST Aug 14 22:07:14] error    : 'mqm_TBCOM' process is not running
> ...
>
> For all other services which are dependent of the base service monit correctly calls the START method.
> Any ideas why this happens?

As long as a "restart" command is available and "if not exist then" use
"restart", the restart command will called if the service does not exists.
I prefer "if not exist for 3 cycles then start" to start an application.
The short delay of "3 cycles" is useful to prevent some tcp/ip problems.

With regards,
I check my monit logs tomorrow,
Lutz