<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
Thanks Randy; I think we have different root causes here.
<div><br>
</div>
<div>/opt/zimbra/stat is only 200MB on this particular server, built in December. </div>
<div><br>
</div>
<div>Before tonight\u2019s reboot I manually stopped Zimbra. </div>
<div><br>
</div>
<div>All the best,</div>
<div>Mark<br>
<br>
<div id="AppleMailSignature" dir="ltr">___________________________
<div>L. Mark Stone</div>
<div>Sent from my iPhone</div>
</div>
<div dir="ltr"><br>
On Jan 13, 2019, at 6:14 PM, Randy Leiker <<a href="mailto:randy@skywaynetworks.com">randy@skywaynetworks.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000">
<font face="arial, helvetica, sans-serif"><span style="font-size: 12pt;">Hi Mark,</span></font>
<div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;">
The issue with the Zimbra MTAs I eventually tracked down & solved could be different than what you're describing. In my case, I experienced these symptoms:</div>
<div>
<ol>
<li><font face="arial, helvetica, sans-serif">On Zimbra MTA 8.8.x servers, a "zmcontrol restart" or a reboot of the server would often, but not always, cause the Postfix & zmconfigd services to take anywhere from 10-20 minutes before they would start.</font></li><li><font face="arial, helvetica, sans-serif">The disk wait time in "top" was well above its normal baseline.</font></li><li><font face="arial, helvetica, sans-serif">The CPU usage was continuously at or near 100%.</font></li><li><font face="arial, helvetica, sans-serif">Long running processes using 20-40% CPU each were present in atop with names starting with "postconf -d" with their output piped to the Zimbra configuration folder.</font></li><li><font face="arial, helvetica, sans-serif">A unusual number of inotify processes would appear in "atop", each using around 10-15% CPU.</font></li></ol>
<div></div>
<div><font face="arial, helvetica, sans-serif">The root cause was due to the Zmstats (<a href="https://wiki.zimbra.com/wiki/Zmstats">https://wiki.zimbra.com/wiki/Zmstats</a>) process generating very large files in the /opt/zimbra/zmstats directory, where files
like io.csv & io-x.csv could be in the 3-5 GB size range per file. Zimbra has a built-in process to periodically rotate & archive all of the performance metrics found in the </font><span style="font-family: arial, helvetica, sans-serif;">/opt/zimbra/zmstats
directory in to sub-folders, with names based on the archive date. This archiving process uses the Linux cat command to pipe each file to gzip. Each time the 3-5 GB files were run through cat to gzip, it resulted in considerable disk I/O. This in turn caused
the CPU usage to peg at near 100%, as the CPU was endlessly waiting for the disk I/O to finish. Making matters worse, the anti-virus product installed on my Zimbra MTAs was scanning that data as it was being written to the Zmstat archive using the inotify
processes. All of this in turn left very little system resources left for zmconfigd to finish doing its rebuild of the Postfix configuration files (the postconf -d processes), as it normally does for any Zimbra service restart. This then caused zmconfigd
to appear non-responsive & where a "zmcontrol status" would report that it failed to start.</span></div>
<div><span style="font-family: arial, helvetica, sans-serif;"></span></div>
<div><span style="font-family: arial, helvetica, sans-serif;"><br>
</span></div>
<div><font face="arial, helvetica, sans-serif">The Zmstat archive folders are never removed by Zimbra & will accumulate indefinitely, eventually consuming a great deal of disk space left unchecked. I know there was an enhancement request in Bugzilla dating
back to Zimbra 7.x to auto delete the old Zmstats archive folders, but that's never been implemented. This means that Zimbra admins will need to create a separate script, or manually, purge those old archive folders. It's not clear to me yet why the file
size on some of the CSV files in that directory became so large over short periods of time (24-48 hours), but since manually erasing those files, stopping the zmstats service & restarting it, it's been behaving well for about a week now with no further recurrences.</font></div>
</div>
<div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;">
In answer to your question about file permissions & ownership, here's what I show for the /opt/zimbra/data/postfix/spool/pid directory on one of the Zimbra 8.8.x MTA servers here:</div>
<div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;">
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Mar 20 2018 inet.[127.0.0.1]:10025</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Mar 20 2018 inet.[127.0.0.1]:10030</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Mar 20 2018 inet.465</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Apr 11 2018 inet.submission</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 33 Jan 6 18:00 master.pid</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Mar 4 2018 pass.smtpd</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Mar 15 2018 unix.bounce</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Mar 15 2018 unix.cleanup</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Sep 19 23:30 unix.defer</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Sep 24 04:09 unix.error</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Apr 11 2018 unix.lmtp</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Sep 26 23:48 unix.retry</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Feb 11 2018 unix.showq</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Mar 15 2018 unix.smtp</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Mar 20 2018 unix.smtp-amavis</div>
<div style="font-size: 12pt;">-rw-------. 1 postfix postfix 0 Dec 6 08:58 unix.trace</div>
<div></div>
</div>
<div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;">
<br>
<div><span name="x"></span>
<div>
<div>
<div><span style="color: rgb(255, 102, 0); font-weight: bold;"><br>
Randy Leiker (</span><span style="font-weight: bold;"> <span style="color: rgb(51, 51, 255); background-color: rgb(255, 255, 255);">
<a href="mailto:randy@skywaynetworks.com">randy@skywaynetworks.com</a></span> <span style="color: rgb(255, 102, 0);">
)</span></span><br>
<span style="color: rgb(0, 0, 153);">Skyway Networks, LLC</span><br>
<span style="color: rgb(0, 0, 153);">1.800.538.5334</span> <span style="color: rgb(255, 102, 0);">
/</span> <span style="color: rgb(0, 0, 153);">913.663.3900 Ext. 100</span><br>
<span style="color: rgb(0, 0, 153);"></span><a href="http://www.skywaynetworks.com" target="_blank">https://skywaynetworks.com</a><br>
</div>
</div>
</div>
<span name="x"></span><br>
</div>
<hr id="zwchr">
<div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">
<b>From: </b>"L Mark Stone" <<a href="mailto:lmstone@lmstone.com">lmstone@lmstone.com</a>><br>
<b>To: </b><a href="mailto:users@lists.zetalliance.org">users@lists.zetalliance.org</a><br>
<b>Sent: </b>Sunday, January 13, 2019 4:21:30 PM<br>
<b>Subject: </b>[Users] Postfix Fails To Start On Reboot<br>
<br>
<style style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Several of us are seeing the issue where Postfix fails to restart on reboot. Randy I believe had some good information on this on a recent Zeta call.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">So, this happened (again) to me today, and I did my usual zmmtactl stop, move the
<span>~/data/postfix/spool/pid</span>/master.pid someplace, then zmmtactl start (which gets saslauthd started but not postfix) then "postfix stop" and "postfix start" and everything is OK again. All commands executed as the Zimbra user.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Except now I'm seeing the ownership permissions of
<span>~/data/postfix/spool/pid</span> all over the place, so asking what others are seeing and what should the ownership be?</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">On one system that has never had an issue with Postfix restarting, all of the files are owned by postfix:postfix with 600 perms.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">On the system where I just executed my hack, the master.pid, unix.error and unix.trace files are owned by root:root (with all other files owned by postfix:postfix). The old master.pid was owned by postfix:postfix.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">On another system which has had this issue intermittently, unix.trace is owned by root:root and everything else by postfix:postfix.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">So before I open a Support Case with Zimbra, I thought I'd ask here what others are seeing, and what your workaround has been.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Thanks,</p>
<p style="margin-top:0;margin-bottom:0">Mark<br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: Calibri, Arial, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p><strong>_________________________________________________</strong></p>
<p><strong>Another Message From... L. Mark Stone</strong></p>
<p><br>
</p>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>