Monit Bug Impacting AppScale Deployments

Posted by AppScale Staff on 8/29/18 11:19 AM

tl;dr: Due to a recently discovered bug in Monit, you may have encountered issues with running AppScale or applications within AppScale. A workaround is to downgrade Monit and temporarily prevent future Monit upgrades on *all* instances of your deployment with the following:

sudo apt-get install monit=1:5.16-2
sudo apt-mark hold monit

Once the next version of AppScale is available, future upgrades of Monit should be re-enabled (with 'unhold').

AppScale Cloud Images

AppScale cloud images are based on Ubuntu LTS. As the LTS acronym infers, these images have the benefit of long-term support, which allows you the flexibility to plan and decide when to perform a full system upgrade.

A feature of the Ubuntu LTS images is that the packages auto-update for critical security fixes. A package called unattended-upgrades is installed and configured by default; the package periodically updates the system with the latest security patches available for that version.

We believe the long-term support and the auto-upgrade for security patches are important for cloud images, due to their volatile nature. In particular, AppScale’s ability to autoscale with the application’s needs implies that instances can be created at any time, so there is a chance that these instances are based off of older cloud images.

In the standard AppScale configuration, these autoscaled instances have limited access to the external world (i.e., they typically do not have a public IP address on public clouds) and they have a strict firewall configuration in place. Nonetheless, the auto-update mechanism ensures the instances are secure.

The Flip Side of the Coin

If you believe that auto-updates are a good feature for cloud images, it is time to mention the flip side of the coin. Since security updates arrive at unexpected intervals, and most of the time with little warning ahead of the release, it is difficult for AppScale and our users to plan for possible side effects. For example, the security update of Monit created issues with all AppScales deployments, preventing newer deployments from working correctly and older deployments from being able to autoscale applications.

We test the latest Ubuntu cloud images and typically detect issues within the same day of the security patch deployment. This latest issue with Monit has been difficult to address since an important component of AppScale’s code needed to be changed. The next version of AppScale fixes the issue. At this time, it is going through regression testing.

Disabling unattended-upgrades

In case you decide that you do not want auto-updates to run and would like to disable them, you can do so by either uninstalling the unattended-upgrades package or modifying its configuration /etc/apt/apt.conf.d/20auto-upgrades to include the following lines:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "0";

Alternatively, you can also instruct unattended-upgrades to blacklist some packages, thus preventing them from being upgraded automatically. Below is an example of how you can blacklist packages:

Unattended-Upgrade::Package-Blacklist {
 	 "monit";
	};

For more information, view the full documentation for Ubuntu Automatic Security Updates.

Summary

To reiterate, if you are experiencing issues with running AppScale or applications within AppScale, that may be due to the recently discovered bug in Monit. The solution is to downgrade Monit and temporarily prevent future Monit upgrades on *all* instances of your deployment with the following:

sudo apt-get install monit=1:5.16-2
sudo apt-mark hold monit

Once the next version of AppScale is available, future upgrades of Monit should be re-enabled (with 'unhold'). We’ll also update this post with any new information once it becomes available.

If you have questions or need help, contact us on #appscale on IRC or post a message in our community forum.

Topics: Best Practices

Subscribe to Email Updates

Most Popular

Recent Posts