AppScale 3.1 saw the introduction of the Azure agent in a beta capability. AppScale 3.2 brings a new and improved Azure agent which graduates to production ready our support of Azure. One of the main changes, was the introduction of Scale Sets in the agent code. Scale Sets are an Azure resource which allows us to deploy and manage identical sets of virtual machines using the same template across all of them. This transition to Scale Sets, allowed AppScale to deal with large deployments (with thousands of running AppServers) more quickly and efficiently. In this blog, we will cover the agent code found in the AppScale Tools repository. This agent code is shared between the tools and AppScale, where it is used for auto scaling.
Credentials to access the Resource
The agent needs the Azure credentials and the resource names/URIs where you would like to start the deployment. AppScale uses this information on your behalf to interact with the cloud and operate correctly on the desired deployments.
Azure Credentials: These parameters are used to identify the agent with the Azure platform, and can be shared across multiple deployments.
Azure Deployment specifics: These parameters are used to identify and interact with a specific deployment.
All the above is configured in the AppScalefile.
Scale Sets instances do not have public IPs associated with them, which is safer for backend services. However, we still need the ability to create instances with public IPs as the entry points to each AppScale deployment. With the release 3.2, we empowered the agents with the capability to specify whether or not a public IP address is needed for the machine being created based upon the machine’s role. For the Azure agent specifically, only the load balancer machines require public IPs. Public IPs are used by browsers and software to reach your applications from the outside world.
A single node deployment is setup using a single instance with the load balancer role and a public IP address. The agent logic to create single instances is different than the logic used to manage Scale Sets. For a single instance we create or update a virtual machine after having created the required resources Virtual Network, Public IP Address and a Network Interface in this specific order.
Creating Scale Sets
The Azure agent automatically uses Scale Sets for multi-node deployments. The AppScale Tools makes two separate agent calls, one to create and provision the nodes with load balancer roles and another to create the rest of the nodes.
The Azure agent code added few methods to handle Scale Sets. First we need to create the Scale Set:
def create_scale_set(self, count, parameters, resource_name,
The creation of a Scale Set has a few pre-requisites:
- Define LinuxConfiguration: It describes the Windows Configuration of the OS profile. Here we pass the SSH key we want to set up the deployment with and disable username/password type authentication.
public_keys = [SshPublicKey(path=auth_keys_path,
ssh_config = SshConfiguration(public_keys=public_keys)
- Define VirtualMachineScaleSetVMProfile: It sets the Scale Set virtual machine’s OS, storage and network profile.
- VirtualMachineScaleSetOSProfile: Here we set the name of the Scale Set Virtual Machine, the admin username and the linux configuration defined above.
- VirtualMachineScaleSetStorageProfile: Here we set the VirtualMachineScaleSetOSDisk, where we pass in the custom AppScale image we need to start the virtual machine with and the operation system and caching types.
- VirtualMachineScaleSetNetworkProfile: Here we pass in the reference to the Subnet and the Virtual Network that we created for the load balancer VM in order that the Scale Set virtual machines and the load balancer can communicate with each other.
- Define Virtual Machine Scale Set Compute Sku: Here we specify the instance type or the size and number of virtual machines being created within the Scale Set.
sku = ComputeSku(name=parameters[self.PARAM_INSTANCE_TYPE],
- Set over provisioning: Here by default we set over provisioning to false, because we use instead, AppScale’s autoscaling features to determine whether or not and when additional are required.
vm_scale_set = VirtualMachineScaleSet(sku=sku,
resource_group, scale_set_name, vm_scale_set)
Once the Scale Set is created, modifying it is a simple matter of updating the current number of machines. For example :
def add_instances_to_existing_ss(self, count, parameters)
creates an updated ComputeSku and passes it to VirtualMachineScaleSet and updates the number of instances within the Scale Set with
resource_group, vmss.name, scaleset)
The example below shows a multi-node deployment. The node/s with the load balancer role are created as a regular virtual machine outside of the Scale Sets and would be assigned a public IP. The rest of the non-load-balancer machines are created within a Scale Set with a maximum capacity of 20 machines each and assigned just the private IPs.
For a three node deployment, with an ‘appscale up’ using the latest 3.2 AppScale Tools, you will see output similar to the following.
Starting AppScale 3.2
Verifying that SSH key exists locally.
Configuring network for machine/s under resource group 'appscale-scale-sets' with storage account 'appscalescalesets' in zone 'West US'
Creating/Updating the Virtual Network 'appscale-keyname' ...
Creating/Updating the Public IP Address 'wandering-disk-5567' ...
Creating/Updating the Network Interface 'wandering-disk-5567' ...
Creating a Virtual Machine 'wandering-disk-5567' ...
Azure load balancer VM is available at 126.96.36.199
Please wait for AppScale to prepare your machines for use. This can take few minutes.
Creating/Updating the Virtual Network 'appscale-keyname' ...
Creating a Scale Set wispy-limit-3344-scaleset-2vms with 2 VM(s)...
Scale Set 'wispy-limit-3344-scaleset-2vms' with 2 VM(s) has been successfully configured!
As seen in the above example output, the Scale Sets and the Load balancer virtual machine share the same Virtual Network between them. Any changes made to the Scale Set results in an update to the Virtual Network.
With the AppScale Tools 3.2, the command ‘appscale status --verbose’ will give the following output, with the load balancer having the public IP and all other roles having private IPs.
[email protected]:~# appscale status --verbose
Below is a screenshot of the Azure Portal which gives the overview of the resource group under which the resources were created. You can see how the ‘polished-tooth-1606’ is our load balancer virtual machine and the ‘royal-king-8375-scaleset-2vms’ is the Scale Set created and contains two virtual machine instances.
In the screenshot below, you can see an overview of the instances of the Scale Set ‘royal-king-8375-scaleset-2vms’. The instance IDs and the computer/instance names are the two important aspects of the Scale Set instances which AppScale uses to update a Scale Set to add or remove specific instances (as in case of autoscaling).
The Azure agent creates Scale Sets with a capacity of 20 instances each. The Azure Scale Set has a create/update policy, so any changes in the parameters during an update can add or remove instances. This new agent deals with autoscaled instances by checking if any of the existing Scale Sets have the capacity to accommodate them and if not, it creates a new Scale Set with the number of instances to be autoscaled.
As an example, we use the same three node deployment as shown in the screenshots above, and increase the minimum appengine count to trigger autoscaling. When the autoscaler recognizes the need to autoscale an instance to accommodate more appservers, it makes a call to the Infrastructure Manager which in turn calls the agent from tools to spin up additional VM instances. The Azure agent does an evaluation of the present capacity of the Scale Sets to accommodate the additional VMs and when/if Scale Set is not at full capacity (as in this case), the agent just updates it by passing the new capacity to the existing Scale Set as seen below:
In the below screenshots, we can see how the scale set instance count is being updated from 2 -> 3 when autoscaled instances are added to existing scale sets.
As always we would love to hear any feedback to make this agent better! This includes suggestions for a better random name generator (we use Haikunator now) which gives better, deeper & whackier names than ‘polished-tooth’ or anything similar.