Add query group ASG metrics to query group CW dashboard

Thu, 27 May 2021 17:22:27 +0000

When looking at the query group dashboard, it is helpful to know what "state" the ASG was in when referencing the various dashboard metrics. Currently, I must do one of the following: 1) reference these metrics in the ASG monitoring tab (e.g., https://us-west-2.console.aws.amazon.com/ec2autoscaling/home?region=us-west-2#/details/my-query-group?view=monitoring), 2) create a CW dashboard of my own, 3) modify the existing Datomic CW dashboard to add this chart. The ASG metrics answer questions like: 1) how many instances was my query group running? 2) was my query group trying to scale out or in? 3) why did my query group not scale out or in?

I would find the following ASG metrics (in order of importance) useful additions to the current query group dashboard.

In Service Instances (Count)
Minimum Group Size (Count), Maximum Group Size (Count), and Desired Capacity (Count)
Pending Instances (Count)

Make DesiredCapacity an optional parameter in the query group CF template

Wed, 24 Mar 2021 17:24:25 +0000

Currently (March 2021) the Datomic query group CF template has the DesiredCapacity parameter marked as required. Because of this, during a CF template update, CF will always set the DesiredCapacity to the CF parameter for DesiredCapacity. This behavior does not work well in the following situation.

In the query group template I have the DesiredCapacity at 2, and MaxCount at 4. I have an ASG scaling policy to target track 50% average CPU utilization. My query group is currently at the max, because the ASG scaling policy scaled up the group DesiredCapacity to 4 (to match my load). My load has increased to the point where 4 nodes is no longer sufficient. I must update the query group CF template to increase the MacCount. The problem is that when you update the MacCount from 4 to 6 via a CF template update and leave the DesiredCapacity at 2, CF will actually set the DesiredCapacity on the ASG to 2 during the high load situation. This worsens the high load situation.

One option is to query the ASG for its current DesiredCapacity prior to running a CF update and set the CF parameter to the value you retrieved. This approach complicates the update procedure. It also has a "read-modify-write" race condition -- the DesiredCapacity could have changed from the time you read the DesiredCapacity to the time you are performing the CF update.

A second option is to make the DesiredCapacity parameter optional. As per the CF docs, DesiredCapacity is not a required parameter. When it is not present, it defaults to the minimum size of the group. Leaving DesiredCapacity unset will make it so during a CF update, CF will not modify the DesiredCapacity at all -- leaving the ASG "in control."

Datomic Knowledgebase - Recent questions tagged query groups

Add query group ASG metrics to query group CW dashboard

Make DesiredCapacity an optional parameter in the query group CF template