Deploying security patches to servers is almost as much fun as managing backup jobs. But everyone has to do it, including companies that have moved their infrastructure to AWS. As we’ve learned with previous posts, Amazon EC2 Systems Manager allows us to use some native AWS tools for management of our EC2 instances, and patch management is no exception.

EC2 Systems Manager allows you to do patch compliance where you can set a baseline and then based on a defined maintenance window a scheduled scan and deployment can be initiated on those EC2 instances. This assumes that you’ve already installed the SSM Agent and setup the basic IAM permissions for the instances to communicate with the Systems Manager service. The details can be found in the previous post.

Create A Baseline

To get started with patch compliance we’ll want to configure a baseline. Open your EC2 dashboard and scroll down to find the “Patch Baseline” service on the left hand side. You’ll notice that a few patch baselines will be created by Amazon to use right away but we’ll create our own so that we understand what they’re doing. You’ll notice that those baselines are listed at “Default Baselines” meaning that if a patch baseline is not assigned to an EC2 instance, the default baseline would be used.

Note: Three of the baselines specifically state that they’re for Amazon, Ubuntu or RHEL. The non-descriptive baseline is a Windows baseline.

Click the “Create Patch Baseline” button.

Give the Baseline a name and a description that will be easily identifiable by your administrative team. Then select the Operating System that the patches will be used for. I’ve chosen Windows for this post.

Next we want to go through the approval rules. This step requires us to pick operating system(s) that we’ll want to scan and/or patch.

Move on to the Classification of the update such as critical, security, rollups etc.

Then select the severity such as Critical or Important. The last section allows you to place a delay in so that your patches aren’t deployed immediately after the release date. The default is 0 days to ensure that patches get applied as soon as possible. Beneath this there is an option to exclude certain patches as well so they aren’t included.

Add a Patch Group

Once your patch baseline is created, select the baseline and select the “Modify Patch Groups” option.

Add at least 1 Patch Group. This is really just a tag that you’re placing on the baseline, but it will be important later.

Next, go to your EC2 instances. Select one or more of your instances and add a tag. In the screenshot below I already have a “Name” tag used to identify this Windows Server instance. I want to add another tag named “Patch Group”. This is important! Amazon usually lets you create any key value pairs that you want, but the “Patch Group” tag is used specifically for patch management. The value of the tag should match the baseline patch group you entered earlier.

Create an IAM Role

In order to run patches on registered instances a new role will need to be created. Open the Identity and Access Manager (IAM) service in AWS and create a new role.

Select the AmazonSSMMaintenanceWindowRole” policy from the list. Type in SSM to filter the list if that helps you out.

Give the role a name and description.

Once you’ve created the role, find the role in the roles page and click trust relationships tab. Click “Edit Trust Relationships”. You’ll need to add a comma to the end of the ec2.amazonaws.com line and then add a new line with:

“Service”:”ssm.amazonaws.com” to the existing policy as shown below. The click update trust policy.

One other piece to mention is that if the user you’re logged in as isn’t an administrator, you’ll need permissions to assign the role we created previously to a maintenance window. Essentially you have to have permissions to add permissions. So if you’re not an administrator, you’ll need to add the PassRole policy to the user account. This post assumes the user logged in is a super user.

Create a Maintenance Window

As we know, our systems can’t usually be shut down whenever we want them to be. We have to define proper outage windows. We can define these in the Maintenance Windows section of the EC2 console under the “System Manager Shared Resources” group.

On the opening screen select the “Create a Maintenance Window” button.

Give the window a name that will identify it. After that, you’ll want to build a schedule for this maintenance window to run. I’m being pretty lazy here and selected a window of every 30 minutes. I imagine you’ll have a much smaller window available to you. You can also set a duration for the window to last and when to stop maintenance routines before the window closes. This ensures that maintenance tasks don’t run over your window by accident.

Once you’ve created your maintenance window, select the object and select the “Register targets” action from the drop down.

Enter an owner for record keeping purposes and then you can either specify the instances associated with this maintenance window, or add them based on tags. Since I’ve already added a tag for my Patch Group, I’m going to associate all instance with my patch group with this maintenance window. You can slice and dice these however you need to.

Next, click the action window again and select the “Register task” option. Here you’ll see a familiar screen if you went through the Run Command post on this site. Select the “AWS-ApployPatchBaseline” document.

In the registered targets section, you should see the maintenance window target has already been selected.

Below this, select the operation, which will be either Scan, or Install. For this post, I’m only doing scans, but most likely you’ll want to do both. One window for scanning which can be done on a regular basis and an install window maybe monthly where the systems will be rebooted to apply patches. You will need to assign the Maintenance Window role with permissions to execute these command which we created earlier. Then select how many instances to run the commands on at once and when to stop if errors occur.

Monitor Patch Compliance

You’re set up at this point to scan your instances for missing patches. If you go to the Patch Compliance service from the EC2 console, you’ll start getting info about your instances once you’ve gone through at least 1 maintenance window. The scan task we’ve created in this post will have run and you can see if you have out of data instances. In my case I have only 1 instance and it’s missing patches.

If you do nothing else, you’ll only be able to see instances that are missing patches. But you can create another maintenance window with a task to install (instead of scan) patches for you. This will require a reboot of the instance so plan your install window accordingly. When your’edone, you’ll see that the instances are up to date and your next scan window will show everything in the green again.

Also, for any maintenance window you can look at the history to see if the tasks ran successfully or not. This is helpful at first to ensure that your permissions and roles were created correctly.

Summary

Patch management and compliance is a pain, but a necessary part of most environments. EC2 Systems Manager Patch Compliance can help teams manage their patches without the need for a more robust tool like Microsoft’s System Center Configuration Manager or WSUS server in the cloud. Stay tuned for more EC2 Systems Manager Posts.