Infrastructure as Code CloudFormation

A presentation at Amazon Web Services User Group Rome in June 2020

In this session, I will talk about AWS CloudFormation and its main features, advantages and disadvantages. I will discuss the YAML syntax, frameworks that run on top of AWS CloudFormation and last but not least, I'll talk about the ways you can extend CloudFormation functionality through Macros and Resource Providers.

AWS CloudFormation is a service that uses Infrastructure as Code to provision and manages a collection of related AWS and third-party resources. 

YAML syntax improves the readability of templates, this is a very common choice and is not a prerogative of AWS CloudFormation, Ansible and Kubernetes, for example, uses very similar syntax, moreover, the indentation is very familiar to Python programmers.

The JSON syntax instead is very useful for the automatic template generation thanks to Jsonnet, Kapitan, Jinja or Python.

AWS provides the cfn-template-flip tool that allows you to flip JSON templates into YAML and vice versa along with the ability to perform optimizations, cfn-python-lint can instead be used to perform syntax validation and template formatting.

The syntax of a CloudFormation template is very simple, this makes the tool very easy to use, however, to get the best from CloudFormation you need to know how Amazon Web Services works. If you are just getting started you can go very fast with CloudFormation and spent more time understanding Amazon Web Services. 

Knowing an agnostic cloud tool will not make you a multi-cloud DevOps, instead, knowing AWS CloudFormation can help you to understand AWS best practices.

CloudFormation uses a declarative syntax. This means that everything in your template is a declaration of what should be in CloudFormation's state. Since you use resource attributes to declare your infrastructure you must store your configuration in it.

When you launch an instance in Amazon EC2, you have the option of passing user data to the virtual machine to perform automated configuration tasks, but it is hard to declare a configuration in it because you will end on in doing a lot of shell programming which is not as reliable as configuration management should be, instead AWS::CloudFormation::Init can be used.

AWS::CloudFormation::Init is a way to provide metadata on an Amazon EC2 instance for the cfn-init helper script. The configuration is separated into sections. The cfn-init helper script processes the configuration sections in the following order: packages, groups, users, sources, files, commands, and then services but you can change the default order by using configsets.

A configset block consists of a configuration that specifies a set of actions to perform on the machine, you can have many configset in the same init-resource. These include: 

• packages:  to download and install pre-packaged applications and components.
  
• groups: to create Linux/UNIX groups and to assign group IDs.
  
• users: to create Linux/UNIX users on the EC2 instance.
  
• sources: to download an archive file and unpack it in a target directory on the EC2 instance. 
• files: to create files on the EC2 instance.
• commands: to execute commands on the EC2 instance.
  
• services: to define which services should be enabled or disabled at launch.
  

Let's try to make comparisons. User data give you greater control but for the file to parse as valid JSON/YAML code, any special characters such as double quotes must be escaped within the template, on the other hand, AWS::CloudFormation::Init is easiest to read, roll back automatically on failure, but give you less control. User data script is executed only when the virtual machine is launched while cfn-init can upgrade the machine without recreating the resource.

The AWS::CloudFormation::Init block can be specified in the Metadata field in a virtual machine definition, UserData can be specified in the Properties field. You can provide both AWS::CloudFormation::Init and UserData at the same time.

Resources: 
  Ec2Instance: 
    Type: AWS::EC2::Instance
    Metadata: 
      AWS::CloudFormation::Init: 
        config: 
          packages: 
            :
          groups: 
            :
          users: 
            :
          sources: 
            :
          files: 
            :
          commands: 
            :
          services: 
            :
    Properties: 
      UserData:
        Fn::Base64:
            !Sub |
              #!/bin/bash -x
              # Signal the status from instance
              /opt/aws/bin/cfn-signal -e $? -r "Build Process Complete" '${WaitHandle}'
      ImageId: 'ami-1234'

AWS::CloudFormation::WaitCondition can be used when you need to coordinate stack resource creation with configuration actions that are external to the stack creation or to track the status of a configuration process. You need to create also a WaitConditionHandle that returns a pre-signed URL you can use with cfn-signal to notify execution state to CloudFormation.

WaitHandle: 
  Type: AWS::CloudFormation::WaitConditionHandle
  Properties: {}
  
WaitCondition: 
  Type: AWS::CloudFormation::WaitCondition
  DependsOn: 'Ec2Instance'
  Properties: 
    Handle: 
      Ref: "WaitHandle"
    Timeout: '300'

Parameters allow you to create many stacks from the same template. A template is a specification of the AWS resources to be provisioned, a stack instead is a collection of AWS resources that have been created from a template. You can write a generic template and control which resources to create into a stack by using parameters. For example, you can enable the high availability in production and disable it in the development environment.

Below is an example of an input parameter to a template. You can define constraints over length and type, you can use regular expressions to validate the input value.

Rules allow you to define constraints over different parameters, these constraints can also be used to define best practices.

In the below example you can recognize two rules, one check that m1.small machine is used in the test environment, another ensures that m1.large machine is used in production.

Templates in which best practices are defined can be made available as products in AWS Catalog. This allows developers to create a compliant infrastructure, you can prevent any change to the rules by using IAM policies.

Rules:
  testInstanceType:
    RuleCondition:
      Fn::Equals:
      - Ref: Environment
      - test
    Assertions:
    - Assert:
        Fn::Contains:
        - - m1.small
        - Ref: InstanceType
      AssertDescription: For the test environment, the instance type must be m1.small
  prodInstanceType:
    RuleCondition:
      Fn::Equals:
      - Ref: Environment
      - prod
    Assertions:
    - Assert:
        Fn::Contains:
        - - m1.large
        - Ref: InstanceType
      AssertDescription: For the prod environment, the instance type must be m1.large

Outputs are the return values of your CloudFormation template, they can be exported and then used in another template. The exported name must be unique in the region, it can be imported into any CloudFormation template of the same region. When you use ImportValue intrinsic function only the value is imported not the whole state.

Mappings allow you to create generic templates, and define a complex configuration using a key-value map. For example, you can have a region-independent template as it is shown in the below slide. As you know each AMI has its identifier that is only valid in the given region, Mappings allows you to specify for each Region which AMI id to use.

CloudFormation fully manages the infrastructure state. It manages concurrent state changes and resources creation, update, and deletion.

You can create change-sets and have a full overview of the operations that will be performed during the execution of the template. In case of an error, the system rollbacks the infrastructure to the last working state.

CloudFormation has a Drift Detection feature that gives you the capability to discovery out-of-band changes in your infrastructure, it does not perform reconciliation. CloudFormation does not reconcile the state CloudFormation knows about with the real-world infrastructure.

CloudFormation ecosystem is very large, there are many tools and framework build on top of CloudFormation. Also, thanks to AWS's strong focus on backwards compatibility there are many ready-to-use CloudFormation templates, so you rarely have to start from scratch. This greatly reduces time to market.

StackSets feature gives you the capability to create, update, or delete stacks across multiple accounts and regions with a single operation. This feature is useful for those organizations that need to manage hundreds of accounts, deploy global infrastructures or create a disaster recovery solution.

Below are the advantages and disadvantages of using CloudFormation that I have collected from the community:

Advantages:

• stack sets: allow you to have multi-account and multi-region deployments;
• ui debugging: you can use the GUI to perform debugging activities;
• runs in the cloud: operations are performed even in case of network issues;
• ecosystem: you can use many tools and frameworks to meet your needs;
• tons of examples: you will never start from scratch;
• cross-referencing: you can reference value without to import the whole state;
• custom resources: you can extend CloudFormation capabilities with AWS Lambda;
• rollback: restore the last working state when an error occurs, improves resilience and reduces down-time;
• macros: allows you to define transformations that can be applied to the template;
• resource provider: allows you to create third-party resources.
  

Disadvantages:

• confused cli: Wrappers/Makefile are usually used to avoid commands with many parameters, you can also use the GUI or SDK to deploy your stacks;
• no reconciliation: although it gives you the ability to detect out-of-state changes, it does not restore them;
• no open-source: you cannot contribute to CloudFormation's core capability, it has a plug-and-play capability that gives you the ability to extend the functionality by using AWS Lambda functions;
• no modules: you can import existing templates from Amazon S3 and create a Nested Stack, but you cannot import a template as a module from a git repository;
• verbose: the result template has many repetitions, this because of missing modules;
• no programmable: you cannot use software constructs, it is more Infrastructure as Text than Infrastructure as Code;
  

AWS Specific vs Not Multi-Cloud. CloudFormation is not generally used as a multi-cloud tool, although you can use it as such. I think there are advantages and disadvantages to using a multi-cloud tool or native-cloud one.

Native-tools have better integration with the system for which are designed - CloudFormation, for example, is integrated with others services such as AWS Catalog, AWS Lambda, you can ask AWS Support to inspect your templates - on the other hand, a multi-cloud tool allows you to design the whole infrastructure by avoiding Frankenstein solution. I am therefore very excited about the announcement of CloudFormation Providers and third-party resources support.

The advantages and disadvantages in the previous slide are from the IaC community, actually thanks to features such as custom resources, macros and resource provider many of the drawbacks no longer apply, or almost apply.

Macros enable you to perform custom processing on templates. First, you need to create the macro itself, and then you can use the macro to perform your transformations.

To create a macro definition, you need to define the following:

Transform: AWS::Serverless-2016-10-31

Resources:
  MacroFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.6
      CodeUri: source/
      Handler: macro.handler
      Role: !GetAtt TransformExecutionRole.Arn
      Timeout: 300

  Macro:
    Type: AWS::CloudFormation::Macro
    Properties:
      Name: MyCustomMacro
      FunctionName: !GetAtt MacroFunction.Arn

Whereas you can see there is an AWS Lambda function to perform the template processing and a resource of type AWS::CloudFormation::Macro, which enables users to call the Lambda function from within AWS CloudFormation templates.

If you need to process a section, or snippet, of a template, reference the macro in a Fn::Transform function located relative to the template content you want to transform, instead if you need process an entire template, reference the macro in the Transform section of the template. 

The previous snipped for example uses AWS::Serverless-2016-10-31 transform to create MyCustomMacro.

CloudFormation does not have native support for modules, that's why I have created the Template Macro.

Template Macro adds the ability to create CloudFormation resources from existing templates, you can reuse your configurations and manage a group of related resources as if they were a single resource. Check the Template Macro docs if you need further informations. 

Below a Virtual Private Cloud with two private and public Subnets is created, together with a NAT Gateway, NACLs, Security Groups, an Internet Gateway, Route Tables and so on, all this in a single resource.

CloudFormation does not have by default the capability to use any software constructs, that's why I have created the Troposphere Macro.

Troposphere Macro adds the ability to define your infrastructure by using Python language in a CloudFormation template, you can use if-then-else, for-loops and so on. Check Troposphere Macro docs if you need further information. 

In the below example four virtual machines are created by looping over a list of names.

There are more AWS CloudFormation macros you can use, please check Macro Repository if you need further information.

• Count macro allows you to specify multiple resources of the same type.
• PyPlate macro allows you to run arbitrary python code in your CloudFormation templates.
  
• Boto3 macro adds the ability to call AWS API using boto3.
  
• AWS::Serverless is a macro to use AWS Serverless Application Model (AWS SAM).
  

By using the AWS CloudFormation Registry, you can submit, discover, and manage resource providers, it can be used to manage third party resources in the same way as native AWS resource providers.

To create third-party resources in your infrastructure you need to execute cfn-init-command and provide the name of your resource type, then it asks for the language to use in code generation and finally, it initializes the project by using docker environment.

Resource providers are treated as first-class citizens within CloudFormation; you can use CloudFormation capabilities to create, provision, and manage these custom resources in a safe and repeatable manner, just as you would any AWS resource. 

If you have any question please use the comments section.