Secured EMR Cluster Deployment

Cluster Configuration

• Select Security Configurations from Amazon EMR.
• Provide a name in Name section.
• Enable Enable at-rest encryption for EMRFS data in Amazon S3. Select SSE-S3.
• Select Enable at-rest encryption for local disks and provide the respective key provider type. For example, select Key provider type as AWS KMS and AWS KMS customer master key as EMR/KMS.
• Select Enable in-transit encryption, and provide the certificates.
• Enable authentication by selecting the Enable Kerberos Authentication checkbox.
• Select Authentication as External KDC, set the ticket lifetime and (optional) cross-realm between two different cluster KDCs.

• Click Create.

Software and Steps

• Select the EMR release version as 5.28.1 from drop-down list and select the required applications for Infoworks.
• Select the Use multiple master nodes to improve cluster availability checkbox.

• To enable EMRFS, select Enter Configuration under Edit software settings and add the following:

[{"classification":"emrfs-site","properties":{"fs.s3.consistent.retryPeriodSeconds":"10","fs.s3.consistent":"true","fs.s3.consistent.metadata.read.capacity":"600","fs.s3.consistent.retryCount":"5","fs.s3.consistent.metadata.write.capacity":"300","fs.s3.consistent.metadata.tableName":"EmrFSMetadata"}}

 ,{

      "Classification": "hive-site",

      "Properties": {

        "javax.jdo.option.ConnectionURL": "jdbc:mysql://aurora-iw-test-312.cluster-cwldlowxaaaa.us-east-1.rds.amazonaws.com:3306\/hive?createDatabaseIfNotExist=true",

        "javax.jdo.option.ConnectionDriverName": "org.mariadb.jdbc.Driver",

        "javax.jdo.option.ConnectionUserName": "admin",

        "javax.jdo.option.ConnectionPassword": "password"

      }

    }

]

Hardware

The hardware blade includes the networking, AZ/Subnet and instance type sections.

• Select Uniform instance groups for same type of instances and purchasing options for each node type. Other Option - instance fleets for different instance type (mostly used for Spot Instances type).
• Select VPC from the drop-down list in the Network section and select the appropriate subnet in the Subnet section.
• Root device EBS volume size - Set the root device volume from the range of 10-100 GB.
• Configure the machine type of Master and Core nodes. Infoworks recommends minimum of m4.4xlarge for Master Node and m4.2xlarge for Core nodes and Task nodes.

Following are the recommended Minimum Machine type for Master, Core and Task nodes.

M4.4xlarge or similar of 32 vCPUs with 64GB of RAM for Master node.

M4.2xlarge or similar of 16 vCPUs with 32GB of RAM for Core and Task nodes.

• Enable Cluster Scaling.

NOTE For setting the scaling metrics, see the AWS Documentation.

• Click Next to proceed with General Cluster Configurations blade.

General Cluster Configurations

• Provide the naming convention to the cluster in Cluster name section.
• Termination protected - This option avoids accidental deletion of the instances.
• Create tags for the resources in the Tags section.

• Download and store the following file in S3: https://infoworks-setup.s3.amazonaws.com/emr-configurations/emr-5.28.1/usercreation.sh.
• In the Bootstrap Actions section, select the Custom actions and click Configure and add.

• In Script location, select the S3 location where the downloaded script is stored.

• Click Next to proceed to the final blade, Security.

Security

• In the Security Options section, select the respective EC2 key pair to access the cluster through SSH.
• Select default for Permissions and EC2 Security Group to be automatically created/updated by EMR.
• Enlarge Security Configuration to configure security to EMR cluster.
• Select the name of security configuration as created in the Cluster Configuration section.
• Enter the realm and password for the KDC.

• Click Create Cluster.
• To deploy Infoworks edge for EMR, see Deploying Infoworks for EMR Using Infoworks AMI.