You can create an Amazon EC2 deployment for an active AWS environment. When you create an EC2 deployment, you define the engine type, version, and configuration to deploy to the Amazon VPC specified in the environment. You also specify the number of engine instances to deploy. Each engine instance runs on a dedicated
EC2 instance. When you start an EC2 deployment, Control Hub connects to the Amazon VPC specified in the environment and then creates an AWS CloudFormation stack. AWS CloudFormation provisions the group of EC2 instances in the VPC and then deploys and launches one StreamSets engine instance on each EC2 instance. AWS CloudFormation manages the provisioning and monitoring of the EC2 instances. Control Hub simply receives the status of the deployed
StreamSets engine instances and sends any updates to CloudFormation. When you stop an EC2 deployment, CloudFormation deletes the existing EC2 instances. Important: You are responsible for all costs from AWS incurred by the resources provisioned by Control Hub. StreamSets strongly advises against directly modifying the provisioned resources in AWS. Doing so may cause unexpected errors. For more information about AWS
CloudFormation, see the AWS CloudFormation documentation. Before you create an Amazon EC2 deployment, you must complete several prerequisites. Each provisioned Amazon EC2 instance is set up with the following software, based on the selected engine
type. Note: If you need to set up the provisioned instances with additional software, you can define an
initialization script for the deployment.
PrerequisitesBefore you create an Amazon EC2 deployment, complete the following prerequisites: Create an AWS environmentCreate and activate an AWS environment in Control Hub, as described in AWS Environments.Configure an instance profileAsk your AWS administrator to configure an instance profile in AWS to associate with the provisioned EC2 instances. If a default instance profile is defined for the parent AWS environment, you can skip this prerequisite and simply use the default. If a default is not set or if you'd like to override the default for the deployment, see Configure Instance Profiles for EC2 Instances.Optionally, create an EC2 key pairControl Hub does not use or require an EC2 key pair to access the EC2 instances. However, if you plan to connect to the instances using SSH, ask your AWS administrator to create an Amazon EC2 key pair to associate with the provisioned EC2 instances. You can use an existing key pair or create a new key pair to be used exclusively by the deployment. You’ll select the key pair when you create the deployment. For details on creating a key pair, see the Amazon EC2 documentation.Optionally, set up an external resource archiveWhen your pipelines require external resources and when you plan to deploy multiple engine instances, you must set up an external resource archive that all engine instances can access. When your pipelines do not require external resources or when using a single engine instance to get started with StreamSets, you do not need to complete this prerequisite.You typically configure a deployment to use an external resource archive when you are ready to move to production, after you have finished building your pipelines and have finalized the list of external resources that your pipelines require. For more information, see External Resources.Creating an Amazon EC2 DeploymentCreate an Amazon EC2 deployment to define the group of engine instances to deploy to an AWS environment. Important: Before creating a deployment, you must complete the required prerequisites. To create a new Amazon EC2 deployment, click in the Navigation panel, and then click the Create Deployment icon: .Define the DeploymentDefine the deployment essentials, including the deployment name and type, the environment that the deployment belongs to, and the engine type and version to deploy. Once
saved, you cannot change the deployment type, the engine version, or the environment.
Configure the EngineDefine the configuration of the engine to deploy. You can use the defaults to get started.
Configure the EC2 Autoscaling GroupConfigure details about the EC2 instances that will be provisioned.
Configure EC2 SSH AccessOptionally, select the Amazon EC2 key pair to associate with the provisioned EC2 instances.
Share the DeploymentBy default, the deployment can only be seen by you. Share the deployment with other users and groups to grant them access to it.
Review and Launch the DeploymentYou've successfully finished creating the deployment.
Editing an Amazon EC2 DeploymentYou can edit an Amazon EC2 deployment while it is deactivated or active. When you stop a deployment, all existing EC2 instances are deleted. After you edit properties and then restart the deployment, Control Hub uses AWS CloudFormation to provision a new group of EC2 instances and launch a new StreamSets engine instance on each EC2 instance. When you edit a deployment while it is active, existing EC2 instances might be deleted, depending on the following types of edited properties: General deployment or engine propertiesWhen you edit general deployment or engine properties while the deployment is active, AWS CloudFormation continues running the existing EC2 instances. Changes are replicated to all StreamSets engine instances on the next restart of the engines. For example, let's say you edit the deployment to install additional stage libraries on the engine instances, and then you instruct Control Hub to restart all engine instances. Control Hub restarts the StreamSets engine instances on the running EC2 instances, which triggers the installation of the additional stage libraries and the engine property changes. EC2 propertiesWhen you edit EC2 properties while the deployment is active, AWS CloudFormation might replace all of the existing EC2 instances, depending on the change. If a replacement is needed, CloudFormation deletes the EC2 instances in batches to prevent engine downtime. Each batch can contain up to 25% of the total number of instances in the deployment.For example, if you edit the deployment to increase the number of engine instances from 2 to 3, CloudFormation provisions a new EC2 instance. If you edit a deployment with 12 instances to change the instance type, CloudFormation deletes the existing EC2 instances, three at a time, and provisions new EC2 instances to replace them.To edit a deployment, locate the deployment in the Deployments view. In the Actions column, click the More icon ( ) and then click Edit.Tracking URLWhen you view the details of an active Amazon EC2 deployment, you can access a tracking URL to the AWS Management Console. Use the URL to view additional information about the AWS resources automatically provisioned for the StreamSets deployment. To access the tracking URL, click an Amazon EC2 deployment name in the Deployments view and then locate the Tracking URL property in the deployment details. Click the URL to open the AWS Management Console. The console displays details about the AWS CloudFormation stack created for your StreamSets deployment. Explore the following tabs to find information about the provisioned resources:
For example, the following image displays the Events tab for a sample CloudFormation stack: Use the AWS Management Console to explore details about each resource and locate errors that might have occurred. Important: Viewing details about provisioned resources in the console can help you troubleshoot deployment configuration issues. However, StreamSets strongly advises against directly modifying the provisioned resources using the console. Doing so may cause unexpected errors. The following topic provides brief tips on finding the most useful information about the provisioned resources. For more details about monitoring an AWS CloudFormation stack, see the AWS CloudFormation documentation. Auto Scaling GroupIn the AWS CloudFormation stack details page, click the Resources tab and then click the ASG link. The AWS Management Console displays details about the auto scaling group. Explore the following tabs to find information about the provisioned resources:
For example, the following image displays the Instance management tab that includes one EC2 instance with a Healthy status: In the Instances section, click an instance ID to view specific details about the EC2 instance, such as the private IP address. For example, the following image displays a sample EC2 instance summary page: Which AWS service is used to monitor EC2 Instances?You can monitor your instances using Amazon CloudWatch, which collects and processes raw data from Amazon EC2 into readable, near real-time metrics.
Which of the following services can be used to track the CPU usage of an EC2 instance?Using Cloudwatch you can monitor your EC2 instances by several criteria, such as network usage, CPU usage, and so on…
What happens when EC2 instance is stopped?When you stop an instance, the following is lost: Data stored in the RAM. Data stored in the instance store volumes. The public IPv4 address that Amazon EC2 automatically assigned to the instance on launch or start.
Which AWS service allows you to monitor the performance of your EC2 Instances to assist in troubleshooting?For Windows, Amazon EC2 offers EC2Rescue, which customers can use to examine their Windows instances to help identify common problems, collect log files, and help AWS Support to troubleshoot your issues. You can also use EC2Rescue to analyze boot volumes from non-functional instances.
|