Leverage a Fully Managed Solution using AWS Batch
During this section we will use Cloud9 to build up a docker image that we will then use to upload to Elastic Container Registry.
We will use this docker image later on with AWS Batch to run our portfolio valuation simulations and leverage a Spot fully managed solution.
Using the Cloud9 Environment
AWS Cloud9 comes with a terminal that includes sudo privileges to the managed Amazon EC2 instance that is hosting your development environment and a preauthenticated AWS Command Line Interface. This makes it easy for you to quickly run commands and directly access AWS services.
An AWS Cloud9 environment was launched as a part of the CloudFormation stack (you may have noticed a second CloudFormation stack created by Cloud9). You’ll be using this Cloud9 environment to execute the steps in the workshop.
Find the name of the AWS Cloud9 environment by checking the value of cloud9Environment in the CloudFormation stack outputs.
Sign in to the AWS Cloud9 console.
Find the Cloud9 environment in Your environments, and click Open IDE.
Please make sure you are using the Cloud9 environment created by the workshop CloudFormation stack!
Take a moment to get familiar with the Cloud9 environment. You can even take a quick tour of Cloud9 here if you’d like.
Create and upload the docker image to ECR
On the terminal run the following commands. This will download the repository of code we will use to build our image.
Login into Elastic Container Registry (ECR) on the terminal. The result of the command will show a Login Succeeded
Create a repository named monte-carlo-workshop.
When executing next section you will need to use use the image generated. Please take note the result above for the Monte carlo image name. It should be something like: ACCOUNT_NUMBER.dkr.ecr.AWS_REGION.amazonaws.com/monte-carlo-workshop:latest
Create the docker image and upload it to the ECR registry
You are now ready to use the generated docker image within AWS Batch. If you are already familiar with docker you can try to run the docker image on cloud9 as an optional exercise.
Leveraging AWS Batch
Before we set up AWS Batch environment. Let’s submit a valuation to the SQS queue. Go to the website URL we used in Lab3 and place a new simulation. The website url can be found on the cloudformation -> monte-carlo stack -> Outputs -> WebInterface. You should be able to see at least one message in the SQS console.
Go to the AWS Batch Console. The following instructions use the first-run wizard. If the wizard does not show, replace the path at the end of the URL with /wizard. e.g. https://ap-southeast-2.console.aws.amazon.com/batch/home?region=ap-southeast-2#/wizard
Select/Enter the following values
- How would you like to run your job ? : No job submission and hit Next
- Compute environment name : montecarlo-batch-worker
- Service role and EC2 instance role : Leave it defaulted to “Create a new Role”
- Provisioning Model : Spot
- Maximum bid price : 100
- Spot fleet role : Select the role containing your workshop name
- Allowed instance types : optimal
- Minimum vCPUs : 0
- Desired vCPUs : 0
- Maximum vCPUs : 20
- VPC Id : VPC as created earlier
- Subnets : Any two subnets in the VPC
- Security groups : Security Group as created earlier
- Job queue name : montecarlo-batch-worker
Click Create . It will take less than one minute for the setup to complete. Once complete, click on View Dashboard
Go to Job Definition , hit Create and enter the following details
- Job definition name : montecarlo-queue-processor
- Job Attempts : 3
- Execution timeout : 120
- Job role : Select the one that appears in drop down, as created during setup
- Container image : ACCOUNT_NUMBER.dkr.ecr.AWS_REGION.amazonaws.com/monte-carlo-workshop:latest
The container image name value should be the one you captured from in the previous step. if required, go back to the cloud9 console in another tab and copy the content from there.
- vCPUs : 2
- Memory (MiB) : 512
- Add an environment variable with Key: REGION and Value name the region you are using, example us-east-1
- Add an environment variable with Key: BATCH_MODE and Value = true
- Leave everything as default and click Create job Definition
Now we are ready to submit a job (with the definition created above) and run it against the compute environment created above. Go to Jobs, select Submit job and enter the following details
- Job name : montecarlo-batch-first-run
- Job definition : Select the one created above
- Job queue : Select the one created above
- Job Type : Select Single
- Leave everything as default and click Submit Job
This will create the EC2 Instances using Spot price as bid during creating the compute environment. This process may take 2-3 minutes. When you refresh the screen, you will see the status of the job getting transitioned from submitted > pending > runnable > starting > running.
Evaluate the Results
- Once the job reaches Running state, check your S3 Bucket. In a few minutes you should see results start appearing the bucket.
- If you monitor the SQS queue for messages you should see them being picked up by the worker container.
- Once the job is completed, check the AWS Batch -> Jobs Dashboard and click on the job to display the job summary showing the number of attempts and link to the logs. Click on View Logs
- Click on the links to the logs and confirm everything went as expected
- Use AWS QuickSight to build visualizations, perform ad-hoc analysis, and quickly get business insights from your data. You will need to create a json manifest file with your Amazon S3 data location. Use the following template as a starting point:
You’ve completed Lab 4, Congrats!
Teams set bespoke change management standards per service, underpinned by standard AWS guidelines.
All production environment changes are reviewed, tested and approved. Stages include design, documentation, implementation (including rollback procedures), testing (non-production environment), peer to peer review (business impact/technical rigour/code), final approval by authorised party.
Emergency changes follow AWS incident response procedures. Exceptions to change management processes are documented and escalated to AWS management.
AWS Security monitors newsfeeds/vendor sites for patches and receives customer intelligence via http://aws.amazon.com/security/vulnerability-reporting/.
AWS customers are responsible for all scanning, penetration testing, file integrity monitoring and intrusion detection for their Amazon EC2 and Amazon ECS instances/ applications. Scans should include customer IP addresses (not AWS endpoints). AWS endpoint testing is part of AWS compliance vulnerability scans.
• Port scanning attacks
• Usage (CPU, processes, disk utilization, swap rates, software-error generated losses)
• Application metrics
• Unauthorized connection attempts
Near real-time alerts flag incidents, based on AWS Service/Security Team- set thresholds.
Requests to AWS KMS are logged and visible via the account’s AWS CloudTrail Amazon S3 bucket. Logs provide request information, under which CMK, and identify the AWS resource protected through the CMK use. Log events are visible to customers after turning on AWS CloudTrail in their account.
1. Activation and Notification Phase
2. Recovery Phase
3. Reconstitution Phase
To ensure the effectiveness of the AWS Incident Management plan, AWS conducts incident response testing, providing excellent coverage for the discovery of defects and failure modes as well as testing the systems for potential customer impact.
The Incident Response Test Plan is executed annually, in conjunction with the Incident Response plan. It includes multiple scenarios, potential vectors of attack, the inclusion of the systems integrator in reporting and coordination and varying reporting/detection avenues.
There are always the kind of tasks that need to run periodically, either to analyze and process information (like fraud detection), or to simply do things like send email reports. But for that, we need to have a tool to schedule computer resources and, of course, the script.
But what if we only need to worry about coding the script?
Introducing AWS Batch. It’s a free service that takes care of batch jobs you might need to run periodically or on-demand. And you only pay for the resources you use.
In this tutorial, you’ll learn how to kick off your first AWS Batch job by using a Docker container.
What Is AWS Batch? A Quick Overview
Before we dive in, let’s do a quick overview of the subject at hand. AWS Batch is a service that lets you run batch jobs in AWS. You don’t have to worry about installing a tool to manage your jobs. AWS Batch will do that for you.
There are a lot of features you might not need when you’re first starting out, but let’s explore a few of them anyway:
- Instances will run only for the time that’s needed, taking advantage of the per-second billing. You can also lower your costs by using spot instances.
- It’s possible to configure how many retries you’d like for any job.
- It offers queues where you send the jobs. Each queue could be configured with a certain priority so you can configure which jobs will run first. You can also have queues that use better resources to speed up the process.
- It supports Docker containers so that you can focus only on your code.
- And more…
So, enough theory. Let’s get our hands dirty.
Kick Off Your First Job
Before we start, there are some prerequisites that will make this tutorial easy to follow, and it will include some good practices in regards to security. If you think you need more details, you can check the setup page in AWS’s official docs.
- Have an AWS account.
- Create an IAM user with administrator permissions. To do this, you can just follow this tutorial. I recommend you give granular permissions to the user that will do the provisioning.
- Install and configure AWS CLI.
If something from the above doesn’t work, it might be because a permission is missing, or the CLI is not configured properly. I’ll let you know exactly what’s needed in the following steps.
Go to AWS Batch
Log in to your AWS account and look for AWS Batch in the initial screen, or you can go directly by using this link.
You’ll see a screen like the following:
Click the “Get started” button. Then, this next screen will appear:
Click the “Skip wizard” button. We’re not going to follow this wizard because I want to explain each step to you. Also, after this, you’ll probably use AWS CloudFormation or something else to provision, not the wizard.
Create a Compute Environment
The jobs will run on a compute environment. Here, you’ll configure the instance type, family, and some other things that we’ll see in a bit.
It’s important that you know we’re not going to create any instances now. AWS Batch will create one when it’s needed. You can also configure things to create instances right away, speeding up job scheduling, but we won’t tackle that in this post.
Click the “Compute environments” link that appears in the left menu. You’ll see the following screen:
Instance Type and Permissions
Now click the “Create environment” blue button so you can start defining the compute environment. You’ll start configuring the environment in the following screen:
For simplicity, we’re going to choose all default values. You just need to name the environment. I called it “first-compute-environment.”
You don’t have to worry about creating a service or instance role right now. Just choose the option “Create new role” for both, and AWS will create them for you with the proper permissions. It will help you see which permissions are needed and adjust them if you want to.
Leave the EC2 key pair blank because we don’t need to access the servers for now.
Scroll down a little bit, and let’s talk about the compute resources section. You’ll see the following screen:
This is where you get to choose if you want to use on-demand or spot instances. For simplicity, let’s choose “On-demand.”
The “Allowed instance types” field is where you define which family type you’d like these environments to create. This is where things get fun because you can create compute environments that are CPU-intensive and choose between C family instance types. Or if there are jobs that are memory intensive, you can choose M family instance types. You’re limiting which instance types can be created. I chose “optimal,” so AWS decides for me which instance is better based on the configuration of job queues.
Now, vCPUs are one of the most important things here in order for your first job to run.
If you’re familiar with running workloads using ECS, you might get confused here. You might configure so many vCPUs that AWS won’t be able to create the environment. And even if there are a few instances running, jobs won’t run until the environment is ready. So keep in mind that vCPUs are virtual CPUs, not CPU units that you configure in a container when running in ECS.
I configured a maximum of four vCPUs. It means that if at some point the cluster has four vCPUs among all instances, it won’t create more. Jobs will run slowly, but your costs will remain controlled. I also put one vCPU as desired, just so it starts creating an instance right now. AWS will adjust this later if needed, and you can change it when submitting a job if you’re in a hurry.
Scroll down a little bit, and you’ll now configure the networking section and tags. You’ll see a screen like this:
Leave the VPC and subnets as default for now. Click the “Create” blue button and wait a bit while the environment is created.
Create a Job Queue
Now you need a queue where you’ll send the jobs to get executed. This queue will be attached to a compute environment so the AWS Batch service will create the resources needed based on the load of the queue. It will use the min, max, and desired vCPUs configuration to know how many instances to create.
Click the “Job queues” link in the left menu and you’ll see the following screen:
Then, you can click the “Create queue” blue button. You’ll see this:
Let’s put a name to the queue so it’s easy to identify. I called it “first-job-queue.”
In the priority, make sure you type a value that lets you play with lower priority queues later. I put “100” in case I need to create a lower priority queue later—say, for example, one with 50.
Enable the job queue. By default, this checkbox will be checked. You should leave it that way.
You now need to connect this queue to one or more compute environments. I chose the one I just created, the “first-compute-environment” one. If there were any other environment, this is where you’d choose it.
Why would I like to have more than one compute environment? Well, it’s useful if you want to speed up a job’s processing time by creating more instances using the spot market. You can have an on-demand compute environment where you always have resources available. And if the load increases, you can create spot instances if there are any available, based on the bid you configured.
Click the “Create queue” blue button.
Create a Job Using Docker
We’re going to use a “hello world” job that AWS evangelists have used for demo purposes. I couldn’t find a repository with all the files they’ve used, so I created one with all the files we’re going to need. You can find it on GitHub here.
Let’s explore what’s in there, as well as why and how to use those files to create our first job in AWS Batch.
We’re going to create a simple job that will pull a Bash script from S3 and execute it. The Dockerfile and the script that does what I just described is located in the “job” folder of the repository.
I won’t explain either the script or the Dockerfile just yet—we’ll just use it. So let’s build the Docker image and push it to the Docker hub. You need to have Docker installed on your machine, a Docker hub account, and a login for your computer.
Let’s build the Docker image. You can skip this step and use my image located here, or you can run the following command and tag the image using your username instead of mine:docker build -t christianhxc/aws-batch-101:latest .
Now, let’s push the image. You need to be logged in with your user ID. And make sure you push the image that has your username in the tag. Run the following command:docker push christianhxc/aws-batch-101:latest
That’s it! You now have the Docker image that will download a Bash script from S3 and run it.
A Bash Script
Let’s create the Bash script. You can use the one I have in the repo. That script simply puts a Fibonacci sequence in a DynamoDB table. It uses an environment variable called FOO to create the series of numbers, and it uses an argument just to print it in the console.
This script is in the root of the GitHub repository I linked before, and it’s called mapjob.sh
Now, because this is outside the scope of AWS Batch, I’m just going to list the actions you’ll need for this guide to work. We’ll need to do the following:
- Create a DynamoDB table in the Virginia region with primary key of “jobID”. Mine is called “fetch_and_run.” If you decide to enter a different name, make sure you change it at the end in the mapjob.sh script.
- Create an S3 bucket in the Virginia region. Mine is called “cm-aws-batch-101.” Don’t make it public.
- Upload the mapjob.sh script in the bucket you just created.
- Create an IAM role for an ECS service task with permissions to the S3 bucket and the DynamoDB table. If you don’t know how to do that, follow these instructions. I called my IAM role “aws-batch-101.” We’ll use this one next.
You’re almost ready to kick off your first job. You already have a script and a Docker image to use.
Let’s create the job definition in AWS and then submit a job.
Create a Job Definition
At this point, you’ve defined the environment where your jobs will run and the queue, which means AWS takes care of creating resources only when they’re needed. Now you need to run the job definition. And this is where things get more interesting.
Click the “Job definitions” link in the left menu and you’ll see the following screen:
Click the “Create” blue button and let’s start defining the job.
Enter any name you’d like. I put “first-job.” We set job attempts to 1. Job attempts is the maximum number of times to retry your job if it fails. And Execution timeout, is the maximum number of seconds your job attempts would run. For this example, we set it to 60 seconds.
Scroll down a bit and let me explain what’s there:
Job role provides a drop down menu where you select the job role. choose the IAM role you created previously; mine is “aws-batch-101.”
Note that: Only roles with Amazon Elastic Container Service Task Role trust relationship will be shown. You can learn more about creating roles with AWS ECS trust relationship here.
Now pick a name for the container image. Like I said before, for simplicity, you can use mine. I called it “christianhxc/aws-batch-101:latest.” These values can’t be changed when submitting a job, but the ones we’re about to explore can be changed.
The command field describes the command passed to the container. It maps to the COMMAND parameter to docker run. Here, we’ll type the name of the script that will run the container and its parameters. Because we can override this value, we’ll leave it as it is right now.
Now, here’s another trick to be able to run a job. Unfortunately, you can’t configure CPU units to a container, only vCPUs. It means that, at minimum, the container will have 1024 CPU units because that’s the equivalent to one vCPU. You can configure the CPU, then, in blocks of 1024. This is important because I entered 256, thinking that this was CPU units, and the job never started. It sticks in the RUNNABLE state if there’s nowhere to run it.
Configure how much memory this container will need. I put 256. Leave the rest as it is.
Submit a Job
You’re now, finally, able to submit a job.
Click the “Jobs” link in the left menu, and you’ll see the following screen:
Click the “Submit job” blue button. Let’s submit one!
Next, name your job submission. I called it “my-first-job.” Choose the job definition and the queue we just created, and choose “Single” as a job type.
Scroll down a little and let’s override some values here:
In here, you’ll need to put the name of the script in the S3 bucket and the Fibonacci number as parameter. But these are only for reference. I used “mapjob.sh 60.” Type in “1” for vCPU and “256” for memory.
Scroll down some because our scripts need environment variables in order to work. Let’s add the corresponding values:
Let’s add the environment variables. For FOO, enter the Fibonacci number. I used 60. For BATCH_FILE_TYPE, put “script”, and for BATCH_FILE_S3_URL, put the S3 URL of the script that will fetch and run.
Click the “Submit job” blue button and wait a while. You can go to the computer environment and changed the desired vCPUs to 1 to speed up the process.
It will start creating one instance. When the instance is ready to process a job, the job will transition from RUNNABLE to SUCCEEDED.
And you’ll see a new entry in the DynamoDB table.
You can keep submitting jobs and change the FOO var to generate a different sequence of numbers. When you don’t submit any other job, AWS Batch will terminate the instance it created.
It’s Your Turn Now
You now have the basics to kick off a job in AWS Batch. Once you’ve finished this guide, it’s up to you which scripts or code you’ll put in a container. AWS Batch will manage all the infrastructure, scheduling, and retries for you.
Now the challenge is in how to code your application so that you can submit several instances of a job. AWS Batch will run them as you submit them and will scale out/in when it’s needed, saving you some money. You can start by migrating any existing cronjob, but don’t stop there. Many things could go wrong with your job execution.
The container may be buggy. Performance may be slow. Sometimes, you may need to provision more memory for the job.
To investigate and debug issues like these, you need monitoring tools.
This post was written by Christian Meléndez. Christian is a technologist that started as a software developer and has more recently become a cloud architect focused on implementing continuous delivery pipelines with applications in several flavors, including .NET, Node.js, and Java, often using Docker containers.
About Christian MelendezChristian Meléndez is a technologist that started as a software developer and has more recently become a cloud architect focused on implementing continuous delivery pipelines with applications in several flavors, including .NET, Node.js, and Java, often using Docker containers.
10-15 minutes passed and Natalya was already leaving her office. They went down to the 1st floor. There Natalya asked if Olesya was very busy. To this question, Olesya replied that no, then Natalya suggested celebrating 1 week of work.
And her husband relatively easily reached the remote village where her mother-in-law lived. But that time it was a day off, and at least someone went to the city to shop. Today was Tuesday and it was empty around.AWS Tech Conference
She flirted with stiletto heels to the place of honor at the table, allocated to her by the dude Harry. He dressed up in a tuxedo with a bow tie, poured sparkling champagne into cocked glasses. We got up from the couch.
You will also like:
And now, on a carefree Saturday evening, finished with training, I enter into an unequal battle. With my desire. After taking a shower and completing all the necessary procedures, I leave the bath naked in the hall, where a rug is already spread on the.