- [Deen] Hello everyone, I'm Deen.
I'm a Senior Analytics Specialist Solution Architect,
focusing on big data services like AWS Glue.
Joining me today is Arun.
He's also an Analytics Specialist Solution Architect.
In this session, we are going to talk about Glue blueprints
and how it supports creating and sharing Glue workflow
for similar use cases within your organization.
Data engineers spends significant effort and time
building data integration pipeline,
performing tasks like extracting, cleaning,
and loading data.
Every time they onboard a new source they need to rinse
and repeat the whole process from the scratch.
Similar to building data integration pipeline,
there are requests to build reusable functionalities
like data partitioning, file format conversion, et cetera,
from various teams.
Every time a request from a new team comes,
data engineer will spend cycles duplicating
and retesting already build functionality for that team.
This approach is slow, expensive, and ineffective.
AWS Glue blueprint addresses this problem
by offering framework for you to build
and share reusable Glue workflows.
Now let's take a closer look
at the process of building Glue blueprints.
AWS Glue blueprint provides organization
a mechanism to develop, share
and reuse complex ETL workflow.
They assist ETL developer to generate
and publish templates of commonly asked ETL workflow
that analysts or other non-developer can consume
and execute jobs without having to write code.
You can see Glue blueprint lifecycle in this slide
with series of steps.
Step one, data engineer identifies data injection pattern
and creates blueprints.
A blueprint in AWS Glue is a zip archive
that consists of a blueprint configuration file,
a layout generator script,
and optional ETL scripts and supporting file.
Step two, data engineers shares this blueprint
via source control tool or S3.
Step three, non-developers persona can easily register
and use this blueprint using a user interface
where they provide input as parameters.
Finally, blueprint uses these parameters
to generate AWS Glue workflow
and users can simply execute these workflow
to ingest and transform data.
Now that you know what is Glue blueprint,
let's look at the scenario and data set for this demo.
One of the common tasks in building data lake
is to convert row file to columnar file format
like Apache Parquet before consumption.
You may already know that Parquet file format
provides multiple times better performance
and lower cost compared to row-based files
like CSV or JSON for analytical workloads.
One of the main challenges
is that source data are being generated
in various file format.
and because of that you need to develop
and test and deploy pipeline for each file format.
You can significantly reduce the effort for development
and testing by using Glue blueprints.
We are going to demonstrate the same
using Amazon product data which is in CSV file format
and reviews data which is in JSON file format.
We will be converting them to Parquet using Glue blueprints.
If you had to do this without Glue blueprint,
you would have to manually create multiple jobs and workflow
for each file format.
But Glue blueprint simplifies this process.
We will reuse the same blueprint
to convert both JSON and CSV files to Parquet file format.
The same blueprint can be used in future
to ingest other file format.
Now I'll hand over to Arun to provide the demo
of the scenario.
- [Arun] Hello everyone.
For the purpose of this demo,
we will be using a Glue blueprint
that performs data format conversion.
This blueprint can convert files in any format.
That is, format supported by the Glue crawler,
such as CSVs or JSONs
into a more data lake optimized format, Apache Parquet.
This blueprint and many more
are available on our Public GitHub page.
At a later part in this demo,
we will take a deeper look at the contents of a blueprint.
Until then, as Deen already mentioned,
it is a collection of files under a ZIP archive
that helps generate a repeatable workflows.
We have our conversion blueprint already uploaded
to an S3 location in our demo account.
Before we head over to the AWS Glue console,
let us explore our data set.
For this demo
we will be using the publicly available Amazon reviews data
set for magazines.
The dataset consists of two types of files,
a CSV metadata file
with Amazon standard identification number
that correlates this data set with reviews data
which is in JSON format.
Let us use Glue blueprint to transform both of these dataset
into a more data lake optimized format, Apache Parquet.
We start by logging into the Glue console.
On the left hand side of the console
there is a link Blueprints,
which we will be using to create them.
We click on Add blueprints,
which exposes a standard blueprint creation form.
Let us create our data format conversion blueprint,
starting with an appropriate name.
The blueprint names can contain alphabets A to Z,
or numbers zero to nine.
Underscore and hyphen
are the only special characters allowed.
Find the blueprint to our ZIP file
which has already been uploaded to an S3 location.
You can add an optional description
and then click the Add blueprint button.
The blueprint will be created,
you can wait until the status turns to active.
Now that our blueprint has been created and active,
our next step is to create workflows.
Follow the UI options to create our first workflow,
which will convert the CSV reviews metadata to Parquet.
We enter the workflow name
which in our case is csv_to_parquet.
Next, we enter a pre-created IAM role.
This is the role that will be attached
to our Glue ETL jobs and crawlers,
and it should help really just to create databases,
access S3 files, et cetera.
We now point the input to the S3 location
where our CSV data is located.
Next, we enter a database and table name
in which the converter data will be stored.
In case the DB or table doesn't exist it will be created.
We also mentioned a destination location
to store the converted Parquet files.
The blueprint allows you to schedule the workflow
using the frequency fields.
However, for the purpose of this demo
we will skip this optional section
and run the workflow on demand.
Additionally, you can configure the number of Glue workers
for your workflow
which again we will leave it for the defaults.
Finally, we provide a pre-existing IAM role
that Glue can assume, and the role should have permissions
to create workflow and AWS resources associated to it.
We click on the Submit button
to initiate the workflow creation process.
We will now check on the status of our workflow.
We see it changed from running to succeeded.
We will follow the same set of instructions
to create our next workflow
which will convert the JSON reviews dataset to Parquet.
When you compare the step to the previous what changes are,
the name of the workflow,
the location of the input and output,
and also the destination table name.
Using the same blueprint
we were able to create two workflows
working on two different data formats.
Let us trigger them both and wait for it to complete.
Now that both the workflows have successfully executed,
let us take a look at the results.
We see that the S3 or Parquet location has been populated
with the Parquet equivalent of the source files.
Additionally, the Glue catalog holds entries
to both the source and converted files,
making it easier for the files to be consumed
in a tableau manner by other analytics services,
such as Athena or EMR.
Before we conclude let us take a deeper look
at the contents of a blueprint,
and for this, let us extract our archive file.
As you can see this particular archive contains four files
of which we will take a closer look.
The first file named blueprint.cfg
is a mandatory file that defines the entry point
into the blueprint.
It contains the parameters that form the user input,
their data types and associated properties.
It is this file that drives the workflow configuration form
that we used earlier to create our workflows.
The configuration file also contains a pointer
to the layout descriptor.
The layout.py, which is the layout descriptor file
is yet another mandatory file in the blueprint archive.
This is a Python file that contains a function
that defines the Glue workflow.
It parses the form field we entered earlier.
For example, you can see the workflow name
being extracted out to a variable in the step.
The code creates any necessary databases and tables.
Similarly, it creates the crawler and Glue job.
Scheduling information if any is parsed and used.
Finally, the workflow is created.
In addition to the configuration file and layout descriptor,
the blueprint archive can have multiple Glue jobs scripts
such as conversion.py in our example.
It may also contain a README and other supporting files.
If we take a step back and inspect one of our workflows,
we see that it consists of a trigger, a crawler
and an ETL job.
You can navigate to the respective sections
of the Glue console to individually inspect them.
The job section for example
has an auto-generated PySpark code
responsible for the data transformation.
This is created from the script file
that we had in our ZIP archive.
To summarize, we were able to use Glue blueprints
to create and share AWS Glue workflows.
Rather than creating a separate workflow
for each similar use case, the logic was abstracted
into a single repeatable Glue blueprint.
We hope this session helped you familiarize
the concept of Glue blueprints.
Please feel free to gather more information
from our documents, using the links on screen.
Thank you for watching.