Welcome to "This is My Architecture".
I'm Orit, and I have with me Yotam from Go Global. -Hi, Orit.
Tell us about Go Global. Go Global is a hotel consolidator
and technology supplier for the tourism industry.
We have over 230,000 hotels and apartments around the world
spread across 15,000 destinations in over 200 countries.
Sounds great. What are we going to talk about today?
Today we'll be talking about the data platforms that our company has developed.
We receive an average of 40 million searches on our platform each day
and generate 20 million events on a daily basis, which we send to Kinesis Firehose.
The events are formatted in AVRO, we write them in AVRO
because it's easier for us to send them out in this format using our app right from the start.
They all reach one single Kinesis Firehose, and from there continue flowing through the platform.
What happens after the Firehose?
Okay, so all of the events reach the Firehose,
which then aggregates different kinds of events into the object that is written into S3.
Once it's written, we use CloudWatch Events
to trigger the Step Function,
which basically does two things: Firstly, it manages the life cycle
of the object itself, that is to say, it moves it from the appropriate folders,
and secondly, all done with Lambdas, we take the incoming data
and split it.
As mentioned, we get many different kinds of data here,
many data types, so now we split according to the data type we receive.
Since we sent it through AVRO, we also know the schema,
and each data type has several mandatory fields according to which we know
which data type it is. So we split each data type
and from there we can do - the Lambda does several things.
Firstly, it writes it into the proper place in S3 for further processing.
Secondly, we update an index in Elasticsearch as required.
And we also know to maintain all the partitions in Glue in Athena,
which Athena uses thanks to this Lambda.
Sounds great. How many different types of events do you have?
Today we have about 80 events, with the aim of adding many more.
What happens when a new event is created, when the app decides to create a new event?
The developer is in charge of adding new events.
They need to define the class - which is actually the event -
define it with code, fill it with data
and use a standard tool in order to send the data to Firehose.
That's all you need to do code-wise, it's a matter of minutes.
Another thing they need to do is define the table in Athena,
there's a small utility in the code that generates the DDL needed to create the table,
but other than that, you don't need to do anything else,
and the distance between needing a data type and a data type that's already in S3
and is queryable in Athena is a matter of about five minutes.
So a developer who decides to add a new event doesn't need to know how the data platform is built
and doesn't need to change the infrastructure. They simply add the event, update it,
and the system already knows to receive it and... -Right.
To bring it to the data lake. -The whole idea, or one of the needs we had
when building this system was actually, as you said,
so that the developers won't need to do anything in the system or even know it.
This only creates friction, which slows down development. They need things to just work:
define the data type, define the table in Athena if required, and run with it.
Great. Who are the users of the data lake? Who are the consumers?
We have four main consumers: Firstly, R&D who use Elasticsearch
and Athena both for immediate needs as well as for investigating issues.
The second group is Operations, who also use both Elasticsearch and Athena
and to a certain extent also QuickSight.
We should mention that the data passes from S3 to Redshift through an Airflow Cluster
which runs ETLs hourly or daily, as required.
The third group of users is our Customer Success team
who actually make use of all these tools, both for customers with a pressing issue -
for which they use Elasticsearch - as well as for things requiring targeted investigation
or to get an aggregation that isn't currently calculated in Airflow. This is done on Athena,
through the console. And also for data - we use QuickSight dashboards
that read the data from Redshift written through Airflow.
And the fourth group is business users, who also use QuickSight
with the dashboards we have there, and also use a third party app
called ClickView in order to read Redshift data that is aggregated through Airflow.
Sounds amazing. You've actually built a generic data platform
to be used on top of data lake architecture and S3, with a materialized view in Elasticsearch,
Redshift, different consumers in the organization who can consume the data. Sounds great. Thank you, Yotam,
and thanks for watching "This is My Architecture".
Thank you for watching For more information visit aws.amazon.com/this-is-my-architecture
aws