Welcome to another episode of 'This is My Architecture'.
Today, I'm joined by Carlos from TrueBlue.
- Hi, Carlos. - Hi, Adrian.
Thanks for joining me.
Happy to be here.
Tell me about what TrueBlue does.
TrueBlue is a staffing company that connects people to blue collar jobs
on demand across North America.
Right. It's a buoyant economy at the moment, so I bet you're busy.
Tell us about some of the challenges that you have.
Some of the biggest challenges are matching the right people to the right jobs
and for that we need our data for jobs, job requests, and for workers
to be up-to-date in the present state, so we can make the most accurate match,
and then be able to send the right people those jobs.
Right. Accurate matches are really important for your customers, I bet.
Let's dive in.
Okay. So, it all starts here with SQL Server.
We use DMS to migrate our transactional databases to AWS, specifically into S3.
So, we have a landing bucket,
and we have Parquet files insert sub dates and deletes
for our transactional tables being accumulated here in this bucket.
So, the data migration service,
native integration with all that SQL Server farm that you have there.
Once the files land in S3, what happens next?
So, what happens next is we set up S3 by notifications on this bucket,
and now this is actually feeding to SQS queues
that are capturing those events
and accumulating here in these queues for downstream consumption.
Awesome. So, event driven architecture all kicks off from here.
Let's go into Glue here,
and I see you've got a number of services,
- serverless services, in fact, - Mhm.
that support it.
How does it work?
So, we have different latency requirements for different tables,
and not all of them need to be near real-time.
So, what we have here is three SQS queues,
one for each category of tables based on latency.
And then we have here three EventBridge rules as well.
We have a 10-minute rule, a 60-minute rule, and a 3-hour rule.
What EventBridge is doing is,
let's look at the 10-minute one, for example.
It gets invoked every 10 minutes.
It invokes a Step Function directly.
Native integration EventBridge to Step Functions.
And then this Step Function is orchestrating a workflow of Glue jobs
where the first Glue job, the extractor, is going read from this SQS queue.
Basically, those are S3 locations for the landing buckets.
And then it's going to write or update tables for our transactional databases.
So, a common pattern that we see for data lakes like this is to use EMR.
Now, I understand you were using that before, right?
Yes, EMR is great.
It allowed us to achieve this outcome of combining inserts, updates and deletions
to consistent views of tables very, very fast.
But at TrueBlue we are serverless first.
Also, our developers and our engineers were serverless developers,
and that's just a personal preference for us.
We were having some challenges based on skill set and complexity
to manage the EMR clusters and the autoscaling,
and then debugging when something went wrong
because we have long-running clusters.
It was just challenging for us
and we wondered is there a serverless option that we could use.
Immediately, one of our engineers said, "Can we do this with Glue?"
Like, "I don't know".
So, we actually decided to use Glue
and it actually worked perfectly fine for us.
It accomplishes the same outcome as EMR for us,
but in a serverless fashion where we read the events from SQS
and then we write those to our Delta Lake in S3.
Right. So, it's also a familiar environment for your developers
to be able to write jobs directly into Glue, right?
Yes, both the engineers and data scientists.
Everyone is using Glue all the time.
Well, that's where I wanted to go to next.
How do your data scientists now start querying it?
Yes. So, the data scientists actually are over here,
and they use Spark SQL, typically,
to read the tables from this Delta Lake.
We have three tiers in our Delta Lake: bronze, silver and gold.
So, we have raw and then we have views
that we create by joining those raw tables,
and that are feature store for ML.
Now, they query from any one of those three stages.
It's very nice for them because it's very reliable how Glue
is basically performing those upserts and keeping those tables up to date.
They're able to perform accurate matches as well
because they're able to extract their datasets by querying the Delta Lake.
I love the fact that you're using event driven architecture
to keep up with the volume of changes that are going on in this.
This is fantastic.
Thanks for sharing your architecture with me, Carlos.
Thank you, Adrian.
And thank you for joining us on 'This is My Architecture'.