This is a guest post by Eunice Do, Data Engineer at TripleLift, a technology company leading the next generation of programmatic advertising.
What is the name of your system and where can we find out more about it?
The system is the data pipeline at TripleLift. TripleLift is an adtech company, and like most companies in this industry, we deal with high volumes of data on a daily basis. Over the last 3 years, the TripleLift data pipeline scaled from processing millions of events per day to processing billions. This processing can be summed up as the continuous aggregation and delivery of reporting data to users in a cost efficient manner. In this article, we'll mostly be focusing on the current state of this multi-billion event pipeline.
InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
Need excellent people? Advertise your job here!
Cool Products and Services
Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how engineering teams are using products like StackHawk and Snyk to add security bug testing to their CI pipelines.
Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years.
Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!
Learn to balance architecture trade-offs and design scalable enterprise-level software. Check out Educative.io's bestselling new 4-course learning track: Scalability and System Design for Developers. Join more than 300,000 other learners.
Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.
Advertise your product or service here!
If you are interested in a sponsored post for an event, job, or product, please contact us for more information.
Make Your Job Search O(1) — not O(n)
Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.
We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.
InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
Need excellent people? Advertise your job here!
Cool Products and Services
Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years.
Learn to balance architecture trade-offs and design scalable enterprise-level software. Check out Educative.io's bestselling new 4-course learning track: Scalability and System Design for Developers. Join more than 300,000 other learners.
Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!
Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.
Advertise your product or service here!
If you are interested in a sponsored post for an event, job, or product, please contact us for more information.
Essilen Research Free Video Series on Hiring
Modern tech companies understand that hiring is the most important on-going activity they undertake. However, a lot of companies still do their hiring in an ad-hoc way. Thrown-together job specs, random resume screening, unfocused interviews, and a lot more. The result is a process that doesn't get you the best employees you could. World-class tech companies do things very differently. In Essilen Research's video series on tech hiring, you'll learn the secrets that the best companies use to hire the best engineers, developers and designers.
We take you through the hiring process from start to finish. Learn how to write the right job spec. How to screen candidates efficiently, effectively, and without bias. Learn how to train interviewers so everyone's on the same page. No more arguments at the hiring meeting! And there's more: how to offer and close candidates, so they don't slip through your fingers. And of course, how to onboard new-hires the right way so they hit the ground running.
Essilen Research has decades of experience starting, building, running, and scaling great tech organizations. We've conducted thousands of interviews as hiring managers, hired hundreds, and trained dozens of interviewers ourselves. There really IS a better way to do tech hiring and this video series will start you on the path. Level up your tech hiring!
PA File Sight monitors file access on a server in real-time.
It can track who is accessing what, and with that information can help detect file copying, detect (and stop) ransomware attacks in real-time, and record the file activity for auditing purposes. The collected audit records include user account, target file, the user’s IP address and more. This solution does NOT require Windows Native Auditing, which means there is no performance impact on the server. Join thousands of other satisfied customers by trying PA File Sight for yourself. No sign up is needed for the 30-day fully functional trial.
Make Your Job Search O(1) — not O(n)
Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.
We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.
Zoom scaled from 20 million to 300 million users virtually over night. What's incredible is from the outside they've shown little in the way of apparent growing pains, though on the inside it's a good bet a lot of craziness is going on.
Sure, Zoom has made some design decisions that made sense as a small spunky startup that don't make a lot of sense as a defacto standard, but that's to be expected. It's not a sign of bad architecture as many have suggested. It's just realistically how products evolve, especially when they must uplift over weeks, days, and even hours.
Sudden success invites scrutiny, so everyone wants to know how Zoom works. The problem is we don't know much, but we do have a few information sources:
LOL. Who knew a birthday service could lead to an existential crisis?
Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the smart and thoughtful world.
Know someone who wants to understand the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 111 mostly 5 star reviews. Here's a recent 100% organic non-GMO review:
Number Stuff:
Don't miss all that the Internet has to say on Scalability, click below and become eventually
consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...
InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
Need excellent people? Advertise your job here!
Cool Products and Services
Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years.
Learn to balance architecture trade-offs and design scalable enterprise-level software. Check out Educative.io's bestselling new 4-course learning track: Scalability and System Design for Developers. Join more than 300,000 other learners.
Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!
Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.
Advertise your product or service here!
If you are interested in a sponsored post for an event, job, or product, please contact us for more information.
Essilen Research Free Video Series on Hiring
Modern tech companies understand that hiring is the most important on-going activity they undertake. However, a lot of companies still do their hiring in an ad-hoc way. Thrown-together job specs, random resume screening, unfocused interviews, and a lot more. The result is a process that doesn't get you the best employees you could. World-class tech companies do things very differently. In Essilen Research's video series on tech hiring, you'll learn the secrets that the best companies use to hire the best engineers, developers and designers.
We take you through the hiring process from start to finish. Learn how to write the right job spec. How to screen candidates efficiently, effectively, and without bias. Learn how to train interviewers so everyone's on the same page. No more arguments at the hiring meeting! And there's more: how to offer and close candidates, so they don't slip through your fingers. And of course, how to onboard new-hires the right way so they hit the ground running.
Essilen Research has decades of experience starting, building, running, and scaling great tech organizations. We've conducted thousands of interviews as hiring managers, hired hundreds, and trained dozens of interviewers ourselves. There really IS a better way to do tech hiring and this video series will start you on the path. Level up your tech hiring!
PA File Sight monitors file access on a server in real-time.
It can track who is accessing what, and with that information can help detect file copying, detect (and stop) ransomware attacks in real-time, and record the file activity for auditing purposes. The collected audit records include user account, target file, the user’s IP address and more. This solution does NOT require Windows Native Auditing, which means there is no performance impact on the server. Join thousands of other satisfied customers by trying PA File Sight for yourself. No sign up is needed for the 30-day fully functional trial.
Make Your Job Search O(1) — not O(n)
Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.
We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.
InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
Need excellent people? Advertise your job here!
Cool Products and Services
Learn to balance architecture trade-offs and design scalable enterprise-level software. Check out Educative.io's bestselling new 4-course learning track: Scalability and System Design for Developers. Join more than 265,000 other learners.
Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!
Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.
Advertise your product or service here!
If you are interested in a sponsored post for an event, job, or product, please contact us for more information.
Essilen Research Free Video Series on Hiring
Modern tech companies understand that hiring is the most important on-going activity they undertake. However, a lot of companies still do their hiring in an ad-hoc way. Thrown-together job specs, random resume screening, unfocused interviews, and a lot more. The result is a process that doesn't get you the best employees you could. World-class tech companies do things very differently. In Essilen Research's video series on tech hiring, you'll learn the secrets that the best companies use to hire the best engineers, developers and designers.
We take you through the hiring process from start to finish. Learn how to write the right job spec. How to screen candidates efficiently, effectively, and without bias. Learn how to train interviewers so everyone's on the same page. No more arguments at the hiring meeting! And there's more: how to offer and close candidates, so they don't slip through your fingers. And of course, how to onboard new-hires the right way so they hit the ground running.
Essilen Research has decades of experience starting, building, running, and scaling great tech organizations. We've conducted thousands of interviews as hiring managers, hired hundreds, and trained dozens of interviewers ourselves. There really IS a better way to do tech hiring and this video series will start you on the path. Level up your tech hiring!
PA File Sight monitors file access on a server in real-time.
It can track who is accessing what, and with that information can help detect file copying, detect (and stop) ransomware attacks in real-time, and record the file activity for auditing purposes. The collected audit records include user account, target file, the user’s IP address and more. This solution does NOT require Windows Native Auditing, which means there is no performance impact on the server. Join thousands of other satisfied customers by trying PA File Sight for yourself. No sign up is needed for the 30-day fully functional trial.
Make Your Job Search O(1) — not O(n)
Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.
We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.
In that talk Rick revealed for the first time the inner arcana of single-table design. Minds were blown. Weaknesses were revealed. Futures were changed.
As a mere novice in the ways of DynamoDB I realized there were many levels of understanding needed before one could become a true AWS Data Hero. For that we need a guide.
Our guide on the Hero’s Journey that is mastering DynamoDB is a wise young wizard named Alex DeBrie. Alex wrote what you might consider to be the Gnostic Gospels of DynamoDB: The DynamoDB Book.
You will know something after reading this book
But it's more than just a book. You can’t buy it on Amazon. Instead, Alex uses Gumroad to offer packages at three different price points along with a team option. Each level provides additional content:
Basic package ($79). A 450 page book and six cheatsheets.
Plus package ($129). 60-page Analytics supplement guide. 60-page Operations supplement guide. Five deployable code implementations.
Premium package ($249). Video walkthroughs of every chapter.
Use the code "HIGHSCALABILITY" at checkout and you’ll shave off $20 for Basic, $30 for Plus, and $50 for Premium. You’re welcome.
When Alex asked for advice about what to include in the book, I made a big deal about including complete working code examples. Alex delivered. So I thought it was only fitting that I buy the Premium package.
But it’s so much money, you might complain. As someone who has made not much money writing a book I have some idea how much effort Alex put into creating these materials. It’s a lot of work. As a profession we must be willing to pay for our tools. So as a professional programmer wanting to create professional products—I’m very happy with the result. Here’s why.
What’s great about SQL is you can find an answer to damn near any question you have. Not so with DynamoDB. Working code examples are rare.
I remember Rick talking about using “between” when making queries. Never heard of such a thing and it took me forever to find a complete working code example of what he was talking about. Don’t even talk about the documentation. What you find are ridiculous command line examples like “aws dynamodb query…” What use is the command line for this sort of stuff?
Show the code! And Alex does.
I think you’ll find the 5 code examples—Big Time Deals, E-Commerce, Github Migration, Github Node, and Session Store Node—worth the price of admission. First, they tackle complex domains so they might already solve a problem you’re having. That’s gold. Second, you’ll learn a lot by just reading the code. As a not so great Javascript developer I know I learned a lot.
What about the Holy Grail of single-table design? If you have that sneaking suspicion you’re using DynamoDB incorrectly and you’ve tried to learn single-table design before, but got frustrated, Alex has your back. The book is filled with clear examples backed by strategies for various use cases.
Against my expectations the videos turned out to be very useful. Alex’s tone was great. It was just like he was next to me explaining what he was doing. And that’s exactly what you want from a teaching product.
Here's my email interview with Alex DeBrie on The DynamoDB Book. Enjoy.
InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
Need excellent people? Advertise your job here!
Cool Products and Services
Learn to balance architecture trade-offs and design scalable enterprise-level software. Check out Educative.io's bestselling new 4-course learning track: Scalability and System Design for Developers. Join more than 265,000 other learners.
Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!
Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.
Advertise your product or service here!
If you are interested in a sponsored post for an event, job, or product, please contact us for more information.
Essilen Research Free Video Series on Hiring
Modern tech companies understand that hiring is the most important on-going activity they undertake. However, a lot of companies still do their hiring in an ad-hoc way. Thrown-together job specs, random resume screening, unfocused interviews, and a lot more. The result is a process that doesn't get you the best employees you could. World-class tech companies do things very differently. In Essilen Research's video series on tech hiring, you'll learn the secrets that the best companies use to hire the best engineers, developers and designers.
We take you through the hiring process from start to finish. Learn how to write the right job spec. How to screen candidates efficiently, effectively, and without bias. Learn how to train interviewers so everyone's on the same page. No more arguments at the hiring meeting! And there's more: how to offer and close candidates, so they don't slip through your fingers. And of course, how to onboard new-hires the right way so they hit the ground running.
Essilen Research has decades of experience starting, building, running, and scaling great tech organizations. We've conducted thousands of interviews as hiring managers, hired hundreds, and trained dozens of interviewers ourselves. There really IS a better way to do tech hiring and this video series will start you on the path. Level up your tech hiring!
PA File Sight monitors file access on a server in real-time.
It can track who is accessing what, and with that information can help detect file copying, detect (and stop) ransomware attacks in real-time, and record the file activity for auditing purposes. The collected audit records include user account, target file, the user’s IP address and more. This solution does NOT require Windows Native Auditing, which means there is no performance impact on the server. Join thousands of other satisfied customers by trying PA File Sight for yourself. No sign up is needed for the 30-day fully functional trial.
Make Your Job Search O(1) — not O(n)
Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.
We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.
Monday, June 15, 2020 at 9:04AM 
