ATMCash Exploits Virtualization for Security - Immutability and Reversion

This is a guest post by Ran Grushkowsky, Head of Technology at ATMCash.

Virtualization and cloud-based systems are very hype in the industry; however, most financial companies stray from those solutions. At ATMCash, we’ve approached virtualization not for the usual reason of scalability, but for the usually missed value of security.

In this article, I will introduce the concept of security added value in the utilization of virtualization and why people should consider deploying mini-clouds for those use-cases. 

How do virtual machines help mitigate risk?

I am sure most of you have heard of the recent hacking in the financial sector. Financial companies are under constant hacking attempts and security is of the utmost importance. One of the bigger risks in system deployment is a breach in one of the stack components. Regular system patches and maintenance fix known exploits and issues, however, sometimes it may be too late and the component has already been breached. If the system has already been compromised in a natural environment where patches are applied to existing systems, sometimes the patch may come too late and a Trojan horse or some sort of malicious code has already been injected (as may have been seen in recent cases). Virtual machines provide a great hidden gem: immutability and reverting images. 

Example of how ATMCash uses those features for security in the stack:

Click to read more ...

TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data

This is a guest post by Andy Gelfond, VP of Engineering for TripAdvisor. Andy has been with TripAdvisor for six and a half years, wrote a lot of code in the earlier days, and has been building and running a first class engineering and operations team that is responsible for the worlds largest travel site. There's an update for this article at An Epic TripAdvisor Update: Why Not Run On The Cloud? The Grand Experiment. 

For TripAdvisor, scalability is woven into our organization on many levels - data center, software architecture, development/deployment/operations, and, most importantly, within the culture and organization. It is not enough to have a scalable data center, or a scalable software architecture. The process of designing, coding, testing, and deploying code also needs to be scalable. All of this starts with hiring and a culture and an organization that values and supports a distributed, fast, and effective development and operation of a complex and highly scalable consumer web site.

Stats as of 6/2011

Click to read more ...

A TripAdvisor Short

Sometimes I get article proposals and then there's no follow up. Though these TripAdvisor data points are from 2010, I thought them worth sharing:

Our site serves in excess of 100M dynamically generated page view a day (all media and static content goes through CDN), and we do this with about 100 machines, no single point of failure, supported by distributed service architecture that that responds to over 2B requests a day, and a data warehouse of over 20TB that is used to drive email campaigns, SEM, and general reporting. We are a Linux/Java/Apache/Tomcat/Postgres/Lucene shop, and have built our own distributed computing architecture. We also maintain duplicate data centers (one active, one standby) for redundancy and maintenance purposes.

Too bad, it sounds like it would have been a good article.

Click to read more ...

Evernote Architecture - 9 Million Users and 150 Million Requests a Day

The folks at Evernote were kind enough to write up an overview of their architecture in a post titled Architectural Digest. Dave Engberg describes their approach to networking, sharding, user storage, search, and some other custom services.

Evernote is a cool application, partially realizing Vannevar Bush's amazing vision of a memex. Wikipedia describes Evernote's features succinctly: 

Evernote is a suite of software and services designed for notetaking and archiving. A "note" can be a piece of formattable text, a full webpage or webpage excerpt, a photograph, a voice memo, or a handwritten "ink" note. Notes can also have file attachments. Notes can then be sorted into folders, tagged, annotated, edited, given comments, and searched. Evernote supports a number of operating system platforms (including Android, Mac OS X, iOS, Microsoft Windows and WebOS), and also offers online synchronization and backup services.

Key here is that Evernote stores a lot of data, that must be searched, and synced through their cloud to any device you use. 

Another key is the effect of Evernote's business model and cost structure. Evernote is notable for their pioneering of the freemium model, based on the idea from their CEO: The easiest way to get 1 million people paying is to get 1 billion people using. Evernote is designed to become profitable at a 1% conversion rate. The free online service limits users to a hefty 60 MB/month while premium users pay $45 per year for 1,000 MB/month. To be profitable they most store a lot of data without spending a lot of money. There's not a lot of room for extras, which accounts for the simple practicality of their architecture. 

The article is short and succinct, so definitely read it for details. Some takeaways:  

Click to read more ...

Facebook: An Example Canonical Architecture for Scaling Billions of Messages

What should the architecture of your scalable, real-time, highly available service look like? There are as many options as there are developers, but if you are looking for a general template, this architecture as described by Prashant Malik, Facebook's lead for the Messages back end team, in Scaling the Messages Application Back End, is a very good example to consider. 

Although Messages is tasked with handling 135+ billion messages a month, from email, IM, SMS,  text messages, and Facebook messages, you may think this is an example of BigArchitecture and doesn't apply to smaller sites. Not so. It's a good, well thought out example of a non-cloud architecture exhibiting many qualities any mom would be proud of:

Click to read more ...

Viddler Architecture - 7 Million Embeds a Day and 1500 Req/Sec Peak

Viddler is in the high quality Video as a Service business for a customer who wants to pay a fixed cost, be done with it, and just have it work. Similar to Blip and Ooyala, more focussed on business than YouTube. They serve thousands of business customers, including high traffic websites like FailBlog, Engadget, and Gawker.

Viddler is a good case to learn from because they are a small company trying to provide a challenging service in a crowded field. We are catching them just as they transitioning from a startup that began in one direction, as a YouTube competitor, and pivoted into a slightly larger company focussed on paying business customers.

Transition is the key word for Viddler: transitioning from a free YouTube clone to a high quality paid service. Transitioning from a few colo sites that didn't work well to a new higher quality datacenter. Transitioning from an architecture that was typical of a startup to one that features redundancy, high availability, and automation. Transitioning from a lot of experiments to figuring out how they want to do things and making that happen. Transition to an architecture where features were spread out amongst geographically distributed teams using different technology stacks to having clear defined roles.

In other words, Viddler is like most every other maturing startup out there and that's fun to watch. Todd Troxell, Systems Architect at Viddler, was kind enough to give us an interview and share the details on Viddler's architecture. It's an interesting mix of different technologies, groups, and processes, but it somehow seems to all work. It works because behind all the moving parts is the single idea: making the customer happy and giving them what they want, no matter what. That's not always pretty, but it does get results.

Site: Viddler.com

The Stats

Click to read more ...

Did the Microsoft Stack Kill MySpace?

Robert Scoble wrote a fascinating case study, MySpace’s death spiral: insiders say it’s due to bets on Los Angeles and Microsoft, where he reports MySpace insiders blame the Microsoft stack on why they lost the great social network race to Facebook.  

Does anyone know if this is true? What's the real story?

I was wondering because it doesn't seem to track with the MySpace Architecture post that I did in 2009, where they seem happy with their choices and had stats to back up their improvements. Why this matters is it's a fascinating model for startups to learn from. What does it really take to succeed? Is it the people or the stack? Is it the organization or the technology? Is it the process or the competition? Is the quality of the site or the love of the users? So much to consider and learn from.

Some conjectures from the article:

Click to read more ...

Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events Per Day

Facebook did it again. They've built another system capable of doing something useful with ginormous streams of realtime data. Last time we saw Facebook release their New Real-Time Messaging System: HBase To Store 135+ Billion Messages A Month. This time it's a realtime analytics system handling over 20 billion events per day (200,000 events per second) with a lag of less than 30 seconds. 

Alex Himel, Engineering Manager at Facebook, explains what they've built (video) and the scale required:

Social plugins have become an important and growing source of traffic for millions of websites over the past year. We released a new version of Insights for Websites last week to give site owners better analytics on how people interact with their content and to help them optimize their websites in real time. To accomplish this, we had to engineer a system that could process over 20 billion events per day (200,000 events per second) with a lag of less than 30 seconds. 

Alex does an excellent job with the presentation. Highly recommended. But let's take a little deeper look at what's going on...

Click to read more ...

Medialets Architecture - Defeating the Daunting Mobile Device Data Deluge

Mobile developers have a huge scaling problem ahead: doing something useful with massive continuous streams of telemetry data from millions and millions of devices. This is a really good problem to have. It means smartphone sales are finally fulfilling their destiny: slaughtering PCs in the sales arena. And it also means mobile devices aren't just containers for simple standalone apps anymore, they are becoming the dominant interface to giant backend systems.

Click to read more ...

Stack Overflow Architecture Update - Now at 95 Million Page Views a Month

A lot has happened since my first article on the Stack Overflow Architecture. Contrary to the theme of that last article, which lavished attention on Stack Overflow's dedication to a scale-up strategy, Stack Overflow has both grown up and out in the last few years.

Stack Overflow has grown up by more then doubling in size to over 16 million users and multiplying its number of page views nearly 6 times to 95 million page views a month.  

Stack Overflow has grown out by expanding into the Stack Exchange Network, which includes Stack Overflow, Server Fault, and Super User for a grand total of 43 different sites. That's a lot of fruitful multiplying going on.

Click to read more ...