Making Hadoop Run Faster
Tuesday, August 28, 2012 at 9:15AM Making Hadoop Run Faster
One of the challenges in processing data is that the speed at which we can input data is quite often much faster than the speed at which we can process it. This problem becomes even more pronounced in the context of Big Data, where the volume of data keeps on growing, along with a corresponding need for more insights, and thus the need for more complex processing also increases.
Batch Processing to the Rescue
Hadoop was designed to deal with this challenge in the following ways:
1. Use a distributed file system: This enables us to spread the load and grow our system as needed.
2. Optimize for write speed: To enable fast writes the Hadoop architecture was designed so that writes are first logged, and then processed. This enables fairly fast write speeds.
3. Use batch processing (Map/Reduce) to balance the speed for the data feeds with the processing speed.
Batch Processing Challenges
Cloud Bursting between AWS and Rackspace
Friday, June 15, 2012 at 9:30AM Cloud bursting is an application deployment model in which an application runs in a private cloud or data center and bursts into a public cloud when the demand for computing capacity spikes. The advantage of such a hybrid cloud deployment is that an organization only pays for extra compute resources when they are needed. ([Definition by SearchCloudComputing])
Neal Sample the former CTO - X.commerce at eBay gave an interesting talk last year on the economic benefit of Cloud Bursting. Neal pointed out eBay traffic statistics and showed some real numbers of the business impact of bursting peak load activities using on-demand cloud resources as presented in the diagram below.
In order to use cloud bursting effectively we need to address the following set of challenges:
Build your own twitter like real time analytics - a step by step guide
Thursday, May 24, 2012 at 9:15AM Major social networking platforms like Facebook and Twitter have developed their own architectures for handling the need for real-time analytics on huge amounts of data. However, not every company has the need or resources to build their own Twitter-like solution.
In this example we have taken the same Twitter/Facebook-like blueprint, and made it simple enough for developers to implement. We have taken the following approach in our implementation:
- Use In Memory Data Grid (XAP) for handling the real time stream data-processing.
- BigData data-base (Cassandra) for storing the historical data and manage the trend analytics
- Use Cloudify (cloudifysource.org) for managing and automating the deployment on private or pubic cloud
The example demonstrate a simple case of word count analytics. It uses Spring Social to plug-in to real twitter feeds. The solution is designed to efficiently cope with getting and processing the large volume of tweets. First, we partition the tweets so that we can process them in parallel, but we have to decide on how to partition them efficiently. Partitioning by user might not be sufficiently balanced, therefore we decided to partition by the tweet ID, which we assume to be globally unique. Then we need persist and process the data with low latency, and for this we store the tweets in memory.
Big Data In the Cloud Using Cloudify
Tuesday, March 27, 2012 at 9:35AM Edd Dumbill wrote an interesting article on O’Reilly Radar covering the current solutions for running Big Data in the Cloud
Big data and cloud technology go hand-in-hand. Big data needs clusters of servers for processing, which clouds can readily provide.
Big PaaS
Edd touched briefly on the role of PaaS for delivering Big Data applications in the cloud
Beyond IaaS, several cloud services provide application layer support for big data work. Sometimes referred to as managed solutions, or platform as a service (PaaS), these services remove the need to ucale things such as databases or MapReduce, reducing your workload and maintenance burden. Additionally, PaaS providers can realize great efficiencies by hosting at the application level, and pass those savings on to the customer.
Architecting Massively-Scalable Near-Real-Time Risk Analysis Solutions
Thursday, December 22, 2011 at 9:34AM Constructing a scalable risk analysis solution is a fascinating architectural challenge. If you come from Financial Services you are sure to appreciate that. But even architects from other domains are bound to find the challenges fascinating, and the architectural patterns of my suggested solution highly useful in other domains.
Five Misconceptions on Cloud Portability
Thursday, November 17, 2011 at 10:06AM The term "cloud portability" is often considered a synonym for "Cloud API portability," which implies a series of misconceptions.
If we break away from dogma, we can find that what we really looking for in cloud portability is Application portability between clouds which can be a vastly simpler requirement, as we can achieve application portability without settling on a common Cloud API.
In this post i'll be covering five common misconceptions people have WRT to cloud portability.
- Cloud portability = Cloud API portability. API portability is easy; cloud API portability is not.
- The main incentive for Cloud Portability is - Avoiding Vendor lock-in.Cloud portability is more about business agility than it is about vendor lock-in.
- Cloud portability isn’t for startups. Every startup that is expecting rapid growth should re-examine their deployments and plan for cloud portability rather than wait to be forced to make the switch when you are least prepared to do so.
- Cloud portability = Compromising on the least common denominator.Application portability doesn't require compromise on the least common denominator as most of the interaction with the cloud API happens outside of our application code anyway, to handle things like provisioning, setup, installation, scaling, monitoring, etc.
- The effort for achieving cloud portability far exceed the value. The effort to achieve cloud portability is far less than it used to be, in most cases, making it a greater and more valuable priority (with less investment) than it used to be.
Building your own Facebook Realtime Analytics System
Monday, July 18, 2011 at 9:02AM Recently, I was reading Todd Hoff's write-up on FaceBook real time analytics system
. As usual, Todd did an excellent job in summarizing this video from Engineering Manager at Facebook Alex Himel.
In the first post, I’d like to summarize the case study, and consider some things that weren't mentioned in the summaries. This will lead to an architecture for building your own Realtime Time Analytics for Big-Data that might be easier to implement, using Facebook's experience as a starting point and guide as well as the experience gathered through a recent work with few of GigaSpaces customers. The second post provide a summary of that new approach as well as a pattern and a demo for building your own Real Time Analytics system..