If You're Programming a Cell Phone Like a Server You're Doing it Wrong

Power on a cell phone is like water in a desert. It’s the very stuff of life. If you take the same naive programming techniques you learned when programming on a server in a datacenter your cell phone will die of thirst.

This is dramatically shown by Reto Meier, Tech Lead for the Android Developer Relations Team, in a remarkable series of instructional videos:

Click to read more ...

Sponsored Post: Apple, Couchbase, Evernote, MongoDB, Stackdriver, BlueStripe, Surge, Booking, Rackspace, AiCache, Aerospike, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• Evernote is hiring a Senior DevOps Engineer in our mission to help the world remember everything. Our work environment is collaborative and relaxed, our benefits and perks are fantastic, and we enrich the lives of more than 65 million users worldwide every day! Please apply here.


• Stackdriver is looking for systems + cloud + dev + ops guru to serve as our liaison within the DevOps community. If you are passionate about monitoring and automation, enjoy working on open source, and are excited by the prospect of sharing your expertise with your peers, get in touch with us today! http://bit.ly/143ARmy


• We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.


• Apple Applications Architect. Apple's Customer Systems group, within the global Information Systems and Technology (IS&T) organization, helps design and implement all of the critical infrastructure that allows Apple to provide the best support in the world. Imagine what you could do here. Please apply here.


• Apple Software Engineer, Researcher. The OS X Analytics group is looking for a skilled software engineer. In this position you will develop tools to collect, process, and analyze data from millions of systems. Imagine what you could do here. Please apply here.


• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 


• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• Surge - The Scalability & Performance Conference, presented by OmniTI, Sept. 12th-13th, features speakers from Joyent, Fastly, Dyn, Netflix, Linkedin and Amazon. Special, High Scalability Reader Rate: $50 off registration--through Sept. 10! Book hotel and get $50 off, from OmniTI. 

Cool Products and Services

• The leading technology companies use Couchbase as their NoSQL database. Download the free open-source version of Couchbase Server and make something awesome today.


• MongoDB Management Service (MMS) is a cloud-based suite of services for managing MongoDB deployments. In addition to monitoring and alerting, now you can seamlessly back up your MongoDB deployment to the cloud using using MMS. To get started with monitoring and backup, visit mms.mongodb.com.


• BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.


• AppDynamics is an easy-to-use application performance management solution that offers code-level insight into Java, .NET and PHP applications. Get the free trial.


• NEW! Aerospike 3 - Download FREE. Introducing the new Aerospike 3 database that builds off of Aerospike's legacy of speed, scale, and reliability, adding an extensible data model that supports complex data types, large data types, queries using secondary indexes, user defined functions (UDFs) and distributed aggregations using Stream UDFs for real-time data.


• The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.


• aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required . http://aicache.com/


• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.


• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit  http://www.appdynamics.com/freetrial.


• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.


• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

The Hidden DNS Tax - Cascading Timeouts and Errors

This is a guest post by Nick Burling, VP of Product Management of Bluestripe.

Readers of High Scalability know are well versed in performance optimization techniques. Reverse proxies, Varnish, Redis — you hear about them daily. But what you may not realize is that one of the oldest technologies in your stack can be one of your biggest bottlenecks: DNS.

People don't spend a lot of time thinking about DNS. It's not sexy. It's an infrastructure service, and it's just supposed to work.

At BlueStripe, we work with many teams running applications that support millions of web requests a day. We keep seeing DNS delays and errors that the platform operations team never knows about. It's so common we've start calling it the Hidden DNS Tax.

What is the Hidden DNS Tax?

Click to read more ...

Stuff The Internet Says On Scalability For September 13, 2013

Hey, it's HighScalability time (this week is a fall harvest basket overflowing with good nutritious wisdom):

(Voyager: 36 yrs & 11B miles has reached interstellar space. So it begins...)

• 170 million: metrics Twitter collects every minute; 350 million: Snapchat daily photo shares
• Quotable Quotes:
  • @blowmage: OH: “Guys, databases don't know how to sort things. That's why NoSQL uses JavaScript.”
  • Nokia insider: I look back and I think Nokia was just a very big company that started to maintain its position more than innovate for new opportunities.
  • Paulo Siqueira: Ignoring scalability is not as bad as it sounds—if you use the proper tools.
  • David Rosenthal: The relationship between diversity and risk is very complex.
  • Jaime Teevan: the exact same result list will seem more relevant to you if it is returned just a fraction of a second faster.
  • @aphyr: I use Redis as a queue #leaveDBalone


• Hey, I've always thought this about security, it's just too damn complex. But I always thought much smarter people than me understood it all, so it must be OK. Now I know that's not true. Nobody really understands security. NIST's Ridiculous Non-Response Response To Revelation That NSA Controlled Crypto Standards Process: "the NSA made sure that the standards were so complicated that no one could actually vet the security."


• Observability at Twitter. Great post on how Twitter captures, stores, queries, visualizes and automates monitoring data to enable the debugging of hundreds of distributed systems across multiple datacenters. Very detailed and helpful. 170 million individual metrics are collected every minute. Metrics are stored in and queried from a time series database developed at Twitter. Queries are written using a declarative, functional inspired language. Visualization use cases include hundreds of charts per dashboard and thousands of data points per chart. Our monitoring system allows users to define alert conditions and notifications in the same query language they use for ad hoc queries and building dashboards. Note how everything at this level of game play is a custom job.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Need Help with Database Scalability? Understand I/O

This is a guest post by Zardosht Kasheff, Software Developer at Tokutek, a storage engine company that delivers 21st-Century capabilities to the leading open source data management platforms.

As software developers, we value abstraction. The simpler the API, the more attractive it becomes. Arguably, MongoDB’s greatest strengths are its elegant API and its agility, which let developers simply code.

But when MongoDB runs into scalability problems on big data, developers need to peek underneath the covers to understand the underlying issues and how to fix them. Without understanding, one may end up with an inefficient solution that costs time and money. For example, one may shard prematurely, increasing hardware and management costs, when a simpler replication setup would do. Or, one may increase the size of a replica set when upgrading to SSDs would suffice.

This article shows how to reason about some big data scalability problems in an effort to find efficient solutions.

Defining the Issues

Click to read more ...

Stuff The Internet Says On Scalability For September 6, 2013

Hey, it's HighScalability time:

(Unidentified Ivy Bridge processor using 22 nanometer Tri-Gate transistors)

• Quotable Quotes:
  • @pbailis: Big ups to AWS folks for following up re: all of my questions on cr1 provisioning. We saw a huge win moving from m1.xl to cr1.8xl
  • @rob_carlson: Packet switching via containers --> almost 8X increase in trade; what will #drones bring? What is optimal mesh size?
  • @mrtazz: “an Open Source, Clojure-based DevOps platform” congratulations, I now have no idea what you’re talking about
  • @KentBeck: If you can't make engineering decisions based on data, then make engineering decisions that result in data.
  • @cassandralondon: Cassandra on AWS SSDs - a perfect fit because you don't get write amplification 


• If you think about it, a cloud as a rule driven, capability rich environment, accessible over a large surfaced API, plays the same role as physics in biology. Software must specify every little detail. Biology relies on the laws of physics to do the work. A cloud provides the response to the call of programmers. Complex grunt work is just done, as if by nature. Peter M. Hoffmann: The amount of information contained in our DNA is staggering, but it is not nearly enough to specify each molecule’s or cell’s location, or even the shape of an organ. Rather than being a blueprint (as DNA is often mistakenly called), DNA is more like a cooking recipe. When I make a cake, I don’t have to specify where each starch or sugar molecule goes. I just follow the instructions, and the molecules go where they are supposed to. Much of the information to make a cake or a human being is contained in the laws of physics and chemistry. Molecules “know” how to put themselves together.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Paper: MillWheel: Fault-Tolerant Stream Processing at Internet Scale

Ever wonder what powers Google's world spirit sensing Zeitgeist service? No, it's not a homunculus of Georg Wilhelm Friedrich Hegel sitting in each browser. It's actually a stream processing (think streaming MapReduce on steroids) system called MillWheel, described in this very well written paper: MillWheel: Fault-Tolerant Stream Processing at Internet Scale. MillWheel isn't just used for Zeitgeist at Google, it's also used for streaming joins for a variety of Ads customers, generalized anomaly-detection service, and network switch and cluster health monitoring.

Abstract:

MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework’s fault-tolerance guarantees.

This paper describes MillWheel’s programming model as well as its implementation. The case study of a continuous anomaly detector in use at Google serves to motivate how many of MillWheel’s features are used. MillWheel’s programming model provides a notion of logical time, making it simple to write time-based aggregations. MillWheel was designed from the outset with fault tolerance and scalability in mind. In practice, we find that MillWheel’s unique combination of scalability, fault tolerance, and a versatile programming model lends itself to a wide variety of problems at Google.

Click to read more ...

Wide Fast SATA: the Recipe for Hot Performance

This is a guest post by Brian Bulkowski, CTO and co-founder of Aerospike, a leading clustered NoSQL database, has worked in the area of high performance commodity systems since 1989.

This blog post will tell you exactly how to build a multi-terabyte high throughput datacenter server. A fast, reliable multi-terrabyte data tier can be used for recent behavior (messages, tweets, plays, actions), or anywhere that today you use Redis or Memcache.

You need to know:

• Which SSDs work
• Which chassis work
• How to configure your RAID cards

Intel’s SATA solutions – combined with a high capacity storage server like the Dell R720xd and a host bus adapter based on the LSI 2208, and a Flash optimized database like Aerospike, enables high throughput and low latency.

In a wide configuration, with 12 to 20 drives per 2U server, individual servers can cost-effectively serve at high throughput with 16T at $2.50 per GB with the s3700, or $1.25 with the s3500. Other SSD offerings – from Crucial (Micron) and Samsung (S843) – are at other densities and price-performance points.

This is in-memory computing at a stunningly new, accessible price level – but there are some details you need to know...

Click to read more ...

Sponsored Post: Apple, Couchbase, Evernote, 10gen, Stackdriver, BlueStripe, Surge, Booking, Rackspace, AiCache, Aerospike, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• Evernote is hiring a Senior DevOps Engineer in our mission to help the world remember everything. Our work environment is collaborative and relaxed, our benefits and perks are fantastic, and we enrich the lives of more than 65 million users worldwide every day! Please apply here.


• Stackdriver is looking for systems + cloud + dev + ops guru to serve as our liaison within the DevOps community. If you are passionate about monitoring and automation, enjoy working on open source, and are excited by the prospect of sharing your expertise with your peers, get in touch with us today! http://bit.ly/143ARmy


• We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.


• Apple Applications Architect. Apple's Customer Systems group, within the global Information Systems and Technology (IS&T) organization, helps design and implement all of the critical infrastructure that allows Apple to provide the best support in the world. Imagine what you could do here. Please apply here.


• Apple Software Engineer, Researcher. The OS X Analytics group is looking for a skilled software engineer. In this position you will develop tools to collect, process, and analyze data from millions of systems. Imagine what you could do here. Please apply here.


• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 


• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• Surge - The Scalability & Performance Conference, presented by OmniTI, Sept. 12th-13th, features speakers from Joyent, Fastly, Dyn, Netflix, Linkedin and Amazon. Special, High Scalability Reader Rate: $50 off registration--through Sept. 10! Book hotel and get $50 off, from OmniTI. 

Cool Products and Services

• The leading technology companies use Couchbase as their NoSQL database. Download the free open-source version of Couchbase Server and make something awesome today.


• MongoDB Management Service (MMS) is a cloud-based suite of services for managing MongoDB deployments. In addition to monitoring and alerting, now you can seamlessly back up your MongoDB deployment to the cloud using using MMS. To get started with monitoring and backup, visit mms.mongodb.com.


• BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.


• AppDynamics is an easy-to-use application performance management solution that offers code-level insight into Java, .NET and PHP applications. Get the free trial.


• NEW! Aerospike 3 - Download FREE. Introducing the new Aerospike 3 database that builds off of Aerospike's legacy of speed, scale, and reliability, adding an extensible data model that supports complex data types, large data types, queries using secondary indexes, user defined functions (UDFs) and distributed aggregations using Stream UDFs for real-time data.


• The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.


• aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required . http://aicache.com/


• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.


• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.


• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.


• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

Stuff The Internet Says On Scalability For August 30, 2013

Hey, it's HighScalability time:

(Nerd Power: Paul Kasemir software engineer AND American Ninja Warrior)

• Two billion documents, 30 terabytes: Github source code indexed
• Quotable Quotes:
  • David Krakauer: We fail to make intelligent machines because engineering is about putting together stupid components to make smart objects. Evolution is about putting together smart components into intelligent aggregates. Your brain is like an ecosystem of organisms. It's not like a circuit of gates.
  • @spyced: At this point if you depend on EBS for critical services you're living in denial and I can't help you. 
  • @skilpat: TIL Friedrich Engels, not Leslie Lamport, invented logical clocks in a 1844 letter to Karl Marx
  • Dan Geer: Risk is a necessary consequence of dependence.
  • @postwait: OS Rule 1. The version of /usr/bin/X you want today will never be what your OS ships. Use /keep/your/shit/to/yourself/X instead.
  • @antirez: the moral is: "Hey, that's a completely harmless commit about error reporting! It will never affect behavior". hehe
  • Robert Scoble: My next-door neighbor was on the first iPhone team and he told me  he almost killed himself working for Steve Jobs, because he demands so much from you. He did not take substandard performance, and he would keep you up, and he would call you on a Sunday when you’re having family time … and essentially randomize your whole life
• LOL. Interview with an Ex-Microsoftie: "Program Files", is that one of yours? —Sure is. I thought, "This is where we'll put the programs. On the file system. And since they're gonna be made out of files, I'll stick the word "Files" on the end. With a space in the middle. People are going to be typing this all the time, so they'll want it to be as long and descriptive as possible."

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...