Entries by HighScalability Team (1576)

Wednesday
Oct302013

Strategy: Use Your Quantum Computer Lab to Tell Intentional Blinks from Involuntary Blinks

Oh, you don't have a Quantum Computer Lab staffed with researchers? Well, Google does. Here they are on G+. To learn what they are up to the Verge has A first look inside Google's futuristic quantum lab. The lab is partnership between NASA, Google, and a 512-qubit D-Wave Two quantum computer.  

One result from the lab is:

The first practical application has been on Google Glass, as engineers put the quantum chips to work on Glass's blink detector, helping it to better distinguish between intentional winks and involuntary blinks. For engineering reasons, the quantum processor can never be installed in Glass, but together with Google's conventional server centers, it can point the way to a better blink-detecting algorithm. That would allow the Glass processor to detect blinks with better accuracy and using significantly less power. If successful, it could be an important breakthrough for wink-triggered apps, which have struggled with the task so far.

Google thinks quantum computing has a major role in machine learning:

Click to read more ...

Tuesday
Oct292013

Sponsored Post: Apple, NuoDB, ScaleOut, FreeAgent, CloudStats.me, Intechnica, MongoDB, Stackdriver, BlueStripe, Booking, Rackspace, AiCache, Aerospike, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

  • Apple is hiring for multiple positions. Imagine what you could do here. At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly.
    • Sr. Software Engineer. You will primarily work with the domain team including project managers and engineers, as well as a large team of consultants in California and India. You will also work with many cross-functional and infrastructural teams during software delivery life cycle. Assignments can include design, delivery and oversight of incremental functionality, as well as a multi-year re-architecture of a complex in-flight application. Please apply here.
    • Enterprise Software Engineer.  Ability to develop detailed design and deliver a scalable implementation. Hands-on development of new code and/or managing existing code as part of a group and/or alone. Evaluating products including open-source modules and if need be incorporating them into projects. Should be able to lead a small group of developers to develop and maintain systems. Please apply here
    • Senior Cocoa Engineer. Apple is seeking a senior Cocoa engineer to join the IS&T Client Frameworks team. Are you someone looking to solve technically challenging problems involving a wide range of technologies (client, server, web)? Please apply here.
    • Web Application Engineer. We are looking for a team player with focus on designing and developing WWDR’s web-based applications. The successful candidate must have the ability to take minimal business requirements and work pro-actively with cross functional teams to obtain clear objectives. Please apply here.
    • Web Application Engineer. We are looking for a team player with focus on designing and developing WWDR’s web-based applications. The successful candidate must have the ability to take minimal business requirements and work pro-actively with cross functional teams to obtain clear objectives. Please apply here.
    • Senior Web Developer: Worldwide Developer Relations. Responsible for the architecture, design and development of the user interface of WWDR’s web applications. Work with UI designer and marketing to analyze business requirements and contribute to functional requirements. Collaborate with server-side software engineers on design, document technical specifications, and implement proper solutions. Please apply here
    • Sr Software Engineer. The iOS Systems Team is looking for a Software Engineer to work on operations, tools development and support of worldwide iOS Device sales and activations. Please apply here
    • Sr. Software Engineer. The Identity Management Services team at Apple is in search of a motivated Senior Software Engineer who is self-driven and has a proven track record in design and development of complex, highly available and scalable systems. Please apply here
    • SQE and Operations Manager, iOS Systems. The iOS Systems team is looking for an experienced hands-on manager to lead the Quality Engineering, Build and Release Engineering team. Please apply here
    • Senior Engineer: Emerging Technology. Apple’s Emerging Technology group is looking for a senior engineer passionate about exploring emerging technologies to create paradigm shifting cloud based solutions. Please apply here. 
    • Senior Storage Engineer. Software Engineering Operations (SEO) is seeking an experienced storage engineer to join our team. This role will focus on designing, deploying and maintaining critical SAN and NAS storage solutions. Please apply here

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.

  • FreeAgent are looking for a talented Operations Engineer to come and work on the FreeAgent app, internal services and supporting infrastructure. You'll be working alongside our Ops team squashing single points of failure, fixing bottlenecks, profiling load and solving interesting scaling and automation problems. Please apply here

  • Intechnica is looking for Performance Architects, Performance Engineers, a Lead Automation Engineer, and a Solution Assurance Analyst. If making super-fast systems is your forte, send your CV with covering letter to careers@intechnica.co.uk.

  • Stackdriver is looking for systems + cloud + dev + ops guru to serve as our liaison within the DevOps community. If you are passionate about monitoring and automation, enjoy working on open source, and are excited by the prospect of sharing your expertise with your peers, get in touch with us today! http://bit.ly/143ARmy

  • We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.

  • LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online

  • New Relic is looking for a Java Instrumentation Engineer, Java Scalability Engineer,  Distributed Systems Engineer and Android app engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter? 

Fun and Informative Events

  • Your event here.

Cool Products and Services

  • NuoDB Blackbirds Release 2.0 Birthday. They grow up so fast these days! What people love about NuoDB is that it’s stable, always there for you and its flexible. Which is why it’s winning all kinds of popularity competitions, from “Most Likely to Succeed” through “Least Likely To Fall Over Sharding” to “Most Likely to Be ACID Compliant”. 

  • Rapidly Develop Hadoop MapReduce Code. With ScaleOut hServer™ you can use a subset of your Hadoop data and run your MapReduce code in seconds for fast code development and you don’t need to load and manage the Hadoop software  stack, it's a self-contained Hadoop MapReduce execution environment. To learn more check out www.scaleoutsoftware.com/prototypehadoop/

  • CloudStats.me - Monitor all your VPS, Dedicated and Cloud servers from one place. Whether you have only one server or hundreds of them, you will be able to check their status in seconds from the dashboard. Try server monitoring now for free.

  • MongoDB Backup Free Usage Tier Announced. We're pleased to introduce the free usage tier to MongoDB Management Service (MMS). MMS Backup provides point-in-time recovery for replica sets and consistent snapshots for sharded systems with minimal performance impact. Start backing up today at mms.mongodb.com.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • NEW! Aerospike 3 - Download FREE. Introducing the new Aerospike 3 database that builds off of Aerospike's legacy of speed, scale, and reliability, adding an extensible data model that supports complex data types, large data types, queries using secondary indexes, user defined functions (UDFs) and distributed aggregations using Stream UDFs for real-time data.

  • The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.

  • aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required. http://aiscaler.com/

  • LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

Monday
Oct282013

Design Decisions for Scaling Your High Traffic Feeds

Guest post by Thierry Schellenbach, Founder/CTO of Fashiolista.com, follow @tschellenbach on Twitter and Github

Fashiolista started out as a hobby project which we built on the side. We had absolutely no idea it would grow into one of the largest online fashion communities. The entire first version took about two weeks to develop and our feed implementation was dead simple. We’ve come a long way since then and I’d like to share our experience with scaling feed systems.

Feeds are a core component of many large startups such as Pinterest, Instagram, Wanelo and Fashiolista. At Fashiolista the feed system powers the flat feed, aggregated feed and the notification system. This article will explain the troubles we ran into when scaling our feeds and the design decisions involved with building your own solution. Understanding the basics of how these feed systems work is essential as more and more applications rely on them.

Furthermore we’ve open sourced Feedly, the Python module powering our feeds. Where applicable I’ll reference how to use it to quickly build your own feed solution.

Introduction to Feeds

The problem of scaling feed systems has been widely discussed, but let me start by clarifying the basics:

Click to read more ...

Friday
Oct252013

Stuff The Internet Says On Scalability For October 25th, 2013

Hey, it's HighScalability time:


Test your sense of scale. Is this image of something microscopic or macroscopic? Find out.
  • $465m: Amount lost in 45 minutes due to a software bug. Where? Where else...the finance industry.
  • Quotable Quotes:
    • FCC: Fiber-to-the-home, on average, has the best performance in terms of latency, with 18 ms average during the peak period, with cable having 26 ms latency and DSL 44 ms latency.
    • @CompSciFact: "About 1,000 instructions is a reasonable upper limit for the complexity of problems now envisioned." -- John von Neumann, 1946
    • @anildash: healthcare.gov got 20M unique visitors in 20 days, faster than Google+ launch. Took Pinterest 2 years & BuzzFeed 4 years to hit 20M.
    • Thomas A. Edison: I start where the last man left off.
    • @brycebaril: I've never had a tech conference toy with my emotions like this year's #realtimeconf

  • Great explanation of the Netflix people don't know, their CDN. Chaos Kong is Coming: A Look At The Global Cloud and CDN Powering Netflix:  Netflix sees about 2 billion requests per day to its API, which serves as the “front door” for devices requesting videos, and routes the requests to the back-end services that power Netflix. That activity generates about 70 to 80 billion data points each day that are logged by the system.

  • Healthcare.gov Didn’t Work in Tests, Launched Anyway. This is just getting silly. When have projects built and released like this ever worked? Especially under huge huge initial loads. Never (or close to). This stuff is complicated for many reasons on every level. To compare such a product with a website is the height of technical ignorance. I recall Captain Renault: I'm shocked, shocked to find that gambling is going on in here!

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Wednesday
Oct232013

Strategy: Use Linux Taskset to Pin Processes or Let the OS Schedule It?

This question comes from Ulysses on an interesting thread from the Mechanical Sympathy news group, especially given how multiple processors are now the norm:

Ulysses:

  1. On an 8xCPU Linux instance,  is it at all advantageous to use the Linux taskset command to pin an 8xJVM process set (co-ordinated as a www.infinispan.org distributed cache/data grid) to a specific CPU affinity set  (i.e. pin JVM0 process to CPU 0, JVM1 process to CPU1, ...., JVM7process to CPU 7) vs. just letting the Linux OS use its default mechanism for provisioning the 8xJVM process set to the available CPUs?
  2. In effrort to seek an optimal point (in the full event space), what are the conceptual trade-offs in considering "searching" each permutation of provisioning an 8xJVM process set to an 8xCPU set via taskset?

Given taskset is they key to the question, it would help to have a definition:

Click to read more ...

Monday
Oct212013

Google's Sanjay Ghemawat on What Made Google Google and Great Big Data Career Advice

In a People of ACM interview with Sanjay Ghemawat, a Google Fellow in the Systems Infrastructure Group (MapReduce, BigTable, Spanner, GFS, etc), talks about a few interesting aspects of Google's culture.

What Made Google Google

Progress is a modern idea. The conviction that future can be changed for the better through individual advancement and action has over hundreds of years driven an exponential growth in the technome.

What drives progress? Challenges. Individuals finding and defeating a challenge. There's usually something someone wants to do so badly that they put in the effort, the thought, and the money into solving all the problems. The results are often something new and amazing.

And so it was for Google:

Click to read more ...

Friday
Oct182013

Stuff The Internet Says On Scalability For October 18th, 2013

Hey, it's HighScalability time:


Test your sense of scale. Is this image of something microscopic or macroscopic? Find out.
  • $3.5 million: Per Episode Cost of Breaking Bad
  • Quotable Quotes:
    • @GammaCounter: "There are 400 billion trees in the Amazon River basin, close to the number of stars in the Milky Way galaxy." 
    • @rbranson: Virtualization has near-zero overhead, unless the VM spends most of it's time copying between RAM and network… like memcached or haproxy.
    • @HackerNewsOnion: Programming is 1% inspiration, 99% trying to get your environment working.
    • @aneel: "roundtrips, not bandwidth, is now often the bottleneck for most applications"
    • @jamesurquhart: Not to mention the fact that auto-scaling should happen above IaaS layer. Think multi-cloud.
    • Sheref Mansy: A machine keeps sort of chugging away, without worrying about its environment. But a living system has to.
    • V.D. Veksler: it just came to my attention that Javascript v8 is faster than Python. I could not believe it, thought it might just be CPython.
    • Doron Rajwan: For the past 30 years, computer performance has been driven by Moore’s Law; from now on, it will be driven by Amdahl’s Law.
    • Bjarne Stroustrup: There are only two kinds of languages: the ones people complain about and the ones nobody uses.
  • Steve Souders and John Allspaw, the Laurel and Laurel of the DevPerfOps world had a really good interview at Velocity. Some trends...mobile is huge; there's now a big focus on rendering performance; institutionalizing failure - planning and doing something with failure, failure is a friend, not a scary monster; don't panic, when there's a problem figure out what's going on first; Humans and Machines are buds, they are cooperative, not John Henry like adversaries.
  • Forget the history of Kings and peoples. Here's a far more interesting history, the History of Packets. Cool look at how TCP has changed over time, a description of how packets work, and the history of the Internets. Kind of boring as there are no beheadings.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Wednesday
Oct162013

Interview With Google's Ilya Grigorik On His New Book: High Performance Browser Networking

If you are Google you don't just complain about performance on the web, you do something about it. Doing something about web performance is the job of one Ilya Grigorik, Developer Advocate, Make the Web Fast at Google, and author of a great new book: High Performance Browser Networking: What every web developer should know about networking and web performance

That's a big topic you might be saying to yourself. And it is. The book is 400 plus information packed pages. But never fear. Ilya writes in a very straightforward style. It’s like a man page for the web. Which is a good thing.

In case you are not familiar with Ilya, he's the perfect choice for writing such an ambitious book. For years Ilya has been producing excellent content on his blog and if you search YouTube you'll find presentation after presentation on the topics found in the book. Authority established.

Reading the book I was struck by what a complicated beast or little World Wide Web has become. That's clear from just the chapter titles: Primer on Latency and Bandwidth, Building Blocks of TCP, Building Blocks of UDP, Transport Layer Security, Introduction to Wireless Networks, WiFi, Mobile Networks, Optimizing for Mobile Networks, Brief History of HTTP, Primer on Web Performance, HTTP 1.X, HTTP 2.0, Optimizing Application Delivery, Primer on Browser Networking, XMLLHttpRequest, Server Side Events, WebSocket, and WebRTC. 

I've often imagined there's a white board at Google with "Don't Erase!" written in red at the top. On it is a complicated diagram of every part of the web and Google's master plan for making that part better. The book reads a little like that imagined diagram. The Primer on Latency is a real eye opener. If you wonder why people are always going on about latency then this chapter is for you. The chapter on Mobile Networks is really good, it fills in a lot of details in an excessively complicated space. With HTTP 2.0 on the horizon you can learn about the entire thing here in full on gory detail. I'd never heard of Server Side Events before so that was enlightening. And the WebRTC chapter has an amazing amount of detail on an exciting new browser capability. 

If you've been burned by books that were just reprints of manuals and specification documents, that's not the case with this book. It's full of practical real-world advice. For example, there' one example of how "Apple engineers saw a 300% performance improvement for users on slower networks once they made better reuse of existing TCP connections within iTunes, via HTTP keepalive and pipelining!" Bits like this can be found throughout the book.

And many chapters end with a practical set of things you can do to implement all the stuff you've learned part. For example, the WebRTC chapter ends with a Performance Checklist that begins with a few sections like:

Signaling service
• Use a low-latency transport.
• Provision sufficient capacity.
• Consider using signaling over DataChannel once connection is established.
Firewall and NAT traversal
• Provide a STUN server when initiating RTCPeerConnection.
• Use trickle ICE whenever possible—more signaling, but faster setup.
• Provide a TURN server for relaying failed peer-to-peer connections.

 

Almost every chapter has solid practical advice like this on how to make the web faster for your application.

Here's my email interview with Ilya Grigorik on High Performance Browser Networking. Enjoy.

Please Tell Us Who You Are And What You've Brought To Show And Tell Today?

Click to read more ...

Tuesday
Oct152013

Sponsored Post: Apple, ScaleOut, FreeAgent, CloudStats.me, Intechnica, Couchbase, MongoDB, Stackdriver, BlueStripe, Booking, Rackspace, AiCache, Aerospike, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

  • Apple is hiring for multiple positions. Imagine what you could do here. At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly.
    • Sr. Software Engineer. The Identity Management Services team at Apple is in search of a motivated Senior Software Engineer who is self-driven and has a proven track record in design and development of complex, highly available and scalable systems. Please apply here
    • SQE and Operations Manager, iOS Systems. The iOS Systems team is looking for an experienced hands-on manager to lead the Quality Engineering, Build and Release Engineering team. Please apply here
    • Senior Engineer: Emerging Technology. Apple’s Emerging Technology group is looking for a senior engineer passionate about exploring emerging technologies to create paradigm shifting cloud based solutions. Please apply here. 
    • Senior Storage Engineer. Software Engineering Operations (SEO) is seeking an experienced storage engineer to join our team. This role will focus on designing, deploying and maintaining critical SAN and NAS storage solutions. Please apply here

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.

  • FreeAgent are looking for a talented Operations Engineer to come and work on the FreeAgent app, internal services and supporting infrastructure. You'll be working alongside our Ops team squashing single points of failure, fixing bottlenecks, profiling load and solving interesting scaling and automation problems. Please apply here

  • Intechnica is looking for Performance Architects, Performance Engineers, a Lead Automation Engineer, and a Solution Assurance Analyst. If making super-fast systems is your forte, send your CV with covering letter to careers@intechnica.co.uk.

  • Stackdriver is looking for systems + cloud + dev + ops guru to serve as our liaison within the DevOps community. If you are passionate about monitoring and automation, enjoy working on open source, and are excited by the prospect of sharing your expertise with your peers, get in touch with us today! http://bit.ly/143ARmy

  • We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.

  • LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online

  • New Relic is looking for a Java Instrumentation Engineer, Java Scalability Engineer,  Distributed Systems Engineer and Android app engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter? 

Fun and Informative Events

  • Your event here.

Cool Products and Services

  • Rapidly Develop Hadoop MapReduce Code. With ScaleOut hServer™ you can use a subset of your Hadoop data and run your MapReduce code in seconds for fast code development and you don’t need to load and manage the Hadoop software  stack, it's a self-contained Hadoop MapReduce execution environment. To learn more check out www.scaleoutsoftware.com/prototypehadoop/

  • CloudStats.me - Monitor all your VPS, Dedicated and Cloud servers from one place. Whether you have only one server or hundreds of them, you will be able to check their status in seconds from the dashboard. Try server monitoring now for free.

  • The leading technology companies use Couchbase as their NoSQL database. Download the free open-source version of Couchbase Server and make something awesome today.

  • MongoDB Management Service (MMS) is a cloud-based suite of services for managing MongoDB deployments. In addition to monitoring and alerting, now you can seamlessly back up your MongoDB deployment to the cloud using using MMS. To get started with monitoring and backup, visit mms.mongodb.com.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • NEW! Aerospike 3 - Download FREE. Introducing the new Aerospike 3 database that builds off of Aerospike's legacy of speed, scale, and reliability, adding an extensible data model that supports complex data types, large data types, queries using secondary indexes, user defined functions (UDFs) and distributed aggregations using Stream UDFs for real-time data.

  • The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.

  • aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required . http://aicache.com/

  • LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

Friday
Oct112013

Stuff The Internet Says On Scalability For October 11th, 2013

Hey, it's HighScalability time:


In honor of Twitter's Cha-Ching moment, here's Twitter By the Numbers
  • Quotable Quotes:
    • @BrandonBloom: Mutable data structures are only faster b/c they have fewer features: ie no persistence. You must manually recover that feature w/ copying.
    • @hayesdrumwright: Scale breaks hardware Speed breaks software Scale and speed breaks everything Adrian Cockcroft - Netflix #TechSummit
    • Gladwell, Malcolm: Saul thinks of power in terms of physical might. He doesn’t appreciate that power can come in other forms as well— in breaking rules, in substituting speed and surprise for strength.
  • Now here's an irony. The East India Company established a major trading post at Bantam in Java. They called their trading posts "factories." Get it? Java. Factories. That's stranger than fiction.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...