Paper: A Web of Things Application Architecture - Integrating the Real-World into the Web

How do you layer a programmable Internet of smart things on top of the web? That's the question addressed by Dominique Guinard in his ambitious dissertation: A Web of Things Application Architecture - Integrating the Real-World (slides). With the continued siloing of content, perhaps we can keep our things open and talking to each other?

In the architecture things are modeled using REST, they will be findable via search, they will be social via a social access controller, and they will be mashupable. Here's great graphical overview of the entire system:

Abstract:

Click to read more ...

Sponsored Post: Rackspace, Simple, Fitbit, Amazon, Booking, aiCache, Aerospike, Percona, ScaleOut, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 
• We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.
• Help build the platform that powers a better, fairer banking experience at Simple. Join a talented team that chooses its own tools; works across web, Android, iOS, and Ruby/Scala/Clojure backend apps; and develops a secure and scalable banking service on AWS. Learn more at careers.
• Fitbit is hiring a Site Operations Lead to help us on our mission to make the world a healthier place! Fitbit's wearable fitness devices are worn by people across the world, each syncing with the web site, wirelessly and automatically, every 15 minutes. Join our mission here!
• The AWS Relational Database Service (RDS) automates management of relational databases in the cloud. We have a wide variety of customers and are part of many mission-critical applications, like the ones built by the 2012 Obama re-election campaign. If you're interested in joining a fast-growing service and team, please send your resume to rds-jobs@amazon.com.
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs
• Aerospike is Hiring! You dream in C - and like it? Then join us as a Senior Distributed Systems Engineer or Client / Application Engineer. People covent your bag of tricks for troubleshooting systems and network issues? Join our Operations and QA team. See if these positions are a fit for you! 

Fun and Informative Events

• It's back! Join the MySQL Community at the annual Percona Live MySQL Conference and Expo in Santa Clara, April 22-25. This year's conference features an outstanding lineup of 92 speakers delivering 112 breakout sessions over three days! 

Cool Products and Services

• The Rackspace Cloud Application Programming Interface (API) has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code.  Read about three of the most popular things that customers do with the Rackspace AP.
• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free. No sign-up required. http://aicache.com/deploy
• New Benchmark shows Aerospike nearly 10x Faster than the Competition. Thumbtack Technology YCSB Benchmark shows Aerospike nearly 10x faster than Cassandra, Couchbase and Mongodb. Read it now!
• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

Khan Academy Checkbook Scaling to 6 Million Users a Month on GAE

Khan Academy is a non profit company started by Salman Khan with the Big Hairy Audacious Goal of providing a free, world class education to anyone, anywhere, anytime. That’s a lot of knowledge. Having long been inspired and captivated by the Khan Academy, I was really curious to know how they plan to do it. Ben Kamens, lead developer at Khan Academy, gives the somewhat surprising answer in an interview: How to Scale your Startup to Millions of Users.

The short answer: develop a strong team, focus on features, let Google App Engine do the heavy lifting.

Some people seem to be turned off by all the GAE love in the interview. Part of it is that the interviewer is Fred Sauer, Developer Advocate for Google App Engine, so there’s a level of familiarity between the two. But the biggest part is simply that they really like GAE, for all the reasons your are supposed to like GAE. And that’s OK. In this day and age you are free to love whichever platform you choose.

Biggest surprise:

• A profile on 60 Minutes drove more traffic than TechChrunch, HackerNews, and everything else combined. Old media is not dead.

Part I liked the best:

• GAE is an abstraction over all the typical scalability issues and that let’s you focus on business problems. All abstractions leak, you are going to have to deal with problems no matter what you choose, but you are choosing the type of problems you want  to deal with by the platform you select. It's all about understanding the tradeoffs you're making.

Here’s my gloss on the major takeaways from the interview:

Click to read more ...

Stuff The Internet Says On Scalability For March 29, 2013

Hey, it's HighScalability time:

(Ukrainian daredevil scaling buildings)

• 44.6 billion - Tumblr posts; 300 Gb/s - DDoS DNS amplification attacks; 100 million - Eventbrite tickets processed.
• Quotable Quotes:
  • @tveskov: Alan Kay: “The past 30 years have been completely mundane. It’s all been scaling (of old technology) and Angry Birds” 
  • @phrawzty: OH "Complexity is accelerating. We must augment our ability to manage it." #monitorama
  • @stallent: new wave of apps that are bringing the server doing actual legit valuable work back into vogue know scaling is more than hard
  • @calvdee: Fending off a 300Gb/s DDoS attacks would constitute a feat of #highscalability 
  • @solarce: "A spider farted in Finland and screwed up my IOPS!" -- @lusis #monitorama
  • @cra: Listening to Shawn Pearce talk about scaling #git at Google with JGit... Android AOSP repos: 19.4GB, 2.5Mreq/day, 5.0TB/day #eclipsecon
  • @codemonkeyism: Sometime in the future Erlang people will realize scalability comes from async not actors and then hell freezes over.
• InfoQ in China has graciously offered to translate some of my articles into Chinese. So here's Iron.io Moved From Ruby to Go: 28 Servers Cut and Colossal Clusterf**ks Prevented in the original Chinese.


• To justify high margin devices you need user data to sync automagically between all devices a user owns, otherwise you would just leave everything on the cloud and stream it to much cheaper devices. But Apple is finding device syncing in a complex error prone world is not so simple: The promise of iCloud’s Core Data support is that it will solve all of the thorny issues of syncing a database by breaking up each change into a transaction log. Except it just doesn’t work. 


• Food trucks are the cloud of the restaurant world. Dream friendly, flexible hours, low startup costs, making good money requires selling alcohol, and you can just pack up and move whenever you want. Oh, well, maybe not.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

The Changing Face of Scale - The Downside of Scaling in the Contextual Age

Robert Scoble is a kind of Brothers Grimm for the digital age. Instead of inspired romantics walking around the country side collecting the folk tales of past ages, he is an inspired technologist documenting the current mythology of startups.

One of the developments Robert is exploring is the rise of the contextual age. Where every bit of information about you is continually being prodded, pulled, and observed, shoveled into a great learning machine, and turned into a fully actionable knowledge graph of context. A digital identity more real to software than your physical body ever was. 

Sinner or saviour, the Age of Context has interesting implications for startups. It raises the entrance bar to dizzying heights. Much of the reason companies are tearing down the Golden Age of the Web, one open protocol at a time, is to create a walled garden of monopolized information.

To operate in this world you will have to somehow create a walled garden of your own. And it will be a damn big garden. So you must scale. Facebook and Google have an insane number of servers. Then you must gather to yourself the people capable of running these servers, creating the applications and cutting edge devices that tease out the data, store the data, learn from the data, and create a market for the data. A serious capital and technological barrier to entry. 

It's popular to say don't worry about scaling until you need to. It's a wonderful problem to have, etc. A curious aspect of these new contextual systems, Robert noticed, is that they need to plan on scale from the start. Not only must the infrastructure scale to gather and process context and turn it into smart data, the cost of each new customer is now huge.

One of the upsides of your typical software service is new customers are incrementally cheap to service, so they are highly profitable. The data is typically dead. It just sits in inexpensive storage and comes alive on demand, only when a user needs it. Economies of scale are your friend.

In the contextual age economies of scale are not your friend...

Click to read more ...

AppBackplane - A Framework for Supporting Multiple Application Architectures

Hidden in every computer is a hardware backplane for moving signals around. Hidden in every application are ways of moving messages around and giving code CPU time to process them. Unhiding those capabilities and making them first class facilities for the programmer to control is the idea behind AppBackplane.

This goes directly against the trend of hiding everything from the programmer and doing it all automagically. Which is great, until it doesn't work. Then it sucks. And the approach of giving the programmer all the power also sucks, until it's tuned to work together and performance is incredible even under increasing loads. Then it's great.

These are two different curves going in opposite directions. You need to decide for your application which curve you need to be on.

AppBackplane is an example framework supporting the multiple application architectures we talked about in Beyond Threads And Callbacks. It provides a scheduling system that supports continuous and high loads, meets critical path timing requirements, supports fair scheduling amongst priorities; is relatively easy to program; and supports higher degrees of parallelism than can be supported with a pure tasking model.

It's a bit much for simple applications. But if you are looking to go beyond a basic thread per request model and think of an application as a container where a diverse set of components must somehow all share limited resources to accomplish work, then some of the ideas may prove useful.

In case you are still wondering where the name AppBackplane comes from, it's something I made up a while ago as a takeoff of computer backplane:

Click to read more ...

Stuff The Internet Says On Scalability For March 22, 2013

Hey, it's HighScalability time: 

• 1 Billion/Month : Active Mostly Mobile YouTube Users
• Quotable Quotes:
  • @NeckbeardHacker: "Wait...he reimplemented swatch and rsync in chef, node and mongo?""Yup." "......Why?""Go easy on him...he's a Ninjipsterstar."
  • @b6n: Scale myth #1: Your service is a unique snowflake.
  • @polotek: If you don't care about bugs, design or scalability, it only takes 2 days to build anything.
  • @NasHope: Even with the rapid scalability of Rackspace, I've still waited for over an hour for Dominos to actually deliver my pizza.
  • George Dyson: Computers may turn out to be less important as an end product of technological evolution and more important as catalysts facilitating evolutionary processes through the incubation and propagation of self-replicating filaments of code.
  • George Dyson: Von Neumann believed that all fields of science, including pure mathematics, derive their sustenance through contact with real problems in the physical world.
• For those of us on the outside looking in, Sebastian Stadil gives a rare view from the cool kids table with an early look at How Google Compute Engine stacks up to Amazon EC2. Is it as wonderful as we all imagined? The differences: AWS has a much richer set of services; GCE is on-demand only, so AWS can be cheaper; GCE faster disk; GCE faster network IO, especially between datacenters; GCE faster boot times, GCE can mount read-only partitions multiple machines; GCE shares images across regions. Sebastian ponders what new architectures Google's feature set will encourage to flourish? Interesting was the idea that because the inter-datacenter network IO is so fast it will be possible to put read slaves in multiple datacenters, replicate in real-time. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Dart - Is it the Future of the Web?

John McCutchan, after a long career spent working on the Linux kernel and being hired out as a code optimization guru, joined Google's Dart team. A curious hire until you watch Bringing SIMD to the Web via Dart, where John makes a programmer accessible explanation of why he likes Dart: performance, performance, performance.

Dart is an open-source Web programming language developed by Google. The motivation for Dart is twofold: provide a language capable of scaling up to the complex web applications that are becoming the norm. Think Gmail. And provide a single language capable of working on both the client and server. Towards those ends Dart is a complete language, full tool environment, and provides an advanced Web UI framework for building web applications at a high level of abstraction.

Why isn't JavaScript good enough? The fear is for large web apps the nature of JavaScript puts crippling limits on potential performance improvements, which will cause web apps to lose out to mobile apps. Before you scream in protest, Dart is created by the same guys who worked on V8, so all their JavaScript experience has fed into Dart.

John’s pitch is that despite many assertions to the contrary, the web is not fast. For example, when comparing the performance of a 2D physics engine across C and the web, the web version is 20x slower than C. Java is twice as slow as C. So the browser is not fast. Dart and the new libraries it supports can close that gap, allowing for the development of high performance applications in the browser.

You may be wondering which of the gazillion web frameworks to use and Dart may just be on the edge of your radar. If so this is an excellent video to take a look at.

Why might you like Dart? Here’s John’s list:

Click to read more ...

Sponsored Post: Fitbit, OLO, Amazon, aiCache, Aerospike, Percona, ScaleOut, New Relic, Logic Monitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• Fitbit is hiring a Site Operations Lead to help us on our mission to make the world a healthier place! Fitbit's wearable fitness devices are worn by people across the world, each syncing with the web site, wirelessly and automatically, every 15 minutes. Join our mission here! 
• OLO's food ordering platform powers some of the largest restaurant chains and feeds millions of consumers. We're looking for Senior C# Software Engineers and DevOps Engineers to help us scale our system. Apply here.
• The AWS Relational Database Service (RDS) automates management of relational databases in the cloud. We have a wide variety of customers and are part of many mission-critical applications, like the ones built by the 2012 Obama re-election campaign. If you're interested in joining a fast-growing service and team, please send your resume to rds-jobs@amazon.com.
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs
• Aerospike is Hiring! You dream in C - and like it? Then join us as a Senior Distributed Systems Engineer or Client / Application Engineer. People covent your bag of tricks for troubleshooting systems and network issues? Join our Operations and QA team. See if these positions are a fit for you! 

Fun and Informative Events

• It's back! Join the MySQL Community at the annual Percona Live MySQL Conference and Expo in Santa Clara, April 22-25. This year's conference features an outstanding lineup of 92 speakers delivering 112 breakout sessions over three days! 

Cool Products and Services

• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free. No sign-up required. http://aicache.com/deploy
• New Benchmark shows Aerospike nearly 10x Faster than the Competition. Thumbtack Technology YCSB Benchmark shows Aerospike nearly 10x faster than Cassandra, Couchbase and Mongodb. Read it now!
• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

Beyond Threads and Callbacks - Application Architecture Pros and Cons

There's not a lot  of talk about application architectures at the process level. You have your threads, pools of threads, and you have your callback models. That's about it. Languages/frameworks making a virtue out of simple models, like Go and Erlang, do so at the price of control. It's difficult to make a low latency well conditioned application when a power full tool, like work scheduling, is taken out of the hands of the programmer.

But that's not all there is my friend. We'll dive into different ways an application can be composed across threads of control.

Your favorite language may not give you access to all the capabilities we are going to talk about, but lately there has been a sort of revival in considering performance important, especially for controlling latency variance, so I think it's time to talk about these kind of issues. When it was do everything in the thread of a web server thread pool none of these issues really mattered. But now that developers are creating sophisticated networks of services, how you structure you application really does matter.

In the spirit of Level Scalability Solutions - The Conditioning Collection, we are going to talk about the pros and cons of some application architectures with an eye to making sure high priority works gets done with low enough latency, while dealing with deadlock and priority inheritance, while making the application developer's life as simple as possible...

Click to read more ...