Wolfram|Alpha Architecture
Friday, May 15, 2009 at 1:13AM Making the world's knowledge computable
Today's Wolfram|Alpha is the first step in an ambitious, long-term project to make all systematic knowledge immediately computable by anyone. You enter your question or calculation, and Wolfram|Alpha uses its built-in algorithms and growing collection of data to compute the answer.
Answer Engine vs Search Engine
When Wolfram|Alpha launches later today, it will be one of the most computationally intensive websites on the internet. The Wolfram|Alpha computational knowledge engine is an "answer engine" that is able to produce answers to various questions such as
- What is the GDP of France?
- Weather is Springfield when David Ortiz was born
- 33 g of gold
- LDL vs. serum potassium 150 smoker male age 40
- life expectancy male age 40 finland
- highschool teacher median wage
Wolfram|Alpha excels at different areas like mathematics, statistics, physics, engineering, astronomy, chemistry, life sciences, geology, business and finance as demonstrated by Steven Wolfram in his Introduction screencast.
The Stats
- Abour 10,000 CPU cores at launch
- 10+ trillion of pieces of data
- 50,000+ types of algorithms
- Able to handle about 175 million queries per day
- 5+ million lines of symbolic Mathematica code
The Computers Powering Computable Knowledge
There is no way to know exactly how much traffic to expect, especially during the initial period immediately following the launch, but the Wolfram|Alpha team is working hard to put reasonable capacity in place.
As Stephen writes in the Wolfram|Alpha blog Alpha will run in 5 distributed colocation facilities. What computing power have they gathered in these facilities for launch day? Two supercomputers, just about 10,000 processor cores, hundreds of terabytes of disks, a heck of a lot of bandwidth, and what seems like enough air conditioning for the Sahara to host a ski resort.
One of their launch partners, R Systems, created the world’s 44th largest supercomputer (per the June 2008 TOP500 list - it is listed as 66th per the latest Top500 list). They call it the R Smarr. It will be running Wolfram|Alpha on launch day! R Smarr has a Sum Rmax of 39580 GFlops using Dell DCS CS23-SH, QC HT 2.8 GHz computers, 4608 cores, 65536 GB of RAM and Infiniband interconnect.
Dell is another of the launch partners with a data center full of quad-board, dual-processor, quad-core Harpertown servers. What does it all add up to? The ability to handle 175 million queries (yielding maybe a billion) per day—over 5 billion queries (encompassing around 30 billion calculations) per month.
The Launch of Wolfram|Alpha
Watch a live webcast of the Wolfram|Alpha system being brought online for the first time on
- Friday, May 15, beginning at 7pm CST
The First Killer App of The New Kind of Science
The Genius behind Wolfram|Alpha is Stephen Wolfram. He is best know for his ambitious projects: Mathematica and A New Kind of Science (NKS).
May 14, 2009 marks the 7th anniversary of the publication of his book A New Kind of Science
Status
That it should be possible to build Wolfram|Alpha as it exists today in the first decade of the 21st century was far from obvious. And yet there is much more to come.
As of now, Wolfram|Alpha contains 10+ trillion of pieces of data, 50,000+ types of algorithms and models, and linguistic capabilities for 1000+ domains. Built with Mathematica—which is itself the result of more than 20 years of development at Wolfram Research—Wolfram|Alpha's core code base now exceeds 5 million lines of symbolic Mathematica code. Running on supercomputer-class compute clusters, Wolfram|Alpha makes extensive use of the latest generation of web and parallel computing technologies, including webMathematica and gridMathematica.
How Mathematica Made Wolfram|Alpha Possible?
Wolfram|Alpha is a major software engineering development to make all systematic knowledge immediately computable by anyone. It is developed and deployed entirely with Mathematica—in fact, Mathematica has uniquely made Wolfram|Alpha possible. Here's why.
- Computational knowledge and intelligence
- High-performance enterprise deployment
- One coherent architecture
- Smart method selection
- Dynamic report generation
- Database connectivity
- Built-in, computable data
- High-level programming language
- Efficient text processing and linguistic analysis
- Wide-ranging, automated visualization capabilities
- Automated importing
- Development environment
Information Sources
- Wolfram|Alpha
- About Wolfram|Alpha
- Wolfram|Alpha Blog
- Introduction to Wolfram|Alpha screencast
- A New Kind of Science
- Wolfram Research - Mathematica
- How Mathematica Made Wolfram|Alpha Possible?
- The Secret behind the Computational Engine in Wolfram|Alpha
- More info expected after the launch... stay tuned!
Congratulations Stephen!
Reader Comments (12)
An interesting debate: http://seekingalpha.com/article/124787-wolfram-alpha-google-killer-or-google-acquisition-target">Wolfram Alpha: Google Killer or Google Acquisition Target?
10,000 cores aren't that much in today's server design. What is surprising is that they chose harpertown platform from dell which uses fbdimms. San Clemente would have been better on power. I also wonder why nehalems were not picked given the performance advantage.
What it amounts to is about 1200 servers, with 8 penryn cores each. That is about 30 racks of servers, probably cost only 3 million dollars. Commodity servers are dirt cheap today. Let's see how much performance you can get out of that.
Thanks geekr. Going the super computer route is pretty interesting. It's nice to have money!
Sounds like HAL9000:
I'm sorry Dave, I'm afraid I can't do that...
Wolfram|Alpha ha temporarily exceeded its current maximum test load.
Looks like 10000 cores will not be enough for now :-)
NKS is available online here: http://www.wolframscience.com/nksonline/toc.html
Chapter 1 The Foundations for a New Kind of Science 1
Chapter 2 The Crucial Experiment 23
Chapter 3 The World of Simple Programs 51
Chapter 4 Systems Based on Numbers 115
Chapter 5 Two Dimensions and Beyond 169
Chapter 6 Starting from Randomness 223
Chapter 7 Mechanisms in Programs and Nature 297
Chapter 8 Implications for Everyday Systems 363
Chapter 9 Fundamental Physics 433
Chapter 10 Processes of Perception and Analysis 547
Chapter 11 The Notion of Computation 637
Chapter 12 The Principle of Computational Equivalence 715
Sample from the blog: It’s 3am on the East Coast and we can see from the sampling of our geoIP data that plenty of people are awake and using Wolfram|Alpha. Here’s a sample of 5 seconds on the map:
http://blog.wolframalpha.com/2009/05/18/do-wolframalpha-users-ever-go-to-sleep/
Wolfram|Alpha serves 124 Queries per Seconds as of now.
Do you know the software components of the Wolfram Alpha architecture? Is it based purely on Mathematica? Do they use open source software like apache, mysql, memcached, hadoop or others discussed here on highscalability.com?
http://www.infoq.com/news/2009/05/Wolfram-Alpha
Partial answer to your question about open source software in the Wolfram|Alpha Architecture from the http://www19.wolframalpha.com/faqs.html">FAQ:
What web technology does Wolfram|Alpha use?
Its server technology is based on Apache Web servers accessing clusters of webMathematica servers. On the client side, it's using AJAX(JavaScript).
Is this the start of a new cloud market: Computation as a Service? Wolfram Alpha + Developer APIs: a nice combo!
http://web2.sys-con.com/node/970448">Wolfram|Alpha's Computation as a Service
http://blog.wolframalpha.com/2009/05/26/the-first-week-of-wolframalpha-thank-you/">The First Week of Wolfram|Alpha: Thank You!:
It’s now a week since we officially launched Wolfram|Alpha into the world.
It’s been a great first week.
Approaching 100 million queries. Lots of compliments.
But for me the most striking thing is how many people want to help Wolfram|Alpha succeed.
Making the world’s knowledge computable is a huge undertaking.
And it’s wonderful to see all the help we’re being offered in doing it.
We’ve worked hard to construct a framework. But to realize the full promise of computable knowledge, we need a lot of input and support.
And it’s remarkable that in just one week, there’s already a whole community springing up around Wolfram|Alpha—with our own Community site and many independent sites.
We’ve had lots of feedback from lots of people. In fact, in the past 7 days, we’ve had no less than 55,000 feedback messages posted to the site.
Suggestions. Encouragements. Corrections. Obvious things. Incredibly obscure things.
At some level, it’s quite overwhelming. But it’s also incredibly helpful.
...
From the W|A Blog: Building Rome in a Day
If you’ve been following the launch of Wolfram|Alpha, then you have probably heard that two supercomputer-class systems are a big part of what is behind the scenes. One of them is the R Smarr system, belonging to our good friends at R Systems, which is featured in http://www.youtube.com/watch?hl=en&v=umEKJrlbw9s">this video. The other is our custom Dell system, highlighted in the http://www.youtube.com/watch?v=qUFmqshsnXU">Rack ‘n’ Roll video. Between the two of them, we can handle around 1800 queries per second (qps). Many people have asked about how we pulled together all of this infrastructure.
See details on the blog.