Monday
Mar302009
Ebay history and architecture
Monday, March 30, 2009 at 11:30PM
Ebay[1]
Starts in 1995, initial name AuctionWeb (V1) :
- very simple architecture
- based on perl
- no database, for data persistence they used plain files
Because of rapid growth they needed to improve their architecture and so V2 (clever name) was born:
- replaced perl with C/C++
- started using a database in a master-slave configuration
- C++ back-end
- XSLT front-end
Any request will lead to an XML file being created in C++ and the XLST processor will transform that into html.
*pretty sophisticated architecture for the 90s, XLST was cutting-edge back then*
That hold out pretty well for a while but in the late 90s ebay experienced an exponential growth.
They started having some trouble with outages and needed improvements, so V3 was developed:
- based on java
- search engine still used C++
- proof that relational databases can scale (aggressive caching)
- developed a messaging layer for making a lot of asyncronious calls, they actually ended up being sued because of the delay in which images appear on the site after an item gets posted :-)
- although they switched to java the basic principle of generating xml files for each request was still used.
Combining the need for a multilingual website with the boom of AJAX technologies and flash applications they started to doubt their XSL system and moved on to what became in 2006 V4 of ebay:
- remove everything that could be replaced with java, Ebay loves Java
Everything is Java (Code):
- Image - Java class
- Link - Java class
- Javascript - Java classes
- Content - Java classes
=> lot's of code to write, they're using Eclipse for developing.
For more details check:
Eclipse at Ebay
Tailoring Eclipse to the eBay architecture
[1] Images and ideas/info are from the links above.
Starts in 1995, initial name AuctionWeb (V1) :
- very simple architecture
- based on perl
- no database, for data persistence they used plain files
Because of rapid growth they needed to improve their architecture and so V2 (clever name) was born:
- replaced perl with C/C++
- started using a database in a master-slave configuration
- C++ back-end
- XSLT front-end
Any request will lead to an XML file being created in C++ and the XLST processor will transform that into html.
*pretty sophisticated architecture for the 90s, XLST was cutting-edge back then*
That hold out pretty well for a while but in the late 90s ebay experienced an exponential growth.
They started having some trouble with outages and needed improvements, so V3 was developed:
- based on java
- search engine still used C++
- proof that relational databases can scale (aggressive caching)
- developed a messaging layer for making a lot of asyncronious calls, they actually ended up being sued because of the delay in which images appear on the site after an item gets posted :-)
- although they switched to java the basic principle of generating xml files for each request was still used.
Combining the need for a multilingual website with the boom of AJAX technologies and flash applications they started to doubt their XSL system and moved on to what became in 2006 V4 of ebay:
- remove everything that could be replaced with java, Ebay loves Java
Everything is Java (Code):
- Image - Java class
- Link - Java class
- Javascript - Java classes
- Content - Java classes
=> lot's of code to write, they're using Eclipse for developing.
For more details check:
Eclipse at Ebay
Tailoring Eclipse to the eBay architecture
[1] Images and ideas/info are from the links above.
Reader Comments (14)
I thought that Ebay was running some custom ASP.NET Stuff, at least for its frontend.
Wouldn't it be a performance hit to have all static content as java class files? Why would you want to wrap an image file into a java class and process it through the JVM?
I have never heard of a company moving from C++ to Java for scalability reasons.. That must have been a performance hit. I wonder how much extra hardware ebay had to purchase.
They used to run IIS but that was back in version 2 of ebay (V2)
the isapi dll piece of the url is there for backward compatibility reasons.
Scalability != Performance
@Marcelb Glad you liked my slides from EclipseCon :-)
Just to answer a few of the questions...
@Shoan There is a Java class to reference any static image. This is just used for forming the URL to the image when the page (HTML) is created on the server. The image itself is completely separate, and in most cases is pushed out to a CDN. The Java class represents the physical resource, so if somebody deleted the resource, code would not compile. The class also helps us to smartly resolve the URL (different URL for dev/QA for example.)
@Jasonfreyt In our experience, Java vs. C++ speed is not much of an issue for a long running server application running on a JVM with a good JIT and with the right tuning options enabled. YMMV. It also matter a lot if your app is bound by memory vs. CPU vs. IO. Java may hurt for memory bound apps. For us at least, it was much easier to horizontally scale the databases with Java in front of them, more than making up for any memory overhead.
@Steve That is right. Any ISAPI.dll you see is for historical reasons. Our front end remained Windows in the V3 world, as the best XSLT we found was MSXML. Unfortunately it also has some ugly memory leak problems (4 bytes/transformation.)
Indeed I like them, thank you for that great presentation which btw is linked in the article.
A few of the sites (I forget which, but I think Poland and Malaysia and a few others) are written using a new Perl code-base. Much more light-weight design when compared with the monstrosity that is the main eBay site.
They threw out Perl back in V1 thinking that it was the problem, wondered why C/C++ didn't come with all the solutions in V2, by V3 went to Java looking for the answers. They now have a situation where it takes > 6 months to get anything out the door and no-one wants to even begin to touch the XSLT for fear of breaking it.
This isn't quite accurate, historically:
V2 didn't use XSL, at all.
V2, which was written in 1997, was an ISAPI app running on Windows NT servers (natch), talking via SQL*Net to an Oracle backend. The ISAPI app emitted straight HTML (unfortunately). NO XML was generated.
Although that architecture sounds peculiar today, at the time neither PostGres nor MySQL had matured enough to compare with Oracle, and it was much faster to develop and debug C++ under Visual C++ than gdb, etc.
That architecture actually carried ebay through it's IPO and into the early 2000's, when V3, which was Java based, replaced it. The biggest problems, scaling wise, was use of a single database instance, which ran on a Sun E10000.
Search was completely different. In V1, it was...grep spawned by an Apache process. In V2, it was Microsoft's search service for a while, then a product called "Thunderstone" for many many years. In the early 2000s it was replaced by an in-house search which continues to serve them well (AFAIK).
You may be right, although some ebay architecture presentations do not mention the old V1, perl based version, and V2 becomes V1.
Did you post this comment just to link to there.com ? :)
Thanks Marcel, really interesting article. It is great to see the development of eBay from a startup using Perl to a very large scale web entity and to see what technologies were used in eBay in a chronological order.
I don't think Michael Wilson sent that comment just to link to there.com :)
http://www.there.com/companyInfo.html">Here writes that he was an early employee at eBay from 1997 to 2001. So he probably has something to talk about on this subject.
Emre is correct, thanks :-).
Also, I'd say that the exponential growth really came under V2.
Also remember that when V3 came out, it "just" covered the Auction business. PayPal was a completely different infrastructure and animal all its own (that's just a footnote).
Thank you for your notes, Michael