Tuesday
Sep162008
EE-Appserver Clustering OR Terracota OR Coherence OR something else?
Tuesday, September 16, 2008 at 3:01AM
Hi,
I am very glad that this site exists, as I have learned more about clustering on this site than for quite some time reading stuff elsewhere. Oftentimes, one can find lots of material about clustering, but the practical real-life information is missing. Not so wih this site.
I am currently planning the development of an application which has a lot of enterprise features and requirements. On the other side (if the tiny chance of success might strike us), this application would not be an in-house application of a financial institution, or something like that, but some kind of communit/web 2.0 web site. Thus it is an enterprise application with (hopefully, but surely unlikely) the user numbers of a social networking site. Each user initiated transaction involves huge resssources business logic wise (including insane amounts of encryption oprations).
Of course, I do not intend to induldge into premature scaling, but to invest every minute I have into the implementation of business logic features. Nevertheless, I do not want to make some extremely bad choices which would force a complete reimplementation straight after the first tiny success - i.e. I want to start with the right technology and architecture, but wait with the implementation of the scalability and high availyability features.
Because of the enterprise aspects of this software, my first thought was to use Java SE 6 and Java EE 5 technologies only in order to get all the JEE features and to be vendor independent at the same time.
For implementation and testing purposes I thought of Glassfish v2UR2, Postgresql 8.3 and Solaris 10. As all of the major JEE-Appserver vendors advertise the clustering capabilities, I thought that this could not be a bad move. Hopefully, Glassfish would provide HA and scalability, if not there would always be Geronimo, JBoss, Weblogic, or Websphere.
Now it seems that there are vast differences between different products:
- JEE-Application servers are scaling only to some degree(?). It seems that JEE is almost exclusively used for enterprise applications like SAP ERP or applications at financial institutions? Therefore, there is no need for extreme scalability.
- Terracotta seems to be very nice, as one do not have to learn the insanely huge JEE-technology stack, but can just write a mostly Java-SE-only threaded application(?). But Terracotta does not seem to scale very well either (bottleneck with write-operations caused by the master-worker architecture?) and we would be dependend on the future of the Terracotta Corporation. JEE on the other side is vendor neutral.
- Oracle Coherence. This product seems to be the best distributed caching product and the holy grail of scalability(?). But it is oracle-expensive. Absolutely nothing for a tiny start-up with no financing. JEE is vendor neutral and thus possibly much cheaper.
Do you think that it is possible that one could produce a JEE-Architecture which could provide massive scalability (many hundreds of AppServer) using only the Glassfish clustering features?
Or am I on a completely wrong track? Do we have to plan for Oracle Coherence usage? Are there other possibilities?
Thanks a lot for any opinions or hints!
regards,
mike
I am very glad that this site exists, as I have learned more about clustering on this site than for quite some time reading stuff elsewhere. Oftentimes, one can find lots of material about clustering, but the practical real-life information is missing. Not so wih this site.
I am currently planning the development of an application which has a lot of enterprise features and requirements. On the other side (if the tiny chance of success might strike us), this application would not be an in-house application of a financial institution, or something like that, but some kind of communit/web 2.0 web site. Thus it is an enterprise application with (hopefully, but surely unlikely) the user numbers of a social networking site. Each user initiated transaction involves huge resssources business logic wise (including insane amounts of encryption oprations).
Of course, I do not intend to induldge into premature scaling, but to invest every minute I have into the implementation of business logic features. Nevertheless, I do not want to make some extremely bad choices which would force a complete reimplementation straight after the first tiny success - i.e. I want to start with the right technology and architecture, but wait with the implementation of the scalability and high availyability features.
Because of the enterprise aspects of this software, my first thought was to use Java SE 6 and Java EE 5 technologies only in order to get all the JEE features and to be vendor independent at the same time.
For implementation and testing purposes I thought of Glassfish v2UR2, Postgresql 8.3 and Solaris 10. As all of the major JEE-Appserver vendors advertise the clustering capabilities, I thought that this could not be a bad move. Hopefully, Glassfish would provide HA and scalability, if not there would always be Geronimo, JBoss, Weblogic, or Websphere.
Now it seems that there are vast differences between different products:
- JEE-Application servers are scaling only to some degree(?). It seems that JEE is almost exclusively used for enterprise applications like SAP ERP or applications at financial institutions? Therefore, there is no need for extreme scalability.
- Terracotta seems to be very nice, as one do not have to learn the insanely huge JEE-technology stack, but can just write a mostly Java-SE-only threaded application(?). But Terracotta does not seem to scale very well either (bottleneck with write-operations caused by the master-worker architecture?) and we would be dependend on the future of the Terracotta Corporation. JEE on the other side is vendor neutral.
- Oracle Coherence. This product seems to be the best distributed caching product and the holy grail of scalability(?). But it is oracle-expensive. Absolutely nothing for a tiny start-up with no financing. JEE is vendor neutral and thus possibly much cheaper.
Do you think that it is possible that one could produce a JEE-Architecture which could provide massive scalability (many hundreds of AppServer) using only the Glassfish clustering features?
Or am I on a completely wrong track? Do we have to plan for Oracle Coherence usage? Are there other possibilities?
Thanks a lot for any opinions or hints!
regards,
mike
Reader Comments (7)
Hi Mike,
I haven't seen Glassfish et al used for the largest websites so I can't comment on its capabilities. If you are interested in in-memory datagrid solutions you have a few options: Coherence, Terracotta, GridGain, GigaSpaces, ScaleOut Server, Gemstone or something custom built on memcached. Which to pick depends on your language choice, budget, desired features, yada yada. There's enough reading on all these players to keep you busy! Keep in mind that clustering doesn't always mean scalability, it can just mean high availability.
I think it really depends on what you want to cluster. If you have 100's of servers, the overhead of sending all data to all servers in a cluster would be too much. I'd suggest partitioning your application so that different parts are served by separate clustered components. Or look at a partial data replication solution like buddy replication e.g. jboss cache.
There are a number of ways of making sites handle large volumes of user even with encryption. Take SSL for example most large banks would have some hardware SSL accelerator running in front of their boxes handling the SSL part and passing HTTP to the app servers. This can also be done in software (but not as well) by using Apache infront of an app server handling SSL and passing requests to the app servers via mod_jk. A number of app servers support this method.
Hi Todd, hi noelo_unreg,
thank you very much for your hints! I still have to start to dive into the in-memory datagrid articles, as I needed some time to realize what kind of fundamental understanding problems I still had about (java) clustering: I did not know the difference between Session-HA/Load-Balancing by the JEE AppServer and distributed caching solutions. Now I think that I have understood the whole (or at least a part) of the misery of java clustering(?):
There are two types of clustering:
1. Clustering for HTTP&EJB-Session HA and scalability. This can be done by most JEE Appservers. Older implementations used DBs to to store the session data (which was more secure for HA). Newer implementations mostly use in-memory session handling.
2. Additionally, there is the problem that each component/service of the application might be clustered and that this component by itself (not the application server) wants to store data in a RDBMS. The RDBMS cannot be clustered very easily, so it becomes a 1-node bottleneck for the whole cluster of this component - additionally to the fact that a RDBMS has a bad performance anyway. This is why as much 'scratch data' as Terracotta calls it (data that is only valuable during this session, but does not have to be archived) and maybe even persistent business data has to be kept in a cache in memory (the heap of the JVM).
This caching solution could potentially be the cluster-free solution for the RDBMS-bottleneck problem for small/medium scalability problems: There are affordable Intel/AMD machines available with 8-24 Xeon cores or 32 Opteron cores and 32 to 256GB RAM (Sun Fire X4600 M2 Server). A cache of 200GB in a single JVM accessed by 32 Opteron cores surely would deliver a much better performance than an application server hooked up to a RDBMS. Many medium sized problems could be solved this way: by non-distributed caching.
But there is the garbage collection problem of the JVM. It seems that even with every possible GC-JVM-optimization there is still a full stop GC every few hours that could last minutes if the heap has a size of many GB - totally unaceptable for any interactive (especially web) application.
So it seems that all low to medium scalability requirements could be fulfilled by using JEE AppServer clustering and caching the RDBMS transactions. But because of the GC problems, the in-memory datagrid / distributed caching products are needed not only for huge scalability tasks, but also for small to medium.
I am amazed how a seemingly small technical detail like the implementation of the GC leads to such drastic design decisions like a vendor-lock-in to an in-memory datagrid vendor.
Also thanks a lot for the partial data replication solution tipp!
We do have a heavily component fragmented architecure and intend to cluster the components some day. But I just cannot guess the scalability needs of each component cluster yet.
I will now read all the great in-memory datagrid product articles to understand the differences of the offerings and in order to see which architecture modifications these solutions would require. If they aren't huge, I could hopefully propagate this decision to the time after the implementation of the most important business logic features.
On other thing which may be of use is selective replication where the only the items that have changed in the HTTP session are replicated across the cluster i.e. if only one session attribute has changed the only that is replicated and not the entire session. I only have experience with clustering on JBossAS but I know that you can setup HTTP session replication to only distribute the changes rather than the full session each time.
If the database is a major point of contention you could consider looking at some clustered caches (JBoss cache, Terracotta) or even consider sharding and physically distributing you database to different servers.
Adding to Todd reference one of the things that makes GigaSpaces different than the suggested alternatives is our end to end solution and integration with both Glassfish and Jetty. The integration enables scalability not just at the data layer but include session level clustering and as well as integration with load balancer scaling.
You can see the results of one of our recent benchmarks that shows http://www.gigaspaces.com/WebAppScale">1.4 Billion transaction a day using that model.
Nati S.
Hi,
I'd like to suggest that you evaluate the Resin 4.0 JEE 6 (lite) app serverwhich supports dynamic clustering and includes a distributed persistent cache. The cache is an implementation of the JCache api, is available for applications and also supports session caching.
http://www.caucho.com
Regards,
Fred.
Hi,
I'd like to suggest that you evaluate the Resin 4.0 JEE 6 (lite) app serverwhich supports dynamic clustering and includes a distributed persistent cache. The cache is an implementation of the JCache api, is available for applications and also supports session caching.
http://www.caucho.com
Regards,
Fred.