No to SQL? Anti-database movement gains steam – My Take

In this post i wrote my view on the anti SQL database movement and where the alternative approach fits in:

- SQL databases are not going away anytime soon.
- The current "one size fit it all" databases thinking was and is wrong.
- There is definitely a place for a more a more specialized data management solutions alongside traditional SQL databases.

In addition to the options that was mentioned on the original article i pointed out the the in-memory alternative approach and how that fits into the puzzle. I used a real life scenario: scalable Social network based eCommerce site where i outlined how in-memory approach was the only option they could scale and meet their application performance and response time requirements.

Comments

[Adron]

I work at Webtrends, and since we're talking billions of page views per day that we collect a traditional relational database just doesn't cut it for collection of that data. The continuous nature and speed of this volume being collected really draws up a lot of questions in regards as what and how we work around these needs.

With the number of sites out there that have massive hit counts and massive volumes of data I see the demand for something besides traditional relational databases. They still serve tried and true in Enterprises and hundreds of other roles, but the web realm is really hammering a need for something new, consistent, and different.

[Bradford]

I've been focusing a lot on this topic lately on my blog at http://www.roadtofailure.com

I have a new article, "Decline of the Data Warehouse", about how vendors are terrified of the revolution in Scalable Data Warehousing. Hadoop represents the first radical shift in a long time in an industry that was built around “pay big or go home

I also have an article called "Social Media Kills the Database", about the coming irrelevance of the ACID, transaction-oriented DB for a lot of situations. You can find it at http://www.roadtofailure.com/2009/06/19/social-media-kills-the-rdbms/

[natis]

Bradford thanks for the reference - I just posted the following comment to your blog:

That’very insightful analysis - I’m going to add it to the links of references on my recent post No to SQL? Anti-database movement gains steam – My Take

Based on your requirements i was wondering if you also considered any of the In-Memory approaches? In my post i was specifically referring to a social network application that needed to find a network of 2500 friends of friends out of a database more x100,000 users in every query and was able to do that in just few msec.

[Fazal Majid]

I expect RDBMSes running on SSDs will replace many in-memory DB applications.

[natis]

I don't think so. The RDBMS overhead is inherit with the model itself i.e. the fact that data needs to be flattened into stream of bytes every time you need to store it or read it.

The benefit of In-Memory alternative iss not just the speed of I/O but also the fact that it enables you to take advantage of the fact that your data lives withing the application memory space. Storing your data on SSD would still goes through serialization and deep copy of data every time you access it. You can't inject code that will access this data by reference and the ability to manage and trigger events in an effective way doesn't really exist.

So i would say that SSD is going to improve RDBMS i/O performance and scalability but still inherit large part of the overhead associated with RDBMS such as serialization, indexing as well the fact that the data is always going to be remote to your application.

[sudsmenon]

Relational databases were developed as disk page based storage systems and on top of that, they tacked on more overhead by way of strict adherence to ACID semantics. Additionally they evolved from traditional big iron hardware where everything ran on a big honking box.

In-memory grids have evolved from the ground up to be distributed as well as highly concurrent and use the fact that in-memory storage is the primary mechanism for storing and accessing data, leading to more granular synchronization for the same data consistency.

SSDs will certainly improve access rates but nothing much beyond that.

We have to really separate relational databases from SQL in this discussion however. SQL is a language that is expressive, powerful and can easily apply to in-memory data management solutions as much as it can be applied to traditional databases.

By the same token, indexing is an optimization that allows the query optimizer to improve performance on heavily used queries and not really a limitation of the traditional relational database systems. An in-memory solution can come up with multiple intelligent indexing schemes, especially ones that work on partitioned data and distributed querying.

Last but not the least, databases, because of how they evolved, treat anyone accessing data as a client (in the classic client-server parlance), whereas in-memory data management solutions are able to provide behavior routing and execution on the grid (again, partly an artifact of the evolution of in-memory systems)

But not all behavior should be executed on the data node and an invocation model where the client is indeed invoking queries or other behavior on the server or registering subscriptions or continuous queries is the right way to go if you are looking for extreme scaleout (10s of thousands of clients accessing the data fabric).

So what's my point in all this? I agree that relational databases have gotten a bit long in the tooth and it is time to look at other ways of managing data. But jettisioning SQL as part of the make over is perhaps an extreme step. There are places where key-value stores work great and we should use key-value stores in such cases, but applications that need the rich expressiveness of SQL should be able to use it and continue to derive the benefits that in-memory data management brings to the picture (like redundancy, dynamic rebalancing, transparent fail over and in memory data access speeds)

Amen
Suds

[rc]

I think SQL is used more than before. Look at all the data that used to be stored in flat files! Nowadays more and more devs use Sqlite because it is much more feature rich than a comma separated file. And Sqlite is very simple to deploy. Mobile Phone devs now learn SQL because they see how fast Sqlite is. Sqlite is a really powerful RDBMS.

At my previous job we used to store geographical data in ESRI (an example of a specialized database) but after a few years we switched to Oracle Locator. Oracle makes it possible to combine "normal" data and geographical data with one simple select-statement and you can declare referential integrity between your "normal" data and your geographical data or store it in the same table.

And we used to store our code in Visual Source Safe (an example of a specialized database), but we switched to Team System and Team System uses MS SQL.

Sharepoint is becoming more and more popular and it uses MS SQL.

A lot of companies use the wiki software to start their own 'wikipeda' and this wiki software uses MySQL.

Anjo, The Netherlands

[rc]

Interesting, there is a new fish in the sea and it is called HadoopDB. http://dbmsmusings.blogspot.com/2009/07/announcing-release-of-hadoopdb-longer.html

[rc]

By the way, why is the shown number of comments often wrong on this site? Sometimes I see seven, sometimes I see five. Never the right amount. Has some trade offs between scalability and accuracy been made?