How is Berkely DB fare against other Key-Value Database

I want to know how is Berkeley DB compared against other key-value solution. I read it from Net that Google uses it for their Enterprise Sign-on feature. Is anyone has any experience using Berkeley DB. Backward compatibility is poor in Berkley DB but that is fine for me. How easy to scale using Berkeley DB.

How to handle so many socket connection

In my application, we receive request from many clients through socket.Client can connect to this socket and send data. this connection has to be maintained for indefinite hours. Client are continously sending data . There are so many clients simultaneously to server. I am using java to make application which listen on port and do processing on data. How can i scale this socket overhead. Is there any product which helps in maintening socket .

Scaling MySQL on Amazon Web Services

I've recently started working with a large company who is looking to take one of their heavily utilized applications and move it to Amazon Web Services. I'm not looking to start a debate on the merits of EC2, the decision to move to aws is already made (and is a much better decision than paying a vendor millions to host it).

I've done my reasearch and I'm comfortable with creating this environment with one exception, scaling MySQL. I havent done much work with MySQL, i'm more of an Oracle guy up to now. I'm struggling to determine a way to scale MySQL on the fly in a way so that replication works, the server takes its proper place in line for master candidacy, and the apache servers become aware of it.

So this is really three questions:

1. What are some proven methods of load balancing the read traffic going from apache to MySQL.
2. How do I let the load balancing mechanism know when I scale up / down a new Mysql Server?
3. How to alert the master of the new server and initiate replication in an automated environment?

Personally, I dont like the idea of scaling the databases, but the traffic increases exponentially for three hours a day, and then plummets to almost nothing. So this would provide a significant cost savings.

The only way I've read to manage this sort of scaling I read here on slides 18-25:
http://assets.en.oreilly.com/1/event/21/Tricks%20and%20Tradeoffs%20of%20Deploying%20MySQL%20Clusters%20in%20the%20Cloud%20Presentation
Has anyone tried this method and either had success or have scripts available to do this? I try not to remake the wheel when I dont have to. Thanks in advance.

VMware to bridge a DMZ.

Hey guys,

There is a renewed push at my organization to deploy vmware...everywhere.

I am rather excited as I know we have a lot of waste when it comes to resources.

What has pricked my ears up however, is the notion of using this technology in our very busy public facing DMZ's.

Today we get lots of spikes of traffic and we are coping very well. 40x HP blades, apache/php/perl/tomcat/ all in HA behind HA F5's and HA Checkpoint FW's. (20 servers in 2 datacentres).

The idea is, we virtualise these machines, including the firewalls onto hosts vmware clusters that span the public interface to our internal networks. This is something that has gone against the #1 rule I have ever lived by while working on the inet. No airgaps from the unknown to the known!

I am interested in feedback on this scenario.

From a resource perspective, our resource requirements in the DMZ will be lowered over time due to business change and we still have a lot of head room in our capacity.

Do you think this is change for change sake? All I can see is more complexity, higher risk and more skill required to manage what today is a very simple and resilient setup with no security flaws.

VMware and some big name companies/gov agencies stand by the notion the software dividing the host machine is more than capable are keeping the DMZ's in check. It just doesn't sit well with me, knowing we may have a public facing website on the same host machine which is running a critical safety or customer management tool.

Apart from the ease of management to grow/shrink (something we don't need todo in any rush), what are the advantages to increase risk and complexity?

Are any of you in the same position?

Costs wise - our website costs are minuscule compared to the revenue we generate thru them - Would you risk what is a sound and stable environment because it sounds cool to 'virtualise' or is there something I am missing?

Kind regards,
Foodie

ps. I don't post much on here but I love reading your articles. The website I am referring to in my post hits a peak of $250/second and is responsible for 90% of revenue to the business.

Dependency Injection and AOP frameworks for .NET

We're looking to implement a framework to do Dependency Injection and AOP for a new solution we're working on. It will likely get hit pretty hard, so we'd like to chose a framework that's proven to scale well, and operates well under pressure.

Right now, we're looking closely at Spring.NET, Castle Project's Windsor framework, and Unity. Does anyone have any feedback on implementing any of these in large, high traffic environments?

Writing about cisco loadbalancer?

Guys,

At one of my jobs I have to administer a CISCO ACE (application control engine) hardware load-balancer.
I don't particularly love this beast, but it's very very powerful.

There appears to be little real-world info out there, so it could be interesting writing an article on that.

But I don't have other HW LB's to compare it to and I don't want to rehash the product page.


What would interest you in a 'product review' of a loadbalancer?
No replies means it's not an interesting topic, so no article then ;-)

1dbase vs. many and cloud hosting vs. dedicated server(s)?

Me and my partner are making a blueprint for an online webshop service. The purpose of this project is to make webshops available for small company's/ individuals automatically just by creating an account with us. Our webapp can be used to add products/pages/... to the store and we'll handle secure checkout by paypal.

Our app should be scalable and manageable. Because we also want to offer free webshops, the amount of webshops could be +10.000 within a few years. We are building on the Zend framework and are using mysql for database.

From the start we want to build our application for optimal and easy scalability in the future, to avoid a lot changes to our app/database in the future.

Now our questions are:

Should we use?:
* one database for all shops (or limited to X shops );
* one database for each new shop (each having products, orders... tables);

I think both approaches have PRO/CONS. What do you think ? Does anyone has experience with this kind of structure ?

PRO:
one database: easier to make changes to database layout
multiple databases:more scalable, easier to backup/restore

CON:
one database: harder to code because of extra keyfields in tables , slower or more difficult to backup/restore.
multiple databases: Takes longer to push changes to database layout

We are totally not clear on what hosting we should use. Would it be a solution to use a cloud service such as mosso/amazon/gogrid? Or is it better to just start with one dedicated server and expand 'manually' later?

Thanks in advance for your help!

SPHiveDB: A mixture of the Key/Value Store and the Relational Database.

The Key/Value Store becames more and more popular. When we use the Key/Value Store to store objects, we need to serialize/deserialize the objects as binary buffer. We have many ways to serialize/deserialize objects. A possible way is to use the Relational Database. Every value we store in the Key/Value Store is a SQLite instance. We can use the power of the Relational Database to manipulate the value. The SQL is very powerful for processing query request.

SPHiveDB = TokyoCabinet + SQLite
http://code.google.com/p/sphivedb/

SPHiveDB is a server for sqlite database. It use JSON-RPC over HTTP to expose a network interface to use SQLite database. It supports combining multiple SQLite databases into one file ( through tokyo cabinet ). It also supports the use of multiple files.

Google Voice Architecture

Hi High Scalability community!

Do you have any information on the architecture behind Google Voice, the new service by Google that offers one Google Number for all your calls and SMS? It is based on GrandCentral who has been acquired by Google 2 years ago.

Thanks!

starting small with growth in mind

Hello all,

I'm working on a web site that might totally flop or it might explode to be the next facebook/flickr/digg/etc. Since I really don't know how popular the site will be I don't want to spend a ton of money on the hardware/hosting right away but I want to be able to scale it easily if it does grow rapidly. With this in mind, what would be the best approach to launch the site?

Thanks,
Dan