Thursday
Feb192009
Heavy upload server scalability
Thursday, February 19, 2009 at 12:39AM
Hi,
We are running a backup solution that uploads every night the files our clients worked on during the day (Cabonite-like).
We have currently about 10GB of data per night, via http PUT requests (1 per file), and the files are written as-is on a NAS.
Our architecture is basically compound of a load balancer (hardware, sticky sessions), 5 servers (Tomcat under RHEL4/5, ) and a NAS (nfs 3).
Since our number of clients is rising, (as is our system load) how would you recommend we could scale our infrastructure? hardware and software? Should we go towards NAS sharding, more servers, NIO on tomcat...?
Thanks for your inputs!
We are running a backup solution that uploads every night the files our clients worked on during the day (Cabonite-like).
We have currently about 10GB of data per night, via http PUT requests (1 per file), and the files are written as-is on a NAS.
Our architecture is basically compound of a load balancer (hardware, sticky sessions), 5 servers (Tomcat under RHEL4/5, ) and a NAS (nfs 3).
Since our number of clients is rising, (as is our system load) how would you recommend we could scale our infrastructure? hardware and software? Should we go towards NAS sharding, more servers, NIO on tomcat...?
Thanks for your inputs!
Reader Comments (2)
Usually people with a lot of files to store use a distributed file system like MogileFS on the backend to handle the write load. Then you need as much bandwidth and as many servers as you want to handle simultaneous users. Messing with Tomcat probably won't buy you much, but that's when benchmarking is for.
Another option would be to load into S3 or some other cloud service. Then you can drop the computers and NAS and let them worry about scaling, backup, etc.
Bob,
How are the files transfered to you system? From what I understood, its via http-PUT.
Why not use ftp to transfer the files and then store then in your NAS via batch?