Product: GlusterFS
Sunday, March 16, 2008 at 2:24AM Adapted from their website:
GlusterFS is a clustered file-system capable of scaling to several peta-bytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. Storage bricks can be made of any commodity hardware such as x86-64 server with SATA-II RAID and Infiniband HBA).
Cluster file systems are still not mature for enterprise market. They are too complex to deploy and maintain though they are extremely scalable and cheap. Can be entirely built out of commodity OS and hardware. GlusterFS hopes to solves this problem.
GlusterFS achieved 35 GBps read throughput. The GlusterFS Aggregated I/O Benchmark was performed on 64 bricks clustered storage system over 10 Gbps Infiniband interconnect. A cluster of 220 clients pounded the storage system with multiple dd (disk-dump) instances, each reading / writing a 1 GB file with 1MB block size. GlusterFS was configured with unify translator and round-robin scheduler.
The advantages of GlusterFS are:
* Designed for O(1) scalability and feature rich.
* Aggregates on top of existing filesystems. User can recover the files and folders even without GlusterFS.
* GlusterFS has no single point of failure. Completely distributed. No centralized meta-data server like Lustre.
* Extensible scheduling interface with modules loaded based on user's storage I/O access pattern.
* Modular and extensible through powerful translator mechanism.
* Supports Infiniband RDMA and TCP/IP.
* Entirely implemented in user-space. Easy to port, debug and maintain.
* Scales on demand.
Related Articles
Reader Comments (5)
I have tested Gluster, and I was quite satisfied with it. Handles node failures well (although the healing could be automatically triggered), and the configurations allows for some really flexible brick setups. It gives a feeling of a very robust DFS, but I haven't tested it in a high-traffic production environment yet.
I've tested this a couple times through out its life cycle, but kept getting disappointed. With every release another issue arises that seemed like a decent developer would catch while coding. Their wire protocol overhead is way to much (upwards 500bytes for a header?). The software constantly falls on its face under pressure.
I set it up in 2 shadow clusters (shadow meaning the cluster works off the shadowing of ports on our routers to get "real" world traffic without effecting the real cluster). The usage was alot of images and video streaming at about 500mbit. Had 4 glusterfs servers talking to 18 glusterfs clients. One of the glusterfs servers would always crash within an hour (which one crashed was random). The mount point would constantly "disappear" on the clients.
It sounds like a great idea, but I'll stick with lustre and/or the many ways to turn linux into a san.
Second that. I am attempting to use this as a production filesystem for large datasets and it is completely useless. I've had to do significant scripting to catch crashes and respawn, mounts disappear hourly and instantly under load... The idea is sort-of good, theoretically, but the implementation of it so far is complete balls. Some adults need to step in and take over development of this software if it's going to go anywhere.
I've been using glusterFS for hosting XEN images, other than having to use --disable-direct-io it works pretty well, it's very flexible, and easy to use.
Been using it for sharing video content between a bunch of web servers, and so far so good, no complaints.
Looking forward to version 2.1 though ;)
ciao/Riaan
We avoided using Gluster and went with Luster due to the fact that we would have to take the system down to add brinks. Now that this problem has been solved and a number of user space tools have been added with version 3.1 we have changed our file system backends to Gluster.
Luster was complex and required a lot of baby sitting, Gluster is simple and even though it's not quite as fast as Lustre it is well worth loosing the speed for something that can be fixed quickly when something goes wrong.