Tachyon - Fault Tolerant Distributed File System with 300 Times Higher Throughput than HDFS

Tachyon  (github) is interesting new filesystem brought to by the folks at the UC Berkeley AMP Lab:

Tachyon is a fault tolerant distributed file system enabling reliable file sharing at memory-speed across cluster frameworks, such as Spark and MapReduce.It offers up to 300 times higher throughput than HDFS, by leveraging lineage information and using memory aggressively. Tachyon caches working set files in memory, and enables different jobs/queries and frameworks to access cached files at memory speed. Thus, Tachyon avoids going to disk to load datasets that is frequently read.

It has a Java-like File API, native support for raw tables, a pluggable file system, and it works with Hadoop with no modifications.

 

It might work well for streaming media too as you wouldn't have to wait for the complete file to hit the disk before rendering.

Discuss on Hacker News