IBM Debuts Fast Storage System

With an eye toward helping tomorrow’s data intensive organizations, IBM researchers have developed a super-fast storage system capable of scanning in 10 billion files in 43 minutes.

This system easily bested their previous system, demonstrated at Supercomputing 2007, which scanned 1 billion files in three hours.

Key to the increased performance was the use of speedy flash memory to store the metadata that the storage system uses to locate requested information. Traditionally, metadata repositories reside on disk, access to which slows operations.

“If we have that data on very fast storage, then we can do those operations much more quickly,” said Bruce Hillsberg, director of storage systems at IBM Research Almaden, where the cluster was built. “Being able to use solid-state storage for metadata operations really allows us to do some of these management tasks more quickly than we could ever do if it was all on disk.”

IBM foresees that its customers will be grappling with a lot more information in the years to come.

“As customers have to store and process large amounts of data for large periods of time, they will need efficient ways of managing that data,” Hillsberg said.

For the new demonstration, IBM built a cluster of 10 eight-core servers equipped with a total of 6.8 terabytes of solid-state memory. IBM used four 3205 solid-state Storage Systems from Violin Memory. The resulting system was able to read files at a rate of almost 5 GB/s (gigabytes per second).

Read More….