Thursday, September 01, 2011

RAM disk is already in Linux and nobody told you (a.k.a Shared memory for ordinary folk)

The Linux 2.6 Kernel these days comes with an in memory file system. This is really shared memory across processes ("Everything old is new again!").

The beauty of the Linux implementation (like everything else) is that this shared memory looks like a regular file system - /dev/shm (but is really Tmpfs). So, your application - yes even a Java application can use this shared memory system across processes as if it were just writing to a regular file. You can create directories, setup a memory quota for this file system, tail files, change to the directories etc like any other directory, open-write-close files that can be read by other processes. Convenient!

RAM disks are not a new concept. But a RAM disk driver that is built into the kernel adds a totally different level of credibility and trust to the concept.

All the contents of this directory are in memory. Nothing gets written to the disk, no flush, no IO wait times. Naturally, this being in-memory you will loose all the contents when your OS reboots. What possible use can this be to us, I hear you ask? Well well.. where do I begin:

  • Memory is cheap. 96GB RAM is quite common these days on server class machines
  • You can run a big RDBMS on this file system. Typically these databases are anyway replicated/clustered over the network
    • So you can run the entire DB in-memory and still have HA because the replica is running on another machine anyway (low "disk" latency + high TPS + HA)
    • Why write to a local disk which can crash anytime
    • Why spend so much on expensive SSDs
    • 10GigE is already here
  • Run medium sized JVMs and push all the heavy data to this shared memory filesystem
    • You free up the heap to do simple in-JVM caching and reduce the pressure on GC by moving all the data to /dev/shm
    • If your JVM crashes, another JVM can be notified to pick up that data since it is just stored as a bunch of files and directories
  • People used to do IPC all the time using old fashioned shared memory constructs but it fell out of favor because networks and network drivers became quite fast
    • Also moving away from IPC to TCP-over-localhost gave you freedom to spread your processes across machines (TCP-over-GigE)
    • Perhaps it is now worthwhile to re-examine that approach and shave precious milliseconds by not copying data back and forth between processes over sockets
To that end I wrote a simple Java class to measure the performance of writing to file on /dev/shm compared to a regular disk based file on /tmp. The results are interesting and I'm hoping will make you readers re-think about your software systems ("Everything old is new ...").

The program is a simple Java file writer that forces a flush every few tens of bytes (which means nothing when writing to /dev/shm). The destination file and its path can be specified as a command line argument. So, it's easy to compare performance. I ran these tests on a Cloudera-Ubuntu VMWare image running on my 64-bit Windows 7 laptop with 4GB RAM and 2.3 GHz duo core Intel i5. In-memory is not surprisingly 7x faster for a 112KB file. Also, laptop was running on batteries means processor speed steps down to save power.


Why people do not talk about this relatively new Linux feature out loud is perplexing.

Other interesting commands you can run against this file system:
  • mkdir
  • cat
  • tail, grep
  • ipcs -a
  • df -k
Follow up articles:
Detailed log:


Until next time,
Ashwin.

5 comments:

Ashwin Jayaprakash said...

Kernel docs on shmfs

Ashwin Jayaprakash said...

Tmpfs

Peter Lawrey said...

I am not sure why it is not used more often either. Being an old SunOS administrator I use it for /tmp if for no other reason that it is automatically cleared on a re-boot.

tmpfs was added to SunOS in 1987.

http://en.wikipedia.org/wiki/Tmpfs

Hering said...

I second Peter's comments. Are files created in /dev/shm accessible by processes using System V IPC? Are shared memory segments created by IPC accessible in /dev/shm?

Ashwin Jayaprakash said...

http://slacy.com/blog/2008/08/tmpfs-vs-ext3-performance-on-large-file-sets/