Kotlin, Unix Philosophy, Cypher and loads of tech stuff to read

There are so many JVM based languages out there and yet none of them seem to be able to take over from good old (has it really been that long) Java. From my point of view (I'm sure it's shared by many others) most of these languages like Scala, JRuby, Groovy, Gosu, Clojure, Ceylon etc feel too far away from the center. They either propose radical syntax and idioms, seem overly clever, too flexible and hence not clear/safe for the average developer, or just not performant enough.

Take Scala for example. If you pay no heed to the sensation Yammer's leaked email to TypeSafe caused and just read between the lines, there are some very real truths there about what a large population of developers, team leads, managers expect from a language before they can invest time, training, money, people and their careers into it.

Which is why, I find JetBrain's Kotlin and to some extent for non-server side Google's Dart to be promising. Neither of these languages are about revolutionary/radical new concepts or "paradigm" shifts (ahem) nor do they hide their Java roots. Dart is definitely not JVM based but neither is Kotlin planning to stick to just the JVM (LLVM plans). They are both intended to be evolutionary cleanups with some state of the art language features. Only time will tell if they succeed.

To add to our distractions we have Node.js with some appeal if you are just out of school and easily impressed or have a very specific need. Sigh...I'm already sounding like a weather beaten veteran who's seen too many "flash in the pans".

If you didn't know and are wondering what the actual JVM team has been doing in response to the demand from these new JVM-based languages in Java 7, here's a hint (JSR 292, MethodHandles, InvokeDynamic ..). Be warned, this is not for the faint of heart.

Enough of that. Now on to BloomFilters, a wonderful data structure heavily used by all NoSQL systems like Hive, Cassandra etc. Here's a very nice visualization.

I've written about Neo4J the graph database before. I really like the concept but I haven't found an excuse yet (partially because it's LGPL) to use it. They even have a nicely done graph access language, sort of like SQL for graphs called Cypher. Here's a nice demo of Neo4J, Gephi and spatial data. Makes sense for desktop use but I don't know if they have many server/enterprise grade users with many concurrent transactions.

If you are looking for some general tech reading, like while on a place, here's a nice set of essays (also a book) - The Architecture of Open Source Applications.

Speaking of Open Source, I had not read or heard about The Unix Philosophy. I like the simplicity and focus. Obviously it has had great success but it applies to other software too. Sort of like Deiter Rams ideas on design but for software.

If you've made it till here and are dying for some hard core tech, then here's some on CAS concurrency, Azul's proposed Linux kernel patch and thread affinity.

These days, any blog is incomplete without mention of BigData. So, here's some BigData reading material - slides from the Cassandra 2011 summit.

Ashwin.

Some interesting talks from Camelone, Eurocon, HadoopWorld, distributed joins and more

Apache Camel, if you haven't used or even heard about this lovely little integration framework then I highly recommend watching this video by one of its founders. It's best described as a mini-ESB, an implementation of the Enterprise Application Integration Patterns (EAI) with API/programmatic access, a zillion connectors and a delight to use.

Also worth watching is the EAI intro video by Gregor Hohpe at Camelone 2011.

Some interesting HadoopWorld 2011 slides:

• Hadoop World 2011 Presentation Slides: Advanced HBase Schema Design - Lars George, Cloudera
• Hadoop World 2011 Presentation Slides: Replacing RDB/DW with Hadoop and Hive for Telco Big Data - Jason Han, NexR
• Hadoop World 2011: Hadoop and Graph Data Management: Challenges and Opportunities

 Apache Eurocon 2011 slides, worth reading - Lucene at Twitter.
Joins - the Achilles heel of NoSql, distributed systems. Possible, but difficult and not always feasible. Yet, several solutions are available being offered in addition to the big data warehouse vendors and do-it-yourselfers.

• 70x Faster Joins with AQL in MySQL Cluster 7.2 DMR
• 25x: MySQL Cluster and pushdown joins (in pursuit of the holy grail)
• Hadapt is moving forward

More lessons on IO - Disk I/O abstraction layer X-rays.

You thought System.nanoTime() was fast than using System.currentTimeMills() for profiling? Well.. it's all relative.

The old JCache API seems to have been revived - JSR107: The new Caching Standard. It's also going to be a part of the next JEE spec. More signs of commoditization - a good sign of progress.

Until next time!

Garbage collection, memory, IO and other scary stories

Happy Halloween! Want to read some scary, low-level, systems related stuff? Here's a short list of very useful articles I read recently:

• What Every Data Programmer Needs to Know about Disks
• Linux IO Scheduler
• Do You Really Get Memory?
• Quake 2 Source Code Review
• The Architecture of Open Source Applications
• Understanding Memory
• Playing with Virtual Memory
• Other links I've mentioned previously

SQL anti-patterns.

GC horror stories in the .Net world.

Still not scared? How about some math and statistics:

• Ranking data at LinkedIn
• Some more math stuff and other useful miscellaneous Java articles 
• Basic R/statistics every programmer should know
  

Boo!
Ashwin.

Offloading data from the JVM heap (a little experiment)

Last time, I wrote about the possibility of using Linux shared memory to offload cacheable/reference data from the JVM. To that end I wrote a small Java program to see if it was practical. The results were better (some even stranger) than I had expected.

Here's what the test program does:

• Create a bunch of java.nio.ByteBuffers that add up to 96MB of storage
• Write ints starting from the first buffer, all the way to the last one - that's writing a total of 96MB of some contrived data
• For each test, the buffer creation, writing and deletion is done 24 times (JIT warm up)
• For each such test iteration, measure the memory (roughly) used in the JVM heap, the time taken to create those buffers and the time taken to write 96MB of data
• Obviously, there are things here that sound fishy to you - like why use ByteBuffers instead of just writing to an OutputStream or why write to the buffers in sequence. Well, my intentions were just to get a ballpark figure as to the performance and the viability of moving data off the JVM heap

About the test:

• There are really 5 different ways to create the buffers. Then there are 2 variations of these tests in which the buffer sizes vary (blocks), but the total bytes written are the same
• The buffers (blocks) for each variation are created as:
• Ordinary HeapByteBuffers inside the JVM heap itself - as a baseline for performance
• DirectByteBuffers
• A file created on Ext4fs using RandomAccessFile and parts of the file are memory mapped using the FileChannel. The file is opened in "rw" mode. Other options are "rwd" and "rws"
• The same as above but the file resides in /dev/shm the in-memory based, shared memory virtual file system (Tmpfs)
• The buffers are created using Apache's Tomcat Native Libraries which in turn use Apache Portable Runtime libraries. The Shared memory (Shm) feature was used to create the buffers. This is similar to DirectByteBuffers but the buffers reside in a common area, in OS memory and not owned by any but shared between processes (Similar to /dev/shm but without the filesystem wrapper overhead)
• The machine used to test was my moderately powered Windows 7 home laptop with 8GB RAM, 2.3GHz i5 running a Cloudera Ubuntu Linux VMWare Player. There were a few other processes running, but nothing that was using CPU extensively. 500MB+ memory was free and available
• The VM had 1GB RAM and the JVM heap was 256MB

Results:

• The test program was run once for each configuration, but each test itself ran 24 times to allow the JIT to warmup and even the file system caches to stay warm where needed
• The test prints out the timings with headers which were then compiled into a single text file and then analyzed in RStudio

summary_mem_used_time_taken_millis

  block_size                      test_type perctile95_buffer_create_and_work_time_millis perctile95_mem_bytes
1       4096                         direct                                       1555.65              3047456
2       4096 file_/dev/shm/mmap_test.dat_rw                                        661.70              3047632
3       4096          file_mmap_test.dat_rw                                       2055.75              3047632
4       4096                           heap                                       1071.15            102334496
5    4194304                         direct                                        653.85                 3008
6    4194304 file_/dev/shm/mmap_test.dat_rw                                        561.40                 3184
7    4194304          file_mmap_test.dat_rw                                       3878.25                 3184
8    4194304                           heap                                       1064.80            100664960
9    4194304                            shm                                        678.40                 2496

Interpretation of the results:

• The test where block size was 4KB had quite a lot of memory overhead for the non-Java-heap ByteBuffers. Memory mapping was also slow for these small sizes as the Javadocs itself says for FileChannel.map()
• The JVM heap test was slower than I had expected (for larger ByteBuffers). I was expecting that to be to fastest. Perhaps it was the small memory (1GB) virtualized OS it was running in. For smaller block sizes approaching 1K or less, the JVM heap performance is unbeatable. But in these tests, the focus was on larger block sizes
• The Apache shared memory test would not even start for the 4KB tests as it would complain about "not enough space"
• Almost everything fared well in the larger 4MB test. The per-block overhead was less for the off-heap tests and also the performance was nearly identical for /dev/shm, Apache Shm and DirectByteBuffer

Sources:

• The sources for this test are available here.
• To run all the tests except Apache Shm you only need to compile JavaBufferTest and run it with the correct parameters
• To run all tests, you can use the sub-class AprBufferTest which can test Apache Shm and also the remaining tests. To compile this you'll need tomcat-coyote.jar from apache-tomcat-7.0+. To run this you'll need the Jar file and the Tomcat Native bindings - tcnative.dll or libtcnative for Linux

There are advantages to using ByteBuffer outside the Java heap:

• The mapped file or the shared memory segments can outlive the JVM's life span. Another process can come in and attach to it and read the data
• Reduces the GC pressure
• Zero-copy transfers are possible to another file, network or device using FileChannel.transferTo()
• Several projects and products have used this approach to host large volumes of data in the JVM
• HBase-3455
• Details of HBase Slab allocator
• Cassandra-2252

Disadvantages:

• The data has to be stored, read and written using primitives to the ByteBuffer - putInt(), putFloat(), putChar() etc
• Java objects cannot be read/written like in a simple Java program. Everything has to be serialized and deserialized back and forth from the buffers. This adds to latency and also makes it less user friendly

Misc notes and references:

• These tests can also be run on Windows, except for the Linux Tmpfs tests. However a RAM Drive can be used to achieve something similar

Ashwin.

RAM disk is already in Linux and nobody told you (a.k.a Shared memory for ordinary folk)

The Linux 2.6 Kernel these days comes with an in memory file system. This is really shared memory across processes ("Everything old is new again!").

The beauty of the Linux implementation (like everything else) is that this shared memory looks like a regular file system - /dev/shm (but is really Tmpfs). So, your application - yes even a Java application can use this shared memory system across processes as if it were just writing to a regular file. You can create directories, setup a memory quota for this file system, tail files, change to the directories etc like any other directory, open-write-close files that can be read by other processes. Convenient!

RAM disks are not a new concept. But a RAM disk driver that is built into the kernel adds a totally different level of credibility and trust to the concept.

All the contents of this directory are in memory. Nothing gets written to the disk, no flush, no IO wait times. Naturally, this being in-memory you will loose all the contents when your OS reboots. What possible use can this be to us, I hear you ask? Well well.. where do I begin:

• Memory is cheap. 96GB RAM is quite common these days on server class machines
• You can run a big RDBMS on this file system. Typically these databases are anyway replicated/clustered over the network
• So you can run the entire DB in-memory and still have HA because the replica is running on another machine anyway (low "disk" latency + high TPS + HA)
• Why write to a local disk which can crash anytime
• Why spend so much on expensive SSDs
• 10GigE is already here
• Run medium sized JVMs and push all the heavy data to this shared memory filesystem
• You free up the heap to do simple in-JVM caching and reduce the pressure on GC by moving all the data to /dev/shm
• If your JVM crashes, another JVM can be notified to pick up that data since it is just stored as a bunch of files and directories
• People used to do IPC all the time using old fashioned shared memory constructs but it fell out of favor because networks and network drivers became quite fast
• Also moving away from IPC to TCP-over-localhost gave you freedom to spread your processes across machines (TCP-over-GigE)
• Perhaps it is now worthwhile to re-examine that approach and shave precious milliseconds by not copying data back and forth between processes over sockets

To that end I wrote a simple Java class to measure the performance of writing to file on /dev/shm compared to a regular disk based file on /tmp. The results are interesting and I'm hoping will make you readers re-think about your software systems ("Everything old is new ...").

The program is a simple Java file writer that forces a flush every few tens of bytes (which means nothing when writing to /dev/shm). The destination file and its path can be specified as a command line argument. So, it's easy to compare performance. I ran these tests on a Cloudera-Ubuntu VMWare image running on my 64-bit Windows 7 laptop with 4GB RAM and 2.3 GHz duo core Intel i5. In-memory is not surprisingly 7x faster for a 112KB file. Also, laptop was running on batteries means processor speed steps down to save power.


Why people do not talk about this relatively new Linux feature out loud is perplexing.

Other interesting commands you can run against this file system:

• mkdir
• cat
• tail, grep
• ipcs -a
• df -k

Follow up articles:

• Bigger pages
• In memory file system commands
• Optimizing Oracle

Detailed log:


Until next time,
Ashwin.