The best thing coverage can tell you is that code is unused, and should therefore be deleted.

After a move to new hardware and a new kernel, a pair of hosts in two of our production environments started seeing out of order traffic. Had we set something up wrong, or was the hardware/kernel change the cause?

Our new production exchange recently produced an impossible looking NullPointerException. At the same time, we saw another application in the same deployment throw an OutOfMemoryError. Both problems turned out to have the same root cause. This post tells the story of how we found that out.

A section on our big office whiteboard has this mysterious series of markings on it: How did it get there? What does it mean?

Some cute round trip test tricks In the last post, we looked at layering our deserialization code to keep things simple. This time, we’ll enjoy the delightful testing benefits this effort yields.

…indeed, your life might get simpler if you don’t. This post will talk through two examples where clever serialization would have been an option, but stupid alternatives actually turned out to be preferable.

We have recently added an extra (optional) call back to the disruptor library. This post will walk through one of our motivations for doing this: monitoring. Before we start – what are we monitoring, and why? At LMAX Exchange, the vast majority of our applications look a bit like this: Here, events are received from the network on a single thread, and placed in an input (or application) ring buffer. Another thread (we’ll call it the application thread) processes these messages, and may emplace events in several output (or ‘publisher’) ring buffers.

Recently at LMAX Exchange we’ve had a couple of services suffer from memory leaks. In both cases, we noticed the problem much later than we’d like. One stricken application started to apply backpressure on (really rather important) upstream components. Another caused an account related action to be intermittently unavailable. What we’d like to be able to do is detect when any of our java services is in acute heap distress as soon as it enters that state.

In part one, we discovered that our multicast receipt thread was being stalled by page faults. In part two, we’ll dig down into the causes of those page faults, and with some help from our friends at Informatica, get to the bottom of things. Systems Cavalry Arrives Let’s focus on just the first stack trace to begin with. We can have a bit of a guess at what’s going on just by looking at the symbol names.

We recently fixed a long standing performance issue at LMAX Exchange. The path we followed to fixing it was sufficiently windy to merit a couple of posts. In this first post we’ll define our issue and then attempt to figure out its cause. Problem Identified At LMAX Exchange we have an application named the market data service. It’s multicast receipt thread is not keeping up – it occasionally stalls for around 150ms. This causes data loss; the service has to request retransmission (it NAKs) from upstream.

This time we’re going to explore how functional programs express variance. We’ll consider the same case as we did for TDA – getting a stored value from a Map, and storing a new value in a Map. Retrieval lookup :: Ord k => k -> Map k a -> Maybe a In Haskell, the variance is expressed in the return type. data Maybe a = Nothing | Just a We might express this as follows:

This time we’re going to explore how tell don’t ask (TDA) programs express variance, and hopefully come across an interesting corollary of our TDA definition. We’ll consider two ubiquitous operations: getting and storing a value into a map. In both cases we’ll discuss the standard Java implementation before creating a speculative TDA version. Example one: value retrieval We want to lookup the value associated with a particular key. This operation has entropy two. Either the map contains the value or it does not.

I have recently started a major stream of work centered on a particular application in the LMAX stack. This application has had plenty of features added to it over the last few years, but nothing has really required an overhaul. Our work, however, is somewhat more involved; even finishing the simplest of our requirements has been taking a week or so – that’s a long time, for us. Hitting the buffers Our method, to begin with, looked something like the following:

Events have conspired against me and I find myself writing an iOS app. Off the back of some relatively complete android work, I was interested to see how the platform compared. First steps After the gigantic pain of upgrading a dual boot Leopard/Ubuntu macbook to Mountain Lion, installing XCode was relatively trivial (albeit I had to enter my password roughly five or six hundred more times than I’d hoped). I dived straight in hoping I would soon emerge with some useful knowledge; you will have to be the judge.

Before we begin – I really feel like “The Class Path” ought to be the name of a TNG episode. Perhaps something to suggest to the TNG_S8 twitter account. Anyway… There comes a point in every java program’s lifetime where it must answer once of life’s fundamental questions: how am I going to talk to my config file (or, indeed, any file like resource that’s not another class)? As good test driven developers, we’ll usually have a short digression about the definition of unit testing before agreeing that there should be at least one test that reads from a real file in the same way that the production code will, and then start to write that test.

A post about closing resources in Java. First though, an aside. “Is it a good idea for destructors to throw?” – Common C++ interview question A quick bit of code can show us. #include <iostream> #include <stdexcept> namespace { class Foo { public: void someUsage() { std::cout << "Using foo" << std::endl; } ~Foo() { std::cout << "Destroying foo" << std::endl; throw std::runtime_error("Exception 2"); } }; } int main() { std::cout << "Program started" << std::endl; try { Foo f; f.

The right time to build We’ve talked at length about the costs of builders, and yet… they persist. I can think of at least three examples from popular java libraries that work well: Hamcrest’s Description Java’s own StringBuilder Guava’s ImmutableList.Builder Although, having looked at StringBuilder‘s doc – augh! It’s so big! Five or six builders could come out of that, starting with AppendOnlyStringBuilder. What do these master builders (That’s enough linguistic hyphens between the Builder pattern and actual builders – Ed) have in common?

Well, really, this should be obvious. My mental image of factories is the Toyota plant; an efficient production line of high tech robots automatically executing specialized constructive tasks as part of a well honed process (perhaps with a nearby overseer sternly observing, making regular marks on some sort of handheld checklist device). Whereas my builder image is a bloke leering at young ladies from the safety of some dodgy looking scaffolding. Ahem Uh. Yeah. I actually wanted to talk about the Factory and Builder patterns.

Without even a mention of the ‘M’ word! One of the things that attracted me to SPA 2013 was the Neural Net workshop on the Sunday. It turned out to be even better than I hoped; although we didn’t get quite as far as we could have done. The first thing we wanted to do was implement a Neuron. In this particular case, a neuron took two inputs (let’s say they are doubles, for the time being) and then performs a computation on those inputs to determine a single output.

So, this week on Distilled Derivatives, some more laziness. And also, exceptions. Everyone loves them, really useful things for letting you move error handling into a single location that they are. Now and again though, their misuse can really get in the way of getting something done. Retrieve the content of the first functioning url Given n urls in some ordered collection, get the content of the first “working” url, where working is defined to mean “has http code 200 and non empty response content”.