The surprise for me is that the core component is Redis.
My first guess would have been custom C code. Yeah, you have to do everything yourself. Yeah, it would be hard to write. But you'd control every little bit of it.
Obviously, I must not fully understand the problem and what Redis buys.
Sam Puralla (if you are reading) -- do you know why didn't Twitter go with a full custom system at its heart?
I'm probably a better person to answer this than Sam - I'm a former lead on this project - so I'll take a swing:
We chose Redis because it gave us the specific, incremental improvements over our existing memcached-based system that we required, without requiring us to write (yet another) component. There was enough to do, and this choice has turned out to be good enough, I think.
As the project progressed though, we treated Redis much like we did own it. We altered the wire protocol, changed the eviction strategy, and reduced protocol parsing overhead, for example. Much of that work has long since made it upstream.
Sounds like good engineering choices. I think of Twitter as having unlimited resources but, of course, that can't be true.
Two follow-on questions:
1. Did your changes make it back into open source or were they only relevant to Twitter? When you say upstream to you mean on Redis or earlier in the Twitter pipeline?
2. How much is Redis on the critical path? Is it 90% of the processing work in the large fanout cases?
My first guess would have been custom C code. Yeah, you have to do everything yourself. Yeah, it would be hard to write. But you'd control every little bit of it.
Obviously, I must not fully understand the problem and what Redis buys.
Sam Puralla (if you are reading) -- do you know why didn't Twitter go with a full custom system at its heart?
Josh