At the end of April, Brian and I took a trip to scenic Denver, CO to represent clypd at the first-ever GopherCon. The list of talks was absolutely incredible, including many of the members of the Go team and representatives from companies using Go in a variety of interesting ways. We were excited for an opportunity to meet members of the burgeoning Go community (that it provided an excuse to check out the numerous craft breweries that call Denver their home was a nice bonus).
At clypd, we have been using PostgreSQL and it has been a great experience. We are impressed with its simplicity, good documentation, active community, abundance of functions and lot more. This post jots lessons learned and insights acquired during a recent journey in improving performance of one of our queries. Most resources and tools in this post point to other sites, however we think it would be helpful to list various ‘performance improvement’ resources in one page. Read More
Despite the title, this isn’t another sales excellence screed, travel policy dictat, or a washroom deconfliction methodology. Rather, this is about our journey to select a new programming language and platform for our performance-sensitive services.
We began with a set of web services successfully built on the Ruby/Rails ecosystem. Given an extremely short timeframe to deliver version 1 and that the majority of the early product requirements were focused on the user interfaces, Ruby made a lot of sense.
As the scaling requirements of some of our services grew beyond the initial prototype, we found (to little surprise) that Ruby MRI just couldn’t keep up. We did some investigation and benchmarking with JRuby, hoping that it would offer a clean transition to a more performant services platform. Not so much… so we found ourselves at a crossroads.
Picking A New Path
Choosing a new programming language often bears more resemblance to a religious debate than to a purely logical process. Each of us has our blacklist (“Java is too verbose”, “Python’s dynamic typing makes you lazy”, “Scala is too hard to learn”), our hot list (“Ooh, let’s try Racket”, “Haskell looks like fun”, “Why not Erlang?”), and our unique perspective (embedded, web apps, enterprisy, whatever). Despite all of this, we intentionally took a more balanced and reasoned approach.
Given our small team size and aggressive schedule, we simply couldn’t afford to evaluate every possible platform. Instead, we immediately narrowed the field down to a reasonable set of candidates by ignoring some less likely scenarios:
- Clojure/Haskell/Erlang – too esoteric, not enough industry traction, too slow for certain workloads
- C/C++/Objective-C – not enough abstraction, very complex syntax
- Python – same architectural deficiencies as Ruby
We knew this effort would require extensive research, so we wanted to focus our energy by first defining a detailed evaluation criteria. Obviously, we demanded that our new platform provide an order of magnitude increase in performance (at least!) along with very strong support for concurrency and parallelism. In addition, we also wanted to understand:
- ramp-up – How quickly can people learn the platform? What kind of safeguards does the platform have to inhibit new people from implementing bad code?
- open source and community involvement – What does the open source community look like? How mature are the “gem” equivalents? Do these open source components use best practices around testing, like rubygems?
- TDD/BDD – What tools are in place to support TDD/BDD? Is there something like RSPEC for the platform? Do strong mocking tools exist?
- build time and effort – What effort is involved in the build? Does the build time slow down as the source code becomes larger and more complex?
- commercial adoption – What are other folks in our industry using?
- tooling – What tools exist for developing (IDEs), testing, documenting, building, profiling, deploying, and monitoring? What about static analysis?
- scalability – How easily can the platform scale horizontally and vertically?
- recruiting – How likely will we be able to attract and excite new developers?
We avoided the approach of writing small toy programs in each language to attempt to extrapolate conclusions. We felt that these quick “Hello World” spikes didn’t provide enough exposure to grok the nuances and pain points and we specifically avoided deriving conclusions from them.
After defining our evaluation criteria we were off to the races evaluating the platforms! Those that remained on our short list were Java, Scala, Go, and D.
Java was the most well understood platform since several clypd senior developers had deep experience with that platform. Java received strong marks for ramp-up, community, testing, tooling, and commercial adoption. Unfortunately, Java’s deficiencies quickly doomed its selection as the same senior developers didn’t even advocate for it. Their concerns included code verbosity, long build times, and a lack of a consistent strategy for concurrency.
None of the developers had any previous exposure to D besides the occasional thread from Andrei @ Facebook. Although D appears to address many of the shortcomings in Java and C++, ultimately we were challenged finding enough public information – anecdotes, critiques, and analysis – to give D anything but failing marks for community and adoption.
The final two remaining platforms, Scala and Go, both scored high in many areas. Both have
- strong concurrency models
- active communities and existing commercial installations
- proven scalability
- positive trending to help with recruiting
Scala’s functional model appealed in principle, but the language’s flexibility/complexity and associated ramp-up required by the team was a significant concern. Further, Scala’s lack of stability, ridiculously long build times (too long to support TDD), and clunky build tools are in stark contrast to Go’s simplicity, 1.x compatibility guarantee, and blazingly fast build times. Lastly, we’d heard some mixed feedback from folks in our network who were already using Scala. Some loved it, some didn’t. As a team, we met one final time to review our findings and recommend a path forward. After a contentious but educational process, we could finally say that we knew which way to Go. [groan]
Stay tuned for more updates as we learn about Go, its ecosystem and the Gopher community.
On a brief holiday from my regular activities, I recently spent a few days in Sidetrackistan attempting to answer a simple question: how far can we push bare-metal hardware using the simplest of web services and a standard deployment configuration? I quickly developed an experiment and collected data that exposed some interesting – though somewhat expected – limits of parallelism in some popular web platforms. I also identified a newer platform that seems better poised to exploit multicore to its theoretical limit. Read More
There has been a lot of back and forth lately about why Real-time Bidding (RTB) will or won’t work in Television. This is an interesting debate, but it’s not about why it may or may not work, it’s simply a question of when, and what are the first steps we must take on our path to RTB in TV?
So, what is the difference between RTB and programmatic buying and selling? Programmatic buying and selling is a technique used to trade media using business logic to acquire individual impressions for the delivery of a marketing campaign. RTB is a timeline laid on top of programmatic. To further understand this, let’s first dissect how RTB works in a digital world. Read More
Ad Tech is notoriously rife with acronyms on both the business and technology sides of the fence. clypd’s experience in the television ad tech world has proven that acronym issues are not only widespread, but the concepts represented by these multi-letter concepts in the nascent world of programmatic television advertising take on modified meanings, rendering our space one that is impossible for Grandma to understand.
clypd, along with other technology companies, ad agencies, DSPs, ATDs and SSPs are strategizing on how to leverage the programmatic buying methods that have been successful in the digital world to deliver on marketing goals in the world of the larger screen. While benefits of media buying in this medium are similar, many of the strategies must be augmented for use in television. Read More
One of the challenges of setting up a development process and workflow for a startup is to fit the process to the business needs. Process for process sake is a waste of time and drives everyone crazy. The other thing that drives everyone crazy is chaos. Product hates chaos. Engineers hate chaos. I’m a big fan of “just enough process to keep us from descending into chaos”. Read More