An Introduction to Television Data Management Platforms (DMP)

By clypd, clypd Blog, Technology No Comments

In the ad tech world, there are a lot of acronyms. In this series, we’ll discuss the DMP – or Data Management Platform. In the digital landscape, DMPs are fairly established and standardized, but in the television world, they operate differently, simply because television and digital media are different.

A DMP is a centralized computing system for collecting, integrating, and managing large sets of structured and unstructured data from disparate sources. First, data goes in, then it gets manipulated, normalized, and prepared in an easy to use manner that makes it actionable. In television, the design, functionality, and utility of a DMP are different than in other environments.

Read More

clypd does Denver: GopherCon 2014

By clypd Blog, Technology No Comments

At the end of April, Brian and I took a trip to scenic Denver, CO to represent clypd at the first-ever GopherCon. The list of talks was absolutely incredible, including many of the members of the Go team and representatives from companies using Go in a variety of interesting ways. We were excited for an opportunity to meet members of the burgeoning Go community (that it provided an excuse to check out the numerous craft breweries that call Denver their home was a nice bonus).

Read More

clypd’s Lessons Learned in PostgreSQL Performance

By clypd, Engineering, Technology No Comments

At clypd, we have been using PostgreSQL and it has been a great experience. We are impressed with its simplicity, good documentation, active community, abundance of functions and lot more. This post jots lessons learned and insights acquired during a recent journey in improving performance of one of our queries. Most resources and tools in this post point to other sites, however we think it would be helpful to list various ‘performance improvement’ resources in one page. Read More

Go + clypd

Getting to “go”

By clypd Blog, Technology No Comments

Despite the title, this isn’t another sales excellence screed, travel policy dictat, or a washroom deconfliction methodology. Rather, this is about our journey to select a new programming language and platform for our performance-sensitive services.

Starting Point

We began with a set of web services successfully built on the Ruby/Rails ecosystem. Given an extremely short timeframe to deliver version 1 and that the majority of the early product requirements were focused on the user interfaces, Ruby made a lot of sense.

As the scaling requirements of some of our services grew beyond the initial prototype, we found (to little surprise) that Ruby MRI just couldn’t keep up. We did some investigation and benchmarking with JRuby, hoping that it would offer a clean transition to a more performant services platform. Not so much… so we found ourselves at a crossroads.

Picking A New Path

Choosing a new programming language often bears more resemblance to a religious debate than to a purely logical process. Each of us has our blacklist (“Java is too verbose”, “Python’s dynamic typing makes you lazy”, “Scala is too hard to learn”), our hot list (“Ooh, let’s try Racket”, “Haskell looks like fun”, “Why not Erlang?”), and our unique perspective (embedded, web apps, enterprisy, whatever). Despite all of this, we intentionally took a more balanced and reasoned approach.

Given our small team size and aggressive schedule, we simply couldn’t afford to evaluate every possible platform. Instead, we immediately narrowed the field down to a reasonable set of candidates by ignoring some less likely scenarios:

  • Clojure/Haskell/Erlang – too esoteric, not enough industry traction, too slow for certain workloads
  • C/C++/Objective-C – not enough abstraction, very complex syntax
  • Python – same architectural deficiencies as Ruby

We knew this effort would require extensive research, so we wanted to focus our energy by first defining a detailed evaluation criteria. Obviously, we demanded that our new platform provide an order of magnitude increase in performance (at least!) along with very strong support for concurrency and parallelism. In addition, we also wanted to understand:

  • ramp-up – How quickly can people learn the platform? What kind of safeguards does the platform have to inhibit new people from implementing bad code?
  • open source and community involvement – What does the open source community look like? How mature are the “gem” equivalents? Do these open source components use best practices around testing, like rubygems?
  • TDD/BDD – What tools are in place to support TDD/BDD? Is there something like RSPEC for the platform? Do strong mocking tools exist?
  • build time and effort – What effort is involved in the build? Does the build time slow down as the source code becomes larger and more complex?
  • commercial adoption – What are other folks in our industry using?
  • tooling – What tools exist for developing (IDEs), testing, documenting, building, profiling, deploying, and monitoring? What about static analysis?
  • scalability – How easily can the platform scale horizontally and vertically?
  • recruiting – How likely will we be able to attract and excite new developers?

We avoided the approach of writing small toy programs in each language to attempt to extrapolate conclusions. We felt that these quick “Hello World” spikes didn’t provide enough exposure to grok the nuances and pain points and we specifically avoided deriving conclusions from them.

After defining our evaluation criteria we were off to the races evaluating the platforms! Those that remained on our short list were Java, Scala, Go, and D.

Java was the most well understood platform since several clypd senior developers had deep experience with that platform. Java received strong marks for ramp-up, community, testing, tooling, and commercial adoption. Unfortunately, Java’s deficiencies quickly doomed its selection as the same senior developers didn’t even advocate for it. Their concerns included code verbosity, long build times, and a lack of a consistent strategy for concurrency.

None of the developers had any previous exposure to D besides the occasional thread from Andrei @ Facebook. Although D appears to address many of the shortcomings in Java and C++, ultimately we were challenged finding enough public information – anecdotes, critiques, and analysis – to give D anything but failing marks for community and adoption.

The final two remaining platforms, Scala and Go, both scored high in many areas. Both have

  • strong concurrency models
  • active communities and existing commercial installations
  • proven scalability
  • positive trending to help with recruiting

Scala’s functional model appealed in principle, but the language’s flexibility/complexity and associated ramp-up required by the team was a significant concern. Further, Scala’s lack of stability, ridiculously long build times (too long to support TDD), and clunky build tools are in stark contrast to Go’s simplicity, 1.x compatibility guarantee, and blazingly fast build times. Lastly, we’d heard some mixed feedback from folks in our network who were already using Scala. Some loved it, some didn’t. As a team, we met one final time to review our findings and recommend a path forward. After a contentious but educational process, we could finally say that we knew which way to Go. [groan]

Stay tuned for more updates as we learn about Go, its ecosystem and the Gopher community.

Brian Fallik is Principal Developer and Joel Melby is the CTO at clypd.

Go Speed Racer Go

By clypd, clypd Blog, Technology No Comments

On a brief holiday from my regular activities, I recently spent a few days in Sidetrackistan attempting to answer a simple question: how far can we push bare-metal hardware using the simplest of web services and a standard deployment configuration? I quickly developed an experiment and collected data that exposed some interesting – though somewhat expected – limits of parallelism in some popular web platforms. I also identified a newer platform that seems better poised to exploit multicore to its theoretical limit. Read More

Programmatic vs. RTB

By clypd, Product, Technology No Comments

There has been a lot of back and forth lately about why Real-time Bidding (RTB) will or won’t work in Television. This is an interesting debate, but it’s not about why it may or may not work, it’s simply a question of when, and what are the first steps we must take on our path to RTB in TV?

So, what is the difference between RTB and programmatic buying and selling? Programmatic buying and selling is a technique used to trade media using business logic to acquire individual impressions for the delivery of a marketing campaign. RTB is a timeline laid on top of programmatic. To further understand this, let’s first dissect how RTB works in a digital world. Read More