clypd’s workflow involves offline processing of files from our clients and partners. They drop compressed ZIP files to our SFTP servers which are then picked and processed by our workers. The processed data is crucial in powering clypd’s advanced audience targeting platform. Because these files are critical to our business, we have monitors that ensure correct parsing and transformation of the files.
At clypd, almost every feature we build is data-driven. It is therefore important for the data access layer of our Go platform to be powerful, yet easy to build upon, maintain, and debug. Today we dive into the design of the clypd data access layer, the frameworks we chose, and how we augmented them.
A huge benefit provided by Go is simplicity throughout the clypd stack. Dependencies, particularly third-party libraries, are often opaque in other programming languages due to varying code style, source code that is difficult to find, or abstraction layers that are challenging to see through. The source code for any library is just a click away in Go. Design idioms found in dependency libraries are usually similar to the ones we use, which ensures that the entire system is easy for our team to comprehend.
Almost a year ago, we blogged about our reasoning and methodology for choosing Go as our next generation platform here at clypd. A year is a long time, both in software technology and in the lifetime of a startup. Since paper is the traditional gift for a one year anniversary, it seems appropriate to write down our learnings so far.
When we posted “Getting to Go,” we included a list of the selection factors we used to evaluate potential platforms. The list includes
Thorough testing continues to be a key tenet of software development at clypd. Quality is paramount, but shipping code quickly requires us to be efficient in how we search for bugs. We’ve found innovative ways to unit test individual Go packages of our programmatic television ad platform as it has grown in its capabilities. The construction of a similarly efficient mechanism to test large portions of the whole system has taken our test automation to the next level.
At clypd, we place a lot of value on testing as a mechanism for ensuring code correctness and our approach to testing constantly evolves. We’ve spent the past few months learning and creating a new approach to mocking and testing functions that need to access data, whether from a database, a file, or over the network.
At the end of April, Brian and I took a trip to scenic Denver, CO to represent clypd at the first-ever GopherCon. The list of talks was absolutely incredible, including many of the members of the Go team and representatives from companies using Go in a variety of interesting ways. We were excited for an opportunity to meet members of the burgeoning Go community (that it provided an excuse to check out the numerous craft breweries that call Denver their home was a nice bonus).