clypd’s workflow involves offline processing of files from our clients and partners. They drop compressed ZIP files to our SFTP servers which are then picked and processed by our workers. The processed data is crucial in powering clypd’s advanced audience targeting platform. Because these files are critical to our business, we have monitors that ensure correct parsing and transformation of the files.
At clypd, almost every feature we build is data-driven. It is therefore important for the data access layer of our Go platform to be powerful, yet easy to build upon, maintain, and debug. Today we dive into the design of the clypd data access layer, the frameworks we chose, and how we augmented them.
A huge benefit provided by Go is simplicity throughout the clypd stack. Dependencies, particularly third-party libraries, are often opaque in other programming languages due to varying code style, source code that is difficult to find, or abstraction layers that are challenging to see through. The source code for any library is just a click away in Go. Design idioms found in dependency libraries are usually similar to the ones we use, which ensures that the entire system is easy for our team to comprehend.
Almost a year ago, we blogged about our reasoning and methodology for choosing Go as our next generation platform here at clypd. A year is a long time, both in software technology and in the lifetime of a startup. Since paper is the traditional gift for a one year anniversary, it seems appropriate to write down our learnings so far.
When we posted “Getting to Go,” we included a list of the selection factors we used to evaluate potential platforms. The list includes
At clypd, we place a lot of value on testing as a mechanism for ensuring code correctness and our approach to testing constantly evolves. We’ve spent the past few months learning and creating a new approach to mocking and testing functions that need to access data, whether from a database, a file, or over the network.