Sneak peek into Playment’s Engineering
How core tech was shaped and how tech-stack/architecture we evolved with.
Playment, One stop solution for all your Data Labeling needs. You might be curious to know how the product was engineered for highly dynamic and varied requirements from our capricious clients and how to support the product for high-scale, dynamic loads(very large content) and fully automated pipeline.
In the early days, Playment was just a simple crowdsourcing platform and its data management (upload, download, transformations) were being carried out manually. With a quantum boom in data units and clients, it was no longer feasible for us to cater to solutions manually. Hence, we started formulating a plan to help us better cater to the needs of the market.
Problems
- Automating the whole pipeline from client input/output, task creation, problem-solving, data management. This will help streamline the project
- Whimsical project management will not be generic process management as each project was unique on its own and the workflow process had to be tailor-made to suit the client’s specifications and uses
- Sandboxing — where project managers could experiment their various workflow ideas
- Plug and Play components — This feature will provide us with immense flexibility and power to ship fast, experiment with AI components, and for PM’s to have pluggable components in their workflow which they can edit modify remove or create any combined combination/flow. But each step in the workflow has to be individually scalable.
Approach
Not only were we new to this domain but since this particular market was just starting up, the clients were also going through a phase of experimentation
We had lots of ideas pouring in and we were on the way to fabricate a very complicated idea.
We wanted to start from scratch as it was not just a simple data flow management pipeline but something that could be custom-built to suit the needs of the client. We were also facing a lot of changes in project guidelines, definitions and processes at regular intervals from our clients. And thereby we build a solution that would cope with the frequent changes and would be highly accurate.
Solutioning
Listing down above problems and expected features, We were very sure that we need a new language/framework which is highly efficient, easy to write Concurrency based components, developer-friendly as a lot of these components would grow large and it should be easy for any developer to integrate and write code in it, Strong language design principles etc.
Apart from this, our very focus was on preparing a good code architecture which will allow us to have pluggable components.
So breaking the above into 2 parts
1) Language Selection
These points were very important for us to narrow down on most premium selection for us which was GOLANG amongst others were Java, Rust, Scala, Kotlin, Ruby. There were lot of factors for this but most imporant were industry approval, performace and no JVM reliance(as it requires lot of unnecessary tweaks). Language inherited lot of good design principles which we liked, although error handling pattern, lack of generics were a bit painful in early adoption. But golang was wat we needed. Channels and goroutines (a high performance [CSP and goroutines] with low memory footprint, an optimized GC and inbuilt libraries such as net/HTTP) were serious design improvements for a programming language.There was a drastic system improvement both in terms of memory and speed (in comparison to our existing stack of JAVA and NODE)
2) Code Architecture
RabbitMq (needed ingredient) — Our aim to build the Seamless, highly scalable, persistent, dynamic data management pipeline became the way to easy after we added Rabbitmq to our tech stack. It gave us the power to throttle the traffic according to the various pluggable services and their limit constraints for example if crowdsourcing service can only handle 30 inputs/sec then this pluggable service will have this constraint set to the orchestrator service which will limit the speed to not more than 30.
Manual Ack feature of RabbitMQ help us to only Ack when we are done with the current step processing and then only We push into another queue. So even after the system restart, we can process unacknowledged as data will persist in the queue and would again be subscribed by the service.
// Setting autoAck parameter (3rd param) to false which means we have to manualy Ack it
deliveries, err := rabbitmq.Consume(queueName, "", false, false, false, false, nil)
if err != nil {
return
}
...
...
...// Here we are manual ack the input received from the queue after we are done with the processing
err := input.delivery.Ack(false)
if err != nil {
panic(err)
}
Boundary Creation — After starting with a firm prototype, we made sure development was easy and not complicated. We designed it to enable a new developer to spend lesser time and resources on the new pluggable he was working on. Then we worked on our interfaces along with well-specified boundaries with a goal to make this product survive for as long as possible. The code contains a lot of interfaces which helped us a lot in the loose coupling of the dependencies. And in return, it helped in code testing, mockable and better readability of the code. see this example for golang advanced testing.
Dockerize Applications — The final step was to dockerize all the services which help our CI/CD pipeline and auto-scaling along with the other containerization benefits.