Creating a new pipeline architecture & saving a failing labeling operation
Guidance from | Michael Woo, Ruben Mayer |
Owner | Shawn Farsai |
Since customers at Scale are confidential, I will refer to it by its codename Alpaca.
Alpaca was a leading autonomous vehicle company. They were Scale's largest customer at the time. IIRC it was something like $20M 12-month deal.
When we came in, the margins were negative and the client was unhappy with the accuracy of the data. There were thousands of annotators working 24/7 behind the scenes, and debugging this pipeline wasn't going to be easy.
Through lots of grit and some creative ideas, in 2-months we turned it around from -10% margins to +80% margins, making it the most profitable operation at Scale. But more importantly, we fixed the recall accuracy and improved customer sentiment enough to secure a renewal.
The two insights were (1) triangulating low-value scenes that were artificially lowering our accuracy, and (2) creating a new pipeline architecture that allowed us to scale the operation to handle these scenes much more effectively. There's lot to go into here, but its more interesting for a live conversation.