Creating a new pipeline architecture & saving a failing labeling operation

Guidance fromMichael Woo, Ruben Mayer
OwnerShawn Farsai

Since customers at Scale are confidential, I will refer to it by its codename Alpaca.

Alpaca was a leading autonomous vehicle company. They were Scale's largest customer at the time. IIRC it was something like $20M 12-month deal.

When we came in, the margins were negative and the client was unhappy with the accuracy of the data. There were thousands of annotators working 24/7 behind the scenes, and debugging this pipeline wasn't going to be easy.

Through lots of grit and some creative ideas, in 2-months we turned it around from -10% margins to +80% margins, making it the most profitable operation at Scale. But more importantly, we fixed the recall accuracy and improved customer sentiment enough to secure a renewal.

The two insights were (1) triangulating low-value scenes that were artificially lowering our accuracy, and (2) creating a new pipeline architecture that allowed us to scale the operation to handle these scenes much more effectively. There's lot to go into here, but its more interesting for a live conversation.