Scaling Woven’s Hiring Assessments with AI & Machine Learning Development

To maximize our impact, we identified a key metric we could meaningfully improve: the percentage of skill assessment scenarios that could be automatically scored. 

Woven was able to automatically score 64% of assessments before we started.

Focusing on Accuracy

Woven’s system prompts candidates with a set of scenarios to answer as if they are on the job. For example, responding to a support ticket about a bug or writing code to calculate a billing invoice.

There were several areas of the product we could work on to improve this metric. To boost the accuracy of entire groups, we focused on groups of similar scenarios that weren’t performing well, rather than individual scenarios.

We identified several low-accuracy scenarios and analyzed their scores. Our approach incorporated two key AI techniques:

• Natural Language Processing techniques, including stemming, summarization, padding, and topic extraction, normalize the text, remove human bias, and make it more machinable
• Machine Learning development techniques, including coarsening and binning, to improve the ML system’s accuracy

Getting to Machinable Data

One scenario wasn’t scorable by AI because the test questions had recently been changed, so data from the old questions was not usable when scoring the new questions. We used automatic code rewriting and transfer learning to make the old data relevant to the updated test questions. This spared Woven dozens of hours of manual labor and provided a roadmap for using existing data when test questions change.

For several other scenarios, we found problems with the scoring data, leading to lower accuracy. These problems were often the result of human graders having different interpretations of the grading rubric. In these instances, AI wasn’t the most helpful approach. Instead, we wrote regular expressions to improve these groups of scenarios. We further identified several commonly misunderstood scoring rubric items that could be replaced by unit tests, eliminating ambiguity.

To wrap up our engagement, we summarized our findings and delivered prototypes Woven could implement. We also provided a plan with Woven’s next steps in the short, medium, and long term and cautioned against potential pitfalls.