So You Wanna Keep Your Job in Data Science… Part 3

by Kevin Nutter

At System2, we’re big on feedback—upward feedback, downward feedback, and sideways feedback (I think the proverbial “they” has a name for this: 360 feedback). It’s the best way to get better, so we routinely ask our clients in the investment space to do the same by scoring our projects.

We do this semi-annually to understand what System2 work was helpful and what wasn't. This helps shape the initiative long-term and ensures we spend time on things that matter. Our suggested rubric for "scoring" projects is:

For each project, how valuable would you rate System2’s output?

20% of our projects are a waste of time

Almost 20% of our projects receive the score of Uninformative (these are most often described as ”a waste of time”).

1 = Imperative, 2 = Actionable, 3 = Informative, 4 = Uninformative

However, those are just 3% of overall time spent for the year. Since we know that one project can pay for the year, we often take risks to investigate potential, highly complex projects. We mitigate the risk by finding the quickest way to identify if there is any hope of success.

One great project pays for the initiative

An “Imperative” score is very difficult to achieve. We average 25 - 100 projects per year per client (depending on the size of the client), and we strive for 1 - 5 projects to receive an Imperative score. Below is the distribution from one of our clients. We had 4 of 70 projects, roughly 6%, meet the Imperative criteria.

Balance long-shot projects with informative work

System2 tries to strike a balance between safe and risky projects. As seen below, roughly 50% of the projects we did scored “Informative.” It’s important to provide consistent, high-quality work products to multiple investment teams. We focus on building a consensus across many teams that System2 is a valuable part of their investment process.

Great work takes time

We attempt to align the time spent per project with the score received (e.g., spend the most time on projects that score Imperative), and then monitor our time to evaluate how successful we are. Typically, as noted in the previous blogs in this series, high-complexity projects can take a few weeks to a few months.

Asking for feedback can be difficult, but being diligent about collecting it enables System2 to make decisions intra-year so our time is put towards projects that our clients find most valuable. It ensures that we’re taking enough risks to find and complete the few projects that separate us from any other service provider on the market.

Recap of System2’s five-step process

Building an indispensable data science department requires much more than great data science. Unlike an in-house data science team, which only needs to build one successful data science department, at System2, we build a new department for every client.

Five things must happen to manage and develop a data science department effectively:

Prioritize - actionable analyses above informative ones.
Track - the time the data science team spends on different analyses.
Score - ask the client’s team for a score on each analysis.
Present - communicate the scores and efficiency to the senior stakeholders.
Iterate - optimize the data science team’s time on the activities with the most impact.

I’m diving into each of these steps in more detail through 2024.

Disclaimer: All opinions expressed by System2 employees and their guests are solely their own and do not reflect the opinions of System2. This post is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of System2 may maintain positions in the securities discussed in this post.