Part 2: Begin with a Decision that Many People Make Often, and Make Quickly
You’ll shoot yourself in the foot if you try to solve the sort of problem only a 10th-level wizard specializing in conjuration makes the first Tuesday of every prime-numbered year. Decisions made rarely or by few tend to be very difficult or to have little generalizable utility.
- What is the ideal flux density be for each individual magnet in a particle collider with maximum experimental resolution around the 125 GeV range?
- Is Dragon’s Egg or Mission of Gravity a better first novel to study in a Hard Science Fiction class?
Sure, it might be fun to build an AI that could actually solve these problems, but for now it’s much more efficient to leave rare problems to humans. Remember – we spend most of our time mired in decisions everyone makes.
Begin with a decision problem where
- it takes less than a minute to make the decision
- lots of people make this sort of decision, and
- people who make this decision tend to do it often.
This is really a litmus test for deciding whether a problem meets the requirements mentioned in Part 1. A problem that passes this test will satisfy many of the requirements.
Choose a decision problem where it takes less than a minute for a person to make this decision.
It should take less than a minute to make this sort of decision. If it takes one minute to make the decision, you can only manually check about 500 samples per day. Beyond this point you lose the ability to reasonably manually verify the correctness of your results. That is, assuming you have requirements calling for over 90% accuracy. If you don’t have a lot of labeled data, this small of a return on time invested in labeling data that takes long to label isn’t usually worth it. I’d also question: if it takes more than a minute to make the decision, is it really not reducible to a set of smaller decisions?
Say you’re looking to make an AI to help decide what expensive watch to buy. Things that might go through your head when making the decision might include:
- Is it within my budget?
- Is it the color I want?
- Is it available in my size?
These are simple problems that an AI assistant could use to automatically discard watches that aren’t worth considering, leaving you to focus on:
- Is it comfortable?
- Do I like the style?
Further, the easily automatable pieces of the problem are generalizable. They aren’t just applicable to watches, but to shoes, shirts, and a variety of other clothing items and accessories.
Choose a decision problem where lots of people make this sort of decision.
If many people can make the decision, you can easily check your work by collaborating with them. You’ll know you’ve found one if there is an entire profession with people constantly making this decision.
The decision-makers are your source of requirements (hint: you’re making this algorithm to automate away the more tedious of the decisions they make; you’re building it for them). They’re a great source for labeled data. If labeled data isn’t easy to come by, you can usually
- outsource data labeling to them,
- consult them on tricky cases, and
- literally use their daily work as training data.
Most preferably, as the designer of the algorithm YOU should know how to make this sort of decision. If you learn how to make the decision, or at least get some intuition, it saves you a lot of time when verifying system performance.
Choose a decision problem where people who make this decision tend to do it often.
This gets at the amount of data which may exist, or is easily generatable. If it’s a once-per-year decision, you aren’t likely to have a lot of historical data to base your model on.
If your system is designed to be used in users’ everyday lives, they will see the improvements from your system constantly. Highlighting common potential errors in documents as they’re typed is a prime example: people need help ensuring they typed a word correctly or haven’t made some easily-catchable mistake. Instead, they’re free to focus on what they actually want to say rather than whether it’s really spelled “concured” or “concurred”.
If it cost me nothing, given the choice between not having to make an almost-mindless near-daily decision and one I’d have to spend hours contemplating but may only do once in my life, I would choose to do away with the former. What’s great is it usually ends up being simpler to automate as well, so not only is the cost lower, but the reward is greater.
