In many industries, sales is used to ‘selling’ the vision of the product in advance of it being built, and customers assume vapourware by default. No one bats an eye because we’re accustomed to the idea that engineering will always be able to fulfill whatever we’re selling, given enough time and money. New technologies like machine learning and blockchain offer a world of possibilities, but many of these possibilities may not actually be able to be implemented in practice, even with a huge budget. It's easy to promise "The product will automatically predict X with high accuracy." where X could be anything from detecting a security breach to predicting stock prices to finding the perfect outfit for you wear. But even if the prototype is already 70% accurate, it may never get to 80%, or whatever you need it to be to be commercially viable. Don't oversell the vision, be honest. Be honest with customers, and be honest within the company walls too. Imagine slightly implying or exaggerating results to your manager, who then implies or exaggerates a little further to their manager, etc. and pretty soon the board is hearing that the product is 110% accurate and can slice bread too!
Be Absolutely Clear: Is This a Prototype or a Commercial Product? When starting a traditional project with new technology, it’s common to start with a prototype. Although best practice is to "throw away" that prototype and start fresh on the commercial development, management often pushes teams to continue work directly on the prototype to turn it into a commercial product, skipping the throw away step. The result is a shaky development foundation that can result in code debt and bugs for the lifetime of the product. Far from ideal, but it can be liveable in a business sense. But let's take the example of an AI/ML project. The mechanics of delivering such a program are very similar to traditional projects on the face of it, so it is tempting to think of the implementation as being the easy part. But because of the non-deterministic nature of AI/ML, all the pitfalls of ordinary software development programs - misaligned objectives, underestimation, lack of process, skimping on QA, ignoring risks, etc. - are amplified x10. To deliver successfully, you need to ratchet up the program diligence (see Delivering an AI Program: What Could Possibli Go Wrong?) Skipping the throwaway step is no longer possible. When a data scientist is prototyping a model, their focus is truly to validate that the product could ever have a chance of being delivered, and if not, what other related things it could do instead to deliver value. Prototyping AI involves many shortcuts - working on a "clean" data set (artificially clean, biased, small sample), where the availability and timing of the data may not be an issue. The work is done on a Jupyter notebook or other simulated environment that is vastly different from a real-world environment, and using models and techniques that could never be used in a commercial context. The business problem itself may be fluid, as the data scientist may realize that the algorithm can't solve X but can solve Y instead. When the prototype has met its objectives, a new project should be started from scratch for the commercial development. In commercial development, the team's work is focused on entirely different things such as:
So, is the project a prototype or a commercial endeavour? Alignment from the team to upper management is critical. The answer can't be both. Related reading:
3 Comments
5/24/2021 08:23:39 am
12/1/2021 06:27:45 am
Excellent article! Your post is essential today. Thanks for sharing, by the way. Your comment will be posted after it is approved.
Leave a Reply. |