Evaluating the success of an AI&ML use case

              Data science team has finished development of the current version of the ML model & has reported an accuracy or error metric. But you are not sure how to put that number in context. Whether that number is good or not good enough.

                In one of my previous blogs, I have addressed the issue of AI investment and how long before the business can know if the engagement has some potential or it is not going anywhere. This blog can be considered an extension of the above-mentioned blog. If you haven’t checked it out already, please visit : https://anirbandutta-ideasforgood.blogspot.com/2023/07/investment-on-developing-ai-models.html

                In my previous blog, I spoke about the KPIs like Accuracy and Error as thumb-rules to quickly assess on the potential success of the use case. In this blog, I will try to add more specificity or relativeness to it.

                Fundamentally, to evaluate the latest performance KPI of your AI&ML model, there are 3 ways you can go about it, in independence or in combination.

Consider human level performance metric.

For AI use cases which has the primary objective of replacing human effort, this can be considered the primary success metric. For example, if for a particular process the current human error stands at 5%, and the AI can have less or equal to 5% error rate, it can be determined a valuable model. Because AI with the same error rate, bring along with it - smart automation, speeding up the process, negligible down-time etc.

Example: Tasks which needs data entry can easily be replicated by AI. But the success criteria for AI does not need to be 100% accuracy for adoption, but just must match the accuracy which the human counterpart was delivering, to be adopted for real word deployment.

Base Model metric

In use cases for which problem areas getting addressed are more theoretical in nature, or the discovery of the business problem that can get addressed is in progress, its best to create a quick simple base model and then try to improve the model with each iteration.

For example: Currently I am working on a system to determine if a content is created by AI or not. For the lack of any past reference based to which the accuracy can be compared, I have taken this approach to determine the progress.

Satisfying & optimizing metric

We outline both, a metric that we want the model to do as good as possible (we call this optimizing metric) while also meeting some minimum standard which makes it functional and valuable in real life scenarios (we call this satisfying metric)

Example: For Home Voice Assistant, the optimizing metric would be the accuracy of a model hearing exactly what someone said. The satisfying metric would be that the model does not take more than 100 ms to process what was said.

Investment on developing AI&ML models – timelines & diminishing return

 

One of the most popular questions that I often get asked by the stakeholders is about the timelines required for a ML model to finish development. I will try to address the subtlety of this topic in this writeup.

AI development is a unique scenario where you are expected to deliver an innovation. It is a special case where the resource required is uncertain. And hence it is sometimes very difficult to understand when & where to “STOP”.

When I talk to businesses one of the questions, what I stress about the most, is for them to define what an “MVP” solution is to them. That is with what minimum accuracy or maximum error rate the AI solution would still be useful for their business.

If you are investing on AI use cases one of the concepts, I would recommend you understand is – AI resourcing & diminishing return. Please look at the graph below –

 

            So, what I suggest to the AI investors are if you haven’t reached an MVP by the point of maximum return, “STOP”. For example – By the end of PoMR if the model is still with an error rate of 30%, and that is something that does not work for your business, may be AI cannot solve this for you. Maybe it needs a completely different approach to solve this. Whatever is the case, deploying more resource is not the solution.

            Driving from my experience with all the AI&ML use cases I have worked for almost a decade now, a general thumb rule which I recommend is – The accuracy or error rate, that you get at the end of 3 months is your Point of maximum return. You should reach an MVP by then. Beyond that it should be fine tuning or customizing to specific business needs. By then if it it’s still miles apart from your business objective, may be its time to pull the plug.

            This is again an opinion piece, and these has been my experience. Will be glad to hear how the journey has been for you.

AI/ML and human brain - Similarities and their implication in Corporate and Political campaigns

 The premise of this article is how our brain is similar to a machine, and the biases and errors the machine experiences are experienced by the brain too, and in some cases this may be leveraged to drive success at a marketing or electoral campaign.

1. The good orator - No election has ever been won without a good orator at the helm. However good your policies are at the end of the day it needs to be sold. A good orator is like quality data to a machine. To drive learning in a machine, if we even want to propagate biases in some way, it needs to be clearly and strategically delivered. This delivery of data, to steer the mind and machine in a certain way, have also been called Propaganda in warfare.

2. The 360 degree delivery – To drive a target candidate to take a certain action, stimulus must be provided from every direction. It is widely accepted the same objective can be achieved with much less effort with stimulus from multiple dimension, rather than larger one-dimensional effort. It is the same reason we come across our favorite brand across multiple platforms like TV, radio, newspaper, social media, online ads etc. According to a successful marketing platform ‘personalized messaging across email, SMS, direct mail, and more, alongside personalized online response’ leads to a much successful marketing campaign. AI systems inspired by this characteristic of the brain is always advised to be built around with data from as many diverse source systems as possible.

3. What is in it for me - Human beings by the very nature of their existence and survival instinct mostly react to news and events that directly affect their well-being. The same idea is implemented in reinforcement learning where the agent takes measure to fulfill objective which in our case is survival. So to make your target audience to take notice of any policy or idea, it should be narrated as tightly coupled with the audiences’ well-being. It should answer their basic question how it will affect me.

4. Relative rather than absolute – Human brains intuitively understand something relative much better than anything absolute. If you ask most people if a deal is good, they will generally say it is good or bad based on how other people are getting deals. In the same way you can manipulate a machine to label something on a particular high or low range, by strategically infusing data on the other end of the scale. In the same way during a political or marketing campaign it is not enough to advertise your positives but also important to emphasize your opponents weaknesses.

5. Confirmation bias – Definition - “Confirmation bias occurs from the direct influence of desire on beliefs. When people would like a certain idea or concept to be true, they end up believing it to be true. They are motivated by wishful thinking. This error leads the individual to stop gathering information when the evidence gathered so far confirms the views or prejudices one would like to be true.” This is a psychological error that can always be used to one’s advantage during a political or marketing campaign. Political and Corporate organizations have at various times taken advantages of this by implementing a biased belief system at an early age of life or consumption cycle.

6. ML bias – Machine Learning applications develop inherent biases when fed data which may be tilted towards a certain stereotypical trend due to the flawed nature in which society develops. Like real-world example of a machine learning model that’s designed to differentiate between men and women in pictures. When the training data contains more pictures of women in kitchens than men in kitchens, or more pictures of men writing computer code than women writing computer code, then algorithm is trained to make incorrect inferences about the gender of people engaged in those activities. Human brain can be manipulated in the same way too. If you give enough example to a brain, associating people of certain characteristic to certain nature of actions – either good or bad, human brains inherently start associating these people to those activities without extensive thought.

But however amazing our brain is, it still have certain flaws which have been inherited by ML and AI processes as these are inspired by the brain itself. But I guess all these imperfections are what keeps us human.

Demonstrating Flask to create a simple Web-API OCR program

 Most of the time the real use of the Machine Learning code lies at the heart of an application which integrates the ML code to becomes a smart application.

“Consider the following situation:

You have built a super cool machine learning model that can predict if a particular transaction is fraudulent or not. Now, a friend of yours is developing an android application for general banking activities and wants to integrate your machine learning model in their application for its super objective.

But your friend found out that, you have coded your model in Python while your friend is building his application in Java. So? Won't it be possible to integrate your machine learning model into your friend's application?

Fortunately enough, you have the power of APIs. And the above situation is one of the many where the need of turning your machine learning models into APIs is extremely important.”- - https://www.datacamp.com/community/tutorials/machine-learning-models-api-python

The below code demonstrates how you can use Flask - A web services' framework in Python, to wrap a machine learning Python code into an API.

Github link: https://github.com/anirbanduttaRM/FLASK-OCR

Few things to note:

· Here it is assumed the image is already in the path of the python code. That is we are not inputting the image from API. Since that was not the primary object of the exercise.

· You will find 2 OCR functions –

image = Image.open("screenshot.jpg")

image.filter(ImageFilter.SHARPEN)

new_size = tuple(2*x for x in image.size)

image = image.resize(new_size, Image.ANTIALIAS)

txt1 = pytesseract.image_to_string(image)

#return jsonify({'Converted OCR A': str(txt1)})

and

image = cv2.imread("screenshot.jpg")

image = cv2.resize(image, None, fx=1.2, fy=1.2, interpolation=cv2.INTER_CUBIC)

image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)

kernel = np.ones((1, 1), np.uint8)

image = cv2.dilate(image, kernel, iterations=1)

image = cv2.erode(image, kernel, iterations=1)

cv2.threshold(cv2.bilateralFilter(image, 5, 75, 75), 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

txt2 = pytesseract.image_to_string(image)

return jsonify({'Converted OCR ': str(txt1)})

Use any of the one. Add return at the end of the one you prefer to use. I will suggest you should try the accuracy of both.

· I used postman to check the result. You can do the same.

· After running flask if you need to check if it has been installed properly again refer to the article I have mentioned above from datacamp to check the screen shots. And from the same link you can refer how to run and check your code in postman.

· Please note here since the objective was to demonstrate the flask I haven’t chosen a model that needed to be trained. IF you have a model which needs to be trained write and train the model in a separate py file and call that file from the flask py file.

Steps to run:

Run the flask python file -

Place the image in the python file location

For me the text image was -

Open Postman and run the API

Resource that was referred:

https://www.datacamp.com/community/tutorials/machine-learning-models-api-python

Transformer and its impact on Language Understanding in Machine Learning

‘Transformer’ a deep learning model was proposed in the paper ‘Attention is all you need’ which relied on a mechanism called attention, and ignored recurrence (which lot of its predecessors depended on) , to reach a new state of the art in translation quality, with significant more parallelization and much less training.

Due to its non-reliance on recurrence, the transformer does not require sequence data be processed in order, that allows transformers to allow parallelization and thus being able to train on much larger dataset on much lesser time. Transformer has enabled development of pre-trained models like BERT which has enabled much needed transfer learning. Transfer learning is a machine learning method where a model trained for a task is reused as starting point for another task. And as with BERT this approach delivers several advantages considering the vast compute and resource that is required to train the base model. BERT is a large model (12-layer to 24-24 layer transformer) that is trained on a large corpus like Wikipedia for a long time. It can then be fine-tuned to specific language tasks. Though Pre-training is expensive, fine-tuning is inexpensive. The other aspect of BERT is it can be adopted to many type pf NLP tasks. This enables the organizations to leverage large models build on transformers trained on massive datasets without spending significant resource on time, computation and effort and applying them on their own NLP tasks.

One of the challenge of using pre-trained transformer enabled models are they are trained and thus learns language representation of general English in case the language is English. Thus even strong base model might not always perform accurately when applied to domain specific context. Like – Credit could mean different thing when applied to general English or financial context. Significant fine tuning on specific tasks and customer use cases may go a long way on solving this.

Research is being done if similar transformer based architecture can be applied to video understanding. The thought process is if video can be interpreted as sequences of image patches extracted from individual frames, similar to how sentences are sequences of individual words in NLP.

Azure AI portfolio and its offerings - cheat sheet

 Courtesy: Almost all the information in this blog has been compiled from these 2 YouTube videos. So thanks to the original creators.

https://www.youtube.com/watch?v=qJGRd34Hnl0

An introduction to Microsoft Azure AI | Azure AI Essentials

https://www.youtube.com/watch?v=8aMzR8iaB9s

AZ-900 Episode 16 | Azure Artificial Intelligence (AI) Services | Machine Learning Studio & Service

Azure AI portfolio has options for every developer may it be in the form of

Pre-built AI models

• Advanced machine learning capability or

• Low code/ no code development experience

Azure cognitive services provide the most comprehensive portfolio of customizable AI models in the market. It includes

• Vision,

• Language,

• Speech &

• Decision.

It just needs an API call to integrate them to our applications.

Users can customize AI models using one’s own data without any machine learning expertise required. These models can also be deployed to containers so it can be run from anywhere.

For Business users Azure provides access to the same AI models through AI Builder which provide a no-code experience to train models and integrate them into apps within Microsoft Power Platform.

For common solution like chatbot and AI powered search, services are provided, which accelerate development for these solutions. These scenario specific services often bring together multiple cognitive services along with business logic and a user interface to solve for a common use case.

If we are looking to develop advanced machine learning models, Azure Machine Learning enables to quickly build, train and deploy machine learning models with experiences for all skill levels ranging from code first to a drag and drop no code experience.

It provides services that empowers all developers. It helps in the entire process by providing us with a set of tools. The processes include –

• · Training the model

• · Packaging and validating the model

• · Deploy the model as web services

• · Monitoring those web services

• · Retraining the model to get even better results.

Set of tools mentioned above include –

• Notebooks written in python/R

• Visual designer which allows us to build machine learning models using a simple drag and drop experience directly in our browsers.

• Machine learning model allows us to manage all the compute resources where train, package, validate and deploy those models so that we don’t have to worry about Azure infrastructure and underlying resources ourselves.

• Additionally, Azure machine learning comes with something called automl. This automated process allows us to perform different algorithms with our data and see which one scores the best and deploy that as our designated web service.

• Features of pipelines which allows us to build the entire process end-to-end.

Complete end to end solution for building machine learning models.