Offering - Private NLP development

Abstract: This post is intended for business users who are looking forward to understand and prepare for future investment in AI development for their businesses. It covers the need and way forward or high level architectural plan to build custom NLP (Natural Language Processing) application for a business. 

Foreword: ChatGPT is magic. It is very unlikely that you, the reader of this, have not used or experienced it at least once till now. ChatGPT is a LLM means Large Language Model. It is trained with almost all the writings and non text data of the internet. It is like a human who has consumed the whole of internet and can use that as a basis of knowledge to analyze content, situation etc. So, it a super intelligent probably very high IQ person who knows about many or all the things. Its base is logical reasoning, following patterns, voting. Means it does not know what is right or wrong or it does not have any subjective inference of its own. That's the basic difference between AI and human being. A human have consciousness. Means he/she can infer very different conclusion than what is logical, practical and mostly popular.

In short, the minority report may be just obsoletely dumped. Even AI models like ChatGPT, may conclude very different conclusions or output artifacts like document, image etc than what is shown to the user. But it is most likely that they are dumped, never seen.

Here starts the problem. What is right or expected output or which one has most potentially be a viral content, may not be the one corporate wants! Business may demand very different thing. They need their AI to follow a discipline, give more importance to the internal documents, processes and history rather than what is believed to be the one. As for example, for a "CRITICAL" problem, the resolution time may be 8 hours instead of 2 hours as suggested by the LLM. Here is a link of an IBM article about the same point.  

For large corporates, they already know it and already investing and ahead in few places for this. It is for the MSME who does not even dare to dream about it now, as on December 2024. They are afraid of very huge investment, lack of clarity about their own use cases, why they require it at the first place. 

Prompt Engineering sits here. You try different prompts, learn secrets sometimes like superstitious beliefs which probably nobody knows the actual value. But no matter whatever you try, it is very unlikely that you will get predictably good results every time.  

Another problem of these models is that they hallucinate! 

AI models like ChatGPT or Gemini etc may very smartly produce garbage. All wonderfully packaged. It may infer something from something and use that as a base to produce another thing. Like i have a piece of code in C written in 1992, in a specific environment like X-Windows, and AI may use that as a base to predict a python equivalent of that in short cut! You never know if what AI is saying is right or wrong, you should double check it. And that is called Ethical AI! AI ethics say professional should be sure about the correctness of the data before forwarding it to the business! It is because of the limitation in the data the AI is trained on. 

All of these makes it practically impossible to use it as part of automated workflow. You can never allow AI to respond to customer automatically until you are sure. So, the solution is to develop your own NLP. 

Thanks to the collective work towards AI by various institutions, corporates and researchers, we may dare to think about it even as an individual! We don't need to build a complete model ChatGPT. We can use an existing LLM model as a base and customize or add on to it with our own data. Schematically, the scenario is like the following:

Hence, building a custom NLP seems to give much better peace of mind, relevance and control over the automated response. And surprisingly, cost is not as high as it seems from a distance. Roughly a Dev environment, that can be shared among 2/3 contributors, can be obtained at roughly 20K per month. So, a 6 month window for such an ambitious project would hardly cost 1.2 Lakh INR. 

Unfortunately base models in other Indian languages like Hindi, Bengali etc are either not yet available or not upto the mark for this kind of project. We may need another breakthrough in algorithm  

beyond AIAYN (Attention Is All You Need). 

Overall, for ambitious and conscious biz houses, this project is very much appropriate.