The Solution for Gen AI May Not Work the Way People Want

BrainBox AI, a vendor of energy management systems, told GlobeSt.com in March that it had found a solution to a big generative AI problem: hallucination. That’s when the software makes things up.

“For us it’s super important,” company co-founder and chief technology officer Jean-Simon Venne said about its new generative AI interface for customers. “As soon as we have an hallucination, the building engineer will stop using it.”

“When you think about it, it was obvious that we were going to end up where we are with the way we train these models,” Venne added. “You train it on all the information on the Internet. You’re training a model on information that is exact information, that is factual, and a lot of false information. When you ask a question, guess what you’re going to get? You’re going to get both.”

BrainBox AI only trains its software on its data. There isn’t anything external. This sounds something like retrieval augmented generation, or RAG. Here’s how Microsoft describes it:

“Retrieval Augmented Generation (RAG) is an architecture that augments the capabilities of a Large Language Model (LLM) like ChatGPT by adding an information retrieval system that provides grounding data. Adding an information retrieval system gives you control over grounding data used by an LLM when it formulates a response.”

Grounding is an AI technique in which the software is restricted to using verifiable sources of information, often internal vetted sources. But is that always viable?

Not necessarily, says TechCrunch. “Things get trickier with ‘reasoning-intensive’ tasks such as coding and math, where it’s harder to specify in a keyword-based search query the concepts needed to answer a request — much less identify which documents might be relevant,” writes Kyle Wiggers. “Even with basic questions, models can get ‘distracted’ by irrelevant content in documents, particularly in long documents where the answer isn’t obvious. Or they can — for reasons as yet unknown — simply ignore the contents of retrieved documents, opting instead to rely on their parametric memory [a name for the stored information the system was trained on].”

The systems are complicated and somewhat mysterious. Even the developers don’t necessarily know how the software makes decisions at any given time. If they always did, why and how could the software do unexpected things?

This isn’t to say that generative AI isn’t worth consideration. Like many technologies, of course it can be useful. But nothing is so incredible that it can do everything someone might want. An old saying in technology, and life, is that there are no magic solutions that make everything better. Take care in choosing tools and realize that there is always work to do.