Confronting the 6 Challenges of LLMs: What Every Business Needs to Know

How can an LLM make such a simple mistake?

Because LLMs generate responses based on patterns in data they’ve been trained on, not on verified facts.

The problem of hallucination is not limited to popular LLMs such as ChatGPT alone. It extends to all types of LLMs, whether they are free, paid, or custom-integrated solutions. The fundamental method they use to learn from large amounts of data is often the same, and input from users can sometimes be ambiguous, which may result in such errors.

A different type of hallucination occurs when an LLM’s training data doesn’t match the query’s context.

For example, imagine you are discussing strategies to increase sales in a software company that sells to enterprise customers. You mention, “Our biggest challenge in closing deals is the lengthy ___.”

If the LLM was trained on mostly general business data, it might predict “sales cycle.”

However, if your specific context is that you are trying to highlight approval processes within large organizations as the key bottleneck, the more relevant completion would be “procurement processes.”

That’s why it’s important that the LLM be trained on data that is relevant to you and your needs, and to be diligent when reviewing LLM output.

2. Cost Uncertainty

When it comes to budgeting for LLMs, many businesses find themselves in uncharted waters.

Sure, platforms like ChatGPT or GitHub Copilot might have standard rates. But the cost of using LLMs in custom business solutions can vary significantly.This is largely due to the token-based charging model used by LLMs, where the cost depends on the number of tokens (basic units of text, such as words or parts of words) processed. Estimating the exact cost becomes challenging, as it depends heavily on usage patterns and the specific applications of the technology.

3. Context Limitations

LLMs work a bit like us humans — they have short-term and long-term memory, called context windows. To answer questions well, they need to load up relevant info into their “memory.” Bigger context windows help them understand better and respond more accurately, but they also mean processing more tokens, which can hike up costs.On average, there are about 0.7 words per token. For perspective, Google Gemini’s model has a token limit of 1 million tokens.

A token limit of 1 million tokens (equivalent to approximately 700,000 words) may seem sufficient, but tokens encompass various aspects:

• The immediate question 
• Relevant background information 
• The ongoing conversation’s history 

As models extend their token limits to address these needs, businesses must be prepared for escalating costs.

What challenges might employees face with LLMs?

Deploying LLM solutions is just the first step. The real work begins with training employees on how to properly utilize the technology. Simply having access to a large language model is not enough to unlock its full potential.

Using LLMs effectively requires users to overcome a significant learning curve in understanding the models’ capabilities and limitations. It’s like having a powerful motorcycle but no idea how to operate it.

Organizations must invest efforts into educating their teams on these key challenges:

4. The Importance of the Right Prompts

The effectiveness of a LLMs depends significantly on the questions you ask. If the prompts aren’t well-crafted, the responses you get may not be useful.

Think of crafting the right prompt as similar to programming: you’re essentially coding with words. This task requires a clear understanding of your problem and an insight into how LLMs interpret and process the information you provide.

5. Probabilistic Nature and Consistency

LLMs are probabilistic, which means their responses can vary even when given the same input, whether you repeat the query minutes or days later.

This variability can be confusing and may seem unreliable, especially in business settings where consistency is key. Users must understand this aspect to set realistic expectations and know when and how to rely on LLM outputs.

6. The Black Box Problem

Often, LLMs operate as a “black box,” providing answers without transparent explanations.

For example, if you ask, “Where did this info come from?” they might not explain, making it difficult to verify the accuracy.

I’ll show an example of this using our conversation from earlier: