EQL LLM

Demo of LLM response and classifier updating the web site view..
MLflow experiment loss logs
Unity Catalog with registered model
Workflow job
Logged Experiments

Inspiration

We were inspired by the desire to provide better customer service and the opportunity to develop skills in LLMs. With a great deal of hype being peddled by the media and consultants, this was a great opportunity to actually quantify the levels of effort required to annotate data sets and fine tune models in comparison to the accuracy / quality of the outcomes.

We have a passion for emerging technologies that will provide future business benefits.

What it does

Our solution provides conversational responses to customer queries using a Llama2-chat-hf based LLM, fine-tuned with corporate content. It then performs zero-shot classification of the prompt and LLM response on a BERT model to determine which page of the customer facing website best enhances the information being presented to the customer.

As the public facing Energex website doesn't permit embedding, each relevant page was exported as html files, and these are displayed in the client application vs live embedding of the Energex website.

How we built it

Our hackathon submission was built through excellent teamwork! 4 members all contributing to annotating and planning all implemented features. Daily stand-ups were used to review and allocate tasks where specific skills where needed. (e.g. creating an Interface, fine-tuning, lang chain, content classification, business engagement etc.)

The solution was built in a Hackathon provisioned Databricks workspace. The LLM fine tuning started off with the smaller 3B dolly model, then expanded to larger and different models that are better suited to our purpose (chatting to a customer)

Challenges we ran into

The challenges we encountered were:

Annotating content is laborious and critical for good results
Challenges in fine-tuning models to avoid random responses differing from our official content
Distributed compute with multi-GPU clusters
Not only challenges but more opportunities
Consolidate different opinions and thoughts from team members

Accomplishments that we're proud of

We were able to produce and fine-tune an LLM model with no prior experience in LLM’s.

We were able to use this opportunity to demonstrate capability to our internal customers and seed some thoughts around future use cases. Final delivery of the submission was made by the absolutely great team efforts and team leadership.

What we learned

Lots... But most significantly:

Logging experiments with MLFlow
creating model serving endpoints
importance of prompt engineering
using zero-shot classifiers
Annotating content is laborious and critical for good results

We also identified many more potential use cases for our business that we could deliver with this same general Databricks pipeline.

What's next for EQL LLM

The next key steps for us are to navigate the cyber constraints and firewalls to enable LLM tuning and hosting within our own existing Databricks environment. This is a very significant milestone as we can then demonstrate complete control over the environment and allow corporate sensitive data to be used in self-hosted LLM's.

We will then focus on how to constrain LLM responses to locally indexed Vector databases where we have use cases where responses can only contain Corporate Content.

Through our LLM Hackathon experience we have not only uncovered a large range of LLM opportunities at Energy Queensland, but importantly have developed the understanding of different LLM concepts, the effort required and accuracy that could be achieved and whether this can fully solve a business problem.