Don't GenAI Everything
In case you missed it, for the last few blogs posts, I had been using Stability AI's Stable Diffusion model to generate the banner images, and I have been mostly successful. Did I really need to? Perhaps not. For this post, I spent almost 30 minutes trying to figure out the "right" prompt to generate the image I wanted: "A car stopped in front a stop sign."
Fortunately, I am able to run this code on a Google Colab notebook for free. However, it still takes time and I still couldn't get what I wanted. This was close, but the "stop" sign resembles but isn't one that we would recognise.
I finally did a search on two stock art websites that I had been using previously - Unsplash and Pixabay - and found something suitable within 3 minutes. All I needed was a little cropping and that's it!
When seeking a solution, especially one involving digital technologies, we should always be mindful not to use a sledgehammer to crack a nut. As exciting and revolutionary Generative AI might be, not every solution requires a large language model. Artificial intelligence and machine learning are fields of study that are more than half a decade old. The transformer architecture itself is based on artificial neural networks at its core, and these have been around since 1943. Along with ANNs, other ML algorithms have been used successfully in computer vision, decision making, and predictive analytics. Amazon, since its early days, has used association rule learning to perform market basket analysis to sell products to online shoppers. Many large supermarkets did the same when deciding that cigarette packets (in the old days), some magazines, and chewing gums should be displayed at the checkout aisles.
When building out solutions involving large language models, it is very important to consider issues like data privacy and sovereignty, and the computational resource requirements when performing tasks like fine-tuning. More often than not, you are looking to use a model that has a large number of parameters, and these tasks may require quite a bit of GPU compute resources to complete the operation in a decent amount of time. These resources are often scarce and more expensive than regular compute instances.
If you're looking to infuse AI into your business, there are several options that Oracle has made available for a while now. Oracle in their Oracle Artificial Intelligence 2024 Sales Specialist learning path, presents three general categories of AI solutions:
- Ready-to-go
- Ready-to-work
- Ready-to-build
Oracle AI Solutions
Ready-to-go
For a developer, these are probably the least interesting. Here, Oracle refers to AI solutions that are already integrated in to Oracle's products, and don't require any customisations. Products such as Oracle's Fusion Apps suite of SaaS offerings, and systems like the Oracle Cloud Infrastructure (OCI) observability and monitoring, and their flagship database, the Oracle Autonomous Database (ADB).
Ready-to-work
The ready-to-work solutions are where I think most web developers and businesses can exploit to develop AI-based solutions that are tailored to the organisation's unique data and operational needs. These services require a minimal amount of knowledge about machine learning to begin. Let's look at some typical scenarios.
If you had a dataset with well-defined "inputs" and "outputs", and is of reasonably high quality, the AutoML UI is available to ADB customers and it allows users to execute a typical ML model training workflow without getting involved in every aspect of the process, such as, algorithm selection, adaptive sampling, feature selection, and model tuning. This easy-to-use interface allows users to train ML models specific to their datasets, and then deploy them as REST endpoints. These ML models can then be consumed and used for inferencing in your favourite application development platform, Oracle APEX.
Next, we have a variety of Oracle AI services that offers pre-trained ML models for different types of tasks. Simply understand the input and output parameters of the REST endpoint, and voila, you have an intelligent application. Current pre-trained models include:
- Language
- Speech
- Vision
- Document Understanding
- Anomaly Detection (deprecated); and more recently
- Generative AI
Look for processes that are manual, tedious, and time-consuming. For example, accounts payable (AP) processing. Are invoices, received either in the mail or a PDF sent via e-mail, being rev. Automated classification in a document management workflow. Labelling photos taken by field engineers. Automatic tagging of helpdesk tickets by extracting entities. These are just a handful of situations where business processes can be enhanced and assisted using ML without a lot of effort.
Pre-trained models are typically trained over large data sets, and while they might be good enough for some general tasks such as identifying a car given a photo of a street, there are situations where the use case demands a prediction model that is specific to the domain, for example, identifying a custom electronic component on a printed circuit board. And in such situations, some services like Vision and Language allow developers to train a custom model using the customer's labelled datasets. In a previous blog posts, I described how I trained a custom model that could identify an Oracle APEX challenge coin.
Ready-to-build
When the ready-to-work solutions do not sufficiently meet the requirements, then consider using the OCI's various infrastructure platforms for supporting machine learning. You can choose either to use the OCI Data Science service to experiment, train, deploy, and manage your ML models, or build your own MLOps strategy using computes or the Oracle Container Engine for Kubernetes (OKE). In previous blog posts, I had demonstrated both approaches, two involving LLMs, and another, using traditional algorithms used for classification tasks.
For this post, I decided to experiment with LangChain to do in-context learning instead of fine-tuning a LLM to score user-submitted reviews like I did in a previous post. And instead of working with open source models from Hugging Face, I opted to use the OCI Generative AI service. A "blend" of ready-to-work and ready-to-build solutions. Here's a sample code to make sure things were working as hoped.
import os
from langchain_community.llms import OCIGenAI
from langchain_core.prompts import ChatPromptTemplate, \
FewShotChatMessagePromptTemplate
from langchain_core.output_parsers import StrOutputParser
sample_reviews = [
{
"review": """The machine brews tasty coffee drinks. It is also stylish
and nice. But the sticker that warns about machine testing drove me nuts
because I couldn't just unstick it. I tried for hours to remove it cleanly.
Eventually, soap and water and some time helped to soften the remaining sticker
paper/glue mixture and I could peel it off with my nails.
""",
"score": 5
},
{
"review": """The machine is pretty, but overall hard plastic. The water
tank and filter are easy to remove and replace. But it uses water before and
after each coffee, so you have to refill the water more often than I expected.
The noise is loud and the top cover for the coffee grains looks cheap and does
not break any of the sound. The strengh of the coffee is not for European
coffee people. I have to always use the strongest grindr and flavor and still
does not taste better than a Nespresso pod (which I still have). Also, the
products images in the 5400 look like a sticker. The easygo latte is the best
piece, in my opinion, because it is easy to remove and clean. However, for the
price, it could overall be a better, more solid, stainless steel material. It
is also sometimes slow to turn on, start, brew and turn off. I won't say it is
a bad product. It is great, but not EXCELLENT, as one would expect. Thinking
wheter is worth keeping it or not. So just wanted to share a transparent
review.
""",
"score": 3
},
{
"review": """I’ve had a love/hate relations with this machine since we
got it. It’s loud and slow, and I really struggled to find a coffee setting I
like. It doesn’t brew full cups unless you double them. The macchiato setting
was the best one…..until the frother broke (2 years after purchase). It leaks
from the bottom (common issue when I tried to troubleshoot) and also now the
steam comes from the top instead of the steamer (another common issue). No
repair shops close to me. To replace the lattego, it’s impossible to find a
Philips version, and any ones I’ve found that would be compatible are over
$150! So now, I have a $1300 coffee maker that brews 3/4 cups of coffee. When
it worked, it was maybe a 3.5 - 4 star, but I personally would never buy this
machine again.
""",
"score": 1
}
]
sample_prompt = ChatPromptTemplate.from_messages(
[
("human", "{review}"),
("ai", "{score}")
]
)
few_shot_prompt = FewShotChatMessagePromptTemplate(
example_prompt=sample_prompt,
examples=sample_reviews
)
prompt = ChatPromptTemplate.from_messages(
[
("ai", """You are an expert at rating user reviews. Predict and
return only the score and nothing else."""),
few_shot_prompt,
("human", "{review}")
]
)
llm = OCIGenAI(
auth_type="API_KEY",
auth_profile=os.environ["OCI_CLI_PROFILE"],
model_id="meta.llama-2-70b-chat",
service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
compartment_id=os.environment["COMPARTMENT_ID"],
model_kwargs={"temperature": 0.2, "max_tokens": 5 }
)
output_parser = StrOutputParser()
chain = prompt | llm | output_parser
reviews = [
"""Fantastic addition to the kitchen. We use it heavily everyday with zero
issues. Easy to maintain and all parts easily available. My only wish is that
custom programming was more flexible and there was more manual control
especially regarding the steam/milk function.
""",
"""Good when it works, had it 3 weeks now and fails to work more than it
does. Water leaks into the drip pan all the time overflowing constantly. When
it works its ok, but for that kind of money it should work constantly.
"""
]
for review in reviews:
response = chain.invoke({"review": review})
print("{review}\n{response}\n===".format(review=review, response=response))
This generated the following outputs:
Fantastic addition to the kitchen. We use it heavily everyday with zero issues. Easy to maintain and
all parts easily available. My only wish is that custom programming was more flexible and there was
more manual control especially regarding the steam/milk function.
AI: 4
===
Good when it works, had it 3 weeks now and fails to work more than it does. Water leaks into the drip
pan all the time overflowing constantly. When it works its ok, but for that kind of money it should
work constantly.
AI: 2
===
The results were pretty accurate for this very small proof-of-concept. If you'd like to validate, these reviews were taken from this Amazon product. 😉
In future posts, we will look at various options for operationalising LangChain applications that includes using Oracle Accelerated Data Science (ADS) to deploy them to the OCI data science platform, and LangServe to host these applications. For now, I wanted to wrap up the initial discussion and point out that, yes, I seemed to have strayed back to using LLMs to create an automated review scoring application. Not all business problems will benefit from a GenAI-based solution, however, some might. Trying to score a review is essentially a sentiment analysis task. While the OCI language service has a pre-trained model for this task, it is limited to performing aspect-based and sentence-level analyses, while I need it it to score the review in its entirety. For results, the model returns the probabilities of whether the text is negative, mixed, neutral, or positive, but I needed it to provide a score from 1 to 5. With these factors in mind, I decided to pursue a LLM-based solution.
Ready-to-work HOL at APEX World 2024
Unfortunately, due to conflicting commitments, I will not be able to attend the NLOUG APEX WORLD 2024 conference that will be held later this week, on March 21 and 22. I was privileged to have been selected to present "Build Intelligent APEX Applications Now!", and the Oracle LiveLabs workshop "Build an Intelligent Oracle APEX Application". Both topics would have examined the ready-to-work" scenarios. Fortunately, Heli Helskyaho had agreed to help deliver the hands-on-lab at the conference, so if you are interested in learning how to use Oracle's AutoML UI to train, deploy, and then perform inferences by simply calling a REST API, then you are in extremely good hands. Heli is the real expert, and I can only walk in her shadow!
Before I end this post, I wanted to take the opportunity to thank the abstract and organising committee for their invitation this year. I sincerely hope that the Sun and Moon align better next year. I wish the organisers, presenters, and the attendees a very successful learning and networking event!
Credits