When the worlds of language and numbers converge: Considerations for building with LLMs 

For the past 70 years, the world of software was one of comprehending and analyzing numbers. 

Early ML native applications required extremely sophisticated teams and mountains of clean, highly structured data. They could consume tables, join data, and perform complex numerical analyses. However, akin to a train running on a single track, it was difficult to deviate from a narrowly intended range of motion. Systems were brittle, needing precise connection through APIs. Even quantitative data was of limited utility if not highly structured. Applications could not digest unstructured data such as text, images, audio and video – or express complex ideas that require their usage. 

Enter Large Language Models. With the rise of GPUs, cloud computing, and innovation in model architectures, LLMs can ingest and analyze multimodal data (i.e., pictures, videos, voice) to produce integrated responses. They do this at scale while accessing a corpus of knowledge across the web or relevant information within a company’s firewall.  

We expect LLMs to increase the surface area and value of the software industry 

In this piece, we outline some characteristics of AI applications:

Applications can live in the space between structured and unstructured data for the first time.

Multimodal data (e.g., pictures, videos, voice) can build on and amplify the value of structured data. Integrating such data with traditional structured data enables software companies and leveraging the power of LLMs to “reason” on the expanded corpus enables software to radically expand scope and produce outputs at near submission-ready state (though we still *strongly* recommend a human review cycle.) 

Multi-modal ingest can massively improve specialized industry applications 

First wave LLM applications emerged with key word aggregation, synthesis and output capabilities. One example is Hebbia’s tool, Matrix, that automates much of a financial analysts’ rote work by surfacing key investment insights from SEC filings, public analyst reports and other written documents.

As models get better, the next tide of applications will connect relationships across diverse data types such as voice, text, and images. Healthcare diagnosis, management, and research is a compelling case where unstructured data (clinical notes, medical imaging) can account for up to 80% of a patient’s medical record (National Library of Medicine).

Several non-clinical AI applications are tackling the enormous administrative burdens of healthcare, such as prior authorization (Anterior, Develop, Latent), coding, billing, and utilization. Akasa accelerates prior auth mentioned above, as well as automates claim attachment and submission to payer portals. Zentist’s Revenue Cycle Management AI solution can fast track the 8+ hours a week Dental Service Organizations spend on manually managing claims.

LLMs can also aid in clinical decisioning support by surfacing medical research to develop diagnoses, while still giving the clinician authority over to accept, edit, or reject the LLM output (Glass Health, Carenostics). AI can analyze large volumes of unstructured molecular, clinical, and genomic data to provide us a more granular level of understanding of the bodies’ response to therapy (Tempus Health). Healthcare companies with ambitions to train proprietary models that use new types of data seek previously untapped health system datasets. Startups are tackling next generation healthcare data businesses, marketplaces (Segmed, Protege) and platforms that speed-up backend processes like cleaning, harmonization and error audit.

Humans no longer need to manually create comprehensive deliverables bit by bit, they review AI outputs

Dashboards, drafts, and other outputs historically required many cycles of human input. Lawyers metabolized hundreds of long documents; plan reviewers surfaced thousands of codes across changing codebooks. Now, AI applications can automate a large percentage of that workflow. Humans add value by shaping, validating, and updating AI outputs.

Companies can reduce time and dependence on manual processing, saving cognitive load and freeing workers to focus on higher value-added processes. EvenUp software learns from a dataset of 250K+ public verdicts and private settlements to auto generate entire demand packages for industry lawyers; Vic.ai uses optical character recognition and natural language processing to cut through hours of accounting processes – even submitting invoices for approval. With TrunkTools search, construction workers ask questions about specifics within project docs such as contract details, what type of home fixtures are present within drawings, and get comprehensive answers back, saving time spent continuously searching and rework. 

Human-in-the-loop is still required for most solutions today, but as LLMs improve, feedback on quality and desirability of model outputs via in-context learning will guide the learning process. Human interaction creates a data flywheel that trains the model and deepens a moat of proprietary data advantages. 

Early-stage learnings on building with LLMs

Here are some key implications for builders, based on what we’ve experienced in successful AI startups to date:

1. Search for business processes that are very expensive, often with high human processing costs doing relatively rote or repetitive work
2. Look for places where there are significant multimodal inputs and outputs 
3. Focus on the most important and hardest tasks first. Break down the workflow into smaller chunks. Assess whether you can truly solve the problem.
4. Determine what solution is best suited for your problem. LLMs and machine learning models may not be appropriate. Take time to determine if ML solutions are better-suited.
5. Experiment judiciously with different models and approaches. Measure outcomes – and even costs – systematically so you can tune and improve.
6. Treat model choices as architectural decisions where you’re looking for bottlenecks of performance such as accuracy and latency
7. Continue to look for additional or better structured/labeled data to improve your process. Good data is nearly as important as picking the right model. Note that this might affect model or architectural choices down the road.
8. Expand and automate advanced pieces of workflow bit by bit. Note that each time you add a module or capability you need to repeat steps 1 through 5. 
9. Grab opportunities to innovate on user experience. LLM capabilities enable much more complex and complete outputs.
   1. Different types of outputs can include visual as well as textual or quantitative information e.g. images, text, chats, even maps. Innovate continuously on how best to represent data while enabling users to consume and edit/modify the output as part of their workflow.
   2. LLM chatbots are familiar on the consumer-facing end, but organizations can also leverage LLMs to power sophisticated backend processes.
10. Structure the business to benefit from ongoing model improvements, recognizing that today’s “impossible” problems may be far easier feats in the future.

We’ve already made a dozen investments in and around AI applications. We’re on the lookout for more. If that’s you, let’s chat!

Written by Greg Sands and Nicole Seah. Special thanks to Yang W. (Co-founder of Tensec), and part of the Costanoa Team: Rebecca Li, Amy Cheetham, Martina Lauchengco and Taylor Bernal for their contributions to this piece.