Venture Bytes #104: LLM Orchestrators Enhancing AI Models

LLM Orchestrators Enhancing AI Models

‍

Generative AI has become a competitive and operational imperative for businesses, but  companies  struggle  with  scale, production, and expertise. Additionally, businesses want to benefit from model- agnostic approaches to get the best of multiple models – an approach that adds another layer of complexity. Large language models (LLM) orchestration is the answer.

LLMs  are  becoming  increasingly commoditized with over 30 LLMs now in the arena including open-source models. Market  dynamics  have  dramatically shifted, as evidenced by an Arize survey indicating OpenAI’s market share in LLM adoption plummeting from 83% in April 2023 to just 13% six months later, amidst rising alternatives. This market realignment underscores the burgeoning competition, and shifts focus beyond model capabilities to the robustness of the infrastructure that supports them.

LLM  orchestration  is  essentially  the process  of  managing  and  controlling LLMs  to  maximize  their  performance and impact. Organizations are grappling with  the  complexities  of  effectively leveraging LLMs, which involves selecting optimal models, unifying various LLMs into integrated services, and deploying applications in a cost-efficient manner. Addressing these challenges and improving AI workflows requires combining several AI models, algorithms, and tools into a single framework. Orchestration startups span a broad spectrum, offering services from  model  refinement  to  application rollout and computational acceleration. AI Orchestration Market size is likely to reach $35.2 billion in 2030 from $6.9 billion in 2022, g      2.5 % CAGR, per SNS Insider.

AI Infrastructure Opportunity Is Enormous

The future of AI promises complex applications, and the bedrock for this progress is a robust infrastructure, which stands as a vast and vital opportunity. The trajectory for AI applications leans heavily towards leveraging fine-tuned foundation models, tailored through additional data or parameter adjustments for specific use cases. This shift emphasizes the infrastructure’s role over mere model capabilities, highlighting a burgeoning market for AI infrastructure that is rapidly expanding and becoming increasingly complex.

The majority of enterprises prefer to operate their LLMs over external API providers, driven by considerations of cost, latency, and data privacy. For example, OpenAI’s pricing model requires 12 cents for every 1000 tokens (around 700 words) for a fine- tuned model on Davinci. Conversely, by leveraging a combination of HuggingFace, DeepSpeed, and Ray, companies can build their own fine-tuning and serving systems for LLMs. This approach not only proves to be cost-efficient—under $7 for tuning a 6 billion parameter model in just 40 minutes—but also addresses the speed bottleneck. External LLMs like GPT-3.5 can take up to 30 seconds per query, which is impractical for many applications. By internalizing the process, companies can significantly enhance performance, often achieving latency improvements of 5x or more, tailored specifically to their application needs.

Promising Startups in LLM Orchestration

Startups such as Anyscale, Run.ai, and Pinecone are making significant strides in the LLM orchestration area. California-based Anyscale, which was valued at $1 billion in its $199 million Series C round, eases the scaling of AI applications through its Ray framework. Leading AI organizations such as OpenAI, Cohere and EleutherAI use Anyscale’s Ray to train LLMs at scale. With Anyscale’s fine-tuning capabilities, the Llama-2-7B model has demonstrated task-specific performances reaching 86%, eclipsing G  4’s 78% and doing so at a fraction of the cost—merely 1/300th.

Run:ai another Series C company, assists organizations in advancing their AI projects economically by virtualizing high-cost hardware resources. This enables the pooling, sharing, and strategic allocation of these resources. The company provides an array of solutions, including GPU optimization, cluster management, and the orchestration of artificial intelligence and machine learning workflows.

Pinecone, which raised a $100 million Series B round in April 2023 at a $750 million valuation, also holds a crucial role in facilitating large-scale similarity searches. This capability is key for LLMs that rely on contextual and semantic understanding for information retrieval. Pinecone’s offerings enhance LLMs’ ability to process and analyze extensive data sets efficiently and accurately.**

‍

Extended Reality Gets Much Needed Boost

Will Apple’s Vision Pro do to the extended reality category what Apple’s iPhone did to smartphones? The Vision Pro’s launch and its sell-out in a mere 18 minutes are a testament to the market’s eagerness to step into new dimensions of digital experiences. With competitors such as Google, Meta, Sony, and Microsoft in the VR headset market, Apple aims to redefine the landscape by pushing the boundaries of the use cases for the device, which may become a catalyst for an industry-wide revolution.

We know Apple did not invent smartphones, but the iPhone turbocharged the smartphone revolution by introducing innovative use cases. The ecosystem created by Apple, directly and indirectly, created immense business opportunities for app developers, smartphone accessories developers, and other smartphone manufacturers that offered a more economical device with similar features. Similarly, is Apple’s Vision Pro poised to redefine VR headset applications - something Meta’s Quest 3 is struggling to do - creating substantial business opportunities for startups in this burgeoning space? We think so.

Vision Pro transcends being a mere product; it stands as a pivotal platform, intrinsic to Apple’s prospective expansion in extended reality, elevating its existing product ecosystem. Apple’s emphasis on designing products for widespread adoption set it apart from the competition’s priority of being fast and first. The market entry of such a company, boasting a unified product portfolio, an expansive user base, and unrivalled brand trust, carries substantial weight. The repercussions will include technologists intensifying their efforts, markets responding dynamically, investors regaining confidence, and strategic acquisitions coming to fruition. It serves as a resounding signal of confidence, igniting a renewed surge of innovation in the extended reality industry. Notably, the extended reality industry is projected to grow from $40.1 billion in 2023 to $111.5 billion in 2028, at a CAGR of 22.7%, per MarketsandMarkets.

‍

At $3,499 the Vision Pro is significantly more expensive than the competition, but the device is more than a gaming accessory, it is a device meant for spatial computing with the potential to replace gaming consoles, televisions, tablet PCs, Laptops, and more. Drawing a parallel from the past, the first version of the iPhone had 15 inbuilt apps while the Appstore was launched with 500 apps in 2008. In contrast, Vision Pro has 600 dedicated spatial apps, 20% higher than the 2008 Appstore. Additionally, the visionOS ensures seamless compatibility with existing iPad and iPhone apps, therefore most apps that work on iPad and iPhone will also work on Vision Pro. However, dedicated versions of apps for VR headsets would be paramount for the success of the device.

Beyond mere entertainment, these VR headsets have the potential to redefine corporate meetings, foster interactivity, and elevate overall workplace productivity. Potential collaborations with Edtech firms are positioned to reshape the educational landscape through immersive 3D interactive content. Moreover, VR headsets are primed to deliver real-time expert guidance in healthcare, industrial settings, training, education, and flight simulation, among various other applications. Such versatile applications create a substantial total addressable market of over $1 trillion, VR analysis, signifying vast opportunities across industries.

‍

As Apple spearheads innovation, more economical alternatives are anticipated to thrive, including promising startups such as Magic Leap, Real Wear, and Varjo:

• Magic Leap’s VR headset and its collaboration with NVIDIA for further enhancements is finding many takers, the most prominent being Audi to create an artificial reality-based driver interface.
• Real Wear, offering VR headsets to industrial workers for document navigation, virtual assistance, remote expert guidance, and industrial IoT data, are poised to become highly valuable companies soon. In 2023, Real Wear won ‘Best Manufacturing and Industrial Solution’ at the XR Today Awards, underscoring its credibility.
• Varjo, offers VR headsets for applications in flight simulation, medical expert guidance, education, and more, exemplifying the diverse market. Recent adoptions of Varjo VR headsets by BAE Systems, a defence tech company, Leonardo, a military pilot training company, and MachineMD, a medical device company, exemplify the diverse use cases.

‍