Redpanda serverless streaming option targets cost control

New capabilities

Real-time decision-making is a requirement to navigate the current uncertain economic climate and repeated supply chain disruptions brought on by worldwide events in recent years.

In addition, real-time decision-making can be a competitive advantage for enterprises in an environment in which data alone is no longer a differentiator. That includes feeding AI models and applications with current data to keep them up to date.

The pace of business continues to accelerate, so expectations to see near-real-time data continue to grow.

As a result, the importance of streaming data is rising, according to Doug Henschen, an analyst at Constellation Research.

"The pace of business continues to accelerate, so expectations to see near-real-time data continue to grow," he said. "It's not every application that demands subsecond data latency, but use of streaming data platforms and streaming data analysis is becoming much more routine."

There are other ways to ingest data, such as processing batch files, but those don't enable the delivery of information in real time, Henschen continued.

"When latency expectations get down to the subsecond realm, batch and microbatch technologies no longer suffice," he said. "Transactional applications, in particular, demand streaming data speeds."

Stephen Catanzano, an analyst at TechTarget's Enterprise Strategy Group, similarly noted that the need for streaming data is on the rise.

"It is increasing in importance as more organizations want to feed real-time data into AI models to create fast and ongoing data insights for the organizations' decision-makers," he said. "It's a different source of data, but anyone in [the analytics] space believes that all data should be delivered as a stream in real time."

However, at the same time as streaming data is gaining importance, the cost of the cloud computing needed to run real-time analysis is expensive. Controlling the cost of cloud computing has therefore become an emphasis for many organizations.

Some technology vendors have responded as well by improving the efficiency of their tools to enable customers to consume less compute power when running their workloads. For example, database vendor Rockset recently made cost control the main focus of its January 2024 platform update, including autoscaling compute capabilities and improved CPU ratios.

Beyond improving the efficiency of their tools, another way vendors are attempting to help customers control costs is by introducing serverless versions of their platforms.

Serverless architectures still require the use of servers. However, they eliminate the need to manage infrastructures when creating and running applications and services. As a result, they have the potential to improve efficiency and save money.

Redpanda Serverless is designed for that purpose. The new version of the vendor's streaming data platform is designed to enable customers to scale up their use of Redpanda when their data workloads demand and scale down their use when workloads are lighter.

The result is better cost control, which is significant, according to Catanzano.

Not only will it enable existing Redpanda customers to better manage cloud spending, but it could also enable those customers to expand use without incurring increased costs. In addition, it could make the vendor's streaming data platform accessible to a new audience of smaller companies.

"The big news is that their serverless platform is now available as a pay-as-you-go offering, which enables organizations to scale up as needed," Catanzano said. "It may open the door for smaller companies and new workloads [by existing users] such as AI implementations."

Henschen likewise noted that the primary benefits of serverless tools such as Redpanda's new release are more efficient use of compute power and the resulting cost control.

"Serverless capabilities are increasingly in demand because they support elastic scaling, eliminating administrative tasks and downtime associated with adding or reducing compute and storage capacity," he said. "Serverless capabilities also reduce costs in cases where workloads are spiky and capacity can be shut down automatically when not in use."

In addition to Redpanda, data management and analytics vendors that have recently introduced serverless options include AWS, which unveiled a series of serverless tools in November 2023, and vector database specialist Pinecone.

Meanwhile, as part of Redpanda Serverless, developers will have access to the streaming data vendor's ecosystem of partnerships and integrations to customize applications.

For event-driven application design, Redpanda customers can use platforms such as AWS Lambda. Stream processing and analytics engines that are part of Redpanda's network include DeltaStream and MongoDB's Atlas Stream Processing. And among the databases and real-time analytics pipelines that Redpanda works with are QuestDB and SingleStore.

Next steps

Despite the general availability of Redpanda Serverless, cost control remains an appropriate area of focus for the streaming data vendor, according to Catanzano. By improving the efficiency of its existing capabilities and finding other means of helping customers predict and manage spending, Redpanda has the potential to broaden its audience.

"The adoption [of serverless] seems to be focused on reducing latency and the costs of streaming across the network," Catanzano said. "If they can continue to tackle this, their adoption rate should continue to grow."

Henschen, meanwhile, said that despite developing an ecosystem of partnerships and integrations, the streaming data vendor could still add more to provide customers with choice as they select the platforms they want to use as part of their data operations. In particular, Redpanda could align more closely with Microsoft Azure to add to its relationships with AWS and Google Cloud.

"Redpanda collaborates with a broad ecosystem of vendors, and it's available on both AWS and Google Cloud, but more partners and availability on Microsoft Azure as well couldn't hurt," Henschen said. "It also never hurts independent vendors to have formal joint solutions and sales partnerships with big mainstream tech vendors and cloud providers, but those relationships are hard to win."

Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.