It’s been a busy spring and summer here at Dell. In March, on the first day of spring, we launched the Dell Data Lakehouse. This turnkey solution features a powerful query engine powered by Starburst, which provides high-performance, high-concurrency access to distributed data, regardless of the data source. The foundation of Dell Data Lakehouse is built on Dell S3-compatible storage providing a highly performant, high availability storage layer to store and query data in open formats like Iceberg.
In today’s fast-paced world, IT and data leaders face a tough challenge, accelerating analytics and AI while keeping costs in check. The tradeoff between speed and cost can prove expensive on either end. While adopting the data lakehouse offers performance advantages at lower costs, data engineering and IT teams still grapple with deciding which data to optimise and cache and which to leave as is in the data lake.
Consider this feedback from data teams:
“I’m the data engineer, and the analysts keep asking me to make copies and re-partition data (by date, by customer) when they have a new question so that response times are tolerable. It’s a waste of my time and storage isn’t free!”
“I’m the Data Leader for the data org, and I can’t just keep throwing money at more and bigger clusters to meet my query response time SLAs! Data sizes are growing faster than my budget.”
This is why we’re thrilled to introduce Warp Speed in the Dell Data Lakehouse for data on Dell’s S3-compatible storage!
What is Warp Speed?
Warp Speed is a new feature in the Dell Data Lakehouse that autonomously learns query patterns and identifies frequently accessed data to create optimal indexes and caches while keeping infrequently accessed data where it is.
What does it do?
It delivers on the promise of accelerating query performance while keeping costs in the check. With Warp Speed, the same cluster can run data lake queries 3x to 5x faster without requiring any change in the query by the end user. It can also help reduce cluster sizes by up to 40%.
More simply put, organisations can run more queries on large clusters or run the same volume of queries on smaller clusters.
- Accelerating data lakes: Autonomously index the data lake and on-demand accelerate exploratory datasets without involving data engineering.
- Building high-performance dashboards: Faster drill-down on TBs to PBs of data, without any change to the end user experience. The same queries now just run faster.
How does it do that?
Warp Speed employs a combination of acceleration technologies to achieve its remarkable performance:
- Autonomous Indexing: This feature creates appropriate index types (bitmap, dictionary, tree) tailored to each data block, accelerating operations such as joins, filters, and searches. Indexes are stored on an SSD in the compute nodes for rapid access.
- Smart Caching: Smart caching is a proprietary SSD columnar block caching that optimises performance based on frequency of data usage. Caching eliminates unnecessary table scanning and provides more reuse of data between queries thus saving compute costs.
How do I get it?
Warp Speed is available to all Dell Data Lakehouse customers and supported for those who are using Dell S3-compatible storage as their data lake. There is no change to the software license – this is now built-in! The configuration of the compute nodes will be modified to include SSDs that have been fully tested and benchmarked by Dell to support the Warp Speed index and cache.
Accelerated Innovation in the AI Era
Dell Data Lakehouse with Warp Speed sets a new benchmark in data lake analytics, empowering organisations to derive insights from their data more quickly and efficiently than ever before. Warp Speed unlocks the full potential of the Dell Data Lakehouse, paving the way for accelerated and budget-friendly innovation and growth in the AI era.
To get a full, hands-on experience, visit the Dell Demo Centre to interactively explore the Dell Data Lakehouse with labs hand-picked for you by Dell Technologies’ experts.