Welcome to FastLanes

We’re excited to announce the launch of FastLanes, a next-generation big-data file format designed for modern hardware, including SIMD and GPUs. FastLanes enables high-throughput, low-latency data access for analytical and AI workloads.

What is FastLanes?

FastLanes originated during my PhD research at CWI in the Database Architectures (DA) group and continues to evolve in my postdoctoral work, with a focus on GPU decoding and AI workloads.

FastLanes introduces a new compression layout that supports parallel decoding at absolutely crazy speed—over 100 billion integers per second with scalar code¹. It’s inspired by the concept of lanes, where data is organized in separate lanes to minimize dependencies and maximize independent—and at the same time, same—tasks, perfect for SIMD and GPU data parallelism.

Why FastLanes?

The state-of-the-art file format, Parquet, struggles to keep up with modern processors and accelerators. As compute capabilities scale, especially on GPUs and SIMDized CPUs, there’s more opportunity to decode data faster. FastLanes addresses this by:

• Aligning with hardware: Designed for SIMD units, multicore CPUs, and GPUs.
• Enabling extreme parallelism: Each lane is independently decodable.
• Reducing latency: Optimized for low-overhead, high-speed decompression.
• Supporting analytics and AI: Tailored for workloads involving large-scale numerical data.

What’s Next?

FastLanes is open-source and actively maintained. We’re working on:

• GPU-accelerated decoding
• Interoperability with Apache Arrow and DuckDB

Stay tuned for updates, performance numbers, and deep dives into the internals.

Get Involved

We invite the community to explore, use, and contribute to FastLanes. Join the GitHub project, star the repository, and hop into the FastLanes Discord to discuss ideas and provide feedback.

Let’s build the future of data formats—together.