Duck DB

DuckDB is an amazing tool. Even with tens millions of records you can create advanced analysis in seconds or less of without setting up a database server using fairly straightforward SQL commands.

It’s a self contained application (no libraries required!), though you can write apps that use it in a wide variety of languages like C, Rust, Java, Python and R that either use it statically or dynamically linked.

I’ve been mostly using the C interface with it, but for most data manipulation you can just create lists of SQL commands to run an pipe it into the duckdb app – in my testing it’s slightly faster with then the C though granted I played with relatively simple apps and I didn’t optimize my build or exclude debugger symbols and used dynamic linking.