Introducing sqlleaf: column-level lineage for SQL queries

[u/juppets]

https://github.com/sqlleaf-io/sqlleaf

sqlleaf is a tool to help you to understand how your data flows across your systems by producing detailed, column-level lineage across any SQL query and dialect.

It extends beyond traditional lineage tools by mapping relationships across all kinds of expressions, such as columns, functions, literals and variables.

It addition, it can process many kinds of statements (insert, update, merge, copy, unload, execute, perform, put, and many more), as well as all kinds of objects (tables, views, procedures, types, UDFs, stages, sequences, etc).

It even has a PL/pgSQL parser, allowing you to generate lineage for dynamic, procedural SQL in Postgres.

sqlleaf builds on top of sqlglot, the popular SQL transpiler, and so it inherits support for all of its dialects (30+). However, there are still many dialect-specific features that need to be implemented, as the list of features for each dialect is extremely long.

I hope some of you find this project useful. It still has a long way to go. My vision is to see it become a library, or even a layer, in the data engineering stack that opens up a range of new applications that can be created, such as SQL validators, impact analysis, and tracing/auditing systems.

URL: https://github.com/sqlleaf-io/sqlleaf

Contributions and feedback are most welcome!

P.S. why I made this: I recently had to trace and delete sensitive data across an enterprise's large data warehouse, but the tooling that existed (both open source and proprietary) had limitations that prevented my team from doing so. After months of experimentation, I eventually realized that I had no choice but to roll my own.