Stumbled upon these problems while doing the agentic pipeline

• CREATE TABLE DDLs are not enough. To understand what's in the table you need to have basic per-column stats and see examples of rows
• some tables (e.g., migrations, empty tables, old stuff, etc.) are useless for real analytics and real users, yet agent kept rediscovering them
• some tables only have less than 10 rows. Better see all rows instead of barebone column names and types
• some tables are huge, agent's queries were trying to get too much data. Good man won't query billions of rows
• agents needs to sample data from tables every time he researches a table
• if you have many columns, presenting data as a `transposed table` helps improve LLM's understanding

So I built two tools to solve these problems

• db-snooper: generates compact Markdown with schema, value distributions, and samples.
• schema-linker: finds possible undeclared joins and checks them against the data.

Full blog posts: db-snooper and schema-linker

I have been a DE for over 10 years working on SSIS and later building pipelines on databricks and dbt. Earlier this year I took up a new job as a manager of an analytics team. I hate my role and the lack of ownership and hands on work. I am primarily tasked with leading a team on reporting and data delivery and semantic layer while depending on the DE team for all data. I am also not enjoying middle management. Current job is a snowflake, dbt and AWS shop but I am my team primarily just write sql on snowflake. Recently I have been in conversations for a data architect role at another team in my organization and it’s a more engineering and architecture focused role which I want but they use Fabric. Wondering if that makes a difference. I am inclined to take the new role if it works out just so I can move out of analytics and go back to a more data engineering focused IC role.

Sometimes I use AI to write SQL faster, but sometimes I still prefer writing it myself just to make sure I haven't forgotten the basics.

AI can save a lot of time, especially with complex queries, but I feel like relying on it for everything can make you a bit rusty.

Curious how others feel about this. Do you still write SQL by hand for practice, or do you let your AI agent handle most of it now?

Hi, as the title says, I'm probably moving from a data developer in one company to an analytics engineer in another. Mostly using dbt, but I'm wondering if there are any major things I should factor in to my designs/thinking between GCP (Bigquery) and AWS (Redshift).