I have been a DE for over 10 years working on SSIS and later building pipelines on databricks and dbt. Earlier this year I took up a new job as a manager of an analytics team. I hate my role and the lack of ownership and hands on work. I am primarily tasked with leading a team on reporting and data delivery and semantic layer while depending on the DE team for all data. I am also not enjoying middle management. Current job is a snowflake, dbt and AWS shop but I am my team primarily just write sql on snowflake. Recently I have been in conversations for a data architect role at another team in my organization and it’s a more engineering and architecture focused role which I want but they use Fabric. Wondering if that makes a difference. I am inclined to take the new role if it works out just so I can move out of analytics and go back to a more data engineering focused IC role.

Sometimes I use AI to write SQL faster, but sometimes I still prefer writing it myself just to make sure I haven't forgotten the basics.

AI can save a lot of time, especially with complex queries, but I feel like relying on it for everything can make you a bit rusty.

Curious how others feel about this. Do you still write SQL by hand for practice, or do you let your AI agent handle most of it now?

Hi, as the title says, I'm probably moving from a data developer in one company to an analytics engineer in another. Mostly using dbt, but I'm wondering if there are any major things I should factor in to my designs/thinking between GCP (Bigquery) and AWS (Redshift).

Stumbled upon these problems while doing the agentic pipeline

• CREATE TABLE DDLs are not enough. To understand what's in the table you need to have basic per-column stats and see examples of rows
• some tables (e.g., migrations, empty tables, old stuff, etc.) are useless for real analytics and real users, yet agent kept rediscovering them
• some tables only have less than 10 rows. Better see all rows instead of barebone column names and types
• some tables are huge, agent's queries were trying to get too much data. Good man won't query billions of rows
• agents needs to sample data from tables every time he researches a table
• if you have many columns, presenting data as a `transposed table` helps improve LLM's understanding

So I built two tools to solve these problems

• db-snooper: generates compact Markdown with schema, value distributions, and samples.
• schema-linker: finds possible undeclared joins and checks them against the data.

Full blog posts: db-snooper and schema-linker

When comparing source and target tables with millions of rows, what SQL techniques do you use to identify missing records and differences in column values?

I’m designing a self-service AWS sandbox environment for my organization. Employees should be able to request temporary, isolated AWS accounts for learning, testing, and prototyping.

Each sandbox should have:

-Strict spending and time limits

-Controlled IAM access

-Restrictions preventing access to production or other AWS accounts

-Protection against disabling logging and security controls

-Automated cleanup and account recycling

-Centralized monitoring and cost alerts

I’m considering using separate AWS Organizations member accounts and evaluating Innovation Sandbox on AWS. Has anyone deployed it in production, or used another open-source/project-based solution? I’d appreciate advice about security pitfalls, SCP design, budgeting limitations, account provisioning, and cleanup automation.

I've been a fan of this project for a while now, ngl. I thikn they've made some really smart design decisions with regard to where the space is headed. But they just released a distributed benchmark and it's better than every Spark accelerator out there... Previously it was Clickbench and some single node TPC-H benchmarks that showed really promising results (beyond ppl's personal benchmarking) but they just released a TPC-DS at 3TB and 1TB across 8 nodes. Is it time we actually move past the JVM and go fully Rust-native? I just don't see many reasons why not to run Sail when it's this much more performant, and runs python natively, and runs my pyspark and spark SQL code... of course having more years under its belt to be battle tested is important, but man the advantages of Rust, Datafusion, Arrow, all packaged together feels very good to run. And yea maintaining the Spark API is clutch...

Hey guys

I'm relatively new to Databricks but want to lead the charge to move to it in my company instead of just using ADF for everything.

In regards to Delta loads (loading data incrementally based on when it was last updated) what I would originally do is run a BINARY_CHECKSUM() on all columns and then track what has changed before bringing in the differences. This however is not supported in databricks and I want to do things properly so how would you do this?

Source Data (Azure SQL)

Bronze Layer (Azure SQL)

• What is different between them?
• What new rows have been inserted?
• What rows in Bronze no longer appear in Source

What I love about Databricks is that you can do joins of different data sets within the same activity in a notebook but when I tried to do this with manually comparing every single column, it takes forever to load. I just want to know the best way of doing this so any help or resources would be appreciated.

Many Thanks!

Hi r/dataengineering,

Wondering if anyone has had any experience navigating a similar situation or has any insights to offer.

My org has a central Data & Analytics department where all the Data Engineers/Analysts/Scientists sit under a technical engineering manager.

Each manager is given say 3 - 4 business areas (marketing, sales, ops, call centre etc..) to service. In each team, a team member is given a business area to manage and run with.

What essentially happens is, all the team members end up being more involved in the business rather than with the data team. From the business stakeholder point of view, this is great since the DE is embedded into the team and working day to day with the stakeholder.

However, from the Data side it feels like we're just a team by org chart and not how we work with each other. We end up going to work, sitting next to each other but barely collaborating since we're working on completely different things.

We can't even review each other's SQL code because we don't have enough context into other business areas to understand if the code is actually doing what it's intended to do.

Would appreciate any insight on this. Thankyou

After combing through many posts in this sub, and speaking to colleagues who has stood up airflow, the consensus is to use MWAA. The cost pays for itself in the required staff overhead needed to manage OSS.

Does that statement remain valid with the help of LLMs? Teams who have access to Claude can spin up the framework much quicker and debug much faster. The time spent troubleshooting an OSS setup should be smaller with the help of LLMs.

Can anyone speak to this personally?

I’ve programmed casually for about 10 years in a variety of languages, but I’ve only been doing it professionally (DE/AE) for around 3. I love this career. I used to enjoy the problem-solving of writing code myself, but what I really love is building: smart pipelines, better ways to collect metadata, simple apps that let the business interact with its data.

So naturally, AI has me feeling a sense of dread. The most senior dev on my team (mostly an app developer, 20+ years of experience) now believes AI can out-code him 99% of the time. He’s smart and I trust his judgment, which makes it more disappointing. He doesn’t even think we need to review code anymore, since GitHub Copilot will flag anything problematic.

On the learning side, I’ve read Fundamentals of Data Engineering, I’m partway through Designing Data-Intensive Applications, and I’ve learned Kimball’s dimensional modeling through video courses. Now I’m wondering:

• Is it still worth continuing this kind of deep learning?
• Does system-level and architectural knowledge still set you apart, or is that being automated too?
• Should I just learn alongside AI, questioning why it makes the choices it does?

What level of learning is actually worth it these days, and how are you staying ahead?

me irl

The data being received in csv or zip, after opening all the data is encrypted. I got to know that the file is pgp and its encrypted. Manager suggested using pyspark to decrypt the file. Is anyone aware of decrypting pgp files or pop in general.

What do we need to start things off, I requested about keys but he is yet to provide!

Would definitely love to learn from ur experience?

Update: got the key as .asc file. How to decrypt? Suggest some libraries that work with zip/csv with asc passphrase!

Hello, sorry if this subreddit isn't the place for noob questions.

I have some postgres tables with millions of rows that I'm moving into python for processing via psycopg2. I've been using cursor.fetchall/cursor.fetchone, and have discovered the copy_to/copy_from syntax.

It appears that copy_to is faster than fetchall: https://stackoverflow.com/a/68130908 , https://www.psycopg.org/docs/usage.html#using-copy-to-and-copy-from

Why? Is it that serializing the data takes time? If so, then does that mean if I ingest the data serially from the file on the python side then I lose any advantages?

"Use the Index Luke" is where I started my research into databases and the advantages of pipelining are repeated ad nauseam in that article, so doesn't that mean the data is already serial coming out of the query? It seems to me that "copy_to" would mean I force postgres to retrieve all my data, write it to the file, and then only start reading the data once its already written to the file, which is a complete lack of pipelining.

I understand that "copy_to is faster than fetchall" but it doesn't line up with my existing understanding of how query results are produced so something in my existing understanding must be wrong.

One of the things that got me into DuckDB was a series of technical articles posted to their site. This one was particularly eye opening:

Faster Dashboards with Multi-Column Approximate Sorting

I hadn't fully appreciated how important row order was to OLAP performance. In it they cover space filling curves as a method for mapping multiple columns (dimensions) onto a single "order dimension" to maintain proximity from the higher dimensions.

This works very well, but it means threading a curve though a set of potential values, at equal density, rather than what actually exists in the dataset.

Alternatively, by first approximating the rank of each value within its column (quantiles), and space-filling on that, the curve threads through the set of existing values. This handles skew a lot better.

This is a two-column ordering over the TPC-DS SF100 store_sales table, where ss_net_paid has reasonable skew. The rank-based methods manage to read far less of the table for a 1% range query.

I wrote up a more detailed post here:

https://www.duckstring.com/blog/clustering-for-pipelines

I'll probably be using this to set the row order whenever I overwrite a table and I have a reasonable idea as to what might be queried from it.

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension \[spark cell runner\](https://marketplace.visualstudio.com/items?itemName=PrashanthReddyMunagala.spark-cell-runner) that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated cluster

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github

Senior analytics engineer - spent last 5 years in FAANG mostly doing BI/DE work across small and large teams, and since last year i'm mostly doing fullstack for small internal multiservices app (data/stream intensive). WLB is ok but no career progression in current role.

Using AI tools extensively - both for internal purposes as well as for minor system integrations here and there, as well as maintaining knowledge repos enabling better collaboration using AI. Got cloud certifications, but no tool-specific certifications. Spent 2 years doing internships/ or unrelated corporate jobs before that.

I assume no career growth in my current role, but WLB is good and now I mostly work on software stuff (though i am still much much stronger in DE part), the only concerning thing is stagnation and comfort zone. Been contemplating a lot between:

- Rest & vest? ( I am fine with that)

- Sr. DE in scale-ups/startups?

- Attempt downgrading to some AI-company for a mid-DE role?

- Try to pivot internally to PMT / TPM job families? (Exciting, but concerned about the skill gap)

- Try to pivot internally into higher scope but more stress role? (No excited about that)

- Try to pivot into a more AI-parallel role? (Exciting, but I assume knowledge gap)

I want to get a better understanding of the market conditions? How realistic/unrealistic are these assuming some time to prepare? How hard is it to get interviews these days? How code-intensive vs design/architectural vs behavioral are they?

I had this conversation with one of our account manager but they seem to give me so many random answers on how it can be used.

Here are my doubts,

1. How do you define one semantic view, is it based on multiple ideas coming together into a unified view or multiple ideas into multiple views
   
2. After the change in orchestration, Agent tends to write its own SQL definition for the questions asked, how is it utilising the semantic view in this case?
   
3. Following up on the second question, how do verified queries work together. I have noticed it recognises the verified query but doesn’t want to use the definition which is correct.
   
4. Which model is being used for the agent to answer business-related questions?
   
5. Is it better to bring your own LLM and use cortex analysts ability to write code and let the LLM answer?

Let me know if you have found anything useful or good practice while building these views and agents. I want to understand it better.

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension spark cell runner that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github

What's your take on the new Claude Dashboard by Anthropic? I still think open-source BI-as-Code will win, and Claude Dashboards are the Jupyter Notebooks of the age of agents. Meaning they are easy to set up quickly, but impossible to manage or maintain in the long run (if needed). A decade ago we discussed no-code vs. code, now it's in-built in frontier models/apps vs. open source or code-first.

Hi Folks - PM from Databricks here and just wanted to share a useful technical blog on how to conduct unit tests with SDP Pipelines today! Here's the use case:

Nightmare scenario - Your new data pipeline updates look good before going to production - every table builds, every expectation passes, and the gold tables have plausible values in them. Looks good right?

But if you push to production, the dashboards would still be showing inflated numbers and an unhappy business partner will be knocking on your DM's

For cases like this you need to utilize Unit Tests to ensure your transformation logic is sound. Expectations help ensure your data looks right, but it still misses what your transformations are doing.

With unit tests, you directly validate your transformations on data inputs you control before the real tables are built. Testing edge cases and finding those annoying "silent" errors

Checkout the blog for a deep dive tutorial on how to use unit testing with Databricks SDP Pipelines!

I'm testing various semantic layers and would love to test a canonical example from MetricFlow. Looking for something that is more than a toy. Any suggestions?