​

I am m curious how different teams handle this, especially when a pipeline has multiple transformation steps and some of them have already completed. How do you avoid duplicate records or inconsistent data when rerunning..

We've been running nightly ETL jobs for years without many complaints, but now the ML team wants data that's available almost immediately for fraud detection and recommendations models.

I'm trying to figure out whether it makes more sense to keep shrinking our batch windows or introduce a separate streaming layer for the workloads that actually need it.

If you've added **real-time AI** capabilities to an existing ETL stack, what ended up working?

jobdataapi.com v4.38 / API version 1.36

We’ve improved job description handling to preserve meaningful technical terms such as C#, F# and snake_case more reliably as new listings are processed. Markdown descriptions retain readable formatting for display and provide a useful starting point for search, analysis and other downstream processing. Request description_md=true to include the description_md field in job responses.

After November 1, 2026, the description_str parameter and its corresponding description_string response field will be removed. If your integration uses either, switch to description_md=true before then and convert Markdown to plain text in your application if needed. The default HTML description remains available.

I have an ETL job that runs successfully every day, with no errors. But sometimes, the target table has fewer records than the source.

The job status is green, and the column mappings look correct. I’m not sure whether the issue is with filtering, incremental loading, or duplicate handling.

How would you troubleshoot this? What checks would you perform first to find where the records are missing?

What are you actually building on day-to-day? Are you running hybrid models (mainframes/on-prem relational DBs syncing to cloud data lakes), or are legacy constraints forcing you to stick strictly to on-prem stacks (like Hadoop, Oracle/SQL Server, or traditional ETL tools like Informatica/DataStage)?

A follow up article from my jev fact extraction pipeline where we take the classified prose segments and convert them to structured JSON using purpose built LLMs.

Source code: https://github.com/s0cks/prose2json

~ Tazz

I maintain Scriptella, an open-source Java ETL tool, and wanted to establish a reproducible performance baseline for CSV imports.

I compared 11 import configurations using PostgreSQL and MariaDB, with 1 million rows per test.

Some results from the latest CI run:

Native COPY was fastest for PostgreSQL, as expected. Handwritten JDBC also outperformed Scriptella, while both were substantially faster than the tested csvsql workflow.

These are end-to-end timings, not isolated framework overhead measurements. The comparison uses one workload and one CI run, with different import mechanisms.

I've published the source code, Docker Compose setup, raw measurements, charts, and reproduction instructions:

https://github.com/scriptella/scriptella-benchmarks

I'd be interested in feedback from people working with ETL pipelines, particularly on which additional workloads would make this comparison more useful.

Got a slow Spark pipeline with two very believable explanations. The code has grown for years and has joins/transforms that probably deserve a hard look. But data volume has also gone up a lot while the underlying setup has basically stayed put.

A rewrite might pay off, but it is risky and eats engineering time we do not really have. Changing the compute path is quicker to test, but it will not magically save a bad data model. Both arguments sound sensible on their own, which is annoying.

How do you call this? What tells you the job design is the problem versus the infrastructure just being tapped out..

I use Grafana OSS for dashboards, I want to maintain historical dashboard data for archive purposes, beyond our data source's (Prometheus) retention period.

Mimir and Thanos feel like overkill for this. Add infrastructure, storage costs, but all i need is to look back at historical data.

Any lighter approaches that is working for you?

One thing I've noticed in ETL discussions is how often people talk about rising bills, unexpected usage costs, and the time spent troubleshooting pipelines when something goes wrong.

The pricing can look fine when you're setting things up, but once the data volume grows, you need to understand what you're actually being charged for. And when a pipeline breaks in production, getting useful help matters a lot more than it did during setup.

Full disclosure, I work at Hevo, so I'm obviously biased. But I don't see it mentioned much in these discussions, and I thought it might be useful for people comparing tools.

A couple of things worth knowing about Hevo:

• Pricing: It's consumption-based, using credits tied to data events. Historical data loading is free, and the pricing page has a calculator to estimate costs based on your usage.
• Support: live chat on the Free plan to priority 24x7 support on small companies or big enterprise.
• Pipeline troubleshooting: You can monitor pipelines, inspect failed events, and replay events after fixing the underlying issue.

It's not automatically the cheapest option for every workload, and usage still matters. I'd recommend checking the actual numbers for your data volume and looking at the support coverage you'd get before comparing it with another tool.

Here's the pricing page for anyone who wants to look at the details:

Hevo pricing and plans

Curious what matters more when you're evaluating an ETL tool: keeping the bill under control, or knowing you can get help when a production pipeline breaks?
Also would love to understand what other problems you guys face, maybe Hevo has a solution?

Friends,

I am working on improving some of my substack's publication pipeline and figured I would share with you some of the work I have been doing.

To provide some reading aides to my substack readers I created a tool for generating clean information dense infographic cards from Markdown, JSON, YAML and Jsonnet.

Then I thought about ways to automate the process of generating them, which led me to try experimenting w/ Jev.

I wanted something that was deterministic, reproducible, auditible and easy to create.

Jev's role here is to classify fact from long-form prose and rank the facts so i can extract them and present them in the infographic cards.

You can find the infographic compiler on my GitHub here: https://github.com/s0cks/docker-chromium-screenshot

And you can find the jev (tev1) classifier discussed in this article here: https://github.com/s0cks/tev1-datapoint-extractor

They are slightly vibe coded with Claude so I can experiment with the idea quickly.

I plan on more updates just figured folks might be interested.

I also have plans to adapt some of this work into my prose -> audiobook pipeline down the road, and maybe eventually fine tune a jev model specifically for this kind of classification.

Thoughts, questions, comments, concerns are welcome.

Best,

~ Tazz

how to handle every file format during ingestion without rewriting the entire pipeline.

After combing through many posts in this sub, and speaking to colleagues who has stood up airflow, the consensus is to use MWAA. The cost pays for itself in the required staff overhead needed to manage OSS.

Does that statement remain valid with the help of LLMs? Teams who have access to Claude can spin up the framework much quicker and debug much faster. The time spent troubleshooting an OSS setup should be smaller with the help of LLMs.

Can anyone speak to this personally?

we've reached the point where our python scripts are causing more problems than they're solving

everything lands in snowflake eventually, but bw scheduled CSV exports and snowpipe we've accepted more latency than the business wants, oracle is the main source, so i'm looking for data integration tools that handle the full CDC pipeline instead of stitching together separate pieces.

what are people happy with these days?

I've been thinking about a problem with sources that only provide periodic snapshots instead of CDC or a reliable updated_at field.

If a record is missing from today's extract, it could mean the record was deleted. But it could also be an incomplete export or a pagination issue. Treating every missing record as a delete can mess up the data downstream.

How do you usually handle this? Do you compare snapshots, wait until a record is missing from multiple extracts, or use another approach before marking it as deleted?

what has worked well for people running these pipelines in production??

I’m new to data engineering and have mostly worked in analytics engineering. I’m curious about how streaming pipelines work in practice, especially with Kafka, Flink, and Iceberg.

I’d love to hear from people who’ve built or operated these systems: What concepts should I learn first? How do you handle watermarks and late-arriving events, quarantine bad data, and manage schema evolution or unexpected schema drift?

Any practical examples, learning resources, or advice on what’s most useful to understand would be appreciated.