if you've ever sent out a number that was wrong because of a query that looked fine, i'd like to hear what the query was. collecting these to turn into puzzles.

im wokring TsundereSQL, a visual novel where the answers are real SQL, graded on the sample data plus a few hidden sets using duckdb.

here's the one in the demo right now. page 2 of a december BTC trade list, ranked by price * quantity (quote currencies are mixed, which a real desk would yell about):

SELECT trade_id,
  CASE WHEN side = 'buy' THEN 'BUY FLOW' ELSE 'SELL FLOW' END AS flow,
  ROUND(price * quantity, 2) AS notional_value,
  STRFTIME(executed_at, '%Y-%m-%d') AS executed_on
FROM trades
WHERE base_asset = 'BTC'
  AND executed_at BETWEEN '2025-12-01' AND '2025-12-31'
  AND price * quantity > 50000
ORDER BY notional_value DESC, trade_id
LIMIT 10 OFFSET 10

passes the sample and two of the hidden sets. the third has a trade at dec 31 17:36. executed_at is a timestamp, so DuckDB casts '2025-12-31' to 2025-12-31 00:00:00 and that trade falls outside the range. it was the 4th biggest of the month. everything below it moves up one and page 2 comes back with 2 rows instead of 3, and the missing row is a dec 11 trade that got pushed onto page 1, so the two rows you do get look fine.

fix is a half-open range: executed_at >= '2025-12-01' AND executed_at < '2026-01-01'.

anything that can be checked by comparing result sets has a shot at becoming a level.

https://store.steampowered.com/app/5213730/

Hey guys,
I'm a data engineer, and I have been building pipelines for small to mid sized clients (with less than 1 terabytes of data) and most of the time the workload isn't really worth paying the premium for Saas products like Snowflake, Databricks .. etc, and can be done on a single beefy machine but stitching open sources tools usually comes with maintenance, networking and configuration headache, also these tools aren't unified under one control plane and don't know that other tools exists.
so I built a data platform that can be deployed on a single machine or server, and since I'm Egyptian I called it Obelisk 😄
Under the hood Obelisk uses:

1. DuckDB, so you can do queries that joins multiple different sources in the same query (local or remote lakehouse, dbs, erp systems, mongodb .. etc)
2. the platform also act as a daemon manager, so you can deploy tools you need with a single click and they will integrate directly into the platform (or you can connect remote instances)
3. for variables and connection strings, you just add them to the platform once and every tool can access them.
4. built-in support for cronjobs and advanced orchasteration tools like dagster or airflow
5. a browser based dbt studio
6. reactive notebooks
7. I also added file manager and terminal access to the machine, so you don't need to SSH directly.

GitHub Repo: https://github.com/amrelsawalhi/obelisk

this isn't a finished product, I still have a lot of ideas to implement like auth and access control for teams
I’d love to get your feedback on the platform and contributions are most welcome.

My background includes data science, machine learning and AI, and I've also worked directly in customer support.

I'm interested in how data analysis can help support teams improve their operations, rather than just create dashboards that look good.

Imagine a support team has thousands of tickets containing categories, timestamps, resolution outcomes, customer feedback, and escalation histories.

I'd want to answer questions such as:

• Which issue categories generate the most tickets?
• Which types of cases take the longest to resolve?
• Which issues are most likely to be escalated?
• Are customers repeatedly contacting support about the same problems?
• Can text analysis help identify recurring complaints that aren't captured by existing categories?

I'd probably begin with data cleaning, exploratory analysis, and a few clear visualizations before considering more advanced NLP or machine learning.

I'm curious how other analysts would approach this problem.

What would your first three analyses be, and which findings do you think would be most useful to a support operations manager?

I'm particularly interested in practical analysis that can lead to an operational improvement.

a week ago, I've been asking on reddit on how other B2B SaaS teams moved away from signed Metabase iframes once they hit multi-tenant role permissions, slow morning dashboard load times also escalating Snowflake compute bills and since then, I've noticed nearly four distinct paths to investigate (i’ve formatted the spreadsheet itself with ai to keep it readable enough):

1. staying with Metabase or Apache Superset, but placing a dedicated caching tier or materialized rollups in front of the database.
2. Moving to a dedicated spreadsheet-first or workbook-centric embedded BI platform like Sigma or Omni.
3. Decoupling the frontend completely using a headless semantic layer (specifically testing Cube Core and Cube Cloud) to serve pre-aggregations into our own React components while keeping an optional embedded builder for enterprise users.
4. Writing custom dbt rollups into ClickHouse or DuckDB and maintaining our own FastAPI aggregation endpoints.

so I also spent the last two weeks running a sandbox bake-off with an anonymized slice of our Snowflake production dataset. below I've placed my exact test setup, the comparison spreadsheet + where each option actually landed

Test Environment and Methodology

• Source data: 50 million rows of B2B event, order, and billing telemetry in Snowflake.
• Multi-tenancy: 900 simulated tenant accounts, with realistic volume skew (top 5 enterprise tenants account for 30% of total rows).
• Snowflake cluster: Standard X-Small warehouse, auto-suspend set to 60 seconds.
• Test queries:
  • Query 1: 30-day KPI summary with two COUNT(DISTINCT) calculations.
  • Query 2: Daily time-series trend with dynamic date-grain switching (day, week, month) across 90 days.
  • Query 3: Multi-column tabular breakdown with role-based column masking on unit margin fields.
• Load profile: 50 concurrent tenant sessions running random filter combinations over a 15-minute burst window to simulate morning login traffic.

Master Comparison Spreadsheet

What We Learned From The Runs

1. Where Metabase and Superset still make sense

If your customer base is under 100 accounts, your dashboard has uniform permissions, and your engineering team does not have frontend cycles to touch chart components, Metabase remains the fastest path to production. However, relying on question-level result caching falls apart the moment customers use dynamic date filters, because every unique date range wakes up your cloud warehouse.

2. Where Sigma and Omni excel

If your product serves finance, accounting, or operations users who want an Excel-style workbook directly inside your application, Sigma is very difficult to beat from a UX standpoint. Their spreadsheet model feels natural to non-technical analysts. Omni is an impressive evolution for teams that like LookML's modeling rigor but want modern speed and cleaner embedded authoring. The trade-off with both is that you are paying ongoing compute costs to your warehouse for every complex slice-and-dice query, unless you scale warehouse sizes.

3. Why the semantic layer approach with Cube worked best for our architecture

Cube gave us two specific architectural capabilities that resolved our initial bottleneck:

• Aggregate Awareness in Cube Store: We defined a pre-aggregation rollup partitioned by month on tenant_id, transaction_date, and service_tier. When a tenant loads a 7-day dashboard, switches to a 90-day view, or aggregates by week, Cube Store rewrites the query to hit the local sorted Parquet partition. It served 94% of our simulated morning burst queries in under 450 ms (p90), consuming only 1.1 Snowflake credits over the 15-minute test window.
• Decoupled Surfaces on One Metric Model: We do not have to rebuild all 14 dashboard cards in React immediately. We can render high-traffic KPI cards natively using our own React design system through the REST API, while dropping in the embedded Creator Mode interface for our top enterprise accounts so their power users can build custom reports against the exact same governed measures.

4. The real trade-offs and gotchas with Cube

This was not a free lunch, and there are two operational complexities anyone considering this path should know:

1. Data modeling discipline: You have to explicitly model your cubes, primary keys, and join relationships in YAML or JavaScript. If your underlying database schema has messy joins or unindexed foreign keys, Cube will force you to fix your join logic before your metrics compile correctly.
2. Index sorting and warmup overhead: In our first pre-aggregation attempt, we naively added three high-cardinality string dimensions to a single rollup. The pre-aggregation build time took over 25 minutes and consumed noticeable worker memory. We had to restructure the rollup to order indexes strictly by tenant_id, date, and low-cardinality status codes, using partition_granularity set to month with refreshKey tracking our updated_at timestamp.

my question for the community.

now, for those of you who do run pre-aggregations / rollups in production embedded environments: i’d like to know how you guys work with late-arriving data (like payment refunds or backfilled usage events arriving 48 hours later) and do not trigger complete rebuild of historical partitions? like, are you running targeted partition refreshes on an incremental window, or maybe relying on live union fallbacks?

Hey everyone!

I’ve been working on a game called DataMart, where you manage a corner shop and everything runs on real SQL. You build it all yourself: the databases, schemas, tables, relationships and queries, your way, on a real PostgreSQL server that runs inside the game.

Customers will ask you things like “Where can I find the apples?” or other questions about the store. Once a week an auditor comes by to check you’re tracking everything properly, and asks you a few questions you’ll answer with SQL.

There are also mysteries and stories involving the people in town, and of course you unravel those with SQL too.

The game is in alpha and free at https://rodrigoavf.itch.io/datamart

I’d love to hear your thoughts!

Ps. Yes, the game is being created with the help of AI, and what is not now days?

We process 68000 billing chats a week and our production classification kept splitting 4216 family plan transfer conversations between 2 existing buckets. 2170 landed in cancellation while 2046 landed in general billing. The topic distribution barely moved, the labels were wrong. (I recognized the wording from a support review.)

So I ran semantic clustering over the production traces and the family plan transfer cluster showed up in Braintrust with the mislabeled examples grouped together. I pulled that trace cohort into one review and added a custom facet to track the pattern without replacing the whole taxonomy. A useful catch. The dashboard now shows the emerging request but I still haven’t decided whether it deserves a durable category or a temporary watch label.

I’m a data engineer and previously did ERP implementations for 3 years. Saw how much of a crap shoot most F500 companies data is, anyways took this job to go back to school at smaller firm.

Company is finally realizing how bad their data is and have exec support, but I don’t think they know how bad it really is. I mainly work with the external data for business decisions.

Basically the internal data is split across 6 systems and they have no standard documentation/ideas of how their business operates or what data each unit consumes etc.

My manager/engineering team are building out a new software and haven’t even cracked open the data foundation. I’ve seen this a million times in implementation where execs and management assume the data is all good and then it’s go-live / go-time and realize their data is up the wall.

I know for a fact this companies internal data is messed up to a new level (without even seeing it)… manager thinks this will just be a light lift we just go in write out some documents and map some schemas and it’s all good happy day…

How do I communicate to him and the other executives that if they want to really do this it’s not like a week to a month project???

I mean we are talking about establishing data maturity/ MDM from scratch

Hi all,

I work in credit risk at an auto lender. Because of new regulation, we need simple, fixed rules that decide which applicants must provide extra documentation (proof of expenses). My task is to find out which customer groups are risky enough to justify that.

Setup: ~250k loan contracts, binary target = default within 12 months (~1%). Features include loan amount, total exposure, income, DTI, LTV, vehicle value, new vs. used car, online vs. dealer application, and our internal scorecard.

What I've done so far: default rates per bucket and per feature combination, confidence intervals, a scan for cut-offs on loan amount, and a logistic regression with the scorecard as a control.

My questions:
1. What's your general approach to deriving thresholds that are both statistically sound and simple enough for business and regulators?
2. How do you validate that segments or cut-offs are real and not just noise, especially with a low default rate?
3. How do you show a rule adds value beyond an existing scorecard?
4. Are there methods, papers, or tools you'd recommend for this kind of problem?

Any experience or pointers are appreciated. Thanks!

This is a utility that is part of the framework I had posted about earlier, the post discusses the ideas and the utility behind the tool. The TL;DR version is that it helps you move from a conceptual business use case to a data representation that can be consumed by the tool, the required context is also presented.

https://r2dssanity.substack.com/p/from-business-use-case-to-technical?r=3ncgel&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Do you use Qt and Python to visualize and interact with data?

Then you might like this library that I've been working on for the past couple years. It provides an easy way to add parameters for different types: float, int, string, but also Qt types such as QPointF, QSizeF etc.

I use it to load pydantic models from configs and save them back out.

https://github.com/beatreichenbach/qt-parameters

In Retail Sales Analytics project in Power BI, I started investigating different categories, quantities, and discounts to understand why.
But I found myself wondering: Am I getting closer to answering the business question, or am I just analyzing more data?
There are always more charts to create, more numbers to compare and more questions to ask. But not every finding helps us understand the actual problem.
I am curious to know how experienced data analyst approach this.
How do you decide which direction to investigate and how do you avoid spending time on analysis that does not add much value?
I love to hear your thoughts.

Guys i am thinking of conducting an experiment. I have a masters degree in a language (sanskrit) and have no background in statistics.

My experiment will include around 100 participants, each rating 30 objects in 1-7 likert scale (6 scales). I asked claude and it suggested linear mixed model in R.

But as i said i have no experience. Can you guys recommend some courses or YouTube videos to get started. I feel so overwhelmed looking at different type of tests or models.

Any resources for getting started in forcasting?

I have a work project in mind but feel like I'm both over complicating and simplifying this process. I seem to keep landing on extremes when I try to search it up with either machine learning/ai overkill or generic overviews.

For context I'm just aiming to forcast whether locations will hit monthly budget goals based on historical data. Ive considered either:

1. pulling average daily services and dragging that out to see what the likely total is.

Or

1. Calculating cummulative % of services completed by day for the previous month and use that to assume estimate the end point for the current month.

Again I'm pretty sure Im over complication and simplifying this. So any help would be greatly appreciated.

Over the years, I've collected practical notes, examples, and references while working on real-world data analytics projects.

I recently organized them into a free learning pack covering:

Power BI & DAX – Data modeling, calculations, and reporting.

SQL – Queries, data manipulation, and analysis.

Tableau – Data visualization and dashboards.

Python – A compact guide to data analysis with practical examples.

I also included DataBridge AI, a Python-based tool I've been developing for data cleaning, validation, mapping, and analytics workflows.

These materials started as personal working references, so the focus is on practical application rather than theory alone.

I'm sharing everything for free in case it's useful to other analysts or people learning data analytics.

Free resources: https://aabdelghaffar650-cloud.github.io/portfolio/#free-pack

I'd also be interested to hear which topics or practical examples others find most useful when learning data analytics

I don't know if others agree, but I think given the data oriented nature of this subreddit, it is bound to resonate with some. I learned the craft from a statistics view point, my formal training is statistics based, but I worked in software engineering for the predominant part of my career. Rather than hit everything with a "foundation model" hammer, I think leveraging the libraries that have emerged in the last two decades for data analysis with LLM tools is a more productive and tested path.

https://r2dssanity.substack.com/p/beyond-the-agent-illusion-leveraging?r=3ncgel&utm_campaign=post-expanded-share&utm_medium=web

Hello fellow Redditors, so I've caught myself in a tough situation; would be glad to hear your advice on this. Since, I believe there are people who have way more experience in this domain that I do.

With around 3.5+ years of experience in the data & analytics space, I believe this is a good time to select a niche because, at the end of the day, no good company looks out for a generalist data analyst; they're looking for specialists in a certain niche. But I'm confused...

And honestly, I do believe I have found the right niche where my web analytics skills (GA4, GTM, etc.) can complement my data analysis skills (SQL, Python, Power BI, etc.) to create a strong offer and solve a specific problem for my ICP. However, I'm still not sure how to execute this and turn it into a running business model. I think I can confidently say ecommerce is my niche.

I have been testing a lot lately, tweaking my profile every other day because it's quite confusing. At one point, I want to select a niche and a specific problem and keep posting about that until I start getting inbound leads.

Meanwhile, I want to keep my profile general instead of selecting a niche because it would have an impact on recruiters or HR skimming through my profile. Like, whenever they have an open analyst role, they'd probably skip my profile the moment they see 'e-commerce' or some other niche in my headline, especially if they're hiring for a different niche/industry.

I want to hear from experienced folks here, what do you suggest? Should I go narrow, or keep it broad and keep talking about different topics instead of sticking to 2-3? If you were in my shoes, what would have been your strategy? My eventual goal is to get inbound leads from LinkedIn or even just work on a side-hustle. I have been posting consistently for three months now, but that was with a generalist LinkedIn profile, and not targeting any ICP, and I never got a single inbound message except for 2-3 messages from recruiters.

After completing an analysis, what checks do you perform to ensure that your calculations, aggregations, and conclusions aree correct? Do you independently validate results using SQL, spreadsheets, or another tool?

We are implementing Collibra as part of the data governance initiative and honestly I’m struggling to see the end of the tunnel.

The tool seems huge and overwhelming: business glossary, document data model, mapping relationships feels like an enormous amount of work that could go forever.

I’d love to hear from people that have Collibra in the companies they work for:
\- where are you in implementation? Have you reached a point where collibra genuinely is part of your day?
\- are you getting any value from it?
\- what are you documenting?

Let me know your honest opinions!

I am in the process of learning data analytics and Google Data Studio. I'd appreciate any feedback you'd be willing to offer. Thank you. Here is the full link, as the dashboard is interactive

Hi everyone,

With all the talk about AI and the fast-shifting market, I’ve been questioning whether a traditional data analysis toolkit is still viable and in demand for solving real-world business challenges:

Microsoft Excel: For data cleaning, auditing sales records, and tracking daily core metrics.

SPSS: For survey analysis, evaluating customer satisfaction, and identifying key statistical drivers behind buying behavior.

Power BI: For building interactive dashboards that connect sales data directly to decision-making, identifying bottlenecks, and spotting revenue opportunities.

For business owners, managers, and working analysts:

In practice, is this stack still enough to solve concrete sales problems and land real consulting or job opportunities? Or has the market moved past these tools, making them insufficient for today's business needs?

I'd really appreciate your honest feedback and practical experiences.

I’ve been recording encounters with a particular category of vehicle during everyday journeys. I’m examining whether these encounters are concentrated near departures and arrivals, after accounting for how much time I observe in those periods.

I have timestamped encounter records and journey start/end times. My comparison uses the first five minutes after departure and the last five minutes before arrival, counting overlapping windows only once.

I calculate:

Share of encounters within those windows ÷ share of observation time within those windows.

I’d appreciate feedback on whether this is a reasonable comparison and how to improve it. Routes, road conditions and times of day vary, and some vehicles travel together in groups. How should I account for those factors when assessing the apparent concentration?

Anonymised endpoint example

This is a real 20-leg movement block from four recorded visit days. All legs in that block are included, including seven with no target-category encounters. It is a selected block, not a complete diary of all journeys.

Names, locations, vehicle-category names, calendar dates and clock times are omitted. D1-D4 indicate different recorded days; legs with the same day label may be related.

An endpoint window is the first 300 seconds after departure or the final 300 seconds before arrival. Overlap is counted once. Window time is therefore the smaller of 600 seconds and the leg duration. Event classification uses the original timestamps and includes the exact five-minute boundary.

Durations below are elapsed leg intervals calculated from recorded start/end timestamps. Treating every second as available observation time is an assumption for review; continuous observation coverage has not been independently verified from recordings.

Leg table

All times are seconds. Inside and Outside count individual vehicles as recorded; Groups counts encounter rows. A single encounter row can contain several vehicles travelling together.

Encounter offsets

Each row is one recorded encounter group. The offset is seconds after departure on the specified leg. The vehicle count preserves groups rather than treating every vehicle as an independent encounter.

The totals are 31 recorded vehicles, of which 29 are inside the windows, and 22 encounter groups, of which 20 are inside. Total elapsed time is 546.58 minutes; window time is 195 minutes.

Vehicle-count enrichment = (29 / 31) / (11700 / 32795) = 2.62. This is the endpoint encounter rate divided by the overall encounter rate. It is a descriptive comparison, not a probability or causal conclusion.

Useful review questions: Is elapsed leg time an adequate exposure measure? How should repeated legs on one day and grouped vehicles be handled? What additional traffic, route and time information would be needed to assess ordinary variation?

Hi guys, I'm an analyst, not in IT but was recently tasked with finding, classifying, and removing duplicates for sensitive data at my company and want to do this in the least tedious manner. Do you have any tips?