[ Removed by Reddit on account of violating the content policy. ]

I ran my schema-selection layer for SQL agents on Spider 2.0 and BIRD this week. Four things I didn't expect, and the numbers.

Context: I maintain schemagate, an open-source retriever / MCP server for SQL agents. It picks the tables a question needs from the ones the caller is allowed to read, and checks the generated SQL against the same list before it runs. Everything below is one model call per question (no voting, no agent loop), scored by each benchmark's own evaluator. Predictions and evaluator output are in the repo.

Results

• BIRD dev, all 1,534 questions: 77.1% execution accuracy (Claude Opus 5.5). Hidden test set submitted, waiting on the BIRD team.
• Spider 2.0-lite, the 135 SQLite questions: 71.9% (Sonnet 5.5). The best published agent I know of on that subset reports 57.8% Pass@1, with a different model.
• Spider 2.0-lite, the 205 BigQuery questions: 45.9%. Much harder; Snowflake isn't run yet, so no leaderboard row.

What surprised me

1. Opus did worse than Sonnet on Spider 2.0. Same pipeline, same 135 questions: Sonnet 71.9%, Opus 67.4%. On BIRD Opus was clearly better. Don't assume the bigger model wins on your data; measure it.

2. Voting didn't help. Several candidate queries + majority vote + a judge looked like +3 points on half the questions, and exactly +0 on the held-out half, at 5.5x the cost. I'm glad I split the set before tuning.

3. BigQuery's date-sharded tables broke table selection. One database has 366 tables named ga_sessions_20160801 ... ga_sessions_20170801. Offered one by one, they filled all 12 slots with random days, and the model refused ("no table for January 2017"). Collapsing each family into one wildcard table (ga_sessions_*, filter on _TABLE_SUFFIX) fixed most of those refusals. If your warehouse has sharded tables, your retriever probably has this bug.

4. The benchmark found a real bug in my access check. 3 answers came back blank. My SQL scope check read WITH RECURSIVE periods(n) AS (...) as a query against a table called periods the caller couldn't see, and refused it. It failed closed (nothing leaked), but every recursive query was wrongly refused for every user. Fixed, with a test. Benchmarks you didn't write are good at this.

Honest caveats: single runs, so about ±1–2 points; the Spider 2.0 numbers are two of three parts; BIRD Mini-Dev rows are self-reported until the leaderboard PR is merged.

Repo (Apache-2.0, pip install schemagate): https://github.com/ashishsinha1602/schemagate Benchmarks page with every caveat: https://ashishsinha1602.github.io/schemagate/benchmarks/

Happy to answer questions about the setup, or about what didn't work.

every time we change the prompt or the model, something else breaks. one answer gets better, two get worse. we only find out when users complain.

do you guys run evals for this? and how do you track what the agent is doing step by step when it gives a bad answer? right now we just read logs and it takes forever.

I've been building AgentRein for about a year, and this is one of the problems I've been trying to solve.

Let's say an agent creates a GitHub issue, generates a Stripe invoice, and updates something in your CRM. Then it realizes something went wrong halfway through.

The API calls themselves might have all worked fine, but the overall result is a mess. Now you're left figuring out what to undo and what to keep.

I've been working on a way to handle this without having to rewrite existing integrations.

With AgentRein, you can wrap existing SDK clients using agentrein.wrap() and add a safety layer around their operations.

A few things it supports:

• Rolling back previous actions in reverse order when something fails, where undo operations are available.
• Requiring human approval before risky API calls.
• Detecting when an agent starts drifting away from its original intent.
• Rules to allow, block, or require approval for specific operations.
• Audit logs to track what happened during a session.

I'm curious how other people are handling this, especially outside of demos.

Do you use custom rollback logic, checkpoints, compensating transactions, or something else entirely?

And how do you deal with actions that succeeded technically but shouldn't have happened in the first place?

I've been building this for a while, so I'd appreciate honest feedback on the approach.

https://agentrein.com/docs

​

I've spent the last month trying to turn my independent AI engineering projects into paid work, without much success.

I've completed two public RAG evaluation betas and built tooling for retrieval testing, reproducible bug reports, AI workflow inspection, and MCP-based development.

I tried Upwork, other freelance platforms, and direct outreach. The biggest lesson so far is that selling standalone technical audits is difficult when companies primarily want their problems fixed.

I'm now considering subcontracting for agencies that already have paying clients.

Instead of offering a large consulting package, I'd prefer to start with something small and measurable: reproducing an AI agent failure, testing a workflow, investigating a retrieval regression, or verifying a specific fix.

For people who run AI automation agencies or have done this kind of subcontracting:

Where are these small paid tasks actually distributed?

Are there agency partner networks, developer communities, vendor lists, or other channels where new subcontractors can demonstrate their work and earn an initial paid trial?

I'm especially interested in experiences from people who have secured real paid work this way, rather than general advice about building a personal brand.

Thanks for sharing what actually worked.

I kept running into the same problem building agents. Something would go wrong three or four steps downstream, and by the time I noticed, I had no real way to trace back what the agent actually believed at each step or what depended on it. Logs tell you what happened, not what justified it.

So I built Sevenbox. Every decision an agent makes gets recorded as a Warrant: what it claimed, what prior claims it depended on, and whether acting on it can be undone. If a belief turns out wrong later, Sevenbox walks the dependency chain and tells you exactly what still needs fixing instead of you grepping through logs trying to reconstruct it by hand.

It works with LangGraph, CrewAI, OpenAI Agents SDK, and eleven other frameworks, so you can wrap an agent you already have instead of rewriting it. The SDK, the framework adapters, and a local UI for inspecting a run are all open source.

I am the person who built this, not a company account posting on its behalf. I would genuinely like people actually running agents in production to try it against something real and tell me where it breaks or where the idea does not hold up. Happy to answer anything about how it works.

Hey everyone,

When building agents that interact with external APIs (GitHub, Jira, Stripe, or internal microservices), tool outputs frequently flood the context window with hundreds of tokens of metadata: avatar URLs, HATEOAS links, node IDs, and null keys.

The common pattern of using an auxiliary model (like GPT-4o-mini) to summarize tool outputs before feeding them to the primary agent introduces 500ms–2s of external network latency and recurring API charges.

I built TokenPrune, an open-source, zero-dependency Python package that executes deterministic, in-memory pruning and table compression:

bash pip install tokenprune

Quick Example

```python from tokenprune import prune_tool

@prune_tool def fetch_system_logs(): return { "cluster_id": "us-east-1", "service_url": "https://internal.aws/k8s/node/12", "health": "degraded", "errors": [ {"code": 500, "reason": "Database connection pool exhausted", "doc_url": "https://docs.internal/500"}, {"code": 503, "reason": "Upstream timeout", "doc_url": "https://docs.internal/503"} ], "debug_trace": None }

print(fetch_system_logs()) ```

Pruned YAML output injected directly into the LLM prompt: yaml cluster_id: us-east-1 service_url: [https://internal.aws/k8s/node/12](https://internal.aws/k8s/node/12) health: degraded errors: [table: code, reason, doc_url] - 500, Database connection pool exhausted, [https://docs.internal/500](https://docs.internal/500) - 503, Upstream timeout, [https://docs.internal/503](https://docs.internal/503)

Benchmarks

• Token reduction: 70–90% on enterprise API payloads.
• Runtime overhead: < 0.5 ms (AST tree-shaking in pure Python standard library).

• External API calls: 0.

• GitHub: https://github.com/GeekyMK/tokenprune

• PyPI: https://pypi.org/project/tokenprune/

I would love to hear feedback from anyone running agent workflows on where the heuristics break or what schema formats you deal with most often.

I'm researching how teams handle evidence and quality for AI agents that make or assist decisions.

Three things I'm trying to understand:

1. What record do you keep per decision (policy applied, data accessed, who was responsible), and who actually asks for it?

2. How do you learn later that a decision was wrong, and how long does that feedback take?

3. If you use LLM judges for evals, how do you check they still agree with humans as things change?

I'm trying to learn what's really painful versus what only sounds painful. Comments or DMs both welcome, and I'll post a summary of what I learn.

We are building an internal assistant that reaches Slack and Jira and a few internal APIs through MCP and something needs to sit in the middle before it goes wider than the team. Two options are on the shortlist and maybe a third if neither fits. One looks quicker to connect with prebuilt integrations and the other leans harder on prompt injection checks and per user rate limits and logging what the agent called. Pricing once usage grows matters too since this runs all day. Which held up for you in production?

Agents retry. Usually that's fine, but not when the tool issues a refund, sends an email or
deletes records. I built aobs (Apache-2.0, Python) to handle that, plus the observability
around it:
Guard risky tools: checked against your real records (not the model's arguments), approval
above a threshold, no duplicates across retries/restarts. Recipes for refunds, emails,
deletes and deploys. [there's a demo GIF in the README]

Trace runs: OpenAI-compatible clients, Anthropic, LangChain/LangGraph; export to
OpenTelemetry.

CI gate: a TOML policy → PASS/FAIL with evidence, as a GitHub Action.
docker compose up for a local demo with a dashboard.

Alpha, honest about limits (in the README). I'd love feedback, and there are a few "good first
issue" if you want to contribute.
https://github.com/AshmitJain932/ai-agent-observability

Say an agent submits a request but the tool times out before confirming it worked. The agent retries and now there are two tickets, bookings or payments.

How are you handling that uncertainty? Does the agent check whether the first attempt succeeded, does the API prevent duplicates, or does it stop and ask someone?

Interested in how you handle cases where the tool never gives a clear answer.

I run an AI agent daily and it writes to its own memory and skills. Three failure modes kept showing up, plus one bug I hit in the streaming layer while watching it happen. I built one small thing for each, measured them, and sent them upstream. Two plugins are in the upstream catalog, the third is in review, and the core fix is open. The three plugins are Apache-2.0, no paid tier, no account.

1. It writes its loop into a file

Local models and stuck streams repeat themselves. Hermes truncates runaway text, but by design it lets a provider-confirmed tool call through even when its arguments are repetition. A tool call is also the only part of a turn that changes anything outside the process — a file, a post, a shell command.

hermes-degeneracy-guard sits at pre_tool_call, before the tool executes. It scores the arguments; when they look like a loop rather than data, it hands the call to Hermes' own approval gate instead of guessing. A refusal, an error in the gate, or a timeout all refuse the call, and cron defaults to deny, so an unattended run fails closed rather than writing a corrupted file.

Evidence: five hooks, none of them editing a file shipped by Hermes; hermes plugins validate . passes at the pinned commit. Catalog PR #141341 merged 2026-10-20.

2) It records a decision the user never made

"The user decided X" is the most dangerous sentence an agent can write, because the next session believes it. hermes-provenance-gate checks every memory and skill write for that construction and looks for the user's own words behind it: a quote that appears verbatim in what the user typed this session, that shares key words with the claim, and that does not drop a negation. By default it only logs writes that fail this check; block mode refuses them.

Evidence: on 133 lines written and labelled by a separate model session that had not seen the plugin's code — 49 claims, 84 not — precision 1.000 (34/34), recall 0.694 (34/49). The misses are published next to the hits, and two independent review passes by separate model sessions turned their findings into regression tests. Catalog PR #141935 merged 2026-10-20.

3) It can't say where a skill's text came from

To trust a skill that rewrites itself, you need to answer two questions afterwards: which text went into it, and whether the record of that is still intact.

hermes-skill-lineage records, per skill write, the content fingerprint before and after, the sources present earlier in the same session, the features linking a source to the added lines, and a seq → prev_hash → hash chain over the entries. trace answers where a version came from; descendants goes the other way — give it a page or a skill and get every skill whose content it ended up inside; report lists versions carrying lines of no traceable origin instead of smoothing them over; verify re-walks the chain and head prints a value to write down elsewhere, so a later run can tell truncation from an intact chain.

Evidence: 48 tests pass and one module skips on both Python 3.11.16 and 3.14.7 — that module needs Hermes itself importable — and hermes plugins validate . passes with no warnings on Hermes 0.22.0 (2026.9.24, local e06df9c2eae6). A manual end-to-end run in an isolated data root created a skill through skill_manage, then ran status / trace / descendants / report / verify / head — raw log committed in the repo. Catalog PR #142381 is open.

4) Two streaming paths the loop guard doesn't reach yet

Hermes added a guard that cuts a stream as soon as it turns into a repetition loop. It is wired into the chat_completions and Anthropic Messages streams; the Bedrock Converse and Codex Responses streams don't use it, so on those an uncapped endpoint can stream a loop to the output limit. #141477 is the report. #141479 wires the existing guard into both, reusing the same detection logic and the same end-of-turn hand-off, with no new detection logic. Both are open.

What I did not solve, and would like your take on:

• The source→line link is computed from shared shingles, anchors and session order. It is not causal proof. If you know a cheaper way to separate "this line came from that page" from "they happen to share words", I would rather replace my heuristic than defend it.
• Keyed hashes mean a third party can verify internal consistency but cannot re-derive my links from a public artifact. Publishing an unkeyed digest is the obvious alternative and it leaks content to dictionary attack on short strings. I chose privacy over third-party reproducibility. I am not sure that is the right trade.
• The three defaults do not agree with each other. degeneracy-guard acts: it hands a suspicious call to the approval gate. provenance-gate watches: it logs what it would have refused as would_block and refuses nothing until you set PROVENANCE_GATE_MODE=block. skill-lineage only records; it never blocks. I set them that way because the cost of a false block differs — a corrupted file is expensive to notice, a refused memory write is cheap to retry, and losing a fingerprint costs nothing at all — but three sibling tools with three defaults is a smell, and I would rather hear why it is wrong than defend it.

Links:

• https://github.com/simin-yuan/hermes-degeneracy-guard
• https://github.com/simin-yuan/hermes-provenance-gate
• https://github.com/simin-yuan/hermes-skill-lineage
• https://github.com/NousResearch/hermes-agent/issues/141477 and https://github.com/NousResearch/hermes-agent/pull/141479
• Upstream record with the PR texts and raw API payloads: https://github.com/simin-yuan/upstream-contributions

Disclosure: I use an AI agent in my own workflow — that is why these tools exist. Parts of this write-up were drafted with AI help and edited by me; English isn't my first language. The measurements above are runs I can point at; where something failed, was skipped, or is still unmeasured, I have said so.

most LangGraph demos stop at the happy path. in production the interesting problems are what happens when a node fails halfway, a step gets retried, or a human needs to approve something before the next action runs.

there’s a workshop on Oct 30 that builds these patterns around an enterprise claims triage case study. Fri Oct 30, 9:30 AM to 12:30 PM ET, live and online, led by Ben Auffarth (AI consultant, best selling author, 15+ years).

what it covers:

• idempotent LangGraph state machines, so retries don’t repeat side effects
• human approval gates for uncertain or high impact decisions
• compensation handlers that undo earlier steps when a later one fails
• a routing layer built on small encoders (ModernBERT, SetFit) instead of an LLM call, running on CPU under 15ms
• split conformal prediction to decide when the router should hand off to a person or an LLM
• Wilson score intervals and clustered standard errors to evaluate it properly

notebooks are runnable, intermediate Python is enough, and a laptop or free Colab works.

Full details here

Hi there, El from Cohere here. Today we’re introducing Rerank 5, and since it’s available in LangChain, I thought i’d put it here. 

Rerank 5 available in Pro and Fast tiers just like Rerank 4 but far outdoes it – along with every other model we tested. Pro is number 1 on the nanoMTEB and ViDoRe V3 benchmarks and Fast outdoes other speedy models like Qwen3-Reranker-0.6B and Qwen3-Reranker-8B (which is a lot bigger).

Both are also leaders across multilinguality, financial document performance, healthcare docs, and more (I’ll let you read the full blog for all that info). 

We’re also introducing a totally new feature to Rerank that (to our knowledge) we’ve never seen in any other reranking model, Context Focus. Enterprise docs are only getting longer, so more and more context is getting passed to LLMs=more tokens. Context Focus grabs only the specific passages in a document relevant to your query, and it reduces token costs pretty significantly. It’s an opt-in feature and works like a dial, so you can choose exactly how much context you want. 

Like I said, it’s available through LangChain (and a bunch of other places like our API). See our blog for more info and lmk what you think! 

Question for people using LangChain in real products: what does the rest of your backend look like?

Once the chain/agent works, I still see teams needing auth, app data, vector storage/search, files, document parsing, usage tracking and background execution.

I'm building Singlebase around that layer - basically the backend underneath the AI orchestration. One thing I'm experimenting with is normalizing documents to Markdown + extracting images before chunking/indexing, so ingestion isn't tied to one file format. Do you prefer your RAG/agent framework to own more of that infrastructure, or keep LangChain focused on orchestration and put persistence/search/etc. behind a separate backend API?

Hi all,

I took the LangChain Certified Agent Engineer exam on Sunday morning (Singapore time). The exam details say results arrive within 72 hours, but that window has passed and I haven’t heard anything yet.

Has anyone else had a delay like this? If so, how long did your results take, and did you have to contact support?

Also, two questions referred to a dataset that was supposed to be in the org shared with me, but the dataset wasn’t there. Did anyone else run into this? I’m wondering if it’s a known issue with the exam environment.

The exam FAQ mention email addresses that aren't monitored now. Hope someone at LangChain read this and update the FAQ's

Thanks in advance!