r/altprog • u/ewigerus • 22h ago
HQL (Hyper Query Language): any jq, pyspark or Kusto fans here? you might help
I've been working on HQL (Hyper Query Language), a small statically typed language built around functional expressions and data pipelines.
The basic idea is fairly conventional: values enter a pipeline, functions transform them, and the result can be collected, written somewhere, or kept as a stream.
A simple place to start is using it roughly where I might otherwise reach for awk, perl or jq.
Suppose birds.json contains some records and I want to select and transform them:
source(file("birds.json"), Json)
| filter(b => b.count > 2)
| map(b => {
species: b.species,
count: b.count,
})
| sort(by = b => b.species)
| target(file("popular-birds.yaml"), Yaml)
The pipeline reads JSON, filters it, constructs new records, sorts them and writes YAML.
The same thing can be used from a shell:
cat birds.jsonl | hql '
source(stdin, Json)
| filter(b => b.count > 2)
| map(b => b.species)
'
This part already works. HQL currently supports the CLI/data-processing use case, reusable .hql programs, typed expressions, collections and first-class streams.
The part I'm working on now is making the same model useful for actual data plumbing (access data sources and targets are configured via shared hql.toml).
Instead of introducing a separate workflow language, external systems should just be sources and sinks. Conceptually, a Kafka → PostgreSQL pipeline should look approximately like this:
source("orders")
| filter(o => o.status == "completed")
| map(o => {
order_id: o.id,
customer_id: o.customer_id,
total: o.total,
completed_at: o.timestamp,
})
| collect(500)
| sink("analytics")
where configuration could bind:
orders -> Kafka topic orders.v1
analytics -> PostgreSQL table completed_orders
Named external sources and targets are part of the HQL I/O model. The current design resolves them through runtime configuration (hql.toml), while the connector/configuration layer is still being developed.
Kafka is naturally a Stream<T>. collect(500) changes that into Stream<Seq<T>>, so the PostgreSQL sink can consume batches rather than individual rows.
This is deliberately part of the type model rather than a special streaming syntax:
Order
Seq<Order>
Stream<Order>
Stream<Seq<Order>>
The same distinction applies to files and standard I/O.
A finite result can be written once:
result
| target(file("report.json"), Json)
while a stream is consumed incrementally:
events
| sink(stdout, Json)
and a stream can be batched:
events
| collect(100)
| sink(stdout, Json)
There is considerably more in the language — functions, pattern matching, Option<T>, graph traversal, semantic retrieval, executable Markdown, etc. — but I'm trying to keep the core boring: typed expressions, collections, streams and composable transformations.
The implementation is in Rust and is still small enough that fairly fundamental parts of the language and standard library are open to change.
Repo:
Docs:
https://hql.hypermarkdown.org/
I'm particularly interested in feedback from people who have worked with Unix text-processing tools, functional/dataflow languages, Kafka-style stream processing, PySpark/Databricks, or ETL systems.
And if you look at it and think “this is missing X”, please open a feature request rather than assuming X is intentionally absent:
https://github.com/ewiger/hql/issues
Concrete examples and awkward edge cases are especially useful. I'm interested in finding out where this model stops composing cleanly. Or where this all can lead. Ty, wbr YY
