Add GenDB#932
Open
alexey-milovidov wants to merge 2 commits into
Open
Conversation
GenDB (https://github.com/SolidLao/GenDB) is an LLM-powered query engine that synthesizes a custom C++ binary per SQL query. We ran its multi-agent code-generation pipeline once against the ClickBench schema + 43 queries and ship the resulting .cpp files in generated/ so the benchmark itself needs no LLM access — install just compiles them. Storage layout is a raw per-column binary store the agents picked; ingest.py converts hits.parquet into that layout (DuckDB does the parquet decode). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- gendb/results/20260518/c8g.24xlarge.json — full 43-query cold/warm triple, load time, data size, concurrent QPS. - gendb/generation_times.json — wall-clock cost of synthesizing each per-query .cpp file (sum ≈ 41 min across 43 queries). make_result.py adds these to each cold try (warm tries unchanged — they reuse the binary on disk). - gendb/query: replace the awk needle-match with a bash string compare so Q29's regex backslashes don't get reinterpreted as awk escapes. Q29's [null,null,null] from the first benchmark run was the only query the awk version mishandled; results JSON has the post-fix [cold, warm, warm] for it. - gendb/utils/timing.h: print bare seconds so benchmark-common.sh's `^[0-9]+(\.[0-9]+)?$` last-line regex captures it. - gendb/storage_layout.json: trim to the 25 columns the 43 queries reference; HitColor is a 1-char string in hits.parquet that the int8 cast couldn't handle and no query needs it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR - it's not good, and kind of pointless. It generates low-quality code and sometimes produces incorrect answers (e.g., it misinterprets a SELECT * query). I hope students will defend their PhD.
Summary
.cppfiles were generated once, offline, by spawning Claude code-generator agents (mirroring GenDB's pipeline), then committed undergendb/generated/. Running the benchmark only compiles + executes them; no LLM access required at run time.gendb/ingest.pyto decodehits.parquetinto that layout (the upstream GenDB storage-designer normally writes its own ingest, but a battle-tested parquet decoder is more reliable than a generated one).gendb/results/20260518/c8g.24xlarge.json(data 37 GB, QPS 0.642, 43 queries).gendb/generation_times.jsonrecords the wall-clock cost of each.cppsynthesis;gendb/make_result.pyadds those into the cold tries when assembling the result JSON (warm tries are unchanged — they reuse the binary on disk).Test plan
.cppfiles compile underg++ -std=c++17 -O3 -march=native -fopenmp../benchmark.shruns end-to-end against a fresh checkout:install→bench_download→load→ 43 × 3 query tries → concurrent QPS test →Data size:output.generate-results.shpicks upgendb/results/20260518/c8g.24xlarge.jsonand produces a validdata.generated.jsentry.