One commit per remember
Each remember call adds the text that was remembered and the entities and facts it yielded, and edits the files of the facts it confirmed, ended or invalidated.
What it is
The history of a dataset becomes the history of a repository. The export reads the databases only and makes no LLM call, so it works even when no LLM is configured.
Deterministic
The same dataset gives the same commit IDs. Dates are taken from the data, and author and committer are always gligabao <gligabao@localhost>.
Generated and read-only
The repository is built from the dataset when you ask for it, and is not imported back. The dataset stays the source of truth.
No git binary needed
The server writes git's loose objects (blobs, trees and commits) itself and streams them as a tar.gz. You only need git to read the result.
Exporting a dataset
The endpoint is GET /v1/gb/datasets/{name}/git. gb-user-id is the dataset's owner, and only the owner's live dataset can be exported. The answer is 200 with application/gzip: a tar.gz with one bare repository in <name>.git/.
# over HTTP
$ curl -H 'gb-user-id: u1' http://localhost:7878/v1/gb/datasets/main/git | tar xz
$ git clone main.git main && cd main && git log --stat
# or with the CLI: downloads and unpacks the bare repository into DIR (default NAME.git)
$ gligabao dataset git main main.git
$ git clone main.git outErrors are {"error": "..."}: 400 for a missing or invalid user, 404 if the user has no live dataset with that name, 500 for a database error. The archive is streamed after the data is read; a failure while writing it cuts the download short (a corrupt gzip) and is logged.
What a commit is
Commits are in time order, oldest first (by created_at, then ID).
Root commit
Adds README.md: the dataset's name, description and tags, and the layout below. Dated at the dataset's creation, or at the first call if that is earlier.
One commit per remember call
A call is a root artifact (one that is nobody's child) with its descendants: the chunks and summaries of a long text, the web searches and the series observations made for it. The commit date is the artifact's created_at: system time, when gligabao learned the text, not when what it says was true.
Query and failed runs
A query is a commit of its own, query #<run id>: <question>, that only adds runs/query/<id>.json. A remember call that stored no artifact (failed, stopped, or its artifact was deleted) is remember run #<id> failed: <error>.
Last commit, if needed
Adds the entities and facts that have no source artifact any more (it was deleted), in their final state.
The message of a remember commit is remember #<artifact id>: <first line of the text>, then a summary (+2 entities, +3 edges, 1 facts confirmed, 1 facts changed), a line for each fact that was ended, invalidated or superseded, and the trailers Artifact-Id, Kind, Session-Id (if any), Valid-At, Remember-Run-Id and Artifacts (how many artifacts the call stored, if more than one).
Reading the history
Plain git works on the knowledge. The commit below is an illustration of the format: a second call that told gligabao Alice changed jobs.
$ git log -1 --format=fuller
commit 3f9c1e0a…
Author: gligabao <gligabao@localhost>
AuthorDate: Mon Nov 3 09:14:22 2025 +0000
remember #2: Alice left Acme and joined Initech in March.
+1 entities, +1 edges, 1 facts changed
- invalidated edge 1: Alice works_at Acme
Artifact-Id: 2
Kind: text
Valid-At: 2025-03-01T00:00:00Z# how did the facts evolve?
$ git log -p edges/
# every commit that touched Alice
$ git log -- entities/person/
# which call said this fact?
$ git blame edges/works_at/1.mdLayout
Files have a YAML front matter between --- lines; strings are double-quoted. File names are lowercase slugs (letters, digits, _, -; at most 40 characters) plus the ID, so they never collide.
| Path | Content |
|---|---|
artifacts/<id>.md | The text of a root artifact: front matter (id, kind, session, classes, valid_at, created_at) and the text. Its chunks, summaries, web searches and observations are in artifacts/<id>/<child id>-chunk-<n>.md, -summary.md, -web-search.md, -series-observation.md, with parent (and for a chunk chunk_index, start_offset, end_offset, tokens). |
entities/<type>/<name>-<id>.md | An entity: id, name, type, model, artifacts (the artifacts that mention it so far), and its description as the body. |
edges/<relation>/<id>.md | A fact: source, relation, target (and their IDs), valid_at, invalid_at, created_at, expired_at, model, artifacts and the description as the body. |
runs/remember/<id>.json, runs/query/<id>.json | The log of a remember call or of a query (see below). |
How changes become file changes
The export replays the edge events of each call (gb_edge_events), so the history of the facts is exact: when a fact was learned, ended or contradicted is in the diffs.
| Event | In the commit |
|---|---|
a new fact (created) | adds edges/<relation>/<id>.md (with invalid_at if the text said it already ended) |
a stored fact stated again (confirmed) | adds the artifact to the fact's artifacts |
a stored fact that the text ended (ended) | adds invalid_at |
a stored fact contradicted by a new one (invalidated) | adds invalid_at and expired_at |
the new fact is older than a stored one (superseded) | the new fact's file has invalid_at and expired_at |
| an entity mentioned again | adds the artifact to the entity's artifacts |
Graph classes and run logs
The repository also keeps how the knowledge was made, not only what it is.
Graph classes
The classes an LLM gave to the graph of an artifact (the classify_graph stage) are the classes of that artifact's front matter: a sorted list, once each (classes: ["career_change", "hr"]). For a chunked text they are on the chunk files.
Remember runs
runs/remember/<run id>.json, in the commit of the artifact it stored: kind, status, error, params, the times, the stages in pipeline order with the model calls they made (model, provider, reasoning, tokens, cost_usd) and the kept stage outputs.
Query runs
runs/query/<run id>.json: the question, session_id, status, error, params, times and stages. A query does not change the knowledge, so its commit only adds this file. The answer is not kept by the database, so it is not in the log.
Scoped to the dataset
Runs are matched by dataset name. Runs of an earlier, deleted dataset of the same name, and of other users, are left out. A run that is still going shows the stages it has reached so far.
Limits
The export is a view of what the database keeps, so some things are approximate or missing.
Good to know
- Entity history is approximate: an entity has only its latest description, so its file appears in the commit of the first call that mentions it, already with that description, and then changes only in
artifacts. Facts are exact. - It needs
gb_edge_events: a fact with no events (a database made before they existed) is shown as created and never ended. Events of a deleted artifact are lost, and its facts go to the last commit. - Commit dates are system time: a text about 2019 remembered today is a commit of today. Calls with the same
created_atare ordered by artifact ID. - Not exported: embeddings, timeseries, geo data, images, sessions as such and the usage counters. The README of the repository says so.
- Run logs are those of the end of the call, not of the commit's time: stage outputs may have been garbage collected, and the
runs/files can be big. - Privacy: the repository contains every text of the dataset, so it is as sensitive as the dataset.
- Scale: the whole archive is built from loose objects and the data is read into memory first, so for a very large dataset run
git gcafter cloning.
In the code
For people who embed or extend gligabao.
Packages
internal/memory/memory_gitexport: writes git's loose objects itself (no git binary or library).Writetakes anInputand aSink(DirSinkorTarGzSink).memory.LoadGitExport: reads a user's dataset into theInput.internal/api/datasets(Git, the handler) and thedataset gitcommand of thecli.