240b5214dd
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
zhenyi
|
240b5214dd |
feat(archive): add on-disk archive/bundle cache with streaming persist
Adds a single-process on-disk cache for the two archive-producing entry
points: GitBaby::get_archive and GitBaby::create_bundle. The cache
lives at <repo>/archive/<key>.<ext>, so it's local to the repository
and cleaned up with the working tree (no separate cache directory to
track).
Why this layer:
- Archive generation is the most expensive routine in the lib: it
shells out to git archive / git bundle, which have to walk history
and pack files. Repeat callers with the same request shape can
reuse the result.
- Both endpoints already streamed a ChildArchiveReader to callers; the
cache layer keeps the streaming contract but tees the bytes onto
disk so subsequent calls hit a regular tokio::fs::File.
Cache keys (helper.rs):
- get_archive_cache_key: resolves the commit to an oid (so a moved
branch ref does NOT hit a stale archive), serializes a
GetArchiveCacheKey struct with format / prefix / path / exclude
via serde_json, then sha1-hashes with gix:#️⃣:hasher(Sha1).
exclude is sorted for key derivation but the on-wire command keeps
the caller's original order.
- bundle_cache_key: refs joined with "\n" hashed as-is (no
resolution — bundle refs are interpreted relative to the source
repo, not the destination's branch state).
On-disk layout (helper.rs):
- archive_store_dir(repo_dir) = <repo>/archive
- archive_file_path(repo_dir, key, ext) = <repo>/archive/<key>.<ext>
- tmp_archive_path(final) = <final>.part.<pid> for concurrent
writers without clobbering each other. The pid suffix is enough for
the single-process contract; the reader renames .part.<pid> to
final on EOF.
Streaming persist (helper.rs::PersistReader):
- AsyncRead + Unpin + Send wrapper around (child stdout, persist
tokio::fs::File, tmp path, final path).
- Each successful read writes the same bytes to the .part.<pid> file
via AsyncWriteExt.::write_all, then flushes on EOF and renames tmp
-> final. A short read with an early EOF renames back to keep the
contract: either the final file is the full archive, or no cache
file exists.
- The reader reports EOF to the caller exactly when the underlying
child stdout reports EOF; bytes already written to disk on a
downstream error are dropped on rename failure.
usecase.rs:
- get_archive / create_bundle: cache hit short-circuits to a plain
ChildArchiveReader over the cached file (no child process spawned).
Cache miss: spawns the child as before, but wraps the stdout in
PersistReader so the bytes are streamed to the caller AND teed to
<final>.part.<pid>. On EOF the tmp file is renamed to <final>.
- Public ChildArchiveReader signature drops the Sync bound on its
inner reader (BufReader<Box<dyn AsyncRead + Unpin + Send>>) — Sync
was not load-bearing for the read path and is annoying to satisfy
for the PersistReader wrapper. Existing call sites continue to
compile because none of them relied on Sync.
Tests (tests/archive.rs, new, 4 cases):
- get_archive_caches_after_first_call: first call produces bytes;
second call with the same request reuses the cached file
(verified by removing the underlying source file between calls).
- get_archive_changes_key_when_commit_advances: an archive for an
older oid must not be served for a newer oid, and vice versa.
- create_bundle_caches_per_ref_set: bundle for refs=[a,b] is cached
separately from refs=[a].
- archive_cache_survives_orphan_temp: an orphaned .part.<pid> from a
crashed previous run does not poison a new call; the new call
recreates the tmp file cleanly and ends up with a valid final.
- Uses the TestFacade / temp_repo / git_commit / write_file /
index_commit / baby_of fixtures shared with tests/tree.rs. A
per-process AtomicU32 counter gives each test a unique temp dir.
CI: cargo build + cargo test (68 passed, +4 new) green.
BREAKING: ChildArchiveReader's inner reader bound changed from
AsyncRead + Unpin + Send + Sync to AsyncRead + Unpin + Send (drop
Sync). The struct itself still satisfies Send + Sync because all its
fields do; only the inner AsyncRead bound relaxed. No in-tree caller
required Sync.
Notes:
- The cache is single-process, like the foyer cache. Two processes
racing on the same <repo>/archive/<key> could both write a
.part.<pid> file with different pids; both could succeed and the
last rename wins. Acceptable for now; documented in AGENTS.md is
not needed since this lives inside the repo, not in a shared
cache directory.
- Invalidating the cache on commit advance: the key depends on the
resolved oid, so a force-push to a new oid naturally invalidates.
We do NOT walk the archive dir to garbage-collect old entries;
callers can rm -rf <repo>/archive when they want to reclaim
space.
|
||
|
zhenyi
|
61f3faed38 |
feat(tree): add tree_entries_with_latest + tighten RepositoryFacade sync bound
Public API additions (src/tree):
- src/tree/types.rs: new TreeEntryWithLatest { entry, latest } pair
(entry: TreeEntry, latest: CommitInfo) with Serialize/Deserialize
derives so it can flow through the cache payload path.
- src/tree/helper.rs: parse_ls_tree_entries — NUL/newline-delimited
ls-tree stdout -> Vec<TreeEntry>. Mirrors the git ls-tree -r -t
layout (mode SP type SP oid TAB name). Errors are reported with the
1-based line index via BabyError::CommitParse for parity with
parse_commit_output.
- src/tree/usecase.rs: GitBaby::tree_entries_with_latest(revision, path)
-> Vec<TreeEntryWithLatest>. Implementation:
1) spawn a single `git ls-tree -r -t <rev> -- <path>` to enumerate
blobs/trees,
2) parse with parse_ls_tree_entries,
3) fan out N parallel `tree_latest_commit(rev, path)` lookups via
tokio::task::JoinSet,
4) reassemble preserving the ls-tree order.
Subtree paths are normalized: tree entries get a trailing `/` for
the lookup so callers don't have to know whether a name is a tree.
Path safety reuses share::validate_local_no_escape, and revision
reuses share::validate_revision_non_empty.
- src/tree/mod.rs: re-export TreeEntryWithLatest alongside the
existing tree public surface.
Trait bound change (src/repo.rs):
- RepositoryFacade now requires Send + Sync in addition to 'static.
This is needed so the JoinSet-driven fan-out above can share the
Arc<dyn RepositoryFacade> across spawned tasks. Existing single-task
implementations are unaffected; the only callers that needed to
add Sync explicitly are those keeping non-Sync state inside their
facade (none in-tree).
Tests (tests/tree.rs, new):
- tree_entries_with_latest_whole_tree: enumerate a flat repo and
verify every (path, latest_commit_id) pair round-trips against the
revwalk.
- tree_entries_with_latest_subtree: scope to a sub-path and check
only entries under it are returned, in ls-tree order.
- tree_entries_with_latest_rejects_bad_input: empty revision and
path-traversal attempts both surface BabyError without spawning a
child process.
- Uses the existing TestFacade skeleton, temp_repo helper, and
git_commit / index_commit / write_file fixtures. Counter uses
AtomicU32 + COUNTER.fetch_add for unique temp dirs across tests,
matching the pattern from other integration tests.
CI: cargo build + cargo test (64 passed, +3 new) green.
Notes:
- TreeEntryWithLatest has #[serde(default)] on no fields today, but
it follows the additive-only convention documented in AGENTS.md
(cached serde payloads only ever add new fields).
- BREAKING for RepositoryFacade implementors that are not Sync.
None ship in-tree; downstream consumers must add Sync if they hold
non-Sync interior state.
|
||
|
zhenyi
|
5d32290cf1 |
feat(cache): split cache module into submodules and wire invalidation hooks
Two-part change:
1) Submodule split of src/cache/ (was a single ~150-line mod.rs)
- src/cache/config.rs: CacheConfig, defaults, default_cache_dir(),
DEFAULT_NAME / DEFAULT_MEMORY_CAPACITY_BYTES /
DEFAULT_DISK_CAPACITY_BYTES constants
- src/cache/key.rs: CacheKey / CacheValue type aliases and key
derivation helpers (rev -> oid, blob-exists negatives)
- src/cache/store.rs: CacheStore — the foyer HybridCache wrapper
(open / close / get / insert / remove / contains), all errors
mapped to BabyError::Cache with op-tagged source
- src/cache/pools.rs: CachePools — multi-store aggregator used by
the global singleton; replaces the old monolithic CacheStore
- src/cache/stats.rs: CacheStats + counter accessors
- src/cache/invalidation.rs: hook_ref_updated / hook_blob_written
(single-process invalidation hooks, fired from usecase impls)
- src/cache/mod.rs: now just submodule re-exports of
(config, global, invalidation, key, pools, stats, store)
- src/cache/global.rs: GITBABY_CACHE singleton + init / try /
shutdown now operate on Arc<CachePools> instead of Arc<CacheStore>
2) Cache integration in usecase modules
- src/blob/usecase.rs: read paths go through cache.get / insert;
invalidation hooks fire on write
- src/commit/usecase.rs: commit-list results cached, invalidated on
commit write
- src/branch/usecase.rs: branch-list results cached, invalidated on
ref update
- src/refs/usecase.rs: lookup-cache + ref-update hook
- src/tags/usecase.rs: tag-info cache + write hook
3) AGENTS.md
- Document that cached serde payloads (CommitInfo, Vec<TreeEntry>,
etc.) follow additive-only compatibility: never rename/remove a
field, every new field gets #[serde(default)] so stale payloads
still deserialize.
- Note that the foyer cache is single-process: fixed on-disk dir
per config + in-process invalidation hooks. Sharing across
processes on the same repo can leave stale metadata; do not do
that.
CI: cargo build + cargo test (61 passed) green.
BREAKING: the public cache surface moved. Anyone reaching into
cache::CacheStore directly should switch to cache::CachePools (or
cache::global::try_global_cache). The high-level helpers in
cache::global keep the same name; only the inner type changed.
|
||
|
zhenyi
|
4d5e5ba5f9 |
feat(server): add streaming-first server scaffolding
Introduce src/server/ as a streaming-first git/lfs server surface. All
request and response bodies cross the API as
Box<dyn AsyncRead + Unpin + Send + Sync> (alias BoxedAsyncRead) — no
Vec<u8> bodies, no read_to_end convenience, no http crate.
What's in the box:
- src/server/mod.rs: re-exports + pipe re-export of
command::cmd::DEFAULT_OUTPUT_CAP_BYTES as server::DEFAULT_OUTPUT_CAP_BYTES
- src/server/backend.rs: ServerBackend async-trait (authorize,
list_refs_stream, produce_pack_stream, ingest_pack_stream,
lfs_get_stream, lfs_put_stream). Every method currently returns
ServerError::Unimplemented("...") placeholders — the wiring is
real, the bodies are intentionally empty.
- src/server/request.rs: ServerRequest (not Clone, body is a boxed
reader), HeaderMap = HashMap<String, String>, RefUpdateBatch
(oid strings, no gix::ObjectId), BoxedAsyncRead alias
- src/server/stream.rs: StreamGuard (Clone + Send + Sync) carrying
cancel() / handle() so spawned workers can race the request
- src/server/endpoint.rs, error.rs, limits.rs: EndpointKind enum,
ServerError, DEFAULT_LFS_OBJECT_CAP_BYTES (5 GiB cap, separate from
the 64 MiB output cap)
- tests/server.rs: PlaceholderBackend skeleton + ENV_LOCK +
200 ms cancel convention reused from tests/env.rs and tests/pipe.rs
Public API impact on GitBaby:
- Drop pipe from GitBaby::new — the server owns its own stream
lifecycle, so the Pipe parameter is no longer required at
construction time. Existing callers must be updated:
- before: GitBaby::new(facade, pipe)
- after: GitBaby::new(facade)
Documentation:
- AGENTS.md gains a "Planned but unimplemented (streaming-first
server)" section documenting the deliberately-empty contracts so
future contributors don't "fill them in" without a concrete
git/lfs protocol task.
CI: cargo build + cargo test (61 total, 23 new from server suite)
green.
|
||
|
zhenyi
|
66045dd971 |
feat(cache): add hybrid memory/disk cache layer on foyer
Introduce a two-tier cache for blob payloads backed by the foyer crate
(memory + disk). The cache is configured via CacheConfig and exposed
through src/cache/{mod,global}.rs.
Why foyer:
- HybridCache gives us bounded RAM plus disk overflow without bringing
in our own eviction / spill logic.
- The runtime-tokio feature plays cleanly with the existing tokio stack.
Touches:
- src/cache/mod.rs: CacheConfig, CacheStore, CacheKey/Value aliases,
default-into-disk fallback (CacheConfig::default)
- src/cache/global.rs: process-wide singleton accessor for the default
cache
- src/error.rs: new BabyError::Cache { op, source } variant wrapping
foyer::Error so cache failures share the existing error pipeline
- Cargo.toml: add foyer = "0.22.3" with runtime-tokio feature
- src/lib.rs: pub mod cache; (wiring only — no public API on GitBaby
changes yet)
- src/tree/types.rs: drive-by import reorder (serde before std::path)
to match rustfmt
CI: cargo build + cargo test (38 passed) green.
Skipped intentionally: no git/lfs protocol logic is added here; this
commit is purely the storage substrate.
|
||
|
zhenyi
|
916b5f3176 |
feat(types): add Serialize/Deserialize derives to all types.rs
Add the serde 'twins' (Serialize + Deserialize) to every public struct and enum in src/*/types.rs so data types can be serialized via JSON, bincode, postcard, etc. Dependency changes: - enable 'serde' feature on gix (for gix::ObjectId) - enable 'serde' feature on time (for time::OffsetDateTime) Skipped: - src/archive/types.rs::ChildArchiveReader<R> — contains a tokio::process::Child field, semantically not serializable Coverage (18 modules): archive, blame, blob, branch, cleanup, commit, compare, config, conflict, diff, merge, refs, remote, setup, submodule, tags, tree Notes: - field names kept snake_case (no rename_all remap) - existing PartialEq / Eq / Default / Copy derives preserved - cargo build + cargo test (38 passed) green |
||
|
zhenyi
|
680411c4fb |
chore: initialize gitbaby repository
gitbaby is an asynchronous Rust library wrapping git operations, built on gix + tokio + async-trait. Module layout: - archive, blame, blob, branch, cleanup, commit, compare, config, conflict, diff, merge, refs, remote, setup, share, submodule, tags, tree (feature modules) - command: command scheduler (cmd/env/pipe/context/error) - repo: RepositoryFacade trait (consumers implement, exposing a gix::Repository) Features: - async API on tokio 1.53 - order-preserving Env (Vec<(String, String)>) - hand-written BabyError / CmdError, not using thiserror derive - integration tests covering cmd_run / context / env / pipe (env keys prefixed for isolation) CI: none, standard cargo build / test / clippy / fmt only |