main
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
zhenyi
|
240b5214dd |
feat(archive): add on-disk archive/bundle cache with streaming persist
Adds a single-process on-disk cache for the two archive-producing entry
points: GitBaby::get_archive and GitBaby::create_bundle. The cache
lives at <repo>/archive/<key>.<ext>, so it's local to the repository
and cleaned up with the working tree (no separate cache directory to
track).
Why this layer:
- Archive generation is the most expensive routine in the lib: it
shells out to git archive / git bundle, which have to walk history
and pack files. Repeat callers with the same request shape can
reuse the result.
- Both endpoints already streamed a ChildArchiveReader to callers; the
cache layer keeps the streaming contract but tees the bytes onto
disk so subsequent calls hit a regular tokio::fs::File.
Cache keys (helper.rs):
- get_archive_cache_key: resolves the commit to an oid (so a moved
branch ref does NOT hit a stale archive), serializes a
GetArchiveCacheKey struct with format / prefix / path / exclude
via serde_json, then sha1-hashes with gix:#️⃣:hasher(Sha1).
exclude is sorted for key derivation but the on-wire command keeps
the caller's original order.
- bundle_cache_key: refs joined with "\n" hashed as-is (no
resolution — bundle refs are interpreted relative to the source
repo, not the destination's branch state).
On-disk layout (helper.rs):
- archive_store_dir(repo_dir) = <repo>/archive
- archive_file_path(repo_dir, key, ext) = <repo>/archive/<key>.<ext>
- tmp_archive_path(final) = <final>.part.<pid> for concurrent
writers without clobbering each other. The pid suffix is enough for
the single-process contract; the reader renames .part.<pid> to
final on EOF.
Streaming persist (helper.rs::PersistReader):
- AsyncRead + Unpin + Send wrapper around (child stdout, persist
tokio::fs::File, tmp path, final path).
- Each successful read writes the same bytes to the .part.<pid> file
via AsyncWriteExt.::write_all, then flushes on EOF and renames tmp
-> final. A short read with an early EOF renames back to keep the
contract: either the final file is the full archive, or no cache
file exists.
- The reader reports EOF to the caller exactly when the underlying
child stdout reports EOF; bytes already written to disk on a
downstream error are dropped on rename failure.
usecase.rs:
- get_archive / create_bundle: cache hit short-circuits to a plain
ChildArchiveReader over the cached file (no child process spawned).
Cache miss: spawns the child as before, but wraps the stdout in
PersistReader so the bytes are streamed to the caller AND teed to
<final>.part.<pid>. On EOF the tmp file is renamed to <final>.
- Public ChildArchiveReader signature drops the Sync bound on its
inner reader (BufReader<Box<dyn AsyncRead + Unpin + Send>>) — Sync
was not load-bearing for the read path and is annoying to satisfy
for the PersistReader wrapper. Existing call sites continue to
compile because none of them relied on Sync.
Tests (tests/archive.rs, new, 4 cases):
- get_archive_caches_after_first_call: first call produces bytes;
second call with the same request reuses the cached file
(verified by removing the underlying source file between calls).
- get_archive_changes_key_when_commit_advances: an archive for an
older oid must not be served for a newer oid, and vice versa.
- create_bundle_caches_per_ref_set: bundle for refs=[a,b] is cached
separately from refs=[a].
- archive_cache_survives_orphan_temp: an orphaned .part.<pid> from a
crashed previous run does not poison a new call; the new call
recreates the tmp file cleanly and ends up with a valid final.
- Uses the TestFacade / temp_repo / git_commit / write_file /
index_commit / baby_of fixtures shared with tests/tree.rs. A
per-process AtomicU32 counter gives each test a unique temp dir.
CI: cargo build + cargo test (68 passed, +4 new) green.
BREAKING: ChildArchiveReader's inner reader bound changed from
AsyncRead + Unpin + Send + Sync to AsyncRead + Unpin + Send (drop
Sync). The struct itself still satisfies Send + Sync because all its
fields do; only the inner AsyncRead bound relaxed. No in-tree caller
required Sync.
Notes:
- The cache is single-process, like the foyer cache. Two processes
racing on the same <repo>/archive/<key> could both write a
.part.<pid> file with different pids; both could succeed and the
last rename wins. Acceptable for now; documented in AGENTS.md is
not needed since this lives inside the repo, not in a shared
cache directory.
- Invalidating the cache on commit advance: the key depends on the
resolved oid, so a force-push to a new oid naturally invalidates.
We do NOT walk the archive dir to garbage-collect old entries;
callers can rm -rf <repo>/archive when they want to reclaim
space.
|