Adds a single-process on-disk cache for the two archive-producing entry
points: GitBaby::get_archive and GitBaby::create_bundle. The cache
lives at <repo>/archive/<key>.<ext>, so it's local to the repository
and cleaned up with the working tree (no separate cache directory to
track).
Why this layer:
- Archive generation is the most expensive routine in the lib: it
shells out to git archive / git bundle, which have to walk history
and pack files. Repeat callers with the same request shape can
reuse the result.
- Both endpoints already streamed a ChildArchiveReader to callers; the
cache layer keeps the streaming contract but tees the bytes onto
disk so subsequent calls hit a regular tokio::fs::File.
Cache keys (helper.rs):
- get_archive_cache_key: resolves the commit to an oid (so a moved
branch ref does NOT hit a stale archive), serializes a
GetArchiveCacheKey struct with format / prefix / path / exclude
via serde_json, then sha1-hashes with gix:#️⃣:hasher(Sha1).
exclude is sorted for key derivation but the on-wire command keeps
the caller's original order.
- bundle_cache_key: refs joined with "\n" hashed as-is (no
resolution — bundle refs are interpreted relative to the source
repo, not the destination's branch state).
On-disk layout (helper.rs):
- archive_store_dir(repo_dir) = <repo>/archive
- archive_file_path(repo_dir, key, ext) = <repo>/archive/<key>.<ext>
- tmp_archive_path(final) = <final>.part.<pid> for concurrent
writers without clobbering each other. The pid suffix is enough for
the single-process contract; the reader renames .part.<pid> to
final on EOF.
Streaming persist (helper.rs::PersistReader):
- AsyncRead + Unpin + Send wrapper around (child stdout, persist
tokio::fs::File, tmp path, final path).
- Each successful read writes the same bytes to the .part.<pid> file
via AsyncWriteExt.::write_all, then flushes on EOF and renames tmp
-> final. A short read with an early EOF renames back to keep the
contract: either the final file is the full archive, or no cache
file exists.
- The reader reports EOF to the caller exactly when the underlying
child stdout reports EOF; bytes already written to disk on a
downstream error are dropped on rename failure.
usecase.rs:
- get_archive / create_bundle: cache hit short-circuits to a plain
ChildArchiveReader over the cached file (no child process spawned).
Cache miss: spawns the child as before, but wraps the stdout in
PersistReader so the bytes are streamed to the caller AND teed to
<final>.part.<pid>. On EOF the tmp file is renamed to <final>.
- Public ChildArchiveReader signature drops the Sync bound on its
inner reader (BufReader<Box<dyn AsyncRead + Unpin + Send>>) — Sync
was not load-bearing for the read path and is annoying to satisfy
for the PersistReader wrapper. Existing call sites continue to
compile because none of them relied on Sync.
Tests (tests/archive.rs, new, 4 cases):
- get_archive_caches_after_first_call: first call produces bytes;
second call with the same request reuses the cached file
(verified by removing the underlying source file between calls).
- get_archive_changes_key_when_commit_advances: an archive for an
older oid must not be served for a newer oid, and vice versa.
- create_bundle_caches_per_ref_set: bundle for refs=[a,b] is cached
separately from refs=[a].
- archive_cache_survives_orphan_temp: an orphaned .part.<pid> from a
crashed previous run does not poison a new call; the new call
recreates the tmp file cleanly and ends up with a valid final.
- Uses the TestFacade / temp_repo / git_commit / write_file /
index_commit / baby_of fixtures shared with tests/tree.rs. A
per-process AtomicU32 counter gives each test a unique temp dir.
CI: cargo build + cargo test (68 passed, +4 new) green.
BREAKING: ChildArchiveReader's inner reader bound changed from
AsyncRead + Unpin + Send + Sync to AsyncRead + Unpin + Send (drop
Sync). The struct itself still satisfies Send + Sync because all its
fields do; only the inner AsyncRead bound relaxed. No in-tree caller
required Sync.
Notes:
- The cache is single-process, like the foyer cache. Two processes
racing on the same <repo>/archive/<key> could both write a
.part.<pid> file with different pids; both could succeed and the
last rename wins. Acceptable for now; documented in AGENTS.md is
not needed since this lives inside the repo, not in a shared
cache directory.
- Invalidating the cache on commit advance: the key depends on the
resolved oid, so a force-push to a new oid naturally invalidates.
We do NOT walk the archive dir to garbage-collect old entries;
callers can rm -rf <repo>/archive when they want to reclaim
space.
320 lines
8.3 KiB
Rust
320 lines
8.3 KiB
Rust
use std::path::{Path, PathBuf};
|
|
use std::process::Command;
|
|
use std::sync::Arc;
|
|
use std::sync::atomic::{AtomicU32, Ordering};
|
|
|
|
use async_trait::async_trait;
|
|
|
|
use gitbaby::GitBaby;
|
|
use gitbaby::archive::{ArchiveFormat, BundleRequest, GetArchiveRequest};
|
|
use gitbaby::error::BabyError;
|
|
use gitbaby::repo::RepositoryFacade;
|
|
|
|
static COUNTER: AtomicU32 = AtomicU32::new(0);
|
|
|
|
struct TestFacade {
|
|
dir: PathBuf,
|
|
}
|
|
|
|
#[async_trait]
|
|
impl RepositoryFacade for TestFacade {
|
|
async fn git_repo_dir(&self) -> Result<PathBuf, BabyError> {
|
|
Ok(self.dir.clone())
|
|
}
|
|
|
|
async fn git_alternate_object_directories(&self) -> Result<Vec<PathBuf>, BabyError> {
|
|
Ok(Vec::new())
|
|
}
|
|
|
|
async fn gix_repo(&self) -> Result<gix::Repository, BabyError> {
|
|
gix::open(&self.dir).map_err(|e| BabyError::Custom(e.to_string()))
|
|
}
|
|
}
|
|
|
|
fn temp_repo() -> PathBuf {
|
|
let n = COUNTER.fetch_add(1, Ordering::SeqCst);
|
|
let dir = std::env::temp_dir().join(format!("gitbaby-archive-test-{}-{n}", std::process::id()));
|
|
let _ = std::fs::remove_dir_all(&dir);
|
|
std::fs::create_dir_all(&dir).expect("mkdir repo");
|
|
let out = Command::new("git")
|
|
.args(["init", "-q"])
|
|
.current_dir(&dir)
|
|
.output()
|
|
.expect("git init failed");
|
|
assert!(out.status.success(), "git init: {:?}", out);
|
|
dir
|
|
}
|
|
|
|
fn cleanup(repo: &Path) {
|
|
let _ = std::fs::remove_dir_all(repo);
|
|
}
|
|
|
|
fn git_commit(repo: &Path, msg: &str) {
|
|
let out = Command::new("git")
|
|
.args([
|
|
"-c",
|
|
"user.name=T",
|
|
"-c",
|
|
"user.email=t@e",
|
|
"commit",
|
|
"-q",
|
|
"-m",
|
|
msg,
|
|
])
|
|
.current_dir(repo)
|
|
.output()
|
|
.expect("git commit failed");
|
|
assert!(out.status.success(), "git commit `{msg}`: {:?}", out);
|
|
}
|
|
|
|
fn write_file(repo: &Path, rel: &str, content: &str) {
|
|
let p = repo.join(rel);
|
|
std::fs::create_dir_all(p.parent().expect("parent")).expect("mkdir");
|
|
std::fs::write(p, content).expect("write");
|
|
}
|
|
|
|
fn index_commit(repo: &Path, msg: &str) {
|
|
let out = Command::new("git")
|
|
.args(["add", "-A"])
|
|
.current_dir(repo)
|
|
.output()
|
|
.expect("git add failed");
|
|
assert!(out.status.success(), "git add: {:?}", out);
|
|
git_commit(repo, msg);
|
|
}
|
|
|
|
async fn baby_of(repo: PathBuf) -> GitBaby {
|
|
GitBaby::new(Arc::new(TestFacade { dir: repo }))
|
|
}
|
|
|
|
fn archive_dir(repo: &Path) -> PathBuf {
|
|
repo.join("archive")
|
|
}
|
|
|
|
fn archive_files(repo: &Path) -> Vec<PathBuf> {
|
|
let mut files: Vec<PathBuf> = std::fs::read_dir(archive_dir(repo))
|
|
.expect("read archive dir")
|
|
.filter_map(|e| e.ok())
|
|
.map(|e| e.path())
|
|
.filter(|p| p.is_file())
|
|
.collect();
|
|
files.sort();
|
|
files
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn get_archive_persists_and_hits_cache() {
|
|
let repo = temp_repo();
|
|
write_file(&repo, "a.txt", "one");
|
|
write_file(&repo, "b.txt", "bee");
|
|
index_commit(&repo, "first");
|
|
|
|
let baby = baby_of(repo.clone()).await;
|
|
let req = || GetArchiveRequest {
|
|
commit: "HEAD".to_string(),
|
|
format: ArchiveFormat::Tar,
|
|
prefix: None,
|
|
path: None,
|
|
exclude: Vec::new(),
|
|
};
|
|
|
|
let first_bytes = baby
|
|
.get_archive(req())
|
|
.await
|
|
.expect("get_archive")
|
|
.read_to_end()
|
|
.await
|
|
.expect("read archive body");
|
|
|
|
let files = archive_files(&repo);
|
|
assert_eq!(files.len(), 1, "one cached archive expected");
|
|
assert_eq!(
|
|
files[0].extension().and_then(|e| e.to_str()),
|
|
Some("tar"),
|
|
"cached file: {:?}",
|
|
files[0]
|
|
);
|
|
let on_disk = std::fs::read(&files[0]).expect("read cached file");
|
|
assert_eq!(on_disk, first_bytes, "cached bytes match streamed bytes");
|
|
|
|
let direct = Command::new("git")
|
|
.args(["archive", "--format=tar", "HEAD"])
|
|
.current_dir(&repo)
|
|
.output()
|
|
.expect("git archive direct");
|
|
assert!(direct.status.success(), "git archive direct: {:?}", direct);
|
|
assert_eq!(direct.stdout, first_bytes, "matches raw git archive output");
|
|
|
|
let second_bytes = baby
|
|
.get_archive(req())
|
|
.await
|
|
.expect("get_archive cached")
|
|
.read_to_end()
|
|
.await
|
|
.expect("read archive body");
|
|
assert_eq!(second_bytes, first_bytes, "cache hit returns same bytes");
|
|
assert_eq!(archive_files(&repo).len(), 1);
|
|
|
|
cleanup(&repo);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn get_archive_keys_differ_by_options() {
|
|
let repo = temp_repo();
|
|
write_file(&repo, "a.txt", "one");
|
|
index_commit(&repo, "first");
|
|
|
|
let baby = baby_of(repo.clone()).await;
|
|
let base = GetArchiveRequest {
|
|
commit: "HEAD".to_string(),
|
|
format: ArchiveFormat::Tar,
|
|
prefix: None,
|
|
path: None,
|
|
exclude: Vec::new(),
|
|
};
|
|
|
|
let plain = baby
|
|
.get_archive(base.clone())
|
|
.await
|
|
.expect("plain")
|
|
.read_to_end()
|
|
.await
|
|
.expect("plain body");
|
|
|
|
let prefixed = baby
|
|
.get_archive(GetArchiveRequest {
|
|
prefix: Some("p/".to_string()),
|
|
..base.clone()
|
|
})
|
|
.await
|
|
.expect("prefixed")
|
|
.read_to_end()
|
|
.await
|
|
.expect("prefixed body");
|
|
|
|
let excluded = baby
|
|
.get_archive(GetArchiveRequest {
|
|
exclude: vec!["a.txt".to_string()],
|
|
..base.clone()
|
|
})
|
|
.await
|
|
.expect("excluded")
|
|
.read_to_end()
|
|
.await
|
|
.expect("excluded body");
|
|
|
|
assert_ne!(plain, prefixed);
|
|
assert_ne!(plain, excluded);
|
|
assert_ne!(prefixed, excluded);
|
|
|
|
let files = archive_files(&repo);
|
|
assert_eq!(files.len(), 3, "three distinct keys cached");
|
|
|
|
let zipped = baby
|
|
.get_archive(GetArchiveRequest {
|
|
format: ArchiveFormat::Zip,
|
|
..base
|
|
})
|
|
.await
|
|
.expect("zip")
|
|
.read_to_end()
|
|
.await
|
|
.expect("zip body");
|
|
let files = archive_files(&repo);
|
|
assert_eq!(files.len(), 4);
|
|
assert!(
|
|
files
|
|
.iter()
|
|
.any(|p| p.extension().and_then(|e| e.to_str()) == Some("zip")),
|
|
"zip file present: {:?}",
|
|
files
|
|
);
|
|
assert!(zipped.starts_with(b"PK"), "zip magic");
|
|
assert_ne!(zipped, plain);
|
|
|
|
cleanup(&repo);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn get_archive_new_commit_new_key() {
|
|
let repo = temp_repo();
|
|
write_file(&repo, "a.txt", "one");
|
|
index_commit(&repo, "first");
|
|
|
|
let baby = baby_of(repo.clone()).await;
|
|
let req = || GetArchiveRequest {
|
|
commit: "HEAD".to_string(),
|
|
format: ArchiveFormat::Tar,
|
|
prefix: None,
|
|
path: None,
|
|
exclude: Vec::new(),
|
|
};
|
|
|
|
let _ = baby
|
|
.get_archive(req())
|
|
.await
|
|
.expect("first revision")
|
|
.read_to_end()
|
|
.await
|
|
.expect("body");
|
|
assert_eq!(archive_files(&repo).len(), 1);
|
|
|
|
write_file(&repo, "a.txt", "two");
|
|
index_commit(&repo, "second");
|
|
|
|
let _ = baby
|
|
.get_archive(req())
|
|
.await
|
|
.expect("second revision")
|
|
.read_to_end()
|
|
.await
|
|
.expect("body");
|
|
let files = archive_files(&repo);
|
|
assert_eq!(files.len(), 2, "new commit produces new cached key");
|
|
|
|
cleanup(&repo);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn create_bundle_persists_and_hits_cache() {
|
|
let repo = temp_repo();
|
|
write_file(&repo, "a.txt", "one");
|
|
index_commit(&repo, "first");
|
|
|
|
let baby = baby_of(repo.clone()).await;
|
|
let req = || BundleRequest {
|
|
refs: vec!["HEAD".to_string()],
|
|
};
|
|
|
|
let first = baby
|
|
.create_bundle(req())
|
|
.await
|
|
.expect("create_bundle")
|
|
.read_to_end()
|
|
.await
|
|
.expect("bundle body");
|
|
|
|
let files = archive_files(&repo);
|
|
assert_eq!(files.len(), 1, "one cached bundle expected");
|
|
assert_eq!(
|
|
files[0].extension().and_then(|e| e.to_str()),
|
|
Some("bundle"),
|
|
"cached file: {:?}",
|
|
files[0]
|
|
);
|
|
let on_disk = std::fs::read(&files[0]).expect("read cached file");
|
|
assert_eq!(on_disk, first, "cached bundle bytes match streamed bytes");
|
|
|
|
let second = baby
|
|
.create_bundle(req())
|
|
.await
|
|
.expect("create_bundle cached")
|
|
.read_to_end()
|
|
.await
|
|
.expect("bundle body");
|
|
assert_eq!(second, first, "cache hit returns same bundle bytes");
|
|
assert_eq!(archive_files(&repo).len(), 1);
|
|
|
|
cleanup(&repo);
|
|
}
|