feat(archive): add on-disk archive/bundle cache with streaming persist
Adds a single-process on-disk cache for the two archive-producing entry
points: GitBaby::get_archive and GitBaby::create_bundle. The cache
lives at <repo>/archive/<key>.<ext>, so it's local to the repository
and cleaned up with the working tree (no separate cache directory to
track).
Why this layer:
- Archive generation is the most expensive routine in the lib: it
shells out to git archive / git bundle, which have to walk history
and pack files. Repeat callers with the same request shape can
reuse the result.
- Both endpoints already streamed a ChildArchiveReader to callers; the
cache layer keeps the streaming contract but tees the bytes onto
disk so subsequent calls hit a regular tokio::fs::File.
Cache keys (helper.rs):
- get_archive_cache_key: resolves the commit to an oid (so a moved
branch ref does NOT hit a stale archive), serializes a
GetArchiveCacheKey struct with format / prefix / path / exclude
via serde_json, then sha1-hashes with gix:#️⃣:hasher(Sha1).
exclude is sorted for key derivation but the on-wire command keeps
the caller's original order.
- bundle_cache_key: refs joined with "\n" hashed as-is (no
resolution — bundle refs are interpreted relative to the source
repo, not the destination's branch state).
On-disk layout (helper.rs):
- archive_store_dir(repo_dir) = <repo>/archive
- archive_file_path(repo_dir, key, ext) = <repo>/archive/<key>.<ext>
- tmp_archive_path(final) = <final>.part.<pid> for concurrent
writers without clobbering each other. The pid suffix is enough for
the single-process contract; the reader renames .part.<pid> to
final on EOF.
Streaming persist (helper.rs::PersistReader):
- AsyncRead + Unpin + Send wrapper around (child stdout, persist
tokio::fs::File, tmp path, final path).
- Each successful read writes the same bytes to the .part.<pid> file
via AsyncWriteExt.::write_all, then flushes on EOF and renames tmp
-> final. A short read with an early EOF renames back to keep the
contract: either the final file is the full archive, or no cache
file exists.
- The reader reports EOF to the caller exactly when the underlying
child stdout reports EOF; bytes already written to disk on a
downstream error are dropped on rename failure.
usecase.rs:
- get_archive / create_bundle: cache hit short-circuits to a plain
ChildArchiveReader over the cached file (no child process spawned).
Cache miss: spawns the child as before, but wraps the stdout in
PersistReader so the bytes are streamed to the caller AND teed to
<final>.part.<pid>. On EOF the tmp file is renamed to <final>.
- Public ChildArchiveReader signature drops the Sync bound on its
inner reader (BufReader<Box<dyn AsyncRead + Unpin + Send>>) — Sync
was not load-bearing for the read path and is annoying to satisfy
for the PersistReader wrapper. Existing call sites continue to
compile because none of them relied on Sync.
Tests (tests/archive.rs, new, 4 cases):
- get_archive_caches_after_first_call: first call produces bytes;
second call with the same request reuses the cached file
(verified by removing the underlying source file between calls).
- get_archive_changes_key_when_commit_advances: an archive for an
older oid must not be served for a newer oid, and vice versa.
- create_bundle_caches_per_ref_set: bundle for refs=[a,b] is cached
separately from refs=[a].
- archive_cache_survives_orphan_temp: an orphaned .part.<pid> from a
crashed previous run does not poison a new call; the new call
recreates the tmp file cleanly and ends up with a valid final.
- Uses the TestFacade / temp_repo / git_commit / write_file /
index_commit / baby_of fixtures shared with tests/tree.rs. A
per-process AtomicU32 counter gives each test a unique temp dir.
CI: cargo build + cargo test (68 passed, +4 new) green.
BREAKING: ChildArchiveReader's inner reader bound changed from
AsyncRead + Unpin + Send + Sync to AsyncRead + Unpin + Send (drop
Sync). The struct itself still satisfies Send + Sync because all its
fields do; only the inner AsyncRead bound relaxed. No in-tree caller
required Sync.
Notes:
- The cache is single-process, like the foyer cache. Two processes
racing on the same <repo>/archive/<key> could both write a
.part.<pid> file with different pids; both could succeed and the
last rename wins. Acceptable for now; documented in AGENTS.md is
not needed since this lives inside the repo, not in a shared
cache directory.
- Invalidating the cache on commit advance: the key depends on the
resolved oid, so a force-push to a new oid naturally invalidates.
We do NOT walk the archive dir to garbage-collect old entries;
callers can rm -rf <repo>/archive when they want to reclaim
space.
This commit is contained in:
parent
61f3faed38
commit
240b5214dd
@ -1,9 +1,230 @@
|
||||
use std::path::Path;
|
||||
use std::io;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use std::task::{Context, Poll};
|
||||
|
||||
use serde::Serialize;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
use crate::archive::types::{ArchiveFormat, BundleRequest, GetArchiveRequest};
|
||||
use crate::command::cmd::Cmd;
|
||||
use crate::command::env::Env;
|
||||
use crate::error::BabyError;
|
||||
|
||||
/// 缓存目录名,位于仓库根下(裸仓库无工作区,无污染风险)。
|
||||
const ARCHIVE_DIR_NAME: &str = "archive";
|
||||
|
||||
/// 已打包产物在仓库内的落盘目录。
|
||||
pub fn archive_store_dir(repo_dir: &Path) -> PathBuf {
|
||||
repo_dir.join(ARCHIVE_DIR_NAME)
|
||||
}
|
||||
|
||||
/// 归档格式对应的文件扩展名(bundle 固定为 `bundle`)。
|
||||
pub fn format_ext(format: ArchiveFormat) -> &'static str {
|
||||
match format {
|
||||
ArchiveFormat::Tar => "tar",
|
||||
ArchiveFormat::TarGz => "tar.gz",
|
||||
ArchiveFormat::TarBz2 => "tar.bz2",
|
||||
ArchiveFormat::Zip => "zip",
|
||||
}
|
||||
}
|
||||
|
||||
/// 缓存文件最终路径:`<repo>/archive/<key>.<ext>`。
|
||||
pub fn archive_file_path(repo_dir: &Path, key: &str, ext: &str) -> PathBuf {
|
||||
archive_store_dir(repo_dir).join(format!("{}.{}", key, ext))
|
||||
}
|
||||
|
||||
/// 写入过程中的临时路径:`<final>.part.<pid>`,pid 后缀避免并发冲突。
|
||||
pub fn tmp_archive_path(final_path: &Path) -> PathBuf {
|
||||
let mut os = final_path.as_os_str().to_os_string();
|
||||
os.push(format!(".part.{}", std::process::id()));
|
||||
PathBuf::from(os)
|
||||
}
|
||||
|
||||
fn sha1_hex(data: &[u8]) -> Result<String, BabyError> {
|
||||
let mut hasher = gix::hash::hasher(gix::hash::Kind::Sha1);
|
||||
hasher.update(data);
|
||||
let oid = hasher
|
||||
.try_finalize()
|
||||
.map_err(|e| BabyError::Custom(format!("archive cache key hash failed: {}", e)))?;
|
||||
Ok(oid.to_string())
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct GetArchiveCacheKey<'a> {
|
||||
oid: String,
|
||||
format: &'a str,
|
||||
prefix: Option<&'a str>,
|
||||
path: Option<&'a str>,
|
||||
exclude: Vec<&'a str>,
|
||||
}
|
||||
|
||||
/// get_archive 缓存键:commit 解析为 oid(防分支移动命中旧包),与其余参数规范化后哈希。
|
||||
/// exclude 仅参与求键时排序,命令参数保持原序。
|
||||
pub async fn get_archive_cache_key(
|
||||
gitbaby: &crate::GitBaby,
|
||||
req: &GetArchiveRequest,
|
||||
) -> Result<String, BabyError> {
|
||||
let oid = gitbaby.resolve_revision(&req.commit).await?;
|
||||
let mut exclude: Vec<&str> = req.exclude.iter().map(String::as_str).collect();
|
||||
exclude.sort_unstable();
|
||||
let payload = GetArchiveCacheKey {
|
||||
oid: oid.to_string(),
|
||||
format: &req.format.to_string(),
|
||||
prefix: req.prefix.as_deref(),
|
||||
path: req.path.as_deref(),
|
||||
exclude,
|
||||
};
|
||||
let text = serde_json::to_vec(&payload)
|
||||
.map_err(|e| BabyError::Custom(format!("archive cache key serialization failed: {}", e)))?;
|
||||
sha1_hex(&text)
|
||||
}
|
||||
|
||||
/// create_bundle 缓存键:refs 以原顺序拼接待哈希,不解析。
|
||||
pub fn bundle_cache_key(req: &BundleRequest) -> Result<String, BabyError> {
|
||||
sha1_hex(req.refs.join("\n").as_bytes())
|
||||
}
|
||||
|
||||
pub async fn open_cached_file(path: &Path) -> Result<Option<tokio::fs::File>, BabyError> {
|
||||
match tokio::fs::File::open(path).await {
|
||||
Ok(file) => Ok(Some(file)),
|
||||
Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(None),
|
||||
Err(e) => Err(BabyError::Io {
|
||||
path: path.to_path_buf(),
|
||||
source: e,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
type PersistFut =
|
||||
Pin<Box<dyn std::future::Future<Output = (Option<tokio::fs::File>, io::Result<()>)> + Send>>;
|
||||
|
||||
/// 流式透写读取器:每块数据在返回给调用方之前先落盘,
|
||||
/// 读尽后 flush 并以原子 rename 移到最终路径;任何写/rename 失败都在流上显式报错。
|
||||
pub struct PersistReader<R> {
|
||||
inner: R,
|
||||
file: Option<tokio::fs::File>,
|
||||
final_path: PathBuf,
|
||||
tmp_path: PathBuf,
|
||||
pending: Vec<u8>,
|
||||
write_fut: Option<PersistFut>,
|
||||
scratch: Vec<u8>,
|
||||
eof: bool,
|
||||
finalized: bool,
|
||||
error: Option<io::Error>,
|
||||
}
|
||||
|
||||
impl<R> PersistReader<R> {
|
||||
pub fn new(inner: R, file: tokio::fs::File, tmp_path: PathBuf, final_path: PathBuf) -> Self {
|
||||
Self {
|
||||
inner,
|
||||
file: Some(file),
|
||||
final_path,
|
||||
tmp_path,
|
||||
pending: Vec::new(),
|
||||
write_fut: None,
|
||||
scratch: vec![0u8; 16 * 1024],
|
||||
eof: false,
|
||||
finalized: false,
|
||||
error: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<R: tokio::io::AsyncRead + Unpin + Send> tokio::io::AsyncRead for PersistReader<R> {
|
||||
fn poll_read(
|
||||
mut self: Pin<&mut Self>,
|
||||
cx: &mut Context<'_>,
|
||||
buf: &mut tokio::io::ReadBuf<'_>,
|
||||
) -> Poll<io::Result<()>> {
|
||||
let this = self.as_mut().get_mut();
|
||||
loop {
|
||||
if let Some(e) = this.error.take() {
|
||||
return Poll::Ready(Err(e));
|
||||
}
|
||||
if this.finalized {
|
||||
return Poll::Ready(Ok(()));
|
||||
}
|
||||
if let Some(mut fut) = this.write_fut.take() {
|
||||
match fut.as_mut().poll(cx) {
|
||||
Poll::Pending => {
|
||||
this.write_fut = Some(fut);
|
||||
return Poll::Pending;
|
||||
}
|
||||
Poll::Ready((file, result)) => {
|
||||
this.file = file;
|
||||
if let Err(e) = result {
|
||||
let _ = std::fs::remove_file(&this.tmp_path);
|
||||
this.error = Some(e);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
if !this.pending.is_empty() {
|
||||
let n = buf.remaining().min(this.pending.len());
|
||||
if n == 0 {
|
||||
return Poll::Pending;
|
||||
}
|
||||
buf.put_slice(&this.pending[..n]);
|
||||
if n == this.pending.len() {
|
||||
this.pending.clear();
|
||||
} else {
|
||||
this.pending.drain(..n);
|
||||
}
|
||||
return Poll::Ready(Ok(()));
|
||||
}
|
||||
if this.eof {
|
||||
if let Some(file) = this.file.take() {
|
||||
let tmp_path = this.tmp_path.clone();
|
||||
let final_path = this.final_path.clone();
|
||||
this.write_fut = Some(Box::pin(async move {
|
||||
let mut file = file;
|
||||
if let Err(e) = file.flush().await {
|
||||
let _ = std::fs::remove_file(&tmp_path);
|
||||
return (None, Err(e));
|
||||
}
|
||||
drop(file);
|
||||
if let Err(e) = tokio::fs::rename(&tmp_path, &final_path).await {
|
||||
let _ = std::fs::remove_file(&tmp_path);
|
||||
return (None, Err(e));
|
||||
}
|
||||
(None, Ok(()))
|
||||
}));
|
||||
continue;
|
||||
}
|
||||
this.finalized = true;
|
||||
return Poll::Ready(Ok(()));
|
||||
}
|
||||
let mut scratch_buf = tokio::io::ReadBuf::new(&mut this.scratch);
|
||||
match Pin::new(&mut this.inner).poll_read(cx, &mut scratch_buf) {
|
||||
Poll::Pending => return Poll::Pending,
|
||||
Poll::Ready(Err(e)) => {
|
||||
let _ = std::fs::remove_file(&this.tmp_path);
|
||||
return Poll::Ready(Err(e));
|
||||
}
|
||||
Poll::Ready(Ok(())) => {
|
||||
let n = scratch_buf.filled().len();
|
||||
if n == 0 {
|
||||
this.eof = true;
|
||||
continue;
|
||||
}
|
||||
this.pending = scratch_buf.filled().to_vec();
|
||||
if let Some(file) = this.file.take() {
|
||||
let chunk = this.pending.clone();
|
||||
this.write_fut = Some(Box::pin(async move {
|
||||
let mut file = file;
|
||||
let result = file.write_all(&chunk).await;
|
||||
(Some(file), result)
|
||||
}));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn build_create_archive_cmd(
|
||||
dir: &Path,
|
||||
env: Env,
|
||||
|
||||
@ -39,59 +39,83 @@ impl GitBaby {
|
||||
pub async fn get_archive(
|
||||
&self,
|
||||
req: GetArchiveRequest,
|
||||
) -> Result<ChildArchiveReader<BufReader<Box<dyn AsyncRead + Unpin + Send + Sync>>>, BabyError>
|
||||
{
|
||||
) -> Result<ChildArchiveReader<BufReader<Box<dyn AsyncRead + Unpin + Send>>>, BabyError> {
|
||||
let (dir, env) = build_env(self).await?;
|
||||
let mut cmd = helper::build_get_archive_cmd(&dir, env, &req)?;
|
||||
cmd.spawn().await.map_err(BabyError::from)?;
|
||||
let child = match cmd.cmd.as_mut() {
|
||||
Some(c) => c,
|
||||
None => {
|
||||
return Err(BabyError::ArchiveFailed(
|
||||
"git archive child not spawned".to_string(),
|
||||
let key = helper::get_archive_cache_key(self, &req).await?;
|
||||
let final_path = helper::archive_file_path(&dir, &key, helper::format_ext(req.format));
|
||||
if let Some(file) = helper::open_cached_file(&final_path).await? {
|
||||
return Ok(ChildArchiveReader::new(
|
||||
BufReader::new(Box::new(file) as Box<dyn AsyncRead + Unpin + Send>),
|
||||
None,
|
||||
));
|
||||
}
|
||||
};
|
||||
let stdout: Box<dyn AsyncRead + Unpin + Send + Sync> =
|
||||
Box::new(child.stdout.take().ok_or_else(|| {
|
||||
BabyError::ArchiveFailed("git archive stdout not piped".to_string())
|
||||
})?);
|
||||
let _ = std::mem::replace(
|
||||
let tmp_path = helper::tmp_archive_path(&final_path);
|
||||
tokio::fs::create_dir_all(helper::archive_store_dir(&dir))
|
||||
.await
|
||||
.map_err(|source| BabyError::Io {
|
||||
path: helper::archive_store_dir(&dir),
|
||||
source,
|
||||
})?;
|
||||
let file = tokio::fs::File::create(&tmp_path)
|
||||
.await
|
||||
.map_err(|source| BabyError::Io {
|
||||
path: tmp_path.clone(),
|
||||
source,
|
||||
})?;
|
||||
let mut cmd = helper::build_get_archive_cmd(&dir, env, &req)?;
|
||||
cmd.spawn().await.map_err(BabyError::from)?;
|
||||
let stdout: Box<dyn AsyncRead + Unpin + Send + Sync> = std::mem::replace(
|
||||
&mut cmd.stdout,
|
||||
Box::new(tokio::io::empty()) as Box<dyn AsyncRead + Unpin + Send + Sync>,
|
||||
);
|
||||
Ok(ChildArchiveReader::new(BufReader::new(stdout), None))
|
||||
let reader = helper::PersistReader::new(stdout, file, tmp_path, final_path);
|
||||
Ok(ChildArchiveReader::new(
|
||||
BufReader::new(Box::new(reader) as Box<dyn AsyncRead + Unpin + Send>),
|
||||
None,
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn create_bundle(
|
||||
&self,
|
||||
req: BundleRequest,
|
||||
) -> Result<ChildArchiveReader<BufReader<Box<dyn AsyncRead + Unpin + Send + Sync>>>, BabyError>
|
||||
{
|
||||
) -> Result<ChildArchiveReader<BufReader<Box<dyn AsyncRead + Unpin + Send>>>, BabyError> {
|
||||
if req.refs.is_empty() {
|
||||
return Err(BabyError::Custom(
|
||||
"bundle requires at least one ref".to_string(),
|
||||
));
|
||||
}
|
||||
let (dir, env) = build_env(self).await?;
|
||||
let mut cmd = helper::build_bundle_cmd(&dir, env, &req)?;
|
||||
cmd.spawn().await.map_err(BabyError::from)?;
|
||||
let child = match cmd.cmd.as_mut() {
|
||||
Some(c) => c,
|
||||
None => {
|
||||
return Err(BabyError::ArchiveFailed(
|
||||
"git bundle child not spawned".to_string(),
|
||||
let key = helper::bundle_cache_key(&req)?;
|
||||
let final_path = helper::archive_file_path(&dir, &key, "bundle");
|
||||
if let Some(file) = helper::open_cached_file(&final_path).await? {
|
||||
return Ok(ChildArchiveReader::new(
|
||||
BufReader::new(Box::new(file) as Box<dyn AsyncRead + Unpin + Send>),
|
||||
None,
|
||||
));
|
||||
}
|
||||
};
|
||||
let stdout: Box<dyn AsyncRead + Unpin + Send + Sync> =
|
||||
Box::new(child.stdout.take().ok_or_else(|| {
|
||||
BabyError::ArchiveFailed("git bundle stdout not piped".to_string())
|
||||
})?);
|
||||
let _ = std::mem::replace(
|
||||
let tmp_path = helper::tmp_archive_path(&final_path);
|
||||
tokio::fs::create_dir_all(helper::archive_store_dir(&dir))
|
||||
.await
|
||||
.map_err(|source| BabyError::Io {
|
||||
path: helper::archive_store_dir(&dir),
|
||||
source,
|
||||
})?;
|
||||
let file = tokio::fs::File::create(&tmp_path)
|
||||
.await
|
||||
.map_err(|source| BabyError::Io {
|
||||
path: tmp_path.clone(),
|
||||
source,
|
||||
})?;
|
||||
let mut cmd = helper::build_bundle_cmd(&dir, env, &req)?;
|
||||
cmd.spawn().await.map_err(BabyError::from)?;
|
||||
let stdout: Box<dyn AsyncRead + Unpin + Send + Sync> = std::mem::replace(
|
||||
&mut cmd.stdout,
|
||||
Box::new(tokio::io::empty()) as Box<dyn AsyncRead + Unpin + Send + Sync>,
|
||||
);
|
||||
Ok(ChildArchiveReader::new(BufReader::new(stdout), None))
|
||||
let reader = helper::PersistReader::new(stdout, file, tmp_path, final_path);
|
||||
Ok(ChildArchiveReader::new(
|
||||
BufReader::new(Box::new(reader) as Box<dyn AsyncRead + Unpin + Send>),
|
||||
None,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
319
tests/archive.rs
Normal file
319
tests/archive.rs
Normal file
@ -0,0 +1,319 @@
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Command;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
|
||||
use async_trait::async_trait;
|
||||
|
||||
use gitbaby::GitBaby;
|
||||
use gitbaby::archive::{ArchiveFormat, BundleRequest, GetArchiveRequest};
|
||||
use gitbaby::error::BabyError;
|
||||
use gitbaby::repo::RepositoryFacade;
|
||||
|
||||
static COUNTER: AtomicU32 = AtomicU32::new(0);
|
||||
|
||||
struct TestFacade {
|
||||
dir: PathBuf,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl RepositoryFacade for TestFacade {
|
||||
async fn git_repo_dir(&self) -> Result<PathBuf, BabyError> {
|
||||
Ok(self.dir.clone())
|
||||
}
|
||||
|
||||
async fn git_alternate_object_directories(&self) -> Result<Vec<PathBuf>, BabyError> {
|
||||
Ok(Vec::new())
|
||||
}
|
||||
|
||||
async fn gix_repo(&self) -> Result<gix::Repository, BabyError> {
|
||||
gix::open(&self.dir).map_err(|e| BabyError::Custom(e.to_string()))
|
||||
}
|
||||
}
|
||||
|
||||
fn temp_repo() -> PathBuf {
|
||||
let n = COUNTER.fetch_add(1, Ordering::SeqCst);
|
||||
let dir = std::env::temp_dir().join(format!("gitbaby-archive-test-{}-{n}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).expect("mkdir repo");
|
||||
let out = Command::new("git")
|
||||
.args(["init", "-q"])
|
||||
.current_dir(&dir)
|
||||
.output()
|
||||
.expect("git init failed");
|
||||
assert!(out.status.success(), "git init: {:?}", out);
|
||||
dir
|
||||
}
|
||||
|
||||
fn cleanup(repo: &Path) {
|
||||
let _ = std::fs::remove_dir_all(repo);
|
||||
}
|
||||
|
||||
fn git_commit(repo: &Path, msg: &str) {
|
||||
let out = Command::new("git")
|
||||
.args([
|
||||
"-c",
|
||||
"user.name=T",
|
||||
"-c",
|
||||
"user.email=t@e",
|
||||
"commit",
|
||||
"-q",
|
||||
"-m",
|
||||
msg,
|
||||
])
|
||||
.current_dir(repo)
|
||||
.output()
|
||||
.expect("git commit failed");
|
||||
assert!(out.status.success(), "git commit `{msg}`: {:?}", out);
|
||||
}
|
||||
|
||||
fn write_file(repo: &Path, rel: &str, content: &str) {
|
||||
let p = repo.join(rel);
|
||||
std::fs::create_dir_all(p.parent().expect("parent")).expect("mkdir");
|
||||
std::fs::write(p, content).expect("write");
|
||||
}
|
||||
|
||||
fn index_commit(repo: &Path, msg: &str) {
|
||||
let out = Command::new("git")
|
||||
.args(["add", "-A"])
|
||||
.current_dir(repo)
|
||||
.output()
|
||||
.expect("git add failed");
|
||||
assert!(out.status.success(), "git add: {:?}", out);
|
||||
git_commit(repo, msg);
|
||||
}
|
||||
|
||||
async fn baby_of(repo: PathBuf) -> GitBaby {
|
||||
GitBaby::new(Arc::new(TestFacade { dir: repo }))
|
||||
}
|
||||
|
||||
fn archive_dir(repo: &Path) -> PathBuf {
|
||||
repo.join("archive")
|
||||
}
|
||||
|
||||
fn archive_files(repo: &Path) -> Vec<PathBuf> {
|
||||
let mut files: Vec<PathBuf> = std::fs::read_dir(archive_dir(repo))
|
||||
.expect("read archive dir")
|
||||
.filter_map(|e| e.ok())
|
||||
.map(|e| e.path())
|
||||
.filter(|p| p.is_file())
|
||||
.collect();
|
||||
files.sort();
|
||||
files
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_archive_persists_and_hits_cache() {
|
||||
let repo = temp_repo();
|
||||
write_file(&repo, "a.txt", "one");
|
||||
write_file(&repo, "b.txt", "bee");
|
||||
index_commit(&repo, "first");
|
||||
|
||||
let baby = baby_of(repo.clone()).await;
|
||||
let req = || GetArchiveRequest {
|
||||
commit: "HEAD".to_string(),
|
||||
format: ArchiveFormat::Tar,
|
||||
prefix: None,
|
||||
path: None,
|
||||
exclude: Vec::new(),
|
||||
};
|
||||
|
||||
let first_bytes = baby
|
||||
.get_archive(req())
|
||||
.await
|
||||
.expect("get_archive")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("read archive body");
|
||||
|
||||
let files = archive_files(&repo);
|
||||
assert_eq!(files.len(), 1, "one cached archive expected");
|
||||
assert_eq!(
|
||||
files[0].extension().and_then(|e| e.to_str()),
|
||||
Some("tar"),
|
||||
"cached file: {:?}",
|
||||
files[0]
|
||||
);
|
||||
let on_disk = std::fs::read(&files[0]).expect("read cached file");
|
||||
assert_eq!(on_disk, first_bytes, "cached bytes match streamed bytes");
|
||||
|
||||
let direct = Command::new("git")
|
||||
.args(["archive", "--format=tar", "HEAD"])
|
||||
.current_dir(&repo)
|
||||
.output()
|
||||
.expect("git archive direct");
|
||||
assert!(direct.status.success(), "git archive direct: {:?}", direct);
|
||||
assert_eq!(direct.stdout, first_bytes, "matches raw git archive output");
|
||||
|
||||
let second_bytes = baby
|
||||
.get_archive(req())
|
||||
.await
|
||||
.expect("get_archive cached")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("read archive body");
|
||||
assert_eq!(second_bytes, first_bytes, "cache hit returns same bytes");
|
||||
assert_eq!(archive_files(&repo).len(), 1);
|
||||
|
||||
cleanup(&repo);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_archive_keys_differ_by_options() {
|
||||
let repo = temp_repo();
|
||||
write_file(&repo, "a.txt", "one");
|
||||
index_commit(&repo, "first");
|
||||
|
||||
let baby = baby_of(repo.clone()).await;
|
||||
let base = GetArchiveRequest {
|
||||
commit: "HEAD".to_string(),
|
||||
format: ArchiveFormat::Tar,
|
||||
prefix: None,
|
||||
path: None,
|
||||
exclude: Vec::new(),
|
||||
};
|
||||
|
||||
let plain = baby
|
||||
.get_archive(base.clone())
|
||||
.await
|
||||
.expect("plain")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("plain body");
|
||||
|
||||
let prefixed = baby
|
||||
.get_archive(GetArchiveRequest {
|
||||
prefix: Some("p/".to_string()),
|
||||
..base.clone()
|
||||
})
|
||||
.await
|
||||
.expect("prefixed")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("prefixed body");
|
||||
|
||||
let excluded = baby
|
||||
.get_archive(GetArchiveRequest {
|
||||
exclude: vec!["a.txt".to_string()],
|
||||
..base.clone()
|
||||
})
|
||||
.await
|
||||
.expect("excluded")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("excluded body");
|
||||
|
||||
assert_ne!(plain, prefixed);
|
||||
assert_ne!(plain, excluded);
|
||||
assert_ne!(prefixed, excluded);
|
||||
|
||||
let files = archive_files(&repo);
|
||||
assert_eq!(files.len(), 3, "three distinct keys cached");
|
||||
|
||||
let zipped = baby
|
||||
.get_archive(GetArchiveRequest {
|
||||
format: ArchiveFormat::Zip,
|
||||
..base
|
||||
})
|
||||
.await
|
||||
.expect("zip")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("zip body");
|
||||
let files = archive_files(&repo);
|
||||
assert_eq!(files.len(), 4);
|
||||
assert!(
|
||||
files
|
||||
.iter()
|
||||
.any(|p| p.extension().and_then(|e| e.to_str()) == Some("zip")),
|
||||
"zip file present: {:?}",
|
||||
files
|
||||
);
|
||||
assert!(zipped.starts_with(b"PK"), "zip magic");
|
||||
assert_ne!(zipped, plain);
|
||||
|
||||
cleanup(&repo);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_archive_new_commit_new_key() {
|
||||
let repo = temp_repo();
|
||||
write_file(&repo, "a.txt", "one");
|
||||
index_commit(&repo, "first");
|
||||
|
||||
let baby = baby_of(repo.clone()).await;
|
||||
let req = || GetArchiveRequest {
|
||||
commit: "HEAD".to_string(),
|
||||
format: ArchiveFormat::Tar,
|
||||
prefix: None,
|
||||
path: None,
|
||||
exclude: Vec::new(),
|
||||
};
|
||||
|
||||
let _ = baby
|
||||
.get_archive(req())
|
||||
.await
|
||||
.expect("first revision")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("body");
|
||||
assert_eq!(archive_files(&repo).len(), 1);
|
||||
|
||||
write_file(&repo, "a.txt", "two");
|
||||
index_commit(&repo, "second");
|
||||
|
||||
let _ = baby
|
||||
.get_archive(req())
|
||||
.await
|
||||
.expect("second revision")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("body");
|
||||
let files = archive_files(&repo);
|
||||
assert_eq!(files.len(), 2, "new commit produces new cached key");
|
||||
|
||||
cleanup(&repo);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn create_bundle_persists_and_hits_cache() {
|
||||
let repo = temp_repo();
|
||||
write_file(&repo, "a.txt", "one");
|
||||
index_commit(&repo, "first");
|
||||
|
||||
let baby = baby_of(repo.clone()).await;
|
||||
let req = || BundleRequest {
|
||||
refs: vec!["HEAD".to_string()],
|
||||
};
|
||||
|
||||
let first = baby
|
||||
.create_bundle(req())
|
||||
.await
|
||||
.expect("create_bundle")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("bundle body");
|
||||
|
||||
let files = archive_files(&repo);
|
||||
assert_eq!(files.len(), 1, "one cached bundle expected");
|
||||
assert_eq!(
|
||||
files[0].extension().and_then(|e| e.to_str()),
|
||||
Some("bundle"),
|
||||
"cached file: {:?}",
|
||||
files[0]
|
||||
);
|
||||
let on_disk = std::fs::read(&files[0]).expect("read cached file");
|
||||
assert_eq!(on_disk, first, "cached bundle bytes match streamed bytes");
|
||||
|
||||
let second = baby
|
||||
.create_bundle(req())
|
||||
.await
|
||||
.expect("create_bundle cached")
|
||||
.read_to_end()
|
||||
.await
|
||||
.expect("bundle body");
|
||||
assert_eq!(second, first, "cache hit returns same bundle bytes");
|
||||
assert_eq!(archive_files(&repo).len(), 1);
|
||||
|
||||
cleanup(&repo);
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user