Git at Any Scale: Overcoming the Hosting Nightmare

Git’s design treats every clone of a repository as a complete, identical copy of the source data. The same on‑disk layout that a developer’s laptop uses is also what a production Git server must expose. This uniformity sounds simple, but it forces the server to handle the same low‑level operations that a single‑user client performs, which quickly becomes a scalability bottleneck.

All objects—blobs, trees, commits—are stored in packfiles. A packfile is a binary container that compresses objects and stores many of them as deltas against other objects. Packfiles are the fundamental unit for both storage and network transfer: every push, fetch, and clone sends or receives a packfile. Because the Git protocol cannot be altered without breaking compatibility, a server must always read and write these large binary files, even if the underlying storage system is changed.

The consequences are twofold:

• Limited parallelism. A single packfile must reside on a filesystem that supports the locking and sync semantics Git expects. When many clients request objects simultaneously, the server’s I/O subsystem becomes a contention point.
• Fragile availability. If the host machine or its disk fails, the entire repository becomes unavailable because the packfile cannot be reconstructed on the fly.

Enterprises have tried three broad strategies to overcome these limits:

1. Distribute the filesystem. Projects such as early GitHub deployments placed repositories on NFS, GFS, or DRBD. Git’s assumptions about local‑disk behavior (e.g., atomic renames, reliable locking) broke over networked filesystems, leading to slow performance and corruption.
2. Distribute the packfiles. Some teams attempted to shard individual packfiles across storage nodes. Because objects inside a pack are stored arbitrarily and often as deltas, locating a needed object may require fetching multiple packfiles, negating any I/O gain.
3. Distribute Git itself. A more radical approach replaces on‑disk packs with a distributed hash table (DHT). While JGit can map SHA‑1 keys to a DHT, the Git protocol still forces packfile transfer, so clone operations remain slow and the design was abandoned in practice.

In practice, the combination of an immutable, binary packfile format and a protocol that mandates packfile exchange makes “just add more machines” an ineffective scaling strategy. Any solution must either redesign the storage layer while preserving packfile semantics or accept the inherent limits of Git’s original design.

Git stores every object (blobs, trees, commits, tags) as a compressed binary blob inside a .pack file. When a packfile is created, the pack‑generation algorithm first selects a set of objects to include, then orders them to minimise total size. This ordering is unrelated to the logical structure of the commit DAG, so objects are placed essentially at random positions within the file.

To achieve high compression, most objects are not stored whole; instead they are saved as a delta against another object in the same packfile. A delta records only the byte‑wise differences between a base object and the target object. During a fetch or a checkout, Git must:

1. Locate the packfile on disk.
2. Read the index (.idx) to map a SHA‑1 to an offset.
3. Seek to that offset, then, if the object is a delta, recursively read its base object(s) until a full (non‑delta) object is reached.
4. Apply each delta in turn to reconstruct the requested object.

This process introduces two orthogonal performance bottlenecks for large, distributed deployments:

• Random I/O pattern: Because the physical layout does not follow the DAG, a single logical operation (e.g., listing recent commits) can trigger many non‑sequential disk seeks, exhausting SSD/ HDD throughput and increasing latency.
• Delta chain depth: Deep delta chains require multiple reads and in‑memory recombinations, inflating CPU usage and memory pressure, especially when many concurrent clone or fetch operations target the same packfile.

Consider a repository with a src/ directory that changes frequently. A recent commit may store the new src/main.rs as a delta against the version from three commits earlier. If a developer checks out the latest commit, Git must read the delta, then the base object, then possibly the base’s own delta, resulting in three separate seeks and three decompression steps before the final file appears on disk.

Practical mitigation strategies (to be applied after understanding the above mechanics) include:

• Periodically repack with git gc --aggressive to limit delta chain length.
• Store packfiles on storage that offers low‑latency random reads (e.g., NVMe SSDs) and sufficient IOPS.
• Shard very large repositories so that each shard contains a subset of the DAG, reducing the number of objects per packfile.
• Cache frequently accessed packfile indexes in memory to avoid repeated index reads.

Git’s native design stores every object as a content‑addressable blob inside a packfile. Because a packfile is just a binary blob on a local filesystem, the simplest way to scale a Git service is to make that filesystem available to many front‑end servers. This “distribute the filesystem” approach keeps the Git implementation unchanged; the web tier can read and write the same on‑disk repository as a developer’s laptop would.

• Pros: No code changes, easy to add more application instances, leverages existing Git tooling.
• Cons: Network filesystems (NFS, GFS, DRBD) break Git’s assumptions about locking, atomicity, and read‑write ordering, leading to latency spikes and corruption. The random layout of objects inside packfiles means that even a single object read may trigger many physical disk seeks, which is amplified over a remote mount.

The next step is to “distribute the packfiles” themselves. Instead of sharing a raw filesystem, each server holds a copy of the packfiles and a coordination layer (e.g., a CDN or object store) serves them to clients. Servers can parallelize fetches, and a failed node does not make the repository unavailable because other nodes still have the same packfiles.

• Pros: Improves read scalability, reduces contention on a single storage device, and isolates failures.
• Cons: Packfiles are immutable only while being written; synchronizing new packfiles across nodes adds operational complexity. Because Git always transfers data as packfiles, any change still requires a full packfile write, and delta compression can cause large files to be rewritten frequently.

The most complex path is to “distribute Git itself.” Here the storage layer is abstracted away—objects are stored in a distributed key‑value store keyed by their SHA‑1, and a custom Git implementation (e.g., JGit with a DHT backend) serves requests directly from that store. This removes the packfile bottleneck at the storage level.

• Pros: Enables horizontal scaling of storage, can leverage existing distributed databases that provide replication, durability, and compliance features (e.g., SOC 2, ISO 27001).
• Cons: The Git protocol still requires packfiles for network transfer, so clone and fetch operations incur extra packing steps. Traversing the commit DAG forces many round‑trips to the key‑value store, dramatically increasing latency unless aggressive caching is added. Implementations are non‑trivial and have historically shown poor clone performance, leading many organizations to abandon this path.

Choosing a path therefore depends on the organization’s tolerance for operational complexity versus the need for high availability and read throughput. Small teams often accept the filesystem approach, mid‑size services move to packfile replication, and only large, highly‑available platforms consider a full Git‑logic distribution despite its steep engineering cost.

Git stores every object (blob, tree, commit, tag) under a SHA‑1 key, which makes the repository a natural candidate for a content‑addressable key‑value store. In theory a distributed hash table (DHT) could replace the on‑disk packfiles: the key is the object hash, the value is the raw object data. This idea was explored in a Google project that used JGit, a pure‑Java implementation of Git, to plug a DHT behind the repository abstraction.

While the DHT layer succeeded in serving individual objects, the Git network protocol still requires the client to receive and send data as packfiles. A packfile is a binary container that groups many objects, applies delta compression, and is the only format understood by git fetch, git push, and git clone. Consequently, even when the server stores objects in a DHT, the server must materialise a packfile for every network operation, serialize it, and stream it to the client.

Performance degraded for two intertwined reasons:

• Graph walk latency: Git operations walk the commit DAG step‑by‑step. Each step requires the previous object's hash to locate the next object. When every lookup incurs a remote DHT round‑trip, the number of network hops grows from a few to dozens or hundreds, inflating latency dramatically.
• Packfile generation overhead: Building a packfile on‑the‑fly from dispersed objects forces the server to fetch many fragments, apply delta compression, and write a temporary binary file before it can be streamed. This extra I/O and CPU work outweighs any storage‑distribution benefits, especially under heavy clone or fetch traffic.

Practical attempts to mitigate these issues—such as caching generated packs or pre‑computing common packfiles—still could not overcome the fundamental mismatch between a DAG‑centric data model and a flat, binary packfile transfer protocol. The result was clone times that were unacceptable for large repositories, leading the project to abandon the DHT approach and revert to traditional packfile storage on scalable filesystems.

GitHub’s first production system was a single‑machine Rails monolith that stored every repository on the local disks attached to the web server. Because the application code and the Git data lived side‑by‑side, scaling the web tier was straightforward: launch more Rails processes. The real problem emerged when additional web nodes needed read‑write access to the same on‑disk repositories.

The engineering team’s initial solution was to “distribute the filesystem” rather than redesign Git’s storage format. They assumed that a network file system would present the same semantics as a local ext4 volume, allowing the existing Ruby code to remain unchanged.

• NFS – The first attempt used a central NFS server. Git’s core assumes atomic file‑level locking, immediate fsync, and consistent read‑after‑write ordering. Over NFS these guarantees break: lock files are not respected reliably, write‑through caching introduces latency, and occasional “stale‑handle” errors caused repository corruption.
• GFS – A block‑level distributed filesystem was tried next. While GFS offered higher throughput, its write‑path required complex chunk replication that conflicted with Git’s need to rewrite packfiles atomically. The result was frequent “partial pack” files that could not be read by the Git client.
• DRBD – A longer‑lived deployment mirrored block devices between two nodes. DRBD’s synchronous replication added a full round‑trip for every disk write, turning a simple git push into a multi‑second operation. Moreover, DRBD did not expose the POSIX‑level locking semantics Git expects, leading to deadlocks during concurrent pushes.

All three approaches failed because they ignored the semantics baked into Git’s packfile layout. Packfiles are compressed, delta‑encoded collections of objects placed arbitrarily to minimize size. Accessing a single object requires:

1. Reading the pack index to locate the object’s offset.
2. Seeking to that offset, then possibly following a chain of delta bases.
3. Performing a final decompression step.

When the underlying filesystem cannot guarantee fast, atomic seeks and reliable locking, each of these steps becomes a source of latency or corruption. The lesson for large‑scale Git hosting is that the storage layer must preserve local‑disk semantics; otherwise the cost of “distributed” filesystems outweighs any benefit.

Practical guidance derived from GitHub’s experience:

• Prefer local SSD storage for active packfiles and replicate at the repository level, not the block level.
• Use read‑only mirrors for historic packs; write‑only paths should be confined to a single node to avoid lock contention.
• When scaling, consider distributing the packfiles themselves (e.g., sharding by SHA‑1 prefix) rather than attempting to virtualize the entire filesystem.