Joel Marcey oversees the technology and engineering programs and initiatives of the Rust Foundation. Prior to joining the Rust Foundation as a founding staff member, Joel worked at Facebook/Meta as a Developer Advocate and ecosystem lead with prominent participation in high-impact standards organizations including Open Web Docs and Ecma.
Open source is a wonderful thing and has been an integral part of the software industry for over the last quarter century. Many of the most used and most critical software of today are entirely open source, and most software has open source dependencies.
The powerful idea behind open source is that the code is available for anyone to use, at any time, and generally for free (there are license and copyright considerations, of course, but for all intents and purposes, let’s just call it free use). But many open source projects rely on both the hard work of volunteers and free or donated infrastructure to keep them afloat and moving forward. The Rust ecosystem is no exception.
There are various components that make up the Rust ecosystem: the Rust Project, the global community of Rust contributors and developers, Rust releases, crates.io, and more. In this post, we’ll focus on crates.io as we examine the intricacies of observing the true cost of open source.
crates.io and Package Repositories
crates.io is the Rust ecosystem’s package repository. This is where all of the libraries and code live in order to do useful things with Rust. There are currently over 283K crates, with a cumulative 2.5M versions, that have been downloaded over 332B (that’s billion with a B) times.
The use of crates.io is experiencing an upward trajectory in both the amount of crates and the volume of usage; this has been the trend for years. The growth of Rust, the increased usage of CI in development, and, of course, AI, have made the trend more exponential than linear. Just look at our monthly bandwidth volume over the last several years and where we calculated 2026 would land when we did our estimation late last year. As you will see later in the article, the figure we landed on was a dramatic underestimation.
It is not just crates.io that is seeing a massive level of growth. Many of the popular package repositories across the open source ecosystem are having to deal with this type of unprecedented growth. In fact, several major package repositories have put out joint statements around sustainability and put out a call for the industry to address what is quickly becoming a crisis.
Datadog to the Observability Rescue
In order to understand the growth of crates.io and how we can plan for its future, we need data to help us make proper decisions. Datadog provides the Rust Foundation and Rust Project with a platform to make use of that data. Let’s talk about some of the questions that Datadog helps us answer.
How much bandwidth is required for crates.io usage?
As stated above, the amount of bandwidth required to run crates.io is growing at unprecedented rates. And we don’t see that slowing down in the near future. So, we need a way to keep tabs on how much bandwidth is being used at any given time in order to understand spikes, trends, and other types of data.
We had estimated about 18-19 PB of necessary bandwidth to support crates.io in 2026 and we have almost reached that amount in just the first half of 2026!
Bandwidth is the most core metric of usage, and Datadog allows us to visualize that and take the information to try to get more granular answers to critical questions, like the ones I am going to talk about next.
Who is using crates.io the most?
This is a difficult question to answer precisely. The biggest obstacles in getting accurate data here are cloud providers. Most of our traffic hits AWS, Azure, and the Google Cloud Platform (GCP). This makes sense because a lot of automated access to crates.io would be utilizing machines in the cloud. While we know that traffic is coming from, say AWS, we don’t know who the actual responsible party might be. It could be AWS or a random developer elsewhere in the world. That said, we can glean some useful information outside of the cloud providers, particularly around how many requests per amount of time are being made.
Take a look at this graph. It shows the number of requests per hour by a given anonymized entity. As you can see, there are ten or so top-heavy users and then a long tail after that.
Having Datadog visualize this can allow us to figure out how we want to handle high usage on crates.io, potentially discussing support options with these top-heavy users.
How much is AI having an impact on our growth?
The AI revolution is upon us. And there are differing opinions on whether that is good, bad, or neutral. I won’t get into those arguments here, but it’s undeniable that AI, primarily through bots and agents, is increasing the burden on our infrastructure. Also, developers using AI are, for better or worse, able to code more quickly and reduce the time between requests to crates.io.
We started collecting AI request data in late 2025. With some peaks and valleys, the number of daily requests as a result of AI is in the millions. And this is just the floor. We only know the declared AI agents: those that explicitly say they are AI. We know there are probably at least an order of magnitude or more hidden AI agents that we cannot track.
Compare the declared AI agents to our total number of requests per day.
For example, on 5 June 2026, we had 2.27M declared AI requests out of a total of 641M requests, meaning on that day, declared AI requests were 0.3% of all our requests. Extrapolating the undeclared AI requests, you can imagine AI requests are probably somewhere between 3-30% of all our requests.
How to handle the influx of AI into our infrastructure is a question that we have not fully answered yet. Can rate limiting work? What are the legitimate uses and illegitimate ones? Handling AI on open source infrastructure might be one of the biggest challenges for infrastructure engineers to tackle because this issue is only going to get worse for the foreseeable future.
How can we track CI?
We estimate that somewhere on the order of 50% of traffic to crates.io is a result of continuous integration (CI) runs that require crates to test for success or failure. There is a lot of Rust development happening. Datadog provides a CI visibility product that gives us great insight into testing within the Rust Project and ecosystem.
Here you can see different views of the CI pipeline and job success running on various architectures, which helps us understand where changed code may be having trouble.
Where are the problem endpoints?
Before joining Datadog’s open source program, we sampled only 5% of requests to reduce costs on our logging, meaning we missed many outliers across our performance and usage. After being able to ingest and index all our web, API and CDN requests, we were quickly able to identify problem points in our infrastructure. And after analyzing all of our logs, we began seeing trends of problem endpoints. Additionally, we are now able to immediately respond to incidents by directly inspecting the traffic associated with them - drastically reducing our response time and letting us zero in on the root cause. For example, we began responding to incidents of unrelated endpoints having disruptions due to unknown factors. This graph shows a problematic endpoint, identified with Datadog. Notice the spike when the endpoint was being abused and our fix being deployed, flattening our response time.
Conclusion
Open source has transformed the software industry. That also comes with a cost. Heavy usage and AI are just a few of the things putting both a people and monetary tax on open source infrastructure. There will need to be changes in order to keep open source package registries available and sustainable. Datadog allows us at the Rust Foundation and Rust Project to make informed decisions and set direction based on easily visible metrics. We sincerely thank Datadog for allowing us to be part of their open source program. We would also like to thank Fastly and AWS for their generous support of in-kind donations, providing much of the bandwidth necessary to support this growth. Learn more about following the lead of Datadog and other organizations helping to support the Rust programming language here.
The author would like to thank Walter Pearce for his contributions and reviews of this blog post.