Java has been a cornerstone of enterprise software for decades, and it continues to play a critical role in the modern cloud-native ecosystem. It powers many performance-critical environments, including data processing and analytics with Apache Spark, streaming data with Apache Kafka, and our own optimized services for processing profiling data. We also see Java widely used among our customers, supporting high-volume systems and critical workloads across a variety of industries.
With 1000s of containers running at any time across our own cloud infrastructure, and many more on our customer’s, we are committed to ensuring that the OpenJDK project benefits from observations and improvements made by our engineering team back to the project, that we use in-house to ensure that Datadog continues to be a cost-effective, reliable platform for our customers.
Java Observability and JDK Mission Control
At Datadog, our open-source library, dd-trace-java, provides robust application performance monitoring for Java-based applications, including distributed tracing and profiling. To enhance observability, we rely on key OpenJDK features like Java Flight Recorder (JFR), JDK Mission Control (JMC), and Java Management Extensions (JMX).
The roots of JDK Mission Control trace back to the JRockit JVM, a project initiated by a team from the Royal Institute of Technology (KTH) in Stockholm. As JRockit evolved, particularly for low-latency workloads (e.g., JRockit Real-Time), the demand for better observability tooling grew. This led to the development of JRockit Mission Control. Marcus Hirt, now at Datadog, was deeply involved in these efforts—contributing to JMAPI (the precursor to java.lang.management) and eventually leading the JRockit Mission Control team. Today, Marcus serves as the project lead for JDK Mission Control, continuing to drive innovation in JVM observability, and is internally responsible for Datadog's continuous profiler projects.
At Datadog, we use JFR in multiple ways — directly accessing its data for insights, and using JMC’s parser to transform JFR data into actionable formats. We also develop custom profilers exporting data in JFR format and heuristics to continuously surface valuable insights. Many of these heuristics are inspired by or extend ideas from the JMC project.
One area Datadog engineers have contributed to in JMC is the continuous enhancement of the visualization capabilities of the desktop client, addressing challenges developers face when analyzing complex performance data. For example, we introduced a call graph visualization to provide clearer insights into method invocation relationships, and developed a new Swing-based flame graph that improves performance and usability when interpreting profiling data. These changes, along with innovations like hierarchical edge bundling and smart graph pruning, reflect our commitment to making JMC a more powerful tool for understanding and diagnosing Java applications for the community at large.
Our efforts have also extended to JMC’s core functionality, for instance, introducing the JMC Rules 2.0 API which extends JMCs ability to provide automatic insights into performance data, and providing a binary format writer so that users can export JFR data for usage elsewhere.
Java Reliability
At Datadog’s scale, we often encounter unique bugs and edge cases that arise from running Java applications in high-throughput, distributed environments. These challenges provide valuable insights that we contribute back to OpenJDK to improve its reliability for all users.
For example, our team has identified and resolved several stability challenges, particularly those tied to our extensive use - and our customers' reliance - on the JVM’s profiling and runtime visibility features. For instance, JDK-8329995 (PR), JDK-8283849 (PR), and JDK-8313816 (PR), all concern issues related to JVM crashes triggered by JVM functions used to introspect the running application - important functionality for the runtime visibility we rely upon.
Knowledge Sharing
As well as our direct contributions to the OpenJDK project, Datadog engineers support the Java ecosystem, speaking regularly with the community about our experiences operating and building around Java at Datadog:
- Future-proofing JVM profiling - Jaroslav Bachorík and Richard Startin at jFokus 24
- Understanding request latency with wallclock profiling - Richard Startin at P99 Conf 2024
- A walk along the complexity-performance curve - Richard Startin at QCon London 2024
We also regularly attend the OpenJDK committers workshop to meet with other committers and share the knowledge we have gained around the use of the JDK in the observability space.
Insights on contributing to OpenJDK
What do you think is the most exciting area to contribute to in the JDK right now?
Contributing to OpenJDK offers a variety of exciting challenges, but one of the most engaging areas is improving Java profiling capabilities. This includes work on tools like Java Mission Control (JMC) and Java Flight Recorder (JFR). These tools are critical for understanding and optimizing application performance, and they have been areas where meaningful contributions can have a wide impact.
Can you highlight some specific contributions you or your team have made to OpenJDK?
One notable contribution was delivering the ‘new’ allocation sampler for JFR a few years ago. This enhancement significantly improved the ability to analyze allocation patterns in Java applications. Beyond that, we’ve focused on raising awareness about profiling labels in JFR. These labels would provide more context during profiling sessions, helping developers better understand what’s happening inside their applications.
We also work closely with the OpenJDK community on initiatives like the new CPU profiler, which is part of JEP 8337789. While much of our involvement here is in testing and providing feedback, it’s a great example of how collaboration within the OpenJDK ecosystem can drive innovation.
Are there any performance or stability improvements your team has contributed to upstream?
Yes, there was a significant performance improvement identified in JDK-8282664, partially implemented by Ludovic Henry in PR #7700, and ultimately completed by another member of the community in #10847.
This change was identified internally as a source of high CPU usage in our logging processing system, but benefits any users of the JDK. Here at Datadog it resulted in noticeable savings, particularly for systems with high logging volumes. It’s an excellent example of how upstream contributions can have tangible benefits for Java users.
What advice would you give to someone interested in contributing to OpenJDK?
Collaboration is key. Engaging with the broader community, working with other contributors, and aligning with ongoing projects can make a huge difference. It’s also important to focus on areas where your expertise aligns with the needs of the platform. Tools like JFR and JMC offer fertile ground for impactful contributions, especially for developers passionate about performance and diagnostics.
Ultimately, contributing to OpenJDK is about improving the Java ecosystem for everyone, and even small changes can have a big ripple effect.