Datadog has been using Temporal to manage complex workflows and some Datadog employees have become Temporal power users and others have become contributors to the project.
Let’s take a look at some examples of the partnership between Datadog and Temporal and what both organizations have done to help improve the experience for Temporal users.
Simplifying Temporal
This partnership really started with a basic idea, one that the Temporal folks had been thinking about for a long time already: simplifying Temporal into a single binary instead of a collection of containers and commands that needed to be orchestrated separately.
Jacob LeGrone, a Senior Software Engineer at Datadog, developed a minimum viable product of Temporal as a single binary, leveraging the power and flexibility of SQLite, an in-memory SQL database. This project was called temporalite.
Temporalite was so successful at streamlining the Temporal experience that the team wanted to go all-in on the temporalite approach as the default for onboarding folks, but having the recommended getting started tooling owned by another company presented a challenge.
In 2022, Datadog transferred ownership of temporalite to the Temporal GitHub Organization and it has become the point of entry into the Temporal ecosystem.
Where are some of the places that Datadog is running temporalite today?
In our CI for integration tests, and we’re also starting to experiment with runbooks to recover from Temporal server outages that require automation built in Temporal (eg. to scale up a Cassandra cluster).
The Datadog/Temporal partnership has been continuing to strengthen because developers at Datadog use Temporal a lot and they also love contributing to open source projects - and temporalite wasn’t the only contribution Datadog employees made to the Temporal project.
Visualizing Workflows
Sebastian Neira, Software Engineer at Datadog, considers himself a “visual person.” When he was new to Datadog and using Temporal frequently, he found that he was frustrated with the existing state of Temporal workflow visualization.
Luckily for Temporal users, Sebastian is good at simplifying complex ideas into something that's easier to parse - and Datadog’s engineering organization runs a couple of internal two-day hackathons each year. He had already started tinkering with a tool to better visualize Temporal workflows, so he put the idea out there as an Request for Comments (RFC) just before the Winter 2022 Hackathon.
The feedback on that RFC gave him direction and motivation for his hackathon efforts.
Forty eight hours later, Sebastian had expanded his MVP into a mostly-finished prototype. This internal tool was such a hit that the team shared it in the Temporal Slack - and everyone was excited about it. It took about 6 months for Sebastian to wrap up other initiatives and have a chance to revisit his hackathon project.
Once he got back into it, for the next few months he found himself not only immersed in a new language ecosystem with Golang, but also focusing full-time on upstreaming this new functionality to the Temporal CLI to create the workflow trace command.
Sebastian had previously contributed to a popular StarCraft 2 mod quite a bit so he was familiar with collaborating with an Open Source community, but working with Temporal was his first time making contributions to a major Open Source project with stringent requirements and multiple rounds of Pull Request (PR) review. This was a new challenge as he learned the ins and outs of Go, and the requirements of the Temporal project, to get his PR ready to merge - and there are some updates in progress.
What are you most excited about regarding the future of Temporal?
I believe Temporal is going to take a big part of what microservices were set out to solve. They’re building an outstanding platform and Datadog has proven how well it can be used to solve a wide variety of problems.
temporalite and workflow trace weren't the only ideas that Datadog had for Temporal, though.
Ongoing Collaboration
Here are some more highlights from the Datadog/Temporal partnership:
- Datadog released a Temporal Server Integration
- Datadog provided a tracing interceptor for the Temporal Go SDK that allows accurate reporting for workflow spans that might remain open for hours, days, or even weeks
- Datadog identified a vulnerability with insecure defaults in the Temporal Server package, CVE-2023-3485, which lead to Temporal becoming a CVE Numbering Authority
temporaltest, a package used for testing temporalite, is also being integrated into the mainline Temporal project- Datadog has created a codec that allows Temporal workflows to work with large payloads that surpass the 4MB payload limit
- Datadog is collaborating with Temporal on a tool to support worker versioning upstream
Looking Ahead
There’s a lot more to come from the collaboration between Datadog and Temporal.
Several Datadog employees will be presenting at Temporal’s 2023 Replay Conference, and there just might be an interesting announcement or two.
If you can’t make it to Replay, we’ll be sure to add links to the session recordings and some notes here, so check back after Replay is over.
Digging Deeper
Want to learn more about how Datadog is using and contributing to the Temporal project?
- Temporalite: the foundation of the new Temporal CLI experience
- Temporal Community Meetup #8 Q&A w/ Jacob LeGrone
Here are some other videos featuring Jacob LeGrone and Temporal at Datadog:
- Software Delivery Building Blocks at Datadog | Jacob LeGrone & Kevin Devroede
- Hacking Temporal to run on SQLite! | Jacob LeGrone
- Temporal @ Datadog | Jacob LeGrone
Jacob was also featured on an episode of Datadog On... in 2020: Datadog on Software Delivery | Jacob LeGrone, Benjamin Smith, & Ara Pulido