Datadog App Builder: Real Answers to Your Top Questions
Thanks to everyone who joined our recent deep-dive into Datadog App Builder (catch the replay here). We covered what it is, how it works, and when to use it—plus we rebooted an EC2 instance live. Unfortunately we ran out of time to get to all your questions during the Q&A time, but we kept a record of them to provide the answers below!
If you’re new to App Builder or want a faster primer before diving into the Q&A, check out our full App Builder breakdown.
Can I connect App Builder to custom internal APIs?
Yes! Datadog Workflows and the App Builder support private locations that you would run within your infrastructure that can then run the supported actions. All you would need at that point is any sort of authentication as well as the IP address / hostname, and then you can run the HTTP actions to retrieve data from an internal source.
I see App Builder & Workflow Automation RBAC permissions under the same heading; What are the main similarities and differences? Are there use cases that naturally lend themselves to one of both of these?
There are certainly use cases for either independent usage or using them together. The App Builder is great for interactive applications that require someone observe some data, make a decision, and push a button; or, for other tasks that need to be run in a controlled manner such as Creating an S3 Bucket, Managing Gitlab Deployments, or Managing Okta Users, Groups, and Roles.
Workflows are a great use case for automated actions, since they can be triggered by monitors. Some examples may be blocking an IP at Cloudflare based on SIEM activity, removing a problematic EC2 instance from an autoscaling group, or restarting a Kubernetes deployment. You can also run Workflows on a schedule, so you can run regular tasks like ensuring monitors can only be edited by the owning team.
The App Builder also has an action that can call Workflows. This can come in handy if there is a pre-existing workflow like some that were just mentioned, or if some more complex logic needs to be handled. Workflows have a flow-style pattern that allows for if/else and loops in a simpler manner than the App Builder does, so combining the two of them allows you more flexibility than either of them alone (depending on your use case, of course).
What performance impact (if any) of large or complex App Builder apps on the Datadog platform? And, what are the limitations of App Builder when building apps such as pricing, execution time, and API rate limits?
Given that Datadog is a SaaS platform, leave the performance concerns to them ;). That said, if you were to do something like polling way too much data and displaying it in a custom chart that has significant complexity, it’s entirely possible that it crashes your browser. But, that would happen with any other amount of large data processing that’s handled client side. Regarding limitations:
Execution time is going to be largely dependent on HTTP timeouts, since (whether it looks like it on the surface or not) these actions are all HTTP calls in one way or another. So you’re not likely going to be able to sit around for an hour waiting for a response, but if that’s something that needs to happen, I’d probably say you have bigger problems anyway…
API rate limits are dependent on the API that you’re hitting. On the Datadog side, the Datadog connection does rely on an API key, so you’d be subject to API key rate limits. This may be different if you are using an app to the local Datadog account, but this is not currently documented and I can’t be sure about that.
What if an EC2 reboot action times out or fails silently, how would you detect and retry?
Something to consider here is using polling intervals for the query that will cause the ListInstances query to re-run on a set interval. If a restart fails, it would not be in a running/started state, and you would have a decent indication that something went wrong. If it fails silently, you’ll likely see it in a stopped state and need to take some sort of additional troubleshooting action.
Can multiple engineers develop on an application synchronously?
Unfortunately, no. However, App Builder apps are typically not so complicated that they require multiple developers to work on them at the same time; it’s not like a typical application development cycle that requires feature branching and such.
Can App Builder connect directly to a database?
Unfortunately this is not currently supported natively. There may be some workarounds for this, such as calling a Lambda function to run the query (if you have that capability), or using some HTTP based interface such as the AWS Data API for RDS.
The pricing page says $35 per published app per month - can apps be run by other users without publishing them? Is there usage-based pricing on the number of app executions per month? Is there a way to track usage?
The app does need to be published for others to use them. The pricing is based on the app itself, and not the number of executions. Currently, the number of apps in an account is exposed in the Plan and Usage section.
What if I have an API I want to get data from that's not supported by a built-in action?
You can utilize the HTTP action to make calls to any HTTP endpoint with various parameters and types of authentication
Based on what you showed, can I use the AWS role I already have hooked up to Datadog? Does your AWS Sandbox appear in the Connections dropdown because it was already configured in the AWS Integration?
(For those of you who haven’t seen the webinar, we used an app to restart EC2 instances) Yes and no…Yes, you can reuse the same role, but you will need to configure the connection separately. It will not be available if you already have the AWS integration setup with Datadog. No, what we did today will not work with the Datadog AWS role. It requires additional permissions to take actions in AWS, as the Datadog role is read only.
How complex can the logic functions get?
If you can do it in vanilla Javascript, you can do it in App Builder. You cannot import other libraries, but beyond that, if you can write it, it should do it. If you want more examples of real-world App Builder apps (and how teams are actually using them), we’ve got a post for that—read it here.
How can I commit these app builder apps to a git repository, and can I use something like Terraform to maintain these?
You can view the JSON of the App Builder app and commit this to a repository. The Datadog Terraform provider can maintain the state of the app, or you can utilize the Datadog API.
Are there any minimum permissions required for connections to become selectable in App Builder?
Yes, you will need the “Connections Read” permission in the App Builder RBAC category.
What happens if a connection is not configured with enough permissions to perform the requested action? (For example, say we don’t have permissions to actually Reboot an Instance)
Anytime an action is run in the editor, you’ll get the output or an error message. If the role is not configured correctly, you’ll get a response (from AWS in this case) that you do not have permission to run XYZ operation based on the role.
Why would people ever use dashboards if you can embed graphs alongside functionality with App Builder?
There’s a few reasons. Primarily, App Builder apps have a cost associated with them, and dashboards do not. Second, the vast majority of dashboards in any given Datadog account do not need actions to be taken on them through something like the App Builder. Third, charts made in the App Builder require the use of Vega charts, so creating them is much more labor intensive than all of the built in dashboard widgets. Lastly, you can embed an app in a dashboard; doing this gives you the ease of built-in dashboard widgets with the power of an app right in a dashboard without the need for compromise.
That’s a wrap. If you’re still stuck or want to build something weird—reach out! We’re always up for making hard stuff easier.
