From Datadog to Prometheus

Datadog is a superb, all-in-one observability platform, and its consumption-based pricing (per host, plus per-GB ingest, plus per-module for APM, logs, RUM, and more) is one of the most unpredictable line items in modern infrastructure. Bill shock at scale is widely reported. The most common open-source replacement is a Prometheus + Grafana stack, often extended with Loki for logs and Tempo for traces. This guide is about doing that swap deliberately, not heroically.

Why teams leave Datadog

It’s almost never dissatisfaction with the product. It’s the cost trajectory: every new host, custom metric, high-cardinality tag, and module compounds the bill, and forecasting it is hard. Teams with the engineering capacity to run their own stack can cut spend dramatically, trading a managed SaaS for operational ownership.

What you’re actually replacing

Datadog bundles several products. Map each before you start:

• Infrastructure metrics → Prometheus (with node_exporter, cAdvisor, and app /metrics endpoints) or the OpenTelemetry Collector.
• Dashboards → Grafana.
• Monitors/alerts → Prometheus alerting rules + Alertmanager (routing, grouping, silences, paging).
• Logs → Loki (or Elasticsearch/OpenSearch).
• APM/traces → Tempo (or Jaeger), instrumented via OpenTelemetry.
• Synthetics/RUM → separate tooling (e.g., Blackbox exporter for uptime; RUM has fewer turnkey OSS options).

The gap to plan for: Datadog’s correlation across metrics/logs/traces and its polished UX take real effort to approximate. Grafana + Loki + Tempo get you most of the way, but you own the integration.

Sizing and cost model

Datadog bills largely per monitored host (plus ingest). Size your migration on the number of hosts/devices under monitoring and your metric/log volume. Self-hosting shifts cost to compute + storage + engineering time, usually far lower at scale, but not zero. Plan retention deliberately: long-retention, high-cardinality metrics are what made Datadog expensive, and they’ll size your Prometheus/Mimir and Loki storage too.

A safe migration flow

1. Inventory what Datadog is doing for you: dashboards, monitors, integrations, retention, and paging/ticketing hooks. Export dashboards and monitors via the Datadog API.
2. Stand up the stack. A common path is the kube-prometheus-stack Helm chart (Prometheus + Alertmanager + Grafana) for Kubernetes, plus Loki/Tempo as needed. Deploy node_exporter and instrument apps with exporters or OpenTelemetry.
3. Recreate the essentials first. Translate your most important monitors into PromQL alerting rules and rebuild the top dashboards in Grafana (or import community equivalents). Don’t try to recreate everything on day one, prioritize what pages humans.
4. Dual-run. Keep Datadog and the new stack running side by side; compare coverage, alert fidelity, and false-positive rates. This is where you find the gaps.
5. Cut over paging. Move Alertmanager → PagerDuty/Opsgenie/email once you trust the alerts, then decommission Datadog agents.

PromQL is a real shift

Datadog’s query language and Prometheus’s PromQL are different models. Rate calculations, histogram_quantile, label matching, and recording rules all need learning. Budget time for your on-call engineers to get fluent, alert quality depends on it. Recording rules and sensible scrape intervals also keep cardinality (and cost) under control.

The PromQL gotchas that catch teams

The specific gotchas that catch teams coming from Datadog:

• Counters need explicit ranges. Datadog often hides the rate math; in PromQL you write rate(http_requests_total[5m]) and pick the window yourself. Too short a window and the graph is noisy, too long and it lags the incident.
• Percentiles require histograms. A p95 monitor in Datadog becomes histogram_quantile(0.95, ...) over a _bucket metric, which only works if your app actually exports that histogram. If it does not, you have an instrumentation task before the alert can exist.
• Rollups and as_count/as_rate behavior have no direct PromQL twin; you decide aggregation with sum by (...), avg by (...), and recording rules that pre-compute the expensive series.
• Composite monitors map to alerting rules that reference other rules or and/unless vector matching, which reads very differently from Datadog’s UI-built composites.

Rewrite the monitors that page humans first, and check each fires on a real condition rather than assuming parity.

What doesn’t port cleanly

Set expectations before you promise a like-for-like migration:

• RUM and Synthetics. Datadog’s real-user monitoring has few turnkey OSS equivalents; uptime checks map to the Blackbox exporter, but front-end session analytics do not port directly.
• Auto-discovery breadth. The Datadog agent auto-detects and instruments many integrations out of the box. With Prometheus you deploy exporters per technology (node_exporter, database exporters, cAdvisor) and own that inventory.
• The correlation UX. Pivoting from a spiking metric straight into the related logs and traces is smooth in Datadog; in Grafana you approximate it with shared labels and trace IDs across Loki and Tempo, and it takes wiring.

Keeping the new bill down

Self-hosting moves the bill from a license to your infrastructure, and cardinality plus retention are the two dials that matter. Keep labels stable and bounded, use recording rules so dashboards read pre-aggregated series instead of scanning raw data, and choose scrape intervals deliberately. For long retention, add Thanos or Mimir so you are not forcing a single Prometheus to hold months of high-resolution data. The metrics that were expensive in Datadog because of cardinality and long retention are exactly the ones that will size your storage here, so this is where the durable savings are won or lost.

Proving it before you switch paging off

Before switching paging off Datadog: fire test alerts end-to-end (trigger → Alertmanager → pager → acknowledgement), do a dashboard parity review against the metrics that matter, and run a retention/scale load test so you’re not surprised when storage fills. Treat “we get paged correctly for the incidents we care about” as the acceptance bar.

Making the call

Datadog → Prometheus/Grafana is primarily a cost and ownership decision. The metrics and dashboards migrate well; alerting needs PromQL fluency; correlated logs/traces and polished UX take the most effort to match. Run both stacks in parallel, prove alert fidelity, then cut over paging last. Model your per-host savings in the calculator above, and validate the numbers against a real quote, since self-hosting trades license cost for engineering time.