DMARC Reporting Pipeline: End-to-End RUA + RUF Ingestion, Format Detection, Parsing, Storage, Retention, and Alerting Reference

The operator-implementation reference for building a working DMARC reporting pipeline: six pipeline stages (ingestion, format detection, parsing, storage and deduplication, retention, analysis and alerting) anchored against the DMARC Analyzer project reference implementation in includes/DmarcParser.php + ForensicReportParser.php + TlsRptParser.php + ImapFetcher.php + GraphFetcher.php. Where the aggregate-reports page documents the RUA XML SCHEMA and the forensic-reports page documents the RUF MIME format, this page answers the question "I have a rua=mailto: address; what does the software behind it actually do?"

This pipeline reference is provided in English. The operational vocabulary (rua=, ruf=, AFRF, MIME envelope, IMAP4rev1, Microsoft Graph, Mail.Read, OAuth 2.0 client-credentials, magic bytes, gzip / zip / tar.gz extension semantics, INSERT IGNORE, ON CONFLICT DO NOTHING, AES-256-GCM, application/feedback-report) is standardised RFC / vendor terminology that does not translate. We keep the body in the source language so the configuration values and code references remain copy-paste safe across locales.

TL;DR

  • A DMARC reporting pipeline turns rua= aggregate and ruf= forensic reports landing in a monitoring mailbox into queryable database records and operational alerts. It has six stages: ingestion, format detection, parsing, storage, retention, and analysis/alerting.
  • Three ingestion patterns are canonical: direct SMTP delivery to the rua=mailto: address, IMAP4rev1 polling of a monitoring mailbox, and Microsoft Graph polling of an Entra ID mailbox via the Mail.Read application permission.
  • Format detection uses magic bytes, not filename extensions. gzip starts with 0x1f 0x8b; zip starts with PK\x03\x04; tar.gz is gzip wrapping a tar archive; raw XML starts with <?xml or <feedback. The DMARC Analyzer project implementation lives in includes/DmarcParser.php.
  • RUA aggregate reports use the RFC 7489 Appendix C XML schema with four blocks: <report_metadata>, <policy_published>, one or more <record>s, each with <row> + <identifiers> + <auth_results>.
  • RUF forensic reports use the RFC 6591 AFRF MIME format with multipart/report containing a human-readable text section, a message/feedback-report key-value block, and a message/rfc822 or message/rfc822-headers attachment. The DMARC Analyzer parser lives in includes/ForensicReportParser.php.
  • Deduplicate on the tuple (org_name, report_id) via a database UNIQUE constraint; INSERT IGNORE or ON CONFLICT DO NOTHING handles concurrent ingestion. Add a SHA-256 content hash as a secondary key for senders that reuse report_id across date ranges.
  • Typical retention: 365 days for RUA records, 90 days for RUF records with message-content fields purged after 30 days. GDPR Article 4(1) treats source IPs as potentially personal data; data minimisation argues for time-bounded retention.
  • Six alert categories close the loop: ingestion-health, alignment-regression, volume-anomaly, sender-inventory drift, reporter-coverage, and policy-disposition shift. See the DMARC Monitoring page for the dashboard discipline that consumes these alerts.
  • The DMARC Analyzer project ships reference implementations of every stage: includes/DmarcParser.php (XML + gzip + zip + tar.gz detection and parsing per RFC 7489), includes/ForensicReportParser.php (AFRF MIME envelope per RFC 6591), includes/TlsRptParser.php (JSON TLS-RPT per RFC 8460 for the related TLS reporting pipeline), includes/ImapFetcher.php (IMAP4rev1 mailbox polling), and includes/GraphFetcher.php (Microsoft Graph mailbox polling via OAuth 2.0 client-credentials).

1. What this guide covers

This is the operator-implementation reference for building a working DMARC reporting pipeline. Where the aggregate reports deep dive documents the RUA XML schema and the forensic reports deep dive documents the RUF AFRF MIME format, this page covers the END-TO-END PIPELINE that ingests, decompresses, parses, deduplicates, stores, analyses, and alerts on those reports. It is the answer to the question "I have a rua=mailto: address; what does the software behind it actually do?"

The page anchors against the DMARC Analyzer project's own reference implementation in includes/DmarcParser.php, includes/ForensicReportParser.php, includes/TlsRptParser.php, includes/ImapFetcher.php, and includes/GraphFetcher.php, so readers can map each documented pipeline stage onto a working code path. The pipeline stages are protocol-agnostic; the design generalises to any stack (Python, Go, Ruby, Node.js, .NET, Java) that can speak SMTP, IMAP4rev1, or HTTPS.

Reading order: start with the architecture overview in Section 2 to internalise the six pipeline stages, then walk through each stage in Sections 3 through 8. Sections 9 and 10 cover the two recommended ingestion-source integrations (IMAP and Microsoft Graph). Section 11 covers the monitoring-mailbox operational concerns that every pipeline depends on. Sections 12 and 13 cover troubleshooting and common mistakes. Section 14 is an 8-question FAQ.

2. Pipeline architecture

A DMARC reporting pipeline has six discrete stages, each with well-defined inputs, outputs, and failure modes. The stages are pipelined - a single attachment flows through every stage exactly once - but each stage can be unit-tested in isolation against a canonical fixture set. 

+---------------+    +-------------+    +-------------+
| 1. Ingestion  |--->|  2. Format  |--->|  3. Parse   |
|    SMTP /     |    |  Detection  |    |  RUA XML /  |
|    IMAP /     |    |  XML/gz/zip |    |  RUF MIME   |
|    Graph      |    |  /tar.gz    |    |             |
+---------------+    +-------------+    +-------------+
                                              |
                                              v
+---------------+    +-------------+    +-------------+
| 6. Analysis & |<---| 5. Retention|<---|  4. Store + |
|    Alerting   |    |  GDPR / cost|    |  Deduplicate|
|    KPIs / SLO |    |  365d RUA   |    |  UNIQUE     |
|               |    |  90d RUF    |    |  (org_name, |
|               |    |             |    |   report_id)|
+---------------+    +-------------+    +-------------+

Each arrow represents a hand-off where the pipeline can be paused, retried, or audited. The DMARC Analyzer project implements this exact shape with upload.php as the manual ingestion entrypoint, imap_fetch.php as the automated IMAP + Graph ingestion entrypoint, includes/DmarcParser.php as the format-detection-plus-XML-parser, and the dmarc_reports + dmarc_records + dmarc_forensic_reports tables as the storage tier.

Designing for the six-stage shape pays off in three ways. (1) Operational observability: each stage emits metrics (records-ingested, records-failed-parse, records-deduplicated, records-stored, records-retained, alerts-fired) so failures are localised. (2) Retry safety: stages are idempotent on the deduplication tuple, so a daemon that crashes mid-batch can safely re-ingest the same emails without producing duplicate records. (3) Vendor portability: a stack that ingests via IMAP today can swap to Graph tomorrow without touching Stages 2-6.

3. Stage 1: Ingestion sources

Three canonical patterns deliver DMARC reports to the pipeline. Pick one as the primary; many programs run two in parallel for redundancy.

3.1 Direct SMTP delivery to the rua=mailto: address

The simplest pattern: the rua=mailto: address in the DMARC TXT record is a real SMTP inbox that receives reports as the mail server delivers them. Receivers send aggregate reports every 24 hours (RFC 7489 §7.2.1) and forensic reports as failures occur (§7.3). The inbox must be DMARC-compliant itself - if the mailbox's own domain has p=reject and the reporter is sending from an unaligned sender, the report bounces and the inbox sees zero traffic. This is the canonical "your DMARC reports stopped arriving because you enforced DMARC on the rua= domain" chicken-and-egg failure.

Implementation: publish the rua= address pointing at a mailbox on a domain you fully control; ensure the mailbox's own DMARC alignment is correct; monitor the mailbox for inbound-mail volume as a reporter-coverage KPI (see Section 8). Direct SMTP is fine for low-volume programs but does not scale to multi-tenant pipelines because each rua= address needs its own mailbox.

3.2 IMAP4rev1 polling

Most production pipelines run an IMAP poller that periodically connects to the monitoring mailbox, fetches new messages, processes attachments, and marks messages as Seen or moves them to a processed folder. The protocol is RFC 3501 IMAP4rev1; the canonical PHP integration is the imap_open() family of functions from the IMAP PECL extension. The DMARC Analyzer project ships includes/ImapFetcher.php as the reference implementation (see Section 9).

Operational concerns: (1) authentication credentials should be encrypted at rest (the DMARC Analyzer project encrypts IMAP passwords with AES-256-GCM using a key derived from the DB_PASS environment variable - see the project's CLAUDE.md security patterns); (2) the poller must handle partial-batch failures by marking only successfully-processed messages as Seen; (3) connection rate limits at the mailbox provider mean the poll interval must be at least 5 minutes for most providers; (4) Microsoft 365 mailboxes are progressively disabling Basic Authentication so the Graph pattern in Section 10 is preferred there.

3.3 Microsoft Graph polling

For Microsoft 365 monitoring mailboxes, the Microsoft Graph API is the preferred ingestion path. Graph exposes /v1.0/users/{user-id}/messages for reading mailbox messages with attachments, uses OAuth 2.0 client-credentials (no mailbox passwords), and does not require Basic Authentication. The DMARC Analyzer project ships includes/GraphFetcher.php as the reference implementation (see Section 10).

Setup requires registering an Entra ID (Azure AD) application with the Mail.Read application permission (NOT delegated), tenant admin consent, and either a client secret or a certificate for the OAuth flow. Each access token is valid for 1 hour by default; the fetcher refreshes tokens on-demand.

4. Stage 2: Format detection and extraction

DMARC aggregate reports arrive as one of four container shapes:

  1. Raw XML - rare, typically only from misconfigured reporters. Content-Type: application/xml or text/xml.
  2. gzip - the most common shape. Content-Type: application/gzip or application/x-gzip or application/octet-stream. Filename extension .xml.gz.
  3. zip - common from Microsoft and certain older Yahoo reporters. Content-Type: application/zip. Filename extension .zip.
  4. tar.gz - rare but seen from a handful of academic mailbox operators. gzip wrapping a tar archive. Filename extension .tar.gz or .tgz.

Filename extensions disagree across reporters, so robust pipelines detect by MAGIC BYTES (the first few bytes of the attachment). The DMARC Analyzer project implementation in includes/DmarcParser.php walks attachments through gzip > zip > raw-XML detection in that order. 

// Magic-byte detection (the canonical pattern):

gzip:    first 2 bytes are 0x1f 0x8b
zip:     first 4 bytes are 0x50 0x4b 0x03 0x04 ("PK\x03\x04")
tar.gz:  gzip header followed by tar archive at decompressed offset 0;
         tar marker "ustar" appears at byte 257 of the decompressed payload
xml:     first non-whitespace bytes are "<?xml" or "<feedback"
         (allow optional UTF-8 / UTF-16 BOM before the first <)

Extraction proceeds in two steps. First, the pipeline decompresses to a raw byte buffer containing the XML payload (or the AFRF MIME for RUF reports). Second, character-encoding detection handles the UTF-8 / UTF-16 / ASCII variants that reporters emit; XML parsers consume the encoding declaration from the <?xml version="1.0" encoding="..."?> prologue if present.

Failure modes: corrupted attachments (gzip CRC mismatch, zip central-directory truncation), nested archives (zip-inside-tar, gzip-inside-zip - rare but seen in the wild), and content-type spoofing where the attachment is text but declared binary. The pipeline must log every parse failure with the source mailbox, message ID, and the first 32 bytes of the attachment so operators can audit a misbehaving reporter.

5. Stage 3: Parsing (RUA XML and RUF MIME)

5.1 RUA XML parsing per RFC 7489 Appendix C

DMARC aggregate (rua) reports use the XML schema defined in RFC 7489 Appendix C. The root element is <feedback> with four top-level children: 

<feedback>
  <report_metadata>
    <org_name>google.com</org_name>
    <email>noreply-dmarc-support@google.com</email>
    <report_id>1234567890</report_id>
    <date_range>
      <begin>1716163200</begin>
      <end>1716249600</end>
    </date_range>
  </report_metadata>
  <policy_published>
    <domain>example.com</domain>
    <adkim>r</adkim>
    <aspf>r</aspf>
    <p>reject</p>
    <sp>reject</sp>
    <pct>100</pct>
  </policy_published>
  <record>
    <row>
      <source_ip>192.0.2.1</source_ip>
      <count>42</count>
      <policy_evaluated>
        <disposition>none</disposition>
        <dkim>pass</dkim>
        <spf>pass</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>example.com</header_from>
    </identifiers>
    <auth_results>
      <dkim>
        <domain>example.com</domain>
        <selector>default</selector>
        <result>pass</result>
      </dkim>
      <spf>
        <domain>example.com</domain>
        <scope>mfrom</scope>
        <result>pass</result>
      </spf>
    </auth_results>
  </record>
</feedback>

Parser design rules:

  1. Use a DOM or SAX parser, not regex. DMARC reports are well-formed XML; PHP's SimpleXMLElement, Python's xml.etree.ElementTree, or any equivalent SAX parser is the right tool. The DMARC Analyzer project uses SimpleXMLElement with LIBXML_NONET | LIBXML_NOENT | LIBXML_NOCDATA flags to disable external entity resolution (defence against XXE).
  2. Ignore unknown elements. Reporters extend the schema with non-standard elements (Google adds <extra_contact_info>, Microsoft occasionally adds organization-specific annotations). Robust parsers ignore unknown children rather than rejecting the report.
  3. Coerce date_range values to UTC timestamps. The begin / end fields are Unix epoch seconds in UTC per RFC 7489 §7.2.1.1; store them as BIGINT for queryability.
  4. Normalise the disposition value. Disposition is one of none, quarantine, reject; normalise to lowercase for consistency.
  5. Accept multiple <dkim> and <spf> blocks inside <auth_results>. Multi-domain DKIM signing is increasingly common; the parser must collect all sub-blocks rather than overwriting on each iteration.

5.2 RUF AFRF MIME parsing per RFC 6591 + RFC 7489 §7.3

DMARC forensic (ruf) reports use the Abuse Reporting Format (AFRF) defined in RFC 6591, with DMARC-specific extensions in RFC 7489 §7.3. The container is a multipart/report; report-type=auth-failure MIME envelope with three parts:

  1. A text/plain human-readable section describing the failure;
  2. A message/feedback-report machine-readable key-value section with the canonical AFRF fields (Feedback-Type, Version, User-Agent, Original-Mail-From, Arrival-Date, Source-IP, Reported-Domain, Auth-Failure, DKIM-Domain, DKIM-Selector, Identity-Alignment, Delivery-Result);
  3. A message/rfc822 or message/rfc822-headers attachment with the headers (and optionally the body) of the failing message.

The DMARC Analyzer project parser (includes/ForensicReportParser.php) uses PHP's mailparse extension to walk the MIME tree, extracts the AFRF key-value block as a normalised array, and stores the originating message-headers fragment as a TEXT field for operator review. Forensic reports are far more privacy-sensitive than aggregate reports (they contain message content); pipelines must surface this to operators via storage-tier access controls.

6. Stage 4: Storage and deduplication

6.1 The canonical schema shape

The DMARC Analyzer project schema separates the report-level metadata from the per-record rows: 

-- dmarc_reports: one row per ingested aggregate report
CREATE TABLE dmarc_reports (
    id            BIGINT      PRIMARY KEY AUTO_INCREMENT,
    org_name      VARCHAR(255) NOT NULL,
    email         VARCHAR(255),
    report_id     VARCHAR(255) NOT NULL,
    date_begin    BIGINT      NOT NULL,
    date_end      BIGINT      NOT NULL,
    domain        VARCHAR(255) NOT NULL,
    policy_p      VARCHAR(32),
    policy_sp     VARCHAR(32),
    policy_adkim  CHAR(1),
    policy_aspf   CHAR(1),
    policy_pct    SMALLINT,
    raw_xml_hash  CHAR(64),         -- SHA-256 for content-hash dedup
    ingested_at   TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE KEY uq_report (org_name, report_id)
);

-- dmarc_records: one row per <record> inside a report
CREATE TABLE dmarc_records (
    id              BIGINT      PRIMARY KEY AUTO_INCREMENT,
    report_id       BIGINT      NOT NULL,
    source_ip       VARCHAR(45) NOT NULL,
    message_count   INT         NOT NULL,
    disposition     ENUM('none','quarantine','reject') NOT NULL,
    dkim_eval       ENUM('pass','fail') NOT NULL,
    spf_eval        ENUM('pass','fail') NOT NULL,
    header_from     VARCHAR(255),
    dkim_domain     VARCHAR(255),
    dkim_selector   VARCHAR(255),
    dkim_result     VARCHAR(32),
    spf_domain      VARCHAR(255),
    spf_result      VARCHAR(32),
    FOREIGN KEY (report_id) REFERENCES dmarc_reports(id) ON DELETE CASCADE,
    INDEX idx_source_ip (source_ip),
    INDEX idx_dkim_eval (dkim_eval),
    INDEX idx_spf_eval (spf_eval)
);

-- dmarc_forensic_reports: one row per ingested RUF report
CREATE TABLE dmarc_forensic_reports (
    id              BIGINT      PRIMARY KEY AUTO_INCREMENT,
    feedback_type   VARCHAR(64) NOT NULL,
    arrival_date    BIGINT,
    source_ip       VARCHAR(45),
    reported_domain VARCHAR(255),
    auth_failure    VARCHAR(255),
    dkim_domain     VARCHAR(255),
    original_headers TEXT,
    ingested_at     TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE KEY uq_forensic (feedback_type, arrival_date, source_ip)
);

Decisions to lock in: source_ip as VARCHAR(45) accommodates IPv6 textual forms; BIGINT for Unix timestamps survives the 2038 problem; per-record FK on dmarc_reports.id with ON DELETE CASCADE makes retention cleanup atomic.

6.2 Deduplication strategies

A DMARC report is uniquely identified by the tuple (org_name, report_id, date_range_begin, date_range_end). The same report can arrive multiple times due to mailbox retries, poller crashes mid-batch, or sender-side reissues. Three deduplication strategies, in increasing robustness:

  1. Database UNIQUE constraint on (org_name, report_id). Combined with INSERT IGNORE (MariaDB / MySQL) or ON CONFLICT DO NOTHING (PostgreSQL), this handles duplicates atomically without any application-side bookkeeping. The most-recommended baseline.
  2. SHA-256 content hash stored in the reports table, checked before insert. Useful when a misbehaving sender reuses report_id across different date ranges (rare but seen). The DMARC Analyzer project schema includes a raw_xml_hash column for this purpose.
  3. Idempotency-key middleware in the ingestion daemon, with a short-TTL cache (Redis, Memcached) of recently-seen (org_name, report_id) tuples. Useful when the deduplication must happen BEFORE any database write (high-volume pipelines where rejected inserts are still expensive).

Most production pipelines combine (1) and (2) for defence in depth.

7. Stage 5: Retention policies

Four considerations drive DMARC report retention.

  1. Operational utility. 90-day rolling aggregate reports support quarterly trend analysis, alignment-pass-rate regression detection, and sender-inventory drift detection. Reports older than 90 days rarely surface in operational queries.
  2. Compliance. Most security frameworks (SOC 2, ISO 27001, PCI-DSS) treat DMARC reports as security-monitoring artifacts and require minimum retention of 12 months. The DMARC Analyzer project defaults RUA retention to 365 days for this reason.
  3. GDPR / data minimisation. Aggregate reports contain source IPs that may qualify as personal data under GDPR Article 4(1); apply data minimisation by deleting records older than the operational + compliance minimum. Forensic (RUF) reports contain message content and headers and are far more privacy-sensitive; retain only as long as needed for the specific investigation.
  4. Storage cost. At enterprise volumes (10,000+ messages per day across all reporters), aggregate-record rows accumulate at ~100 MB per month; cap retention at 365 days for most programs to keep the storage tier cost-bounded.

A typical retention policy: RUA records kept 365 days; RUF records kept 90 days with message-content fields purged after 30 days. Implement via a nightly cron that issues DELETE FROM dmarc_reports WHERE date_end < UNIX_TIMESTAMP() - 31536000 and the equivalent for the forensic table. The ON DELETE CASCADE on dmarc_records.report_id means per-record cleanup is automatic.

8. Stage 6: Analysis and alerting

Analysis turns stored records into the KPIs documented in the DMARC Monitoring deep dive (authenticated volume, alignment pass rate, SPF / DKIM pass rates, distinct sending sources, disposition distribution, reporter coverage). Alerting turns threshold breaches into actionable operator notifications.

Six categories of alert close the operational loop:

  • Ingestion-health alerts: rua= mailbox unreachable, IMAP authentication failure, Graph token refresh failure, attachment parse failure rate above 5 percent.
  • Alignment-regression alerts: alignment pass rate for any enforced domain drops more than 5 percentage points below the 7-day rolling average.
  • Volume-anomaly alerts: authenticated message volume drops more than 20 percent day-over-day or spikes more than 5x against the prior 14-day median.
  • Sender-inventory drift alerts: a new source IP appears with more than 100 messages from an IP not seen in the past 30 days.
  • Reporter-coverage alerts: count of distinct reporting organizations drops below 5 per day for a high-volume sender.
  • Policy-disposition shift alerts: an enforced domain shows quarantine or reject disposition above 1 percent of total volume from a previously-aligned receiver.

The DMARC Analyzer dashboard surfaces all six categories via the source-IP detail view, the reporter breakdown, and the per-domain alignment trend - see reporter detail and source-IP detail in the running application.

9. IMAP mailbox setup (ImapFetcher.php)

The DMARC Analyzer project ships includes/ImapFetcher.php as the reference IMAP4rev1 polling integration. The fetcher connects to a monitoring mailbox over IMAPS (TLS-encrypted port 993), iterates the INBOX for unread messages with attachments, extracts the attachment bytes, hands them off to DmarcParser, and marks the message as Seen on successful ingestion.

Per-user configuration (encrypted at rest via AES-256-GCM in the users table): 

imap_host         = "imap.example.com"
imap_port         = 993
imap_username     = "dmarc-reports@example.com"
imap_password     = "<encrypted>"   // AES-256-GCM
imap_use_ssl      = 1
imap_inbox        = "INBOX"
imap_processed    = "INBOX.Processed"  // optional: move-after-ingest
imap_poll_minutes = 15

Authentication strategies vary by provider. Generic IMAP servers accept Basic Authentication (username + password); Gmail requires an application password when 2FA is enabled; Microsoft 365 has progressively disabled Basic Authentication so the Microsoft Graph path (Section 10) is preferred there. The fetcher logs each ingestion attempt to the api_requests table for audit.

10. Microsoft Graph setup (GraphFetcher.php)

The DMARC Analyzer project ships includes/GraphFetcher.php as the reference Microsoft Graph polling integration. The fetcher uses OAuth 2.0 client-credentials grant (no mailbox passwords) and the Mail.Read application permission.

Tenant administrator setup:

  1. In the Entra ID admin centre, register a new application. Record the Application (client) ID and the Directory (tenant) ID.
  2. Add the API permission Microsoft Graph > Application permissions > Mail.Read. Click "Grant admin consent" on the consent flow.
  3. Generate a client secret OR upload a certificate in Certificates & secrets. Record the secret value at generation time - the portal does not show it again.
  4. Provide the tenant ID, client ID, and secret to the fetcher via the per-user configuration (encrypted at rest):
graph_tenant_id     = "11111111-1111-1111-1111-111111111111"
graph_client_id     = "22222222-2222-2222-2222-222222222222"
graph_client_secret = "<encrypted>"   // AES-256-GCM
graph_mailbox       = "dmarc-reports@example.com"
graph_poll_minutes  = 15

The fetcher requests an access token from https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token with grant_type=client_credentials and scope=https://graph.microsoft.com/.default, caches the token until expiry (1 hour), and uses it to GET /v1.0/users/{mailbox}/messages?$filter=hasAttachments+eq+true&$expand=attachments. Each returned message has its attachments as base64-encoded contentBytes fields; the fetcher decodes, decompresses, parses, and stores via the same Stage 2-4 pipeline used by IMAP. After successful ingestion, the fetcher PATCHes the message with isRead: true.

11. Monitoring mailbox best practices

A dedicated, single-purpose monitoring mailbox is the load-bearing component of a DMARC reporting pipeline. Eight best practices:

  1. Dedicated address per program. Do not use a personal inbox; route to dmarc-reports@<your-domain>. This keeps DMARC reports out of personal mail and provides clean access boundaries for ingestion daemons.
  2. The mailbox's own domain must be DMARC-compliant. If your reporting domain has p=reject and a reporter sends from an unaligned address, the report bounces and never lands. Verify with the domain checker before going live.
  3. No catch-all. Spam routing rules at the receiving server should not intercept DMARC reports. Whitelist the most common reporter sources (noreply-dmarc-support@google.com, dmarcreport@microsoft.com, postmaster@yahoo.com) as belt-and-braces.
  4. Accept large attachments. Aggregate reports from high-volume senders regularly exceed 1 MB compressed; raise the mailbox attachment ceiling above the default 25 MB if hitting limits.
  5. Plain-text content disposition. Disable any inbox-side rich-content rewrite (HTML stripping, link wrapping); DMARC reports are application/xml / application/gzip / application/zip and should pass through untouched.
  6. Retain raw emails for at least 90 days. Even after parsing succeeds, the raw email lets operators re-process if the parser is later updated to extract additional fields. Store at the mail-server tier (cheap) or in a separate raw-archive table.
  7. Monitor the mailbox volume as a reporter-coverage KPI. A sudden drop in daily inbound volume signals a broken DMARC TXT, a bouncing mailbox, or a dropped sending relationship.
  8. Encrypt mailbox credentials at rest. IMAP passwords, Graph client secrets, refresh tokens: all must be encrypted at rest (the DMARC Analyzer project uses AES-256-GCM with a key derived from DB_PASS).

12. Pipeline troubleshooting

The four most-common pipeline failure modes and their diagnostic steps:

12.1 Reports arrive but parser fails on every one

Symptom: the IMAP fetcher logs records ingested = 0 while the mailbox shows inbound traffic. Diagnosis: the format-detection stage is failing. Capture the first 32 bytes of a failed attachment via the parse-error log; compare against the magic-byte table in Section 4. Common cause: a reporter is sending a new container shape (e.g., zstd compression seen from one experimental operator); add detection for it OR exclude that reporter from the polling glob.

12.2 Reporter coverage drops to zero

Symptom: aggregate-report volume goes from 10 reporters/day to 0 within 24 hours. Diagnosis: the rua= mailbox is bouncing OR the DMARC TXT record is malformed OR the reporting domain itself has p=reject and the reporter is sending from an unaligned address. Check inbox bounces; re-validate the DMARC TXT via the domain checker; verify the rua= mailbox's own DMARC alignment.

12.3 Deduplication is failing - duplicate records

Symptom: the same source IP appears with double the expected count for a single day. Diagnosis: the UNIQUE constraint on (org_name, report_id) is missing OR the constraint is present but the parser is extracting different report_id values for the same logical report. Verify the schema; check that the parser is reading the <report_id> element rather than falling back to a per-message hash.

12.4 OAuth token refresh fails (Graph)

Symptom: Graph fetcher logs HTTP 401 - invalid_client at every poll. Diagnosis: the client secret has expired (the portal default is 24 months), the application has lost its Mail.Read permission, OR a tenant Conditional Access policy is blocking client-credentials grants from the daemon's IP. Re-issue the client secret in Entra ID; verify the permission grant; whitelist the daemon IP in the Conditional Access policy if applicable.

13. Common pipeline mistakes

  1. Relying on filename extensions for format detection. Use magic-byte detection; extensions disagree across reporters.
  2. Using regex to parse the XML. Use a DOM or SAX parser; regex fails on unexpected whitespace, CDATA sections, and nested elements.
  3. Resolving external entities during XML parse. XXE injection risk; always disable external-entity resolution (LIBXML_NONET | LIBXML_NOENT in PHP).
  4. No UNIQUE constraint on report_id. Duplicate ingestion produces inflated KPIs and confused operators.
  5. Marking messages as Seen before parse succeeds. A daemon crash mid-batch loses the messages permanently. Mark as Seen only AFTER successful storage.
  6. No retention policy. Tables grow without bound; query performance degrades; GDPR exposure accumulates.
  7. Polling too fast. Most mailbox providers rate-limit IMAP connections at one per 5 minutes; poll less often.
  8. Storing IMAP passwords in plaintext. Always encrypt at rest. The DMARC Analyzer project uses AES-256-GCM with a DB_PASS-derived key.
  9. Treating RUA and RUF identically. RUF contains message content and is far more privacy-sensitive than RUA; apply tighter retention and access controls.
  10. Ignoring parse errors. Each parse failure indicates a misbehaving reporter OR a pipeline gap. Log every failure with the source mailbox + message ID + first 32 bytes.

14. FAQ

What is a DMARC reporting pipeline?
A DMARC reporting pipeline is the end-to-end software workflow that turns DMARC aggregate (RUA) and forensic (RUF) reports landing in a monitoring mailbox into queryable database records and operational alerts. It has six stages: ingestion, format detection, parsing, storage, retention, and analysis / alerting.
How do I ingest DMARC aggregate reports automatically?
Three canonical patterns. Direct SMTP delivery to the rua= mailbox; IMAP4rev1 polling of a monitoring mailbox (the DMARC Analyzer project ships includes/ImapFetcher.php); or Microsoft Graph polling via the Mail.Read application permission and OAuth 2.0 client-credentials (the DMARC Analyzer project ships includes/GraphFetcher.php).
How does the pipeline detect whether a DMARC report attachment is XML, gzip, zip, or tar.gz?
Magic-byte detection. gzip starts with 0x1f 0x8b; zip starts with PK\x03\x04; tar.gz is gzip wrapping a tar archive; raw XML starts with <?xml or <feedback. Filename extensions are advisory only.
What XML schema do DMARC aggregate reports use?
RFC 7489 Appendix C. The root element is <feedback> containing <report_metadata>, <policy_published>, and one or more <record> blocks with <row> + <identifiers> + <auth_results>.
How does pipeline deduplication work for DMARC reports?
A DMARC report is uniquely identified by (org_name, report_id, date_range_begin, date_range_end). Deduplicate via a database UNIQUE constraint on (org_name, report_id) with INSERT IGNORE / ON CONFLICT DO NOTHING. Add a SHA-256 content hash as a secondary key for senders that reuse report_id across date ranges.
What retention policy should I apply to DMARC reports?
Typical: 365 days for RUA records, 90 days for RUF records with message-content fields purged after 30 days. Driven by operational utility (90-day rolling window), compliance (12-month minimum for SOC 2 / ISO 27001 / PCI-DSS), GDPR data minimisation, and storage cost at enterprise volumes.
How do I poll a Microsoft 365 mailbox for DMARC reports via the Graph API?
Register an Entra ID application, grant the Mail.Read application permission with tenant admin consent, generate a client secret or certificate, then use OAuth 2.0 client-credentials to GET /v1.0/users/{mailbox}/messages?$filter=hasAttachments+eq+true. See the DMARC Analyzer project's includes/GraphFetcher.php for the reference implementation.
What alerts should the pipeline fire?
Six categories: ingestion-health, alignment-regression, volume-anomaly, sender-inventory drift, reporter-coverage, and policy-disposition shift. See Section 8 and the DMARC Monitoring deep dive for the dashboard discipline that consumes them.

Last updated: {date}: 2026-10-12