Building an effective n8n sitemap crawl without falling into an infinite loop can be tricky. My goal was to create a daily monitoring system for a live WordPress site using the n8n automation tool. This project involved crawling the XML sitemap to track and verify the status of pages. The challenge was to design a workflow that efficiently processes the sitemap without looping indefinitely or generating false alerts. Here’s how I tackled it.
Designing the Sitemap Crawling Workflow
The workflow consisted of a series of five nodes, each performing a specific function in the sitemap crawling process. The nodes were organized in the following sequence:
- Fetch Sitemap Index (HTTP Request): Retrieves the main sitemap index file.
- List Child Sitemaps (Code): Parses the sitemap index to list all child sitemaps.
- Fetch Child Sitemaps (HTTP Request): Downloads each child sitemap.
- List Page URLs (Code): Extracts all page URLs from the child sitemaps.
- Fetch Each Page (HTTP Request): Fetches each page to perform checks.
Unlike conventional methods, this workflow did not use any loop nodes like “Split In Batches” or “Loop Over Items.” Instead, it leveraged n8n’s ability to automatically repeat actions for each item output by a node, simplifying the process and avoiding potential infinite loops.
Implementing Controlled Batching
Batching was crucial to prevent overwhelming the server with requests. The HTTP Request nodes were configured with built-in batching options. For the “Fetch Child Sitemaps” node, batchSize was set to 1 and batchInterval to 1200 ms. Similarly, the “Fetch Each Page” node was set to batchSize 1 and batchInterval 1500 ms. These settings allowed controlled, timed requests without manual loops.
Bug 1: The Response That Was Always Empty
Symptom
The “List Child Sitemaps” node consistently threw the error “No child sitemaps found in sitemap_index.xml” during every run, despite the sitemap index being valid when opened in a browser.
Investigation
The HTTP Request node’s output was examined field by field, confirming the request’s success and the presence of the XML data. However, the code was reading a non-existent property.
Cause
The node was set to responseFormat text, placing the response in the data property. The code incorrectly read from body, which returned undefined and caused the guard clause to fire.
Fix
The code was adjusted to read data first and fall back to body:
const xml = typeof j.data === 'string' ? j.data : (typeof j.body === 'string' ? j.body : '');Result
The workflow successfully found its child sitemaps on the next run.
Bug 2: A File That Is Not a Page
Symptom
Checks for missing title and meta description erroneously triggered on a .kml map file.
Investigation
Directly opening the URL revealed a valid KML file, not an HTML document, explaining the absence of HTML elements.
Cause
The URL list included all loc entries from child sitemaps without filtering out non-HTML files.
Fix
Implemented two filters: first, exclude URLs with paths ending in known non-HTML extensions (e.g., xml, kml, pdf); second, check the content-type header and skip non-HTML responses.
Result
The URL count reduced from 89 to 88, and the false alerts were resolved.
Why There Is No Loop Node
Loop nodes like “Split In Batches” were deliberately avoided for three reasons:
- Avoiding Infinite Loops: Loop nodes require precise exit conditions, and errors can lead to infinite loops.
- n8n’s Built-in Repeating: n8n automatically repeats actions for each item, making manual loops redundant.
- Simpler Rate Limiting: Using
batchSizeandbatchIntervalis more straightforward than coding a loop with a counter and wait node.
Want this built for you
If you need a custom solution to automate your sitemap crawling or any other task, consider exploring AI Solutions and Automation. We can design and implement workflows tailored to your specific requirements, leveraging tools like n8n to streamline your processes. Get in touch to learn how we can assist with your automation needs.