All well and good. But a lot of these problems are self-inflicted and can be sorted with good old-fashioned technical SEO.
For example, Profound's own pricing page has a features table that's... not a table.
Which means that when it's fetched by a bot/agent that's not layout-aware, it can end up as a jumbled mess of markdown, with missing feature flags, duplicated sections, and values disconnected from their labels.
Not very machine readable.
Third screenshot here is a fetch of their pricing page via Exa, which is used by a lot of agents and I assume they care about. Other extraction pipelines may encounter similar problems.
Yes, there are genuine cases where this (unrendered content) can be an issue. But this particular example is just a straightforward technical SEO problem.
So before you get to fancy solutions and serving different documents to different user agents, start by... making sure your main document is well structured.
Happy to help Profound with the fix, although I assume there are people on their team that can handle it. I do see this regularly on Ecommerce sites, so they're not alone.
But probably quite important that your pricing page is machine readable, particularly if you're marketing a product around all that.
Notes:
1. Image 4 is what ChatGPT was able to reconstruct from the Exa return.
2. ChatGPT's own fetch did return the missing fields.
Sidenote: sometimes a table should be a table 🤷
Introducing Dynamic Bot Rendering.
When an AI bot visits your site, we serve it a fully loaded version of the page. Your human visitors’ experience stays exactly the same.
Most AI bots only read the raw HTML. Anything behind scripts, tabs, or accordions, like pricing and product details, never reaches them.
See your site’s bot readability in Bot View, then choose what gets rendered (the whole site, a group of pages, or a single page) without touching your CMS.
Live today for customers on CloudFront, Cloudflare, or Vercel, with more CDNs on the way.