Bingbot cannot index (or scrape) WPcom Simple sites

1 min read Original article ↗

Predef

Please see this internal P2 for details on our predef: p7DVsv-j5F-p2#comment-48067.

This issue will be updated with additional context as needed.

Quick summary

We have determined that Microsoft is using its generic Bingbot crawler to scrape sites. They have not yet documented a way to block the scraping behavior, so for the moment, we have blocked Bingbot from indexing Simple sites via robots.txt directives.

We've created this issue to track support interactions related to this.

Steps to reproduce

N/A

What you expected to happen

N/A

What actually happened

N/A

Impact

Some (< 50%)

Available workarounds?

No but the platform is still usable

Platform (Simple and/or Atomic)

No response

Logs or notes

No response