Skip to main content

For site owners

OpenEngResearch, the research fetcher

If you found this address in your server logs, the request came from the web fetcher in the OpenEng AI engine’s research code.

It identifies itself with this user agent:

Mozilla/5.0 (compatible; OpenEngResearch/0.1; +https://openeng.app/research)

The product token is OpenEngResearch; the number after the slash is the fetcher’s version.

How it behaves

A polite fetcher, by construction.

  • Reads your robots.txt first

    Before it fetches a page, it requests /robots.txt from the same host once and caches the rules for that host. It follows the Allow and Disallow lines of the User-agent: * group: the longest matching rule wins, and a tie goes to Allow. If robots.txt can’t be fetched or returns an error, it treats the site as allowed, per the usual convention.

  • Spaces out its requests

    It waits at least 700 milliseconds between requests to the same host.

  • Gives up quickly

    A connection times out after 10 seconds and a whole request after 20 seconds. It follows at most four redirects.

  • Caps what it takes

    It only sends GET requests, and it discards any response body larger than 5 MB.

  • Stays off private networks

    It refuses a URL that resolves to a private, loopback, link-local, or cloud-metadata address, and it won’t follow a redirect to such a literal address or to localhost or a metadata host.

  • An HTTP client first

    It fetches pages with a plain HTTP client (HTTPS or HTTP, as the URL says), not a browser.

Limit or block it

Use the User-agent: * group.

The fetcher reads only the rules in your robots.txt’s User-agent: * group. A group addressed to another name — including OpenEngResearch itself — is not read, so put the paths you want it to skip under *:

User-agent: *
Disallow: /private/
Allow: /private/press/

With these rules it skips everything under /private/ except /private/press/, where the longer Allow rule wins. Rules in the * group apply to every crawler that honors it, not just this one.

Questions or concerns?

Write to hello@openeng.app and include the user agent and a sample of the requests from your logs.