Web Crawler Ethics & Robots.txt Compliance

Policy Section 01

Civic Indexation Principles for Search Engines

Demeure Historique embraces the open dissemination of architectural knowledge. We authorize and encourage all accredited, non-commercial, and general-purpose search engine spiders (including Googlebot, Bingbot, Qwant, Mojeek, and academic library indexers) to crawl, index, and cache our public content.

Our objective is to ensure that when citizens, architecture students, historic monument owners, and municipal heritage delegates search for guidance on slaked lime mortar ratios, timber framing dendrochronology, or historic garden management, they discover authentic, peer-reviewed conservation information rather than commercial promotional advertisements.

Policy Section 02

Prohibition of Commercial Exploitation & Speculative Scraping

While general search discovery is fully supported, Demeure Historique strictly forbids the automated extraction of data from this website for commercial purposes, including:

• Real Estate Speculation: Scraping monument locations, ownership structures, or conservation distress indicators to generate commercial real estate sales leads or speculative acquisition databases.
• Vendor Marketing Solicitations: Automated harvesting of email addresses or telephone numbers to compile unsolicited commercial marketing lists targeting historical property custodians.
• Proprietary Commercial Artificial Intelligence: Wholesale scraping of our copyrighted technical monographs, historical architectural drawings, and diagnostic handbooks for inclusion in closed, proprietary commercial LLMs without formal cultural heritage accreditation.

Policy Section 03

Technical Harvest Rate-Limiting & Server Courtesy

To maintain zero disruption for human visitors and prevent unnecessary computational energy consumption, automated crawlers and research spiders must observe courteous crawling behavior:

Policy Section 04

Academic & University Research Exemption Framework

Accredited academic institutions, university laboratories, and non-profit heritage research bodies (such as CNRS, INHA, and European university faculties of architecture) wishing to conduct high-volume computational linguistic or architectural analyses of our digital publications may request an official research pass.

Upon submission of an academic project summary, our Secretariat will provide a dedicated direct API endpoint or tarball archive containing sanitized, pre-indexed documents, eliminating the need for web scraping while preserving our production server resources. Requests should be addressed to ihyippuiuhoppy@gmail.com.

Policy Section 05

Enforcement & Automated Rate-Limiting

Our cloud server infrastructure employs automated edge security firewalls that detect abnormal request velocity, recursive loop querying, and malicious scanning patterns. IP addresses violating these polite crawling directives or ignoring HTTP 429 (Too Many Requests) headers will experience automated temporary or permanent null-routing at the reverse proxy layer.

Policy Section 06

Open Cultural Data Standards & Semantic Ontologies

To aid legitimate non-commercial knowledge graphs, cultural researchers, and public semantic engines, every page published across our digital domain embeds structured microdata formatted according to schema.org guidelines:

Policy Section 07

Crawler Verification & Reverse DNS Validation

Demeure Historique utilizes automated reverse DNS verification protocols to distinguish genuine search engine crawlers from deceptive user-agent spoofers. Spiders presenting user-agent strings identifying themselves as major search engines must resolve to official domain hostnames (e.g., *.googlebot.com, *.search.msn.com). Bots spoofing legitimate search engine identities to bypass security filters will be permanently blacklisted.

Policy Section 08

Heritage Data Sovereignty in the Age of Generative AI

Architectural restoration is a high-consequence discipline. When an incorrect chemical formulation or incompatible modern sealant is applied to an ancient building based on flawed or hallucinated automated summaries, centuries-old masonry can disintegrate within a single winter freeze.

For this reason, Demeure Historique insists upon rigorous data sovereignty. We require that artificial intelligence developers respect our crawler exclusions unless they hold accredited research agreements guaranteeing that historical lime, timber, and structural specifications will not be distorted, misrepresented, or detached from their indispensable scientific caveats.

Policy Section 09

XML Sitemap Synchronization & Dynamic Endpoints

Our web publication pipeline automatically updates our canonical XML sitemap (https://demeure-historique.org/sitemap.xml) whenever new restoration monographs, annual audited balance sheets, or emergency triage bulletins are issued by our conservation committee.

Search engines are invited to poll our sitemap index at weekly intervals. In accordance with Sitemaps.org 0.9 protocols, our statutory pages are assigned high indexing priority (0.9 to 1.0) with monthly change frequencies, while technical archives and research repositories receive regular incremental updates as historical fieldwork progresses throughout the four seasons. By adhering to these standardized discovery protocols, search engines ensure that citizens always discover current, verified conservation knowledge, technical reports, and non-profit statutory documentation across France.