robots.txt carried a UTF-8 em dash in its header comment, which renders as
mojibake ("鈥�") in clients that read text/plain as a legacy codepage. The
file is now pure ASCII (verified with `LC_ALL=C grep '[^ -~]'`), and the
encoding trap is documented in the endpoint so it does not come back.
Policy change to allow-all: the previous version blocked CCBot, Bytespider,
Amazonbot and Applebot-Extended. Training crawlers are now welcome too - the
blog benefits from being read, and robots.txt is advisory anyway (Cloudflare
documents it as unenforced; AI Crawl Control is the enforcement layer).
Drops the Content-Signal directives along with the blocklist.
Sitemap gains both fields Google actually uses:
- lastmod on all 58 post URLs, from updatedDate ?? pubDate, read straight
from the markdown frontmatter at config-eval time (the sitemap runs in
astro:build:done, after the content collection is gone).
- xhtml:link hreflang alternates on all 77 URLs, pairing EN/ZH twins.
/posts/ is special-cased: it has no /zh/posts/ twin, the Chinese post
listing IS the /zh/ homepage, so the pair is declared rather than derived.
Verified programmatically that every emitted alternate resolves to a page
that is actually built and present in the sitemap.