SEO: sitemap lastmod + hreflang alternates; robots.txt allows all crawlers

robots.txt carried a UTF-8 em dash in its header comment, which renders as
mojibake ("鈥�") in clients that read text/plain as a legacy codepage. The
file is now pure ASCII (verified with `LC_ALL=C grep '[^ -~]'`), and the
encoding trap is documented in the endpoint so it does not come back.

Policy change to allow-all: the previous version blocked CCBot, Bytespider,
Amazonbot and Applebot-Extended. Training crawlers are now welcome too - the
blog benefits from being read, and robots.txt is advisory anyway (Cloudflare
documents it as unenforced; AI Crawl Control is the enforcement layer).
Drops the Content-Signal directives along with the blocklist.

Sitemap gains both fields Google actually uses:
- lastmod on all 58 post URLs, from updatedDate ?? pubDate, read straight
  from the markdown frontmatter at config-eval time (the sitemap runs in
  astro:build:done, after the content collection is gone).
- xhtml:link hreflang alternates on all 77 URLs, pairing EN/ZH twins.
  /posts/ is special-cased: it has no /zh/posts/ twin, the Chinese post
  listing IS the /zh/ homepage, so the pair is declared rather than derived.
  Verified programmatically that every emitted alternate resolves to a page
  that is actually built and present in the sitemap.

src/pages/robots.txt.ts

@@ -2,18 +2,20 @@ import type { APIRoute } from 'astro';
 import { SITE } from '../config';
 
 /**
- * robots.txt — served from the origin so Cloudflare merges its managed
+ * robots.txt - served from the origin so Cloudflare merges its managed
  * block instead of substituting the placeholder Content Signals Policy.
  *
- * Policy here: allow classic search (this blog is a name-search / GEO play)
- * and allow the citation crawlers that power AI Overviews / ChatGPT Search /
- * Perplexity, because being cited is the point of the two-lane strategy in
- * docs/seo-reference.md. Block only the training-only harvesters that give
- * no referral traffic back.
+ * Policy: allow everything. This blog is a name-search / GEO play, so
+ * classic search, citation engines AND AI training crawlers are all
+ * welcome - being cited and being read are both wins.
+ * See docs/seo-reference.md (two-lane strategy).
+ *
+ * ASCII-ONLY on purpose: a UTF-8 em dash in a text/plain file renders as
+ * mojibake (e.g. "鈥�") in clients that read it as a legacy codepage.
  */
 export const GET: APIRoute = () => {
-  const body = `# robots.txt — blog.hoelee.com
-# Policy: index freely. Allow citation/answer engines, block training-only harvesters.
+  const body = `# robots.txt - blog.hoelee.com
+# Policy: allow all crawlers. Search, citation and AI training welcome.
 
 User-agent: *
 Allow: /
@@ -21,27 +23,6 @@ Allow: /
 # Pagefind's search shards are build artifacts, not content.
 Disallow: /pagefind/
 
-# AI trainers that send no traffic back (blocked).
-User-agent: CCBot
-Disallow: /
-
-User-agent: Bytespider
-Disallow: /
-
-User-agent: Amazonbot
-Disallow: /
-
-User-agent: Applebot-Extended
-Disallow: /
-
-# Deliberately ALLOWED (cite us, don't train on us — enforced by Cloudflare
-# Content-Signal below): Googlebot / Google-Extended (AI Overviews, Gemini),
-# OAI-SearchBot / GPTBot (ChatGPT Search), PerplexityBot, ClaudeBot, Bingbot.
-
-# Content signals: machine-readable preference statements (contentsignals.org).
-User-Agent: *
-Content-Signal: search=yes, ai-input=yes, ai-train=no
-
 Sitemap: ${SITE.url}/sitemap-index.xml
 `;