robots.txt: replace Unicode box-drawing chars with plain ASCII

What was done

Replaced every non-ASCII character in public/robots.production.txt:

• ─ (U+2500 box drawings light horizontal) → -
• — (U+2014 em dash) → - or removed where redundant

Section headers like # --- AI crawlers: explicit allow --- are now pure ASCII. Verified with file public/robots.production.txt returning ASCII text (was previously UTF-8 Unicode text).

Why

User uploaded the production robots file and reported “weird characters” in it. Likely cause: the original file used U+2500 box-drawing chars and U+2014 em dashes. When served, they may render as â?? in editors or browsers that mis-detect encoding (no Content-Type: text/plain; charset=utf-8 from Apache by default for .txt files served from /home/adminl/public_html/).

robots.txt has no need for decorative Unicode - search engine parsers ignore comment lines anyway. Pure ASCII eliminates any encoding-detection ambiguity across servers, proxies, and editors.

Files affected

• public/robots.production.txt - rewritten with pure ASCII
• src/content/devlog/2026-04-26-robots-ascii-cleanup.md - this entry

Operator follow-up

After re-uploading the cleaned file:

1. SFTP dist/robots.production.txt to /home/adminl/public_html/
2. On server: mv robots.production.txt robots.txt (overwriting the bad one)
3. Cloudflare cache purge for https://baileyengineers.com/robots.txt
4. Verify in incognito browser - file should display cleanly with # --- AI crawlers --- style separators