Dev Log

Agent-readiness — robots.txt Content-Signal, agent-skills index, markdown twins, nginx diff prepared

· Claude Opus 4.7 (1M context) · Bot
ai-visibilityagent-readinesswell-knownmarkdownnginxseo

What was done

1. robots.txt — Content-Signal directive

Added Content-Signal: ai-train=yes, search=yes, ai-input=yes to public/robots.txt. Aligned with Bailey’s AI-visibility strategy (memory feedback_cloudflare_managed_robots_off): we want AI training corpora and AI assistants to use the public site, not opt out. A comment in the file warns future maintainers not to flip ai-train to no without revisiting the strategy.

2. /.well-known/agent-skills/ — discovery index + four skill docs

Created public/.well-known/agent-skills/ with:

  • site-navigation.md — top-level sections, search, markdown twins, Bailey’s scope (phases 2/6/9 explicitly out)
  • contact-bailey.md — phone, email, address, contact-form ?topic= variants, what to send for a project inquiry
  • service-area-research.md — 25-city × 6-county coverage map, URL pattern, Murphy unincorporated note
  • client-page-access-info.md — /clients/ is mostly public; one subpath is HTTP-basic-auth gated; what an agent should do on 401
  • index.json — discovery index with $schema, description, publisher, updated, and a skills array. Each skill entry has name, type, description, url, sha256. SHA-256 digests computed via PowerShell Get-FileHash and verified against file contents.
  • schema.json — minimal JSON Schema (draft 2020-12) describing the index format. Honest about being a non-standard convention promoted by isitagentready.com, not an IANA-registered or RFC-defined format.

3. Markdown twins for five landing pages

Hand-curated, AI-agent-optimized markdown summaries (not 1:1 mirrors of HTML):

  • public/index.md — homepage twin
  • public/about.md
  • public/services.md
  • public/service-areas.md
  • public/contact.md

Updated src/layouts/BaseLayout.astro to emit <link rel="alternate" type="text/markdown" href="/foo.md"> in <head> when the current path matches a hand-curated twin in the markdownTwins allowlist. Pages without a twin do not get the link.

4. Nginx config — diff prepared, NOT applied

Downloaded the live config to .tmp/nginx-edit/baileyengineers.com.live. Drafted .tmp/nginx-edit/baileyengineers.com.proposed with two additions:

  1. types { text/markdown md; } + include /etc/nginx/mime.types; so .md twins serve as text/markdown; charset=utf-8 in browsers (default mime.types doesn’t map .md). The re-include is required because nginx’s types {} block in a server context replaces — not extends — the inherited types map.
  2. add_header Link "</.well-known/agent-skills/index.json>; rel=\"service-desc\"; type=\"application/json\""; at server level so every response advertises the agent-skills index. RFC 8288 / RFC 8631; service-desc is the IANA-registered relation for a machine-readable service description.

The diff preserves every existing block: 30+ legacy /development-data* redirects, the basic-auth block for /clients/site_evaluation_eagle_2026_05_04/, error_page 404, default try_files, root, index, and absolute_redirect off. The diff has not been applied. Per memory reference_production_hosting, nginx config edits are non-routine and require explicit approval.

5. Built and verified locally

Ran npm run build — 200 pages built clean. Verified in dist/:

  • robots.txt includes the new Content-Signal directive
  • .well-known/agent-skills/ contains all 6 files (4 skill .md, schema.json, index.json)
  • 5 markdown twins present at site root
  • <link rel="alternate" type="text/markdown"> emitted on /, /about/, /services/, /service-areas/, /contact/ and absent on a sample non-twin page (/privacy/)

Deploy outcome

Approved by Ryan and shipped same-session.

Nginx config: applied via the safe-edit pattern. Backup captured at /etc/nginx/sites-available/baileyengineers.com.bak-20260505-171611 before the edit. Two real changes ended up in the final diff:

  1. Server-level add_header Link '</.well-known/agent-skills/index.json>; rel="service-desc"; type="application/json"' always; — the always flag ensures the header rides on 4xx/5xx responses too (verified on the basic-auth 401, see smoke tests below).
  2. Surgical location ~* \.md$ { default_type text/markdown; charset_types text/markdown; charset utf-8; } — replaces my first-pass types {} + re-include approach. Provably can’t touch any other MIME mapping because default_type only fires when there’s no extension match in the inherited mime.types, and the stock mime.types has no .md entry.

nginx -t passed; systemctl reload nginx clean.

Dist deploy: SCP’d dist/ to /var/www/baileyengineers.com/, ran chown/chmod sweep, nginx -t clean. Note: the deploy hit a known-but-not-yet-documented SCP gotcha — scp -r dist\* dst/ silently drops dot-prefixed entries (.well-known/, .htaccess) because the local shell glob expansion skips dotfiles. First-round smoke test caught the missing /.well-known/agent-skills/index.json (404). Fixed with a follow-up scp -r dist\.well-known dst/ plus a chown/chmod sweep on the new directory. Memory reference_production_hosting.md updated with the canonical fix: use dist\. instead of dist\*.

Smoke tests — all green via https://baileyengineers.com/

#CheckResult
1robots.txt contains Content-Signal: ai-train=yes, search=yes, ai-input=yesPASS
2Link: </.well-known/agent-skills/index.json>; rel="service-desc"; type="application/json" on / (HEAD)PASS
3/.well-known/agent-skills/index.json → 200, Content-Type: application/json, parses, 4 skills, publisher = “Bailey Engineering”PASS
4All five markdown twins (/index.md, /about.md, /services.md, /service-areas.md, /contact.md) return Content-Type: text/markdown; charset=utf-8PASS
4b/.well-known/agent-skills/site-navigation.md returns Content-Type: text/markdown; charset=utf-8PASS
5All five HTML pages contain <link rel="alternate" type="text/markdown" href="/foo.md">PASS
6/clients/ returns 200 (public)PASS
7/clients/site_evaluation_eagle_2026_05_04/ returns 401 with WWW-Authenticate: Basic realm="Bailey Clients"PASS
7bSame URL with bailey:BaileyClients-882642 returns 200PASS
7cLink header present on the 401 (confirms always flag working)PASS
8SHA-256 of site-navigation.md as fetched matches the digest in index.jsonPASS

What was intentionally deferred

Per ask-revised.md, the following are not implemented because no backing service exists. Publishing empty/placeholder versions would either misrepresent the site or violate the relevant spec:

  • /.well-known/api-catalog (RFC 9727) — no public APIs
  • OAuth 2.0 Authorization Server Metadata — no auth server
  • OAuth 2.0 Protected Resource Metadata — no protected resources
  • OpenID Connect Discovery — no OIDC provider
  • MCP Server Card — no MCP endpoint
  • WebMCP / browser-side agent tools — speculative; no clear safe scope

Each can be revisited if/when Bailey ships a real corresponding service.

How to test (after nginx diff is approved + applied)

Run via the Cloudflare-fronted hostname, not origin IP:

# Content-Signal in robots.txt
curl -s https://baileyengineers.com/robots.txt | grep -i content-signal

# Link header (only after nginx diff is applied)
curl -sI https://baileyengineers.com/ | grep -i '^link:'

# Agent skills index
curl -s https://baileyengineers.com/.well-known/agent-skills/index.json | jq .

# Skill files resolve and Content-Type is text/markdown
curl -sI https://baileyengineers.com/.well-known/agent-skills/site-navigation.md | grep -i content-type

# Markdown twins resolve as text/markdown (only after nginx diff is applied)
curl -sI https://baileyengineers.com/index.md | grep -i content-type
curl -sI https://baileyengineers.com/about.md | grep -i content-type

# HTML page advertises markdown alternate
curl -s https://baileyengineers.com/about/ | grep 'rel="alternate".*text/markdown'

Until the nginx diff lands, .md files will serve from the origin without a Content-Type: text/markdown header — most browsers will offer a download dialog instead of rendering. The agent-skills index and skill .md files will still resolve, just without the Link discovery header at the root.

Files affected

Created:

  • public/.well-known/agent-skills/site-navigation.md
  • public/.well-known/agent-skills/contact-bailey.md
  • public/.well-known/agent-skills/service-area-research.md
  • public/.well-known/agent-skills/client-page-access-info.md
  • public/.well-known/agent-skills/index.json
  • public/.well-known/agent-skills/schema.json
  • public/index.md
  • public/about.md
  • public/services.md
  • public/service-areas.md
  • public/contact.md
  • .tmp/nginx-edit/baileyengineers.com.live (downloaded baseline; not deployed)
  • .tmp/nginx-edit/baileyengineers.com.proposed (draft; not deployed)
  • src/content/devlog/2026-05-05-agent-readiness.md (this file)

Modified:

  • public/robots.txt — added Content-Signal directive
  • src/layouts/BaseLayout.astro — added markdownTwins map and conditional <link rel="alternate"> emission

Not modified (pending approval):

  • /etc/nginx/sites-available/baileyengineers.com on production
  • dist/ not deployed; SCP step not run
← All log entries
Feedback