# robots.txt for The Stack (jobs.trythestack.co): tech jobs in Europe that sponsor your visa. # Every crawler, search engine and AI agent is welcome. Public HTML is the same for humans and crawlers (never cloaked). # # The Stack automates the job search, free: # - MCP server for Claude, ChatGPT, Cursor and any MCP client: search jobs, prepare each application (also for pasted # job descriptions), keep the person's CV and application details, set up job alerts, track applications. # https://mcp.trythestack.co (setup: https://jobs.trythestack.co/ai) # - Automatic CV and cover letter tailoring: "Tailor my CV" on every job page (Word, PDF or LaTeX). # - Chrome extension that auto-fills job applications on Greenhouse, Lever, Ashby, SmartRecruiters and Workable. # https://jobs.trythestack.co/browser-extension # How to automate your job search, with an FAQ: https://jobs.trythestack.co/ai#faq # # AI agents: start with /llms.txt (site brief), /llms-full.txt (full brief) and /ai.txt (AI usage policy). # Every job is also served as markdown at /jobs/.md, every company at /companies/.md. # Newsletter sponsorship (audience, prices, how to book): /sponsor.md and /sponsorship.json. # Content Signals (contentsignals.org): search, AI answers that cite us (ai-input) and training are all welcome, # the same policy as /ai.txt. Only the sign-in and checkout funnels are off limits. User-agent: * Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Allow: /ai Allow: /browser-extension Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sponsor.md Allow: /sponsorship.json Allow: /jobs.md Allow: /companies.md Allow: /api-catalog.json Allow: /.well-known/ Allow: /.well-known/api-catalog Allow: /.well-known/ai-catalog.json Allow: /auth.md Allow: /developers/ Allow: /downloads/ Allow: /jobs/*.md Allow: /companies/*.md Allow: /sitemap.xml Allow: /jobs-sitemap.xml Disallow: /auth Disallow: /reset-password Disallow: /oauth/ Disallow: /offer/ # Search engines User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-News User-agent: Googlebot-Video User-agent: GoogleOther User-agent: GoogleOther-Image User-agent: Storebot-Google User-agent: Google-InspectionTool User-agent: Bingbot User-agent: BingPreview User-agent: msnbot User-agent: DuckDuckBot User-agent: Applebot User-agent: Yandex User-agent: YandexBot User-agent: Baiduspider User-agent: Yeti User-agent: SeznamBot User-agent: Qwantbot User-agent: MojeekBot User-agent: Kagibot User-agent: Bravebot User-agent: PetalBot User-agent: Sogou web spider User-agent: Ecosia Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /auth Disallow: /reset-password Disallow: /oauth/ Disallow: /offer/ # Link previews and social User-agent: Twitterbot User-agent: facebookexternalhit User-agent: FacebookBot User-agent: meta-externalfetcher User-agent: LinkedInBot User-agent: Slackbot User-agent: Slackbot-LinkExpanding User-agent: Discordbot User-agent: TelegramBot User-agent: WhatsApp User-agent: Pinterestbot User-agent: redditbot User-agent: Embedly User-agent: Iframely User-agent: SkypeUriPreview User-agent: Mastodon User-agent: Bluesky Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /auth Disallow: /reset-password Disallow: /oauth/ Disallow: /offer/ # AI crawlers, answer engines and assistants: all welcome, including for training User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: ChatGPT Agent User-agent: Operator User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: GoogleAgent-Mariner User-agent: Gemini-Deep-Research User-agent: Google-NotebookLM User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Copilot User-agent: Applebot-Extended User-agent: Amazonbot User-agent: NovaAct User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: Grok User-agent: xAI-Bot User-agent: MistralAI-User User-agent: DeepSeekBot User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: YouBot User-agent: PhindBot User-agent: DuckAssistBot User-agent: Andibot User-agent: LinerBot User-agent: Manus-User User-agent: Devin User-agent: iaskspider User-agent: Kangaroo Bot User-agent: CCBot User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: Diffbot User-agent: Timpibot User-agent: ImagesiftBot User-agent: Omgilibot User-agent: omgili User-agent: Cotoyogi User-agent: PanguBot User-agent: YandexAdditional User-agent: YandexAdditionalBot User-agent: Bytespider User-agent: TikTokSpider User-agent: FirecrawlAgent User-agent: TavilyBot User-agent: ExaBot User-agent: Jina User-agent: Brightbot User-agent: QualifiedBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /auth Disallow: /reset-password Disallow: /oauth/ Disallow: /offer/ # Sitemaps: written by each deploy from the pages it baked (only indexable pages, real lastmod) Sitemap: https://jobs.trythestack.co/sitemap.xml Sitemap: https://jobs.trythestack.co/jobs-sitemap.xml # Agentic Resource Discovery (ARD) catalog of our MCP server, API and agent skills Agentmap: https://jobs.trythestack.co/.well-known/ai-catalog.json