• 0 Posts
  • 36 Comments
Joined 1 month ago
cake
Cake day: August 13th, 2025

help-circle
  • If you are using Hugo use this robots.txt template that automatically updates every build:

    {{- $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.txt" -}}
    {{- $resource := resources.GetRemote $url -}}
    {{- with try $resource -}}
      {{ with .Err }}
        {{ errorf "%s" . }}
      {{ else with .Value }}
    	{{- .Content -}}
      {{ else }}
        {{ errorf "Unable to get remote resource %q" $url }}
      {{ end }}
    {{ end -}}
    
    Sitemap: {{ "sitemap.xml" | absURL }}
    

    Optionally if lead rouge bots to poisoned pages:

    {{- $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.txt" -}}
    {{- $resource := resources.GetRemote $url -}}
    {{- with try $resource -}}
      {{ with .Err }}
        {{ errorf "%s" . }}
      {{ else with .Value }}
        {{- printf "%s\n%s\n\n" "User-Agent: *" "Disallow: /train-me" }}
        {{- .Content -}}
      {{ else }}
        {{ errorf "Unable to get remote resource %q" $url }}
      {{ end }}
    {{ end -}}
    
    Sitemap: {{ "sitemap.xml" | absURL }}
    

    Check out how to poison your pages for rouge bots in this article

    Repo was deleted and the internet archive was excluded.

    I use Quixotic and a Python script to poison the pages and I included those in my site update script.

    Its all cobbled together in amateur fashion from the deleted article but its honest work.















  • Not mad, the comment was not adding anything of value in my opinion because it was not clear what it meant. So the down vote is for “this comment is not useful and perhaps lazy”. I also upvote things which I do not agree with but they are proposed in a civil manner, and when I’m proven wrong I say " thanks for letting me know this". So… This is not personal.

    And I never claimed that anything (post, comment or whatever) needed words to have meaning. Not sure what your point is though…



  • This is about some people no being able to comprehend anything from the posts beside from symbols. This could be in the most obscure of languages and any person with a brain that knows memes can understand the basic idea. I’m just saying the verbal part in the post is not comprehensible to all people, which happens to break the rule 5.

    I do not care what language it is on, Spanish is my native language, I just want everyone to able to read and understand.