• rosco385@lemm.ee
    link
    fedilink
    English
    arrow-up
    3
    ·
    20 hours ago

    It’d be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don’t stay away. 😂