« Then there are the user-agents who lie about who they are or where they are coming from because they think it's going to get them special treatment somehow »
Actually, it is. As a contributor to a Read-It-Later app (wallabag) I sometimes "craft" the User-Agent in order to go around JavaScript-rendered pages, because web whatever.0 🤷