@mischievoustomato @Junes @phnt @waifu @zero they could, yes. They could also make their user agent something absurd, but they don’t. Why? Because it would look suspicious as fuck if they didn’t. Does that matter to some random git repository or open source wiki? Fuck no. Even the local LLM I’ve been messing with says that:
AI scrapers use normal browser user agents to:
However, this is just one part of a broader strategy (e.g., proxy rotation, request throttling, and behavioral mimicking) to avoid detection.