I have a single page application where I use a headless browser to serve pages to web crawlers, giving them a version to the page that's very close to what actual users will see.
Currently, I'm whitelisting crawler user agents to a few: google, facebook, bing, yahoo, and linkedin.
I'm now looking to expand this into something more robust that targets each user agent, rather than just these. The reason for this is because if I don't accept a particular crawler, all they'll see will be mustache HTML templates. which is no good for me.
Is using a gigantic user agent regex the best approach? This is something that's going to be done for every request, so I need something that's fast, too...