The fix: Get the Browser Caps - v2 by Oliver Hine. Unzip the file
App_Browsers.zip to get the file OtherSpiders.browser - put this in the App_Browsers directory of your DNN site. Note that your DNN site will automatically restart, so perhaps do this at a quiet time.Alternatively (if you want), you should be able to stop these spiders accessing your site using a suitable
robots.txt file.This patch is needed for DNN 4.9.4 and possibly earlier. I think that it may be fixed in DNN 5.1+.
Spiders fixed: This file has fixes for these spiders: Baiduspider, Yandex, ia_archiver, Sphere, Feedfetcher-Google, Yanga, worio, zibber,
Spiders not fixed: However it doesn't fix TweetmemeBot - I failed to add support for this. Read this Tweetmeme.
Update: doesn't seem to work for Twiceler or voilabot.
What's happening: DNN code is asking ASP.NET for details of the browser capabilities. ASP.NET gets this from the UserAgent string. DNN asks for the major and minor version numbers of the browser and causes an exception if these cannot be determined from the UserAgent. ASP.NET uses various
.browser files (in the framework directories and in the web app's App_Browsers directory) to work out what browsers can do. As the UserAgent provided by these spiders doesn't follow a standard regex pattern, ASP.NET cannot get the major and minor versions.Thanks to: Barry Sweeney for his prompt help on Twitter - and Oliver Hine of course.
Below is the DNN exception that I get before applying this fix:
Message: DotNetNuke.Services.Exceptions.PageLoadException: Value cannot be null. Parameter name: String ---> System.ArgumentNullException: Value cannot be null. Parameter name: String at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal) at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info) at System.Web.Configuration.HttpCapabilitiesBase.get_MajorVersion() at DotNetNuke.UI.Utilities.ClientAPI.BrowserSupportsFunctionality(ClientFunctionality eFunctionality) at DotNetNuke.UI.Utilities.ClientAPI.get_ROW_DELIMITER() at DotNetNuke.UI.Utilities.ClientAPI.RegisterClientVariable(Page objPage, String strVar, String strValue, Boolean blnOverwrite) at DotNetNuke.UI.Skins.Controls.Search.Page_PreRender(Object sender, EventArgs e) at System.Web.UI.Control.OnPreRender(EventArgs e) at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) --- End of inner exception stack trace ---
