Server issues resolved, Making Amends and HD3 Launch

Published November 17, 2011 by Richard Uren

Earlier this week Handset Detection experienced intermittent outages, delayed service and high network latency (for the non technical people out there thats basically all bad).

The actual problem was a 3 fold combination of a traffic surge (which we would usually have ample capacity to handle) combined with a caching bug and the accidental removal of a default device profile by one of our handset maintainers.  The system, when looking for this default profile, would rebuild the device cache and save it. However the bug caused caches to be rebuilt 5 times per second (placing an extremly high load on our servers), rather than once every 8 to 10 hours. Combined with the traffic increase it was enough to have a serious effect on our service.

At the time I was sure the problem was traffic based and so ordered lots more servers. Thats what operations guys like me do, I see servers struggling and think, ‘Well, there’s been no software updates, and traffic is up, therefore ‘Get More Servers, That’ll fix it’ – And it did. Then we fixed the real problem and all those shiny new super fast servers went virtually idle .. .. .. So, rather than begging for a refund with our hosting suppliers (good luck) we figured a better use would be to give something back to our community, by way of an apology, or an amends.

So, From Monday November 21 until December 26th Handset Detection will be uncapped ! Thats right unlimited detections from November 21 right up until Christmas (well Boxing Day technically) for everyone. Hooray ! and sorry for the hassle. (Note : You’ll still get quota email notices but you can safely ignore them).

One last thing from me today. Whenever issues like this crop up folks always ask about a stand alone detection service, to run on their own servers. Early in the new year we’ll be releasing our HD3 in memory detection engine and device database for stand alone use. Its wicked fast and now has the caching bug fixed :) To get on the launch list please drop your details into the HD3 Launch Notification form.

Image Credit : Chantal W. http://www.flickr.com/photos/nanjenchan/

New Detection Engine live Wednesday August 3rd

Published July 26, 2011 by Richard Uren

Over the last two months we’ve been secretly developing a new detection engine. Two weeks ago we plumbed it into the service and began testing. On Wednesday August 3rd 2011 I am very excited to announce that we’ll be taking it live.

Its leaner, more accurate and much easier for us to maintain than our current system.

There’s nothing you need to do to use the new engine, all detection services will use it automatically.

Happy Detecting

Richard

Detection quotas rollover issue.

Published July 26, 2011 by Richard Uren

We are experiencing a problem with rollover of detection quotas at the end of account period. All monthly limits have been reset to zero while we trouble shoot the issue to keep the detections a flowin’.

Cheers

Richard

Update : The period rollover issue has been fixed – Keep the reset counters as a few free detections on us. :)

Operations Mailing List, Updates, Outage Notices

Published June 22, 2011 by Richard Uren

We know you rely on our infrastructure and timely information is critical when problems occur, so we’ve set up an Operations mailing list.

The Operations list will get emails when we push major updates, run server maintenance, add new equipment, experience outages or degraded performance. Essentially its anything of an operational nature which might have an impact on the system.

So go ahead, drop your email into the form below to keep in the operations loop. It strictly for operations only, and you can unsubscribe at any time.

Quick Note : Stats problems

Published June 14, 2011 by Richard Uren

Handset Detection is currently experiencing issues resulting in stats loss in some cases and detection double counting in others. We’re on the case and hope to have the problem sorted out shortly.

In the mean while we’ve zero’d everyone’s detection counters, to keep the detections a flowin’.

Sorry for the hassle and we’ll post back here with updates.

Cheers

Richard

UPDATE : The problem with our message queue has been sorted out, unfortunately the stats loss is permanent. Detection services were not effected by this problem. If anything looks strange with your stats please drop us a note and let us know.

UPDATE – JUN 17, 23:12 : There’s good news and bad news. The bad news is that after customers reports we’ve discovered that the fix only addressed some customers and your stats may still be broken. The good news is that we’ve just applied a fix and can re-constitute all stats from server log files over the next few days. All stats should be working fine now and we’ll have the last week of stats loaded up by Tuesday afternoon. As always if anything looks amiss then drop us an email : hello@handsetdetection.com Note: As before, detection services are not effected by this problem.

Quick Note – Server outage

Published November 27, 2010 by Richard

Just a quick note to let you know that we’re experiencing problems with our Server Beach site this morning. Its been taken offline while the problem is being investigated.  There may be minor performance issues for some folks while traffic switches across to the alternate site.

Update : The tech staff at Server Beach just let us know that the problem has been resolved and the site will be back in active duty shortly. Thats a sub 1h resolution time – nice one Server Beach ! :)

ServerBeach Site Upgrade This Weekend

Published October 15, 2010 by Richard Uren

Quick note to let y’all know that the ServerBeach site will be down Sunday evening from 10pm (October 17th, 2010) Houston Time (GMT -5) until 2am Monday morning for upgrades. There will be no service disruption. ServerBeach will be taken out of active duty by DNS and load will be shifted to Codero 12 hours before the maintenance window begins.

Have a great Friday, Cheers, Richard.

Maintenance Sunday 12th September & Sunday 19th September.

Published August 12, 2010 by Richard Uren

Serverbeach have just informed us that there will be up coming power maintenance at their Virginia data center. They’re not anticipating any effect to service during the maintenance windows. Their message is below.

FYI : Handset Detection runs two active sites at any one time. Either site is able to handle our usual load by itself, plus about 50%. This is really just a heads up in case you’re doing something funky with fixed IPs at your site, or if your DNS is slow to update (which is how we take a site out of circulation).

—-

In a continuing effort to improve the ServerBeach service offering, we will be
performing two proactive repairs of the critical electrical system during the
following windows:

Sunday, September 12, 2010 from 7:00 AM EST – 7:00 PM EST
Sunday, September 19, 2010 from 7:00 AM EST – 7:00 AM EST on Monday, September 20, 2010.

The purpose of the two maintenances is to replace two static transfer switches
that feed our Herndon, Virginia Data Center. One static transfer switch will be
replaced during each maintenance window. The projected duration of the
maintenance window scheduled for Sunday, September 12, 2010 is 12 hours and the
maintenance window scheduled for Sunday, September 19, 2010 is 24 hours.

Please be advised that engineers will be doing both maintenances on the
emergency power system that feeds your server(s) while it is still energized.
This means we will not be shutting off power to your solution, but there is a
certain amount of risk involved with working on electrical systems while they
are energized. We will have staff and infrastructure in place to act in the
unfortunate event something does go wrong. We are not expecting any power
interruptions, but we will be prepared to react quickly.

–

Activation Email Delivery Problems with Gmail

Published July 6, 2010 by Richard Uren

Just a quick heads up that we’re experiencing issues with gmail bouncing activation emails.  We’re happy to manually activate your account while the issue gets sorted.

Drop an email to hello@handsetdetection.com and we’ll set things right.

Cheers
Richard

Quota emails delayed 21st – 30th May 2010.

Published May 30, 2010 by Richard Uren

A bug in the quota alert system prevented quota alert email being sent since the 21st until today, the 30th. The issue is now fixed and the alerts a flowing freely. Sorry about the hassle folks.