Streaming the Northern Lights

Despite the ongoing trauma the world is dealing with we have continued to push ahead at a slower pace than perhaps I could have ever anticipated but nonetheless it can still be described as progress.

Shetland Webcams added a brand-new aurora capable camera at Eshaness Lighthouse in Northmavine with the support of the Northern Lighthouse Board and our sponsor Polycrub.

This installation is our most exposed camera and therefore it has been heavily engineered to provide some of the steadiest streaming during the winter storms…that can literally throw rocks up onto the lighthouse building and surrounding area.

The view looking north with the most photographed cliffs in Shetland would be perfect if it would provide the money shot - I think you will agree that it has done this again and again.

Dark skies, crystal clear constellations, moonlit cliffs, meteors, fireballs and the occasional “unidentified thing” have meant that this camera is now the most viewed camera on the network.

We also managed to replace the original ‘Aurora Cam’ – that being Cliff Cam 3 at Sumburgh Head with an upgraded model. This camera has a really difficult job to allow you to see the aurora and also deal with the tremendous light pollution from Sumburgh Airport.

When Cliff Cam 3 looks north it looks right into the heart of the sun…well perhaps a slight exaggeration on my part. Regular viewers will know what I mean but the camera’s dynamic range is seriously impressive and that is what we were testing. This camera continues to deliver fantastic auroral streaming with the airport and local community in the foreground along with cars, planes and choppers going about their normal business.

Both those new cameras are included in the AuroraWatchUK app which I highly recommend you download as we also installed and maintain the Shetland Magnetometer on behalf of Lancaster University from where these early warning auroral alerts are triggered.

The interest in the Northern Lights is widespread – so many people love to watch them and there are so many places where you can do this online these days. Shetland Webcams is just one place where you can, cloud permitting, see this fascinating natural event from a quiet band of green in the darkness of the northern skies to a full sky storm where, to be honest, the aurora is then better seen and photographed from places much further south than here at 60 degrees north.

We do act as an early warning for real photographers though as we are also a visual trigger for camera bags to be packed, flasks of something warm filled and then heading out into the darkness for that classic shot, socially shared.

Just a few years ago an auroral event was much more word of mouth and more often simply stumbled upon via casual viewing of the webcams. I also have to admit that there was a really good chance you would get stuttering video and even more frustration as we quickly ran out of bandwidth. We could not scale-up quickly, and every aurora generated more and more traffic.

Excuse me whilst I use a little geek-speak but this a ‘behind the scenes’ post…

Our webcam network was originally based on a 1gbps UK network running out of one UK datacentre. This was fine when I started out with this madness, but the demand and traffic has tripled in as many years and an unexpected boost since March 2020 as we all spent too much time in front of a screen.

Shetland Webcams now runs out of 10 global datacentres with more coming online this month.

Adding the aurora cams meant that we were unable to deal with the peaks as they were large and rarely anticipated so I really needed some new thinking - I had to make the leap to 10gbps which was going to be a major leap of faith and investment. To think that a website serving a small island community of 22,000 people on the 60th parallel would need access to 10gbps for – well anything – was mind blowing and still is.

Looking around the market there are so many service providers and I think I have tried the majority over the last few years. They are generally fantastic at sales and marketing but when it comes to getting access to those with the technical competence it tends to falter. Well-meaning cut & paste script responses make me glaze over these days and my tolerance to incompetence is almost nil.

I had stumbled upon Mythic Beasts – what does the name mean to you?

This company had taken over another dreadful company I had used early in my network development and were making contact in order to transfer my dreadful servers across to their network. I forgot to cancel and ended up with stuff I thought I wouldn’t need.

I did my research and I actually got a really good feel for them – their history and how they had come together and delivered their own vision. They hosted Mac Mini’s and Raspberry Pi’s for goodness sake.

I needed to hang out with professional ‘geeks with vision’ and who cared about the tech and clearly running their business well…and might have the time to answer a few questions.

I can really be a pain in the backside, and I do feel for any tech support team when an email from me drops but the response was absolutely fantastic from the get-go. A relationship developed and I decided that my platform needed a different approach in building a new Content Delivery Network (CDN) that could deal with the normal webcam traffic, the aurora cam peaks and our live event streaming requirements.

Our entire ‘AuroraCam’ network is currently delivered by Mythic Beasts from their London Docklands datacentre with a bespoke agreement that simply is not available with any of the other providers we have worked with.

As a result of working with such helpful people who just wanted to provide a properly engineered and scalable solution, I am proud to say that our new ‘AuroraCam’ network just delivers and for the first time I no longer break out in a sweat watching the demand increase from that AuroraWatchUK alert or a celebrity weather personality sending out a Tweet.

We moved 3.6TB in a recent auroral event and the graph below shows how quickly the demand can hit our network.

That graph is from the Mythic Beasts Docklands datacentre and this was for a pretty normal auroral event – nothing special but still I do not believe anyone struggled to get the HD stream.

Now these cameras are bookmarked and linked-to on a number of enthusiast websites and the word gets out quicker as people use online prediction and almost real-time auroral data tools such as SpaceWeather.com, but the biggest impact for Shetland Webcams has been our relationship with AuroraWatchUK when an alert is automatically sent out to thousands of like-minded people.

I rarely go into print about tech companies however I know how challenging it is to run Shetland Webcams and when you come across companies who genuinely care and want to use their experience and resources to help another business, I felt it worth giving some of my time to thank in public the fact they are real people whose passion and commitment have been absolutely instrumental in creating the positive experience for those who enjoy streaming the Northern Lights from Shetland.

As I end the first blog post of 2021 with light at the end of the tunnel (and hopefully not another train) I notice that the aurora has once again returned albeit late in the evening when our UK aurora watchers have already headed off to bed.

Something so calm to watch this amazing display in our northern skies and a privilege to be living and working in Shetland especially during the last unforgettable year.

Live clips from recent aurora via our YouTube channel

https://youtu.be/S8NylQg1cF4

https://youtu.be/Udhk2j-B1FQ

https://youtu.be/z_eiasPZPHY