Cloudbass’ Michael Beaumont Discusses IP Truck Builds With Axon
Driven by growing market demand for 4K production, leading UK Outside Broadcast specialist Cloudbass has become the first company in the world to create a truly end-to-end, multivendor UHD IP HDR Outside Broadcast vehicle based on the ST2110 standard and with Axon’s Neuron signal processing technology at its core.
This is the second time Cloudbass has used Axon technology for a groundbreaking IP project – earlier this year the company installed a combination of Axon Neuron signal processing and Axon Cerebrum Control and Monitoring software to transform an existing 20-camera OB truck from HD-SDI to UHD IP.
Here, Cloudbass Technical Director Michael Beaumont explains the thought process behind the company’s most recent IP project and highlights some of the lessons learned along the way.
Why did Cloudbass decide to make this significant investment in an end to end, multivendor fully 4K IP Outside Broadcast vehicle?
As an Outside Broadcast specialist, Cloudbass was starting to see increased demand for UHD productions in the UK, particularly for live sports coverage that forms the bulk of our business. Our existing HD fleet was also in high demand and we knew we needed extra capacity, so our thought process over the last couple of years has involved asking questions about 4K and deciding whether it is a more viable direction to take. The questions we kept asking (without successfully answering them) were:
Should we build another truck with large enterprise SDI routers?
Is IP truly ready for OBs yet?
Would the industry still be using HD-SDI as its primary signal transport method by the end of the lifespan of a new truck?
As far as the last question is concerned my feeling was no, the industry wouldn’t be using HD-SDI at that point so the jump to IP had to happen. But we had to wait until now, in 2019, for the technologies and interoperability to reach a point where we could pick the best products from each manufacturer and get them all working work happily together, just as we would with baseband products. Yes, we knew there would be a large learning curve, but we felt that starting the IP journey now, at this stage, was the right thing to do because it would stand Cloudbass well for the future.
Did the success of the hybrid truck you built earlier this year influence your decision to build a fully IP truck?
Most definitely! When asked to enhance the Speedway coverage from HD to UHD by UK broadcaster BT Sport, we had to upgrade or build a new UHD truck very quickly. At the time we looked into both an SDI rebuild and the option of upgrading an existing HD truck to IP. We already had a proof of concept design using Axon’s Neuron signal processing technology, so this was the ideal opportunity to test it out.
We decided to do this by adding an IP 3G layer to our OB8 truck, which had been delivering the Speedway coverage in HD. We were looking for a future proof scalable solution we could use to effectively ‘add’ an IP layer into any of our existing or future fleet and when Axon showed us the initial ideas for the Neuron we knew it would be an exciting product that could help us achieve what we wanted.
The build schedule for OB8 was very ambitious and at the time (the start of 2019) Axon was only just completing Neuron’s development. But we had confidence that Axon would deliver on time and, in fact, we took delivery of the one of the very first production batches of Neurons. We then worked closely with the Axon team to integrate Neuron and re-control the truck with the company’s Cerebrum Control & Monitoring platform. The success of that project inspired a lot of confidence, both in Axon technology and in the concept of building a fully IP vehicle.
When you decided to build OB9, did you start from scratch with new everything, including the actual vehicle?
We had already invested in a chassis, which was ready to go, but when we began developing the equipment plan we realised that, compared to a traditional design, the quantity of equipment required was much less because the power of Neuron reduced the need for external processing. Also, the core Arista switch was only 2U but had enough capacity for an approximate baseband equivalent of a 1960 x 1960 3G router, or a 4300 x 4300 1.5G router, which would have taken up substantially more precious rack space in the truck.
These revelations did influence the physical design of the truck, not least because what we learnt early on in the OB8 upgrade was that IT equipment is generally deeper and noisier than traditional broadcast equipment, this meant changing where the equipment was positioned. We created an ‘equipment room’ so that we could keep the nosiest kit away from production areas. As a side benefit, we were able to minimise rack depths elsewhere and therefore maximise the circulation space available for people, which in turn provides a more pleasant environment to work in.
Why did you decide to make Axon Neuron/Cerebrum the core of OB9?
Cloudbass has looked at many options over the past few years, but Neuron makes the most sense to us in terms of facilities and form factor. It offers many of the features needed for an OB truck in one box, so we need less hardware and external glue which reduces the weight of the truck and makes is cheaper and easier to move, without impacting on workflow.
Also, we like Neuron’s upcoming potential, which will allow us to add features to the current hardware platform as they are developed. Knowing from the outset that the product has so many planned future capabilities has helped us design a truck that will grow as needed. For example, in the near future HDR conversion is coming to the Neuron platform. We have designed for this capability now so that we can take full advantage of it as soon as it becomes available.
As far as cerebrum is concerned, Cloudbass has been a fan of this software for many years and we were looking forward to the opportunity of installing it as part of a project. I’m very pleased we did so for both OB8 and OB9. In our unique best-of-breed IP environment, interoperability of signals works well, but controlling all the endpoint devices is still more complex than in traditional systems. Cerebrum helps to hide some of the complexities to the operators and in some cases gives us critical work arounds to issues that are still to be resolved in hardware. A great feature of using Cerebrum with the Neuron platform is having the ability to create visual pages that can control and monitor many different signal statuses usually read from multiple devices. This helps our engineers diagnose issues and faults. It also significantly speeds up the configuration of the truck – what previously took hours to complete can now be done in minutes because Cerebrum has fewer processing units to ‘talk’ to.
What specific advantages has Axon Neuron and Cerebrum technology delivered?
Using Neuron as the core of our system removes the need for lots of external processing and gives the truck more flexibility, which is particularly useful as we develop the best uses for this new technology. During the design stage I hadn’t initially realised how useful the up/down/cross conversion available to every path would be, but this proved particularly invaluable during our first UHD job for Sky Sports Boxing. Having up/down/cross conversion available to every path allowed me to run the truck internally UHD/3G and down-convert all of the feeds required for the other broadcasters onsite. Without Neuron, the amount of extra external glue required to provide frame-sync, colour correction and Up/Down/Cross conversion on every router input/output would be unimaginable!
As this is a fully end to end, multivendor truck, how difficult was it tointegration equipment from so many manufacturers?
We worked with systems integrator ES Broadcast to brings together products from many different manufacturers including Arista, Evertz, GVG, Ross, Calrec, Riedel and Sony.
The equipment supplied wasn’t quite as ‘plug and play’ as suggested in various interoperability tests, but as we had made the decision to use ‘best of breed’ equipment from multiple manufacturers (just as we would if we were building a baseband truck), we somewhat expected this. There were a few initial interoperability challenges and the biggest stumbling block was different manufacturers using different block packing methods within the 2110-20 standard. There are two types of block packing – General Packing Mode (GPM) and Block Packing Mode (BPM). Some manufacturers equipment only transmits one type, while other manufacturers equipment can only receive the other.
My advice to others embarking on this journey is to check your vendors’ block packing methods and learn how to use Wireshark. That piece of software will become your new multi-meter or waveform monitor!
To what extent did Axon technology help pull everything together?
As a mature product, Cerebrum was key to best of breed integration because it gives third party control of nearly every system in the vehicle. It also guarantees an unprecedented level of control, all in one GUI.
Axon was great with onsite support during the build phase and we worked very closely with them. Without their help and assistance we’d have had a much harder time completing this project within the timeframe available. The ability to talk directly to Axon’s product development team was key to this project’s success, and we developed a very good working relationship during the process.
One of the key issues Axon addressed was identifying the challenges created by the different Block Packing Modes. For example, the GVG Kahuna 2110-20 outputs were not initially compatible with Axon Neuron 2110-20 listeners and this could have been a major issue that meant the truck would have not made air. But the Axon team worked hard to overcome this and I can now happily report that Neuron will accept streams from any of the manufacturers’ kit in the truck, no matter what the packing method or standard, making it by far the most flexible piece on equipment we have.
What other issues did you have with incorporating the ST2110 standard?
In the past audio was always the technologically simplest part of an Outside Broadcast truck, and as such it often gets forgotten, but it is so important to get audio right. In OB9, we went for a fully 2110-30 audio design with AOIP units for the Calrec Integration and a new Riedel Artist 2110-30/31 comms system with IP panels, which all sit on the same media network as the video. This enables us to send streams from anywhere to anywhere, with the added advantage of simplifying the outboard rigging in the field and one core of fibre can connect Riedel comms panel, router panel and several 2110-20 video streams.
Planning your audio streams is key. The industry is not used to designing trucks with audio streams of eight or more channels that you can’t easily shuffle, so a lot of thought had to go into this to make it still feel familiar for the operator. Previously we have taken it for granted that, if a signal is ‘embedded’, it will pass through a system, but with ST2110 this is not the case unless the audio is correctly handled. We have chosen to keep some paths within the truck as ST2022-6 to allow ‘embedded audio’ to stay with the video – for example, on the return TX monitor paths – but as Neuron can work in both ST2110-20 and ST2022-6 simultaneously, it doesn’t matter and will process it in exactly the same way without user intervention.
Do you think OB9 would have been possible without Axon’s technology?
No, it would be a very different truck. My hope is that once even more new features for Neuron are released, and IP connectivity becomes standardised on other items of equipment, OB9 will continue to evolve to keep up with new production requirements not even dreamt of yet.
Are you happy with OB9 and are you planning to invest in any more fully IP vehicles with Axon onboard?
We are very happy with OB9 and with how the project went. There are, of course, some minor design changes we would make for the future, but that is only to be expected with a ground-breaking project like this. But what we have now is a very powerful next generation IP truck, which will act as a template for future projects.
Overall, the design concept we have for our hybrid OB8 truck allows us to easily ‘add’ an additional IP layer to our existing HD trucks. For a fully IP new build, we would begin with a copy/paste of our OB9 design and go from there.
Within 40 days of its launch in October 2019, OB9 covered eight major projects including Sky Sport Boxing Fight Nights, Sky Sports Boxing Pay Per View – UHD, ITV’s Rugby X Tournament and Sky Sports UEFA Football Euro 2020 Qualifiers. It is now the flagship of the extensive Cloudbass OB fleet and is in constant use across many prestigious live sporting events.