Rescuing a stalled connected product

A stalled connected product arrives in one of three shapes. The first is the orphan: the supplier has closed, or the one engineer who understood the firmware has left, and nobody can build it any more. The second works in the demo and fails in the field: units drop offline, readings drift, and the dashboard shows numbers nobody trusts. The third launched and cannot be changed: every fix is a site visit, every improvement waits on three companies agreeing whose job it is, and the backlog has become a list of things that will never happen.

"Rescuing a product" is one of the five stages on our brief form. This is what happens when someone picks it.

First, one device on a bench

Before anyone offers an opinion, we want a device, the source code and access to the cloud. Can we build the firmware from the source we have been given, flash it onto the device, and watch it behave the way the units in the field behave? Can we see a reading leave the device and arrive in the database? Can we change one line, rebuild, and prove the change reached the device?

This usually takes less than a week, and it answers most of the questions that matter. The source we were given is not always the source running in the field, and that discovery alone changes the plan. Sometimes the hardware is sound and the firmware is the problem, and sometimes it is the other way round. Nobody can tell from a slide deck, and we would rather spend a week finding out than a quarter assuming.

The audit, in four questions

Does the device know its own state? A unit that reports its signal strength, battery voltage, firmware version and why it last restarted can be diagnosed from a desk. A unit that reports only its readings has to be visited to be understood, and in a fleet spread across a province that is a lot of driving.

Can it be updated without a visit? A fleet with over-the-air updates can be fixed from Cape Town, a few units at a time. A fleet without them is frozen in whatever state it shipped. If the hardware can take an update mechanism, adding one comes first, because it pays for itself on the next fix.

Where does the data go, and who wrote the contract? We have written about the gap between what a device sends and what the cloud expects. In a rescue, that gap is often where an "unreliable" product is actually failing. Readings arrive twice, out of order, or not at all after an outage, and the dashboard averages the damage into something plausible and wrong. We trace the whole path, device to screen, and write the contract down once.

Who holds the keys? Source repositories, cloud accounts, the SIM contract, the signing keys and certificates, the app store accounts, the domain. A product you cannot administer is not yours yet, however much you paid for it. Part of every rescue is collecting these into one place with your name on it.

What we keep

The default is to keep what works, and in a stalled product that is often more than it looks. Start with the hardware: if the board is sound and the sensor measures what it should, it stays, because replacing hardware in the field is the most expensive move in the business. Firmware stays if it builds and its architecture can carry the next year of changes. If not, it is rewritten for the same board, which is a far smaller job than new hardware. The real faults tend to live in the cloud and the data model, and fixing them is where the effort goes. The app is the easiest layer to rebuild and the one clients worry about most, which is the wrong way round.

The rule is to change the layer that is blocking the next change, and leave the rest alone. A rescue that rewrites everything is a rebuild wearing a different name, and it carries a rebuild's risk.

When starting again is cheaper

Sometimes it is. If the units cannot be updated remotely, every fix already means visiting every unit. When those visits have to happen anyway, fitting a board that can be updated remotely can cost less than reflashing the old one and going back for every fix after it. If the source is gone and the hardware is undocumented, reverse-engineering it can cost more than designing it properly. If the product has never worked in the field at all, there may be nothing to rescue except the lesson. We will tell you which case you are in during the first weeks, with the numbers, and the decision is yours.

Starting again rarely means starting from nothing. Africam had the cameras and the audience, but not a platform that brought people back every day. The new platform keeps both and rebuilds what sits around them: live streams around the clock, sighting alerts, a community, and the shop that funds it. Keep what the market values, and replace what stands between it and the market.

Taking it over without breaking it

A rescue happens in stages, and the first stage touches nothing. We watch: read-only access to the cloud and the logs, a device on the bench, a list of what fails and how often. Then one device on new firmware. Then a handful in the field, watched for a day, then the rest. It is the same discipline we use on the Woolworths fleet of more than 800 units, because an update that goes wrong everywhere at once is the one failure a rescued product cannot afford.

Documentation is produced as we go, because a rescue that ends with one new supplier who understands the product has only moved the problem. The test we apply to our own work is whether you could take it to someone else: the architecture written down, the data contract in one place, tests that run, keys in your name.

What to bring to the first call

Whatever you have, in whatever state. The source, or the name of whoever has it. Access to the cloud and the accounts, or a list of the ones you cannot get into. A device or two. A list of what fails, how often and what it costs when it does. The contracts for SIMs and hosting. And the history, as it actually happened. We will not spend the call judging the previous team. Products end up here through deadlines, budgets and people moving on, and most of the decisions made sense at the time.

If your product is live and cannot be changed, or half built and nobody can finish it, send us a brief and choose "Rescuing a product", or write to hello@handee.co.za.