Lessons from Communication Problems that Nearly Jeopardized Development of Hardware-Software Support for a 1000-Device IoT Testbed

While prototyping devices dedicated for a 1000-node low-power wireless networking testbed, we encountered over a dozen nontrivial technical problems. This paper analyzes a few selected ones, which we faced during development of the hardware-software support for two different communication channels for supervising experiments on the testbed. The problems arose despite our employing popular standards and reputed components, minimizing risky features, and following modern quality assurance practices. Moreover, if it had not been for our emphasis on dependability, they would have likely passed undetected and doomed our project. In this light, we believe that the presented lessons we learned the hard way when debugging the problems may be of interest to the community.