In this article I tried to summarize our useful experience in designing this system.
It covers what challenges we and our clients encountered when organizing remote software development; how Redd, our software and hardware complex for remote software development, was designed; and discovers the reasons why this “box” appeared and how it might change the routine of millions of developers specializing in embedded systems, IoT devices, and equipment for automobiles, planes, and communication systems.
Figure 1. Redd Complex
We develop system software for clients. A typical customer is a manufacturer of microcontrollers and microprocessors featuring non-standard, extended, or even their own architecture.
We provide them with development tools: IDE, compilers, real-time operating systems, debuggers, emulators, profilers and other SDK components.
We also perform software development for embedded systems. For example, we develop software for modern dashboards for German car manufacturers. We are also involved in avionics projects, software development for providing mesh-network of “smart counters” and UAVs, non-standard software development for access control systems, and IoT projects (e.g., smart sockets). All these unusual and challenging projects results in a wealth of experience in solving non-trivial problems.
Figure 2. Project Scope
For our clients, we are outside developers. Our offices are located in St. Petersburg, Veliky Novgorod, and Krasnoyarsk, while our clients are in Europe (Germany, Sweden), Asia (South Korea), and Russia.
Primarily, we develop software for a particular device, i.e. a developer comes to us with a target device in mind to work with it.
A brief introduction for those who are unfamiliar with the process of embedded development. A developer needs to start a PC with all required development tools pre-installed, the most common one being an IDE. A number of devices, like debugging boards or dashboards (for developing car software) should be connected to the PC. An example a typical embedded developers work station is seen in Figure 3.
Figure 3. Developer’s Working Place
A developer writes code on the PC, downloads it to a device and runs code on the device. Obviously, it is impossible to check code efficiency or perform debugging and optimization without the target device itself. Moreover, a developer has to monitor physical device reactions like which LEDs flash, what is shown on the display, in what direction the wheel spins, or what happens when a button is pressed.
Challenging Issues for Developers
Here is the challenging part. The first problem a developer is likely to encounter: equipment availability.
Many manufactures produce unique products, with very few in number at early production stages. There could be one, two, or five samples at most in the whole world for a new, cutting-edge microcontroller or microprocessor. These devices can also come with steep price tags. Therefore, a client may not have sufficient allocation to share resources with an external development firm (us, for example). There is simply not enough devices available.
A current example is a new processor produced by JSC PKK Milandr: the 1967ВН044. It is still undergoing the R&D stage, however, design of its development tools has to be done now.
Figure 4. Processor 1967ВН044
The second problem: when a device is a scarce product, it might have a number of hardware problems. If a device was delivered from Moscow, and we discovered malfunctioning hardware, we can return the device within a day. However, what would we do if the devices were sent from Germany? It takes time and money to return the device to the manufacturer, to fix the bug, and to get it back. It results in developers’ downtime and project deadline shifts and delays. There also can be cases when a customer is required to transport the device personally due to a security policy. In the real world, hardware problems are common and are simply not affordable for clients.
In general, boundaries between countries is a great handicap. Even equipment delivery from territorially close Europe might turn into a real quest, not to mention much farther nations such as South Korean and the USA… The handicaps might vary from one case to another, but they do affect the deadlines and customer’s expenses. Therefore, they bring down our competitiveness.
Another problem: equipment might be out of order. Transportation increases damage risks as well as spent time and logistics expenses. Equipment should be dismantled, packed, transferred, installed, configured, and integrated to stand elements.
What if you have to develop a gas analyzer software, thus, your working stand would include several massive gas tank, or if you have to develop a helicopter dispersion system.
Figure 5. Development of Gas Analyzer
Figure 6. Helicopter Dispersion System
Figure 7. Helicopter Dispersion System 2
Currently, developers utilize emulators for debugging such systems; however, it would be preferable to check some issues remotely (e.g., PID-regulator for dispenser engine when customer tried alternately fuel-injected and carbureted engines or configured variable resistors in control system).
However, problems might arise even if developers are in one building.
Equipment might get misplaced in the case of poor organization or workflow, namely, formal transfer procedure; equipment might “wander” among developers involved in the project. On the other hand, if the procedures are too bureaucratic, it might take ages for equipment to get to a developer. As for our company, we have never lost customer’s equipment, but there were some incidents when we could hardly trace where the debugging board was and for how long. It is not a crucial step, but when you have many projects, it is better not to waste even the slightest piece of time.
It becomes even more hectic due to the fact that developers can get carried away. Therefore, if multiple developers need the same board (which is frequent as we usually deal with unique hardware), it results in unavoidable time “overlaps” and downtime.
Even if you were a brilliant manager who successfully organized a perfect workflow and logistics, you would still lose to remote development without any relocations in efficiency.
Here is an example. Testing starts at the final stage. As a rule, it comes in parallel with development (not after it). Testers check functionality, find bugs, and then developers fix the bugs and send builds back for testing and so on. In this case, equipment is required for both developers and testers. If your offices are located, for example, in Krasnoyarsk and Veliky Novgorod, equipment can work 24/7. Novgorod developers can write code in the afternoon and evening (Moscow time), while Krasnoyarsk testers can perform testing early in the morning (time zone difference: 4 hours) during their work hours.
The solution seems evident: remote work with equipment can be profitable. You have to place all hardware in one location and provide the team with remote access.
Developers connect to a device in turn, complete their tasks and disconnect to provide access for another developer. The scheme seems perfect, but standard remote desktop to PC connected to the device is insufficient.
In most cases, developers are limited in interfaces for connecting devices: a PC usually had a limited set of interfaces; therefore, it is impossible to connect to a device without adaptors. For example, an external programmer might be required for allowing remote connection.
Once you find a way to overcome that problem, there is another: a developer has no means for controlling device power supply i.e. hard reboot is impossible. A developer has to call a colleague and ask him or her to push the power button or to un-plug the device from a wall socket
This call interferes with the colleague’s work, i.e. a person wastes time for getting instructions, following them, then trying to gather thoughts on what he or she was doing before to this call and return to their normal workflow. It results in several or even dozens of man-hours of downtime over the course of a project, which are not even related to the current one.
Moreover, a developer has no idea what physical changes have been made to a device they are working on. What is shown on display or how LEDs react on the developed device? It is not always required but when it is, it usually comes at the most inappropriate moment.
Surely, you can try to avoid the limitations as much as possible, i.e., buy a relay with remote control to provide remote reboot, install a web-camera to monitor the process, or mantle and configure the whole complex to support the developers needs. Then you have to instruct others how to work, where to connect, in which order connect. In this case, you might get close to the solution, but it is still not enough.
However, if there are more developers than devices, the lack of centralized and transparent procedure of getting access, might ruin the solution completely. You will have to agree with each team member when and for how long they are going to occupy the device.
Redd: Complex for Remote Development
The idea to develop a solution for all above-mentioned problems was so obvious that when we began exploring the market for potential rivals, we were genuinely surprised that no one had done that yet. There were some products, but they were not fully elaborated providing only a partial solution. Those solutions were task-specific: some provided only JTAG debugging, some only emulation, meanwhile debugging is still required as well as stand, observation, power supply, and access control. No one developed it as a single stand-alone device; therefore, there is no full-fledged solution.
That is why we started development of Redd (Remote Development Device). It is an all-in-one hardware and software complex, which provides access to MCU equipment via the Internet or local network using a number of standard and custom communication protocols.
Figure 8. Redd
We did not pursue the concept of “Nano Innovations”. Redd itself is just a sum of usual and comprehensive technologies united in one device, which provides a solution to all problems mentioned before.
Redd can connect to a device using various interfaces; thus, it reduces the problem with numerous devices required for connection. It can also control the power supply; and, thus, rebooting is not a problem anymore. It also provides real-time observation through video camera.
Figure 9. Real-Time Observation
The Management Server allows booking devices using a calendars web-interface and performs access control.
Figure 10. Management Server
Redd consists of two dependent components: a remote terminal and a management server.
Figure 11. Redd Structure
Remote terminal – a standard, off-the-shelf computer based on an Intel processor, equipped with a custom interface extension board, that was developed by our company. The first version (released in March) will support Ethernet and USB Host (JTAG). The second (to be released in June) will support UART, Ethernet, GPIO, SPI (+SPI flash), SDIO (with switch to SD card emulator), I2C, USB 2.0 OTG, PCIe, LVDS, RS232 CMOS, power circuit switch, and logical switches for activating elements (e.g. buttons).
Figure 12. Rear Panel of the Complex
The device supports a number of the most commonly used interfaces and, additionally, a PLD for implementing any non-standard features. As far as the system was designed as a whole, there is no room for mutual conflicts. Interfaces can be transported from one project to another along with the complex, i.e. no additional expenses are required.
The Redd interface board is represented in Figure 13. It is attached to the slot on the front panel and allows the usage of basic interfaces. However another variant, without the board, is also available (see other figures).
Figure 13. Interface Board
In most cases, developed equipment cannot be tested in the field. For example, only flight following and assistance for a helicopter costs 50 euros. A car cannot be run without brakes. Ships are not always at sea. In other words, an emulator is required.
How does the system communicate with the outside world? By using sensors connected to different buses. Signals for executing devices are also transmitted through buses. At the moment, variants of supported busses are numerous. It includes CAN, UART (namely, CMOS, RS232, and RS485), Ethernet, MODBUS (using UART or Ethernet), and DSL with different layers, etc.
A standard solution is to provide emulators’ sensors and executing devices by connecting them to the busses. The developed equipment will consider the received information as real, while a developer can emulate different work cases and refine work algorithms. Obviously, it is reasonable to buy controllers for different interfaces and connect them for each particular project.
Or, you can skip all the routine with the sensors and executing devices and connect Redd, write an emulator for its work (we emulate the outer environment), and then start debugging the solution while the device believes itself to be in a car or anywhere else while we perform the target logic debugging. Testers can utilize this mechanism for checking extreme or known failure cases, which are either hard or even impossible to reproduce during real-world testing.
The Management Server of the complex is located on the terminal itself. Developers can use a web interface to quickly check which devices are available, and when. They can book the devices they require via the calendar and get access to devices automatically. At first, developers establish a connection with a terminal via SSH protocol, and then with a debugger and a device emulator controller via the terminal. Moreover, Redd supports not only multiuser access mode but also provides access to several devices simultaneously.
Figure 14. Multiuser Connection
Thus, Redd allows managing remote access to a device (or group of devices, several devices can be connected to one terminal) for internal and external developers, creating seamless scheduling and access control without physical interaction.
Figure 15. Redd Possibilities
In addition, some bonus features: Redd can be used for demonstrating your products to customers and clients without giving them the physical device. Alternatively, it can be used for organizing device programming distance learning; such courses might be of some interest to MCU manufacturers or educational institutions.
What is going on now?
The primary purpose of this article is to open a dialogue for feedback. Please, leave a comment, if you do not agree with the list of problems listed in the article, or if you think some of them are omitted. Perhaps you know the solutions, and provide functionality to solve all the problems, that we failed to find, or you cope with these challenges in another way in your own development environment. If you would like to encourage us, we will be really grateful.
Looking forward to your comments.