I always have trouble calling platforms that can run a system of this magnitude "embedded." For some reason I always had the term "single-purpose" linked with the idea of an embedded system. Once you add tons of RAM, a relatively powerful processor, and can only last a couple of hours on a battery--I feel like then it's just a small general purpose computer.
I think there are simply 2 views for the systems: One is the developers view, the other the end users view.
From the developers view these Linux based systems are pretty much general purpose computers.
However what mostly counts for the definition of "embedded system" is the end users view. The end user (customer) gets a complete system for a specific purpose, he can't install any own applications and partly can't even update the system.
I work for a firm that sizes various segments of the embedded hardware and software market. Here's our definition:
A specialized or dedicated computer used to control devices (such as automobiles, home and office appliances, handheld units of all kinds, etc.) where the operating system and application functions are often combined in the same program. An embedded system implies a fixed set of functions programmed into a non-volatile memory (ROM, flash memory, etc.) in contrast to a general-purpose computing machine. However, sometimes single board computers and rack-mounted computers are called “embedded computers” if used to control a terminal interface, machine, motor, etc. An embedded device or system may contain more than one operating system and/or processors (microprocessor, microcontroller, etc.). Specifically excluded from this definition are all types of enterprise computing machines deployed as general-purpose computers (i.e., desktop PCs, standard laptop PCs, enterprise servers, etc.). Our definition of embedded system/device is intended to give a good indication of the potential operating system and run-time software royalty opportunity within the embedded systems market.
So there you go. As you can see our definition has grown substantially over the years as the embedded and IT/enterprise markets have converged. It's much more helpful to think in terms of vendors though.
Traditional embedded silicon vendors include NXP/Freescale, Renesas, Microchip, Rockchip, Cypress/Broadcom etc. Traditional embedded OS vendors include Wind River (Intel), Green Hills, Express Logic, Enea, Micrium, etc.
Intel/Samsung/TI and Microsoft attribute small portions of their revenue to embedded systems, but they don't show up often in conversations with embedded vendors in their respective markets...they focus on enterprise servers/computers/software so they are usually less relevant to the day-to-day going ons of embedded vendors (until they swoop in with the intent to acquire).
This is changing though...has been fun advising some larger players on embedded strategies. Interesting time to be in the market for sure.
> TI ... attribute[s] small portions of their revenue to embedded systems
Um, what? Pretty much every chip TI makes goes into either an embedded or softwareless system. Analog, mixed-signal, and DSP have been their bread and butter for decades.
I've started to think that 'embedded' is an abstraction where the chunk of functionality looks like hardware rather than software. For example my WRT-54G router is running Linux and so are my laptop and desktop, but it seems embedded and the others do not and the Raspberry Pi looks both ways depending on whether I squint or cross my eyes.
I feel like in modern usage, "embedded" has shifted to refer less to the constraints imposed by size and power and cost (because, increasingly, there aren't any), and more to the extra peripherals present. Basically "working in embedded systems" now means what "robotics" used to mean.
Has your thing got always-on sensors, or actuators, or both? Your job title is probably "embedded software developer."
For a long time, I struggled with a similar view. The lines for this are blurring very fast though. Example - Amazon's kindle is more or less a single purpose system - but runs linux, lasts for days on battery, has good RAM and a relatively powerful processor.
One useful metric is how far something is along the "general purpose computer - appliance" continuum. Something can be quite powerful with lots of resources, and still function as an appliance that people do a relatively restricted set of things with.
I always loved Arduino as it promoted a few core ideas (apps are highly portable, the environment is standardized over boards and devices) - this allows the hobbyist / stack overflow reader to make some awesome hacks in a few hours and really reduces the barrier to entry for maker projects. However, in the last few years, I've wanted a similar type of environment for all the Linux boards that now clutter my desk - the ability to run different apps on multiple boards without having to understand the depths of GPIO alternative function mapping and quirks of various distros.
Having a docker container approach where the app is sandboxed away is the right approach I think for this, but there should be a "standard API" as well put in place that the apps that be written against (and bindings to python, rust etc...) that is also emulated in a desktop or webpage environment. This container doesn't support any HDMI output or full Debian environment - just a basic API to start with. Add PIP/Cargo etc.. for advanced use cases. A movement away from Arduino might be around the corner if there is something as accessible but much more powerful available. The block is no doubt "power management" - suspending/hibernating Linux boards for battery operation etc...
I hacked up a version of resinOS last year (using Android as the base Linux port) and running docker on a root FS but never took it anywhere as the day job (a different kind of startup) takes most cycles still. Can't wait to install this later on a few Pi's :)
> A movement away from Arduino might be around the corner if there is something as accessible but much more powerful available. The block is no doubt "power management" - suspending/hibernating Linux boards for battery operation etc...
What are RasPi and similar boards lacking? Power management? Not enough GPIO pins?
>What are RasPi and similar boards lacking? Power management? Not enough GPIO pins?
For me its the ability to "blink" (sleep a designated time and then re-awaken for a brief burst of activity) and so perform some task for 6 months straight on a pair of alkaline AA batteries.
If you're looking for a sensor platform that will let you quickly deploy a sensor that will run for a few years on a pair of AAs, check out what we're putting together at Helium [1]. We just rolled out the presale of our Helium Atom Development Board [2] which uses the Atom - our drop in connectivity and compute module - as the basis for a sensor development board that is actually fit for production. The OS that runs on the module - cleverly named Helium OS [3] - is the basis for our edge programmability and abstracts the annoying, hard things like battery/power management, wireless, and security. Also OTA upgrades are built in. :)
Don't know if you'll still see this. Cool. But one thing. It would be easier for me to get a sign-off from management on a one time purchase of $10k than a $4/month recurring. Corporate sales is funny like that.
I'm using Docker more and I do like it (after a fashion), but I don't want to move my embedded software to this model. I've got one OSS project written in Python and it can run on python-mini (used on OpenWRT) on tiny configurations (<8MB rootfs).
It has minimal dependencies and doesn't even need a virtual-env. It's prepackaged in rpms, debs and apks. It needs raw access to GPIO and USB. ResinOS, which technically interesting, seems like way too much and another layer I just don't need.
I don't want to sound like a dick, because I'm sure the authors put a lot of work into this, but I really don't think this is a good pattern at all.
If I had to write my client again today, I'd probably write it in RUST/LLVM, giving me even less of a reason to stick it in a docker container on my device with <8MB storage space.
Our model is not a commandment for everyone. There will always be cases where containers don't make sense. However, every new level of abstraction meets this instinctive reaction at first. People didn't want VMs in the data centre messing with their bare metal either, I'll bet.
The point of containers is to make updateability easy. To bring embedded software closer in line with what's going on in the cloud. To enable the kinds of workflows we have had in every other part of the development world for decades. It may or may not be right for your project. So long as you understand the intention and make informed choices about the tradeoffs, there will always be cases that fall in the one or the other side of the fence. I don't perceive this as being a dick, you're stating a reasonable case for a model that does indeed work for a lot of cases.
We will continue working to reduce overhead, hopefully power requirements and cost will go down, and somewhere along the curve, we may even meet your needs at some point. Whatever the case, we're pretty sure we'll never cover everything so if we gave the impression that resinOS will end all embedded OSes, then it was the wrong impression to give. :)
We used to release anually in the data centre too. If the benefits of incremental and iterative deployment don't move you, consider all the iot security issues recently. And if that doesnt do it either, consider NASA's Mars rovers, Tesla, nest, and the multitude other embedded products that use updating to great effect. You may disagree, but gibberish is a bit harsh don't you think?
I don't agree. Today you can push out packages to embedded device easily; even tiny <4MB rom devices like openwrt.
You can push out regular updates, but you have to be very careful. You need to have full testing environments and real lab units of all generations to test on before you push to production and realize you now have a brick on the roof of an 80 story building that takes a week to get a roof permit form.
The cloud stuff is gibberish. Today's cloud is rebranded hosted platforms. I agree, I don't see the argument of making _embedded like the cloud_ being meaningful either. I know on my embedded projects we had a tight ship pipeline and you need that for the reason stated above. Sure you can do that with Docker/ResinOS, and it might help with building a piece of that, but I have a feeling it adds as much complexity as it might mitigate.
The embedded systems I work on are never internet connected and updated manually by trained service technicians when necessary.
Like the discussion at the top of this thread, we've really muddied up the waters when it comes to defining "embedded devices".
There are devices like mine, which are the early definition of the term (single purpose, low cost, tightly defined I/O) and then we have low cost highly compact computers, literally servers, that can be multi-purpose and are internet connected 100% of the time by design.
I discovered that docker can be helpful during development for embedded stuff even without using docker on the device itself,. E.g. we put the target distribution for an embedded device incl. the whole development toolchain (gcc&co) in an image. That way the build environment can be easily stored and distributed through a docker registry
Hi everyone, resin.io founder here, happy to answer any and all questions. We just released resinOS at ELCE a few hours ago (Embedded Linux Coference, Europe), happy to see it made HN!
Hi! Discovered resin.io already some time ago and think Docker containers on embedded devices is a very interesting approach.
Just skimmed a little bit through the resin-os docs and was left with the following questions:
- What's your plans for audio and video? The base system doesn't seem to provide any audio or video manager. Should those (e.g. wayland) also be ran inside a container, which could make access from other containers difficult? Or should they if required be included through the Yocto image build process? Or is A/V currently out of scope for you and you focus on pure networked devices?
- Haven't found any description on how prebuilt docker images can be packaged and flashed during the installation process or later on? I only found the rdt push description which seems to push the application sources to the device and builds the image there. This might be interesting for development but would not be anything that I want at all for production. E.g. I don't want to give customers my app sources, I need company git repository access (credentials) during image building, etc.
- What's your reasoning about building all those build and development tools in coffeescript for node? From a pure tool user perspective I know that it's a turn-off for lots of users that it requires node6 on the PC (it's not LTS, other stuff might require other versions, what is that node-thing anyway, ...). And from a developer perspective I would rather not prefer an exotic programming without static typing and a rather volatile ecosystem for a reliable long-term system. I don't want to say here that it isn't possible to create solid project with it and yours might work great - it's just not the first option that comes to my mind.
If you run a privileged container, you can add any sort of audio and video manager in the container. We have users running X11 and Electron inside their containers, for instance.
Preloaded containers is something we have built out for resin.io, and will be extending to resinOS standalone as well. The current release is just the development version. Production version with preloading should follow.
We use node.js for the development toolkit, not for the OS components. There will always be programming language disagreements, but we find we are productive in node.js and it works for us. Coffeescript is admittedly getting a bit long in the tooth and we should switch to ES6 or TypeScript, but I personaly see that as a minor change, though a good one.
I think I see where you fit in - deeply embedded systems with real-time constraints tend to use a certified hypervisor solution to separate runtimes. Traditional IT solutions can use Linux KVM or a full Docker implementation. ResinOS fits in the middle.
Familiarity with Docker will allow makers to get multiple runtimes up and running on higher-end (but small) SoCs...but for what type of applications?
There are a lot of companies looking at the intersection of containers and embedded. Some call it "Fog Computing" others "Edge Computing", others still "Industrial Internet" and so forth. We have a number of use cases on the resin.io website of people using this technology in production today, but some of the larger ones we're not allowed to talk about :(. ResinOS does include a full Docker engine however. It's just that a lot more is needed to do containers right on embedded Linux devices.
There's a lot of devices that need to run unsupervised, but still have (relatively) beefy hardware and commodity software. Digital signage stations are one example, some customers want a (relatively modern) Webkit/Blink based browser based system so they can deploy React/Angular/etc based applications on it instead of needing to hire people who know Qt Embedded or similar. The overhead doesn't really matter for such applications: The backlight for a 40+ inches large touchscreen will always consume more power than the other components combined, so you can put in an actively cooled 20W dual/quadcore APU just fine, and having a full OS allows running other services on it (SSH/VPN solutions for remote management, e.g.).
(Disclaimer, we've been doing exactly that since 2011. I've been wanting to move to a containerized approach for years, but our customers are happy enough with the current solution that they don't want to fund it. Alas.)
The hypriot guys are doing great work, and we share a common mission in spreading containers to embedded development. ResinOS is built with production deployments as a first goal, and so has a read-only rootFS, supports atomic host OS updates, is ported (and portable) to a lot of device types (via Yocto) and in general has many more "embedded" characteristics than Debian, on which hypriotOS is based. I do however want to reiterate that we know the Hypriot guys decently well and appreciate the work they're doing to make Docker accessible to embedded hackers.
Excellent response and yes, HypriotOS is based on Raspbian, which is a Debian derivative. Glad to hear you're using Yocto as well. I'll certainly have to check this out.
Do you have anyone using Kubernetes on ResinOS by chance? I've got a small fleet of small hardware and had started looking at running k8s on HypriotOS, but this looks a bit more custom built.
Haven't really tried to go that route. There are some fundamental limitations to Kubernetes and Docker Swarm due to their dependence on etcd, which behaves very badly when the devices are not on the same LAN. Depending on your use case you may want to have a look at resin.io as a management/orchestration service of sorts (pardon the plug), though our focus is purely on embedded/IoT scenarios, so it may or may not apply to your use case.
Nah I also do quite a bit of work with microcontrollers such as the esp8266 (32 bit 80Mhz RISC) and the just released esp32. It's relevant and I'll check them out.
We're generally very active in the Yocto community, the resin.io team already maintains the raspberrypi, CHIP, and Samsung Artik BSPs. Not sure what their interaction has been with the vitrualisation layer and don't want to misrepresent them, but I do know Andrei and the rest of the team are generally good citizens and work hard to feed patches upstream as much as possible.
I have used Yocto for building a new of embedded Linux systems in production. I understand that Yocto is used for the resin hostOS images and Docker is used for the images run as containers.
Is it possible or has any thought been put into making it possible to use Yocto for building the containerized images that are deployed to resin (probably in the form of docker images)? Although this is less convenient than using off-the-shelf docker images and customizing with your own docker files, I believe many of the benefits of Yocto in terms of building a custom embedded linux system might also be recognized within the context of resin.
Yes! We've actually created extremely small base images with Yocto for our own agent at resin.io, and it's a really cool approach, though it is very complex to set up and handle, and has approximately none of the elegance of a Dockerfile. Also, resinOS images are available as containers for some very specific use cases, so that's another instance where we've done something like that. Perhaps we should write up more about the approach.
Awesome, I would love to see hear more about the approach you guys used for building base images with Yocto. I'm willing to give up a little elegance for control and reproducibility.
I just added the BeagleBoard X15 to the board support list.
Do you anticipate supporting loading the other embedded processors? The BeagleBoard X15 has 2 - Cortex M4s and 2 DSPs but I think a simple case would be the single core dsp on the Beaglebone.
You mean that you tried the image and it worked on the X15? That's fantastic! We hadn't tried the board before.
What does support of the other processors entail? Is it a kernel modification? Are you sure the dsp on the Beagle doesn't work? Say hi on the gitter channel ( https://gitter.im/resin-os/chat ) and the resinOS team will be there tomorrow morning (european time) to talk more.
resinOS is designed for embedded devices, already supports many of them running several architectures (all the way down to ARMv5). While I have heard that rancherOS can run somewhat on a raspberry pi, that's a long way from being able to properly function in an embedded environment (read-only rootFS, reliable networking and DNS stack, support for atomic host updates, etc). There's nothing we'd like more than to have someone make the OS for us, but at this point we're pretty sure we'll have to do it ourselves. The variance in the embedded world is simply too large for a cloud OS to handle it without major rearchitecting.
Transactional updates are pretty much standard for embedded devices that care about updates. Android Brillo, ChromeOS, Android, Snappy, and many many others employ similar strategies.
ResinOS does indeed have a smaller footprint, a broader set of supported architectures (Snappy only supports ARMv7 and above), and uses Docker instead of LXD+snaps.
Congrats on launching! Testing & deployment of stable configs is arguably even more of an issue in embedded development than in web, so I'm sure this will make a lot of people very giddy! Are y'all 100% happy with the technical design assumptions & requirements of Docker, or did you mainly pick it so you could bridge with the existing community (or a bit of both?)
Very good question. We find that we have to add functionality for the embedded world in quite a few cases, and part of the resinOS roadmap is to package that up for the OS. But the base Docker, other than its recently growing size, is a very strong foundation, and one that keeps improving. We're particularly excited about the new security features and in general our approach is that we can move a lot faster by filling the gap between Docker and embedded devices, rather than starting from scratch and building Yet Another Container Engine.
I think nerves[1] needs a mention here... it'll pack your elixir application with a minimal linux kernel onto a read only sd card for use with a raspberry pi or BeagleBone.
Very cool stuff! Some boards, like the beaglebone, have specialized hw like PRUs or I2C pins. Is it possible to access these with Docker containers? Also, what is the minimum memory footprint? i.e. how many containers can I fit within say 1 GB.
The version of Docker on the OS right now is 1.10 so the ram footprint of the OS is fairly small. The team is all asleep (being in Berlin and whatnot), but I'd estimate about 50mb is a decent approximation. We're working on getting a newer Docker but its RAM footprint is one of our concerns, though we do have ways forward.
The number of containers is kinda a red herring for a couple reasons. For one, you can make a trivial container and launch as many containers as you like, it really depends on what's in them. For another, embedded devices don't usually need hundreds of containers, they run payloads that are either single-container, or look a lot more like a docker-compose file with a few microservices.
You missed this part of the question: "Some boards, like the beaglebone, have specialized hw like PRUs or I2C pins. Is it possible to access these with Docker containers?"
quick question...would the images have to be specifically built for arm? (we were looking to build a cluster of RPis that we could orchestrate containers on.)
ARM isn't x86. You get a nice generic kernel with enough built in drivers plus an Initrd and it can boot on most Intel/AMD hardware both old an new (at least the Pentium 4+..maybe even the P2 or P1. Haven't tried a newer distro on them).
ARM isn't anywhere near as standardized. There are usually vendor kernel patches, not just for drivers, but dependent on the block layout of that particular ARM board. uBoot is often forked and also specialized. There is no one shoe fits all Grub.
