Hello Javier!

Welcome to the March 2025 edition of the Embedded Artistry Industry Update. This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

• Humane's purchase
• What people expect for smart device support
• HP's customer service strategy
• A summary of other noteworthy industry news
• Technical articles from around the web
• Embedded job postings
• Updates to the Embedded Artistry Website

Humane was not so Humane

HP announced that it has acquired Humane, the company that produced the infamous AI pin device, for $116m - much less than the $750m-$1bn exit price they were seeking in May 2024. As part of the deal, HP will receive the company's 300 patents, pending patent applications, and other IP. Humane's founders will be forming a new division within HP, called "HP IQ," which will be responsible for integrating AI into HP's printers, PCs, and other devices. Several Humane employees received job offers from HP with 30 to 70% pay increases. Others were laid off by Humane.

Continuation of the AI Pin product is not part of the deal, and Humane is shutting down the cloud software. Cloud-related capabilities (that's pretty much all of them) will no longer function after 1200 PST on 28 February, and all consumer data will be deleted. The company notes that "offline features like battery level, etc." will still be available. We wonder whether the person writing that line in the FAQ thought about how ridiculous that sounded. "Your devices aren't actually bricked, you can still charge them and check the battery level!" Accordingly, Humane suggests that users "recycle" their devices, though they provide no actual guidance for how to do so.

Refunds will only be offered to those who purchased the Pin and are still within the 90-day return window offered by the company (products shipped on or after 15 November 2024). Refunds will not be provided to anyone who purchased earlier than that window. The company will also no longer be replacing the Charge Case in accordance with the recall. Those who chose to wait for a replacement case instead of receiving a refund will be refunded for the portion of the original purchase price that was allocated to the charge case. Refunds will be processed after 28 February, though all refund requests needed to be submitted by 27 February.

Surely, Humane did the bare minimum that they were obligated to do in this situation, and perhaps the device was frustrating enough to use that nobody will truly bewail them becoming paperweights. Nonetheless, bricking a $799 device that also required a monthly subscription with no recourse is yet another example of the troubles with cloud-dependent hardware.

Consumer Reports Survey Results

Consumer Reports published the results of their December 2024 "American Experiences Survey". The survey received responses from 2,130 American consumers, which they claim gives margin of error of +/- 2.59 percentage points, with 95 percent confidence. 

This survey included several questions on connected devices (starting on page 9), and the responses reveal the gap in perception between consumer expectations and connected device manufacturer behavior. 

• Respondents generally expect that device manufacturers will support connected devices for at least 3-5 years, with many expecting devices to be supported for 6-10 years (e.g., ~25% of respondents who purchased a smart appliance, thermostat, TV, smartphone, router, speaker, or vacuum) or longer (~25% for smart appliances, thermostats, ~20% for doorbells, security cameras, TVs, smartphones, routers, speakers).
• 43% of respondents were not aware at the time of purchasing their smart device that it might, at some point, lose software support or stop receiving security updates.
• There is a general expectation that hardware should still continue to be useful even when software is not supported any longer, along with a consistent level (~25%) of "not sure whether it will work" across categories.
• For those whose devices lost software support, the top way that respondents found out was "it stopped working" (40%), followed by "I got a notice from the manufacturer" (39%). 
• When devices lost software support, 36% of respondents replaced it with a new smart device, 35% just kept using it the same as they had been, and 14% disconnected devices from the internet and lived without connected functions.
• 72% think that manufacturers should be required to disclose software support. An additional 13% don't feel it should be required, but that it would be a positive indicator.

Consumer Reports released a subsequent report which provides commentary on the results.

For one, we fully agree with the statement, "it’s clear that a lot of Americans who own smart devices don’t understand the relationship between software support and how long a device might retain its usefulness." There's also an exploration of what it means when "devices stop working" - if you're using the connected features, e.g., pre-heating a smart oven remotely, you're going to feel the pain when that support is gone. If you don't use those features, losing software updates and internet support doesn't matter.

Another significant concern, also discussed in the second report, is one that will be familiar to readers of this newsletter. "No longer supported" is not always a case of "it stopped working." Rather, it leaves the devices open to exploitation and recruitment into botnets. For many connected devices, users are not likely to suspect that the devices have been compromised unless they are monitoring network traffic (not your typical consumer!) or observing some obvious symptom. Device makers do not have a clear legal obligation to tell consumers that software support has ended, and even those that note that support has ended might do so on a lone page on the website. So, many devices linger on in vulnerable states, with consumers unaware of the actual threat they pose.

The bottom line is that consumers don’t understand what they are getting into (or signing up for) with connected devices. It is our responsibility as engineers and administrators to make responsible decisions knowing they affect everyone. And, because we have not done a good job at this, we should not be surprised when regulation forces our hands.

HP's Customer Support Strategy

HP tried out a new customer support strategy, and it did not go over well. Anyone calling into the customer support phone lines from the UK, France, Germany, Ireland, and Italy was forced to endure a minimum 15-minute wait time. Even if HP's telephone support center wasn't busy, you would still hear:

We are experiencing longer waiting times and we apologize for the inconvenience. The next available representative will be with you in about 15 minutes.

To quickly resolve your issue, please visit our website support.hp.com to check out other support options or find helpful articles and assistant to get a guided help by visiting virtualagent.hpcloud.hp.com.

The Register cited internal communications that said the move's "Objective is to influence customers to increase their adoption of digital self-solve, as a faster way to address their support question. This involves inserting a message of high call volumes, to expect a delay in connecting to an agent and offering digital self-solve solutions as an alternative.”

Predictably, this move received internal and external backlash, and HP seems to have stopped the practice the day after The Register published their story. HP shared a statement explaining the move (emphasis ours):

We're always looking for ways to improve our customer service experience. This support offering was intended to provide more digital options with the goal of reducing time to resolve inquiries. We have found that many of our customers were not aware of the digital support options we provide. Based on initial feedback, we know the importance of speaking to live customer service agents in a timely fashion is paramount. As a result, we will continue to prioritize timely access to live phone support to ensure we are delivering an exceptional customer experience.

The words that can unironically come out of a corporate communications department are truly remarkable, such as "we're looking to improve our customer service experience [by wasting our customers' time]."

There are, of course, several assumptions involved in this strategy that are worth questioning:

1. That callers are actually capable of finding, understanding, and executing the written online instructions
2. That these customers had internet access at the time they needed support
3. That the online instructions are sufficiently detailed
4. That those calling in didn't already try solving the problem themselves with available resources
5. That the problem could actually be resolved online rather than requiring a representative (e.g., hardware RMA requests)

We can also acknowledge the doublespeak and see the move for what it most likely is: an attempt to further reduce customer support costs. The current trend in contemporary capitalism is relentlessly shaving off costs, and customer support is not immune. The increasingly prevailing attitude seems to be, "Pay us and go away."

You're smart folks reading this newsletter. How often have you tried to solve a problem yourself, only to find:

• A FAQ with softball questions and vague, obvious, or otherwise useless answers
• Web documentation that provides a solution... for an outdated version that has different options than what you see in the latest version
• A "support community" with a notable lack of company input, resulting in unanswered questions (perhaps even the exact same as yours!) or customers reluctantly helping other customers (if they can)
• An AI chatbot that's just regurgitating the same things you already read and making them even less helpful

Maybe you get lucky and some unaffiliated person's blog or forum post gets you moving again. But when that doesn't happen, what we desperately want is to talk to a real person who can help us with our real problem. And this is becoming increasingly difficult.

There is no doubt about it, providing support to our customers is a costly, time-consuming process. But that's what we signed up for when we got into the business of making products. And these foolish moves by companies leave us with a clear, disruptive avenue: taking pride in providing better support than our competitors.

News in Brief

Software

• Ferrocene 25.02.0 is now available. The update includes changes from Rust 1.82 and 1.83, as well as five new experimental targets.

Consumer

• All 50 states in the U.S. have introduced Right to Repair legislation, though not all have passed (yet).
• Qualcomm and Google have partnered to provide up to eight years of update support. The extended support window only applies to Android devices with the latest Qualcomm chipset, which for now is anything running the Snapdragon 8 Elite platform. 

Security and Privacy

• Apple pulls iCloud encryption support instead of caving into the UK government's demand for an encryption backdoor. UK users can still encrypt some data, such as iCloud Keychain, Health, iMessage, and FaceTime. Other iCloud services will not be encrypted.
• China's Salt Typhoon hacking group continues to breach ISPs and telecoms through unpatched Cisco routers. It has also been reported that they are using a custom utility called JumbledPath to monitor network traffic and potentially capture sensitive data.
• AMD has patched a flaw that allowed the loading of malicious microcode patches. The problem was an improper signature verification weakness in the CPU ROM microcode patch loader.
• The U.S. Cybersecurity and Infrastructure Security Agency (CISA) is warning that Contec CMS8000 patient monitoring devices include a backdoor that sends patient data to a remote IP address. The backdoor can also download and execute files on the device. The IP address is not associated with Contec, but an unnamed Chinese university. The IP address also appears to be hard-coded in other medical equipment, including software for a pregnancy patient monitor from another Chinese manufacturer. Currently, there is no available patch for devices that removes the backdoor, and CISA recommends disconnecting these devices from the network.
• U.S. CISA has ordered government agencies to patch Linux and Android devices by 26 February in response to CVE-2024-53103.
• Eight Sleep hardcodes an AWS key in their software for their network-connected bed.
• Netgear warns users to patch several router and access point models. The particular errors were not disclosed, but they grant the ability for remote code execution and authentication bypass.
• Juniper patches an authentication bypass security flaw affecting several devices.
• OpenSSH released security updates to address machine-in-the-middle and denial of service flaws.
• A new Mirai botnet variant, Aquabot, has been observed exploiting a command injection vulnerability in Mitel SIP phones.

Chips

• Apple announced plans to invest $500bn in the U.S. over the next four years. Teams and facilities will expand in Michigan, Texas, Arizona, Nevada, Iowa, Oregon, North Carolina, and Washington. The company will build a new factory in Houston, Texas, and an Apple Manufacturing Academy in Detroit. They will increase investments into AI, silicon engineering, and their "U.S. Advanced Manufacturing Fun."
• Intel, Synopsys, TSMC all unveil record memory densities.
• The White House is seeking to renegotiate U.S. CHIPS and Science Act awards. There are expected to be delays to some of the upcoming semiconductor disbursements.
• Arm allegedly plans to produce its own data center CPUs this year, and Meta will be one of the first customers. Reuters reports that Arm has been recruiting talent from its licensees to aid its efforts.
• It takes twice as long and costs twice as much to build a chip fab in the U.S. vs Taiwan. Significant factors are longer permitting processes and the lack of 24/7 construction schedules.
• Intel has secured a $536m payout from the European Commission. The payout covers interest from the original $1.1bn penalty, which the General Court of the European Union annulled in 2022.
• Reuters reports that the Trump administration is not likely to support Intel's factories being operated by TSMC. There were also concerns in Taiwan about a joint venture between Intel and TSMC.
• DRAM and NAND prices are expected to rise.
• OpenAI is finalizing the design for its first in-house chip, which will be produced by TSMC.
• Nvidia is expected to consume 77% of wafers used for AI processors in 2025.
• AMD is in talks to sell its AI server assembly plants. Compal, Inventec, Pegatron, and Wistron are interested buyers.
• Intel has revised its construction schedule, and its $100bn Ohio fab site will now come online in 2030. 

Trade War

• At a press briefing, U.S. President Trump announced plans for tariffs on semiconductor chips and pharmaceuticals to start at 25%, "and it'll go very substantially higher over the course of a year." No timeline for these tariffs has been given, though he has said that he would give impacted semiconductor and pharmaceutical companies time to build factories in the U.S. before imposing tariffs. He also announced 25% tariffs on EU goods (and made several other thoughtless comments).
• China responds to tariffs by re-opening antitrust investigations into Google and Nvidia. A new probe against Intel is also rumored.
• China accuses Taiwan of giving its chipmaking industry to the U.S. as a "souvenir".
• Japan announces expanded export controls for chips and quantum computer tech. These new restrictions come into effect at the end of May. The Japanese government also added 42 new entities to its export oversight list, which went into effect on 5 Feb.
• Huawei's chairman reported the company's annual revenue exceeded $118.27bn in 2024 despite U.S. sanctions. This revenue would represent 22% growth compared to 2023.
• Trump tariffs will result in at least a 10% price increase for Acer laptops.
• TSMC is restricting Chinese chip design firms from ordering chips made with 16nm processes and below in order to align with the recent U.S. export rule changes.
• China's produces 28% of the world's supply of chips at mature process nodes. Western companies will likely find it increasingly difficult to compete with Chinese companies pricing their products far cheaper than their Western competitors can afford to match.

Automotive and Aviation

• A Jeep owner complained about seeing in-car advertisements, giving us a glimpse into our future.
• BMW has announced increased investments in internal combustion, citing the EV "rollercoaster" in the U.S.. As with many EV-related claims, this ignores the fact that combustion-powered vehicle sales have been in decline since 2017 and that there is more EV charging infrastructure in existence than when BMW first started producing EVs.
• Volvo China has issued a statement apologizing for customers finding off-brand speakers in their new cars.
• Hyundai's NACS adapter will be free for all existing owners.
• Ten new EV battery manufacturing plants are expected to come online this year in the United States. Capacity is expected to grow to 421.5 gigawatt-hours produced per year, an increase of 90% over 2024.
• 376,000+ Tesla Model Y and 3s are under a recall for faulty steering. The NHTSA investigation found that " a condition could result in overstress of circuits on a printed circuit board that could lead to a loss of EPAS [electronic power steering] when vehicle speed reaches 0 MPH. Tesla's design prevents a loss of EPAS while the vehicle was traveling above 0 MPH."
• Honda-Nissan merger appears on the verge of collapse after Honda proposes turning Nissan into a fully owned subsidy. The deal was originally pitched as a "merger of equals." There are also rumors that Nissan is looking to see if Tesla will buy the company.
• U.S. Department of Transportation has ordered states to halt implementation plans related to the National Electric Vehicle Infrastructure program.

Space

• Firefly's Blue Ghost lunar lander is lowering its orbit in preparation for a landing attempt.
• Isar Aerospace's Spectrum rocket is almost ready to attempt the first orbital launch from Norway. The company is currently pending the launch license.
• Just four years after launch, the JWST program is facing a 20% budget cut. Hubble is also facing a 20% cut, and Chandra is facing an operational wind-down.
• • The Spectro-Photometer for the History of the Universe, Epoch of Reionization and Ice Explorer (SPHEREx) mission is set to launch. This telescope will provide an all-sky infrared spectral survey. The full sky will be surveyed every 6 months. The primary instrument is a spectrophotometer with six linear variable filters and six HgCdTe photodetector arrays. These are protected and kept passively cold by three concentric conic ‘photon shields,’ which give the telescope its distinctive appearance and also allows for no moving parts once the deployable components are unfurled in orbit.
• Boeing has now lost $2bn on the Starliner program.
• Astroscale successfully rendezvoused with a piece of space junk. The company has also been selected for a Japanese mission to demonstrate chemical refueling of satellites in low-Earth orbit and a life extension mission for an unidentified customer in geostationary orbit.
• New observations:
  • Rainbow-colored "Feather" cloud in the Martian sky (Curiosity)
  • Star forming region 30 Doradus (Chandra + Hubble)
  • LED 1313424 - a bullseye galaxy (Hubble)
  • Solar storms (GOES-19 weather satellite)
  • A supernova in Gemini (Hubble)
  • A scene from the Large Magellanic Cloud (Hubble)
  • Spiral Galaxy UGC 5460 (Hubble)
  • NGC 6505: an Einstein ring (Euclid)
  • Milky way over Earth's horizon (Astronaut Don Pettit)
  • Veil Nebula (Hubble)

Around the Web

Burkhard Stubert, who published a guide on Embedded Devices Covered by EU Cyber Resilience Act (CRA), has worked through the CRA's essential requirements related to product properties in an attempt to translate them into concrete, actionable requirements with supporting examples.

Eric Daigle discovered that he could easily break into dozens of apartment buildings thanks to the use of default credentials.

Property-based testing (PBT) is on our list of techniques to incorporate this year. We stumbled upon this excellent post providing general strategies for coming up with properties.

"How Core Git Developers Configure Git" covers some nice git configuration options to try out in your own environments.

Dan Mangum published The Miserable State of Modems and Mobile Network Operators, where he documents the debugging journey for a connectivity issue with the nRF9160.

Another debugging story was published on The Interrupt blog, looking at why std::this_thread::sleep_for() is broken on ESP32.

Shafik Yaghmour provided advice on what to do when optimizations change the behavior of your C++ program.

Rainer Grimm has published a series on implementing a lock-free stack using new C++26 capabilities:

• Deferred Reclamation in C++26: Read-Copy Update and Hazard Pointers
• A Lock-Free Stack: A Simplified Implementation
• A Lock-Free Stack: A Complete Implementation
• A Lock-Free Stack: Atomic Smart Pointer
• A Lock-Free Stack: A Simple Garbage Collector

Embedded Jobs

Latitude AI has an exciting opportunity for software managers interested in Integration Test and Infrastructure for supporting ADAS algorithm development.  We are hiring out of Palo Alto, Pittsburgh and Detroit.  Apply at https://job-boards.greenhouse.io/latitude/jobs/6666463.

Hiring Embedded Engineers?

Is your company hiring embedded systems hardware/software engineers? Send us a short job ad with a link to the full job description. We will be happy to include it in our next newsletter.

What's New on the Embedded Artistry Website

The following articles were published on the website in February:

• Creating Deterministic Zip Files with Git Commit Timestamps

We published the following new lessons in our course library:

• CI/CD for Embedded Systems
  • Principles of CI/CD
    • Principles of CI/CD - published an initial outline for this module
  • Developing an CI Pipeline
    • Developing a CI Pipeline - published an initial outline for this module
  • Specialized Pipelines
    • Specialized Pipelines - published an initial outline for this module
    • Clean, Versioned Builds - published initial notes for this lesson
    • Time-triggered CI Builds - published initial notes for this lesson
    • Optimizing Your CI Pipelines - published initial notes for this lesson
  • Continuous Delivery in an Embedded Context
    • Continuous Delivery in an Embedded Context - published an initial outline for this module
  • Hardware in the Loop
    • Hardware in the Loop - published an initial outline for this module
  • Models to Study
    • CI/CD Models to Study - published an initial outline for this module
    • Embedded Artistry CI/CD Infrastructure - collected links to the information about our processes as a starting point
• Consulting Lessons Learned
  • Managing Your Own Business
    • Managing Your Own Business
    • Understand Your Own Business
  • Proposals
    • Proposals
    • You Can't Change Cheap

We added the following entries to our Embedded Systems Field Atlas:

• Differentiating the Dependency Inversion Principle and Dependency Injection

We added the following entries to the Glossary:

• Continuous Delivery

Updates to Existing Content

We updated the following courses:

• Automated Software Quality Enforcement has now been renamed "CI/CD for Embedded Systems"
• Creating a Cross-Platform Build System for Embedded Projects with Meson
  • Updated course code for exercise solutions:
    • Use the latest libc skeleton files
    • Update printf.wrap file contents to match the lesson content
    • Update printf patch build file contents to match the lesson content
  • Getting Started
    • About the Meson Reference Project - updated file listing
  • Meson: Dependencies and Subprojects
    • Meson: Testing - added new files to the test list
• Behind the Scenes at Embedded Artistry HQ
  • Build and Release Processes
    • Embedded Artistry Build/Release Processes - created module introduction page
    • Embedded Artistry CI Process Overview - relocated this lesson into the proper module, relocated the video position on the page, added new links
    • Setting up CI/CD for Embedded Artistry Website Content - added table of contents, updated lesson based on developments that have happened since the lesson was published, added additional links to supporting content
    • A Look at CI/CD for Embedded Artistry Course Code - added table of contents, improved lesson formatting, relocated video position on the page, added new links

We updated the following entries in the Embedded Systems Field Atlas:

• Case Study: Mirai Botnet - added new examples
• Embedded Artistry's Continuous Integration Process - updated with new links, improved organization
• Exclusive Use of the Customer-Facing Firmware Update Mechanism- added new links
• Historical Software - added new links
• Security Case Studies - added new links
• Self-Documenting Code Is an Illusion - added new reference quotes

We updated the following entries in the Glossary:

• git - added new links and new sections
• GNU Debugger - added new tips and new links

Popular Articles

These were our most popular articles in February:

1. Creating a Circular Buffer in C/C++
2. For Beginners
3. printf a Limited Number of Characters from a String
4. Generating Aligned Memory
5. A General Overview of What Happens Before main()
6. Simple Fixed Point Conversion in C
7. Demystifying Microcontroller GPIO Settings
8. Three GCC Flags for Analyzing Memory Usage
9. -Werror is Not Your Friend
10. How to cherry-pick Specific Files from a Commit

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website and support our efforts to educate embedded developers by becoming a member or sponsoring a student membership. Members can also find the full archive of industry updates on our website.

Happy hacking!

-Phillip & Rozi