Hello Javier!

Welcome to the April 2024 edition of the Embedded Artistry Industry Update. This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

• Embedded Artistry's Spring Break
• Course updates
• Boeing's attempt to back out of its guilty plea agreement
• Google's Pixel 4a update
• HP Bricking Printers
• A summary of other noteworthy industry news
• Technical articles from around the web
• Embedded job postings
• Updates to the Embedded Artistry Website

Spring Break

Embedded Artistry missed its usual winter shutdown, so we will be taking a spring break this month instead. We'll still be available for customer support and student questions, but new content will be paused until we're back in May. Our next newsletter is also likely to be sparse.

Course Updates

We have several course updates to announce.

Our planned "Automated Quality Enforcement" course will be split up into smaller courses that focus on distinct topics. The first piece will be CI/CD for Embedded Systems, and there is an initial outline published on the course page. A second course will be added to cover static and dynamic analysis tools, with the potential for a third covering design by contract and assertions.

As part of developing and testing the content for these courses, we will be rebuilding our CI/CD system and updating our analysis tool usage. We're planning a one-time intensive co-working week as part of this process. If you've wanted to setup up a CI system and/or integrate analysis tools into your workflow but haven't yet made time to do it, this is a great opportunity to get started. If you’re interested, reply to this email and we’ll follow up when we schedule the sessions.

Our CMake and Meson build system courses have also seen significant updates over the past month. These updates were initially triggered by the need to update the embeddedartistry/printf fork with the latest Catch2 version to address illegal instruction failures with newer toolchains. However, we've taken the opportunity to update the project files, make build system improvements, adjust exercises, and handle changes in newer build tool versions. We'll continue updating these courses over the next few months. We'll also be revisiting and updating our Reusable CMake Skeleton and Reusable Meson Skeleton follow-on courses.

Embedded Online Conference

The Embedded Online Conference 2025 is taking place 12-16 May, bringing together thousands of engineers from around the world for a week of practical, high-impact presentations focused on embedded systems development. This year’s lineup features industry experts Jacob Beningo, Philip Koopman, James Grenning, Clive Maxfield, and many more! Don’t miss out on this great lineup of talks, and be sure to use the promo code ARTISTRY2025 at checkout to receive a $50 discount on your registration!

Boeing's Appalling Request

In 2018 and 2019, two Boeing 737 MAX planes crashed, resulting in the deaths of 346 people. Boeing avoided prosecution for fraud and negligence by signing a deferred prosecution agreement in January 2021, agreeing to pay $2.5bn in penalties and compensation. 

However, in May 2024, the U.S. Justice Department determined that Boeing violated the terms of its deferred prosecution agreement"by failing to design, implement, and enforce a compliance and ethics program to prevent and detect violations of the U.S. fraud laws throughout its operations." This was triggered by severe quality problems observed by investigators following an incident where a door plug blew off of a 737 MAX 9 jet in flight. The Justice Department brought charges against Boeing for "conspiracy to defraud the the United States," and more specifically, the FAA Aircraft Evaluation Group.

In July 2024, Boeing agreed to plead guilty to a criminal charge in order to avoid a trial. Penalties included a $234.6m fine, $455m investment into compliance and safety programs, and oversight from an external monitor. The company did this even though they felt they honored the terms of the 2021 agreement. Families of the crash victims submitted a filing urging the court to reject the plea deal, arguing that "the plea deal unfairly makes concessions to Boeing that other criminal defendants would never receive and fails to hold Boeing accountable for the deaths of 346 persons." The families also include a quote from Judge Reed O'Connor that "Boeing's crime may properly be considered the deadliest corporate crime in U.S. history." 

In December 2024, Judge Reed O'Connor rejected the plea deal. The Court states their priorities at this point are to ensure Boeing adheres to an ethics and anti-fraud compliance program overseen by an independent monitor. However, the plea agreement a) requires the government to consider DEI factors when selecting a monitor, which the Court expects Boeing will use to exercise its strike of one of the government's chosen monitors, b) specifies that the monitor reports to the government instead of the Court, and, most importantly, c) prohibits imposing a requirement for Boeing to comply with the monitor's anti-fraud recommendations. How convenient to agree to be monitored but not be required to make any changes based on the findings of said monitor.

In March, parties filed a status report that says that the "parties are continuing to meet and confirm on a resolution" that would serve to avoid a trial. Judge O'Connor ordered Boeing and the government to provide an update on how they plan to proceed" by 11 April.

Now, however, it seems that Boeing is seeking Department of Justice support to withdraw its guilty plea agreement (Ars Technica alternate) in order to obtain more lenient treatment (i.e., weaker penalties) from the Trump administration. The Wall Street Journal reports that the Department of Justice "is reviewing numerous pending criminal cases that haven't yet gone to trial or been approved by courts," and "Boeing stands to benefit from fresh eyes at Trump's Justice Department, which is inclined to at least modify parts of the agreement."

This is absolutely appalling. Hundreds of people died due to Boeing's fraud. The U.S. has failed to hold Boeing accountable to date. And when the winds appear favorable, Boeing shamelessly continues to avoid real accountability for its ongoing actions, for the deaths of 346 people, and for the impact on their families and friends.

Google's Painful Pixel 4a Update

After receiving it's "final" software update in November 2023, Pixel 4a owners were surprised with an another software update last month. In January, Google announced a "Pixel 4a Battery Performance Program", which stated an automatic update would be deployed. For some "Impacted Devices," this update would result in reduced battery runtime and charging performance. Impacted customers could choose between three "appeasement" options: a $50 payment, $100 in Google Store credit, or replacing the battery in their phone for free (only for some locations). Notably, the Program document does not mention any safety or hazard concerns.

After the update, complaints abounded about unusable battery life, with some users only getting two hours on a charge. Hector Martin reverse engineered the update and noted (in a thread that no longer exists) that certain battery profiles saw reduced maximum charge voltage (reducing charging speed and maximum charge potential), and that some batteries ("LSN", suspected to be manufacturer Lishen) assigned the "debug" profile would see capacity reduced from 3080 mAh to 1539 mAh. Some owners have reported that, even after a battery swap, their devices have the same limited capacity, which implies that batteries with problematic cells are still being used in replacements. 

Google also took steps to make sure users were unable to avoid or revert the update. It was sent as an automatic update that users could not opt out of. The company also removed all of the Pixel 4a's factory images from its website, making it more difficult to roll back firmware without hunting around for an old image on a third-party site. However, because source code for these changes is unavailable, third-party custom firmware cannot incorporate the changes and remains unaffected.

The battery changes and these actions imply that there is a safety concern with these batteries - but the company remained silent on this front. That was, until Australia released a Product Safety Recall notice for the Pixel 4a that noted the new firmware was released to "mitigate the risk of overheating" and that "an overheating battery could pose a risk of fire and/or burns to a user." Surely the company is remaining quiet out of legal concern, but this is hardly an acceptable approach. Users in one country should not need to learn about battery overheating and fire concerns through another country's regulatory mechanisms.

Device owners have also complained about Google's handling of the "appeasement" options. For one, to receive the $50 you have to provide excessive personal details to a third-party company, who then says they will charge an annual account fee of $29.95 unless you receive $2000 or more (though the details on when this applies are unclear). Users have also had trouble getting repairs completed, especially if there is a secondary problem like a cracked screen or failed camera, since "Google will not perform partial repairs." Others have also learned that the trade-in value for their Pixel 4a has been reduced after the firmware update.

All-in-all, if you're looking for a playbook on how to handle a customer facing problem, don't follow Google's.

HP Bricking Printers

HP is already infamous for issuing firmware updates that brick printers that have tried to use third-party ink. Now, HP is being accused of releasing a firmware update that has broken laser prints using HP-brand toner.

The problematic update is 20250209, issued on 4 March (really now, shouldn't the update match the release date?) for LaserJet MFP M232-M237 models. After the update, users have reported seeing "Error Code 11" and flashing toner lights when trying to print. Even after reinstalling toner, printing fails. HP has stated they are aware of a firmware issue, but they have not released a fix. Of course, this is on par for the company that added mandatory 15-minute wait times to support calls to encourage you to solve the problem yourself.

The company has also moved to settle a class acton suit with customers, who sued the company for issuing firmware updates that prevent their printers from using non-HP ink and toner. The original complaint was filed in December 2020 and focused on a November 2020 firmware update. HP has positioned these updates as a "security" measure, despite there being no credible reason to believe that printers are being hacked through ink cartridges, while the litigants claim the updates are malware that prevent users from selecting their own toner suppliers. The settlement, however, looks to be negligible: HP doesn't admit to any wrongdoing, and impacted customers won't receive monetary relief. The primary litigants will each receive $5,000 "to compensate them for the services they performed on behalf of the classes," and will pay $725,000 in legal fees. One minor boon to HP customers seems to be that users of specific printers would be able to decline firmware updates that would push Dynamic Security. However, the specific printers are only those that were impacted by the November 2020 update in question. Plenty of other printers that make use of "Dynamic Security" are not included on the list.

Another class-action complaint was filed in January 2024 for firmware updates released in 2022-2023. This complaint similarly accuses HP of creating a monopoly in replacement ink cartridges.

News in Brief

Software

• Zephyr 4.1.0 was released, including performance enhancements, experimental IAR compiler support, and initial support for writing Zephyr applications in Rust.

Security

• Switzerland's National Cybersecurity Centre has announced new 24 hour reporting requirements for critical infrastructure organizations experiencing cyberattacks.
• Supply chain attack on popular 'tj-actions/changed-files' Action exposes CI/CD secrets from 23,000 repositories.
• An attack on the 'reviewdog/action-setup@v1' GitHub Action is believed to have led to the attack on 'tj-actions/changed-files' by leaking a GitHub access token to public logs.
• GitHub found 39m secret leaks in 2024
• The U.S. Justice Department charged Chinese hackers linked to cyberattacks on U.S. federal and state agencies, foreign ministries, and other organizations.
• Many DreyTek routers experienced reboot loops and connectivity problems in March. While a bad firmware update was suspected, the problem seems to be the result of an attack on unpatched routers.
• A newly discovered botnet, Eleven11bot, comprises ~30,000 webcams and security and has been associated with recent "hyper-volumetric DDoS attacks." This is suspected to be another in the Mirai family.
• GitLab released security updates to patch authentication bypass vulnerabilities.
• The BadBox Anroid malware botnet has been disrupted again. 24 malicious Google Play apps have been removed, and command-and-control server communications have been blocked for 500k infected devices.
• Apple backported several security patches to older iPhones and Macs.
• Juniper Networks has patched a vulnerability that has allowed backdooring routers.

Chips

• TI Introduced the world's smallest MCU, measuring only 1.38mm².
• In a surprising twist, Donald Trump wants to kill the CHIPS Act in combination with leveraging tariffs on pretty much everything. The Trump administration's purge of government employees has also affected the U.S. Chips Program Office. There are also threats to claw back CHIPS act funding, although this is questionable since the U.S. government remains legally obligated to distribute already allocated funds. However, legality does not seem to be a hurdle for this administration. There was also an announcement regarding the creation of a "US Investment Accelerator" with the mission of negotiating better deals regarding CHIPS Act funding. Whether this applies to future funding or disbursed funds remains unclear.
• TSMC has announced $100B in additional investments for its U.S. chip fabrication facilities, making a total of three U.S. fab sites. Some say this is to avoid Trumps threat of a 25%+ tariff on semiconductor imports. TSMC says it is driven by demand, and that its Arizona fab's chip production is already sold out through late 2027. It is also reportedly speeding up fab construction, targeting the two year timeline that is typical for its Taiwan sites. 
• Qualcomm has launched a global antitrust campaign against Arm, filing complaints with regulators in the U.S., Europe, and South Korea. The core claim is that Arm limits access to its technologies and changes licensing models in a bid to harm competition.
• Intel announced its pick of Lip-Bu Tan as the new CEO.
• Japanese company Rapidus will begin trial production of its 2nm process.
• A potential venture for TSMC to run Intel's manufacturing capacity is reportedly still in the works. AMD, Broadcom, Nvidia, and Qualcomm, the leading U.S. fabless chip designers, would receive stakes in the joint venture to be operated by TSMC. However, Nvidia's CEO denies they have been approached, and a TSMC board member denied claims that the company is considering purchasing Intel's foundry unit.
• GlobalFoundries is reportedly exploring a merger with Taiwanese chipmaker United Microelectronics Corp.
• SoftBank announced that it is acquiring Ampere Computing, a developer of Arm datacenter CPUs, for $6.5bn. 
• Intel plans to continue using TSMC even when 18A production has ramped up.
• NAND Flash market is down 6.2% quarter-over-quarter.

Trade War

• The trade war has expanded well beyond chips now, with tariffs in the U.S. being added for aluminum, steel, vehicles, auto parts, and increased levies on all goods from China. The U.S. is raising tariffs on many of its [former?] allies. Additionally, "reciprocal tariffs" have been announced, with the scope surprising many. Many affected countries have announced plans for retaliatory tariffs. While the current tariffs do not include semiconductors, the president has indicated they are coming for semiconductors too. However, the tariffs do include wafer fabrication equipment (WFE), which seems quite counterproductive to the efforts to on-shore semiconductor fabrication.
• China's retaliation against the new U.S. tariffs includes a reciprocal 34% tariff. China is also restricting export of seven rare earth minerals to the U.S.
• Nintendo has halted Switch 2 preorders in the U.S. in response to tariffs.
• TSMC allegedly manufactured 2m Ascend 910 AI chips for Huawei shell companies in 2024 in violation of U.S. export controls. Other reports are also surfacing along these lines, such as Chinese firms circumventing export controls on Nvidia Blackwell GPUs by routing orders through nearby countries like Malaysia, Vietnam, and Taiwan. The U.S. has also asked Malaysia to 'monitor every shipment' to restrict the flow of GPUs to China.
• The U.S. government added 80 new organizations from China, Taiwan, the UAE, South Africa, Iran, and other countries to the "Entity List", with the intention of restricting the supply of advanced electronic components.
• China appears to be out researching the U.S. and Europe with respect to next-generation chip fabrication. The Ministry of Finance also announced that the 2025 central budget will allocate $55bn for science and technology investment, a 10% increase over 2024.
• The HP CEO says 90% of products for the U.S. will be made outside of China by October 2025. Lenovo will be moving production lines to India.
• Chinese lawmaker proposes law to allow Chinese companies to hide their foreign suppliers.

Consumer Electronics

• The new "Repebble" smartwatches are now available for preorder.
• Sonos has reportedly cancelled its new TV streaming device.

Automotive and Aviation

• The U.S. NHTSA issues the largest Cybetruck recall yet due to the risk of an exterior panel detaching while driving.
• U.S. EV sales are up 28% this year, and global EV sales are up 30%.
• Boeing won its bid to supply the U.S. Air Force with its next air superiority fighter. This will replace the Lockheed F-22 Raptor sometime in the 2030s.

Space

• NASA has turned of two Voyager science instruments to extend the operational lifetime of the probes. Electrical power is running low, and without turning the instruments off, they would only have a few more months of power remaining before end-of-mission is declared.
• Firefly Aerospace's Blue Ghost Lander successfully touched down on the moon on 2 March 2025, carrying 10 NASA instruments as its payload. The mission operated until the solar-powered lander went dark on 16 March. Firefly declared the mission 100% successful.
  • Shadow of Blue Ghost on the moon taken after landing.
  • Sunrise on the moon from Blue Ghost lander.
  • Blue Ghost Lander on the moon, seen from NASA's Lunar Reconnaissance Orbiter.
• Intuitive Machines's Athena lander ended its mission after one day on the moon, again Cnding up on its side like in the first mission. Also, due to the landing problems on this mission, a rover was unable to deploy due to the door being blocked, but nonetheless was able to collect data before lack of power ended the mission.
• Growing concentrations of greenhouse gases are making Earth's upper atmosphere thinner, decreasing its ability to pull space junk out of orbit.
• AroForge lost hope of talking to their Odin private asteroid probe. Their leading theory as to what went wrong with the probe is complications with solar panel deployment.
• Europe has retired the Gaia satellite after 12 years of mapping the Milky Way.
• Ontario Premier Doug Ford canceled the province's Starlink deal over U.S. tariffs.
• New observations:
  • Spiral Galaxy NGC 5530 (Hubble)
  • JADES-GS-z13-1, observed just 330 million years after the big bang (JWST)
  • Auroras on Neptune (JWST)
  • Spiral Galaxy NGC 2283 (JWST)
  • The Veil Nebula (Hubble)
  • Earth and Moon (ispace's Resilience lander)
  • Spiral Galaxy NGC 5042 (JWST)
  • Actively forming stars: L483 (JWST)
  • Open Cluster NGC 460 (Hubble)
  • Abell 370, a galaxy cluster 4 billion light-years away (JWST)
  • Nebula Sh2-284 in infrared (Hubble)
  • Elliptical galaxy NGC 3561B and spiral galaxy NGC 3561A (Hubble)
  • Spiral galaxy NGC 4536 (Hubble)
  • Small Magellanic Cloud (Hubble)
  • Spiral Galaxy NGC 4900 (Hubble)
  • Herbig Haro 49/50, an outflow from a nearby still-forming star (JWST)
  • Flame Nebula (JWST)
  • A planet destroyed by a white dwarf (Chandra)

Around the Web

Alvaro Prieto published his experiments with routing SWD through a USB-C connection for debugging use.

Mike Szczys described how to use your J-Link as a serial port for your device.

Herb Sutter has published an article discussing efforts to tame undefined behavior in C++. Also, he points out a fact that I didn't know: all constexpr/consteval compile-time code is UB-free, and as of C++26 most of the language and standard library is available at compile-time and UB-free when executed at compile-time. 

It looks like defer might actually be coming to C, and JeanHeyd Menade provides a comprehensive look at the specification. In short, this will bring "RAII-like" capabilities to C without actually requiring an object model or "real RAII" support. It solves real, existing problems when working in C, and we're hopeful to see it properly added to the language.

There's a new C programming language reference site that's run by the C Standards Committee.

Ars Technica had a nice article discussing the reason SNES hardware is now running faster than expected.

Maximilian Köhl published a guide on implementing robust OTA updates for Linux devices.

Dan Mangum published a taxonomy of connected device networks that serves as a useful primer.

Rainer Grimm continued his series on implementing a lock-free stack in C++26:

• A Lock-Free Stack: A Hazard Pointer Implementation
• A Lock-Free Stack: A Hazard Pointer Implementation Explained I

Hiring Embedded Engineers?

Is your company hiring embedded systems hardware/software engineers? Send us a short job ad with a link to the full job description. We will be happy to include it in our next newsletter.

What's New on the Embedded Artistry Website

Updates to Existing Content

We updated the following courses:

• Creating a Cross-Platform Build System for Embedded Projects with Meson
  • Change Log reflects the current course status
  • Code updates
    • Meson minimum version now 0.57.0
    • printf subproject now points to f687191
    • CMocka test code has been updated to reflect the latest libc skeleton requirements and the latest course content
    • C++14 is now the minimum C++ standard used by the course
    • Fixed problem with libc tests failing starting with 007
      • '-DNO_IEEE_Scale' Added to gdtoa compiler flags in all course code examples
    • meson_options.txt now renamed to meson.options
    • meson required version now v1.1 to match the use of meson.options
  • Getting Started
    • Installing Meson and Ninja - updated supported Meson versions.
  • Meson: An Introduction
    • Meson: An Introduction - renamed options file to meson.options
    • Anatomy of a Meson Project - renamed options file to meson.options
    • Initial Build Setup - added new required gdtoa compilation flag ('-DNO_IEEE_Scale')
  • Meson: Dependencies and Subprojects
    • Meson: Testing - add environment variable to generate cmocka XML output with new libc files, add note about future step to add removal for generated XML files, general copyediting
    • Exercise: printf Tests - updated instructions to reference latest requirements with the newest printf repo commits. This resolves test program execution issues with newer toolchains.
    • Exercise Solution: printf Tests - updated solution to reflect latest requirements for the exercise
  • Meson: Build Configuration
    • Run Targets - cleaned up text, added callouts, adjusted content to prepare for changes coming in the next lesson.
    • Exercise: libc Test Run Targets - rewrote exercise instructions to account for the fact that run_target now supports an env option and to support deleting the XML results.
    • Exercise Solution: libc Test Run Targets - rewrote of exercise solution to match new approach
    • Meson: Custom Build Options - renamed options file to meson.options
    • Exercise: Debug & Release Options - renamed options file to meson.options
    • Exercise Solution: Build Options - renamed options file to meson.options
  • Supporting Multiple Toolchains and OSes
    • Meson: Generating Linker Map Files - fixed logic around Map file argument detection since GCC 11.x added warnings and broke the logic.
• Building a Reusable Project Skeleton with Meson
  • Course code updates
    • Fixed broken cmocka test example program after a recent libc update
    • Updated CMocka module
    • Updated catch2 examples to match catch2v3
  • Expanding the Skeleton
    • Supporting A New Testing Framework - updated content to match the changes needed for Catch2 v3.
  • Reusable Build Modules
    • Creating Linker and Compiler Modules - updated linker map detection logic
• Creating a Cross-Platform Build System for Embedded Projects with CMake
  • Course code updated to include new gdtoa flag to resolve failing libc tests
  • CMake: An Introduction
    • CMake: Initial Build Setup - added NO_IEEE_Scale definition to resolve test failures with latest libc files
  • CMake: Dependencies
    • Exercise: Integrating External Dependencies - updated printf references to use the latest build approach
    • Exercise Solution: Integrating External Dependencies - updated printf commit reference to use the latest build approach
    • CMake: Testing - updated printf commit reference, add catch2 dependency, update requirement to C++14
    • Exercise Solution: gdtoa Interface Library - add NO_IEEE_Scale definition to resolve test failures with latest libc files
  • CMake: Build Options
    • Exercise Solution: Custom Build Options - update printf references to include Catch2
  • CMake: Cross-Compilation
    • Enabling Cross-Compilation in Libc - updated printf references

We updated the following Field Atlas entries:

• Case Study: Mirai Botnet - added new incident
• Historical Electrical Systems - added new links

We updated the following Glossary entries:

• Over-the-Air Update [OTA] - added new references
• Supply Chain Attack - added new case studies, reorganized content to group related attacks together

Popular Articles

These were our most popular articles in March:

1. Creating a Circular Buffer in C/C++
2. For Beginners
3. printf a Limited Number of Characters from a String
4. Generating Aligned Memory
5. Simple Fixed Point Conversion in C
6. A General Overview of What Happens Before main()
7. Demystifying Microcontroller GPIO Settings
8. Q&A: How We Document Software Projects
9. Three GCC Flags for Analyzing Memory Usage
10. -Werror is Not Your Friend

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website and support our efforts to educate embedded developers by becoming a member or sponsoring a student membership. Members can also find the full archive of industry updates on our website.

Happy hacking!

-Phillip & Rozi