SCA230 -- Issue, root cause, workaround, and requested fix

Embedded Security ChipWhisperer Software
1

While going through SCA203, I kept having trouble with the jupyter notebooks.

Eventually, I was able to find a root-cause and workaround. This post is to document for others who might hit this, and to request a fix by maintainer of the cwtvla module.

How to determine if you are affected

Run pip check – does it list any version conflicts, especially about numpy and chipwhisperer?

Background

  1. the chipwhisperer 6.0.0 module requires numpy <=1.26.4.
  2. the cwtvla 0.1.3 module requires numpy, but does not list any version metadata.
  3. The chipwhisperer 6.0.0 module does not list cwtvla as a prerequisite (which seems correct)
  4. The cwtvla 0.1.3 module does not list chipwhisperer as a prerequisite

As a result, cwtvla must be manually installed. A typical installation would simply be: pip install cwtvla.

However, because cwtvla does not list version metadata in its requirements, that will install the latest available versions. In particular, numpy will be updated to version 2.x.x … even though it breaks the chipwhisperer module’s explicit requirement of numpy <= 1.26.4.

As a result, lots of things break.

Workaround

pip uninstall -y scipy numpy cwtvla
pip install numpy==1.26.4
pip install "scipy<1.14"
pip install cwtvla --no-deps
pip check

Proper Fix

Fix the cwtvla module dependencies:

  • add chipwhisperer == 6.0.0
  • modify Requires: scipy → Requires: scipy<1.14
  • similarly, add Requires: zarr <= ... with proper version metadata, since that module is also required for at least cwtvla.cw_convenience
  • etc.

This would ensure that The cwtvla module should list dependencies that don’t conflict with the chipwhisperer module.

In fact, perhaps cwtvla module should depend on the chipwhisperer module directly?

2

Hi,

We haven’t made any updates to that package in quite a while, so it’s definitely fallen into disrepair a bit, moreso with recent (and incoming) changes to base ChipWhisperer. I’ll try to take a look and get that fixed up after I’m done with my current project.

4

@Alex_Dewar

This is just some thoughts, from someone who’s only recently learned about the limitations of pip install being run more than once (each one will happily ignore any prior constraints).

That said, it seems a few ways exist that may improve the customer experience.

An intermediate step to improve stability

Installation instructions today include:

python -m pip install -e .
python -m pip install -r jupyter/requirements.txt
# and later ...
python -m pip install nbstripout

These separate calls to pip install are a source of instability. The later calls can (and do) upgrade versions, potentially to incompatible versions.

Instead, jupyter/requirements.txt should probably:
a) be alphabetized
b) include cwtrace, cwtvla – from sca courses
c) include nbstripout – high value
d) include pip-tools – see later for why
e) (+ any other modules needed by courses)

With those jupyter/requirements.txt changes, the installation instructions can be simplified to a single pip install invocation:

python -m pip install -e . -r jupyter/requirements.txt

This ensures the critical fix … ensuring pip has visibility into all the dependencies it must simultaneously meet for this virtual env, which helps ensure customer success when going through these (very useful) courses.

Stronger steps to help ensure customer success

First, get to a state where it’s possible to create a fresh venv, follow the instructions, and walk through each of the jupyter notebooks, without needing to modify the list of installed modules.

Then, use pip-tools to generate a stronger listing of which modules are in that virtual environment, in a way that customers can use. For example, generating the list might be:

cd ~/chipwhisperer
source ~/.cwvenv/bin/activate
pip-compile --generate-hashes --output-file requirements.lock pyproject.toml jupyter/requirements.txt

Why this helps? The customer can ensure their own virtual environment is 100% in sync as follows:

cd ~/chipwhisperer
source ~/.cwvenv/bin/activate
pip-sync requirements.lock

This ensures they have the exact same modules in their virtual environment, removing a large set of potential problems, and helping to ensure reproducibility.

in closing…

Just some thoughts, and free advice is often worth what you paid for it. But … something along the above lines might really improve the experience for folks, by removing a whole class of problems from the set of gremlins that can make stuff not work.

5

That’s a good point about combining the installation and requirements pip calls. I’ll update installation instructions to reflect that.

The prerecorded traces can also take up a lot of space, so we don’t want to install those for users who have hardware and don’t want to use them.

The rest of your suggestions seem reasonable. I’ll explore those further and implement if they work well