Issue connecting to ICE40/Neorv32

So I have been working on trying to clock glitch the neorv32. I have an issue I cannot quite explain. I seemingly connect to the target, glitch as you would expect from the first couple courses in fault101, but I consistently get reset/successful runs at 180 seconds into the glitch campaign regardless of glitch setting. I suspect it is an issue with not communicating correctly, but I am not sure what I am doing wrong. If I was not talking to the target I would get an error, but instead I get normal runs than a sudden change into resets/successful runs where the cycle I am glitching does not seem to matter.

I am glitching tiny-aes, test_encrypt_cbc() specifically, where the only real changes made were removing the printf calls. My main looks like below:

    platform_init();
    // The 3 functions below just call neorv32.h functions.
    init_uart();
    trigger_setup();
    simpleserial_init();

// Set callback function(s).
#if SS_VER == SS_VER_2_1
    simpleserial_addcmd(0x01, 5, aes);
#else
    simpleserial_addcmd('p', 5, aes);
#endif

    while (1)
        // Looks for input coming in from Chipwhisperer simpleserial, calls callback function on input.
        simpleserial_get();

The code contains a function that looks like this:

uint8_t aes(uint8_t cmd, uint8_t scmd, uint8_t dlen, uint8_t *data){ 
      trigger_high();
      int val = PASS_FAILURE;
      val = test_encrypt_cbc();
      trigger_low();
      simpleserial_put(‘r’, 1, (uint8_t *)&val);
      return 0x0;

I attached the main script below.

SCOPETYPE = 'OPENADC'
PLATFORM = 'CW308_NEORV32'
SS_VER = 'SS_VER_2_1'
%%bash -s "$PLATFORM" "$SS_VER"
cd ../../../firmware/mcu/simpleserial-glitch-tiny-aes
make PLATFORM=$1 CRYPTO_TARGET=NONE SS_VER=$2 -j OPT=0
...
from tqdm.notebook import tqdm
import re
import struct
import time

sample_size = 1

# Target cycle calculated from RTL simulation.
#494326
IF_CONDITION = 15000 #377108 #371760

# Range of error for target instruction, in cycles.
ERROR_TOLERANCE = 20 # Range found from 100k attempt results


TARGET_CYCLE = IF_CONDITION

# The number of cycles a normal execution takes.
# Note 1: Found running program and dividing the scope.adc.trig_count, after trigger has been lowered, by 4.
# Note 2: The ADC increments four times each cycle—by default, anyway. If your numbers are very off, check!
PROGRAM_MAX_CYCLE_COUNT = 375786
    
# Set up glitch parameters ranges and step.
gc.set_range("width", 3500, 4500)        # Default values from the solution script
gc.set_range("offset", 2000, 3200)       # Default values from the solution script
gc.set_global_step([400, 200, 100])      # Default values from the solution script
#gc.set_range("width", 4300, 4500)       # custom values
#gc.set_range("offset", 2300, 2800)      # custom values
gc.set_range("ext_offset", TARGET_CYCLE - ERROR_TOLERANCE, TARGET_CYCLE + ERROR_TOLERANCE)
#gc.set_range("ext_offset", 40, 200)
#gc.set_range("ext_offset", 1, PROGRAM_MAX_CYCLE_COUNT)
gc.set_step("ext_offset", 1) # We are interested in the whole cycle search space; set step to 1.

scope.glitch.repeat = 1 # This says how many pulses the glitch signal has, not how many times we try. Check docs before touching it!
reboot_flush()
scope.adc.timeout = 1.5 # Number, in seconds, scope will wait before aborting a capture.

hitList = list()        # Holds time and parameters for successful runs
failList = list()       # Holds time and parameters for crashed runs
normalList = list()     # Holds time and parameters for benign runs
counter = 1             # Attempt counter.

clock_ID = time.CLOCK_MONOTONIC # Clock considers time since boot, including time the system has been suspended.
start_time = time.clock_gettime(clock_ID) # Get some time, in seconds. This will be our start time.
end_time = start_time # We will update this value each iteration.

for glitch_settings in gc.glitch_values():
    #start_time = time.time_ns()
    scope.glitch.offset = glitch_settings[1]
    scope.glitch.width = glitch_settings[0]
    scope.glitch.ext_offset = glitch_settings[2]
    for i in range(sample_size):
        if scope.adc.state:
            # can detect crash here (fast) before timing out (slow)
            failList.append((end_time - start_time, counter, scope.adc.trig_count/4, scope.glitch.width, scope.glitch.offset, scope.glitch.ext_offset))
            gc.add("reset")
            #Device is slow to boot?
            reboot_flush()

        scope.arm()
        data = bytearray([0]*5)
        target.simpleserial_write('p', data)
        #target.send_cmd("p", "P", bytearray([0]*5))
        ret = scope.capture()

        # Gather list elements; number of cycles + parameters. ADC counts 4 times each cycle, so we need to divide its count by 4.
        listElement = (end_time - start_time, counter, scope.adc.trig_count/4, scope.glitch.width, scope.glitch.offset, scope.glitch.ext_offset)
        
        if ret:
            listElement = (end_time - start_time, counter, scope.adc.trig_count/4, scope.glitch.width, scope.glitch.offset, scope.glitch.ext_offset)
            failList.append(listElement)
            gc.add("reset")
            
            #Device is slow to boot?
            reboot_flush()
        else:
            val = target.simpleserial_read_witherrors('r', 1, glitch_timeout=10, timeout=50) #For loop check
            if val['valid'] is False:
                listElement = (end_time - start_time, counter, scope.adc.trig_count/4, scope.glitch.width, scope.glitch.offset, scope.glitch.ext_offset)
                failList.append(listElement)
                gc.add("reset")
            else:

                if val['payload'] == bytearray([1]): #for loop check
                    listElement = (end_time - start_time, counter, scope.adc.trig_count/4, scope.glitch.width, scope.glitch.offset, scope.glitch.ext_offset)
                    hitList.append(listElement)
                    print(end_time-start_time)
                    gc.add("success")
                else:
                    listElement = (end_time - start_time, counter, scope.adc.trig_count/4, scope.glitch.width, scope.glitch.offset, scope.glitch.ext_offset)
                    normalList.append(listElement)
                    gc.add("normal")        
        end_time = time.clock_gettime(clock_ID) # Update timer
        counter = counter + 1                   # Update interation counter

Our documentation explains that:

As the iCE40 has no nRST pin by default, resetting the device is done by reconfiguring it. Note that glitch attacks may result in corruption of the configuration RAM (CRAM) of the iCE40, so if using this target for glitching it is suggested to reprogram it after EVERY attempt.

I added cw.program_target(scope, prog, fw_path) just above scope.arm() in the main loop.

Aside from running slower, two things have changed: first, results are more in line with what I would expect from a normal glitch campaign. Second, I will sometimes get the following error:

(ChipWhisperer Target WARNING|File SimpleSerial2.py:385) Unexpected start to command 0x52, expected 0x72
(ChipWhisperer Target WARNING|File SimpleSerial2.py:418) Unexpected length 84, 1

Is this simply the result of a crashed run?

Out of curiosity, is there any explanation behind the 180 second number I was seeing before? I understand I was likely running on a corrupted a core, but the number seemed a little too specific.