Hello, I found an issue on github (https://github.com/usb-tools/ViewSB/pull/22) mentioning stream capture was not supported in October 2019.
Is it supported now in PhyWhisperer and the ViewSB backend (i.e. is it possible to specify size=0 and get a stream of USB traffic)? If so, what is the effective throughput?
Thank you,
Adam
Hi Adam,
Yes stream capture is supported (I think what happened there is that I submitted the PR for the ViewSB changes required while I was still finishing the work required on my side, so that we’d have everything ready in time for our release).
In the PhyWhisperer repo there’s a stream.ipynb notebook to show how to use it. Let me know if you run into any issues with it. Data throughput will depend on the characteristics of your USB traffic, due to the timestamping that we add to every raw USB byte.
Jean-Pierre
Hi Jean-Pierre, thank you for the quick answer!
Any update on this? Does ViewSB support stream capture with the PhyWhisperer-USB today?
The latest commit in ViewSB seems to break PhyWhisperer-USB support for me, I had to go back to commit 5fa0dd8594f9c5ce9f7fed359a42387a953b08b7 for it to work, but setting --size 0 does not do stream capture on this commit.
I try to periodically check that ViewSB and PW interoperate, but it has been a while since I did that… I think the last commit that I checked was fc6d7a.
I’ll have a look, but as far as PW streaming is concerned, it shouldn’t matter to ViewSB whether PW is streaming or not (beyond calling PW with the required parameters). Does the latest work for you when not streaming?
The latest commit does not work with PW. I had to jump back to 5fa0dd8 (Nov 23, 2019), I picked it simply based on changes made to phywhisperer.py.
I just tried fc6d7a1 (Dec 15, 2020) and it also works. So something broke it between fc6d7a1 (Dec 15, 2020) and the tip of master (Jan 24, 2022).
How do you expect streaming to work when using ViewSB? For example, right now I’m using a flash drive as a test before moving on to my hardware; would you expect packets to continuously appear as I read or write files to the device? Like how ViewSB’s continous output behaves when used with usbmon?
As of commit fc6d7a1, using --stream 0 still limits the output, it doesn’t just keep going without end, even if --timeout 0 is given as well.
It seems different devices stop the capturing at different points. There was one instance when testing with the flash drive where ViewSB managed to dump a lot more of the traffic than all other attemps (maybe about 10x more) before stopping. Though I don’t think I did anything that caused that, I just tried it over and over again.
Answers to your questions are here:
https://phywhispererusb.readthedocs.io/en/latest/viewsb.html#viewsb
Keep in mind sniffing is a secondary feature. PW has very little on-board storage, and how much you can pull from streaming will depend on a lot of factors. You’re not going to get a very long stream capture of moving files on a flash drive. stream.ipynb says: “Captured data is read from the PhyWhisperer at up to 40 Mbps”.
Pushing the data to ViewSB for presentation will slow things down too. If you want to maximize the amount of data you can stream, don’t use ViewSB, just follow what the sniff.ipynb notebook shows.
Jean-Pierre
I just verified and streaming works fine for me with the latest ViewSB commit. I ran:
./viewsb.sh phywhisperer --size 0 --timeout 0 tui
With a plain old USB mouse as a target, the session runs perpetually and you see the traffic from mouse activity in real-time in the ViewSB interface.
Can you elaborate on how it doesn’t work for you?
I’m now testing with a (CP210x) TTL UART to USB cable to make sure I’m not exceeding its max capture rate.
For whatever reason tui isn’t working properly for me, so I’ve been testing with the cli and qt front-ends.
Using this serial cable, when I run ./viewsb.sh phywhisperer --size 0 --timeout 0 cli on commit fc6d7a1 it dumps maybe about two seconds of traffic, I see the “Capturing” LED illuminate for about two seconds before it turns off. If I use the tip of master, I can see the “Armed” and “Capturing” LEDs illuminate but ViewSB never spits out any data, it just sits there indefinitely with no output.
I’ve also ran the example code from both stream.ipynb and sniff.ipynb and neither of them seem to capture and dump the data in perpetuity. Though I haven’t gotten around to modifying the script in any meaningful way, perhaps that’s what you intended the user to do.
I found an old optical wired mouse that seems to work decently well (on fc6d7a1 but still not on the latest commit). I’m not sure why some devices just stop capturing without any detail given as to why.
FWIW, cli doesn’t work for me (it used to - dunno what changed); both qt and tui worked well.
When you run the sniff.ipynb notebook with your target, do you get a proper capture?
With the old optical mouse qt was working for maybe about 30 seconds before it gave out, cli seems to last much longer. They both stop capturing at some point though. I haven’t tested it enough times yet to determine how long they each last on average, or how it might depend on the activity (moving, clicking, etc.).
When running the sniff.ipynb example code it does capture and print data. But the way its written seems like it’s not intended to capture and dump data indefinitely like doing ./viewsb.sh phywhisperer --size 0 --timeout 0 cli.
The hardware I intend to sniff is HS, and I don’t yet know what pattern I need to start capture on. So I was hoping to do some continuous capturing in the hopes to figure out what the pattern will need to be. Once I can figure out what pattern I need to start capture on, then I can capture with ViewSB’s burst flag or even my own script produced from your Jupyter examples.
By the way, I’m pretty new this, but I found that in openvizsla’s
README it says it you can visualize the data using usb2sniffer-qt if the data is saved in “ITI1480A” format: ov_ftdi/README.md at master · openvizsla/ov_ftdi · GitHub
Would you know of a way to store the captured data in the ITI1480A format? It would be nice to use usb2sniffer-qt since it seems like a pretty capable tool. Maybe this format is already being used and I just don’t know it…
After taking another look, it seems that every time it stops capturing when testing with the wired mouse it’s due to a FIFO overflow. Is there a way to prevent or reduce the chance of an overflow occuring when doing a continuous capture, or is it just inevitable?
Edit: I managed to modify the script in stream.ipynb in a way that has it print the capture every time the FIFO gets filled repeatedly without end. I’m not sure if what I wrote would give me more or less dead time as compared to using ./viewsb.sh phywhisperer --size 0 --timeout 0 cli.
I’ll try to address all your questions, and point you in some hopefully useful directions; let me know if I’ve missed something:
The phy.read_capture_data() calls in our notebooks are blocking, so you won’t see the captured data until it’s all been read, streaming or no streaming.
To read and process concurrently, you have to use something like the ViewSB approach. But like I mentioned earlier, if your aim is to capture as much data as possible, then stick to how stream.ipynb does it, because the overhead of parsing the incoming data reduces the rate at which your PC is able to read the incoming data.
That said… if your captures work with our notebooks but not with ViewSB, maybe this points to some problem in your Python threading module? Because threading is what’s used to pass data from the PhyWhisperer backend to the rest of ViewSB. Are you able to use ViewSB with usbmon? If the problem lies there, then the folks over on the usb-tools discord (link here: USB Tools · GitHub) should be able to help you.
FIFO overflow occurs when your PC is no longer keeping up with the incoming USB data. The PhyWhisperer FPGA has nowhere else to store the data, so it gives up. Everything you read before the overflow is still valid. If your target is producing USB traffic at a rate greater than what your PC can read from PhyWhisperer, and you’re using --size=0, then an overflow will always happen. That’s ok.
There are many variables that go into determining how fast you can read - it depends on all the layers from Python to your physical USB port, and everything in between.
I’m not familiar with the ITI480A format. PW doesn’t support saving the captured data to any format, but you have the raw USB data and so anything is possible (pull requests welcome!).
I’m not sure what you did here without seeing your code, but it sounds like you would be capturing chunks of USB traffic, with some USB activity missed in between each chunk; this probably isn’t very useful.
PW really isn’t designed to capture long USB sessions, especially at HS rates. There are other tools that are much better for that, and our crowdsupply project page highlights this. Have you considered using usbmon to see what your target traffic looks like, and from that, pick out a pattern to trigger a short capture with PW?
Jean-Pierre
Hi Jean-Pierre,
This is all very useful information, thank you so much for sharing it.
Are you able to use ViewSB with usbmon?
Yes.
I’m not sure what you did here without seeing your code, but it sounds like you would be capturing chunks of USB traffic, with some USB activity missed in between each chunk; this probably isn’t very useful.
That’s exactly how I have it working as of last night. I just got my PW yesterday so I haven’t nailed down my script just yet.
Have you considered using usbmon to see what your target traffic looks like, and from that, pick out a pattern to trigger a short capture with PW?
This sounds like a good starting point. How would monitoring usbmon using ViewSB versus Wireshark compare? Would you expect any difference in their performance/ability to keep up?
Sorry if these are obvious questions.
Good question! I have no idea. See what the folks on the usb-tools discord think?