Debugging
From Ubicom Developer
Contents |
Introduction
This section helps in setting up the debugging environment with Ubicom32 boards. Actual debugging is out of scope for this document. Debugging help is organized into following categories:
- User-Level Debugging
- Core Dumps
- Kernel Debugging
User-Level Debugging
How to connect
Connect your target board and your PC (with Ubicom32 Linux development environment), as shown below. gdbserver runs on the console of Ubicom32 board. For details of connecting console please refer to the Console and Debug Output page.
Please note that kernel will continue to run and only the user process under debug is effected. Also note that debug logs are active and available for use.
Setup (step 1) - Enable gdbserver
Enable gdbserver in your uClinux build by running make menuconfig from uClinux directory.
Kernel/Library/Defaults Selection --->
[*] Customize Application/Library Settings
In the next menu, select gdbserver.
Miscellaneous Applications --->
[*] gdbserver
Rebuild uClinux and load the image.
Setup (step 2) - Start GDB Server
Connect to console on Ubicom board. For details of connecting console please refer to the Console and Debug Output page. Run gdbserver HOST:PORT APPLICATION on the Ubicom Linux console. This command tells that gdbserver is communicating with host host PCs at the port mentioned via TCP. Please use a port number that is not being used by some other application.
Lets say we want debug ls (which is an alias for busybox)
Note: if your Internet interface is not already up, you should bring it up before this point; because once you start gdbserver, it will take over the console input (and cannot be stopped).
gdbserver localhost:1000 /bin/ls
You should see something like this.
Process /bin/ls created; pid = 33. Listening on port 1000
Setup (step 3) - Connect GDB to GDB Server from your PC
Start ubicom32-elf-gdb or ubicom32-elf-insight with the .gdb (elf) file for the application you are debugging. .gdb file contains all the symbols of the application and this file is present in the build directory of corresponding application in uClinux directory.
ubicom32-elf-gdb busybox_unstripped.gdb
HOST is IP Address of the target (e.g. the Router Gateway Board (not the dongle).
(gdb) target remote 192.168.0.193:1000 Remote debugging using 192.168.0.193:1000 0x41540410 in start () (gdb)
You are now debugging userspace.
Please refer to manual pages of gdb, gdbserver and insight for more details about using the tools.
When the kernel panics, you can dump the core for analyzing the problem at a later time. Connect the Ubicom dongle to the board and do the following.
bash$ export dongle_ip=192.168.0.36
bash$ export UBICOM_DONGLE=dongle_ip:5010
bash$ ubicom32-elf-gdb /home/work/Ubicom/sdk/projects/ultra/vmlinux.elf
GNU gdb 6.7.50-20071004-cvs
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=ubicom32-elf"...
(gdb) target ubicom32
system_thread_execute (arg=<value optimized out>) at /home/work/Ubicom/sdk/pkg/ipHAL/include/ip3k/ipHAL-ip3k.h:227
227 if (INT_REG(interrupt) == INT_REG(INT(0, 0))) {
(gdb) coredump
Dumping Caches
Dumping 256k OCM 0x3ffc0000-0x40000000
Dumping 65536k DDR 0x40000000-0x44000000. This takes time.
................................................................................
Dumping 8192k FLASH 0x60000000-0x60800000
................................................................................
(gdb) quit
This will create Ubicom.core and Ubicom.cache files.
Along with Ubicom.core and .elf files, you might need to collect UbicomSDK.h and UbicomSDK.ldi from sdk/projects/ultra/ directory and also uClinux/linux.2.6.x/.config files to make better analysis.
Analyzing Core Dumps
You can use the core file and the ELF file to start your debugger and analyze the crash or bug you are chasing.
bash$ export dongle_ip=192.168.0.36 bash$ export UBICOM_DONGLE=dongle_ip:5010 bash$ ubicom32-elf-gdb Ubicom.core vmlinux.elf GNU gdb 6.7.50-20071004-cvs Copyright (C) 2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=ubicom32-elf"... (gdb)
When do you need this..
The loaded flash image consists of the bootloader, Ultra, and Linux kernel image, complete with RAM file system. When something unexpected happens and the kernel panics, you see something like this:
[ 279.710000] Linux Kernel Trap: Causes: 0x00000808 [ 279.710000] GDB trapped cause decode: src1 address decode error [ 279.710000] GDB trapped cause decode: src1 range error [ 279.710000] regs: 4071bbb4, tid: 5, trap_cause: 00000808, pc: 422448fc [ 279.710000] [ 279.710000] Data registers [ 279.710000] D00: 00023f60, D01: 424b0000, D02: 00004034, D03: 00004034, [ 279.710000] D04: 00023f60, D05: 00000007, D06: 428f012e, D07: 428f012f, [ 279.710000] D08: 00000000, D09: 00000001, D10: 435b4380, D11: 00000001, [ 279.710000] D12: 00000032, D13: ffff0000, D14: 4071bbb4, D15: 000003a4, [ 279.710000] [ 279.710000] Address registers [ 279.710000] A00: 424b5db4, A01: 84965db4, A02: 40c3b280, A03: 424b03a4, [ 279.710000] A04: 40c3b280, A05: 422448f4, A06: 00000000, A07: 4071bc58, [ 279.710000] [ 279.710000] acc0_lo: 00000000, acc0_hi: 00000000, acc1_lo: 00000055, acc1_hi: 9b498bf4 [ 279.710000] mac_rc16: 85acaacf, source3: 00000055, inst_cnt: 9b49a01b, csr: 00000055 [ 279.710000] int_mask0: 00000000, int_mask1: 4040d660 [ 279.710000] frame_type: 2, nesting_level: 1, thread_type 1 [ 279.710000] [ 279.710000] CALL & CALLI on stack: $lt; 0 [ 279.710000] Kernel panic - not syncing: Aiee, killing interrupt handler!
Connecting to the Debugger
You need to start gdb with your Linux kernel image and attach to your dongle_ip_addr:5010, like this:
bash$ export dongle_ip=192.168.0.36 bash$ export UBICOM_DONGLE=dongle_ip:5010 bash$ ubicom32-elf-gdb /home/work/Ubicom/sdk/projects/ultra/vmlinux.elf GNU gdb 6.7.50-20071004-cvs Copyright (C) 2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=ubicom32-elf"... (gdb) target ubicom32
Analysis
Change your thread to the tid printed in the trap output. This is the ID of the thread that trapped. The kernel tends to use 0 base, and GDB wants 1 base; you will need to use 6 (kernel says tid 5 panic) for this example.
(gdb) thread 6
[Switching to thread 6 (Thread 5)]#0 0x4055b320 in udelay (usecs=1000) at arch/ubicom32/lib/delay.c:18
18 {
Now create a variable in GDB that is the regs value printed in the trap dump.
(gdb) set $regs = (struct pt_regs *)0x4071bbb4
Then print out the PC and stack pointer and set the thread so its PC and SP are these values by doing:
(gdb) p $regs->pc (gdb) p $regs->an[7] (gdb) set $pc = $regs->pc (gdb) set $sp = $regs->an[7]
Now you can backtrace to see the call stack, for example:
(gdb) bt #0 cpu_idle () at /home/work/Ubicom/uClinux/linux-2.6.x/arch/ubicom32/include/asm/ldsr.h:21 #1 0x4040daf4 in rest_init () at init/main.c:484 #2 0x4071c96c in start_kernel () at init/main.c:697 #3 0x4040f7bc in panic (fmt=0x40c42118 ") at kernel/panic.c:132 #4 0x00000000 in ?? ()
Then you can look at code or instructions and debug what happened:
(gdb) x/4i $pc 0x422448fc: add.4 d1,24(a1),d2 0x42244900: moveai a5,#%hi(0x40415f80) 0x42244904: calli a5,16(a5) 0x42244908: move.4 a1,(sp)4++
Notice, this trap was src1 address decode error; and that makes sense, because the instruction at the $pc has a1 as src1, and a1 is 0x84965db4, which is illegal.
