If you are into coding driver for Linux, you will know how precious those last few kernel messages are just before your module causes a system hang. So to alleviate that issue, we did to things:
First we started using virtualization. We use a Windows 7 PC (Linux can be used as well) to host Ubuntu as guest using VirtualBox. We use the "oh so awesome" SublimeText editor to edit the source code of the kernel module we are developing on the Windows PC. We shared the source code folder with the Ubuntu virtual machine (VM). This made life easier because it decreased the time to reboot Ubuntu in case of a hang. And whatever few seconds were involved in reboot could be used to look through the code on the unaffected source editor which ran outside the testing machine. This setup of course would only suit you if you aren't working on developing driver for devices connected to your system bus - you will need a real physical PC running Linux exclusively for that.
Secondly, we wanted a way of continuously buffering the kernel messages outside the virtual machine so that in case of a hang, we could sift through the messages and find the bug that caused it to crash. The most useful information was the one given by the kernel messages which came up just before the hang. We had worked on a few embedded ARM board (OLinuXino) and saw that kernel can be configured so that it spews kernel messages out of the serial port. And we figured out how to do the same on our Ubuntu virtual machine. You can even use this method when using a real PC (as opposed to a virtual machine). Here's how its done:
First we started using virtualization. We use a Windows 7 PC (Linux can be used as well) to host Ubuntu as guest using VirtualBox. We use the "oh so awesome" SublimeText editor to edit the source code of the kernel module we are developing on the Windows PC. We shared the source code folder with the Ubuntu virtual machine (VM). This made life easier because it decreased the time to reboot Ubuntu in case of a hang. And whatever few seconds were involved in reboot could be used to look through the code on the unaffected source editor which ran outside the testing machine. This setup of course would only suit you if you aren't working on developing driver for devices connected to your system bus - you will need a real physical PC running Linux exclusively for that.
Secondly, we wanted a way of continuously buffering the kernel messages outside the virtual machine so that in case of a hang, we could sift through the messages and find the bug that caused it to crash. The most useful information was the one given by the kernel messages which came up just before the hang. We had worked on a few embedded ARM board (OLinuXino) and saw that kernel can be configured so that it spews kernel messages out of the serial port. And we figured out how to do the same on our Ubuntu virtual machine. You can even use this method when using a real PC (as opposed to a virtual machine). Here's how its done:
- Establish a serial connection with the PC under test:
- When using a real PC:
Connect a null modem cable between the COM Ports of the PC under test and you development machine. If you are using laptops, you might need USB-to-Serial convertors. Once that is done, use PuTTY and configure it to open a console over serial at 115200 8N1. - When using a virtual machine:
We use virtualization. We have two Ubuntu guest VMs on our Windows PC - one is the PC on which we test our kernel module and the other VM is used to fetch the kernel messages from the PC under test over a virtualized serial connection. To setup a virtualized serial connection between the two virtual machines look here. If you are using such a setup, make sure to start that virtual machine first which is configured to "create the pipe", otherwise the other machine would not startup because it will failt to connect to the as yet non existent pipe. - Configure the Ubuntu PC under test to provide a console over serial port.
Assuming you are using a distribution of Ubuntu later than 9.10, here is what you need to do (in case you are using a USB-Serial convertor, replace ttyS0 with ttyUSB0 or whatever your serial port shows up as under /dev):- Issue the command: sudo gedit /etc/init/ttyS0.conf
- Type the following into the newly created file :
# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
start on stopped rc or RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L 115200 ttyS0 vt102 - Issue the following: command at the terminal
sudo start ttyS0
Gained console access over serial port. The left VM is the machine under test.
The right VM is used to access the console over virtual serial.
(NOTE In case of virtualization, he second VM used to fetch the kernel messages
from the VM under test must be started early too so that it begins clearing out the
buffered messages from the Windows pipe otherwise the VM under
test will pause booting until the pipe is cleared.)
- Configure grub and change the kernel command line parameters:
- On the test PC issue the command: sudo gedit /etc/default/grub
- In that file change GRUB_CMDLINE_LINUX="" to GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 ignore_loglevel"
- Also uncomment the line #GRUB_TERMINAL=console to GRUB_TERMINAL=console and add a line below it:
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1" - Save the file and exit the editor
- Execute the command to update grub: sudo update-grub2
The bootloader (GRUB2) will now configure the kernel to print the kernel messages on the console on every bootup - Now you have console access over serial port to the test machine as well as live real time updates of kernel messages over the same. You can configure PuTTY to even log the captured kernel messages in a file.
Capturing kernel messages over serial port 
Kernel messages before/during hang
PS: The host pipe method is not that stable, so an alternative is to use VM's serial port in Host Device mode. What you have to do is make sure that your hardware has two serial ports - you can add serial ports using USB-Serial converters. Lets say that these are COM11 and COM12. What you have to do is to open the settings page of the Ubuntu virtual machine in VirtualBox and set the COM1 of the virtual machine in "Host Device" mode and set it to receive and forward all data from/to COM11 of the Host machine. Then you can connect a NULL modem between COM11 and COM12 and use PuTTY on the Host OS (in my case Windows 7) to display the data received on COM12. This involves using hardware to feedback the data from the Guest OS's serial port back to the Host OS's serial port.
References:
References:
Comments
Post a Comment