Saturday, April 27, 2013

Redirecting kernel messages (dmesg) out through your serial port

If you are into coding driver for Linux, you will know how precious those last few kernel messages are just before your module causes a system hang. So to alleviate that issue, we did to things:

First we started using virtualization. We use a Windows 7 PC (Linux can be used as well) to host Ubuntu as guest using VirtualBox. We use the "oh so awesome" SublimeText editor to edit the source code of the kernel module we are developing on the Windows PC. We shared the source code folder with the Ubuntu virtual machine (VM). This made life easier because it decreased the time to reboot Ubuntu in case of a hang. And whatever few seconds were involved in reboot could be used to look through the code on the unaffected source editor which ran outside the testing machine. This setup of course would only suit you if you aren't working on developing driver for devices connected to your system bus - you will need a real physical PC running Linux exclusively for that.

Secondly, we wanted a way of continuously buffering the kernel messages outside the virtual machine so that in case of a hang, we could sift through the messages and find the bug that caused it to crash. The most useful information was the one given by the kernel messages which came up just before the hang. We had worked on a few embedded ARM board (OLinuXino) and saw that kernel can be configured so that it spews kernel messages out of the serial port. And we figured out how to do the same on our Ubuntu virtual machine. You can even use this method when using a real PC (as opposed to a virtual machine). Here's how its done:
  1. Establish a serial connection with the PC under test:
    1. When using a real PC:
      Connect a null modem cable between the COM Ports of the PC under test and you development machine. If you are using laptops, you might need USB-to-Serial convertors. Once that is done, use PuTTY and configure it to open a console over serial at 115200 8N1.
    2. When using a virtual machine:
      We use virtualization. We have two Ubuntu guest VMs on our Windows PC - one is the PC  on which we test our kernel module and the other VM is used to fetch the kernel messages from the PC under test over a virtualized serial connection. To setup a virtualized serial connection between the two virtual machines look here. If you are using such a setup, make sure to start that virtual machine first which is configured to "create the pipe", otherwise the other machine would not startup because it will failt to connect to the as yet non existent pipe.
  2. Configure the Ubuntu PC under test to provide a console over serial port.
    Assuming you are using a distribution of Ubuntu later than 9.10, here is what you need to do (in case you are using a USB-Serial convertor, replace ttyS0 with ttyUSB0 or whatever your serial port shows up as under /dev):

    1. Issue the command: sudo gedit /etc/init/ttyS0.conf
    2. Type the following into the newly created file :
      # ttyS0 - getty
      # This service maintains a getty on ttyS0 from the point the system is
      # started until it is shut down again.

      start on stopped rc or RUNLEVEL=[2345]
      stop on runlevel [!2345]

      exec /sbin/getty -L 115200 ttyS0 vt102
    3. Issue the following: command at the terminal
      sudo start ttyS0
      Gained console access over serial port. The left VM is the machine under test.
      The right VM is used to access the console over virtual serial.
      (NOTE In case of virtualization, he second VM used to fetch the kernel messages
      from the VM under test must be started early too so that it begins clearing out the
      buffered messages from the Windows pipe otherwise the VM under
      test will pause booting until the pipe is cleared.)
  3. Configure grub and change the kernel command line parameters:
    1. On the test PC issue the command: sudo gedit /etc/default/grub
    2. In that file change GRUB_CMDLINE_LINUX="" to GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 ignore_loglevel"
    3. Also uncomment the line #GRUB_TERMINAL=console to GRUB_TERMINAL=console and add a line below it:
      GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
    4. Save the file and exit the editor
    5. Execute the command to update grub: sudo update-grub2
      The bootloader (GRUB2) will now configure the kernel to print the kernel messages on the console on every bootup
    6. Now you have console access over serial port to the test machine as well as live real time updates of kernel messages over the same. You can configure PuTTY to even log the captured kernel messages in a file.
      Capturing kernel messages over serial port
      Kernel messages before/during hang
PS: The host pipe method is not that stable, so an alternative is to use VM's serial port in Host Device mode. What you have to do is make sure that your hardware has two serial ports - you can add serial ports using USB-Serial converters. Lets say that these are COM11 and COM12. What you have to do is to open the settings page of the Ubuntu virtual machine in VirtualBox and set the COM1 of the virtual machine in "Host Device" mode and set it to receive and forward all data from/to COM11 of the Host machine. Then you can connect a NULL modem between COM11 and COM12 and use PuTTY on the Host OS (in my case Windows 7) to display the data received on COM12. This involves using hardware to feedback the data from the Guest OS's serial port back to the Host OS's serial port.


Friday, April 26, 2013

Disabling Avahi (mDNS Daemon) on Ubuntu

mDNS (multicast DNS) is the technology which allows you your computer, smart TV, smart phones to discover services on each other when they are connected to the same local network. Its part of the technology that enables you to play music/photos/videos on your TV from your smartphone over WiFi.

mDNS is one of the techniques for enabling Zero Configuration Networking

  • Apple's Zeroconf implementation is called Bonjour and uses mDNS
  • Ubuntu uses Avahi (wiki | home) as its mDNS daemon
  • Windows uses SSDP to implement UPnP
At times, while testing protocol stacks, you might want a quiet network - one without all the service discovery packets being sent every now and then. To achieve that, you will need to disable mDNS and service discovery daemons on all the computers connected to the network.

On Ubuntu 10.10 and up the following method seems to disable Avahi completely:
  1. Execute sudo gedit /etc/init/avahi-daemon.conf and change:
    start on (filesystem
    and started dbus)
    start on (never
    and filesystem
    and started dbus)
  2. Execute sudo gedit /etc/default/avahi-daemon and change:
On Windows, you can disable the SSDP Discovery Service from the Services Panel as shown here: Disable the SSDP Discovery service

Wireshark Capture: Note the SSDP (Green) and MDNS (Red) packets
sent out by Windows and Linux Hosts on the LAN 

Wednesday, April 24, 2013

Serial Ports between Ubuntu Guests on a Windows Host using VirtualBox

To establish a virtual serial connection between two Ubuntu virtual machines running on a Windows 7 host using VirtualBox. They should be able to exchange data over ttyS0 using a terminal emulation software like PuTTY. It like simulating two physical PCs both running Ubuntu, whose RS232 Ports are connected to each other using a null modem cable (TX to RX).

My setup:
Host OS: Windows 7 32 bit
Guest1 OS: Ubuntu 10.10 i386
Guest2 OS: Ubuntu 10.10 i386
VirtualBox: 4.1.16
(The following method should work on newer versions of VirtualBox and the Host and Guest OSes. Note that Ubuntu 10.10 is no longer supported, so it is better that you try this with a new version.)
  1. Setup the environment:

    1. Install VirtualBox on your Windows PC.
    2. Create a virtual machine on it for Ubuntu.
    3. Make sure to enable COM1 and COM2 for this machine under the serial ports option and leave them as disconnected before you install Ubuntu on it.
    4. Install the latest version of Ubuntu on it using the iso image available here:
    5. Install VirtualBox Guest Additions on Ubuntu and make sure that things like bidirectional clipboard copy work between Ubuntu and Windows.
    6. Install PuTTY on Ubuntu (sudo apt-get install putty).
    7. Turn off the Ubuntu virtual machine (VM) and clone it to end up with two similar virtual machines.
      Two Ubuntu Virtual Machines created on Windows 7 PC using VirtualBox
  2. Right click the first Ubuntu virtual machine and perform the following setting:
    Set Port type to "Host Pipe".
    Enter Path as "\\.\pipe\myserial".
    Check "Create Pipe"
  3. Right click the second (Cloned) Ubuntu virtual machine and perform the following setting:
    Set Port type to "Host Pipe".
    Enter Path as "\\.\pipe\myserial".
    Leave "Create Pipe" unchecked on this second VM
  4. Now start the first Ubuntu VM only. This one had the "Create Pipe" checked. Starting this VM before the other one will create the pipe and the other Ubuntu VM will connect to this pipe.
  5. Start the second Ubuntu VM and run PuTTY on both VMs.
  6. Now using PuTTY, open ttyS0 on both VMs. Try typing something in PuTTY window on one VM, it should show up on the other and vice versa. Since we are using pipes, the baudrate (Speed) on PuTTY's configuration page does not matter.
PuTTY config page

Great Success!

Disabling IPv6 on Ubuntu

There might be times when you might want to disable IPv6 support on your PC running an older version of Ubuntu (in my case 10.10) and you try out the following methods:

Method 1)
Simply issuing the command:
sudo sh -c "echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6" might not work on older versions of Ubuntu.

Method 2)
Neither would the alternate method of editing /etc/sysctl.conf as shown here: How to disable IPv6 in Ubuntu 10.04 Lucid Lynx work on older versions of Ubuntu.

Method 3)
To conclusively disable IPv6 (on Ubuntu 10.10 for example) you need to resort to editing the command line options passed to the kernel by the bootloader (grub) as shown here: How to disable IPv6 in Ubuntu 9.10 Karmic Koala
It involves adding the line GRUB_CMDLINE_LINUX="ipv6.disable=1" to /etc/default/grub and running update-grub2

Don't go by the titles of the posts linked to above, editing the kernel command line arguments is still the best way to disable IPv6 on any version of Ubuntu (although Method 1 does work simply and nicely on newer versions of Ubuntu like 12.04.1)

After executing method 3 on Ubuntu 10.10, you will get the following error while trying to manually set IPv6 address on your PC:

anurag@anupc:~$ sudo /sbin/ifconfig eth6 inet6 add 2001:0db8:0:f101::1/64
No support for INET6 on this system.

In case you want  renable IPv6 on Ubuntu 10.10 after executing method 3, do the following
  1. Edit /etc/default/grub by executing:
    sudo gedit /etc/default/grub
  2. Change GRUB_CMDLINE_LINUX="ipv6.disable=1" to GRUB_CMDLINE_LINUX="ipv6.disable=0" and save the file.
  3. Update Grub by running: sudo update-grub2
  4. Reboot the PC
  5. Execute:sudo sh -c "echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6" 
  6. Reboot once more and you are done.

Wednesday, April 10, 2013

Accessing Serial Console on iMX233-OLinuXino-MAXI

The default console for ArchLinux distribution for OLinuXino is directed to /dev/ttyAMA0.
To access this console, connect a FTDI Basic Breakout - 3.3V from SparkFun using Male to Female Jumper Wires like so:

Connect the FTDI board to your PC using a microUSB Cable and install the FTDI Vitual COM Port Drivers if required.
Then using PuTTY (dowload putty.exe from here) on (works on Windows as well as Linux) to create a serial session over the appropriate serial line (COM37 in my case) at 115200 speed :

On Windows, to look up your COM Port number, you have to usually open the Device Manager expand the Ports class and find it under there like so:

If you use Windows Vista or Windows 7, you can install the FTDI gadget which docks in one corner of your screen and displays the COM port numbers of all FT232RL based USB-Serial converters attached to your computer. This is a quicker way to know the COM number assigned to your FTDI Basic Breakout - 3.3V

Friday, April 5, 2013

kevent 2 may have been dropped

If you are using iMX233-OLinuXino-MAXI with Archlinux on it, it is highly likely that you might get the following kernel message when you connect your board to the network:
smsc95xx 1-1.1:1.0: eth0: kevent 2 may have been dropped
Users of Raspberry Pi faced the same issue (it has the same network chip on it as the OLinuXuiono).
Here is how to fix it.

Access the console on OLiniXino's ttyAMA0 serial port using PuTTY from your Windows/Linux PC (instructions here).

Issue the following command and reboot the board:

sh -c "echo vm.min_free_kbytes = 8192 >> /etc/sysctl.conf"
sysctl -p

What this does is limit the system memory used by the network device - and so you won't see those annoying messages anymore.

Taken from the discussion on the forum here: Network and system hanging

Thursday, April 4, 2013

Sharing folders between Ubuntu 12.04.1 Guest and Windows 7 Host when using VirtualBox

If you are running Ubuntu 12.04.1 i386 on your Windows 7 32-bit using VirtualBox, and found yourself wanting to share files between the two, here is how to get that done using the GUI:
  1. Make sure to install the guest addition on your Guest OS (Ubuntu 12.04.1)
    This will allow you to share clipboard (Copy/Paste) between Ubuntu and Windows and allow you to share folders between the two among other things.
    The following screenshots provide hints at how to get that done.
    Select "Install Guest Addition.."
    This will mount the VirtualBox Guest Additions CD image ISO file on Ubuntu
    Autorun dialog will appear in Ubuntu, select Run

  2. Share folders
    Share some folders between Windows and Ubuntu.
    The following screenshots provide hints at how to get that done.
    You will need to reboot the virtual machine every time you want to share a new folder.
    Select "Shared Folders" from the Devices menu of VirtualBox
    Click the Add icon (Blue folder with green sign)
    Make sure to select the "Auto-mount" and "Make Permanent" Options

  3. Install "Users and Groups"
    If you cd to the shared folders (eg /media/sf_GitHub in my case), you will note that you aren't allowed to do so. This is because the user account that you have logged in with does not yet have the permission to access those folders. What we need is a GUI program which will help you to add users to various groups and grant them permission. To install this program, open a terminal windows in Ubuntu (Ctrl+Alt+T) and execute the command:
    sudo apt-get install gnome-system-tools

    For this installation to execute successfully, you will need internet access on your PC. Make sure that the Ubuntu VirtualMachine can access the internet using the Guest OS's connection (enabled by default when you install VirtualBox)

  4. Add user to vboxsf group
    Open a terminal (Ctrl+Alt+T) and execute the command:

    A GUI panel will open up, follow the instructions as below:
    Start "Users and Groups" by running the command users-admin
    Then select your username and click on "Manage Groups"

    Select "vboxsf" from the list of group and click properties

    Tick the checkbox next to your username and click OK.
    Restart your Ubuntu virtual machine.
    You should be able to cd to your shared folder in /media