DebuggingNetworkManager

Differences between revisions 36 and 54 (spanning 18 versions)
Revision 36 as of 2010-09-17 19:33:30
Size: 9518
Editor: modemcable251
Comment:
Revision 54 as of 2023-03-09 08:20:23
Size: 4564
Editor: seb128
Comment: Debug hints for captive portal issues
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Available languages: [[https://wiki.ubuntu.com/DebuggingNetworkManager_it| Italiano]],
Line 3: Line 4:
== Bug Summary ==
= Bug Summary =
Line 11: Line 13:
== Understanding your bug and getting more information == = Understanding your bug and getting more information =
Line 13: Line 15:
 * There is a lot of debugging information available on the GNOME Live wiki: [[http://live.gnome.org/NetworkManager/Debugging|NetworkManager/Debugging]].  * There is a lot of debugging information available on the GNOME Live wiki: [[https://wiki.gnome.org/Projects/NetworkManager/Debugging|NetworkManager/Debugging]].
Line 16: Line 18:
 * The similar reason/status code data for wpasupplicant is available here: [[http://hostap.epitest.fi/gitweb/gitweb.cgi?p=hostap.git;a=blob;f=src/common/ieee802_11_defs.h|ieee802_11_defs.h]]
Line 17: Line 20:
== A Testcase ==
Line 19: Line 21:
A good testcase is a step by step instruction to reproduce your bug starting with driver unloaded and NetworkManager stopped. = Getting debug logs =
Line 21: Line 23:
With Karmic and above: Stop NetworkManager You can then follow developers' intructions on a bug report for the exact command line to use; or run it directly as such:

== Getting NetworkManager debug logs ==
Line 24: Line 28:
  sudo stop network-manager sudo /usr/lib/NetworkManager/debug-helper.py --nm debug
Line 27: Line 31:
Prior to Karmic (9.10): Kill NetworkManager Then get the logs which will be written to /var/log/syslog. To disable it, pass '''info''' instead of '''debug''' in the above command, or reboot.

== Getting ModemManager debug logs ==
Line 29: Line 36:
  sudo killall NetworkManager sudo /usr/lib/NetworkManager/debug-helper.py --mm debug
Line 32: Line 39:
To unload your driver {{{ sudo modprobe -r DRIVER }}}. Then get the logs which will be written to /var/log/syslog. To disable it, pass '''info''' instead of '''debug''' in the above command, or reboot.
Line 34: Line 41:
Then load the driver {{{ sudo modprobe DRIVER }}} and start NetworkManager: See also [[DebuggingModemmanager]].
Line 36: Line 43:
With Karmic and up: == Getting wpasupplicant debug logs ==
Line 38: Line 46:
  sudo start network-manager sudo /usr/lib/NetworkManager/debug-helper.py --wpa msgdump
Line 41: Line 49:
Up until Karmic:
{{{
  sudo NetworkManager
}}}
Then get the logs which will be written to /var/log/syslog. To disable it, pass '''info''' instead of '''debug''' in the above command, or reboot.
Line 46: Line 51:
=== Capture Log === == Getting a capture of syslog ==

Mixing and mashing the above is perfectly acceptable as well if you want to see how NetworkManager and other parts of the stack interact together.
Line 67: Line 74:
=== Serial Log (Mobile Broadband) === == Handling 3G / modem issues ==
Line 69: Line 76:
==== With ModemManager ==== An few extra things that are very helpful to add in case of issues with 3G:
Line 71: Line 78:
{{{
# become root
sudo -i
The output of udevadm for tty devices, and output of lsusb:
Line 75: Line 80:
# stop NM
stop network-manager

# stop ModemManager
killall -TERM modem-manager

# start ModemManager in debug mode
modem-manager --debug 2>&1 | tee ~/modemmanager.log

# In another terminal, start NetworkManager:
NM_PPP_DEBUG=1 /usr/sbin/NetworkManager --no-daemon 2>&1 | tee ~/networkmanager.log
}}}

Then retrieve both networkmanager.log and modemmanager.log from your home directory and attach them to your bug report.

==== Before ModemManager ====
{{{
# become root
sudo su

# stop NM
/etc/init.d/NetworkManager stop

# start with serial log and tee it in a log file
NM_SERIAL_DEBUG=1 NetworkManager --no-daemon 2>&1 | tee /tmp/nm-serial.txt
}}}

... reproduce and attach the nm-serial.txt

== Debugging Crashes ==


To install debug symbols, add the following line to your {{{/etc/apt/sources.list}}}

{{{
deb http://ddebs.ubuntu.com/~ubuntu-archive/ddebs/ hardy main universe
}}}

or

{{{
deb http://ddebs.ubuntu.com/~ubuntu-archive/ddebs/ intrepid main universe
}}}


Then install the appropriate dbgsym packages:

{{{
  sudo apt-get update
  sudo apt-get install network-manager-dbgsym libnm-util0-dbgsym libnm-glib0-dbgsym libglib2.0-0-dbgsym

or from jaunty on:
  sudo apt-get install network-manager-dbgsym libnm-util1-dbgsym libnm-glib0-dbgsym libglib2.0-0-dbgsym
}}}

Then stop NetworkManager (hardy):
{{{
sudo /etc/dbus-1/event.d/25NetworkManager stop
}}}

or stop NetworkManager (intrepid):
{{{
sudo /etc/init.d/NetworkManager stop
}}}

Run NetworkManager in a debugger
{{{
sudo gdb /usr/sbin/NetworkManager
...
(gdb) run --no-daemon
}}}

Once it crashes get a backtrace
{{{
(gdb) bt
...
(gdb) bt full
...
(gdb) thread apply all bt full
...
}}}

and attach the backtrace above together with your {{{/var/log/syslog}}} to the bug.


== Debugging 3G modems ==

'''Note:''' since karmic NM does not handle modems on its own, but uses modem-manager: follow the DebuggingModemmanager instructions.

In case your modem is not detected, open a bug and attach the following information:

if a modem is not detected, but the driver is loaded and so on, you should have /sys/class/tty/ttyUSB or ttyACM or other entries

run
{{{
$ /lib/udev/nm-modem-probe --verbose --export /sys/class/tty/ttyUSB
}}}
 or
{{{
$ /lib/udev/nm-modem-probe --verbose --export /sys/class/tty/ttyACM
}}}

 to see if it detects GSM or CDMA capabilities


File a bug by running
{{{
$ ubuntu-bug network-manager
}}}
 and add the output of the previous nm-modem-probe command, complete syslog, output of
Line 188: Line 83:
 and
Line 190: Line 85:
$ lshal $ lsusb
Line 193: Line 88:
== Captive portal ==
Line 194: Line 90:
== Driver Logs ==

When a bug appears to be driver related or you are asked by a bug triager to submit a driver enabled
log, you need to enable driver logging right before you start to capture your testcase. How to do that depends
on the driver you use and whether it has been with compiled with debug support.


== Driver Specific Info ==
=== IPW (2100,2200, 3945) ===
==== Logging ====
You can check the status from the cli using
Line 205: Line 92:
# either during module load:
 modprobe ipw{2100,2200,3945} debug=65535

# or when already loaded you can change the debug_level through /sys/bus/.../drivers/
 echo 65535 > /sys/bus/pci/drivers/ipw{2100,2200,3945}/debug_level
$ nmcli networking connectivity check
Line 212: Line 95:
==== Compiling Module Sources ====
For debugging purpose or to verify a fix, a developer might ask you to build your driver module from source. for ipwXXXX you can do that by:
The server should return that value
Line 215: Line 97:
# install required headers and build tools
sudo apt-get install module-assistant
sudo module assistant update
sudo module assistant prepare

# build the driver
cd /path/to/ipw-XXXX
make IEEE80211_IGNORE_DUPLICATE=y SHELL=/bin/bash

# backup your old ipw driver:
sudo cp /lib/modules/$(uname -r)/kernel/drivers/net/wireless/ipw3945/ipw3945.ko $HOME

# install the new driver
cp ipw3945.ko /lib/modules/$(uname -r)/kernel/drivers/net/wireless/ipw3945/
depmod -a

# kill regulatory daemon
ipw3945d-$(uname -r) --kill

#load new module
modprobe ipw3945
$ wget -O - http://connectivity-check.ubuntu.com./ -d 2>&1 | grep X-
X-NetworkManager-Status: online
Line 238: Line 101:
== Bug Tags == Since 1.38, set Environment=NM_LOG_CONCHECK=1 in NetworkManager.service and restart the service to get additional debug logging about connectivity checking.
Line 240: Line 103:
These tags allow isolation of bugs into smaller groups, providing an easier and faster way to work on specific issues. = A Testcase =
Line 242: Line 105:
||<rowbgcolor="#FFEBBB"> '''Tag''' || '''Use case''' ||
|| [[https://launchpad.net/ubuntu/+bugs?field.tag=driver-madwifi|`driver-madwifi`]] || Bugs in which the madwifi driver is in use ||
|| [[https://launchpad.net/ubuntu/+bugs?field.tag=driver-ndiswrapper|`driver-ndiswrapper`]] || Bugs in which the ndiswrapper driver is in use ||
|| [[https://launchpad.net/ubuntu/+bugs?field.tag=vpn|`vpn`]] || Bugs related to either openvpn, vpnc or pptp network-manager vpn modules ||
|| [[https://launchpad.net/ubuntu/+bugs?field.tag=encryption-wep|`encryption-wep`]] || Bugs in which WEP encryption is used ||
|| [[https://launchpad.net/ubuntu/+bugs?field.tag=encryption-wpa|`encryption-wpa`]] || Bugs in which WPA encryption is used ||
|| [[https://launchpad.net/ubuntu/+bugs?field.tag=encryption-wpa2|`encryption-wpa2`]] || Bugs in which WPA2 encryption is used ||
A good testcase is a step by step instruction to reproduce your bug starting with driver unloaded and NetworkManager stopped.
Line 250: Line 107:
The previously described tags are specific to the [[NetworkManager]] application, if you need more general tags please visit [[Bugs/Tags]] page.


== Testing a patch ==

It's often a good idea to check in the [[https://edge.launchpad.net/~network-manager/+archive/trunk|NM trunk PPA]] to see if an updated version of NetworkManager fixes the bugs you see. Be careful though, as those are daily builds that may break other things, and are to be considered unstable.

Sometimes a developer might ask you to test a quick patch he came up with. Below an example how to do that for network-manager. If your bug is for network-manager-applet or some other package, just replace the package name

{{{
# get build dependencies
sudo apt-get install build-essential fakeroot devscripts
sudo apt-get build-dep network-manager

# get sources
sudo apt-get source network-manager

# cd into the source directory
cd network-manager-*/

# apply the patch (note that -p1 could also be -p0 if the patch was created without an extra prefix in the diff path)
patch -p1 < /tmp/yourdownloadedpatch.diff

# build binaries only without signing
debuild -b -uc

# packages are produced in parent directory
cd ../

# install all nm packages
sudo dpkg -i network-manager*.deb libnm-*.deb
First, stop NetworkManager: {{{
sudo service network-manager stop
}}} Then unload your driver:{{{
sudo modprobe -r DRIVER
}}} Next, load the driver:{{{
sudo modprobe DRIVER
}}} and finally start NetworkManager:{{{
sudo service network-manager start
Line 282: Line 116:

Note that you didnt change the version of the package. so to get back to the "official" current version you need to do something like:
{{{
  sudo apt-get install --reinstall network-manager libnm-util0 libnm-glib0

or from jaunty on
  sudo apt-get install --reinstall network-manager libnm-util1 libnm-glib0
}}}

if you installed other packages from the build, do the same for them.

----
CategoryBugSquad

Available languages: Italiano,

Debugging Central

This page is part of the debugging series — pages with debugging details for a variety of Ubuntu packages.

Bug Summary

If a network-manager bug report is about not being able to connect the title or summary should be in the format:

"[CHIPSET] cannot connect to (ENCRYPT_METHOD)"

where the CHIPSET is the wireless driver used and ENCRYPT_METHOD is the encryption method used by your wireless network.

Understanding your bug and getting more information

Getting debug logs

You can then follow developers' intructions on a bug report for the exact command line to use; or run it directly as such:

Getting NetworkManager debug logs

sudo /usr/lib/NetworkManager/debug-helper.py --nm debug

Then get the logs which will be written to /var/log/syslog. To disable it, pass info instead of debug in the above command, or reboot.

Getting ModemManager debug logs

sudo /usr/lib/NetworkManager/debug-helper.py --mm debug

Then get the logs which will be written to /var/log/syslog. To disable it, pass info instead of debug in the above command, or reboot.

See also DebuggingModemmanager.

Getting wpasupplicant debug logs

sudo /usr/lib/NetworkManager/debug-helper.py --wpa msgdump

Then get the logs which will be written to /var/log/syslog. To disable it, pass info instead of debug in the above command, or reboot.

Getting a capture of syslog

Mixing and mashing the above is perfectly acceptable as well if you want to see how NetworkManager and other parts of the stack interact together.

In order to understand whats going on and track down issues, its good to have a full log. To do so, capture the complete test case and submit the whole file (don't cut out what you think is important). Please add markers in the log file so the bug triager can easily see what actions the user takes at what point of time (this isn't essential, but helps a lot).

To capture the syslog, do:

 tail -n0 -f /var/log/syslog > /tmp/syslog

and to stop capturing do Ctrl-C (you will have to type your other commands in an other window or tab)

Adding markers is just like adding new lines with an editor that show the triager what happened at what point of time. You can also do this on the fly as you test with the command logger "[ clicked on wireless network 'ubuntu']" .

Example marker:

Sep  6 08:12:30 ...

[ clicked on wireless network 'ubuntu']

Sep  6 08:12:31 ...
...

Handling 3G / modem issues

An few extra things that are very helpful to add in case of issues with 3G:

The output of udevadm for tty devices, and output of lsusb:

$ udevadm info --query=all --path=/sys/class/tty/... --attribute-walk

$ lsusb

Captive portal

You can check the status from the cli using

$ nmcli networking connectivity check

The server should return that value

$ wget -O - http://connectivity-check.ubuntu.com./ -d 2>&1 | grep X-
X-NetworkManager-Status: online

Since 1.38, set Environment=NM_LOG_CONCHECK=1 in NetworkManager.service and restart the service to get additional debug logging about connectivity checking.

A Testcase

A good testcase is a step by step instruction to reproduce your bug starting with driver unloaded and NetworkManager stopped.

First, stop NetworkManager:

sudo service network-manager stop

Then unload your driver:

sudo modprobe -r DRIVER

Next, load the driver:

sudo modprobe DRIVER

and finally start NetworkManager:

sudo service network-manager start

DebuggingNetworkManager (last edited 2023-06-27 20:21:45 by hellsworth)