DebuggingNetworkManager

Differences between revisions 4 and 57 (spanning 53 versions)
⇤ ← Revision 4 as of 2007-09-27 09:28:01 →
Size: 3555
Editor: e177119140
Comment:
← Revision 57 as of 2023-06-27 20:21:45 → ⇥
Size: 5239
Editor: hellsworth
Comment: Added brief section on netplan
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
=== A Testcase === Available languages: [[https://wiki.ubuntu.com/DebuggingNetworkManager_it| Italiano]],
<<Include(Debugging/Header)>>
||<tablestyle="float:right; font-size: 0.9em; width:30%; background:#F1F1ED; background-image: url('https://librarian.launchpad.net/1812570/bugsquad.png'); background-repeat: no-repeat; background-position: 98% 0.5ex; margin: 0 0 1em 1em; padding: 0.5em;"><<TableOfContents>>||
Line 3: Line 5:
A good testcase is a step by step instruction to reproduce your bug starting with driver unloaded and NetworkManager stopped. = Bug Summary =
Line 5: Line 7:
Kill NetworkManager If a network-manager bug report is about not being able to connect the title or summary should be in the format:

"[CHIPSET] cannot connect to (ENCRYPT_METHOD)"

where the CHIPSET is the wireless driver used and ENCRYPT_METHOD is the encryption method used by your wireless network.

= Understanding your bug and getting more information =

 * There is a lot of debugging information available on the GNOME Live wiki: [[https://wiki.gnome.org/Projects/NetworkManager/Debugging|NetworkManager/Debugging]].
 * You can also take a look at [[http://live.gnome.org/DarrenAlbers/NetworkManagerFAQ|Darren Albers' FAQ]].
 * There is additional information on DebuggingNetworkManager/ReasonCodes for disconnection and network changes available.

= Getting debug logs =

You can then follow developers' intructions on a bug report for the exact command line to use; or run it directly as such.

== Getting NetworkManager debug logs ==

By default, the NetworkManager log level is set to info. You can use nmcli to modify the logging level:
Line 7: Line 28:
  sudo killall NetworkManager $ sudo nmcli general logging level DEBUG domains ALL
Line 10: Line 31:
To unload your driver {{{ modprobe -r DRIVER }}}. You do not need to restart NetworkManager to begin seeing debug messages logged to journalctl. You can watch the NetworkManager logs:
Line 12: Line 33:
Then load the driver and start NetworkManager:
Line 14: Line 34:
  sudo NetworkManager $ sudo journalctl -f -u NetworkManager
Line 17: Line 37:
=== Debugging Crashes === == Getting ModemManager debug logs ==
Line 19: Line 39:
To install debug symbols, add the following line to your {{{/etc/apt/sources.list}}} Manually run with debug enabled:
Line 22: Line 42:
deb http://people.ubuntu.com/~ubuntu-archive/ddebs/ gutsy main universe $ sudo /usr/sbin/ModemManager --debug
$ sudo /usr/sbin/NetworkManager --debug --log-level=DEBUG
Line 25: Line 46:
Then install the appropriate dbgsym packages: See also [[https://modemmanager.org/docs/modemmanager/debugging/|DebuggingModemmanager]].

You do not need to restart ModemManager to begin seeing debug messages logged to journalctl. You can watch the ModemManager logs:
Line 28: Line 51:
sudo apt-get update
sudo apt-get install network-manager-dbgsym libnm-util0-dbgsym libnm-glib0-dbgsym libglib2.0-0-dbgsym		 $ sudo journalctl -f -u ModemManager
Line 32: Line 54:
Then restart NetworkManager: == Getting wpasupplicant debug logs ==

Change the log level:
Line 34: Line 58:
sudo /etc/dbus-1/event.d/25NetworkManager restart $ sudo wpa_cli log_level debug
Line 37: Line 61:
Find the Process ID You do not need to restart wpa_supplicant to begin seeing debug messages logged to journalctl. You can watch the wpa_supplicant logs:
Line 39: Line 64:
ps -aef | grep NetworkManager | grep -v NetworkManagerDisp | grep -v grep
root 24349 1 0 11:25 ? 00:00:00 /usr/sbin/NetworkManager --pid-file /var/run/NetworkManager/NetworkManager.pid
           ^^^ this is the Process ID		 $ sudo journalctl -f -u wpa_supplicant
Line 44: Line 67:
Attach the debugger
{{{
sudo gdb /usr/bin/NetworkManager 24349 # <- use the process id found above
...
(gdb) continue
}}}		 == Getting a capture of syslog ==
Line 51: Line 69:
Once it crashes get a backtrace
{{{
(gdb) bt
...
(gdb) bt full
...
(gdb) thread apply all bt full
...
}}}

and attach the backtrace above together with your {{{/var/log/syslog}}} to the bug.


=== Capture Log ===		 Mixing and mashing the above is perfectly acceptable as well if you want to see how NetworkManager and other parts of the stack interact together.
Line 70: Line 75:
 tail -n0 /var/log/syslog > /tmp/syslog
}}}		 $ tail -n0 -f /var/log/syslog > /tmp/syslog
}}} and to stop capturing do Ctrl-C (you will have to type your other commands in an other window or tab)
Line 73: Line 78:
Adding markers is just like adding new lines with an editor that show the triager what happened at what point of time. Adding markers is just like adding new lines with an editor that show the triager what happened at what point of time. You can also do this on the fly as you test with the command {{{logger "[ clicked on wireless network 'ubuntu']" }}}.
Line 80: Line 85:
Line 84: Line 90:
== Handling 3G / modem issues ==
Line 85: Line 92:
=== Driver Logs === Here are a few extra things that are very helpful to add in case of issues with 3G.
Line 87: Line 94:
When a bug appears to be driver related or you are asked by a bug triager to submit a driver enabled
log, you need to enable driver logging right before you start to capture your testcase. How to do that depends
on the driver you use and whether it has been with compiled with debug support.		 The output of udevadm for tty devices, and output of lsusb:
Line 91: Line 96:

== Driver Specific Info ==
=== IPW (2100,2200, 3945) ===
==== Logging ====
Line 96: Line 97:
# either durint module load:
 modprobe ipw{2100,2200,3945} debug=65535

# or when already loaded you can change the debug_level through /sys/bus/.../drivers/
 echo 65535 > /sys/bus/pci/drivers/ipw{2100,2200,3945}/debug_level		 $ udevadm info --query=all --path=/sys/class/tty/... --attribute-walk
Line 103: Line 100:
==== Compiling Module Sources ====
For debugging purpose or to verify a fix, a developer might ask you to build your driver module from source. for ipwXXXX you can do that by:
Line 106: Line 101:
# install required headers and build tools
sudo apt-get install module-assistant
sudo module assistant update
sudo module assistant prepare		 $ lsusb
}}}
Line 111: Line 104:
# build the driver
cd /path/to/ipw-XXXX
make IEEE80211_IGNORE_DUPLICATE=y SHELL=/bin/bash		 == Captive portal ==
Line 115: Line 106:
# backup your old ipw driver:
sudo cp /lib/modules/$(uname -r)/kernel/drivers/net/wireless/ipw3945/ipw3945.ko $HOME		 You can check the status from the cli using:
{{{
$ nmcli networking connectivity check
}}}
Line 118: Line 111:
# install the new driver
cp ipw3945.ko /lib/modules/$(uname -r)/kernel/drivers/net/wireless/ipw3945/
depmod -a		 Since 1.38, you can set Environment=NM_LOG_CONCHECK=1 in NetworkManager.service and restart the service to get additional debug logging about connectivity checking.
Line 122: Line 113:
# kill regulatory daemon
ipw3945d-$(uname -r) --kill		 = A Testcase =
Line 125: Line 115:
#load new module
modprobe ipw3945		 A good testcase is a step by step instruction to reproduce your bug starting with driver unloaded and NetworkManager stopped.

First, stop NetworkManager and unload your driver:
{{{
$ sudo systemctl stop NetworkManager
$ sudo modprobe -r DRIVER
Line 128: Line 122:

Next, load the driver and start NetworkManager:
{{{
$ sudo modprobe DRIVER
$ sudo systemctl start NetworkManager
}}}

= Netplan =

A configuration abstraction mechanism has added to NetworkManager as of Ubuntu 23.10, called Netplan. There is a whole [[https://netplan.readthedocs.io/en/stable/|library of netplan documentation]] available. However here is a very basic amount of netplan debugging.

Netplan uses the files in /etc/netplan/*.yaml to generate the running profiles you find at /run/NetworkManager/system-connections/*.nmconnection. If you suspect a feature defined in your /etc/netplan/*.yaml files is not being used, it is a good idea to check that the content matches up.

If you find a netplan bug, please report it here: https://bugs.launchpad.net/netplan/+filebug

Available languages: Italiano,

Debugging Central

This page is part of the debugging series — pages with debugging details for a variety of Ubuntu packages.

Bug Summary

If a network-manager bug report is about not being able to connect the title or summary should be in the format:

"[CHIPSET] cannot connect to (ENCRYPT_METHOD)"

where the CHIPSET is the wireless driver used and ENCRYPT_METHOD is the encryption method used by your wireless network.

Understanding your bug and getting more information

Getting debug logs

You can then follow developers' intructions on a bug report for the exact command line to use; or run it directly as such.

Getting NetworkManager debug logs

By default, the NetworkManager log level is set to info. You can use nmcli to modify the logging level: 

$ sudo nmcli general logging level DEBUG domains ALL

You do not need to restart NetworkManager to begin seeing debug messages logged to journalctl. You can watch the NetworkManager logs: 

$ sudo journalctl -f -u NetworkManager

Getting ModemManager debug logs

Manually run with debug enabled: 

$ sudo /usr/sbin/ModemManager --debug
$ sudo /usr/sbin/NetworkManager --debug --log-level=DEBUG

See also DebuggingModemmanager.

You do not need to restart ModemManager to begin seeing debug messages logged to journalctl. You can watch the ModemManager logs: 

$ sudo journalctl -f -u ModemManager

Getting wpasupplicant debug logs

Change the log level: 

$ sudo wpa_cli log_level debug

You do not need to restart wpa_supplicant to begin seeing debug messages logged to journalctl. You can watch the wpa_supplicant logs: 

$ sudo journalctl -f -u wpa_supplicant

Getting a capture of syslog

Mixing and mashing the above is perfectly acceptable as well if you want to see how NetworkManager and other parts of the stack interact together.

In order to understand whats going on and track down issues, its good to have a full log. To do so, capture the complete test case and submit the whole file (don't cut out what you think is important). Please add markers in the log file so the bug triager can easily see what actions the user takes at what point of time (this isn't essential, but helps a lot).

To capture the syslog, do: 

$ tail -n0 -f /var/log/syslog > /tmp/syslog

and to stop capturing do Ctrl-C (you will have to type your other commands in an other window or tab)

Adding markers is just like adding new lines with an editor that show the triager what happened at what point of time. You can also do this on the fly as you test with the command logger "[ clicked on wireless network 'ubuntu']" .

Example marker: 

Sep  6 08:12:30 ...

[ clicked on wireless network 'ubuntu']

Sep  6 08:12:31 ...
...

Handling 3G / modem issues

Here are a few extra things that are very helpful to add in case of issues with 3G.

The output of udevadm for tty devices, and output of lsusb: 

$ udevadm info --query=all --path=/sys/class/tty/... --attribute-walk

$ lsusb

Captive portal

You can check the status from the cli using: 

$ nmcli networking connectivity check

Since 1.38, you can set Environment=NM_LOG_CONCHECK=1 in NetworkManager.service and restart the service to get additional debug logging about connectivity checking.

A Testcase

A good testcase is a step by step instruction to reproduce your bug starting with driver unloaded and NetworkManager stopped.

First, stop NetworkManager and unload your driver: 

$ sudo systemctl stop NetworkManager
$ sudo modprobe -r DRIVER

Next, load the driver and start NetworkManager: 

$ sudo modprobe DRIVER
$ sudo systemctl start NetworkManager

Netplan

A configuration abstraction mechanism has added to NetworkManager as of Ubuntu 23.10, called Netplan. There is a whole library of netplan documentation available. However here is a very basic amount of netplan debugging.

Netplan uses the files in /etc/netplan/*.yaml to generate the running profiles you find at /run/NetworkManager/system-connections/*.nmconnection. If you suspect a feature defined in your /etc/netplan/*.yaml files is not being used, it is a good idea to check that the content matches up.

If you find a netplan bug, please report it here: https://bugs.launchpad.net/netplan/+filebug

DebuggingNetworkManager (last edited 2023-06-27 20:21:45 by hellsworth)