Building Linux Kernel Modules Faster: Internal and external

*For Newbie Linux Kernel and Device Driver developers*.

Most repetitive activity for Kernel and Device Driver developers. is building modules they work on and installing them. So lets discuss few tricks to build them faster way.

Traditionally, when we change a source code of a module, we need to 

a) Build the entire kernel again
make; make module;make modules_install;make install
(optionally using "-jN" option to make)
b) reboot and use the new modules.

But we can avoid reboot by 

a) Build the entire kernel again
b) Remove the old module 
c) Insert the new module.

Building the kernel itself is a time consuming process, especially when we just want to update code specific to our module. So for that we make use linux kernel's support and procedures for building external modules  i.e., modules which are not present in the kernel tree for modules in the kernel tree. Sounds Confusing?

Lets say we have written a new module and wanted to compile it, then we write a makefile which invokes the kernel makefile with below option:

make modules M=<Dir_of_new_Module>

But to compile a internal module (module which is present in the kernel tree) we use the same trick, except the makefile is already present. Just follow the below procedure

Procedure: 

1. make silentoldconfig  (Optional)
• whenever Kconfig is changed., For other cases like make and make modules (without M=) this is run by kernel makefile itself.
2. make modules M=<path_to_dir>
3. make modules_install M=<path_to_dir>
4. Remove the old module and Re-Insert the new one.

Still interested of know-how, see the steps in detail:

Details:

1) Whenever we change Kconfig, .config will be updated and we need to tell the kernel makefile to use the new .config instead of using old one.This does that.

2) This uses the makefile in the module sub directory and compiles the files w.r.t #1 .config options.

3) This will install the compiled modules in to the install path /lib/modules//`uname -r`/kernel/ (or) /lib/module/`uname -r`/extra/ depending on whether the module is internal/external. In our case its always extra as we are using "M=" option for building modules. We can add some prefix (extra folder name under the above paths).

Also it used the depmod utility (through a wrapper shell script: scripts/depmod.sh) to generate the below files which will later be used by modprobe.

• /lib/modules/`uname -r`/modules.alias
• /lib/modules/`uname -r`/modules.alias.bin
• /lib/modules/`uname -r`/modules.builtin
• /lib/modules/`uname -r`/modules.builtin.bin
• /lib/modules/`uname -r`/modules.ccwmap
• /lib/modules/`uname -r`/modules.dep
• /lib/modules/`uname -r`/modules.dep.bin
• /lib/modules/`uname -r`/modules.devname
• /lib/modules/`uname -r`/modules.ieee1394map
• /lib/modules/`uname -r`/modules.inputmap
• /lib/modules/`uname -r`/modules.isapnpmap
• /lib/modules/`uname -r`/modules.ofmap
• /lib/modules/`uname -r`/modules.order
• /lib/modules/`uname -r`/modules.pcimap
• /lib/modules/`uname -r`/modules.seriomap
• /lib/modules/`uname -r`/modules.softdep
• /lib/modules/`uname -r`/modules.symbols
• /lib/modules/`uname -r`/modules.symbols.bin
• /lib/modules/`uname -r`/modules.usbmap

Lets discuss the important ones below: 

• /lib/modules/`uname -r`/modules.dep
• Depmod is a utility which will use the "System.map" and try to work out the dependencies between the modules and create a file called "modules.dep", used by modprobe to insert (insmod) the modules in proper order.
• /lib/modules/`uname -r`/modules.order and /lib/modules/`uname -r`/modules.symbols will be used in that process.
• /lib/modules/`uname -r`/modules.alias
• It identifies the HW devices each device driver supports and writes to a file "module.alias" using which the linux kernel device model can insert the driver along with its dependencies (using modprobe and modules.dep) when it  enumerates a new HW device using PCI/USB ..
• /lib/modules/`uname -r`/modules.builtin
• This gives a list of modules which are staically linked to the kernel forming part of vmlinuz.
• /lib/modules/`uname -r`/modules.<BUS>map
• These files will the known drivers for devices and give informtion regarding the device like vendorid, subsystemid, deviceclass etc.
• This is used to create module.alias

4) Now that we have all the necessary files, use below commands "-v" is for verbose and "-r" is for using rmmod. By default it used insmod.

•     modprobe -v -r 
•     modprobe -v 

Installing depmod and modprobe: Install the "https://www.kernel.org/pub/linux/utils/kernel/module-init-tools/" package with your distro specific instructions.

Common Caveats:

1) If you are not seeing the changes in the new module: 

Probably you are re-inserting the old modules add "-v" to the modprobe and check the timestamps in the path from where modprobe is insmod-ing the module and your module in the kernel tree.  

Running "make modules_install" again should do, if all else fails then try copying manually.

2) If you are seeing a warning about module version check, while inserting the module: 

Probably you are using the above procedure for two dependent modules, like changing a function in cfg80211 and using that one in mac80211. 

Its better to use a directory which hold the directory of both of these, in the above case use "make modules  modules_install M=net/". For more details read Module Versioning Section in R#3.

References:

1) Linux kernel Makefile :-) 

2) Building External Modules

3) Kernel Docs: Building Linux kernel

Where is static variable used? A live Example

Till Date this was the question asked in many 'C' programming interviews asking for a live case of static variable, have come across a few cases but nothing was a solid one. Recently while going through spinlock code came across this function used in lockdep which has a solid example of static variable usage.

First, a little background on the context of lockdep and on below function. Lockdep is a lock monitoring utility in the kernel (an userspace version exists too) which keep tracks of all the locks allocated and order in which they are taken and throw necesary warnings if things are wrong.

Refer Introduction to LOCKDEP for details.

Now how do we track these numerous locks in the kernel, we wont. We only track the lock initialization per data structure, any number of the locks taken on the data structure doesnt matter unless we track one remaining show same behavior. With that we need to have a unique ID to identify a lock/data structure

a) If the lock is allocated statically (stack), then we use the address of the lock as the Unique ID (UID)

b) If the lock is allocated dynamically (heap), then we cant use the address as there will be too many addresses

So here's where the "static" comes in, in every lock init we use a "static" variable (its address a UID) so that across the initialization of the same structure the address remains the same.

       # define spin_lock_init(lock) \ 

       do { \ 

           static struct lockdep_type_key __key; \ 

           __spin_lock_init((lock), #lock, &__key); \ 

       } while (0)

Hmm, beauty isn't it. Finally a live use case for static variable, at least which i know :-)

Decrypting 802.11 packets: Secured 802.11 Environment

While testing secured 802.11 networks we face with a common problem of analyzing the data of the WLAN frames  using a sniffer, as they are encrypted.  Especially if you want to debug some higher layer protocol issues (DHCP, ICMP, ARP etc). 

We have some options to overcome this issue, either using a sniffer (or) using a console (in case of Linux), lets take a quick look at them.

For an End-user/Production Environment:

Most of the wireless capturing tools have support for decryption of the WLAN packets taking the credentials from the user.

Omnipeek: 

In "Tools>Decrypt WLAN packets" we can enter the credentials for each type of security and omnipeek will try to decode all encrypted packets in the trace.

Omnipeek WLAN Decryption Procedure Screenshot

Wireshark:  

In the "Preferences>Protcols>IEEE 802.11" there is an option to enter the Decryption Keys and also to enabled the decryption.

For both the tools, We can enter WEP-Key/WPA-Passphrase/WPA-PSK in the below formats:

   "Key examples:

     01:02:03:04:05 (40/64-bit WEP)    010203040506070809101111213 (104/128-bit WEP)    WPA/WPA2-PSK: Use this calcualtor and paste the psk. 

Wireshark WLAN Decryption Procedure Screenshot

But these decryption techniques are not reliable (especially omnipeek :-) and we need to have that costly license as well)  and can only be useful for post-processing of the packets. Live capture any one??

For a Engineering/Development Environment:

Instead if we have access to the console of the device (of course which runs Linux :-)), we can make use of tcpdump. It should be there in all android phones (at least the ones i have tested have it).

tcpdump:

tcpdump is the back end of wireshark it provides almost all main features of the wireshark. Dont go by the name, it not only supports TCP capturing but all protocols captures supported by wireshark.  As we know that the 802.11 encryption "mostly" (Except if you are using software encryption for some testing/initial development purposes) happens in the HW and tcpdump collects the packet before that all the packets can be seen in plain text even though encryption is enabled. 

For a quick check its good to prefer this, its my favorite way to debug the connection issues/traffic issues/DHCP issues in a secured 802.11 environment.

Usage for an Android Phone/Any Linux Device would be:

tcpdump -i <interface> -s 0 -w <Location to save the capture file, make sure it is writable> 

See tcpdump-man page for details on options.

Eg: 

1. tcpdump -i wlan0 -s 0 -w /data/test.pcap
2. adb pull /data/test.pcap .
3. open the pulled pcap file using wireshark and analyze.*

Note: By default tcpdump captures only 96 bytes of every packet, "-s 0" will set the limit 64K (practically the whole packet)

* If you are an omnipeek maniac  and have a omnipeek Pro and want to analyze these tcpdump captures using omnipeek, there's a new plugin called "Remote TCPDump Adapter Enterprise" where in through SSH it captures the packets using tcpdump but displays in the omnipeek in windows.

Happy Sniffing and Debugging.

Revisions:

1) Thanks to sunil pamula for finding the improper command for PSK keys. Its updated now. 8/11/2013.

An Introduction to Wireless Sniffing and Dissection

Lets try to understand and learn how sniffing wireless networks work and how to configure it in different platforms like Windows and Linux.

Our primary concentration will be Omnipeek and Wireshark the 2 most used Packet Capture and Analysis Tools predominant in wireless LAN industry.

1. Sniffing Wireless in Windows
2. Sniffing Wireless in Linux
3. Sniffing through Multiple Interfaces
4. Monitor Vs. Promiscuous Mode (Coming Soon)
5. Automate Sniffing Process (Coming Soon)

Wireless LAN and Linux Together

With the advent of opensource the development time has come down hugely and the quality has improved quickly. For all those involved with WLAN/Wi-Fi and Opensource lets take a look at the wlan architecture in linux based on opensource mac80211 framework.

Block diagram explaining linux WLAN architecture. Please see below for details.

User-Space:

Configuration: wpa_supplicant and hostapd:

All the applications which interact directly with the user lie here. They can a GUI/CLI based ones for eg. network manager in ubuntu/fedora distributions are UI based ones, but the core part are the CLI based ones eg. wpa_supplicant for controlling the STA part of it and hostapd for controlling the AP part.

Both are configuration file based along with their cli versions (wpa_cli, hostapd_cli) to send commands on the fly.

They support different features like SME, MLME, Security, Wifi-Direct (P2P),  AP and STA configurations.

Tools:

We also have tools to send commands to the driver directly to set some parameters such as channel, bandwidth, some custom commands etc.

The Bridge: User and Kernel

Now how the various applications in userspace communicate to the core entities in the kernel? Well, we have different approaches but all are based on different socket interfaces.

1. WExt ==> Generic Wireless Extensions: IOCTL Interface
2. NL80211==> Netlink Sockets
3. HostAP,==> Raw Packet Sockets
4. Chipset specific:

•               Atheros==> IOCTL Interface 
•               Prism,IPW etc.

Kernel Space

Configuration and UMAC

For opensource world the framework in kernel for WLAN is mac80211, it separates itself in to 2 kernel modules

• cfg80211.ko: Which handles all the configuration, user space interaction
• mac80211.ko: Protocol: Upper MAC , driver interaction.

Most of the features and management is handled by the mac80211 module with the help of lower MAC.

Lower MAC Drivers

Lowe MAC drivers act as a bridge between the UMAC and the chipset (Firmware and HW). They do all the device initialization, registration with OS, Bug Registration, Interrupts registration etc through the services provided by the linux kernel.

A well written driver follows these conventions

• Maintains a OS Independent Layer: Easy portability to different OSes.
• Maintains a UMAC Independent Layer: Easpy portability to different UMAC's: Proprietary, opensource, 3rd party etc.
• Bus Abstraction Layer: Maintains compatibility across different Physical Buses like PCI, PCIe, AHB, SDIO etc.

Chipset: Firmware and HW

The full 802.11 protocol functionality is implemented here.

The firmware which probably runs on a separate processor/micro-controller configures and controls the hardware and also interacts with the host(The driver) through a messaging interface specific to the chipset (control path)

The Data path generally involves a DMA controller in the HW which takes care of generating interrupts to the host processor and transferring packets to and from the Host to the HW queues.

References:

1. wpa_supplicant_hostapd_devel_doc
2. For details on these click userspace_configurations.
3. wpa_supplicant