A compilation of several resources and paying it forward to the community.

1. Embedded Linux Network
Device Driver Development

2. Rights to Copy
 License: Creative Commons Attribution - Share Alike 3.0
 You are free:
 to copy, distribute, display, and perform the work
 to make derivative works
 to make commercial use of the work
 Under the following conditions:
 Attribution – Derived from original work of Free Electrons
 Share Alike
 For any reuse or distribution, you must make clear to others the license terms
of this work.
September 7, 2017
Embedded Linux Network Device Driver
Development
2

3. References
 www.free-electrons.com
 Essential Linux Device Drivers
 Linux Device Drivers
September 7, 2017
Embedded Linux Network Device Driver
Development
3

4. Objectives
 Understanding the structure of ethernet and WIFI device drivers
 Developing Linux network device drivers
Embedded Linux Network Device Driver
Development
4 September 7, 2017

5. Prerequisites
 Solid C programming
 Knowledge of Linux commands is a plus
 Knowledge of networking
 Embedded Linux Kernel and Driver Development training or equivalent
Embedded Linux Network Device Driver
Development
5 September 7, 2017

6. Notes
 Ask any time.
 Turn your cell silent.
Embedded Linux Network Device Driver
Development
6 September 7, 2017

7. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
7

8. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
8

9. OSI Model and Linux Kernel
September 7, 2017
Embedded Linux Network Device Driver
Development
9
User Space
Kernel Network
Stack
Device Driver

10. Network Device Model
September 7, 2017
Embedded Linux Network Device Driver
Development
10

11. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
11

12. Socket Buffer (sk_buff)
 struct sk_buff (include/linux/sk_buff.h) represents a network packet
 Support data encapsulation/decapsulation through protocol layers
 In addition to data, sk_buff maintains:
 head, the start of the packet
 data, the start of the packet payload
 tail, the end of the packet payload
 end, the end of the packet
 len, the amount of data of the packet
 These fields are updated when the packet goes through the protocol
layers
September 7, 2017
Embedded Linux Network Device Driver
Development
12

13. SKB Operations
Allocation
 By dev_alloc_skb() function
 Can be done in ISR
 On Ethernet, allocated size = packet
length + 2
 To word align IP header
 IP header = 14 bytes
Reservation
 By skb_reserve() function
 Skipping NET_IP_ALIGN padding
bytes
September 7, 2017
Embedded Linux Network Device Driver
Development
13

14. SKB Operations cont’d
Data Copying
 From DMA buffer to SKB during
reception for example
 OR SKB to DMA buffer during
transmission (may be)
Pointers Update
 After copying payload
September 7, 2017
Embedded Linux Network Device Driver
Development
14

15. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
15

16. Network Device
 struct net_device (include/linux/netdevice.h) represents a network
interface
 Allocation takes place with alloc_netdev() function
 Size of private data is passed as argument
 Pointer to these private data can be read in net_device->priv
 Use alloc_etherdev() for ethernet interfaces
 Use alloc_ieee80211() for WIFI interfaces
September 7, 2017
Embedded Linux Network Device Driver
Development
16

17. Network Device cont’d
 Registration with register_netdev() function
 Unregistration with unregister_netdev() function
 Liberation with free_netdev() function
September 7, 2017
Embedded Linux Network Device Driver
Development
17

18. Network Device Operations
 Defined by struct net_device_ops (include/linux/netdevice.h)
 Set the netdev_ops field in the struct net_device structure to point to the
struct net_device_ops structure
September 7, 2017
Embedded Linux Network Device Driver
Development
18
Operation Description
ndo_open() Called when network interface uped
ndo_close() Called when network interface downed
ndo_start_xmit() Start packet transmission
ndo_get_stats() Get statistics
ndo_do_ioctl() Implement device specific operations
ndo_set_rx_mode() Select promiscuous, multicast, etc
ndo_set_mac_address() Set MAC address
ndo_set_multicast_list() Set multicast filters

19. Network Device Operations cont’d
 Sample operations are:
September 7, 2017
Embedded Linux Network Device Driver
Development
19
Operation Description
ndo_open() Called when network interface uped
ndo_close() Called when network interface downed
ndo_start_xmit() Start packet transmission
ndo_get_stats() Get statistics
ndo_do_ioctl() Implement device specific operations
ndo_set_rx_mode() Select promiscuous, multicast, etc
ndo_set_mac_address() Set MAC address
ndo_set_multicast_list() Set multicast filters
ndo_tx_timeout() Reset unresponsive network interface

20. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
20

21. Flow Control Utility Functions
 include/linux/netdevice.h
September 7, 2017
Embedded Linux Network Device Driver
Development
21
Operation Description
netif_start_queue() Tells kernel that driver is ready to send packets
netif_stop_queue() Tells kernel to stop sending packets (@cleanup or congestion)
netif_queue_stopped() Tells whether queue is stopped or not
netif_wake_queue() Wakes up a queue after a netif_stop_queue()
netif_rx() Gives SKB to kernel

22. Transmission
 Kernel calls ndo_start_xmit() with SKB as argument
 ndo_start_xmit():
1. Sets up DMA buffers and other HW mechanisms
2. Starts transmission
3. Can stop queueing if no more free DMA buffers available using
netif_stop_queue()
4. Returns NETDEV_TX_OK or NETDEV_TX_BUSY
 When N packets have been sent, an interrupt is raised and the driver
should:
1. Acknowledging interrupt
2. Freeing used DMA buffers
3. Freeing SKB with dev_kfree_skb_irq()
4. If the queue was stopped, start it again
September 7, 2017
Embedded Linux Network Device Driver
Development
22

23. Reception: Non-NAPI
 Reception is via an interrupt that should:
1. Allocate an SKB
2. Reserve NET_IP_ALIGN padding bytes
3. Copy packet from DMA buffers to SKB
4. Update SKB pointers
5. Update the skb->protocol field with eth_type_trans(skb, netdevice)
6. Give SKB to kernel by netif_rx()
 netif_rx() use NET_RX_SOFTIRQ to offload the work of posting received data
packets to protocol layers
 Nice and simple
 @high traffic, interrupt rate is high
 Solution: switch to polled mode when interrupt rate is too high (NAPI)
September 7, 2017
Embedded Linux Network Device Driver
Development
23

24. Reception: NAPI
 In network interface private structure, add a struct napi_struct
(include/linux/netdevice.h)
 @ driver initialization, register the NAPI poll operation
 dev: network interface
 &lp->napi: struct napi_struct
 r6040_poll: NAPI poll operation
 64: weight that represents the importance of network interface
September 7, 2017
Embedded Linux Network Device Driver
Development
24

25. Reception: NAPI cont’d
Rx ISR
 Disables Rx interrupt and switch to
polled mode
if (napi_schedule_prep(&lp->napi)) {
/* Disable reception interrupts */
__napi_schedule(&lp->napi);
}
 Kernel calls our poll() operation
regularly
Poll Function
 static int r6040_poll(struct
napi_struct *napi, int budget)
 Receives at most budget packets and
pushes them to the network stack
using netif_receive_skb()
 If less than budget packets have been
received, switch back to interrupt
mode using napi_complete(&lp-
>napi) and reenable interrupts
 Must return the number of packets
received
September 7, 2017
Embedded Linux Network Device Driver
Development
25

26. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
26

27. Connection with PHY
 MAC and PHY are connected using a MII or RMII interface
 This interface contains two wires used for the MDIO bus
 Driver needs to communicate with the PHY to get link information (up,
down, speed, full or half duplex) and configure the MAC accordingly
September 7, 2017
Embedded Linux Network Device Driver
Development
27

28. PHY in Kernel
 Kernel has a framework (drivers/net/phy/) that
 Exposes an API to communicate with PHY
 Allows to implement PHY drivers
 Implements a basic generic PHY driver that works with all PHY
 See Documentation/networking/phy.txt
September 7, 2017
Embedded Linux Network Device Driver
Development
28

29. MDIO Bus Initialization
 Driver creates a MDIO bus struct mii_bus (include/linux/mii.h) to tells PHY
infrastructure how to communicate with the PHY
 mdio_read() and mdio_write are HW specific and must be implemented
by the driver
September 7, 2017
Embedded Linux Network Device Driver
Development
29

30. MDIO Bus Initialization cont’d
 The ->irq[] array must be allocated and initialized
 To use polling, set the values to PHY_POLL
Lp->mii_if->irq = kmalloc(sizeof(int)*PHY_MAX_ADDR, GFP_KERNEL);
for (i = 0; i < PHY_MAX_ADDR; i++)
lp->mii_if>irq[i] = PHY_POLL;
 Finally, register the MDIO bus
 This will scan the bus for PHYs and fill the mii_if->phy_map[] array with the
result
mdiobus_register(bp->mii_if);
September 7, 2017
Embedded Linux Network Device Driver
Development
30

31. Connecting with PHY
 mdiobus_register() function filled the mii_if->phy_map[] array with struct
phy_device * pointers
 Appropriate PHY (usually, only one is detected) must be selected
 Connecting to the PHY allows to register a callback that will be called
when the link changes :
 interface is usually PHY_INTERFACE_MODE_MII or
PHY_INTERFACE_MODE_RMII
September 7, 2017
Embedded Linux Network Device Driver
Development
31

32. Updating MAC Capabilities
 MAC and PHY might have different capabilities
 PHY handling Gigabit speed, but not MAC
 Driver is responsible for updating phydev->advertise and phydev-
>supported to remove any PHY capability that the MAC doesn't support
September 7, 2017
Embedded Linux Network Device Driver
Development
32

33. Handling Link Changes
 The callback that handle link changes should have the following prototype:
void foo_handle_link_change(struct net_device *dev)
 It must check the duplex, speed and link fields of the struct phy_device
structure, and update Ethernet controller configuration accordingly
 duplex is either DUPLEX_HALF or DUPLEX_FULL
 speed is either SPEED_10, SPEED_100, SPEED_1000, SPEED_2500 or
SPEED_10000
 link is a Boolean
September 7, 2017
Embedded Linux Network Device Driver
Development
33

34. Starting and stopping the PHY
 After set up, to make the PHY driver poll regularly the PHY hardware, one
must start it with phy_start() functions
 And stop it using phy_stop(), when no longer needed
September 7, 2017
Embedded Linux Network Device Driver
Development
34

35. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
35

36. Buffers Management
 During open(), driver pre-allocates needed DMA descriptos needed for
transmission and reception
 When totally utilized, the driver tells the kernel to stop sending packets to
the driver
 SKB are allocated and freed as per need
September 7, 2017
Embedded Linux Network Device Driver
Development
36

37. Concurrency
 Access protection needed in the face of multiple execution threads:
 Transmit thread
 Receive thread
 Transmit-complete interrupt
 Receive interrupt
 NAPI polling
September 7, 2017
Embedded Linux Network Device Driver
Development
37

38. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
38

39. Network Benchmarking
 Netperf (free from www.netperf.org ) can set up complex TCP/UDP
connection scenarios to control C/C’s such as protocol parameters,
number of simultaneous sessions, and size of data blocks.
 Benchmarking = Comparing resulting throughput with maximum practical
bandwidth
 Factors affecting throughput:
 Driver implementation
 Protocol used
September 7, 2017
Embedded Linux Network Device Driver
Development
39

40. Driver Performance
 Execution time
 Considering usage of DMA for large packets
 Offloading CPU by moving functionalities to hardware (TCP checksum)
 Taking into consideration upper layers behavior
September 7, 2017
Embedded Linux Network Device Driver
Development
40

41. Protocol Performance
 TCP window size = data that can be transmitted before receiving ACK
 Fast network interfaces + small window size = TCP sitting idle waiting for ACK
 In UDP, the window size is not relevant as there is no ACK
 Packet loss in TCP and UDP
 Mapping protocol block sizes to MTU sizes
September 7, 2017
Embedded Linux Network Device Driver
Development
41

42. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
42

43. Ethtool (include/linux/ethtool.h)
 ethtool is a user space tool that query low-level information from an
Ethernet interface and to modify its configuration
 @ driver, a struct ethtool_ops structure can be declared and connected to
the struct net_device using the ethtool_ops field
 List of operations: get_settings(), set_settings(), get_drvinfo(), get_wol(),
set_wol(), get_link(), get_eeprom(), set_eeprom(), get_tso(), set_tso(),
get_flags(), set_flags(), etc.
 Some of these operations can be implemented using the PHY interface
(phy_ethtool_gset(), phy_ethtool_sset()) or using generic
operations(ethtool_op_get_link() for example)
September 7, 2017
Embedded Linux Network Device Driver
Development
43

44. Statistics
 The driver is responsible for keeping statistics up to date about the
number of packets/bytes received/transmitted, the number of errors, of
collisions, etc.
 To expose these information, the driver must implement a get_stats()
operation, with the following prototype struct net_device_stats
*foo_get_stats(struct net_device *dev);
 The net_device_stats structure must be filled by the driver
 It contains fields such as rx_packets, tx_packets, rx_bytes, tx_bytes, rx_errors,
tx_errors, rx_dropped, tx_dropped, multicast, collisions, etc
September 7, 2017
Embedded Linux Network Device Driver
Development
44

45. Power Management
 suspend() function should:
 Call netif_device_detach()
 Do the hardware dependent operations to suspend the devices (like disable
the clocks)
 The resume() function should:
 Do the hardware dependent operations (like enable the clocks)
 Call netif_device_attach()
September 7, 2017
Embedded Linux Network Device Driver
Development
45

46. Lab: Training Setup
September 7, 2017
Embedded Linux Network Device Driver
Development
46

47. Outline
 Introduction
 Socket Buffers
 Network Devices
 Communication with Network Protocol
 Communication with PHY
 Buffer Management and Concurrency Issues
 Network Throughput
 Other Network Driver Considerations
 Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
47

48. Ethernet vs. WIFI
Ethernet
 CSMA/CD
 Frames are not acknowledged
WIFI
 CSMA/CA
 Frames are acknowledged
 Uses WEP for enhanced security
 2 modes of operation:
 Ad-hoc
 Infrastructure
September 7, 2017
Embedded Linux Network Device Driver
Development
48

49. Configuring WIFI Drivers
 The Wireless Extensions project defines a generic Linux API to configure
WLAN device drivers in a device independent manner
 It also provides a set of common tools to set and access information from
WLAN drivers
 Individual drivers must implement support for Wireless Extensions to
connect themselves with the common interface and, hence, with the tools
September 7, 2017
Embedded Linux Network Device Driver
Development
49

50. Talking to WIFI Drivers
 Using iwconfig utility
 To glue your driver to iwconfig, you need to implement prescribed functions
corresponding to commands that set parameters such as ESSID and WEP keys
 Using iwpriv utility
 To use iwpriv over your driver, define private ioctls relevant to your hardware
and implement the corresponding handler functions
 Using /proc/net/wireless to get driver statistics
 For this, implement the get_wireless_stats() function in your driver, in addition
to the get_stats() function
 WLAN drivers tie these three pieces of information inside a structure
called iw_handler_def, defined in include/net/iw_handler.h
September 7, 2017
Embedded Linux Network Device Driver
Development
50

51. Ethernet/WIFI Famous Device Drivers
September 7, 2017
Embedded Linux Network Device Driver
Development
51
Device(s) Driver Location in Kernel Tree
Intersil Prism2 WLAN Compact Flash Card /drivers/net/wireless/intersil/orinoco
Intel Pro/Wireless Mini PCI (and PCIe Mini) /drivers/net/wireless/intel/ipw2x00
Atmel WLAN USB /drivers/net/wireless/atmel
Intel PRO/1000 /drivers/net/ethernet/intel/e1000

52. Ethernet Driver Example
September 7, 2017
Embedded Linux Network Device Driver
Development
52

53. Lab: Ethernet Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
53

54. WIFI Driver Example
September 7, 2017
Embedded Linux Network Device Driver
Development
54

55. Lab: WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
55

56.  To contact us:
 https://www.facebook.com/groups/EmbeddedSystemsTraining/
 www.swift-act.com
 training@swift-act.com
 (+2)0122-3600-207
September 7, 2017
Embedded Linux Network Device Driver
Development
56