Kernel Boot Delays During Root Filesystem Mounting and Initialization
The issue at hand involves a significant delay during the kernel boot process on an ARM Cortex-A55-based Rockchip RK3568 platform when booting from an SD card. The delay manifests primarily during the mounting of the root filesystem and the subsequent initialization of system services. The kernel log indicates that the system takes approximately 40 minutes to complete the boot process, with the majority of the time spent between the initialization of the root filesystem and the mounting of secondary filesystems such as /proc, /sys, and /dev/pts.
The kernel log reveals that the system successfully initializes the SD card controller (sdhci-dwcmshc) and detects the SD card (mmc0: new high speed SDHC card at address f5cf). However, the delay occurs after the root filesystem is mounted (EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode) and before the initialization of system services such as udev and syslogd. This suggests that the bottleneck is not in the hardware initialization but rather in the software stack responsible for mounting and initializing the root filesystem.
The kernel command line specifies the root filesystem as root=PARTUUID=614e0000-0000 rootfstype=ext4 rootwait, indicating that the root partition is identified by its PARTUUID and is formatted as ext4. The rootwait parameter ensures that the kernel waits for the root device to become available before proceeding with the boot process. Despite this, the system experiences a significant delay, which points to potential issues in the interaction between the kernel, the SD card controller, and the filesystem layer.
Potential Causes: SD Card Controller Configuration, Filesystem Corruption, and Kernel Boot Parameters
Several factors could contribute to the observed delay during the kernel boot process. These include misconfiguration of the SD card controller, filesystem corruption, and suboptimal kernel boot parameters.
SD Card Controller Configuration
The Rockchip RK3568 platform uses the sdhci-dwcmshc driver for its SD card controller. The kernel log indicates that the controller is initialized successfully, but the delay suggests that there may be issues with the clock configuration or the DMA settings. The SD card controller operates at a clock frequency of 50 MHz (clksd: fixed-clock: clock-frequency = <50000000>), which is within the expected range for high-speed SD cards. However, if the clock frequency is not stable or if there are issues with the DMA engine, the data transfer rate could be significantly reduced, leading to delays during the boot process.
Filesystem Corruption
Another potential cause of the delay is filesystem corruption on the SD card. The kernel log shows that the root filesystem is mounted successfully, but it does not provide detailed information about the integrity of the filesystem. If the filesystem is corrupted, the kernel may spend an excessive amount of time attempting to repair it or accessing damaged sectors. This could explain the long delay before the system proceeds with the initialization of secondary filesystems and services.
Kernel Boot Parameters
The kernel boot parameters may also play a role in the observed delay. The rootwait parameter ensures that the kernel waits for the root device to become available, but it does not specify a timeout. If the SD card is slow to initialize or if there are issues with the SD card controller, the kernel may wait indefinitely for the root device, leading to a prolonged boot time. Additionally, the loglevel=7 parameter increases the verbosity of the kernel log, which could exacerbate the delay by consuming additional CPU resources during the boot process.
Troubleshooting Steps: Optimizing SD Card Controller Settings, Verifying Filesystem Integrity, and Adjusting Kernel Boot Parameters
To address the issue, a systematic approach is required to identify and resolve the underlying cause of the delay. The following steps outline the recommended troubleshooting process.
Step 1: Optimize SD Card Controller Settings
The first step is to ensure that the SD card controller is configured correctly. This involves verifying the clock frequency and DMA settings in the device tree and kernel configuration.
Device Tree Configuration
The device tree for the RK3568 platform specifies the clock frequency for the SD card controller as 50 MHz (clksd: fixed-clock: clock-frequency = <50000000>). This value should be verified against the specifications of the SD card to ensure compatibility. If the SD card supports higher clock frequencies, the device tree should be updated accordingly.
Additionally, the DMA settings should be reviewed to ensure that the controller is using the most efficient transfer mode. The kernel log indicates that the controller is using ADMA 64-bit (mmc0: SDHCI controller on 8404000.sdhci [8404000.sdhci] using ADMA 64-bit), which is the preferred mode for high-speed data transfer. However, if there are issues with the DMA engine, the system may fall back to a slower transfer mode, leading to delays.
Kernel Configuration
The kernel configuration should be reviewed to ensure that the SD card controller driver (sdhci-dwcmshc) is enabled and configured correctly. The following kernel configuration options should be verified:
CONFIG_MMC_SDHCI=yCONFIG_MMC_SDHCI_PLTFM=yCONFIG_MMC_SDHCI_OF_DWCMSHC=y
These options ensure that the kernel includes support for the SD card controller and its platform-specific implementation. If any of these options are missing or disabled, the kernel may not be able to initialize the SD card controller correctly, leading to delays during the boot process.
Step 2: Verify Filesystem Integrity
The next step is to verify the integrity of the root filesystem on the SD card. This can be done using the fsck utility, which checks and repairs filesystem inconsistencies.
Running fsck on the Root Filesystem
To run fsck on the root filesystem, the system must be booted from an alternative root device, such as a USB drive or a network filesystem. Once the system is booted, the SD card can be mounted and checked using the following command:
fsck -f /dev/mmcblk0p3
The -f option forces fsck to check the filesystem even if it appears to be clean. If fsck detects any issues, it will attempt to repair them. After the filesystem has been checked and repaired, the system should be rebooted to verify that the delay has been resolved.
Step 3: Adjust Kernel Boot Parameters
The final step is to adjust the kernel boot parameters to optimize the boot process. This involves modifying the rootwait parameter and reducing the verbosity of the kernel log.
Modifying the rootwait Parameter
The rootwait parameter ensures that the kernel waits for the root device to become available, but it does not specify a timeout. To prevent the kernel from waiting indefinitely, a timeout can be added using the rootdelay parameter. For example:
root=PARTUUID=614e0000-0000 rootfstype=ext4 rootwait rootdelay=10
This parameter instructs the kernel to wait for up to 10 seconds for the root device to become available. If the root device is not available within this time, the kernel will proceed with the boot process, potentially reducing the overall boot time.
Reducing Kernel Log Verbosity
The loglevel=7 parameter increases the verbosity of the kernel log, which can consume additional CPU resources during the boot process. To reduce the verbosity, the loglevel parameter can be set to a lower value, such as 3:
loglevel=3 earlycon=cdns,mmio32,0x02030000 swiotlb=1 console=ttyPS0,115200 rw root=PARTUUID=614e0000-0000 rootfstype=ext4 rootwait
This change reduces the amount of logging during the boot process, which can help to minimize the delay.
Conclusion
The slow kernel boot issue on the ARM Cortex-A55 RK3568 platform is likely caused by a combination of factors, including suboptimal SD card controller settings, filesystem corruption, and inefficient kernel boot parameters. By optimizing the SD card controller configuration, verifying the integrity of the root filesystem, and adjusting the kernel boot parameters, the boot time can be significantly reduced. These steps provide a comprehensive approach to diagnosing and resolving the issue, ensuring a faster and more reliable boot process.
