Kernel changes: Back and Forth?

I have been a Linux user (Ubuntu, Mint, Debian) for a few years and recently installed Neon on a spare partition and am getting comfy with it.
I noticed something peculiar that is not a problem but makes me curious.
Usually, Neon kernels follow the rest of the Ubuntu updates. Last June, though, Discover updated it to 5.19 while Ubuntu was still at 5.15. Fine, no problem, and it worked well. Then, booting another day and the updates rolled the kernel back to 5.15.
Now, just a couple of weeks ago, Discover updated to 6.2 and kept it there for two weeks. Today, the update process AGAIN rolled it back to 5.15.
Again, I do not see any change in any program’s behavior, so there is no problem, other than being curious.

1 Like

Are you certain that the other kernels aren’t still installed? On my Mint VM, I have the 5.15 and 6.2 (which started as 5.19) kernels; in fact, the 5.15 branch was updated today, but it left the 6.2 branch alone.

Are you running Grub or systemd-boot? Do you have a boot menu or is it hidden?

I am using legacy BIOS and grub. In the grub menu, there is the “Advanced options” and a long list of other choices. But running the edit command does not tell me the kernel. I tried that when I first noticed this issue, but could not find the kernel in those listings.
So if the kernels are still there, how would I choose from among 10 entries under “Advanced option”?
BTW, my memory fails me, but I THINK that I removed the headers when I ran autoremove. I won’t do that again.

Autoremove should not remove packages like that. What does

awk -F[\'\'|\"\"] '/^menuentry/ {print $2}' /boot/grub/grub.cfg

return?

awk: line 0: regular expression compile failed (bad class – , [^] or [)
[‘’
“”]: command not found

But ls /boot produces

System.map-5.15.0-79-generic efi initrd.img-6.2.0-26-generic memtest86+_multiboot.bin vmlinuz.old
System.map-6.2.0-26-generic grub initrd.img.old vmlinuz
config-5.15.0-79-generic initrd.img memtest86+.bin vmlinuz-5.15.0-79-generic
config-6.2.0-26-generic initrd.img-5.15.0-79-generic memtest86+.elf vmlinuz-6.2.0-26-generic

Hmmm…I copied this command directly from a script that I run on Debian/Ubuntu-based distros; even runs on my Arch-based systems (after adding sudo).

I found the problem. I use fish shell, the errors you’re getting are because of bash. Try this instead:

awk -F[\'\'\|\"\"] '/^menuentry/ {print $2}' /boot/grub/grub.cfg

It is a simplified version of the grub menu:
Mint
Ubuntu, with Linux 5.15.0-79-generic
Ubuntu, with Linux 5.15.0-79-generic (recovery mode)
Ubuntu, with Linux 5.15.0-78-generic
Ubuntu, with Linux 5.15.0-78-generic (recovery mode)
Ubuntu, with Linux 5.15.0-41-generic
Ubuntu, with Linux 5.15.0-41-generic (recovery mode)
KDE neon 5.27 (22.04) (on /dev/sda3)
Ubuntu 22.04.3 LTS (22.04) (on /dev/sdb1)
Windows 10 (on /dev/sdb2)

Those entries for “Ubuntu, with Linux” all refer to Mint, sda1
That Ubuntu on sdb1 is Ubuntu Mate

Neon will keep a version of 5.15, 5.19, and a version of 6.2 and rotate out older ones. You will see updates for the older kernel in Discover on a regular basis, even though you are not using it. It is there should you need to.

Type “uname -r” in a terminal to display the running kernel version. It should be 6.2 in the current Neon.

Well, you named the "kernel " of my question. I use neofetch or uname -a. In May, it was 5.15. In June it was 5.19. In July it was back to 5.15. In early August it was 6.2, and now it is back to 5.15
What the ???

I would guess Mint is rotating it, not Neon. You may need to check your grub.conf file in Mint to see if is booting the right kernel. I suspect its pointing to 5.15. You should be able to find the menuentry about halfway down the file.

menuentry Neon…

linux /boot/vmlinuz-5.15.0-79-generic

have you installed grub customizer?

or been doing any work on your grub setup?

i know there is a setting in grub where it will pick an entry from the list and other where it will pick last saved and these two features could be walking on each other.

Well, great tips here and I can report some info but no solution yet. And a reminder, I am OK with the older kernel, just curious what’s going on.

Yes, I have grub_customizer, and I found the grub entry with kernel 6.2
The main entry reads:

insmod part_msdos
insmod ext2
set root=‘hd0,msdos3’
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3 --hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3 257f0255-786a-4160-ac57-b953d367efde
else
search --no-floppy --fs-uuid --set=root 257f0255-786a-4160-ac57-b953d367efde
fi
linux /boot/vmlinuz root=/dev/sda3
initrd /boot/initrd.img

The entry with 6.2 reads:

insmod part_msdos
insmod ext2
set root=‘hd0,msdos3’
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3 --hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3 257f0255-786a-4160-ac57-b953d367efde
else
search --no-floppy --fs-uuid --set=root 257f0255-786a-4160-ac57-b953d367efde
fi
linux /boot/vmlinuz root=/dev/sda3
initrd /boot/initrd.img-6.2.0-26-generic

Only the last line is different.
When I tried booting to it, the bootup failed at the USB mouse and reported that device /dev had failed. (Sorry, it was a huge list of messages, that is the only one I could see as relevant.)

When I went to the /boot menu (in the 3rd partition but navigated there from Mint, 1st partition), the icon for the 6.2 kernel had a big X through it.

So I STRONGLY suspect that when I ran autoremove after updating in KDE, it did something to the newer kernels that made the system revert to 5.15
Curiouser and curiouser

grub customizer takes your entire grub structure and archives it, replacing it with it’s own structure (based on what is is unclear)… so now when you have a kernel update your grub cannot adjust properly to the new kernel, nor can it prune properly the older kernels.

i suspect the normal grub behavior when a new kernel shows up is trying to operate on a mangled version grub put in place by grub customizer and that’s why you are getting weird results.

unfortunately there does not seem to be any easy way uninstall grub customizer short of reinstalling grub.

Am I correct, then, in understanding @skyfishgoo to be saying that the fault lies with grub_customizer, that a solution now would be more difficult than it is worth? Thus, I am best off waiting for an updated 6.2 kernel, which will supersede 5.15?
I’m cool with that. Is this a “solution”?

if you want to be sure which kernel you are booting into you can select an item from the grub menu and hit E, it will show you the command you are about to execute and you can read the kernel version in the text.

if that version doesn’t match what is showing on the grub screen then, that is the fault of grub customizer, yes.

more info

1 Like

Just in case you don’t know about this…

Those should be booting to symbolic links in /boot. To see what is used, run ls -l /boot

The links are supposed to be maintained by the software updater, f.ex. APT. Using them means a grub entry doesn’t have to change for every new kernel. (IMO the update-grub machinery is awfully complex and very fragile. I don’t let it touch the boot directly on my systems, preferring to maintain by hand a vastly simpler grub.cfg; I can go for years without having to touch it.)

You are correct that I did not know this before.
Here is the readout of your command launched from sda3 (the KDE partition)

-rw-r–r-- 1 root root 261881 Jul 10 06:33 config-5.15.0-79-generic
-rw-r–r-- 1 root root 275465 Jul 13 06:22 config-6.2.0-26-generic
drwxr-xr-x 2 root root 4096 Apr 28 13:16 efi
drwxr-xr-x 3 root root 4096 Apr 20 00:36 grub
lrwxrwxrwx 1 root root 28 Aug 14 18:24 initrd.img → initrd.img-5.15.0-79-generic
-rw------- 1 root root 117420763 Aug 24 19:23 initrd.img-5.15.0-79-generic
-rw------- 1 root root 136191944 Aug 24 19:22 initrd.img-6.2.0-26-generic
lrwxrwxrwx 1 root root 27 Aug 14 18:24 initrd.img.old → initrd.img-6.2.0-26-generic
-rw-r–r-- 1 root root 182800 Feb 6 2022 memtest86+.bin
-rw-r–r-- 1 root root 184476 Feb 6 2022 memtest86+.elf
-rw-r–r-- 1 root root 184980 Feb 6 2022 memtest86+_multiboot.bin
-rw------- 1 root root 6271933 Jul 10 06:33 System.map-5.15.0-79-generic
-rw------- 1 root root 7963097 Jul 13 06:22 System.map-6.2.0-26-generic
lrwxrwxrwx 1 root root 25 Aug 14 18:24 vmlinuz → vmlinuz-5.15.0-79-generic
-rw------- 1 root root 11613672 Jul 10 08:22 vmlinuz-5.15.0-79-generic
-rw------- 1 root root 13770312 Jul 13 08:31 vmlinuz-6.2.0-26-generic
lrwxrwxrwx 1 root root 24 Aug 14 18:24 vmlinuz.old → vmlinuz-6.2.0-26-generic

If I rename initrd.img as old and initrd.img.old as initrd.img, and then the same thing to vmlinuz, would that remove the “back and forth” issue? Can I redirect grub to the more updated kernel symlink?

Vanilla grub doesn’t use the symlinks. I have little idea what has pointed yours at 5.15. On my Ubuntu when there’s a new kernel APT updates the links. The .old links are pointed at the previous kernel.

If you want to make changes to the links, you delete and recreate them; for example:

rm vmlinuz
ln -s vmlinuz-6.2.0-26-generic vmlinuz

You could use the file /boot/grub/custom.cfg to add an entry or entries that you have control over. The vanilla grub sources that if it’s present.