Discussion on baloo defaults

[sanmayjoshi]

Tldr; As of now baloo, by default, indexes the files names and file contents. How might the current defaults compare with a scenario where by default only the file names are set to index?

It’s been a few weeks since I got Fedora KDE installed, and it’s been really nice exp so far. A few days back while I was working, the fans suddenly started spinning at high rpm, I could also notice slight drop in performance. Upon search I found out about baloo. So I searched in the processes, and it was certainly baloo (probably indexing).

Upon a little search on baloo, I was really surprised to find out the number of users who disable baloo the first thing on a fresh KDE install. It’s a really nice software that may have suffered due to demanding defaults, imo.

By default, baloo is set to scan and index the file names and their contents. I was wondering whether having baloo index file contents causes more performance draw as compared to just the file names. Personally, I haven’t had a moment where I needed the to search the contents of a file via (say) krunner. It’s almost always just the file names.

Just wanted to take this opportunity to ask the community and have a discussion on whether the current baloo config may use a slightly lightweight approach with regards to indexing, by default.

PS. I have disabled baloo for now :cough:

[skyfishgoo]

in my experience it’s not the contents that drive baloo over the edge (tho it does make the index bigger)

no, it’s having to index hidden files (which is off by default on my distro) or having to index folders that contain stuff you are never going to search thru (like ~/snap).

[claydoh]

@skyfishgoo

or having to index folders that contain stuff you are never going to search thru (like ~/snap

Yes, this one, for sure.

Once anyone enables hidden files/dirs, then you have a (self-created) mess, just from browser data alone, let alone other items like akonadi database files, flatpaks installed with —user, etc.

The default settings are fine, imo, and the default list of ignored files is sane, though I imagine other things could be added:

exclude filters=*~,*.part,*.o,*.la,*.lo,*.loT,*.moc,moc_*.cpp,qrc_*.cpp,ui_*.h,cmake_install.cmake,CMakeCache.txt,CTestTestfile.cmake,libtool,config.status,confdefs.h,autom4te,conftest,confstat,Makefile.am,*.gcode,.ninja_deps,.ninja_log,build.ninja,*.csproj,*.m4,*.rej,*.gmo,*.pc,*.omf,*.aux,*.tmp,*.po,*.vm*,*.nvram,*.rcore,*.swp,*.swap,lzo,litmain.sh,*.orig,.histfile.*,.xsession-errors*,*.map,*.so,*.a,*.db,*.qrc,*.ini,*.init,*.img,*.vdi,*.vbox*,vbox.log,*.qcow2,*.vmdk,*.vhd,*.vhdx,*.sql,*.sql.gz,*.ytdl,*.tfstate*,*.class,*.pyc,*.pyo,*.elc,*.qmlc,*.jsc,*.fastq,*.fq,*.gb,*.fasta,*.fna,*.gbff,*.faa,po,CVS,.svn,.git,_darcs,.bzr,.hg,CMakeFiles,CMakeTmp,CMakeTmpQmake,.moc,.obj,.pch,.uic,.npm,.yarn,.yarn-cache,__pycache__,node_modules,node_packages,nbproject,.terraform,.venv,venv,core-dumps,lost+found

I think a simple warning when users enable hiddden files/dirs that this could have a very negative effect might be enough, maybe suggesting some places to consider adding as non-indexed locations.

[sanmayjoshi]

@skyfishgoo

it’s having to index hidden files (which is off by default on my distro)

Yep it’s off on my machine as well (by default). I never enabled it.

@skyfishgoo

in my experience it’s not the contents that drive baloo over the edge (tho it does make the index bigger)

Hmm. But wouldn’t having larger index mean that baloo runs for longer than it would have, had it not been set to index the contents as well? (not sure if I am getting it 100% right). I imagine this becoming a genuine issue for those who have significantly larger storage (and consequently the amount of files) on their machine.

@claydoh

exclude filters=*~,*.part,*.o,*.la,*.lo,*.loT,*.moc,moc_*.cpp,qrc_*.cpp,ui_*.h,cmake_install.cmake,CMakeCache.txt,CTestTestfile.cmake,libtool,config.status,confdefs.h,autom4te,conftest,confstat,Makefile.am,*.gcode,.ninja_deps,.ninja_log,build.ninja,*.csproj,*.m4,*.rej,*.gmo,*.pc,*.omf,*.aux,*.tmp,*.po,*.vm*,*.nvram,*.rcore,*.swp,*.swap,lzo,litmain.sh,*.orig,.histfile.*,.xsession-errors*,*.map,*.so,*.a,*.db,*.qrc,*.ini,*.init,*.img,*.vdi,*.vbox*,vbox.log,*.qcow2,*.vmdk,*.vhd,*.vhdx,*.sql,*.sql.gz,*.ytdl,*.tfstate*,*.class,*.pyc,*.pyo,*.elc,*.qmlc,*.jsc,*.fastq,*.fq,*.gb,*.fasta,*.fna,*.gbff,*.faa,po,CVS,.svn,.git,_darcs,.bzr,.hg,CMakeFiles,CMakeTmp,CMakeTmpQmake,.moc,.obj,.pch,.uic,.npm,.yarn,.yarn-cache,__pycache__,node_modules,node_packages,nbproject,.terraform,.venv,venv,core-dumps,lost+found

Nice.

@claydoh

I think a simple warning when users enable hiddden files/dirs that this could have a very negative effect might be enough, maybe suggesting some places to consider adding as non-indexed locations.

+1

[claydoh]

I enable hidden files on some systems, becasue my user configs are something I search through.

I also add external drives as appropriate, and do exclude things like my browser data, and the like. I forgot once, and had three different browsers running, because I do, and had just set up Kontact with multiple resources and 4 email accounts. THAT reminded me pretty quickly. Lots of changes happening, and constantly. It made my old thin client’s fan speed up regularly, and it is a noisy one. No rise in temps or anything, just a fan that needs cleaning or replacing.

My baloo index was multiple Gb in size lol. I added some entries to the not-index list, and now it is quieter, and ~650Mb.

[ben2talk]

@sanmayjoshi

Upon a little search on baloo

You should always be aware that ‘doing a little search’ is most likely to bring up content from people complaining… The rest of us have little or nothing to say about it.

I haven’t touched Baloo for years.

I remember I did have some serious issues with it, and joined the mob posting and complaining… but whilst many of them disabled it I simply realised that it was MY fault, and fixed the issue.

I am completely unaware of any effect of Baloo on my potato computer until I start typing in menus etc…

The difference in performance… after the initial indexing I’d say minimal unless you decide to import a few GB of files to a folder that’s included in the index.

And even then, the indexer doesn’t consume CPU when I’m busy.

I’m very careful to only include important folders (.config is one)… though I have no issues with having added some large storage directories (mostly media, a few thousand ebooks; plus TV/Movies) with no problem (and those are on ancient rusty disks).

[sanmayjoshi]

@ben2talk

You should always be aware that ‘doing a little search’ is most likely to bring up content from people complaining…

So true…lol

@ben2talk

The difference in performance… after the initial indexing I’d say minimal unless you decide to import a few GB of files to a folder that’s included in the index.

And even then, the indexer doesn’t consume CPU when I’m busy.

Any thoughts on when gaming? From what I understand, by default ~/Documents is there under the list of folders to index. And some game launchers may primarily use ~/Documents as the directory to keep the assets in (Heroic Launcher comes to mind). So, when playing a game, is there a chance that baloo might kick off the indexing? I noticed once or twice that baloo was running when gaming (but that may not be a definitive indication imo). I am not sure but, I believe that games mostly run in memory and rarely perform operations in the storage disk, but I may be wrong in which case baloo kicking off indexing when gaming might make sense. Nevertheless, is there an option to automatically temporarily disable baloo when gaming? (Does gamemode do this already?)

Was that too may questions? Oops, I added another one

[krake]

@sanmayjoshi

And some game launchers may primarily use ~/Documents as the directory to keep the assets in (Heroic Launcher comes to mind)

I would consider this a serious misuse of the default document location.

There is a reason we have standard locations for app internal data like $HOME/.local/share

@sanmayjoshi

is there an option to automatically temporarily disable baloo when gaming?

I think it is possible via D-Bus and balooctl so a game launcher could use either.

Although I would no expect a lot of file being created or modified while a game is running.
New emails being downloaded in the background maybe.

[ben2talk]

@sanmayjoshi

From what I understand, by default ~/Documents

I have already two Documents folders… I have one (Dropbox) mounted as ~/Dropbox/Documents and a local ‘archive’ which is ~/mnt/T4/Documents… each with a link to the other.

So feasibly, you can let your toxic game pollute ~/Documents and remove that from Baloo indexing… basically butcher your system to avoid issues caused by the nasty behaviour of the game.

My issue was with Plex - it generates a truly stupendous number of ridiculously long named files and folders and caches and all kinds of stuff that caused Timeshift to choke after attempting for 20 minutes simply to build a list of the 2~3 million files.

Both Timeshift and Baloo can be great tools when respected and used well.

[sanmayjoshi]

@krake

I would consider this a serious misuse of the default document location.

Correction: Heroic creates a new ~/Games folder and stores its things there. My earlier statement was out of ignorance.

[darryl]

Defaults are only part of the problem. Baloo seems to need a lot of work.

Baloo also doesn’t communicate with System Settings properly anymore.

Just checked, in system settings it reports as: Not running; 0% complete, but in System Monitor it says: baloo_file_extractor 894 MiB, and balooctl6 monitor - says: File indexer is running
Idle

In the meantime I reverted to also using Recoll, it doesn’t get stuck like baloo, and even with realtime indexing and no excluded folders it uses less memory.

[jamesfrederik]

Adding Fedora has just fixed the Recoll Krunner and KIO packages for Plasma 6 and they are working great. I’m running a local build but I think it’s in the repo now.

Be cool to have it embedded in Dolphin, I see from someones post that it used to be possible in KDE5, with at least FSearch and Catfish.

[sanmayjoshi]

Before I disabled baloo, I noticed it was writing to disk at 77MB/s to 79MB/s. It was doing so for a good 20 to 30 seconds (shame it didn’t click to me to grab a quick screenshot). When checked the size of index, it was around 800MB (iirc).

Just learning about Recoll. Seems to be a nice, lightweight alternative, that also uses qt. Took me some time to find its repo. Here’s the repo if anyone’s interested: Jean-Francois Dockes / recoll · GitLab

[michaell]

All my problems with Baloo went away when I turned off file content indexing. That was the high-CPU culprit, especially since I have a ton of files. Just indexing the file names is sufficient and quick. I don’t know if this should be the default, because I don’t know if I’m the outlier, but this works well for me now.

[ben2talk]

@michaell

All my problems with Baloo went away when I turned off file content indexing.

Sure, All the problems I had with Vista went away when I installed Linux.

I hope you didn’t sign up for a Debate club in your university

It is possible that content indexing can be a heavier hitter, but really not likely unless you inlude directories that you shouldn’t really include.

[Herzenschein]

Baloo’s defaults vary according to distro, by the way.

Fedora for example patches the default directories to ~/Documents, ~/Music, ~/Pictures and ~/Videos and ~/Downloads instead of scanning ~/ by default, and Debian/Ubuntu disables indexing entirely by default.

Whether it will index only files and not content or both files and content also depends on whether the distribution patches the upstream defaults. I’m pretty sure I’ve seen differences there between distros at some point.

[ngompa]

@Herzenschein

Fedora for example patches the default directories to ~/Documents, ~/Music, ~/Pictures and ~/Videos and ~/Downloads instead of scanning ~/ by default, and Debian/Ubuntu disables indexing entirely by default.

We do this because there’s a surprising number of applications that dump stuff into ~/, and it has a severe impact on performance when baloo looks over them, even if it decides to not index them.

[Herzenschein]

@ngompa

We do this because there’s a surprising number of applications that dump stuff into ~/, and it has a severe impact on performance when baloo looks over them, even if it decides to not index them.

I actually requested upstream to follow this exact scheme for similar reasons years ago.

Either Fedora’s pattern of setting the default to xdg user dirs or openSUSE’s old pattern (I think, I might be misremembering) of keeping ~/ as the default but disabling content indexing would be nice IMO.

[michaell]

I hope you get a better personality You are a terrible communicator. This isn’t a debate, professor.

[sanmayjoshi]

@Herzenschein

Either Fedora’s pattern of setting the default to xdg user dirs or openSUSE’s old pattern (I think, I might be misremembering) of keeping ~/ as the default but disabling content indexing would be nice IMO.

Both might be ok as well I think.