The main use of using an indexing system such as baloo is to search the contents of files. Usually this comes in the form of text, PDFs, etc. Image search however remains something difficult to do locally on a machine.
I would like to propose that baloo does the following when enocuntering image formats
Run a classifier to identify what objects are found in the image (e.g. cat, dog, etc.), and then indexing that. This would allow you to search images.
If it’s practical, it would be amazing if i could do a face search too. But that’s not straightforward, and probably better handled by something like digiKam
Figure out if there is text in the image. If there is, OCR it using whatever models are available with tessaract, and then index the results.
I don’t know if baloo was designed in a way that allows extensibility with plugins, but that might be a good approach.
Though I currently use FSearch, because baloo is unfortunately a memory hog, I would like to use baloo in the future. And this feature would be awesome. Though I imagine it wouldn’t help with how much memory baloo currently uses.
People can sometimes be too tired and forget to tag/organize everything when they get back from a trip or come back home from spending hours at a bank or hospital. So, organizing pictures, documents, etc. can sometimes get lost among all the other things we need to do.
Being able to find those files that you didn’t get to organize on the same day, week, or month you got them by using some of the technology available today is not unreasonable.
There are multiple ways to go about it, really. I think Immich/Ente Photos has working local (offline) facial and geographic recognition built in. It’s open-source, so it could be used as a starting point for that route.
And there is digiKam like mentioned above (or similar solutions).
It is probably a good idea to let one thing handle metadata, another handle content, etc., and then for the system search tool (baloo) to be able to access all of it.
This was also mentioned in the thread about making KDE smarter.
The goal is to do this automatically. I got thousands of pictures and I can’t tag every single one of them.
I could, but that’s a very manual process and will take several hours (if not days). Even if I am a very fast typist and come up with all sorts of tricks to speed up the process (e.g. a script that goes through the files automatically and prompts me to provide tags), it would still take a very long time. Especially when cameras just name every single file DSC0001.jpg or something like that.
If you’ve used something like Google Photos, you’ll find that the image search feature is magical. It just knows what you want because of the analysis it does in the background. My idea is that this is done at the local file indexing level. It is a real time saver if it can do it automatically instead of you having to do it manually. That way, you don’t need to use any cloud system, and search your photos like how you would search the contents of any other file.
I’ve used digkam. From what I know, it can do face identification, but not classification of images. Also, it’s meant for photographs, not other pictures/videos (e.g. diagrams, screenshots, etc.) which the classifier can also identify and make searchable.
It is also not an automatic process that runs in the background, and is a very deliberate process by the user to do.
I didn’t want to mention AI An AI can probably do this, but unless you connect to a cloud system (which means privacy and cost implications), you’ll have to use a local LLM with vision capabilities, which is extremely inefficient and resource heavy, and the same thing can be done using smart code and simpler well understood ML-based classifiers.
google has entire server farms at their disposal to do this in the background for you … these externalities need to be considered.
yes, that sounds exactly like what you are looking for which digikam says it offers… but running a LLM on your own PC is good way to understand how much energy this kind of thing uses.
you’re right. But I would argue that modern computers have enough power to do this too. It can be done in the background (on a laptop, it can be done while it’s plugged in so it won’t affect battery life), and it only needs to be done once per file. It doesn’t require a lot of computing power, unlike an LLM. There are already some self hosted server apps that can do this (e.g. librephotos) and those can run on consumer grade hardware.
I do run LLMs on my own PC (mostly out of curiosity and experiemtnation) and they are certainly very energy and compute intensive. I am certainly not advocating to use an LLM for this work, which is meant as a more general purpose application. There are more specific models that specifically are meant to classify images and they are not nearly as compute/energy intensive, and can do several images per second on normal consumer hardware, and this would do the job.
An AI can probably do this, but unless you connect to a cloud system (which means privacy and cost implications), you’ll have to use a local LLM with vision capabilities, which is extremely inefficient and resource heavy, and the same thing can be done using smart code and simpler well understood ML-based classifiers.