Systematic integration of OCR features into Plasma

Brainstorm
, ,
1

There are efforts to integrate OCR into Plasma applications:

While the intentions are similar, but the way to achieve them almost always comes down to the same thing (e.g. usage of Tesseract), all of these implementations are separate approaches of doing the similar (generally - the same) things.

For some time I also became interested in implementing OCR support in my small Qt/C++ document framework: GitHub - uenatole/QtDocumentView: Like QPdfView, but universal · GitHub.

The more I discover all of these things than more I’m become confident about the necessity of some common solution because:

  • it might replace repeating code in different projects that using the same backend (Tesseract) and providing the same result (extracted text / text geometry);
  • at the same time, he is completely free to use an alternative text recognition pipeline, including additional steps, or not using the Tesseract;
  • there are some common approaches to text extraction that should be used, but which may not be used in the separate project due to ignorance or cost of its implementation.

My personal view to possible structure of this common component

  • core:

    • abstraction of “text probing” that provides the method giving the fast answer to the question “is this image contains text?”. It might be used to skip time-expensive text recognition (for example it the use-cases where it done automatically on image is shown to the user, probably like in the situation when OcrDocument is used on my QtDocumentView project). There are “false positive” and “false negative” scenarios too, but this shouldn’t be a problem it cases it used.

    • abstraction of “text recognizing” that provides the method of text & its geometry extraction. Might be composed of different stages, e.g. “preprocessing → recognition → postprocessing”.

    • abstraction of “OCR text context” that serves as facade over “prober” and “recognizer” with state machine logic and optional “OCR text state” to pass intermediate results between “probing” and “recognizing” (because some high-precision methods might reuse results from low-precision text recognition methods). Can be used for typical workflows following:

      stateDiagram-v2
[*] --> None
None --> ImageSet: setImage(QImage), QImage.isNull() == false
ImageSet --> None: setImage(QImage), QImage.isNull() == true
ImageSet --> Probing: probe()
Probing --> Probed: (done)
ImageSet --> Recognizing: recognize()
Recognizing --> Recognized: (done)
Probed --> Recognizing: recognize()
Probing --> ImageSet: cancel()
Recognizing --> ImageSet: cancel()

  • standard implementation: e.g. Tesseract-based backend, something default that fits into the most common use-cases.

  • misc: support of highlighting text above an image (with a simple rectangle outline or as in MacOS LiveText) on the surface displaying the image or its selection like over regular text document - to not to reinvent the wheel wherever highlighting or selection of the recognized text is required.

The most things I described above is rough approximation of what might be done to provide systematic way of OCR integration in Plasma ecosystem. I wrote this topic to share some of my thoughts on this, start a discussion and, I hope, at least somehow contribute to the translation of some ideas into reality.

1 Like
2

what do you say to ppl who do not want any text recognition capabilities built into plasma out fears it will be turned into a way to exfiltrate our data?

those ppl exist.

3

What the question…

Firstly, the OCR features is already integrated in Plasma (see Spectacle for example). So there is already “a way to exfiltrate our data”. What a pity.

Secondly, it’s not a discussion about the implementation of such things in the future, but about the possible development consolidation aimed to deduplicate the work is being done and will certainly be done right now.

So it’s not about implementation yet-another-hipster-demonic-service that shall pollute the precious bodily fluids steal someone’s data. Just a common component that might ease providing good and useful functionality that significantly improves UX.

those ppl exist.

Yes, and there is nothing we can do. Meaningful questions, meaningful answers.

1 Like
4

Also, besides the jokes, I would like to add that personally I don’t know about any existing concerns about the presence of OCR features in applications. Are they represented in any Internet community or just appeared in your mind right now?

5
2 Likes
6

It’s primarily about “bloat” dependencies people doesn’t want, not security concerns. But yes, if we refer to @skyfishgoo’s theses in post you mentioned, this is indeed a question worthy of attention.

I repeat: that is not relevant to this discussion. Nothing prevents from opting it, on the contrary, separating most of the common OCR code into a separate component benefits such people, because all the “heaviness” of OCR features is outside the application code, all that remains is a small adapter (if you like that name) for their integration. This is how plugins work.