Version control and sharing of project files

[einar]

For me, coming to kdenlive I early asked two questions:

1. How do I version control my projects?
2. How do I share / collaborate with my team?

The obvious answer for someone that like me came from development was, of course, git. Since project files are just xml-files I thought this would “just work” but alas, I was wrong.

Three things made this much more cumbersome than expected:

1. Non-determinstic files as described in this thread by @Ron . Properties seemingly haphazardly changed order for no obvious reason.

2. Prevalence of absolute paths, making portable projects very impractical as described here

<?xml version='1.0' encoding='utf-8'?>
<mlt LC_NUMERIC="en_US.UTF-8" producer="main_bin" root="//wsl$/Ubuntu/home/einar/...

3. Mixture of “proper” project config data and local user config and/or editor app state. Things such as

kdenlive:sequenceproperties.position
kdenlive:sequenceproperties.activeTrack
kdenlive:docproperties.sessionid
kdenlive:docproperties.browserurl
kdenlive:docproperties.renderurl

Why would the browser url be be stored in the project file? And is it really reasonable that a project file is modified as soon as I change the position / active track? Isn’t that just… temporary and ‘nice to have’ application state?

And, when trying to understand these things, a lack of documentation made it even harder. What does docproperties.browserurl even mean?

---

How we solved it

Today, we are actually using git to collaborate on kdenlive projects in my team. We use git-filters (smudge/clean) to normalize the files which makes it possible to have them in one repository and check out files that work. We use git-lfs for any assets.

It works. We are careful, not working on the same project at the same time. But it works.

But it was a hassle and it shouldn’t have to be that way, in my opinion. And there may still be some things we have just not discovered yet. And new changes may come that will render our filters obsolete. New instabilities can come anytime I fear.

Request

Can we please - step by step - make an effort to make the project files

1. Deterministic
2. Portable
3. Free from temporary application state

?

On the kdenlive roadmap I see Collaborative Editing on a medium term time frame. It to me sounds like a fairly big undertaking. Solving the issues above would go a long way to both provide reliable versioning/backups and collaboration among people.

Note: I realize that git is a tool mainly used by developers and alike and perhaps not the average kdenlive user (or?). However, the solution above would also enable other ways of sharing (i.e. mail, dropbox, whatever).

Thank you
/Einar

[jxu]

This is a nice idea. However I don’t know of any video editors that support this, so it would have to be a completely novel implementation.

[Ron]

@einar

Non-determinstic [project] files

I didn’t side-track my other post on this to talk about it, but part of that indeed was that I’d love to be able to version control project files in git. I’ve already been doing that with subtitle files. And long term there isn’t any hard reason kdenlive couldn’t provide a simple front end to version controlled files in git without directly exposing users to it. It already does that with a managed repository of backup files.

I’d actually looked into this a bit more a few weeks back, the code already seems to be setting a deterministic seed for hashing, and the newer versions are better at this than the older ones were, but there’s still a fairly intrinsic assumption in the Qt XML handling code that “XML doesn’t care about order”. So fixing this further might be some work, but it really would be lovely to have at some point.

@einar

Prevalence of absolute paths

Are you seeing any other paths that aren’t relative to the project root (which is the case you showed)? I think some of the ‘global’ resources (like luma files) might be, since they aren’t under the project root - but mostly moving a project should just be a case of updating the root path.

And that one is hard to avoid, because there isn’t necessarily any sense of “current working dir” without that being there to define it. Especially not in the case of things like rendering scripts.

@einar

Mixture of “proper” project config data and local user config and/or editor app state.

True ‘global app state’ is maintained in the rc file. What is in the project file really should be just project specific data/state.

@einar

Why would the browser url be be stored in the project file

So that when you open the media browser again it can be immediately looking at the last location you accessed for this project, not at the resources for some other project or some other random directory that you then need to navigate away from.

There are a few bugs in this, but the intention to operate that way isn’t one of them.

@einar

when trying to understand these things

It’s all there in that universal API which exposes everything

@einar

And new changes may come that will render our filters obsolete

The project files are not documented to be a stable interface, and indeed do change occasionally, though definitely not whimsically, so if they do there is a good reason,
and the code will take steps to sanely update old project files for any changes - but
that is a one-way process, there is nothing implemented to ‘downgrade them’ again
and in most cases that would not be possible without data loss anyway in the common
case where the old file format has no mechanism to represent the new data.

Can we please - step by step - make an effort to make the project files

Deterministic

+1, but it may be non-trivial to actually do.

Portable

For resources under the project root this should already be the case as much as possible, but if your project uses resources outside of that, then simply making those relative paths too just changes what breaks if you move your project to a different location without saving it there using the application itself.

This is another one of those cases where “I want to ask for something that sounds simple using human language” does not translate to “this is a simple, or even satisfiable, problem for a computer to do”.

There may still be a few cases where the correct/best thing is not already being done, but for any non-trivial project, especially in a collection of projects using shared resources, if you want “portability”, you’re going to need to standardise the filesystem structure you contain all of that under among all of the people and machines you want to move it between.

Free from temporary application state

It already is. Temporary application state is held in memory, persistent global state is stored in system configuration files, persistent project state is held in project files, and the project file is only marked ‘modified’ for changes which get pushed onto the undo stack.