I'm the maintainer for various upstream projects [1] which are currently using Fedora's Zanata for translation services and I saw the recent change proposal for Fedora to shift to Weblate in the next release cycle.
First I'm wondering what sort of time frame is being targetted for move of the upstream projects which are independant of Fedora's release cycle ?
Next, and more importantly, I'm trying to understand how we will integrate with Weblate....
With Zanata the maintainers just run "zanata push" and "zanata pull" when desired and we have rules integrated into our build system to do various processing steps to build the .pot file and .po files as needed.
Looking at what was done for ibus-typing-booster, IIUC Weblate is given commit access directly to the git repository.
This is tricky for us as, AFAICT, it assumes the project is storing the .pot files and .po files in git which is not the case for an increasing number of our virt related projects - libvirt, libvirt-glib & virt-viewer. It is also not appealing from a security POV to grant commit access to a 3rd party service.
The .pot file is never stored as it is an huge auto-generated file which would be out of date as soon as it was committed. We would always rebuild this from master source files whenever it is needed so it is fully updated.
We also recently stopped storing full .po files [2], again because they are huge files with a large amount of redundant information in them. Instead we store what we call ".mini.po" files. This is the same file format as a normal ".po" file, but with
- all filenames + line numbers stripped - all non-translated msgids stripped - msgids sorted alphabetically
This is loss-less - all the removed information can be automatically re-added by processing with msgmerge & the latest .pot file.
This cut down size of stored /po directory from 100 MB to just 18 MB.
It also means that when we push updates with new translations, the git patch diffs have high signal/noise ratio and are very small. eg a git commit refreshing translations is now approx 700 KB, instead of the ~80 MB we saw when storing full .po files.
This can also makes distributed tar.xz files smaller as we only need include the .mini.po files.
We have build rules from convert from .po files to .mini.po files which are run when we pull down new translations from zanata, and the inverse to convert from .mini.po files to .po files which we run at build time when creating the binary .mo files for installation into /usr/share/locale
All this means though that Weblate can't simply operate directly on our git repo. There's no .pot for it to find, nor can it push normally formatted .po files.
Thus I'm trying to understand how we'll go about integrating with Weblate.
I'm guessing there's no way to control how it interacts with git - eg no way to tell it to run meson to generate a .pot file, nor any way to tell it to run meson to generate .mini.po files
The 'wlc download' and 'wlc upload' commands appear to be similar in purpose to zanata's 'pull' and 'push' commands, but when I try to run 'wlc download ibus-typing-booster/app' it reports "Error: Not supported". Essentially if there's a way to push a .pot file and pull a .po file from Weblate's REST API we'll be able to continue our current workflow.
Finally in Weblate after authenticating with FAS, I still don't have the ability to create new projects. Is this intentionally restricted ? I was hoping to do some testing with Weblate & one of the virt apps like virt-viewer to try to understand its operation better before any formal transition period.
Regards, Daniel
[1] libvirt, libvirt-glib, virt-viewer, entangle, osinfo-db, osinfo-db-tools, libosinfo, virt-manager, libvirt-sandbox [2] https://www.berrange.com/posts/2018/11/29/improved-translation-po-file-handl...
Daniel P. Berrangé berrange@redhat.com さんはかきました:
Looking at what was done for ibus-typing-booster, IIUC Weblate is given commit access directly to the git repository.
It does not commit directly to the ibus-typing-booster github repo, it does pull requests like this:
https://github.com/mike-fabian/ibus-typing-booster/pull/59
The 'wlc download' and 'wlc upload' commands appear to be similar in purpose to zanata's 'pull' and 'push' commands, but when I try to run 'wlc download ibus-typing-booster/app' it reports "Error: Not supported". Essentially if there's a way to push a .pot file and pull a .po file from Weblate's REST API we'll be able to continue our current workflow.
I could not get that working either, I would like to find out what is wrong.
Le 2019-11-05 12:15, Daniel P. Berrangé a écrit :
I'm the maintainer for various upstream projects [1] which are currently using Fedora's Zanata for translation services and I saw the recent change proposal for Fedora to shift to Weblate in the next release cycle.
Thank you for coming here and engaging this discussion!
First I'm wondering what sort of time frame is being targetted for move of the upstream projects which are independant of Fedora's release cycle ?
Wish: by February, all projects are using the new translation platform. Reality: there is no hard timeframe, but the translators won't go on multiple translation platform, and because most important packages will have migrate, it may lower a lot the number of contributions to non migrated projects.
Next, and more importantly, I'm trying to understand how we will integrate with Weblate....
The .pot file is never stored as it is an huge auto-generated file which would be out of date as soon as it was committed. We would always rebuild this from master source files whenever it is needed so it is fully updated.
Suboptimal is the pot file format, but also is the best we have. Next generation may be the "Fluent" file format from Mozilla, wait and see.
We also recently stopped storing full .po files [2], again because they are huge files with a large amount of redundant information in them. Instead we store what we call ".mini.po" files. This is the same file format as a normal ".po" file, but with
[...]
Thus I'm trying to understand how we'll go about integrating with Weblate.
It's an interesting thing you did!
I see a few solutions: * you create a repository dedicated to pot and po files, * you interact with Weblate using the command line interface or one of the APIs.
Finally in Weblate after authenticating with FAS, I still don't have the ability to create new projects. Is this intentionally restricted ? I was hoping to do some testing with Weblate & one of the virt apps like virt-viewer to try to understand its operation better before any formal transition period.
Yes, project creation is locked on purpose and won't be open. Correct configuration isn't so easy and it requires some interaction with localization team. The way it worked with Zanata was way too loose, nobody was informed when a new project came or when a project created a new document/branch.
I would love to collaborate with you to find a correct workflow using either method of your choice (and your help with DNF and anaconda team would be much appreciated), but first: _we need approval from Brian Exelbierd or Ben Cotton to continue deployment_. Red Hat and Weblate should finish the paperwork (billing + terms of use). Brian is the one responsible for this action.
Note for readers: I will always suggest to keep it simple and include pot/po files in repository for little projects. Don't invest time in automation as it will add maintenance work and lower the proximity between upstream source code and the translation platform.
Jean-Baptiste
On Tue, Nov 05, 2019 at 01:54:24PM +0100, Jean-Baptiste Holcroft wrote:
Le 2019-11-05 12:15, Daniel P. Berrangé a écrit :
I'm the maintainer for various upstream projects [1] which are currently using Fedora's Zanata for translation services and I saw the recent change proposal for Fedora to shift to Weblate in the next release cycle.
Thank you for coming here and engaging this discussion!
First I'm wondering what sort of time frame is being targetted for move of the upstream projects which are independant of Fedora's release cycle ?
Wish: by February, all projects are using the new translation platform. Reality: there is no hard timeframe, but the translators won't go on multiple translation platform, and because most important packages will have migrate, it may lower a lot the number of contributions to non migrated projects.
Yeah, we want our upstream projects to be where the translaters are most active.
Assuming we can figure out the workflow, I don't see a big problem with moving the projects I'm involved with by Feb timeframe.
We also recently stopped storing full .po files [2], again because they are huge files with a large amount of redundant information in them. Instead we store what we call ".mini.po" files. This is the same file format as a normal ".po" file, but with
[...]
Thus I'm trying to understand how we'll go about integrating with Weblate.
It's an interesting thing you did!
I see a few solutions:
- you create a repository dedicated to pot and po files,
- you interact with Weblate using the command line interface or one of the
APIs.
The CLI/API option is the least disruptive to us - separate repo is probably an undesirable burden to maintainers.
Finally in Weblate after authenticating with FAS, I still don't have the ability to create new projects. Is this intentionally restricted ? I was hoping to do some testing with Weblate & one of the virt apps like virt-viewer to try to understand its operation better before any formal transition period.
Yes, project creation is locked on purpose and won't be open. Correct configuration isn't so easy and it requires some interaction with localization team. The way it worked with Zanata was way too loose, nobody was informed when a new project came or when a project created a new document/branch.
I would love to collaborate with you to find a correct workflow using either method of your choice (and your help with DNF and anaconda team would be much appreciated), but first: _we need approval from Brian Exelbierd or Ben Cotton to continue deployment_. Red Hat and Weblate should finish the paperwork (billing + terms of use). Brian is the one responsible for this action.
I'd probably want start off by doing a PoC with the moving the virt-viewer project as its fairly small & low traffic. I can look at it whenever things are ready from POV of the Fedora weblate deployment.
Regards, Daniel
Daniel P. Berrangé berrange@redhat.com さんはかきました:
On Tue, Nov 05, 2019 at 01:54:24PM +0100, Jean-Baptiste Holcroft wrote:
It's an interesting thing you did!
I see a few solutions:
- you create a repository dedicated to pot and po files,
- you interact with Weblate using the command line interface or one of the
APIs.
The CLI/API option is the least disruptive to us - separate repo is probably an undesirable burden to maintainers.
I would also like to use the command line interface if possible. The pull requests I currently get are fine, I like it, but it would be nice if I also could force weblate to update the pull request immediately (Maybe “wlc commit” would do that??). Currently I only get the error message
Error: Object not found on the server (maybe operation is not supported on the server)
whenever I try to use wlc.
Le 2019-11-05 14:21, Mike FABIAN a écrit :
I would also like to use the command line interface if possible. The pull requests I currently get are fine, I like it, but it would be nice if I also could force weblate to update the pull request immediately (Maybe “wlc commit” would do that??). Currently I only get the error message
Error: Object not found on the server (maybe operation is notsupported on the server)
whenever I try to use wlc.
which command do you type?
Le 2019-11-05 14:51, Jean-Baptiste Holcroft a écrit :
Le 2019-11-05 14:21, Mike FABIAN a écrit :
I would also like to use the command line interface if possible. The pull requests I currently get are fine, I like it, but it would be nice if I also could force weblate to update the pull request immediately (Maybe “wlc commit” would do that??). Currently I only get the error message
Error: Object not found on the server (maybe operation is notsupported on the server)
Weblate team answered me, if you want to manually download, your shoud explicitly write the language name: wlc download ibus-typing-booster/app/fr They will improve the error message
but to really interact with Weblate in your workflow (pot and po files are in you repo):
# Commits changes in Weblate object (translation, component or project). wlc commit
# Pushes changes in Weblate object into remote repository (translation, component or project). wlc push
Jean-Baptiste
Jean-Baptiste Holcroft jean-baptiste@holcroft.fr さんはかきました:
Le 2019-11-05 14:51, Jean-Baptiste Holcroft a écrit :
Le 2019-11-05 14:21, Mike FABIAN a écrit :
I would also like to use the command line interface if possible. The pull requests I currently get are fine, I like it, but it would be nice if I also could force weblate to update the pull request immediately (Maybe “wlc commit” would do that??). Currently I only get the error message Error: Object not found on the server (maybe operation is not supported on the server)
Weblate team answered me, if you want to manually download, your shoud explicitly write the language name: wlc download ibus-typing-booster/app/fr They will improve the error message
wlc download ibus-typing-booster/app/fr
works for me ☺.
but to really interact with Weblate in your workflow (pot and po files are in you repo):
# Commits changes in Weblate object (translation, component or project). wlc commit
# Pushes changes in Weblate object into remote repository (translation, component or project). wlc push
These two commands do not work for me:
$ wlc commit Error: You don't have permission to access this object mfabian@taka:/local/mfabian/src/ibus-typing-booster (release-2.7.3 $%) $ wlc push Error: You don't have permission to access this object $
I am also a bit confused what they actually mean. Does “wlc push” update the push request in my ibus-typing-booser github repo?
Le 05/11/2019 à 12:15, Daniel P. Berrangé a écrit :
Next, and more importantly, I'm trying to understand how we will integrate with Weblate....
Hello Daniel,
let's integrate your projects to the new translation platform.
where are the public bug trackers of your projects?
I have a group of question for each repository: https://fedoraproject.org/w/index.php?title=L10N_Move_to_Weblate/Message_for...
thanks for your help
On Mon, Dec 09, 2019 at 09:11:35PM +0100, Jean-Baptiste wrote:
Le 05/11/2019 à 12:15, Daniel P. Berrangé a écrit :
Next, and more importantly, I'm trying to understand how we will integrate with Weblate....
Hello Daniel,
let's integrate your projects to the new translation platform.
where are the public bug trackers of your projects?
For libvirt related projects currently we use
https://bugzilla.redhat.com/enter_bug.cgi?product=Virtualization%20Tools
I have a group of question for each repository: https://fedoraproject.org/w/index.php?title=L10N_Move_to_Weblate/Message_for...
So that I can gain a understanding of the weblate integration, lets start with only the libvirt-glib project first
https://libvirt.org/git/?p=libvirt-glib.git;a=log https://fedora.zanata.org/project/view/libvirt-glib/?dswid=478
* which branch is your development branch?
master, but note weblate won't be allowed to either push or pull, we must manually upload & download .pot/.po files
* have you merged latest translation from Zanata and locked the project?
I've hit the "make this project readonly" assuming that's what you mean by 'locked'.
* Weblate will handle updates when pot file changes, don't edit po files for this [2]
As above, we can't have weblate directly accessing git
* what is the license of translation? (please use a code from https://spdx.org/licenses/)
LGPLv2-or-later
* do you have any announcement/warning you would like to display to the translators? (it will be displayed in Weblate)
Questions/clarifications about translations should be sent to the libvir-list@redhat.com mailing list. For further information consult https://libvirt.org
* do you need us to activate any specific checks? (this is a setting per component [3])
Any .po file validation rules relevant to the C language.
Especially validating that printf format specifiers have not been re-ordered/removed/changed, since we see that screw up alot.
* do you need us to automatically detect new translation files? (typical usecase: website translation with one translation file per page)
Nope
Regards, Daniel
Le 12/12/2019 à 17:12, Daniel P. Berrangé a écrit :
On Mon, Dec 09, 2019 at 09:11:35PM +0100, Jean-Baptiste wrote:
Hello Daniel, let's integrate your projects to the new translation platform.
where are the public bug trackers of your projects?
For libvirt related projects currently we use
https://bugzilla.redhat.com/enter_bug.cgi?product=Virtualization%20Tools
Thanks for your answer, I've created a bug there: https://bugzilla.redhat.com/show_bug.cgi?id=1783579
let's discuss all of this on bugzilla, and let interested people to register to the conversation
-
Daniel P. Berrangé -
Jean-Baptiste -
Jean-Baptiste Holcroft
-
Mike FABIAN