Implementation of msgcat and msgmerge utilities from GNU gettext by soft-suroleb · Pull Request #1161 · python-babel/babel

soft-suroleb · 2024-12-09T19:14:49Z

Implementation of some features for msgcat and msgmerge to work with a compendium in PyBabel

pybabel concat

Allows concatenating multiple .po files into one.
In case of translation conflicts, the translation from the first file is taken, as if the use-first option is set to true. Flags, locations, and other information in messages are combined.
Implemented options:
- output-file
- less-than
- more-than
- unique
- use-first
- no-location
- width
- no-wrap
- sort-output
- sort-by-file

pybabel merge

Enables merging files using a compendium for translation memory.
The compendium can be used in two modes:
- Default mode: Translations from the compendium are taken if absent in the output file.
- Compendium overwrite mode: With the compendium-overwrite option, translations in the compendium are considered primary and overwrite those in the output file. If a translation is taken from the compendium, a comment is added specifying the source.
Implemented options:
- input-files
- compendium
- compendium-overwrite
- no-compendium-comment
- update
- output-file
- backup
- suffix
- no-fuzzy-matching
- no-location
- width
- no-wrap
- sort-output
- sort-by-file

…pybabel * Define the MessageConcatenation class to mimic the functionality of GNU gettext's msgcat * Define the MessageMerge class to mimic the functionality of GNU gettext's msgmerge * Implement placeholders for the main interface functions

* Add validation for main msgcat options - input_files, output_file * Temporarily set use_first option to true to avoid handling cases with different translations for the same messages

* Implement options unique, less-than, and more-than, and validate their dependencies with each other. * These options specify which messages to include in the output file. * Implement and validate options no-wrap and width. * Create a helper function _prepare that collects data on message occurrences across different catalogs. * Mark options that are already implemented #

* Implement basic functionality of msgmerge * Use and validate the main options: input-files and output-file * Use and validate options: no-wrap and width * Use and validate options: sort-output and sort-by-file, both in msgmerge and msgcat * In the basic version of working with a compendium, a translation for a message is taken from the compendium only if the resulting catalog lacks a translation.

* Create basic tests to verify the functionality of msgcat, specifically the concatenation of catalogs, merging of message flags, locations, etc. * Remove the validation of options sort-output, sort-by-file, unique, use-first, as they are initialized in the function initialize_options.

* Create basic tests to verify the functionality of msgmerge, specifically the merging of messages and their integration with a compendium. * Remove the definition of sort-output and sort-by-file, and add an additional check for input-files.

…m handling logic * Implement `update` to update the source file instead of writing to the current output file * Implement `backup` to save a backup of the source file before making any updates * Implement `c_overwrite` to use a new mode of handling the compendium, where translations from the compendium overwrite messages in the output file

* Implement a test for `msgmerge` that validates the new mode where compendium entries overwrite messages in the output PO file. * Include the `no_compendium_comment` option to ensure comments about translations sourced from the compendium are not included. * Utilize the `no-location` option to exclude location comments from the output.

* Implemented a helper function `_get_expected` to standardize the expected PO file structure. * Renamed the option `c-overwrite` to `compendium-overwrite`

* Mark the catalog as fuzzy after msgcat and msgmerge if there is at least one fuzzy message * Remove add-location as it's unnecessary

soft-suroleb · 2025-02-28T13:17:18Z

Hi guys! I see you marked pr but still haven't left any comments. Will it be reviewed?

akx

Thank you! Some initial comments within.

babel/messages/frontend.py

tests/messages/test_frontend.py

akx · 2025-03-17T14:40:32Z

tests/messages/test_frontend.py

        assert expected_content == actual_content


+class ConcatanationCatalogTestCase(unittest.TestCase):


The new test cases could live in new files, I think.

And if you feel like it, they could be Pytest style instead of Unittest?

Is a new file really needed here? The file is test_frontend.py and all other tests for commands are there and it's logical that they will be there too, no?

Well, TBH test_frontend.py could also be split into more files – smaller files tend to be easier to read and review.

* Update _prepare function in ConcatenateCatalog to check conflicting messages and to not parse po-files twice * Add _conflicts field in Catalog to mark conflicts * Update tests

* Delete unused options * Fix multiline options comments * Replace backup logic in MergeCatalog * Rename to ConcatenateCatalog

soft-suroleb · 2025-03-23T16:05:06Z

Made support for conflicting messages from different po-files during concatenation. If one message has different translations in different files, then the corresponding strings marked with a comment about conflict

* Includes .rst file with detailed use cases and practical examples for pybabel's concat and merge utilities, outlining common scenarios, options, and best practices for managing PO files.

akx

Another round of comments :)

babel/messages/catalog.py

docs/concat_merge_usage.rst

docs/cmdline.rst

babel/messages/frontend.py

babel/messages/pofile.py

* Rename file_name to filename * Adding fuzzy flag to message parameterized in 'add_conflict' * Replace usage scenarious to cmdline.rst * Rename to ConcatenateCatalog

* Add frozen_time fixture to use freeze_time in every test

soft-suroleb · 2025-06-13T09:30:54Z

Guys? Will there be another round? :)

akx · 2025-06-24T12:18:29Z

@soft-suroleb Sorry, I was honeymooning in Japan for the last 3 weeks 😄

There's still a bunch of unresolved comments from me from the last round of review, I think?

soft-suroleb · 2025-07-06T13:40:29Z

Oh, my congrats :) Yeah, but there is question for you in the unresolved comment

soft-suroleb · 2025-09-04T09:29:48Z

Guys? @akx

akx · 2025-10-08T10:03:16Z

Oh, my congrats :) Yeah, but there is question for you in the unresolved comment

Looking at this again... Sorry, which comment? There's a bunch of comments unresolved 😅

soft-suroleb · 2025-11-09T16:21:22Z

#1161 (comment)

This comment. And why bunch of comments? I think only that, all others were being resolved

akx · 2025-11-12T12:31:03Z

Things like https://github.com/python-babel/babel/pull/1161/files/dd44348272edb1d8bd86818f16ea54809cfd0851#diff-9b4dc8cb664f244c13aebdf018a3748731477a5bb7903a9677326c7ebc177f7b et al – GitHub can be confusing with the "hidden conversations" marker sometimes hiding unresolved comments

but every outstanding comment should be available on the Files page.

soft-suroleb added 13 commits December 9, 2024 20:11

Implement basic logic for concatenating catalogs

246671a

* Add validation for main msgcat options - input_files, output_file * Temporarily set use_first option to true to avoid handling cases with different translations for the same messages

Write tests for msgmerge

6020107

* Create basic tests to verify the functionality of msgmerge, specifically the merging of messages and their integration with a compendium. * Remove the definition of sort-output and sort-by-file, and add an additional check for input-files.

Refactor test for msgmerge with compendium-overwrite option

7228cea

* Implemented a helper function `_get_expected` to standardize the expected PO file structure. * Renamed the option `c-overwrite` to `compendium-overwrite`

Create a catalog without fuzzy by default, remove add-location

2dababc

* Mark the catalog as fuzzy after msgcat and msgmerge if there is at least one fuzzy message * Remove add-location as it's unnecessary

Add tests for using msgcat with plural message forms

888cdd0

Rename msgmerge to merge and msgcat to concat

5202291

Add discription to all options

efe2502

akx added size/L feature area/messages labels Dec 14, 2024

akx added this to the Babel 2.18 milestone Feb 1, 2025

akx requested changes Mar 17, 2025

View reviewed changes

soft-suroleb and others added 4 commits March 23, 2025 16:21

Merge branch 'master' into master

133c8db

Ability to specify multiple compendiums

4cbe604

Marking conflicting messages

8568e90

* Update _prepare function in ConcatenateCatalog to check conflicting messages and to not parse po-files twice * Add _conflicts field in Catalog to mark conflicts * Update tests

Fix PR issues

3f37414

* Delete unused options * Fix multiline options comments * Replace backup logic in MergeCatalog * Rename to ConcatenateCatalog

soft-suroleb requested a review from akx March 30, 2025 13:34

soft-suroleb added 2 commits April 7, 2025 14:08

Add info about pybable concat and pybabel merge into docs

1817bce

Add usage documentation for pybabel concat and merge commands

dd44348

* Includes .rst file with detailed use cases and practical examples for pybabel's concat and merge utilities, outlining common scenarios, options, and best practices for managing PO files.

akx requested changes Apr 22, 2025

View reviewed changes

soft-suroleb added 2 commits April 27, 2025 15:17

Fix PR issues

0a6388d

* Rename file_name to filename * Adding fuzzy flag to message parameterized in 'add_conflict' * Replace usage scenarious to cmdline.rst * Rename to ConcatenateCatalog

Add '_conflicts' getter in catalog

5828c13

soft-suroleb and others added 2 commits April 27, 2025 15:38

Rework tests from unittest to pytest

bbba96e

* Add frozen_time fixture to use freeze_time in every test

Merge branch 'master' into master

01f3793

akx self-requested a review June 24, 2025 12:15

Merge branch 'master' into master

1871bcf

akx modified the milestones: Babel 2.18, Babel 2.19 Dec 29, 2025

		assert expected_content == actual_content


		class ConcatanationCatalogTestCase(unittest.TestCase):

Conversation

soft-suroleb commented Dec 9, 2024

Implementation of some features for msgcat and msgmerge to work with a compendium in PyBabel

pybabel concat

pybabel merge

Uh oh!

soft-suroleb commented Feb 28, 2025

Uh oh!

akx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

akx Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

soft-suroleb Mar 23, 2025

Choose a reason for hiding this comment

Uh oh!

akx Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

soft-suroleb commented Mar 23, 2025

Uh oh!

akx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soft-suroleb commented Jun 13, 2025

Uh oh!

akx commented Jun 24, 2025

Uh oh!

soft-suroleb commented Jul 6, 2025

Uh oh!

soft-suroleb commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akx commented Oct 8, 2025

Uh oh!

soft-suroleb commented Nov 9, 2025

Uh oh!

akx commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

soft-suroleb commented Sep 4, 2025 •

edited

Loading