Skip to content

Comments

Future Implementations for classes - Measure, Money and Date#257

Closed
ngachchi wants to merge 688 commits intoNVIDIA:mainfrom
ngachchi:hi_tn
Closed

Future Implementations for classes - Measure, Money and Date#257
ngachchi wants to merge 688 commits intoNVIDIA:mainfrom
ngachchi:hi_tn

Conversation

@ngachchi
Copy link
Contributor

@ngachchi ngachchi commented Jan 17, 2025

What does this PR do ?

This pull request introduces enhanced functionality for three classes: Measure, Money, and Date. These improvements are designed to increase usability and flexibility in handling various scenarios related to measurement units, currency transactions, and date manipulations.

Key Changes:

Measure Class:

  • Introduced the "into" operator and added support for various measurement types, enabling users to define and manipulate custom units tailored to their needs.

Money Class:

  • Implemented functionalities for minor currencies, allowing for more precise financial calculations (e.g., handling cents and pennies).

Date Class:

  • Developed a flexible date range functionality, allowing users to manage periods of time confidently, including start and end dates.
  • Integrated support for different eras (e.g., BC/AD, fiscal years) to ensure compatibility with both historical and future date representations.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

zoobereq and others added 30 commits January 17, 2025 16:45
Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
…nd Measure (#241)

* Hindi TN changes

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated date for Hindi TN cache

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* additional whitelist class .tsv files and unused imports removed

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* incorporated suggestions for unused statements and another for closing the file opened

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* Combined Hindi TN and ITN seperate blocks into single

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added init.py files and removed unused commented lines

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* commented irrevelant references and unused snippets from whitelist and word file

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* Whitelist and Word class changes

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* post processor changes with minor fixes

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* remove space before punctuation for sparrowhawk file

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* minor fixes for measure class

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated Jenkinsfile

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* removed unused imports and statements

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated date stamp for HI cache and commented ITN grammars

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* Updates the cache

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Disables Hindi ITN L0 checks

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Reapplies ITN CI Checks

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Adds missing inits

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* resolved the failing sparrowhawk test cases failed

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

---------

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Simon Zuberek <szuberek@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
* Addition of whitelist and word classes

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updation of Jenkins date

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Cleanup

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

---------

Signed-off-by: Tarushi V <tarushiv@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
* Addition of whitelist and word classes

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updation of Jenkins date

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Cleanup

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

---------

Signed-off-by: Tarushi V <tarushiv@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
* ja tn

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* adding ja

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* removing

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* updated tests

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* addressing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addressing ci

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addressing ci

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* addresing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* removing

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* adresing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* removing unused import

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addressing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* addressing comment;

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addressing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* date for ja

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addresing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addressing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* jenkins

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addresing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addressing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* typo

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adressing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* addressing comment

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ci

Signed-off-by: Alex Cui <alexcui1994@gmail.com>

---------

Signed-off-by: Alex Cui <alexcui1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
pre-commit-ci bot and others added 23 commits January 21, 2025 15:40
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
* Fix space issue with ZH ITN

Signed-off-by: Anand Joseph <anajoseph@nvidia.com>

* Update Jenkinsfile

Update FST paths

Signed-off-by: anand-nv <105917641+anand-nv@users.noreply.github.com>

---------

Signed-off-by: Anand Joseph <anajoseph@nvidia.com>
Signed-off-by: anand-nv <105917641+anand-nv@users.noreply.github.com>
Signed-off-by: Simon Zuberek <simon@zuberek.net>
Co-authored-by: Anand Joseph <anajoseph@nvidia.com>
Co-authored-by: anand-nv <105917641+anand-nv@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
* Addition of whitelist and word classes

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updation of Jenkins date

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Cleanup

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

---------

Signed-off-by: Tarushi V <tarushiv@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
* Addition of whitelist and word classes

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updation of Jenkins date

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Cleanup

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

---------

Signed-off-by: Tarushi V <tarushiv@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
* Addition of whitelist and word classes

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updation of Jenkins date

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Cleanup

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

* Updation

Signed-off-by: Tarushi V <tarushiv@nvidia.com>

---------

Signed-off-by: Tarushi V <tarushiv@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Fixes issue with sparrowhawk builds as the original base image is no longer maintained and build breaks

Signed-off-by: anand-nv <105917641+anand-nv@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
 into hi_tn

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Copy link
Contributor

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>
@ngachchi ngachchi closed this Jan 24, 2025
@ngachchi ngachchi deleted the hi_tn branch January 24, 2025 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.