Add new InternVL2-1B model with multi-format output (LaTex, HTML and Markdown) and Efficient inference by LMDeploy

Author: PrinceVictor

README.md

@@ -4,20 +4,22 @@
 
 [[ Related Paper ]](https://arxiv.org/abs/2406.11633) [[ Website ]](https://unimodal4reasoning.github.io/DocGenome_page/) [[ Dataset (Google Drive)]](https://drive.google.com/drive/folders/1OIhnuQdIjuSSDc_QL2nP4NwugVDgtItD) [[ Dataset (Hugging Face) ]](https://huggingface.co/datasets/U4R/DocGenome/tree/main)
 
-[[Models 🤗(Hugging Face)]](https://huggingface.co/U4R/StructTable-base/tree/main)
+[[Models 🤗(Hugging Face)]](https://huggingface.co/U4R/StructTable-InternVL-1B/tree/main)
 
 
 </div>
 
-Welcome to the official repository of StructEqTable-Deploy, a solution that converts images of Table into LaTeX, powered by scalable data from [DocGenome benchmark](https://unimodal4reasoning.github.io/DocGenome_page/).
+Welcome to the official repository of StructEqTable-Deploy, a solution that converts images of Table into LaTeX/HTML/MarkDown, powered by scalable data from [DocGenome benchmark](https://unimodal4reasoning.github.io/DocGenome_page/).
 
 
 ## Overview
 Table is an effective way to represent structured data in scientific publications, financial statements, invoices, web pages, and many other scenarios. Extracting tabular data from a visual table image and performing the downstream reasoning tasks according to the extracted data is challenging, mainly due to that tables often present complicated column and row headers with spanning cell operation. To address these challenges, we present TableX, a large-scale multi-modal table benchmark extracted from [DocGenome benchmark](https://unimodal4reasoning.github.io/DocGenome_page/) for table pre-training, comprising more than 2 million high-quality Image-LaTeX pair data covering 156 disciplinary classes. Besides, benefiting from such large-scale data, we train an end-to-end model, StructEqTable, which provides the capability to precisely obtain the corresponding LaTeX description from a visual table image and perform multiple table-related reasoning tasks, including structural extraction and question answering, broadening its application scope and potential.
 
 ## Changelog
-Tips: Current version of StructEqTable is able to process table images from scientific documents such as arXiv, Scihub papers. Times New Roman And Songti(宋体) are main fonts used in table image, other fonts may decrease the accuracy of the model's output.
-- **[2024/8/22] 🔥 We have released our [latest model](https://huggingface.co/U4R/StructTable-base/tree/v0.2), fine-tuned on the DocGenome dataset. This version features improved inference speed and robustness, achieved through data augmentation and reduced image token num.** 
+- [2024/10/19] 🔥 We have released our **latest model [StructTable-InternVL2-1B](https://huggingface.co/U4R/StructTable-InternVL-1B/tree/main)**! 
+
+  Thanks to IntenrVL2 powerful foundational capabilities, and through fine-tuning on the synthetic tabular data and DocGenome dataset, StructTable can convert table image into various common table formats including LaTeX, HTML, and Markdown. Moreover, inference speed has been significantly improved compared to the v0.2 version.
+- [2024/8/22] We have released our StructTable-base-v0.2, fine-tuned on the DocGenome dataset. This version features improved inference speed and robustness, achieved through data augmentation and reduced image token num.
 - [2024/8/08] We have released the TensorRT accelerated version, which only takes about 1 second for most images on GPU A100. Please follow the tutorial to install the environment and compile the model weights.
 - [2024/7/30] We have released the first version of StructEqTable. 
 
@@ -26,12 +28,10 @@ Tips: Current version of StructEqTable is able to process table images from scie
 - [x] Release inference code and checkpoints of StructEqTable.
 - [x] Support Chinese version of StructEqTable.
 - [x] Accelerated version of StructEqTable using TensorRT-LLM.
-- [ ] Expand more domains of table image to improve the model's general capabilities.
+- [x] Expand more domains of table image to improve the model's general capabilities.
+- [x] Efficient inference of StructTable-InternVL2-1B by [LMDepoly](https://github.com/InternLM/lmdeploy) Tookit.
 - [ ] Release our table pre-training and fine-tuning code
 
-## Efficient Inference
-Our model now supports TensorRT-LLM deployment, achieving a 10x or more speedup in during inference.  
-Please refer to [GETTING_STARTED.md](docs/GETTING_STARTED.md) to learn how to depoly.
 
 ## Installation
 ``` bash 
@@ -47,15 +47,17 @@ python setup develop
 pip install "git+https://github.com/UniModal4Reasoning/StructEqTable-Deploy.git"
 
 # or Install from PyPI
-pip install struct-eqtable==0.1.0
+pip install struct-eqtable==0.3.0
 ```
 
 ## Model Zoo
 
-| Model | Image Token Num | Model Size | Training Data | Data Augmentation | TensorRT | HuggingFace |
-|---------------------|---------------------|------------|------------------|-------------------|----------|-------------------|
-| StructEqTable-base | 4096 | ~300M | DocGenome |  | ☑️ | [v0.1](https://huggingface.co/U4R/StructTable-base/tree/v0.1) |
-| StructEqTable-base | 2048 | ~300M | DocGenome | ☑️ | ☑️ | [v0.2](https://huggingface.co/U4R/StructTable-base/tree/v0.2) |
+| Model | Model Size | Training Data | Data Augmentation | LMDepoly | TensorRT | HuggingFace |
+|---------------------|------------|------------------|-------------------|----------|----------|-------------------|
+| StructEqTable-InternVL | ~1B | DocGenome and Synthetic Data | ✔ | ✔ | | [v0.3](https://huggingface.co/U4R/StructTable-InternVL-1B/tree/main) |
+| StructEqTable-base | ~300M | DocGenome | ✔ | | ✔ | [v0.2](https://huggingface.co/U4R/StructTable-base/tree/v0.2) |
+| StructEqTable-base | ~300M | DocGenome | | | ✔ | [v0.1](https://huggingface.co/U4R/StructTable-base/tree/v0.1) |
+
 
 
 ## Quick Demo
@@ -65,26 +67,34 @@ cd tools/demo
 
 python demo.py \
   --image_path ./demo.png \
-  --ckpt_path ${CKPT_PATH} \
+  --ckpt_path U4R/StructTable-InternVL-1B \
   --output_format latex
 ```
 
-- HTML or Markdown format output
+- HTML or Markdown format output (Only Supported by StructTable-InternVL2-1B)
 
-  Our model output Latex format code by default. 
-  If you want to get other format like HTML or Markdown, 
-  `pypandoc` support convert latex format code into HTML and Markdown format for simple table (table has no merge cell ).
+```shell script
+python demo.py \
+  --image_path ./demo.png \
+  --ckpt_path U4R/StructTable-InternVL-1B \
+  --output_format html markdown
+```
 
+## Efficient Inference
+- Install LMDeploy Tookit
 ```shell script
-sudo apt install pandoc
-pip install pypandoc
+pip install lmdeploy
+```
 
+- Run the demo/demo.py
+```shell script
 cd tools/demo
 
 python demo.py \
   --image_path ./demo.png \
-  --ckpt_path ${CKPT_PATH} \
-  --output_format html markdown
+  --ckpt_path U4R/StructTable-InternVL-1B \
+  --output_format latex \
+  --lmdeploy
 ```
 
 
@@ -100,9 +110,11 @@ python demo.py \
 - [DocGenome](https://github.com/UniModal4Reasoning/DocGenome). An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Models.
 - [ChartVLM](https://github.com/UniModal4Reasoning/ChartVLM). A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning.
 - [Pix2Struct](https://github.com/google-research/pix2struct). Screenshot Parsing as Pretraining for Visual Language Understanding.
+- [InternVL Family](https://github.com/OpenGVLab/InternVL). A Series of Powerful Foundational Vision-Language Models.
+- [LMDepoly](https://github.com/InternLM/lmdeploy). A toolkit for compressing, deploying, and serving LLM and MLLM.
 - [UniMERNet](https://github.com/opendatalab/UniMERNet). A Universal Network for Real-World Mathematical Expression Recognition.
 - [Donut](https://huggingface.co/naver-clova-ix/donut-base). The UniMERNet's Transformer Encoder-Decoder are referenced from Donut.
-- [Nougat](https://github.com/facebookresearch/nougat). The tokenizer uses Nougat.  
+- [Nougat](https://github.com/facebookresearch/nougat). Data Augmentation follows Nougat.  
 - [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). Model inference acceleration uses TensorRT-LLM.
 
 

docs/GETTING_STARTED.md docs/TENSORRT_GETTING_STARTED.mddocs/GETTING_STARTED.md renamed to docs/TENSORRT_GETTING_STARTED.md

@@ -107,7 +107,33 @@ You may get output as follows:
 ```
 total cost time: 0.88s
 Table 0 LATEX format output:
-\begin{tabular}{l@{\hskip 0.4in}ccc@{\hskip 0.3in}ccc}\hline \multicolumn{1}{c}{\multirow{2}{*}{Model}} & \multicolumn{2}{c}{\bf MCQA} & \multicolumn{2}{c}{\bf NSP} & \multicolumn{2}{c}{\bf PI} \\\multicolumn{1}{c}{} & Accuracy & F1 & Accuracy & F1 & Accuracy & F1 \\ \hline FastText & 0.318 & 0.317 & 0.496 & 0.496 & 0.762 & 0.806 \\ELMo & 0.318 & 0.318 & 0.691 & 0.691 & 0.807 & 0.867 \\BERT & 0.346 & 0.346 & 0.514 & 0.514 & 0.801 & 0.857 \\ \hline \end{tabular}
+\begin{tabular}{|c|c|c|c|}
+\hline
+Quantity $\backslash$ Unit System & International System SI (kg-m-s) & Traditional aeronautical (lb-ft-s) & Traditional structural (lb-inch-s) \\
+\hline
+Mass (translational inertia), $m$ & kilogram mass (kg) & slug = lb-s$^2$/f & lb-s$^2$/inch \\
+\hline
+Length, translational motion & meter (m) & foot (ft) & inch (in.) \\
+\hline
+Time, $t$ & second (s) & second (s) & second (s) \\
+\hline
+Force, translational action & newton (N) = kg-m/s$^2$ & pound force (lb) & pound force (lb) \\
+\hline
+Translational stiffness constant, $k$ & N/m & lb/ft & lb/inch \\
+\hline
+Translational damping constant, $c$ & N/(m/s) = N-s/m & lb/(ft/s) = lb-s/ft & lb/(inch/s) = lb-s/inch \\
+\hline
+Angle, rotational motion & radial (rad), which is dimensionless & radial (rad), which is dimensionless & radial (rad), which is dimensionless \\
+\hline
+Rotational inertia, $J$ & kg-m$^2$ & slug-ft$^2$ = lb-s$^2$ - ft & lb-s$^2$ - inch \\
+\hline
+Moment or torque, rotational action & N-m & lb-ft & lb-inch \\
+\hline
+Rotational stiffness constant, $k_\theta$ & (N-m)/rad = N-m & (lb-ft)/rad = lb-ft & (lb-inch)/rad = lb-inch \\
+\hline
+Rotational damping constant, $c_\theta$ & (N-m)/(rad/s) = N-m-s & (lb-ft)/(rad/s) = lb-ft-s & (lb-inch)/(rad/s) = lb-inch-s \\
+\hline
+\end{tabular}
 ```
 
 

setup.py

@@ -7,7 +7,7 @@ def write_version_to_file(version, target_file):
         print('__version__ = "%s"' % version, file=f)
 
 if __name__ == '__main__':
-    version = '0.1.0'
+    version = '0.3.0'
     write_version_to_file(version, 'struct_eqtable/version.py')
     with Path(Path(__file__).parent,
               'README.md').open(encoding='utf-8') as file:

struct_eqtable/__init__.py

@@ -1,11 +1,43 @@
-from .model import StructTable
+from .pix2s import Pix2Struct, Pix2StructTensorRT
+from .internvl import InternVL, InternVL_LMDeploy
 
-def build_model(model_ckpt, **kwargs):
-    tensorrt_path = kwargs.get('tensorrt_path', None)
-    if tensorrt_path is not None:
-        from .model_trt import StructTableTensorRT
-        model = StructTableTensorRT(model_ckpt, **kwargs)
+from transformers import AutoConfig
+
+
+__ALL_MODELS__ = {
+    'Pix2Struct': Pix2Struct,
+    'Pix2StructTensorRT': Pix2StructTensorRT,
+    'InternVL': InternVL,
+    'InternVL_LMDeploy': InternVL_LMDeploy,
+}
+
+
+def get_model_name(model_path):
+    model_config = AutoConfig.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+    )
+
+    if 'Pix2Struct' in model_config.architectures[0]:
+        model_name = 'Pix2Struct'
+    elif 'InternVL' in model_config.architectures[0]:
+        model_name = 'InternVL'
     else:
-        model = StructTable(model_ckpt, **kwargs)
+        raise ValueError(f"Unsupported model type: {model_config.architectures[0]}")
+
+    return model_name
+
+
+def build_model(model_ckpt, **kwargs):
+    model_name = get_model_name(model_ckpt)
+    if model_name == 'InternVL' and kwargs.get('lmdeploy', False):
+        model_name = 'InternVL_LMDeploy'
+    elif model_name == 'Pix2Struct' and kwargs.get('tensorrt_path', None):
+        model_name = 'Pix2StructTensorRT'
+
+    model = __ALL_MODELS__[model_name](
+        model_ckpt, 
+        **kwargs
+    )
 
     return model

struct_eqtable/internvl/__init__.py

@@ -0,0 +1,2 @@
+from .internvl import InternVL
+from .internvl_lmdeploy import InternVL_LMDeploy