Add interactive download functionality for MaStR date selection #696 by fharookshaik · Pull Request #697 · OpenEnergyPlatform/open-MaStR

fharookshaik · 2025-12-16T22:46:32Z

Summary of the discussion

This documentation describes the implementation of interactive download functionality for the open-MaStR project. The enhancement addresses a key limitation where users could only download the latest MaStR export data (today's date) and had no visibility into available historical downloads.

Problem Solved
Before: Users were limited to downloading only today's MaStR export data through bulk method.
After: Users can now browse all available downloads and select any historical date

Key Features Implemented

Interactive Date Selection: Users can browse available downloads and select which date to download
Programmatic Access: Developers can fetch available dates and select downloads programmatically
Backward Compatibility: All existing functionality preserved - no breaking changes
Enhanced User Experience: User-friendly interfaces for both interactive and programmatic workflows

Type of change (CHANGELOG.md)

Added

Add interactive download functionality to browse and select MaStR export dates (#696)
Add get_available_download_links() function to fetch available downloads from MaStR website (#696)
Add list_available_downloads() function to display downloads in user-friendly format (#696)
Add select_download_date() function for interactive date selection (#696)
Add browse_available_downloads() method to Mastr class (#696)

Updated

Enhance download_xml_Mastr() function with optional URL parameter for custom downloads (#696)
Update Mastr.download() method with select_date_interactively parameter (#696)
Update Mastr.download() method docstring with comprehensive documentation for new parameter (#696)

Removed

No functionality removed - maintains 100% backward compatibility (#696)

Workflow checklist

Automation

Closes #696

PR-Assignee

🐙 Follow the workflow in CONTRIBUTING.md
📝 Update the CHANGELOG.md
📙 Update the documentation

Reviewer

🐙 Follow the Reviewer Guidelines
🐙 Provided feedback and show sufficient appreciation for the work done

…nergyPlatform#696

fharookshaik · 2025-12-16T22:53:43Z

Usage Examples

1. Basic Interactive Usage

from open_mastr import Mastr

# Initialize database
db = Mastr()

# Start interactive download selection
db.download(select_date_interactively=True)

Console Output:

================================================================================
AVAILABLE MAStR DOWNLOADS
================================================================================
#    Date         Version    Type         URL
--------------------------------------------------------------------------------
1    2025-01-03   24.2       current      https://download.marktstammdatenregister.de/Gesamtdatenexport_20250103_24.2.zip
2    2024-12-31   24.2       current      https://download.marktstammdatenregister.de/Gesamtdatenexport_20241231_24.2.zip
3    2024-12-30   24.2       current      https://download.marktstammdatenregister.de/Gesamtdatenexport_20241230_24.2.zip
...
================================================================================
Total: 30 downloads available
================================================================================

Options:
1. Select from the list above (enter the number)
2. Cancel

Please enter your choice (1-2): 1
Enter a number (1-30): 1

Selected: 20250103 (Version 24.2, Type: current)
Starting the Download from marktstammdatenregister.de.
...
Download is finished. It took 120 seconds.
MaStR was successfully downloaded to /path/to/xml_download.

2. Programmatic Usage

from open_mastr import Mastr
from open_mastr.xml_download.utils_download_bulk import get_available_download_links

# Get available downloads
links = get_available_download_links()

# Filter for specific criteria
historical_links = [link for link in links if link['type'] == 'stichtag']
recent_links = [link for link in links if link['date'] >= '20241201']

# Download specific date
db = Mastr()
db.download(date="20241215")

# Or download with custom URL
if historical_links:
    db.download(url=historical_links[0]['url'])

3. Backward Compatibility

All existing functionality is preserved. The following code continues to work exactly as before:

from open_mastr import Mastr

# All existing usage patterns work unchanged
db = Mastr()
db.download()  # Downloads today's data (default behavior)
db.download(date="today")  # Downloads today's data
db.download(date="20230101")  # Downloads specific date if available locally
db.download(method="API")  # Uses API method

nesnoj · 2025-12-17T09:03:42Z

Nice feature, thank you @fharookshaik !
Could you please add a changelog entry and appropriate tests?

… readability

fharookshaik · 2025-12-17T18:32:10Z

Commits Overview

This PR introduces interactive download functionality and includes important optimizations to improve robustness and maintainability.

Commit 1: "Add tests for interactive download functionality"
e4cddae - Wed Dec 17 13:15:59 2025 +0100

Added comprehensive test coverage for the new interactive download functionality.

Files Changed

test_interactive_download.py (NEW FILE - 146 lines).
CHANGELOG.md - Added entry for new feature.

Test Cases Created

Core Function Tests:

test_get_available_download_links() - Tests fetching and parsing of download links from MaStR website.
test_list_available_downloads() - Tests formatted output display of available downloads.

Interactive Selection Tests:

test_select_download_date_valid_selection() - Tests user selection of valid download options.
test_select_download_date_cancel()- Tests user cancellation of download selection.

Integration Tests:

test_mastr_download_interactive()- Tests main download method with interactive selection.
test_mastr_download_interactive_cancel() - Tests download cancellation flow.
test_mastr_browse_available_downloads() - Tests browsing available downloads functionality.

Key Features Tested

URL pattern matching for current and historical exports.
Date parsing and version extraction.
User input handling and validation.
Error handling for invalid selections.
Integration with existing download pipeline.

Commit 2: "Refactor date parsing in download_xml_Mastr to improve robustness and readability"
ecd8ba0 - Wed Dec 17 19:04:55 2025 +0100

Optimized the download_xml_Mastr function to fix logic bugs, improve performance, and enhance code maintainability.

Files Changed

utils_download_bulk.py - A minor optimization _parse_date_stringfunction insidedownload_xml_Mastr`to better handle the bulk_date_string param.
mastr.py - Minor parameter fix.
CHANGELOG.md - Added changelog entry.

Test Results

======================================== test session starts ========================================
collected 7 items                                                                                    

tests/test_interactive_download.py::test_get_available_download_links PASSED                   [ 14%]
tests/test_interactive_download.py::test_list_available_downloads PASSED                       [ 28%]
tests/test_interactive_download.py::test_select_download_date_valid_selection PASSED           [ 42%]
tests/test_interactive_download.py::test_select_download_date_cancel PASSED                    [ 57%]
tests/test_interactive_download.py::test_mastr_download_interactive PASSED                     [ 71%]
tests/test_interactive_download.py::test_mastr_download_interactive_cancel PASSED              [ 85%]
tests/test_interactive_download.py::test_mastr_browse_available_downloads PASSED               [100%]

========================================= 7 passed in 0.03s ==========================================

Related Issues

Issue Add interactive download functionality to browse and select MaStR export dates #696 : Add interactive download functionality for MaStR date selection
Issue Currently no one can bulk download older files due to a missed TODO in download_xml_Mastr #695 : Include bulk date string in download process

*Please note that the test cases were generated with the help of LLM's. The working functionality is manually verified.

Feel free to let me know incase of any improvements.

Regards,
fharook

Simon-Will · 2025-12-21T14:26:06Z

open_mastr/xml_download/utils_download_bulk.py

+            return None, None
+
+        else:
+            print("Invalid choice. Please enter 1, 2, or 3.")


This looks like some left-over of a previous version. 3 is not an available choice, right?

Correct @Simon-Will, Thanks for noticing! Let me remove that and recommit with the removed code!

@Simon-Will I have pushed the correction. Please review it and let me know if something's odd. I am happy to help! :)

…choices

Simon-Will · 2026-01-15T11:32:12Z

There's one more thing I'm thinking about here: The returned list of available downloads includes these little dicts of date and filename. The date is formatted as a string. I would prefer if it were a datetime.date object instead and if the Mastr.download function also accepted a datetime.date object.

FlorianK13 · 2026-01-15T13:10:56Z

There's one more thing I'm thinking about here: The returned list of available downloads includes these little dicts of date and filename. The date is formatted as a string. I would prefer if it were a datetime.date object instead and if the Mastr.download function also accepted a datetime.date object.

I would prefer to have this as a seperate issue and PR - allowing datetime objects is a change that affects multiple places in the codebase I suppose.

FlorianK13 · 2026-02-04T11:52:16Z

@fharookshaik How is your status here? Did you find the time to look at the proposed changes yet?

Simon-Will · 2026-02-05T13:12:25Z

@fharookshaik How is your status here? Did you find the time to look at the proposed changes yet?

Tbh, I don't think there's any proposed change anymore. 😅 @fharookshaik has added tests, a changelog entry and the tiny fixes I requested. If we don't want to switch from date in string format to date as date object, this is ready to be merged.

In fact, my branch for #516 / #718 is rebased onto this one here already.

@fharookshaik, I think it's a great feature by the way! So thanks a lot for contributing it. Sorry that we haven't been super fast here; we all have a lot at our hands.

fharookshaik · 2026-02-05T14:24:14Z

@Simon-Will @FlorianK13 Happy to contribute!

On a little side note, I have been working on a tiny project called MaStr_Visualizer, a simple containerized web application to better understand the energy unit distribution nation-wide. Initially, I have concentrated more on the architecture and backend wise, there are not much statistics to be explored on the frontend. I am planning to introduce more stats in the later time. Feel free to checkout the project. If it is noteworthy to be displayed under Examples of Usage in this repository, feel free to do so!

GitHub Link: https://github.com/fharookshaik/MaStr_Visualizer
Blog Post: https://fharook.vercel.app/posts/mastr_visualizer/

FlorianK13 · 2026-02-05T15:00:05Z

Yeah, sorry for the late replies on this PR @fharookshaik - I'd say we merge then, also to not delay the work of you @Simon-Will any further.

FlorianK13 · 2026-02-05T15:00:59Z

Tests are failing as usual, since API key secrets are not revealed to PRs from forked repositories. Someone really needs to fix this 😅

FlorianK13 · 2026-01-15T15:26:26Z

open_mastr/xml_download/utils_download_bulk.py

-    url_time = dt.strptime(bulk_date_string, "%Y%m%d").date().timetuple()
-    url = gen_url(url_time)
+    # Helper function to convert date string to time.struct_time
+    def _parse_date_string(date_str):


Could you reuse parse_date_string() from helpers.py?

FlorianK13 · 2026-01-15T15:33:16Z

open_mastr/mastr.py

+        >>> # User can then choose a date and download with:
+        >>> # db.download(select_date_interactively=True)
+        """
+        from open_mastr.xml_download.utils_download_bulk import list_available_downloads


Suggested change

from open_mastr.xml_download.utils_download_bulk import list_available_downloads

This is better imported at the top of the file I guess.

FlorianK13 · 2026-01-15T15:34:04Z

open_mastr/mastr.py

 # import xml dependencies
 from open_mastr.xml_download.utils_download_bulk import (
    download_xml_Mastr,
+    select_download_date,


Suggested change

select_download_date,

select_download_date,

list_available_downloads,

FlorianK13 · 2026-02-05T15:02:05Z

@fharookshaik yeah, thanks for your first contribution 🎉

Add interactive download functionality for MaStR date selection OpenE…

dfd0cf2

…nergyPlatform#696

fharookshaik mentioned this pull request Dec 16, 2025

Add interactive download functionality to browse and select MaStR export dates #696

Closed

1 task

nesnoj mentioned this pull request Dec 17, 2025

Tests in PR workflow runs fail across forks due to missing MaStR credentials #700

Open

fharookshaik and others added 3 commits December 17, 2025 13:15

Add tests for interactive download functionality

e4cddae

Merge branch 'develop' into feature-696-add-interactive-download

0f2876f

Refactor date parsing in download_xml_Mastr to improve robustness and…

ecd8ba0

… readability

Simon-Will reviewed Dec 21, 2025

View reviewed changes

Fix prompt message in select_download_date function to reflect valid …

8c4e56c

…choices

Simon-Will mentioned this pull request Jan 4, 2026

516: Dynamically create database tables based on XSD files #684

Closed

18 tasks

Merge branch 'develop' into feature-696-add-interactive-download

c291b37

FlorianK13 mentioned this pull request Jan 22, 2026

Remove output_dir function #689 #717

Draft

5 tasks

Simon-Will mentioned this pull request Feb 3, 2026

516 dynamic tables and columns #718

Open

20 tasks

Simon-Will approved these changes Feb 5, 2026

View reviewed changes

FlorianK13 approved these changes Feb 5, 2026

View reviewed changes

FlorianK13 merged commit 0631a94 into OpenEnergyPlatform:develop Feb 5, 2026
0 of 9 checks passed

	select_download_date,
	select_download_date,
	list_available_downloads,

Conversation

fharookshaik commented Dec 16, 2025 • edited by nesnoj Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of the discussion

Type of change (CHANGELOG.md)

Added

Updated

Removed

Workflow checklist

Automation

PR-Assignee

Reviewer

Uh oh!

fharookshaik commented Dec 16, 2025

Uh oh!

nesnoj commented Dec 17, 2025

Uh oh!

fharookshaik commented Dec 17, 2025

Uh oh!

Simon-Will Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

fharookshaik Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

fharookshaik Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Simon-Will commented Jan 15, 2026

Uh oh!

FlorianK13 commented Jan 15, 2026

Uh oh!

FlorianK13 commented Feb 4, 2026

Uh oh!

Simon-Will commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fharookshaik commented Feb 5, 2026

Uh oh!

FlorianK13 commented Feb 5, 2026

Uh oh!

FlorianK13 commented Feb 5, 2026

Uh oh!

FlorianK13 Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

FlorianK13 Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

FlorianK13 Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FlorianK13 commented Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fharookshaik commented Dec 16, 2025 •

edited by nesnoj

Loading

Simon-Will commented Feb 5, 2026 •

edited

Loading