Build middleware adding custom build flags to one source file fails to build on Windows

[willmmiles]

PROBLEM DESCRIPTION

The path shortening code in the Arduino framework breaks the build if a middleware is used to set build flags on a per-source-file basis, as per the PlatformIO middleware example. This seems to be because it does not handle build sources that have already been promoted to SCons Object objects.

TO REPRODUCE

I have uploaded a minimal example based on the PlatformIO middleware documentation to https://github.com/willmmiles/MiddlewareExample

The example has two environments: one that uses the upstream platform-espressif32 and one that uses this repo.

Attempting to build the 'pioarduino' environment in the example produces:

*** While building `['C:\\Users\\Will\\Projects\\MiddlewareExample\\.pio\\build\\espressif_pioardunio\\src\\http_file.cpp.o']' from `['C:\\Users\\Will\\Projects\\MiddlewareExample\\.pio\\build\\espressif_pioardunio\\src\\http_file.cpp.o']': Don't know how to build from a source file with suffix `.o'.  Expected a suffix in this list: ['.c', '.m', '.C', '.cpp', '.cc', '.cxx', '.c++', '.C++', '.mm', '.s', '.asm', '.ASM', '.spp', '.SPP', '.sx', '.S'].
File "C:\Users\Will\.platformio\platforms\espressif32\builder\frameworks\arduino.py", line 806, in apply_include_shortening

EXPECTED BEHAVIOUR

The MRE should build successfully in the pioarduino env on Windows.

ADDITIONAL CONTEXT

My original use case for file-specific build flags was to pass release metadata only to a single source file, so as to avoid breaking build caching for the entire tree when the metadata was updated. I had originally intended to use a layered system with multiple scripts each handling individual tags, but I could not find a way to add flags to a single file from multiple independent middlewares. The only way I found to supply custom flags to exactly one file was to explicitly call env.Object() on that file; and once that was done, I wasn't able to figure out how to update them further in a second middleware call. In the end, I was forced to do all the metadata handling in one place to work around this.

If you have any ideas on how to approach allowing multiple middlewares to do per-file env changes, I'd be very interested in hearing them!

(Please, remember to close the issue when the problem has been addressed)

[TD-er]

What do you want to do?
I used this feature also before, to define some compile-time defines, like the filename of the build and some internally used short-hand form build date (nr. days since X based on the build date) and Git description.

My work-around now is to recreate a header file which is only included once to not trigger a complete rebuild of all.

See: https://github.com/letscontrolit/ESPEasy/blob/mega/tools/pio/generate-compiletime-defines.py
For ESP8266 I'm still using the 'old' method.

[Jason2866]

I see the problem. When removing the path shortening the project will not compile in general with Windows. The Windows path / command line length limit will happen. In other words, you have to find a way without a middleware or Windows support needs to be dropped for the project.

[qniens]

I see the problem and have no solution for. When removing the path shortening the project will not compile in general with Windows. The Windows path / command line length limit will happen. In other words, you have to find a way without a middleware or Windows support needs to be dropped for the project.

Please don't drop Windows support 😅

[Jason2866]

@willmmiles Please try this one. It should work.

platform = https://github.com/pioarduino/platform-espressif32.git#fix_win_middleware

[willmmiles]

Seems to work for my test cases! That's a clever solution.

[willmmiles]

What do you want to do? I used this feature also before, to define some compile-time defines, like the filename of the build and some internally used short-hand form build date (nr. days since X based on the build date) and Git description.

My work-around now is to recreate a header file which is only included once to not trigger a complete rebuild of all.

I considered writing out a header as a workaround; I've used that approach in other build systems, and it tended to interact poorly with the build system's dependency analysis -- rewriting it on every build forced the system to rebuild the objects even if they hadn't changed as the file timestamp was touched. That can in turn be mitigated by validating the contents before rewriting the file, but it adds a fair bit of complexity. In all cases, using a header resulted in slowing down the build -- it was more IO for the build engine to check that file's status and contents, even if it only ended up getting rebuilt rarely. (To be fair, calling git describe in the build setup is very useful and much more costly if it has to do a dirty check, so whether or not that extra IO is worth the expense is definitely situational. Still, my experience is that faster builds enable faster development, so even small gains are not to be discarded without consideration.)

[TD-er]

That's why I mentioned to only include that generated header file once (in a .cpp file obviously).
This way only that single .cpp file wil be recompiled.

[Jason2866]

Fixed #358