Introduction of returns to our system

[snregales]

Introduction of returns to our system. Returns uses railway functions methodology in order to deal with None and Exceptions.

Goal of this PR is to integrate implemented library function with the API while handling all the errors

• Breakdown lib function into more digestible functions
• assemble them in a pipeline
• make public functions safe (containerize the output)
• log unsafe (failure) behavior
• Integrate the safe functions into the API
• Update tests (counts at each point)

[SimoneAriens]

after my comments are resolved, good to merge!

[Raytesnel]

LGTM

[cfs-data]

I don't think we should merge the subsampling and parsing step, since we may want to subsample conditionally on the result of the parsed data (e.g. if the parsed image is larger than X pixels we need to subsample).

My initial idea was something like this:

scan_image = load_scan_image(filepath="/some/filepath")
if scan_image.width > MAX_SIZE:
    scan_image.subsample(step_size=...)  # in-place subsampling
    scan_image = scan_image.subsample(step_size=...)   # or return a new object

Unless we are sure we will never have to do that?

[Raytesnel]

i think what he wanted here. (if that is the case i can agree) that this load_scan_image will be a top level callable. therefore the @impure_safe at the top of the function.
so in the end this function will be split up in different sup functions to load a scan_image

so in the end, to top level called only needs to think about loading a scan image. configs should handle the rest.

so software is not fixed, so i can't garentee you the we will never have to do that. But for now i think the logic here is some babysteps to make eventually here a pipeline for loading a file.

[SimoneAriens]

these are two separate steps, which i would expect in parse_scan_pipeline because loading and subsampling an image are distinct

return run_pipeline(
        scan_file,
        load_scan_image,
        partial(subsample_image, **parameters.as_dict(include={"step_size_x", "step_size_y"}), 
        error_message=f"Failed to parsed given scan file: {scan_file}",
    )

[Raytesnel]

yhea, i didnt saw parse_scan_pipeline. i tought this was is first step to make later when there is more a pipeline. but he already has one. il fix it.

[cfs-data]

Some minor changes requested, but overall looks good! :)

[github-actions]

Diff Coverage

Diff: origin/main..HEAD, staged and unstaged changes

• packages/scratch-core/src/container_models/base.py (95.0%): Missing lines 12
• packages/scratch-core/src/container_models/light_source.py (100%)
• packages/scratch-core/src/container_models/scan_image.py (84.6%): Missing lines 21,26
• packages/scratch-core/src/parsers/init.py (100%)
• packages/scratch-core/src/parsers/loaders.py (100%)
• packages/scratch-core/src/parsers/x3p.py (100%)
• packages/scratch-core/src/renders/init.py (100%)
• packages/scratch-core/src/renders/image_io.py (94.7%): Missing lines 34,53
• packages/scratch-core/src/renders/normalizations.py (100%)
• packages/scratch-core/src/renders/shading.py (100%)
• packages/scratch-core/src/utils/logger.py (94.4%): Missing lines 35,38-39
• src/pipelines.py (100%)
• src/preprocessors/pipelines.py (100%)
• src/preprocessors/router.py (100%)
• src/preprocessors/schemas.py (96.8%): Missing lines 45

Summary

• Total: 351 lines
• Missing: 9 lines
• Coverage: 97%

packages/scratch-core/src/container_models/base.py

Lines 8-16

   8 
   9 
  10 def serialize_ndarray[T: number](array_: NDArray[T]) -> list[T]:
  11     """Serialize numpy array to a Python list for JSON serialization."""
! 12     return array_.tolist()
  13 
  14 
  15 def coerce_to_array[T: number](
  16     dtype: DTypeLike, value: Sequence[T] | NDArray[T] | None

---

packages/scratch-core/src/container_models/scan_image.py

Lines 17-27

  17 
  18     @property
  19     def width(self) -> int:
  20         """The image width in pixels."""
! 21         return self.data.shape[1]
  22 
  23     @property
  24     def height(self) -> int:
  25         """The image height in pixels."""
! 26         return self.data.shape[0]

---

packages/scratch-core/src/renders/image_io.py

Lines 30-38

  30     input_array: ScanMap2DArray, lower: float, upper: float
  31 ) -> ScanMap2DArray:
  32     """Perform min-max normalization on the input array and scale to the [0, 255] interval."""
  33     if lower >= upper:
! 34         raise ValueError(
  35             f"The lower bound ({lower}) should be smaller than the upper bound ({upper})."
  36         )
  37     return (input_array - lower) / (upper - lower) * 255.0

---

Lines 49-57

  49     :param std_scaler: The multiplier for the standard deviation of the data to be clipped.
  50     :returns: A tuple containing the clipped data, the lower bound, and the upper bound of the clipped data.
  51     """
  52     if std_scaler <= 0.0:
! 53         raise ValueError("`std_scaler` must be a positive number.")
  54     mean = np.nanmean(data)
  55     std = np.nanstd(data, ddof=1) * std_scaler
  56     upper = float(mean + std)
  57     lower = float(mean - std)

---

packages/scratch-core/src/utils/logger.py

Lines 31-43

  31 def log_failure(failure_message: str, failure_level: FailureLevel, error: str) -> None:
  32     logger.debug(f"{failure_message}: {error}")
  33     match failure_level:
  34         case FailureLevel.WARNING:
! 35             logger.warning(failure_message)
  36         case FailureLevel.ERROR:
  37             logger.error(failure_message)
! 38         case FailureLevel.CRITICAL:
! 39             logger.critical(failure_message)
  40 
  41 
  42 def _log_io_container(
  43     result: IOResult,

---

src/preprocessors/schemas.py

Lines 41-49

  41         :param exclude: Set of field names to exclude
  42         :param include: Set of field names to include (mutually exclusive with exclude)
  43         """
  44         if exclude and include:
! 45             raise ValueError("Cannot specify both 'exclude' and 'include'")
  46 
  47         fields = set(self.__class__.model_fields)
  48 
  49         if include:

---

[github-actions]

Package	Line Rate	Branch Rate	Health
.	100%	75%	✔
comparators	100%	100%	✔
container_models	94%	100%	✔
parsers	98%	75%	✔
parsers.patches	91%	62%	✔
preprocessors	99%	83%	✔
processors	100%	100%	✔
renders	98%	50%	✔
utils	91%	75%	✔
Summary	97% (462 / 478)	74% (46 / 62)	✔

Minimum allowed line rate is 50%

[snregales]

this is safe not impure_safe, there is no io call here.

[SimoneAriens]

there were some issues regarding the difficulty of debugging without seeing a stack trace, so check this before merging