From 8ab7073affab1cb5b2280ac8fb52ba391e3677a0 Mon Sep 17 00:00:00 2001 From: John Page Date: Mon, 16 Jun 2025 12:43:37 +0100 Subject: [PATCH 1/5] Comments, tidy, refactor, docs --- memex/Class Derscriptions.md | 354 ++++++++++++++++++ .../VehicleInspectionController.java | 34 +- .../kafka/VehicleInspectionKafkaConsumer.java | 2 +- .../memex/model/VehicleInspection.java | 3 +- .../VehicleInspectionRepository.java | 2 +- .../OptimizedMongoLoadRepository.java | 2 +- .../OptimizedMongoLoadRepositoryImpl.java | 2 +- .../MongoDbJsonStreamingLoaderService.java | 9 +- .../memex/util/AnnotationExtractor.java | 1 - .../memex/{model => util}/DeleteFlag.java | 2 +- .../GlobalExceptionHandler.java | 2 +- .../memex/{model => util}/UpdateStrategy.java | 2 +- .../resources/application-test.properties | 4 +- 13 files changed, 388 insertions(+), 31 deletions(-) create mode 100644 memex/Class Derscriptions.md rename memex/src/main/java/com/johnlpage/memex/{model => util}/DeleteFlag.java (92%) rename memex/src/main/java/com/johnlpage/memex/{controller => util}/GlobalExceptionHandler.java (96%) rename memex/src/main/java/com/johnlpage/memex/{model => util}/UpdateStrategy.java (92%) diff --git a/memex/Class Derscriptions.md b/memex/Class Derscriptions.md new file mode 100644 index 0000000..2d041ce --- /dev/null +++ b/memex/Class Derscriptions.md @@ -0,0 +1,354 @@ +# Intro + +This document outlines the purpose of the various classes in the Memex examples and any specific features of note in +them. I decided not to use JavaDoc as that ends up quite sterile and often incomplete - therefore this is a separate +document outlining not just the interfaces but the concepts behind them. It is a good starting point to understand +the nature and purpose ot Memex. + +Memex comes as a working example application with most reusable functionality abstracted into reusable templated +classes. If you cannot simply reuse these as they are, then you can derive from them or modify them as needed. + +# Model Classes + +Model classes represent the data objects used by your application; they also generally map to the objects you persist +in the database and read and write as JSON from services. + +## VehicleInspection.java + +This represents the results of a manual inspection of a specific vehicle at a point in time by one or more mechanics to +ensure it is safe for road use. It is the core entity in the demonstration code and is used to store data from the +UK Vehicle and Operator Service Agency (VOSA). This inspection data is published by VOSA +at [data.gov.uk](https://www.data.gov.uk/dataset/e3939ef8-30c7-4ca8-9c7c-ad9475cc9b2f/anonymised_mot_test). + +_VehicleInspection_ is annotated as a _@Document_, the MongoDB equivalend of a JPA _@Entity_ annotation. It uses +Lombok to avoid boilerplate for getters, setters and constructors. It is a @Data rather than a @Value type, although +it could be either, and various places in the rest of the Memex source show how to deal with both mutable and +immutable models. + +_VehicleInspection_ demonstrates mapping a class member to database field with a different name using the _@Field_ +annotation. + +_VehicleInspection_ demonstrates using Java Bean Validation (JSR-380, JSR-303) to constrain a field, in this case using +@Min on the +vehicle engine capacity, it's set to one(1) as there are some vehicles wutgh 9cc engine capacities, according to the +data! + +It demonstrates the use of a _@Transient_ (Do not store in DB) field flagged as _@DeleteFlag_, this is used to annotateœ +that a +record should be deleted from the databases, if this metadate field is set to true then rahter than load or update the +data Memex will remove it in its bulk loader. + +__Important__ this clas shows the use of _@JsonAnySetter_ and _@JsonAnyGetter_ annotations, these annotations inform +_Jackson_ what to do with any fields in incoming JSON which do not map to members in the class. Handling these in a +traditional RDBMS is hard, and it's normally considered an error but often data can change, or some parts of the +document may be undefined or subject to change. With MongoDB, we can have _Jackson_ map these to an embedded Document ( +Represented as a HashMap in Java) so they are saved, returned and also queryable using the native MngoDB queries +although not with the auto-generated repository queries. This mechanism is one way to get more from flexibility MongoDB +than you get from a traditional RDBMS. + +## Vehicle.java + +Vehicle is a basic model class used to show how a field in one model (VehicleInspection) can be another class and +how MongoDB seamlessly stores this as a nested object in the database. Unlike an RDBMS, this would not be stored in +another table and linked normally, although you can configure it to do so if you need to. + +## DocumentHistory.java + +DocumentHistory is used to store a history of changes to any given record over time; it is a generic class that can +store +field by field modifications for any entity; there is no specific VehicleInspectionHistory — although the type of record +it applies to is helps as a string value within it. Although queryable, it's used more internally in Memex and acesesed +using methods such as the asOf query mechanism. + +# Repository Classes + +## VehicleInspectionRepository.java + +This is a basic and typical repository provided as an example, it has a number of commented out examples of how to +define customer +repository access methods to find data by various fields what makes it more interesting is the additional set of base +repositoriesit is derived from. As well as the normal MongoRepository from Spring Data MongoDB, it also includes the +following base +classes + +### OptimizedMongoLoadRepository.java + +This Interface and it's implmenetation provides two mecahnisms for loading data into MongoDB, unlike Save() and +SaveAll() in the standard JPA/Spring Data interface these optimise writes using minimal calls to the server, +transactions and sophiticated, expression-based updates to prvide up to 200X faster performance as well as trigger +mechanisms and access to "What changed" inside a transactional context. + +```shell + BulkWriteResult writeMany( + List items, + Class clazz, + InvalidDataHandlerService invalidDataHandlerService, + UpdateStrategy updateStrategy, + PostWriteTriggerService postTrigger) + throws IllegalAccessException; + + CompletableFuture asyncWriteMany( + List items, + Class clazz, + InvalidDataHandlerService invalidDataHandlerService, + UpdateStrategy updateStrategy, + PostWriteTriggerService postTrigger); + ``` + +This is provided as both a synchronous and asynchronous version (Both using the Java Sync library, not reactive Java), +as a database write is a network call and can therefore take potential milliseconds there is usually no need to wait for +it to complete when more data is waiting to send. + +Each takes a list of updated or new items of the related Model class (this can be a list of one to replace Save(), +although replacing a single Save() is not much faster it does open up all the additional capabilities. ) , It also needs +to to send the Class of the Model as Java does not let you infer that through a template. + +It take and inbvalidDataHandlerService - which lets you define what to do for any documents that could not be processed. + +It takes an Update strategy, whether to Insert, OverWrite (Replace) , Update or "Update and Return a change document" +for each of the items in the list. All of these will default to inserting if the document does not exist and will delete +if the _@DeleteFlag_ field is true + +PostWiteTriggerService postTrigger - if non null is an interface with a finction that is called after updateing the data +in the database , this function is passed a means to retrieve the nature of all the changes, ddown to a field level and +is used to write change histories as well as any other post-update processing required. It is akin to an RDBMS trigger +but is written in client side Java. + +Each returns a standard MongODB BulkWrite Result which provides information on the number and type of operations +completed. + +### OptimizedMongoQueryRepsitory.java + +_mongoRepository_ provides all the typical Spring/JPA query faciliites - auto generated queries for the form +FindByThisandThat() or FindAllByFieldLessThan() etc. You can also use _@Query_ annotataions and Query by both Exampel +and Criteria as you woudl normally do. You can also use the Native MongoDB driver to construct queries as code using +fluent query building classes. + +What _OptimizedMongoQueryRepsitory_ brings is the ability to run queries defined at runtime and passed form an +application. In a similar maanner to GraphQL we may not always want to create an explicit endpoint for every possible +queriy our API supports, if a User Interface is some form of query builder, perhaps a form with many optional fields +then we may want the UI designer to have control over the queries run. We might also want to offer both Datbase Querys +and Lucene full text index queries - Lucene indexing is available in MongODB atlas and gives a powerful way to perform +fuzzy, best-match, full texts queries among others. + +This Interface, and it's underlying implemention provide the following three functions + +```java + +List mongoDbNativeQuery(String jsonString, Class clazz); + +List atlasSearchQuery(String jsonString, Class clazz); + +int costMongoDbNativeQuery(String jsonString, Class clazz); + +``` + +The first two take a description, in JSON of the required Query, PRojection, Order and Limits - it's very much a general +purpose query function to allow arbitrary queries . Qirty mongoDBNative (QUerying using the MognoDB database and +indexes) or atlasSearch - Querying using any degined Atlas Search Lucenes indexes. + +The final method, which you can and shoudl call from your service before passing a query to mongoDbNativeQuery returns +a 'cost' between 1 and 500 for that query, where 1 represents a fullyindexed and index covered query and 500 represents +a collection scan. This score indicates how much resouce the query will take and you can use that to determine how to +proceeed. You may reject queries that will be harmful, you may allow and log them, you may pass them to a secondary or +analytic replica or you may modify them, for example adding a limited and indexed date range so they only impact a +recent set of data. Allo fo this you have the tools to do in your selvice layer. + +### OprimisedMongoDownstreamRepository.java + +When you want to perform a large extraction of data from a database, through Spring and out as JSON then there is a +simple and objecous solution - perform a query or aggregation to retirve a Stream or the Object model and then convert +those to JSON to stream them out. This is the simplest way to performa a large scale extraction, when we do thsi however +we have to create large numbers of temporary row (Document) objects and Model objects in order to the render them as +JSON. In the case of MongoDB this is a conversion from BSON (The native internal and network format) to Document +classes ( HashMap based) to Model Classes to JSON. + +If you are extracing a few MB now and then this is OK but what if yo want to regularly and quickly extract Gigabytes of +data - you can shortcut a lot fo the process above by telling MongoDB you want JSON, not Objects form the databaee and +this is where the single method in OptimisedMongoDownstreamRepository comes in. + +``` + Stream nativeJsonExtract(String formatRequired, Class modelClazz); +``` + +This is an example it has a single methos to show that instead of `Stream` we can use the much more +efficient `Stream` and in our ser ice and controller simply stream the out to our end used avoiding 99% of +the object createion and garbage colleciton. JsonObject converts from the BSON native format directly to JSON without +creating any intermediate objets. + +To use this we do need to explicitly tell mongodb exactly what our JSON shodul look like, and what we pass in is a +String representation in JSON of the required format, the function performs a find() retriueving the whole collection +and applying this projections. The purpose of this is not to simply use as is but to demonstrate how much more efficient +large scale data downstrewming as JSON (or BSON) can be when you avoid all the SPring obhjectmapping. + +### MongoHistoryRepository.java + +MongoHistoryRepository is used to read one or multipel documents but to apply the changes made to them over time in +reverse order to revert to older versions. It has two methods although the code can be extended and reused for more +sophiticaed cases including querying historical data. + +```java +Stream GetRecordByIdAsOfDate(I recordId, Date asOf, Class clazz); + +Stream GetRecordsAsOfDate(Criteria criteria, Date asOf, Class clazz); +``` + +This assumes the documents, and any changed version sof them were ingested using the OptimizedLoadRepository with the +record history option enabled. These methods take the base document (latest version) perform a $lookup (JOIN) to fetch +the historical changes and then ,merge tthose to wind the document back to it's prior form. + +# Services + +Services sit between Controlers ( Handing interfaces to the service) and Repositories (Interfacing with an underlying ] +database). Services contain business logic and processing that is specific to your business and also how you intend to +use the data - a very simple controller could acess the repository directly but this is not good practise. In the Memex +examples we do not access the repository directly from the controller - only via services, even if in some cases those +service classes perform no additonal processing or logic. + +## MongoDbJsonStreamingLoaderService.java + +The most significant service in Memex is the StreamingJsonLoaderService - this service parses an incoming JSON stream +into the associated Model class and every 200 documents it uses OptimizedMongoLoadRespository to load that batch of +documents as an asynchronous process whilst it parses and processes the next 200 documents. This Stream processing and +parallelism is at the haeart of high performance data loading in Memex. + +It can accept data as a JSON array or simply multiple JSON obejects, optionaly seperated by a newline, it work by +finding the first object start token `{` then finding the corresponding close token `}` and treating what is between +them as and object to load - it then finds the next open token. + +After parsing each document to a model object, the _StreamingJsonLoader_ will pass that object to a _preWriteTrigger_ +class if one is defined to allow for any modifications needed before writing to MongoDB. + +## PreWriteTriggerService.java + +This class defines a single overrideable functiion which is called by StreamingJsonLoaderService +( May be moved to OptimizedLoadRepositor soon). This is called for each Model object just before writing it to +MongoDB and can be used to augment the data - the correct method to override depends on whather you ar e using mutable +or immutable models. The default class does nothing to the object + +```java +public void modifyMutableDataPreWrite(T document) { +} + +public T newImmutableDataPreWrite(T document) { + return document; +} +``` + +## PostWriteTriggerService.java + +This defines a function to be called after a batch of documents have been written to MongoDB inside the transaction +used to write them and before it commits. The dfault function does nothing but can be overridder in a derived class. + +```java + public void postWriteTrigger( + ClientSession session, + BulkWriteResult result, + List records, + Class clazz, + ObjectId updateId) + throws IllegalAccessException { +} +``` + +This function recieves the transactional session used for the write. By using that session inside this function you are +able to see operations that have not yet been comitted. +It recieves the BulkWriteResult object descring how many of each operation took place and the id values for +anything that was inserted. +It revieces the batch of rercords that were presented for updating, deleteing or inserting +It also inclues an updateId value, this can be used to Query MongoDB to retireve the documents as they are post update +if needed. + +This is derived from the greate the HistoryTriggerService that records changes to the data transactionally. + +## HistoryTriggerService.java + +HistoryTriggerService is a tempalted reuasable class, extending PostWriteTriggerService to generate a change hostiry +of any data updated. + +Where a model is stored in collection X, for example `vehicleinspaeciton` this class writes DocumentHistory objects to +`vehicleinspection_history`. These are a reverse delta history showing field values in previous versions of the document +at each stage. To use them you need to take the latest version of the data and apply in reverse order - this is done by +MongoHistoryReportity for you for functions such as asOf (a point in time). + +When a record is first inserted, then an entry is recorded in the history table just saying this is when the record was +first inserted, it does not contain any of the content. + +When a record is modified, then the modified fields previous version are stored alone with an updateid and timestamp + +When a record is deleted then , currently the whole record is stored in the trigger service and removed from the +principle colleciton. +This could be changed to delete the record from the principle collection AND to remove all hisotic versions very easily +but +many users prefer to archive the data or retain a hisotoyry of its existence and deleteion. It woudl also be simple to +have a history that +retained the creation and deleteion records but no record of the content to modifications. + +## InvalidDataHandlerService.java + +This class frines a singel function to be called before a document is loaded into the database if that documnt fails +java bean validation Java Bean Validation. +It is passed the document and a list of ways it fails validation - the default behaviour is to log as a warning. +This function returns a boolean which determines is the data shoudl be loaded anyway or skipped in the data +loading/updaing process. +It shoudl be used to populate a data file of things to be reloaded - it is designed to allow the rest of the load to +complete and errors or exceprions to be caught. + +```java + public boolean handleInvalidData( + T document, Set> violations, Class clazz) { + LOG.warn( + "Invalid data detected ({} violations) in document, but no explicit handler provided, discarding.", + violations.size()); + return false; +} +``` + +## MongoDbPreflightCheckService.java + +This Java Bean based on ApplicationRunner demonstrates one good practise with services built on MongoDB - what the +server validate any database prerequisites, +be they permissions, version or the exisstence of database objects includeing collecitons, indexes, search indexes and +schema validators. + +The way things like indexes are ppopulated through environment can differ based on the system, srchitecture and business +requirements. In a +development environment it is normal to generate indexes that are not present in the preflight service, however +in production, given creation of an index may have an impact on a running system it is better to build index via +management toolling and failover orchestration. In that case pre-flight shoudl fail until the index already exsts. + +By default this is configured to create indexes and search indexes for ease fo developemtn but warn in the logs this is +not for priduciton. + +## VehicleInspectionPreWriteTriggerService.java + +This is provided as an example of a pre-write trigger which is passed makes small changes to the incoming data to +makle it easy to test updates, you can load the same file but this will modify some details to cause an update and +relevant history entry if required. It also serves to demonstrate what a preWrite trigger looks like both for +mutable and immutable data + +## VehicleInspectionQueryService.java + +The class OptimizedMongoQueryRepository provides an endpoint supporting any MongoDB query defined by an EJSON string, +this is intended to allow front end clients to define their own queries and avoid the requiremetn to create an +endpoint for each new frontend feature - especially where some for of GUI query creation is available. + +The OptimizedMongoQueryRepository also provides a cost estimator function for queries to determine if they will take +a few or a lot fo resources from the datbase server. This Service takes each incoming query, performs a cost +estimation (This is efficent and cached in the estimator) logs the cost and runs it anyway - this si where you woudl +add business logic to deny, log , modify or redirect expensive queries to a secondayr server. + +This class shows how to expose a simple hard-coded query and a Spring query-by-example to the controller class. + +## VehicleInspectionXXXXService.java + +Where XXXXX is one of: DownStream,History,HistoryTrigger,InvalidDataHandler,JsonLoader then the service is thin +wrapper over the generic service or to provide a simple call to the repository with space for additional logic. + +# Controllers + +Memex includes an example _@RestController_ to show how the features are acessed, a RestController was chosen as it +is the simplest to test and understand but other controller types are equally usable in your own code. + +## VehicleInspectionController.java + diff --git a/memex/src/main/java/com/johnlpage/memex/controller/VehicleInspectionController.java b/memex/src/main/java/com/johnlpage/memex/controller/VehicleInspectionController.java index 6ef1d06..d419369 100644 --- a/memex/src/main/java/com/johnlpage/memex/controller/VehicleInspectionController.java +++ b/memex/src/main/java/com/johnlpage/memex/controller/VehicleInspectionController.java @@ -3,7 +3,7 @@ import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.ObjectMapper; import com.johnlpage.memex.dto.PageDto; -import com.johnlpage.memex.model.UpdateStrategy; +import com.johnlpage.memex.util.UpdateStrategy; import com.johnlpage.memex.model.Vehicle; import com.johnlpage.memex.model.VehicleInspection; import com.johnlpage.memex.repository.VehicleInspectionRepository; @@ -164,9 +164,10 @@ public ResponseEntity atlasSearchQuery(@RequestBody String requestBody) public ResponseEntity streamJson() { return ResponseEntity.ok() - .contentType(MediaType.APPLICATION_JSON) - .body(outputStream -> - writeDocumentsToOutputStream(outputStream, downstreamService.jsonExtractStream())); + .contentType(MediaType.APPLICATION_JSON) + .body( + outputStream -> + writeDocumentsToOutputStream(outputStream, downstreamService.jsonExtractStream())); } /** @@ -195,10 +196,13 @@ public ResponseEntity streamJsonNative() { """; return ResponseEntity.ok() - .contentType(MediaType.APPLICATION_JSON) - .body(outputStream -> { - try (BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(outputStream); - Stream stream = downstreamService.nativeJsonExtractStream(formatRequired)) { + .contentType(MediaType.APPLICATION_JSON) + .body( + outputStream -> { + try (BufferedOutputStream bufferedOutputStream = + new BufferedOutputStream(outputStream); + Stream stream = + downstreamService.nativeJsonExtractStream(formatRequired)) { boolean isFirst = true; Iterator iterator = stream.iterator(); while (iterator.hasNext()) { @@ -210,19 +214,21 @@ public ResponseEntity streamJsonNative() { isFirst = false; } } catch (IOException e) { - LOG.error("Error during streaming jsonObjects using native mode: {}", e.getMessage()); + LOG.error( + "Error during streaming jsonObjects using native mode: {}", e.getMessage()); } }); } @GetMapping(value = "/inspections/asOf", produces = MediaType.APPLICATION_JSON_VALUE) public ResponseEntity dataAtDate( - @RequestParam(name = "asOfDate") @DateTimeFormat(pattern = "yyyyMMddHHmmss") Date asOfDate, - @RequestParam(name = "id") Long id) { + @RequestParam(name = "asOfDate") @DateTimeFormat(pattern = "yyyyMMddHHmmss") Date asOfDate, + @RequestParam(name = "id") Long id) { return ResponseEntity.ok() - .contentType(MediaType.APPLICATION_JSON) - .body(outputStream -> - writeDocumentsToOutputStream(outputStream, historyService.asOfDate(id, asOfDate))); + .contentType(MediaType.APPLICATION_JSON) + .body( + outputStream -> + writeDocumentsToOutputStream(outputStream, historyService.asOfDate(id, asOfDate))); } private void writeDocumentsToOutputStream( diff --git a/memex/src/main/java/com/johnlpage/memex/kafka/VehicleInspectionKafkaConsumer.java b/memex/src/main/java/com/johnlpage/memex/kafka/VehicleInspectionKafkaConsumer.java index 2dfa30d..29410ac 100644 --- a/memex/src/main/java/com/johnlpage/memex/kafka/VehicleInspectionKafkaConsumer.java +++ b/memex/src/main/java/com/johnlpage/memex/kafka/VehicleInspectionKafkaConsumer.java @@ -2,7 +2,7 @@ import com.fasterxml.jackson.core.JsonFactory; import com.fasterxml.jackson.databind.ObjectMapper; -import com.johnlpage.memex.model.UpdateStrategy; +import com.johnlpage.memex.util.UpdateStrategy; import com.johnlpage.memex.model.VehicleInspection; import com.johnlpage.memex.repository.optimized.OptimizedMongoLoadRepository; import com.johnlpage.memex.service.VehicleInspectionInvalidDataHandlerService; diff --git a/memex/src/main/java/com/johnlpage/memex/model/VehicleInspection.java b/memex/src/main/java/com/johnlpage/memex/model/VehicleInspection.java index b0f7961..403392e 100644 --- a/memex/src/main/java/com/johnlpage/memex/model/VehicleInspection.java +++ b/memex/src/main/java/com/johnlpage/memex/model/VehicleInspection.java @@ -1,6 +1,7 @@ package com.johnlpage.memex.model; import com.fasterxml.jackson.annotation.*; +import com.johnlpage.memex.util.DeleteFlag; import com.johnlpage.memex.util.ObjectConverter; import java.util.Date; import java.util.HashMap; @@ -43,7 +44,7 @@ public class VehicleInspection { Vehicle vehicle; String files; - @Min(49) + @Min(1) Long capacity; Date firstusedate; diff --git a/memex/src/main/java/com/johnlpage/memex/repository/VehicleInspectionRepository.java b/memex/src/main/java/com/johnlpage/memex/repository/VehicleInspectionRepository.java index 2827223..060165c 100644 --- a/memex/src/main/java/com/johnlpage/memex/repository/VehicleInspectionRepository.java +++ b/memex/src/main/java/com/johnlpage/memex/repository/VehicleInspectionRepository.java @@ -25,7 +25,7 @@ public interface VehicleInspectionRepository * * // Find inspections by engine capacity - auto generated query - List findByCapacityGreaterThan(Long engineCapacity); + List findAllByCapacityGreaterThan(Long engineCapacity); // Custom query to find vehicle inspections by vehicle make and model @Query("{ 'vehicle.make': ?0, 'vehicle.model': ?1 }") diff --git a/memex/src/main/java/com/johnlpage/memex/repository/optimized/OptimizedMongoLoadRepository.java b/memex/src/main/java/com/johnlpage/memex/repository/optimized/OptimizedMongoLoadRepository.java index 10ed69d..83a7899 100644 --- a/memex/src/main/java/com/johnlpage/memex/repository/optimized/OptimizedMongoLoadRepository.java +++ b/memex/src/main/java/com/johnlpage/memex/repository/optimized/OptimizedMongoLoadRepository.java @@ -1,6 +1,6 @@ package com.johnlpage.memex.repository.optimized; -import com.johnlpage.memex.model.UpdateStrategy; +import com.johnlpage.memex.util.UpdateStrategy; import com.johnlpage.memex.service.generic.InvalidDataHandlerService; import com.johnlpage.memex.service.generic.PostWriteTriggerService; import com.mongodb.bulk.BulkWriteResult; diff --git a/memex/src/main/java/com/johnlpage/memex/repository/optimized/OptimizedMongoLoadRepositoryImpl.java b/memex/src/main/java/com/johnlpage/memex/repository/optimized/OptimizedMongoLoadRepositoryImpl.java index 169bd94..3db8752 100644 --- a/memex/src/main/java/com/johnlpage/memex/repository/optimized/OptimizedMongoLoadRepositoryImpl.java +++ b/memex/src/main/java/com/johnlpage/memex/repository/optimized/OptimizedMongoLoadRepositoryImpl.java @@ -3,7 +3,7 @@ import static com.johnlpage.memex.util.AnnotationExtractor.*; import static org.springframework.data.mongodb.core.query.Criteria.where; -import com.johnlpage.memex.model.UpdateStrategy; +import com.johnlpage.memex.util.UpdateStrategy; import com.johnlpage.memex.service.generic.InvalidDataHandlerService; import com.johnlpage.memex.service.generic.PostWriteTriggerService; import com.mongodb.bulk.BulkWriteInsert; diff --git a/memex/src/main/java/com/johnlpage/memex/service/generic/MongoDbJsonStreamingLoaderService.java b/memex/src/main/java/com/johnlpage/memex/service/generic/MongoDbJsonStreamingLoaderService.java index c3663f3..e4ff4d0 100644 --- a/memex/src/main/java/com/johnlpage/memex/service/generic/MongoDbJsonStreamingLoaderService.java +++ b/memex/src/main/java/com/johnlpage/memex/service/generic/MongoDbJsonStreamingLoaderService.java @@ -5,7 +5,7 @@ import com.fasterxml.jackson.core.JsonToken; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; -import com.johnlpage.memex.model.UpdateStrategy; +import com.johnlpage.memex.util.UpdateStrategy; import com.johnlpage.memex.repository.optimized.OptimizedMongoLoadRepository; import com.mongodb.bulk.BulkWriteResult; import jakarta.annotation.Nullable; @@ -24,10 +24,6 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.stereotype.Service; -import jakarta.validation.ConstraintViolation; -import jakarta.validation.Validation; -import jakarta.validation.Validator; -import jakarta.validation.ValidatorFactory; @Service @RequiredArgsConstructor @@ -35,6 +31,7 @@ public abstract class MongoDbJsonStreamingLoaderService { private static final Logger LOG = LoggerFactory.getLogger(MongoDbJsonStreamingLoaderService.class); + private static final int BATCH_SIZE = 200; private final OptimizedMongoLoadRepository repository; private final ObjectMapper objectMapper; private final JsonFactory jsonFactory; @@ -79,7 +76,7 @@ public JsonStreamingLoadResponse loadFromJsonStream( count++; toSave.add(document); - if (toSave.size() >= 100) { + if (toSave.size() >= BATCH_SIZE) { List copyOfToSave = List.copyOf(toSave); toSave.clear(); futures.add( diff --git a/memex/src/main/java/com/johnlpage/memex/util/AnnotationExtractor.java b/memex/src/main/java/com/johnlpage/memex/util/AnnotationExtractor.java index 52e8eaf..d4a7b19 100644 --- a/memex/src/main/java/com/johnlpage/memex/util/AnnotationExtractor.java +++ b/memex/src/main/java/com/johnlpage/memex/util/AnnotationExtractor.java @@ -1,7 +1,6 @@ package com.johnlpage.memex.util; import com.fasterxml.jackson.annotation.JsonProperty; -import com.johnlpage.memex.model.DeleteFlag; import jakarta.annotation.Nullable; import java.lang.reflect.Field; import java.util.HashMap; diff --git a/memex/src/main/java/com/johnlpage/memex/model/DeleteFlag.java b/memex/src/main/java/com/johnlpage/memex/util/DeleteFlag.java similarity index 92% rename from memex/src/main/java/com/johnlpage/memex/model/DeleteFlag.java rename to memex/src/main/java/com/johnlpage/memex/util/DeleteFlag.java index cbe9da0..dc646d2 100644 --- a/memex/src/main/java/com/johnlpage/memex/model/DeleteFlag.java +++ b/memex/src/main/java/com/johnlpage/memex/util/DeleteFlag.java @@ -1,4 +1,4 @@ -package com.johnlpage.memex.model; +package com.johnlpage.memex.util; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; diff --git a/memex/src/main/java/com/johnlpage/memex/controller/GlobalExceptionHandler.java b/memex/src/main/java/com/johnlpage/memex/util/GlobalExceptionHandler.java similarity index 96% rename from memex/src/main/java/com/johnlpage/memex/controller/GlobalExceptionHandler.java rename to memex/src/main/java/com/johnlpage/memex/util/GlobalExceptionHandler.java index 8f2380c..0c40431 100644 --- a/memex/src/main/java/com/johnlpage/memex/controller/GlobalExceptionHandler.java +++ b/memex/src/main/java/com/johnlpage/memex/util/GlobalExceptionHandler.java @@ -1,4 +1,4 @@ -package com.johnlpage.memex.controller; +package com.johnlpage.memex.util; import org.apache.catalina.connector.ClientAbortException; import org.slf4j.Logger; diff --git a/memex/src/main/java/com/johnlpage/memex/model/UpdateStrategy.java b/memex/src/main/java/com/johnlpage/memex/util/UpdateStrategy.java similarity index 92% rename from memex/src/main/java/com/johnlpage/memex/model/UpdateStrategy.java rename to memex/src/main/java/com/johnlpage/memex/util/UpdateStrategy.java index 80119cb..4b09065 100644 --- a/memex/src/main/java/com/johnlpage/memex/model/UpdateStrategy.java +++ b/memex/src/main/java/com/johnlpage/memex/util/UpdateStrategy.java @@ -1,4 +1,4 @@ -package com.johnlpage.memex.model; +package com.johnlpage.memex.util; /** * This IS used to set whether we are using a simple replace ro a recursive field update to diff --git a/memex/src/test/resources/application-test.properties b/memex/src/test/resources/application-test.properties index 8164069..1566235 100644 --- a/memex/src/test/resources/application-test.properties +++ b/memex/src/test/resources/application-test.properties @@ -1,7 +1,7 @@ #Keep this empty if you want tests to use Mongo test container +#spring.data.mongodb.uri=mongodb://localhost:63650 spring.data.mongodb.uri= -spring.data.mongodb.database=memex +spring.data.mongodb.database=memextest memex.kafkaexmple.enabled=true - memex.test.data.vehicleinspection-testid-range.start=10000 memex.test.data.vehicleinspection-testid-range.end=11000 \ No newline at end of file From bfa7409590efa91b3140feb8a62e6ddc5706d3f3 Mon Sep 17 00:00:00 2001 From: John Page Date: Mon, 16 Jun 2025 13:13:26 +0100 Subject: [PATCH 2/5] Updated Docs --- memex/Class Derscriptions.md | 440 +++++++++++++++++------------------ 1 file changed, 216 insertions(+), 224 deletions(-) diff --git a/memex/Class Derscriptions.md b/memex/Class Derscriptions.md index 2d041ce..b9d8d9f 100644 --- a/memex/Class Derscriptions.md +++ b/memex/Class Derscriptions.md @@ -1,354 +1,346 @@ -# Intro +# Introduction -This document outlines the purpose of the various classes in the Memex examples and any specific features of note in -them. I decided not to use JavaDoc as that ends up quite sterile and often incomplete - therefore this is a separate -document outlining not just the interfaces but the concepts behind them. It is a good starting point to understand -the nature and purpose ot Memex. +This document outlines the purpose of the various classes in the Memex examples and highlights their specific features. +Rather than using JavaDoc, which can be sterile and often incomplete, this separate document explains not just the +interfaces but the concepts behind them. It serves as a good starting point for understanding the nature and purpose of +Memex. Memex comes as a working example application with most reusable functionality abstracted into reusable templated -classes. If you cannot simply reuse these as they are, then you can derive from them or modify them as needed. +classes. If you cannot simply reuse these as they are, you can derive from them or modify them as needed. # Model Classes -Model classes represent the data objects used by your application; they also generally map to the objects you persist -in the database and read and write as JSON from services. +Model classes represent the data objects used by your application. They typically map to the objects you persist in the +database and read/write as JSON from services. ## VehicleInspection.java This represents the results of a manual inspection of a specific vehicle at a point in time by one or more mechanics to -ensure it is safe for road use. It is the core entity in the demonstration code and is used to store data from the -UK Vehicle and Operator Service Agency (VOSA). This inspection data is published by VOSA +ensure it is safe for road use. It is the core entity in the demonstration code and stores data from the UK Vehicle and +Operator Service Agency (VOSA). This inspection data is published by VOSA at [data.gov.uk](https://www.data.gov.uk/dataset/e3939ef8-30c7-4ca8-9c7c-ad9475cc9b2f/anonymised_mot_test). -_VehicleInspection_ is annotated as a _@Document_, the MongoDB equivalend of a JPA _@Entity_ annotation. It uses -Lombok to avoid boilerplate for getters, setters and constructors. It is a @Data rather than a @Value type, although -it could be either, and various places in the rest of the Memex source show how to deal with both mutable and -immutable models. +_VehicleInspection_ is annotated as a _@Document_, which is MongoDB's equivalent of a JPA _@Entity_ annotation. It uses +Lombok to avoid boilerplate code for getters, setters, and constructors. It is a @Data rather than a @Value type, +although it could be either. Various places in the Memex source show how to deal with both mutable and immutable models. -_VehicleInspection_ demonstrates mapping a class member to database field with a different name using the _@Field_ +_VehicleInspection_ demonstrates mapping a class member to a database field with a different name using the _@Field_ annotation. -_VehicleInspection_ demonstrates using Java Bean Validation (JSR-380, JSR-303) to constrain a field, in this case using -@Min on the -vehicle engine capacity, it's set to one(1) as there are some vehicles wutgh 9cc engine capacities, according to the -data! +_VehicleInspection_ demonstrates using Java Bean Validation (JSR-380, JSR-303) to constrain a field. In this case, it +uses @Min on the vehicle engine capacity, set to one (1) because there are some vehicles with 9cc engine capacities +according to the data! -It demonstrates the use of a _@Transient_ (Do not store in DB) field flagged as _@DeleteFlag_, this is used to annotateœ -that a -record should be deleted from the databases, if this metadate field is set to true then rahter than load or update the -data Memex will remove it in its bulk loader. +It demonstrates the use of a _@Transient_ (do not store in DB) field flagged as _@DeleteFlag_. This is used to indicate +that a record should be deleted from the database. If this metadata field is set to true, rather than load or update the +data, Memex will remove it in its bulk loader. -__Important__ this clas shows the use of _@JsonAnySetter_ and _@JsonAnyGetter_ annotations, these annotations inform -_Jackson_ what to do with any fields in incoming JSON which do not map to members in the class. Handling these in a -traditional RDBMS is hard, and it's normally considered an error but often data can change, or some parts of the +**Important**: This class shows the use of _@JsonAnySetter_ and _@JsonAnyGetter_ annotations. These annotations inform +_Jackson_ what to do with any fields in incoming JSON that do not map to members in the class. Handling these in a +traditional RDBMS is difficult and normally considered an error, but often data can change, or some parts of the document may be undefined or subject to change. With MongoDB, we can have _Jackson_ map these to an embedded Document ( -Represented as a HashMap in Java) so they are saved, returned and also queryable using the native MngoDB queries -although not with the auto-generated repository queries. This mechanism is one way to get more from flexibility MongoDB -than you get from a traditional RDBMS. +represented as a HashMap in Java) so they are saved, returned, and also queryable using native MongoDB queries (although +not with auto-generated repository queries). This mechanism is one way to get more flexibility from MongoDB than you get +from a traditional RDBMS. ## Vehicle.java -Vehicle is a basic model class used to show how a field in one model (VehicleInspection) can be another class and -how MongoDB seamlessly stores this as a nested object in the database. Unlike an RDBMS, this would not be stored in -another table and linked normally, although you can configure it to do so if you need to. +Vehicle is a basic model class used to show how a field in one model (VehicleInspection) can be another class and how +MongoDB seamlessly stores this as a nested object in the database. Unlike an RDBMS, this would not be stored in another +table and linked normally, although you can configure it to do so if needed. ## DocumentHistory.java -DocumentHistory is used to store a history of changes to any given record over time; it is a generic class that can -store -field by field modifications for any entity; there is no specific VehicleInspectionHistory — although the type of record -it applies to is helps as a string value within it. Although queryable, it's used more internally in Memex and acesesed -using methods such as the asOf query mechanism. +DocumentHistory is used to store a history of changes to any given record over time. It is a generic class that can +store field-by-field modifications for any entity. There is no specific VehicleInspectionHistory—although the type of +record it applies to is stored as a string value within it. Although queryable, it's used more internally in Memex and +accessed using methods such as the asOf query mechanism. # Repository Classes ## VehicleInspectionRepository.java -This is a basic and typical repository provided as an example, it has a number of commented out examples of how to -define customer -repository access methods to find data by various fields what makes it more interesting is the additional set of base -repositoriesit is derived from. As well as the normal MongoRepository from Spring Data MongoDB, it also includes the -following base -classes +This is a basic and typical repository provided as an example. It has several commented-out examples of how to define +custom repository access methods to find data by various fields. What makes it more interesting is the additional set of +base repositories it is derived from. As well as the normal MongoRepository from Spring Data MongoDB, it also includes +the following base classes: ### OptimizedMongoLoadRepository.java -This Interface and it's implmenetation provides two mecahnisms for loading data into MongoDB, unlike Save() and -SaveAll() in the standard JPA/Spring Data interface these optimise writes using minimal calls to the server, -transactions and sophiticated, expression-based updates to prvide up to 200X faster performance as well as trigger -mechanisms and access to "What changed" inside a transactional context. - -```shell - BulkWriteResult writeMany( - List items, - Class clazz, - InvalidDataHandlerService invalidDataHandlerService, - UpdateStrategy updateStrategy, - PostWriteTriggerService postTrigger) - throws IllegalAccessException; - - CompletableFuture asyncWriteMany( - List items, - Class clazz, - InvalidDataHandlerService invalidDataHandlerService, - UpdateStrategy updateStrategy, - PostWriteTriggerService postTrigger); - ``` - -This is provided as both a synchronous and asynchronous version (Both using the Java Sync library, not reactive Java), -as a database write is a network call and can therefore take potential milliseconds there is usually no need to wait for -it to complete when more data is waiting to send. - -Each takes a list of updated or new items of the related Model class (this can be a list of one to replace Save(), -although replacing a single Save() is not much faster it does open up all the additional capabilities. ) , It also needs -to to send the Class of the Model as Java does not let you infer that through a template. - -It take and inbvalidDataHandlerService - which lets you define what to do for any documents that could not be processed. - -It takes an Update strategy, whether to Insert, OverWrite (Replace) , Update or "Update and Return a change document" -for each of the items in the list. All of these will default to inserting if the document does not exist and will delete -if the _@DeleteFlag_ field is true - -PostWiteTriggerService postTrigger - if non null is an interface with a finction that is called after updateing the data -in the database , this function is passed a means to retrieve the nature of all the changes, ddown to a field level and -is used to write change histories as well as any other post-update processing required. It is akin to an RDBMS trigger -but is written in client side Java. - -Each returns a standard MongODB BulkWrite Result which provides information on the number and type of operations -completed. +This interface and its implementation provide two mechanisms for loading data into MongoDB. Unlike Save() and SaveAll() +in the standard JPA/Spring Data interface, these optimize writes using minimal calls to the server, transactions, and +sophisticated expression-based updates to provide up to 200x faster performance. They also include trigger mechanisms +and access to "what changed" within a transactional context. + +```java +BulkWriteResult writeMany( + List items, + Class clazz, + InvalidDataHandlerService invalidDataHandlerService, + UpdateStrategy updateStrategy, + PostWriteTriggerService postTrigger) + throws IllegalAccessException; -### OptimizedMongoQueryRepsitory.java +CompletableFuture asyncWriteMany( + List items, + Class clazz, + InvalidDataHandlerService invalidDataHandlerService, + UpdateStrategy updateStrategy, + PostWriteTriggerService postTrigger); -_mongoRepository_ provides all the typical Spring/JPA query faciliites - auto generated queries for the form -FindByThisandThat() or FindAllByFieldLessThan() etc. You can also use _@Query_ annotataions and Query by both Exampel -and Criteria as you woudl normally do. You can also use the Native MongoDB driver to construct queries as code using -fluent query building classes. +``` -What _OptimizedMongoQueryRepsitory_ brings is the ability to run queries defined at runtime and passed form an -application. In a similar maanner to GraphQL we may not always want to create an explicit endpoint for every possible -queriy our API supports, if a User Interface is some form of query builder, perhaps a form with many optional fields -then we may want the UI designer to have control over the queries run. We might also want to offer both Datbase Querys -and Lucene full text index queries - Lucene indexing is available in MongODB atlas and gives a powerful way to perform -fuzzy, best-match, full texts queries among others. +This is provided as both synchronous and asynchronous versions (both using the Java sync library, not reactive Java). +Since a database write is a network call and can take milliseconds, there is usually no need to wait for it to complete +when more data is waiting to be sent. + +Each method takes: + +- A list of updated or new items of the related Model class (this can be a list of one to replace Save(), although + replacing a single Save() is not much faster, it does open up all the additional capabilities) +- The Class of the Model, as Java does not let you infer that through a template +- An InvalidDataHandlerService - which lets you define what to do for any documents that could not be processed +- An UpdateStrategy, whether to Insert, OverWrite (Replace), Update, or "Update and Return a change document" for each + of the items in the list. All of these will default to inserting if the document does not exist and will delete if the + _@DeleteFlag_ field is true +- PostWriteTriggerService postTrigger - if non-null, this is an interface with a function that is called after updating + the data in the database. This function is passed a means to retrieve the nature of all the changes, down to a field + level, and is used to write change histories as well as any other post-update processing required. It is similar to an + RDBMS trigger but is written in client-side Java. + +Each returns a standard MongoDB BulkWriteResult which provides information on the number and type of operations +completed. -This Interface, and it's underlying implemention provide the following three functions +### OptimizedMongoQueryRepository.java -```java +MongoRepository provides all the typical Spring/JPA query facilities - auto-generated queries of the form +FindByThisAndThat() or FindAllByFieldLessThan(), etc. You can also use _@Query_ annotations and query by both Example +and Criteria as you would normally do. You can also use the native MongoDB driver to construct queries as code using +fluent query-building classes. +What _OptimizedMongoQueryRepository_ brings is the ability to run queries defined at runtime and passed from an +application. Similar to GraphQL, we may not always want to create an explicit endpoint for every possible query our API +supports. If a User Interface has some form of query builder, perhaps a form with many optional fields, then we may want +the UI designer to have control over the queries run. We might also want to offer both Database Queries and Lucene +full-text index queries - Lucene indexing is available in MongoDB Atlas and provides a powerful way to perform fuzzy, +best-match, full-text queries among others. + +This interface and its underlying implementation provide the following three functions: + +```java List mongoDbNativeQuery(String jsonString, Class clazz); List atlasSearchQuery(String jsonString, Class clazz); -int costMongoDbNativeQuery(String jsonString, Class clazz); +int costMongoDbNativeQuery(String jsonString, Class clazz); +``` -``` +The first two take a description in JSON of the required Query, Projection, Order, and Limits - it's very much a +general-purpose query function to allow arbitrary queries. Query mongoDBNative (querying using the MongoDB database and +indexes) or atlasSearch - querying using any defined Atlas Search Lucene indexes. -The first two take a description, in JSON of the required Query, PRojection, Order and Limits - it's very much a general -purpose query function to allow arbitrary queries . Qirty mongoDBNative (QUerying using the MognoDB database and -indexes) or atlasSearch - Querying using any degined Atlas Search Lucenes indexes. +The final method, which you can and should call from your service before passing a query to mongoDbNativeQuery, returns +a 'cost' between 1 and 500 for that query, where 1 represents a fully indexed and index-covered query and 500 represents +a collection scan. This score indicates how much resource the query will take, and you can use that to determine how to +proceed. You may reject queries that will be harmful, you may allow and log them, you may pass them to a secondary or +analytic replica, or you may modify them, for example adding a limited and indexed date range so they only impact a +recent set of data. All of this you have the tools to do in your service layer. -The final method, which you can and shoudl call from your service before passing a query to mongoDbNativeQuery returns -a 'cost' between 1 and 500 for that query, where 1 represents a fullyindexed and index covered query and 500 represents -a collection scan. This score indicates how much resouce the query will take and you can use that to determine how to -proceeed. You may reject queries that will be harmful, you may allow and log them, you may pass them to a secondary or -analytic replica or you may modify them, for example adding a limited and indexed date range so they only impact a -recent set of data. Allo fo this you have the tools to do in your selvice layer. +### OptimizedMongoDownstreamRepository.java -### OprimisedMongoDownstreamRepository.java +When you want to perform a large extraction of data from a database, through Spring and out as JSON, there is a simple +and obvious solution - perform a query or aggregation to retrieve a Stream of the Object model and then convert those to +JSON to stream them out. This is the simplest way to perform a large-scale extraction. However, when we do this, we have +to create large numbers of temporary row (Document) objects and Model objects in order to render them as JSON. In the +case of MongoDB, this is a conversion from BSON (the native internal and network format) to Document classes (HashMap +based) to Model Classes to JSON. -When you want to perform a large extraction of data from a database, through Spring and out as JSON then there is a -simple and objecous solution - perform a query or aggregation to retirve a Stream or the Object model and then convert -those to JSON to stream them out. This is the simplest way to performa a large scale extraction, when we do thsi however -we have to create large numbers of temporary row (Document) objects and Model objects in order to the render them as -JSON. In the case of MongoDB this is a conversion from BSON (The native internal and network format) to Document -classes ( HashMap based) to Model Classes to JSON. +If you are extracting a few MB now and then, this is OK, but what if you want to regularly and quickly extract Gigabytes +of data? You can shortcut a lot of the process above by telling MongoDB you want JSON, not Objects from the database, +and this is where the single method in OptimizedMongoDownstreamRepository comes in. -If you are extracing a few MB now and then this is OK but what if yo want to regularly and quickly extract Gigabytes of -data - you can shortcut a lot fo the process above by telling MongoDB you want JSON, not Objects form the databaee and -this is where the single method in OptimisedMongoDownstreamRepository comes in. +```java +Stream nativeJsonExtract(String formatRequired, Class modelClazz); +``` -``` - Stream nativeJsonExtract(String formatRequired, Class modelClazz); -``` +This is an example with a single method to show that instead of `Stream` we can use the much more +efficient `Stream` and in our service and controller simply stream them out to our end user, avoiding 99% of +the object creation and garbage collection. JsonObject converts from the BSON native format directly to JSON without +creating any intermediate objects. -This is an example it has a single methos to show that instead of `Stream` we can use the much more -efficient `Stream` and in our ser ice and controller simply stream the out to our end used avoiding 99% of -the object createion and garbage colleciton. JsonObject converts from the BSON native format directly to JSON without -creating any intermediate objets. - -To use this we do need to explicitly tell mongodb exactly what our JSON shodul look like, and what we pass in is a -String representation in JSON of the required format, the function performs a find() retriueving the whole collection -and applying this projections. The purpose of this is not to simply use as is but to demonstrate how much more efficient -large scale data downstrewming as JSON (or BSON) can be when you avoid all the SPring obhjectmapping. +To use this, we do need to explicitly tell MongoDB exactly what our JSON should look like, and what we pass in is a +String representation in JSON of the required format. The function performs a find() retrieving the whole collection and +applying this projection. The purpose of this is not to simply use as is but to demonstrate how much more efficient +large-scale data downstream as JSON (or BSON) can be when you avoid all the Spring object mapping. ### MongoHistoryRepository.java -MongoHistoryRepository is used to read one or multipel documents but to apply the changes made to them over time in -reverse order to revert to older versions. It has two methods although the code can be extended and reused for more -sophiticaed cases including querying historical data. +MongoHistoryRepository is used to read one or multiple documents but to apply the changes made to them over time in +reverse order to revert to older versions. It has two methods, although the code can be extended and reused for more +sophisticated cases including querying historical data. -```java +```java Stream GetRecordByIdAsOfDate(I recordId, Date asOf, Class clazz); -Stream GetRecordsAsOfDate(Criteria criteria, Date asOf, Class clazz); -``` +Stream GetRecordsAsOfDate(Criteria criteria, Date asOf, Class clazz); +``` -This assumes the documents, and any changed version sof them were ingested using the OptimizedLoadRepository with the -record history option enabled. These methods take the base document (latest version) perform a $lookup (JOIN) to fetch -the historical changes and then ,merge tthose to wind the document back to it's prior form. +This assumes the documents and any changed versions of them were ingested using the OptimizedLoadRepository with the +record history option enabled. These methods take the base document (latest version), perform a $lookup (JOIN) to fetch +the historical changes, and then merge those to wind the document back to its prior form. # Services -Services sit between Controlers ( Handing interfaces to the service) and Repositories (Interfacing with an underlying ] +Services sit between Controllers (handling interfaces to the service) and Repositories (interfacing with an underlying database). Services contain business logic and processing that is specific to your business and also how you intend to -use the data - a very simple controller could acess the repository directly but this is not good practise. In the Memex -examples we do not access the repository directly from the controller - only via services, even if in some cases those -service classes perform no additonal processing or logic. +use the data. A very simple controller could access the repository directly, but this is not good practice. In the Memex +examples, we do not access the repository directly from the controller - only via services, even if in some cases those +service classes perform no additional processing or logic. ## MongoDbJsonStreamingLoaderService.java -The most significant service in Memex is the StreamingJsonLoaderService - this service parses an incoming JSON stream -into the associated Model class and every 200 documents it uses OptimizedMongoLoadRespository to load that batch of -documents as an asynchronous process whilst it parses and processes the next 200 documents. This Stream processing and -parallelism is at the haeart of high performance data loading in Memex. +The most significant service in Memex is the StreamingJsonLoaderService. This service parses an incoming JSON stream +into the associated Model class and every 200 documents it uses OptimizedMongoLoadRepository to load that batch of +documents as an asynchronous process while it parses and processes the next 200 documents. This stream processing and +parallelism is at the heart of high-performance data loading in Memex. -It can accept data as a JSON array or simply multiple JSON obejects, optionaly seperated by a newline, it work by -finding the first object start token `{` then finding the corresponding close token `}` and treating what is between -them as and object to load - it then finds the next open token. +It can accept data as a JSON array or simply multiple JSON objects, optionally separated by a newline. It works by +finding the first object start token `{`, then finding the corresponding close token `}`, and treating what is between +them as an object to load - it then finds the next open token. After parsing each document to a model object, the _StreamingJsonLoader_ will pass that object to a _preWriteTrigger_ class if one is defined to allow for any modifications needed before writing to MongoDB. ## PreWriteTriggerService.java -This class defines a single overrideable functiion which is called by StreamingJsonLoaderService -( May be moved to OptimizedLoadRepositor soon). This is called for each Model object just before writing it to -MongoDB and can be used to augment the data - the correct method to override depends on whather you ar e using mutable -or immutable models. The default class does nothing to the object +This class defines a single overrideable function which is called by StreamingJsonLoaderService (may be moved to +OptimizedLoadRepository soon). This is called for each Model object just before writing it to MongoDB and can be used to +augment the data. The correct method to override depends on whether you are using mutable or immutable models. The +default class does nothing to the object. -```java +```java public void modifyMutableDataPreWrite(T document) { } public T newImmutableDataPreWrite(T document) { return document; -} -``` +} +``` ## PostWriteTriggerService.java -This defines a function to be called after a batch of documents have been written to MongoDB inside the transaction -used to write them and before it commits. The dfault function does nothing but can be overridder in a derived class. +This defines a function to be called after a batch of documents have been written to MongoDB inside the transaction used +to write them and before it commits. The default function does nothing but can be overridden in a derived class. -```java - public void postWriteTrigger( +```java +public void postWriteTrigger( ClientSession session, BulkWriteResult result, List records, Class clazz, ObjectId updateId) throws IllegalAccessException { -} -``` +} +``` + +This function receives: -This function recieves the transactional session used for the write. By using that session inside this function you are -able to see operations that have not yet been comitted. -It recieves the BulkWriteResult object descring how many of each operation took place and the id values for -anything that was inserted. -It revieces the batch of rercords that were presented for updating, deleteing or inserting -It also inclues an updateId value, this can be used to Query MongoDB to retireve the documents as they are post update -if needed. +- The transactional session used for the write. By using that session inside this function, you are able to see + operations that have not yet been committed. +- The BulkWriteResult object describing how many of each operation took place and the ID values for anything that was + inserted. +- The batch of records that were presented for updating, deleting, or inserting. +- An updateId value, which can be used to query MongoDB to retrieve the documents as they are post-update if needed. -This is derived from the greate the HistoryTriggerService that records changes to the data transactionally. +This is derived from the greater HistoryTriggerService that records changes to the data transactionally. ## HistoryTriggerService.java -HistoryTriggerService is a tempalted reuasable class, extending PostWriteTriggerService to generate a change hostiry -of any data updated. +HistoryTriggerService is a templated reusable class, extending PostWriteTriggerService to generate a change history of +any data updated. -Where a model is stored in collection X, for example `vehicleinspaeciton` this class writes DocumentHistory objects to +Where a model is stored in collection X, for example `vehicleinspection`, this class writes DocumentHistory objects to `vehicleinspection_history`. These are a reverse delta history showing field values in previous versions of the document -at each stage. To use them you need to take the latest version of the data and apply in reverse order - this is done by -MongoHistoryReportity for you for functions such as asOf (a point in time). +at each stage. To use them, you need to take the latest version of the data and apply in reverse order - this is done by +MongoHistoryRepository for you for functions such as asOf (a point in time). -When a record is first inserted, then an entry is recorded in the history table just saying this is when the record was -first inserted, it does not contain any of the content. +When a record is first inserted, an entry is recorded in the history table just saying this is when the record was first +inserted; it does not contain any of the content. -When a record is modified, then the modified fields previous version are stored alone with an updateid and timestamp +When a record is modified, the modified fields' previous versions are stored along with an updateId and timestamp. -When a record is deleted then , currently the whole record is stored in the trigger service and removed from the -principle colleciton. -This could be changed to delete the record from the principle collection AND to remove all hisotic versions very easily -but -many users prefer to archive the data or retain a hisotoyry of its existence and deleteion. It woudl also be simple to -have a history that -retained the creation and deleteion records but no record of the content to modifications. +When a record is deleted, currently the whole record is stored in the trigger service and removed from the principal +collection. This could be changed to delete the record from the principal collection AND to remove all historic versions +very easily, but many users prefer to archive the data or retain a history of its existence and deletion. It would also +be simple to have a history that retained the creation and deletion records but no record of the content modifications. ## InvalidDataHandlerService.java -This class frines a singel function to be called before a document is loaded into the database if that documnt fails -java bean validation Java Bean Validation. -It is passed the document and a list of ways it fails validation - the default behaviour is to log as a warning. -This function returns a boolean which determines is the data shoudl be loaded anyway or skipped in the data -loading/updaing process. -It shoudl be used to populate a data file of things to be reloaded - it is designed to allow the rest of the load to -complete and errors or exceprions to be caught. +This class defines a single function to be called before a document is loaded into the database if that document fails +Java Bean Validation. It is passed the document and a list of ways it fails validation - the default behavior is to log +as a warning. This function returns a boolean which determines if the data should be loaded anyway or skipped in the +data loading/updating process. It should be used to populate a data file of things to be reloaded - it is designed to +allow the rest of the load to complete and errors or exceptions to be caught. -```java - public boolean handleInvalidData( +```java +public boolean handleInvalidData( T document, Set> violations, Class clazz) { LOG.warn( "Invalid data detected ({} violations) in document, but no explicit handler provided, discarding.", violations.size()); return false; -} -``` +} +``` ## MongoDbPreflightCheckService.java -This Java Bean based on ApplicationRunner demonstrates one good practise with services built on MongoDB - what the -server validate any database prerequisites, -be they permissions, version or the exisstence of database objects includeing collecitons, indexes, search indexes and -schema validators. +This Java Bean based on ApplicationRunner demonstrates one good practice with services built on MongoDB - having the +server validate any database prerequisites, be they permissions, version, or the existence of database objects including +collections, indexes, search indexes, and schema validators. -The way things like indexes are ppopulated through environment can differ based on the system, srchitecture and business -requirements. In a -development environment it is normal to generate indexes that are not present in the preflight service, however -in production, given creation of an index may have an impact on a running system it is better to build index via -management toolling and failover orchestration. In that case pre-flight shoudl fail until the index already exsts. +The way things like indexes are populated through environment can differ based on the system, architecture, and business +requirements. In a development environment, it is normal to generate indexes that are not present in the preflight +service. However, in production, given that creation of an index may have an impact on a running system, it is better to +build indexes via management tooling and failover orchestration. In that case, preflight should fail until the index +already exists. -By default this is configured to create indexes and search indexes for ease fo developemtn but warn in the logs this is -not for priduciton. +By default, this is configured to create indexes and search indexes for ease of development but warn in the logs that +this is not for production. ## VehicleInspectionPreWriteTriggerService.java -This is provided as an example of a pre-write trigger which is passed makes small changes to the incoming data to -makle it easy to test updates, you can load the same file but this will modify some details to cause an update and -relevant history entry if required. It also serves to demonstrate what a preWrite trigger looks like both for -mutable and immutable data +This is provided as an example of a pre-write trigger which makes small changes to the incoming data to make it easy to +test updates. You can load the same file, but this will modify some details to cause an update and relevant history +entry if required. It also serves to demonstrate what a preWrite trigger looks like both for mutable and immutable data. ## VehicleInspectionQueryService.java -The class OptimizedMongoQueryRepository provides an endpoint supporting any MongoDB query defined by an EJSON string, -this is intended to allow front end clients to define their own queries and avoid the requiremetn to create an -endpoint for each new frontend feature - especially where some for of GUI query creation is available. +The class OptimizedMongoQueryRepository provides an endpoint supporting any MongoDB query defined by an EJSON string. +This is intended to allow front-end clients to define their own queries and avoid the requirement to create an endpoint +for each new frontend feature - especially where some form of GUI query creation is available. -The OptimizedMongoQueryRepository also provides a cost estimator function for queries to determine if they will take -a few or a lot fo resources from the datbase server. This Service takes each incoming query, performs a cost -estimation (This is efficent and cached in the estimator) logs the cost and runs it anyway - this si where you woudl -add business logic to deny, log , modify or redirect expensive queries to a secondayr server. +The OptimizedMongoQueryRepository also provides a cost estimator function for queries to determine if they will take few +or many resources from the database server. This Service takes each incoming query, performs a cost estimation (this is +efficient and cached in the estimator), logs the cost, and runs it anyway. This is where you would add business logic to +deny, log, modify, or redirect expensive queries to a secondary server. This class shows how to expose a simple hard-coded query and a Spring query-by-example to the controller class. ## VehicleInspectionXXXXService.java -Where XXXXX is one of: DownStream,History,HistoryTrigger,InvalidDataHandler,JsonLoader then the service is thin +Where XXXXX is one of: DownStream, History, HistoryTrigger, InvalidDataHandler, JsonLoader, then the service is a thin wrapper over the generic service or to provide a simple call to the repository with space for additional logic. # Controllers -Memex includes an example _@RestController_ to show how the features are acessed, a RestController was chosen as it -is the simplest to test and understand but other controller types are equally usable in your own code. +Memex includes an example _@RestController_ to show how the features are accessed. A RestController was chosen as it is +the simplest to test and understand, but other controller types are equally usable in your own code. ## VehicleInspectionController.java +This simply provides REST endpoints for the various service functions discussed above. The only additional functionality +that isn't a default Spring RestController is a function which takes a Stream of objects and writes them back to the +client output stream. There are two versions here: one that streams back VehicleInspection classes and one that has the +underlying MongoDB database construct the JSON (as BSON) and stream it without creating any of the intermediary client +objects. This could save 50-90% of client CPU cycles with a consequent increase in performance for large data sets. + From 6be9539d065cf81de9c88202e6772d415ffe088a Mon Sep 17 00:00:00 2001 From: John Page Date: Mon, 16 Jun 2025 13:14:14 +0100 Subject: [PATCH 3/5] Updated Docs --- memex/{Class Derscriptions.md => ClassDescription.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename memex/{Class Derscriptions.md => ClassDescription.md} (100%) diff --git a/memex/Class Derscriptions.md b/memex/ClassDescription.md similarity index 100% rename from memex/Class Derscriptions.md rename to memex/ClassDescription.md From 92e7c0e0aa4a52fdec776a272532ad90ea3b639e Mon Sep 17 00:00:00 2001 From: John Page Date: Mon, 16 Jun 2025 13:15:41 +0100 Subject: [PATCH 4/5] Updated Docs --- memex/ClassDescription.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/memex/ClassDescription.md b/memex/ClassDescription.md index b9d8d9f..855f117 100644 --- a/memex/ClassDescription.md +++ b/memex/ClassDescription.md @@ -8,6 +8,9 @@ Memex. Memex comes as a working example application with most reusable functionality abstracted into reusable templated classes. If you cannot simply reuse these as they are, you can derive from them or modify them as needed. +Memex also comes with a simple browser-based UI to demonstrate the query endpoints. This is not intended as reusable +code. + # Model Classes Model classes represent the data objects used by your application. They typically map to the objects you persist in the From 2c9fa94974ffa4dee8e5ef1a2541c422bd8ff9b0 Mon Sep 17 00:00:00 2001 From: Rajat Sharma Date: Tue, 17 Jun 2025 12:42:06 +1000 Subject: [PATCH 5/5] Test --- .../com/johnlpage/memex/cucumber/steps/KafkaConsumerSteps.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/memex/src/test/java/com/johnlpage/memex/cucumber/steps/KafkaConsumerSteps.java b/memex/src/test/java/com/johnlpage/memex/cucumber/steps/KafkaConsumerSteps.java index d4d2cfb..c33712e 100644 --- a/memex/src/test/java/com/johnlpage/memex/cucumber/steps/KafkaConsumerSteps.java +++ b/memex/src/test/java/com/johnlpage/memex/cucumber/steps/KafkaConsumerSteps.java @@ -45,7 +45,7 @@ public void sendVehicleInspectionsToKafka(int count, long startId, String jsonTe vehicleInspection.setTestid(testId); String message = objectMapper.writeValueAsString(vehicleInspection); - kafkaTemplate.send("test", message); + kafkaTemplate.send("test", message); // test } }