Merge pull request #42 from RTGS-Lab/bugfix/configuration-updates

Configuration Updates and SD Card Updates

Author: zradlicz

.github/workflows/release-workflow.yaml

@@ -181,14 +181,18 @@ jobs:
             product_id_secret: "PARTICLE_WINTERTURF_PRODUCT_ID"
           - name: "Plant Pathways"
             product_id_secret: "PARTICLE_PLANT_PATHWAYS_PRODUCT_ID"
-          - name: "LCCMR Irrigation"
-            product_id_secret: "PARTICLE_LCCMR_IRRIGATION_PRODUCT_ID"
+          - name: "Sharma V3 Irrigation"
+            product_id_secret: "PARTICLE_SHARMA_V3_IRRIGATION_PRODUCT_ID"
           - name: "Roadside Turf"
             product_id_secret: "PARTICLE_ROADSIDE_TURF_PRODUCT_ID"
           - name: "PepsiCo"
             product_id_secret: "PARTICLE_PEPSICO_PRODUCT_ID"
           - name: "Precision Ag Pilot"
             product_id_secret: "PARTICLE_PRECISION_AG_PILOT_PRODUCT_ID"
+          - name: "Turf Disease Pilot"
+            product_id_secret: "PARTICLE_TURF_DISEASE_PILOT_PRODUCT_ID"
+          - name: "Legacy Irrigation"
+            product_id_secret: "PARTICLE_LEGACY_IRRIGATION_PRODUCT_ID"
     steps:
       - name: Checkout code
         uses: actions/checkout@v4

CONFIGURATION.md

@@ -4,18 +4,21 @@
 
 Configuration is loaded at startup in the following priority order:
 
-1. **SD Card**: `config.json` file on the SD card
-2. **Default**: Built-in default configuration if SD file not found
+1. **SD Card**: `config.json` file on the SD card (primary storage)
+2. **EEPROM Backup**: Configuration restored from EEPROM if SD card fails
+3. **Default**: Built-in default configuration if both SD and EEPROM unavailable
+
+The system automatically creates EEPROM backups when configurations are successfully loaded from SD card or applied via cloud functions. When EEPROM backup is used, the configuration is automatically restored to the SD card.
 
 #### Updating Configuration
 
 Configuration can be updated through:
 
 1. **Cloud Function**: `updateConfig` Particle function
-     - copy the intended config.json and paste it as and argument into updateConfig
-     - If successful, the device will restart without returning any value.
-     - verify that the first metadata packet after reset matches your configuration
-     - See below for other types of responses.
+     - Configuration is always saved to EEPROM backup for reliability
+     - SD card is updated when possible, but function succeeds if EEPROM update works
+     - Device will restart automatically on successful configuration update
+     - Verify success by checking metadata packet includes new configuration UIDs
 3. **SD Card**: Replace `config.json` file and restart system
 
 #### updateConfig Function Details
@@ -50,15 +53,12 @@ The function returns specific error codes when configuration updates fail:
 
 | Error Code | Description | Troubleshooting |
 |------------|-------------|-----------------|
-| `0` | Success - Configuration removed from SD card | |
-| `1` | Success - Configuration updated, system restarting | |
-| `-1` | Failed to remove configuration from SD card | |
-| `-2` | Invalid configuration format - Missing 'config' element | |
-| `-3` | Invalid configuration format - Missing 'system' element | |
-| `-4` | Invalid configuration format - Missing 'sensors' element | |
-| `-5` | Failed to write test file to SD card | |
-| `-6` | Failed to remove current configuration from SD card | |
-| `-7` | Failed to write new configuration to SD card | |
+| `0` | Success - Configuration removed from SD card and EEPROM | |
+| `1` | Success - Configuration updated and saved to EEPROM, system restarting | May show SD card warnings but still succeeds |
+| `-2` | Invalid configuration format - Missing 'config' element | Check JSON structure |
+| `-3` | Invalid configuration format - Missing 'system' element | Ensure 'system' section exists |
+| `-4` | Invalid configuration format - Missing 'sensors' element | Ensure 'sensors' section exists |
+| `-8` | Failed to parse configuration - Invalid JSON or values | Check JSON syntax and parameter ranges |
 
 ##### Configuration Validation Rules
 
@@ -70,13 +70,14 @@ The system performs several validation checks:
    - Must contain "sensors" section within config (checked by string search)
    - All whitespace, newlines, carriage returns, and tabs are automatically stripped
 
-2. **SD Card Validation**
-   - SD card must be accessible for writing
-   - System tests write capability before attempting configuration update
-   - Current configuration must be removable before writing new configuration
+2. **Configuration Processing**
+   - Configuration is parsed and applied first (automatically saves to EEPROM backup)
+   - SD card is updated when possible, but failures don't prevent success
+   - System succeeds if configuration parsing works, regardless of SD card status
+   - Warning messages indicate SD card issues but don't cause operation failure
 
 3. **Special Commands**
-   - Use "remove" as the configuration string to delete config.json and revert to defaults
+   - Use "remove" as the configuration string to delete config.json and clear EEPROM backup
 
 ##### Example Error Scenarios
 
@@ -92,17 +93,25 @@ particle call device_name updateConfig '{"config":{"system":{"logPeriod":300}}}'
 # Returns: -4
 ```
 
-###### SD Card Issues
+###### SD Card Issues (Non-Critical)
 ```bash
-# If SD card is not available or full
-particle call device_name updateConfig '{"config":{"system":{"logPeriod":300},"sensors":{}}}'
-# May return: -5, -6, or -7 depending on the specific SD card failure point
+# Configuration update succeeds even if SD card fails
+particle call device_name updateConfig '{"config":{"system":{"logPeriod":300},"sensors":{"numSoil":3}}}'
+# Returns: 1 (success) - Configuration saved to EEPROM, may show SD warnings in logs
+# Serial output: "Warning: Failed to write configuration to SD card, but EEPROM backup is available."
 ```
 
 ###### Configuration Removal
 ```bash
 particle call device_name updateConfig 'remove'
-# Returns: 0 (success) or -1 (failed to remove)
+# Returns: 0 (success) - Removes SD config and clears EEPROM backup
+```
+
+###### Parsing Failures (Critical)
+```bash
+# Invalid JSON structure causes complete failure
+particle call device_name updateConfig '{"config":{"system":{"logPeriod":"invalid"}}}'
+# Returns: -8 (failed to parse configuration)
 ```
 
 ##### Best Practices
@@ -111,7 +120,9 @@ particle call device_name updateConfig 'remove'
 2. **Check Parameter Ranges**: Verify all values are within acceptable ranges
 3. **Plan Hardware Requirements**: Ensure sufficient Talons for sensor configuration
 4. **Monitor Device Status**: Watch for restart after successful configuration
-5. **Verify Configuration**: Check UIDs after restart to confirm changes applied
+5. **Verify Configuration**: Check UIDs in metadata packets to confirm changes applied
+6. **EEPROM Backup Reliability**: Configuration updates work even with SD card failures
+7. **Field Recovery**: Insert fresh SD cards - system automatically restores config from EEPROM
 
 #### Configuration UIDs
 

lib/Driver_-_Kestrel

@@ -841,7 +841,9 @@ String getMetadataString()
 	output = output + "\"Update\":" + String(logPeriod) + ",";
 	output = output + "\"Backhaul\":" + String(backhaulCount) + ",";
 	output = output + "\"LogMode\":" + String(loggingMode) + ",";
-	output = output + "\"Sleep\":" + String(powerSaveMode) + "}},";
+	output = output + "\"Sleep\":" + String(powerSaveMode) + ",";
+	output = output + "\"SysConfigUID\":" + String(configManager.updateSystemConfigurationUid()) + ",";
+	output = output + "\"SensorConfigUID\":" + String(configManager.updateSensorConfigurationUid()) + "}},";
 	//FIX! Add support for device name 
 
 	uint8_t deviceCount = 0; //Used to keep track of how many devices have been appended 
@@ -1074,6 +1076,40 @@ void systemConfig()
 					Serial.println("\tDone");
 				}
 
+				if(ReadString.equalsIgnoreCase("Dump SD")) {
+					fileSys.dumpSDOverSerial();
+					Serial.println("\tDump Complete");
+				}
+
+				if(ReadString.startsWith("Dump SD Recent ")) {
+					String countStr = ReadString.substring(15); // "Dump SD Recent " is 15 chars
+					uint32_t count = countStr.toInt();
+					if (count > 0) {
+						fileSys.dumpSDOverSerial(count);
+						Serial.print("\tDump Complete (");
+						Serial.print(count);
+						Serial.println(" recent files per type)");
+					} else {
+						Serial.println("\tInvalid count parameter");
+					}
+				}
+
+				if(ReadString.startsWith("Write SD ")) {
+					String filename = ReadString.substring(9); // "Write SD " is 9 chars
+					filename.trim();
+					if (filename.length() > 0) {
+						Serial.print("\tStarting file write: ");
+						Serial.println(filename);
+						if (fileSys.writeFileOverSerial(filename.c_str())) {
+							Serial.println("\tWrite Complete");
+						} else {
+							Serial.println("\tWrite Failed");
+						}
+					} else {
+						Serial.println("\tInvalid filename parameter");
+					}
+				}
+
 				if(ReadString.equalsIgnoreCase("Exit")) {
 					return; //Exit the setup function
 				}
@@ -1346,15 +1382,17 @@ int setNodeID(String nodeID)
 
 int updateConfiguration(String configJson) {
 	if(configJson == "remove") {
-		// Remove the configuration file from the SD card
-		if (!fileSys.removeFileFromSD("config.json")) {
-			Serial.println("Error: Failed to remove configuration from SD card.");
-			return -1; // Failed to remove config
-		}
-		else {
-			Serial.println("Configuration removed from SD card.");
+		// Remove the configuration file from the SD card and clear EEPROM
+		bool sdRemoved = fileSys.removeFileFromSD("config.json");
+		configManager.clearConfigEEPROM();
+		
+		if (sdRemoved) {
+			Serial.println("Configuration removed from SD card and EEPROM.");
+			return 0; // Success
+		} else {
+			Serial.println("Warning: Failed to remove SD config, but EEPROM cleared.");
+			return 0; // Still success since EEPROM was cleared
 		}
-		return 0; // Success
 	}
 
     Serial.println("Updating configuration...");
@@ -1382,24 +1420,30 @@ int updateConfiguration(String configJson) {
 		return -4; // Invalid format
 	}
 
-	// test write to SD card
-	if (!fileSys.writeToSD("", "config.json")) {
-		Serial.println("Error: Failed to write to SD card.");
-		return -5; // Failed to write config
+	// First, try to parse and apply the configuration (this will also save to EEPROM)
+	bool configParsed = configManager.setConfiguration(configJson.c_str());
+	if (!configParsed) {
+		Serial.println("Error: Failed to parse configuration.");
+		return -8; // Failed to parse config
 	}
+	Serial.println("Configuration parsed successfully and saved to EEPROM.");
 
-	//clear current config.json
-	if(!fileSys.removeFileFromSD("config.json")) {
-		Serial.println("Error: Failed to remove current configuration from SD card.");
-		return -6; // Failed to remove current config
-	}
+	// Try to write to SD card (optional - don't fail if this doesn't work)
+	bool sdSuccess = false;
+	
+	// Remove old config first
+	fileSys.removeFileFromSD("config.json"); // Don't check return value
 
 	// Write new configuration to SD card
-	if (!fileSys.writeToSD(configJson.c_str(), "config.json")) {
-		Serial.println("Error: Failed to write new configuration to SD card.");
-		return -7; // Failed to write new config
+	if (fileSys.writeToSD(configJson.c_str(), "config.json")) {
+		Serial.println("Configuration written to SD card.");
+		sdSuccess = true;
+	} else {
+		Serial.println("Warning: Failed to write configuration to SD card, but EEPROM backup is available.");
 	}
 
+	// Success if config was parsed (EEPROM updated), regardless of SD card status
+	Serial.println("Configuration update completed. System will restart to apply changes.");
 	System.reset(); //restart the system to apply new configuration
 	return 1; //Success
 }
@@ -1524,12 +1568,34 @@ bool loadConfiguration() {
         configLoaded = configManager.setConfiguration(configStr);
     }
 
+    // If SD card config failed, try EEPROM backup
+    if (!configLoaded) {
+        Serial.println("SD config failed, trying EEPROM backup...");
+        configLoaded = configManager.loadConfigFromEEPROM();
+        if (configLoaded) {
+            Serial.println("Configuration loaded from EEPROM backup");
+            // Restore the config to SD card from EEPROM backup
+            std::string eepromConfig = configManager.getConfiguration();
+            if (fileSys.writeToSD(eepromConfig.c_str(), "config.json")) {
+                Serial.println("EEPROM config restored to SD card");
+            } else {
+                Serial.println("Warning: Could not restore config to SD card");
+            }
+        }
+    }
+    
+    // If both SD and EEPROM failed, use defaults
     if (!configLoaded) {
         Serial.println("Loading default configuration...");
         std::string defaultConfig = configManager.getDefaultConfigurationJson();
         configLoaded = configManager.setConfiguration(defaultConfig);
-        fileSys.removeFileFromSD("config.json");
-        fileSys.writeToSD(defaultConfig.c_str(), "config.json");
+        // Only write default config to SD if no config file exists, not if parsing failed
+        if (configStr.empty()) {
+            Serial.println("No config file found, writing default config to SD card...");
+            fileSys.writeToSD(defaultConfig.c_str(), "config.json");
+        } else {
+            Serial.println("Config file exists but parsing failed, keeping existing file on SD card");
+        }
     }
 
     // Set global variables from configuration