feat: implement AI integration services with NER-based data anonymization#13
feat: implement AI integration services with NER-based data anonymization#13Juste-Leo2 wants to merge 4 commits into
Conversation
…tion and multi-model support
There was a problem hiding this comment.
Code Review
This pull request introduces local Named Entity Recognition (NER) based anonymization to mask sensitive data (such as names, locations, emails, and phone numbers) before sending requests to external AI APIs. Key changes include adding @huggingface/transformers and onnxruntime-react-native dependencies, configuring Metro to resolve model assets, creating a model download script, and integrating anonymization toggles in the UI and AI service. The review highlights several critical issues: a file descriptor leak in the download script during redirects, React Native runtime path resolution failures for local assets, permanent caching of rejected promises in the NER service singleton, a regex matching failure due to unhandled SentencePiece space characters, and a privacy risk where NER failures silently leak un-anonymized data to external APIs.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| const downloadFile = (fileUrl, destPath) => { | ||
| return new Promise((resolve, reject) => { | ||
| console.log(`Téléchargement de ${fileUrl} ...`); | ||
| const file = fs.createWriteStream(destPath); | ||
| https.get(fileUrl, (response) => { | ||
| if (response.statusCode === 302 || response.statusCode === 301) { | ||
| // Redirection HuggingFace | ||
| downloadFile(response.headers.location, destPath).then(resolve).catch(reject); | ||
| return; | ||
| } | ||
| if (response.statusCode !== 200) { | ||
| reject(new Error(`Erreur HTTP ${response.statusCode} pour ${fileUrl}`)); | ||
| return; | ||
| } | ||
| response.pipe(file); | ||
| file.on('finish', () => { | ||
| file.close(() => resolve()); | ||
| }); | ||
| }).on('error', (err) => { | ||
| fs.unlink(destPath, () => reject(err)); | ||
| }); | ||
| }); | ||
| }; |
There was a problem hiding this comment.
In the downloadFile function, the write stream file is created on line 25 before checking the HTTP status code. If a redirect (301/302) or an error (non-200) occurs, this stream is left open and never closed or destroyed. This leaks file descriptors and can cause concurrent write conflicts on destPath during the recursive downloadFile call. You should only create the write stream after confirming a successful 200 OK response.
const downloadFile = (fileUrl, destPath) => {
return new Promise((resolve, reject) => {
console.log(`Téléchargement de ${fileUrl} ...`);
https.get(fileUrl, (response) => {
if (response.statusCode === 302 || response.statusCode === 301) {
// Redirection HuggingFace
downloadFile(response.headers.location, destPath).then(resolve).catch(reject);
return;
}
if (response.statusCode !== 200) {
reject(new Error(`Erreur HTTP ${response.statusCode} pour ${fileUrl}`));
return;
}
const file = fs.createWriteStream(destPath);
response.pipe(file);
file.on('finish', () => {
file.close(() => resolve());
});
file.on('error', (err) => {
fs.unlink(destPath, () => reject(err));
});
}).on('error', (err) => {
reject(err);
});
});
};| // Mettre le chemin vers le dossier contenant le modèle (relatif pour le bundler RN) | ||
| // Avec transformers.js et React Native, on peut utiliser des URL ou des chemins d'assets locaux | ||
| // selon la configuration du plugin babel / metro. | ||
| env.localModelPath = 'assets/models/'; |
There was a problem hiding this comment.
In React Native, local assets are bundled into the native binary and are not accessible via standard relative file paths like 'assets/models/' at runtime. Using @huggingface/transformers directly with local paths will fail in production. Consider using a React Native-compatible wrapper like @automatalabs/react-native-transformers or a custom native asset copier to copy the model files to the document/cache directory before loading them.
|
|
||
| for (const entity of sortedEntities) { | ||
| if (entity.entity_group === 'PER' || entity.entity_group === 'LOC') { | ||
| let originalWord = entity.word.trim(); |
There was a problem hiding this comment.
ALBERT uses the SentencePiece tokenizer, which represents spaces as the special character (U+2581). The word returned by the pipeline will contain this character (e.g., Montpellier or Jean-Claude). Since trim() only removes standard whitespace characters, it will NOT remove (U+2581). As a result, the subsequent regex replacement will fail to match the original text because the original text contains standard spaces, not (U+2581). To fix this, we must replace \u2581 with a standard space (or remove it) before trimming and matching.
| let originalWord = entity.word.trim(); | |
| let originalWord = entity.word.replace(/\u2581/g, ' ').trim(); |
| try { | ||
| const entities = await extractEntities(anonymizedText); |
There was a problem hiding this comment.
If the local NER model fails to load or run, extractEntities silently catches the error and returns an empty array. This causes anonymize to silently succeed without masking any names or locations, sending the raw, un-anonymized sensitive data (PII) to the external LLM API. Since the user explicitly enabled anonymization for privacy, this silent fallback is a major privacy risk. Consider propagating the error or returning a status indicating failure, so that the calling service can abort the request or warn the user before sending un-anonymized data.
| class NERInference { | ||
| static instance: any = null; | ||
|
|
||
| static async getInstance() { | ||
| if (this.instance === null) { | ||
| console.log("Chargement du modèle ONNX..."); | ||
|
|
||
| // Initialisation du pipeline | ||
| this.instance = pipeline( | ||
| 'token-classification', | ||
| 'albert-wikiner-fr-onnx', | ||
| { | ||
| quantized: false // Utiliser false car le fichier s'appelle model.onnx (pas model_quantized.onnx) | ||
| } | ||
| ); | ||
| } | ||
| return this.instance; | ||
| } | ||
| } |
There was a problem hiding this comment.
The pipeline function is asynchronous and returns a Promise. Since this.instance is assigned the Promise directly without await, any concurrent calls to getInstance() will return the same Promise, which is good. However, if the Promise rejects (e.g., due to a loading error), this.instance will permanently hold a rejected Promise, preventing any future retries. We can make the singleton pattern more robust by handling concurrent initialization and allowing retries on failure.
class NERInference {
static instance: any = null;
static initPromise: Promise<any> | null = null;
static async getInstance() {
if (this.instance) return this.instance;
if (!this.initPromise) {
console.log("Chargement du modèle ONNX...");
this.initPromise = pipeline(
'token-classification',
'albert-wikiner-fr-onnx',
{
quantized: false
}
).then(classifier => {
this.instance = classifier;
return classifier;
}).catch(err => {
this.initPromise = null; // Allow retry on failure
throw err;
});
}
return this.initPromise;
}
}…s.js and custom model path handling
|
Okay, the system is finally working and anonymizes the entries. Just wanted to mention a minor issue with “\n” that I found, but it's easy to fix. |
No description provided.