openAI compatibility for springAI export

Author: corradodebari

src/client/spring_ai/README.md

@@ -1,7 +1,7 @@
 # Spring AI template
 
 ## How to run:
-Prepare two configurations in the `ai-optimizer`, based on vector stores created using this kind of configuration:
+Prepare two configurations in the `Oracle ai optimizer and toolkit`, based on vector stores created using this kind of configuration:
 
 * OLLAMA: 
   * Embbeding model: mxbai-embed-large
@@ -54,19 +54,23 @@ Start with:
 
 This project contains a web service that will accept HTTP GET requests at
 
-* `http://localhost:8080/v1/chat/completions`: to use RAG via OpenAI REST API 
+* `http://localhost:9090/v1/chat/completions`: to use RAG via OpenAI REST API 
 
-* `http://localhost:8080/v1/service/llm` : to chat straight with the LLM used
-* `http://localhost:8080/v1/service/search/`: to search for document similar to the message provided
+* `http://localhost:9090/v1/service/llm` : to chat straight with the LLM used
+* `http://localhost:9090/v1/service/search/`: to search for document similar to the message provided
 
 
-RAG call example with `openai` build profile: 
+RAG call example with `openai` build profile with no-stream: 
 
 ```
-curl -X POST "localhost:8080/v1/chat/completions" \
-     -H "Content-Type: application/json" \
-     -H "Authorization: Bearer your_api_key" \
-     -d '{"message": "Can I use any kind of development environment to run the example?"}' | jq .
+curl -N http://localhost:9090/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer your_api_key" \
+  -d '{
+    "model": "server",
+    "messages": [{"role": "user", "content": "Can I use any kind of development environment to run the example?"}],
+    "stream": false
+  }'
 ```
 
 the response with RAG:
@@ -82,10 +86,20 @@ the response with RAG:
   ]
 }
 ```
-
+with stream output:
+```
+curl -N http://localhost:9090/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer your_api_key" \
+  -d '{
+    "model": "server",
+    "messages": [{"role": "user", "content": "Can I use any kind of development environment to run the example?"}],
+    "stream": true
+  }'
+```
 or the request without RAG:
 ```
-curl --get --data-urlencode 'message=Can I use any kind of development environment to run the example?' localhost:8080/v1/service/llm | jq .
+curl --get --data-urlencode 'message=Can I use any kind of development environment to run the example?' localhost:9090/v1/service/llm | jq .
 ```
 
 response not grounded:
@@ -150,10 +164,10 @@ llama3.1:latest             a80c4f17acd5    2.0 GB    3 minutes ago
 kubectl -n ollama exec svc/ollama -- ollama run "llama3.1" "what is spring boot?"
 ```
 
-* **NOTE**: The Microservices will access to the ADB23ai on which the vector store table should be created as done in the local desktop example shown before. To access the ai-optimizer running on **Oracle Backend for Microservices and AI** and create the same configuration, let's do:
+* **NOTE**: The Microservices will access to the ADB23ai on which the vector store table should be created as done in the local desktop example shown before. To access the ai-explorer running on **Oracle Backend for Microservices and AI** and create the same configuration, let's do:
   * tunnel:
   ```
-  kubectl -n ai-optimizer port-forward svc/ai-optimizer 8181:8501 
+  kubectl -n ai-explorer port-forward svc/ai-explorer 8181:8501 
   ```
   * on localhost:
   ```
@@ -173,25 +187,29 @@ kubectl -n ollama exec svc/ollama -- ollama run "llama3.1" "what is spring boot?
   ```
 
 
-* the `bind` will create the new user, if not exists, but to have the `<VECTOR_STORE>_SPRINGAI` table compatible with SpringAI Oracle vector store adapter, the microservices need to access to the vector store table created by the ai-optimizer with user ADMIN on ADB:
+* the `bind` will create the new user, if not exists, but to have the `<VECTOR_STORE>_SPRINGAI` table compatible with SpringAI Oracle vector store adapter, the microservices need to access to the vector store table created by the ai-explorer with user ADMIN on ADB:
 
 ```
 GRANT SELECT ON ADMIN.<VECTOR_STORE> TO vector;
 ```
 * then deploy:
 ```
-deploy --app-name rag --service-name myspringai --artifact-path <ProjectDir>/target/myspringai-1.0.0-SNAPSHOT.jar --image-version 1.0.0 --java-version ghcr.io/oracle/graalvm-native-image-obaas:21 --service-profile obaas
+deploy --app-name rag --service-name myspringai --artifact-path <ProjectDir>/target/myspringai-0.0.1-SNAPSHOT.jar --image-version 0.0.1 --java-version ghcr.io/oracle/graalvm-native-image-obaas:21 --service-profile obaas
 ```
 * test:
 ```
 kubectl -n rag port-forward svc/myspringai 9090:8080
 ```
 * from shell:
 ```
-curl -X POST "http://localhost:9090/v1/chat/completions" \
-     -H "Content-Type: application/json" \
-     -H "Authorization: Bearer your_api_key" \
-     -d '{"message": "Can I use any kind of development environment to run the example?"}' | jq .
+curl -N http://localhost:9090/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer your_api_key" \
+  -d '{
+    "model": "server",
+    "messages": [{"role": "user", "content": "Can I use any kind of development environment to run the example?"}],
+    "stream": false
+  }'
 ```
 it should return:
 ```

src/client/spring_ai/src/main/java/org/springframework/ai/openai/samples/helloworld/AIController.java

@@ -6,11 +6,12 @@
 package org.springframework.ai.openai.samples.helloworld;
 
 import org.springframework.ai.chat.client.ChatClient;
-import org.springframework.ai.chat.model.ChatResponse;
+import org.springframework.ai.chat.client.ChatClient.ChatClientRequestSpec;
 import org.springframework.ai.chat.prompt.Prompt;
 import org.springframework.ai.chat.prompt.PromptTemplate;
 import org.springframework.ai.document.Document;
 import org.springframework.ai.embedding.EmbeddingModel;
+//import org.springframework.ai.openai.api.OpenAiApi.ChatCompletionRequest;
 import org.springframework.ai.reader.ExtractedTextFormatter;
 import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
 import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
@@ -25,11 +26,16 @@
 import org.springframework.web.bind.annotation.RequestBody;
 import org.springframework.web.bind.annotation.RequestParam;
 import org.springframework.web.bind.annotation.RestController;
+import org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+
 import org.springframework.ai.vectorstore.oracle.OracleVectorStore;
 
 import jakarta.annotation.PostConstruct;
 
 import org.springframework.core.io.Resource;
+import org.springframework.http.MediaType;
 import org.springframework.jdbc.core.JdbcTemplate;
 
 import java.io.IOException;
@@ -38,14 +44,26 @@
 import java.util.ArrayList;
 import java.util.Map;
 import java.util.HashMap;
+import java.security.SecureRandom;
+import java.time.Instant;
+
 
 import java.util.Iterator;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+
+import org.springframework.model.*;
+
 @RestController
 class AIController {
 
+	@Value("${spring.ai.openai.chat.options.model}")
+	private String modelOpenAI;
+
+	@Value("${spring.ai.ollama.chat.options.model}")
+	private String modelOllamaAI;
+
 	@Autowired
 	private final OracleVectorStore vectorStore;
 
@@ -71,7 +89,8 @@ class AIController {
 	private JdbcTemplate jdbcTemplate;
 
 	private static final Logger logger = LoggerFactory.getLogger(AIController.class);
-
+	private static final int SLEEP = 50; 		// Wait in streaming between chunks
+	private static final int STREAM_SIZE = 5;    // chars in each chunk
 	AIController(ChatClient chatClient, EmbeddingModel embeddingModel, OracleVectorStore vectorStore) {
 
 		this.chatClient = chatClient;
@@ -169,14 +188,16 @@ public Prompt promptEngineering(String message, String contextInstr) {
 				INSTRUCTIONS:""";
 
 		String default_Instr = """
-					Answer the users question using the DOCUMENTS text above.
+				Answer the users question using the DOCUMENTS text above.
 				Keep your answer ground in the facts of the DOCUMENTS.
 				If the DOCUMENTS doesn’t contain the facts to answer the QUESTION, return:
 				I'm sorry but I haven't enough information to answer.
 				""";
 
-		//This template doesn't work with agent pattern, but only via RAG 
-		//The contextInstr coming from AI Optimizer can't be used here: default only
+		//This template doesn't work with re-phrasing/grading pattern, but only via RAG 
+		//The contextInstr coming from Oracle ai optimizer and toolkit can't be used here: default only
+		//Modifiy it to include re-phrasing/grading if you wish.
+
 		template = template + "\n" + default_Instr;
 
 		List<Document> similarDocuments = this.vectorStore.similaritySearch(
@@ -208,25 +229,70 @@ StringBuilder createContext(List<Document> similarDocuments) {
 		return context;
 	}
 
-	@PostMapping("/chat/completions")
-	Map<String, Object> completionRag(@RequestBody Map<String, String> requestBody) {
-
-		String message = requestBody.getOrDefault("message", "Tell me a joke");
-		Prompt prompt = promptEngineering(message, contextInstr);
-		logger.info(prompt.getContents());
-		try {
-			String content = chatClient.prompt(prompt).call().content();
-			Map<String, Object> messageMap = Map.of("content", content);
-			Map<String, Object> choicesMap = Map.of("message", messageMap);
-			List<Map<String, Object>> choicesList = List.of(choicesMap);
 
-			return Map.of("choices", choicesList);
+@PostMapping(value = "/chat/completions", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
+public ResponseBodyEmitter streamCompletions(@RequestBody ChatRequest request) {
+	ResponseBodyEmitter bodyEmitter = new ResponseBodyEmitter();
+	String userMessageContent;
 
-		} catch (Exception e) {
-			logger.error("Error while fetching completion", e);
-			return Map.of("error", "Failed to fetch completion");
+	for (Map<String, String> message : request.getMessages()) {
+		if ("user".equals(message.get("role"))) {
+			
+				String content = message.get("content");
+				if (content != null && !content.trim().isEmpty()) {
+					userMessageContent = content;
+					logger.info("user message: "+userMessageContent);
+					Prompt prompt = promptEngineering(userMessageContent, contextInstr);
+					logger.info("prompt message: "+prompt.getContents());
+					String contentResponse = chatClient.prompt(prompt).call().content();
+					logger.info("-------------------------------------------------------");
+					logger.info("- RAG RETURN                                          -");
+					logger.info("-------------------------------------------------------");
+					logger.info(contentResponse);
+					new Thread(() -> {
+						try {
+							ObjectMapper mapper = new ObjectMapper();
+				
+							if (request.isStream()) {
+								logger.info("Request is a Stream");
+								List<String> chunks= chunkString(contentResponse);
+								for (String token : chunks) {
+									
+									ChatMessage  messageAnswer = new ChatMessage("assistant", token);
+									ChatChoice choice = new ChatChoice(messageAnswer);
+									ChatStreamResponse chunk = new ChatStreamResponse("chat.completion.chunk", new ChatChoice[]{choice});
+									
+									bodyEmitter.send("data: " + mapper.writeValueAsString(chunk) + "\n\n");
+									Thread.sleep(SLEEP);
+								}
+								
+								bodyEmitter.send("data: [DONE]\n\n");
+							} else {
+								logger.info("Request isn't a Stream");
+								String id="chatcmpl-"+generateRandomToken(28);
+								String object="chat.completion";
+								String created=String.valueOf(Instant.now().getEpochSecond());
+								String model=getModel();
+								ChatMessage  messageAnswer = new ChatMessage("assistant", contentResponse);
+								List<ChatChoice> choices = List.of(new ChatChoice(messageAnswer));
+								bodyEmitter.send(new ChatResponse(id, object,created, model, choices));
+							}
+							bodyEmitter.complete();
+						} catch (Exception e) {
+							bodyEmitter.completeWithError(e);
+						}
+					}).start();
+				
+					return bodyEmitter;
+
+				}
+			break; 
 		}
 	}
+	
+
+	return bodyEmitter;
+}
 
 	@GetMapping("/service/search")
 	List<Map<String, Object>> search(@RequestParam(value = "message", defaultValue = "Tell me a joke") String query,
@@ -247,4 +313,77 @@ List<Map<String, Object>> search(@RequestParam(value = "message", defaultValue =
 		;
 		return resultList;
 	}
+
+	@GetMapping("/models")
+	Map<String, Object> models(@RequestBody (required = false) Map<String, String> requestBody) {
+		String modelId = "custom";
+		logger.info("models request");
+		if (!"".equals(modelOpenAI)) {
+			modelId = modelOpenAI;
+		} else if (!"".equals(modelOllamaAI)) {
+			modelId = modelOllamaAI;
+		} 
+		logger.info("model");
+		
+		
+		logger.info(chatClient.prompt().toString());
+		try {
+			Map<String, Object> model = new HashMap<>();
+			model.put("id", modelId);
+        	model.put("object", "model");
+        	model.put("created", 0000000000L);
+        	model.put("owned_by", "no-info");
+
+        	List<Map<String, Object>> dataList = new ArrayList<>();
+        	dataList.add(model);
+
+        	Map<String, Object> response = new HashMap<>();
+        	response.put("object", "list");
+        	response.put("data", dataList);
+
+			return response;
+
+		} catch (Exception e) {
+			logger.error("Error while fetching completion", e);
+			return Map.of("error", "Failed to fetch completion");
+		}
+	}
+
+
+	public List<String> chunkString(String input) {
+		List<String> chunks = new ArrayList<>();
+		int chunkSize = STREAM_SIZE; 
+	
+		for (int i = 0; i < input.length(); i += chunkSize) {
+			int end = Math.min(input.length(), i + chunkSize);
+			chunks.add(input.substring(i, end));
+		}
+	
+		return chunks;
+	}
+
+	public String generateRandomToken(int length) {
+		String CHARACTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
+    	SecureRandom random = new SecureRandom();
+        StringBuilder sb = new StringBuilder(length);
+        for (int i = 0; i < length; i++) {
+            int index = random.nextInt(CHARACTERS.length());
+            sb.append(CHARACTERS.charAt(index));
+        }
+        return sb.toString();
+    }
+
+	public String getModel(){
+		String modelId="custom";
+		if (!"".equals(modelOpenAI)) {
+			modelId = modelOpenAI;
+		} else if (!"".equals(modelOllamaAI)) {
+			modelId = modelOllamaAI;
+		} 
+		return modelId;
+	}
 }
+
+
+
+

src/client/spring_ai/src/main/java/org/springframework/ai/openai/samples/helloworld/Config.java

@@ -18,3 +18,6 @@ ChatClient chatClient(ChatClient.Builder builder) {
         return builder.build();
     }
 }
+
+
+