Feature/add file support for telegram by pulinduvidmal · Pull Request #184 · yaalalabs/agent-kernel

pulinduvidmal · 2025-12-23T15:10:12Z

Description

This change will add file & image support for telegram integrations

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)
Performance improvement
Test update
CI/CD update
Other (please describe):

Testing

Unit tests pass locally
Integration tests pass locally
Manual testing completed
New tests added for changes

Checklist

My code follows the project's style guidelines
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published

Screenshots (if applicable)

Additional Notes

Copilot

Pull request overview

This PR adds multimodal support (images and files) to the Telegram integration, enabling users to send photos and documents alongside text messages for AI analysis. The changes also include session memory improvements to preserve conversation context with multimodal inputs.

Added file and image download functionality with base64 encoding for Telegram messages
Implemented session memory persistence for multimodal conversations in OpenAI framework
Added new Google ADK example (server_adk.py) demonstrating alternative agent framework
Updated documentation with comprehensive multimodal features guide

Reviewed changes

Copilot reviewed 7 out of 23 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`ak-py/src/agentkernel/integration/telegram/telegram_chat.py`	Core implementation adding file/image download, processing, and multi-request handling
`ak-py/src/agentkernel/framework/openai/openai.py`	Session memory fix to manually save multimodal conversations for future reference
`examples/api/telegram/server_adk.py`	New example showing Google ADK framework integration with Telegram
`examples/api/telegram/build.sh`	Added `adk` to package extras for Google ADK support
`examples/api/telegram/README.md`	Comprehensive documentation update with multimodal usage examples
`docs/docs/integrations/telegram.md`	Integration guide updated with multimodal features, supported formats, and limitations
`ak-py/src/agentkernel/integration/telegram/README.md`	Technical documentation for multimodal support and file handling
Multiple `uv.lock` files	Dependency updates including openai-agents downgrade from 0.6.4 to 0.6.3

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

ak-py/src/agentkernel/framework/openai/openai.py

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

…upport_for_telegram

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 14 out of 29 changed files in this pull request and generated 17 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

ak-py/src/agentkernel/core/attachment.py

docs/docs/integrations/telegram.md

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

ak-py/src/agentkernel/core/runtime.py

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

ak-py/src/agentkernel/core/runtime.py

ak-py/src/agentkernel/core/multimodal.py

Copilot

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ak-py/src/agentkernel/core/multimodal/hooks.py

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

ak-py/src/agentkernel/core/multimodal/storage/session_cache.py

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

ak-py/src/agentkernel/framework/openai/openai.py

ak-py/src/agentkernel/core/multimodal/hooks.py

docs/docs/advanced/multimodal.md

ak-py/src/agentkernel/core/multimodal/tools.py

docs/docs/integrations/telegram.md

docs/docs/advanced/multimodal.md

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

ak-py/src/agentkernel/framework/langgraph/langgraph.py:426

StructuredTool.from_function is called with func=None for coroutine tools. In LangChain, from_function typically expects a real callable for func (even when coroutine is provided); passing None risks a runtime error when binding async tools. Consider passing func=func as well, or using the dedicated async tool constructor/pattern supported by your pinned langchain_core version.

            if asyncio.iscoroutinefunction(func):
                tools.append(
                    StructuredTool.from_function(
                        func=None,
                        coroutine=func,
                        name=func.__name__,
                        description=func.__doc__ or func.__name__,
                    )
                )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/docs/advanced/multimodal.md

amithad · 2026-03-04T05:35:05Z

ak-py/src/agentkernel/core/multimodal/storage/in_memory.py

+    AttachmentStorageDriver,
+)
+
+_log = logging.getLogger("ak.core.multimodal.storage.in_memory")


This should be a class variable

amithad · 2026-03-04T05:35:38Z

ak-py/src/agentkernel/core/multimodal/storage/dynamodb.py

+    AttachmentStorageDriver,
+)
+
+_log = logging.getLogger("ak.core.multimodal.storage.dynamodb")


This should be a class variable

amithad · 2026-03-04T05:38:03Z

ak-py/src/agentkernel/core/multimodal/storage/redis.py

+_log = logging.getLogger("ak.core.multimodal.storage.redis")
+
+
+class RedisStorageDriver(AttachmentStorageDriver):


All ak related redis keys should have a ak prefix (Refer to config.session.redis for an example)

amithad · 2026-03-04T05:39:27Z

ak-py/src/agentkernel/core/multimodal/storage/session_cache.py

+        if current and current.id == session_id:
+            session = current
+        else:
+            from ...runtime import Runtime


Add lazy loading only when required

amithad · 2026-03-04T05:42:11Z

ak-py/src/agentkernel/core/multimodal/storage/storage_manager.py

+    """
+    High-level API for attachment storage.
+
+    Resolves the storage driver from ``AKConfig.multimodal.storage_type``


Remove AI generated unrelated comments

amithad · 2026-03-04T05:42:49Z

ak-py/src/agentkernel/core/multimodal/storage/storage_manager.py

+"""
+Attachment storage manager for multimodal memory.
+
+This module provides the ``AttachmentStorageManager`` class — a high-level API


This is outdated

amithad · 2026-03-04T05:48:50Z

ak-py/src/agentkernel/framework/langgraph/langgraph.py

            if asyncio.iscoroutinefunction(func):
                tools.append(
                    StructuredTool.from_function(
+                        func=None,


Why do we need this?

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-04T06:46:26Z

ak-py/src/agentkernel/integration/messenger/README.md

+```bash
+export AK_MULTIMODAL__ENABLED=true              # Enable multimodal support (default: true)
+export AK_MULTIMODAL__MAX_ATTACHMENTS=5         # Keep last N files in session (default: 5)
+export AK_MULTIMODAL__ATTACHMENT_TTL=604800     # File lifetime in seconds (default: 604800 = 1 week)
+```


This README documents AK_MULTIMODAL__ENABLED as defaulting to true and introduces AK_MULTIMODAL__ATTACHMENT_TTL, but the config defaults to enabled=false and there is no attachment_ttl field/env var (TTL is storage-backend specific, e.g. AK_MULTIMODAL__REDIS__TTL). Please update the documented env vars and defaults to match AKConfig.multimodal.

Copilot · 2026-03-04T06:46:26Z

ak-py/src/agentkernel/core/multimodal/storage/redis.py

+    ``{prefix}{session_id}:{attachment_id}``.  An additional sorted-set
+    ``{prefix}{session_id}:_index`` tracks attachment order for pruning.


Docstring says the Redis index is a “sorted-set”, but the implementation uses a Redis list (RPUSH/LPOP/LLEN). Update the docstring to match the actual data structure (or switch implementation to a sorted set) to avoid operational confusion when inspecting Redis keys.

Suggested change

``{prefix}{session_id}:{attachment_id}``. An additional sorted-set

``{prefix}{session_id}:_index`` tracks attachment order for pruning.

``{prefix}{session_id}:{attachment_id}``. An additional Redis list

``{prefix}{session_id}:_index`` tracks attachment order (insertion order)

for pruning.

Copilot · 2026-03-04T06:46:27Z

ak-py/src/agentkernel/core/multimodal/tools.py

+def analyze_attachments(attachment_ids: list[str], prompt: str) -> str:
+    """
+    Analyze attachments (images/files) using LLM and return ONLY the analysis response.
+
+    :param attachment_ids: List of attachment IDs to analyze
+    :param prompt: The question/prompt for analyzing the attachments
+    :return: Only the LLM analysis response text
+    """
+    if not attachment_ids:
+        return "No attachments provided"
+
+    try:
+        from ..tool import ToolContext
+
+        ctx = ToolContext.get()
+        session = ctx.session
+
+        attachments = AttachmentStorageManager(session_id=session.id).get_attachment_data(attachment_ids=attachment_ids)
+
+        if not attachments:
+            return "No attachments found for the given IDs in this session"
+


This new multimodal tool/path (attachment storage + analyze_attachments + prehook injection) isn’t covered by unit tests in this PR. Add tests that (1) store attachments in a session via the prehook, (2) ensure raw attachments are removed/injected as expected, and (3) verify analyze_attachments can retrieve session-scoped attachments (for at least in-memory + one persistent driver stub).

Copilot · 2026-03-04T06:46:27Z

ak-py/src/agentkernel/integration/telegram/telegram_chat.py

        @router.post("/telegram/webhook")
-        async def handle_webhook(request: Request):
+        async def handle_webhook(request: Request, background_tasks: BackgroundTasks):
            """
            Handle incoming Telegram webhook updates.
            """
-            return await self._handle_webhook(request)
+            # Read body first to avoid stream consumption issues in background
+            body = await request.json()
+            background_tasks.add_task(self._process_webhook_body, body, request.headers)
+            return {"ok": True}


Webhook secret validation is now deferred to the background task, but the HTTP handler always returns 200/{"ok": true} even when the X-Telegram-Bot-Api-Secret-Token is invalid. This weakens auth and makes it hard to detect misconfiguration/attacks (Telegram will think delivery succeeded). Validate the secret token in handle_webhook before scheduling the background task and return a 403 on mismatch; only enqueue _process_webhook_body when authorized.

Copilot

Pull request overview

Copilot reviewed 68 out of 81 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-04T19:50:37Z

ak-py/src/agentkernel/framework/langgraph/langgraph.py

@@ -233,6 +237,31 @@ def get_a2a_card(self):
        # TODO extract description from graph
        return A2ACardBuilder.build(name=self.name, description="", skills=skills)

+    def attach_tool(self, tool: Any) -> None:
+        """
+        Accepts a raw Callable and wraps it with LangGraphToolBuilder before storing.
+        Follows the same pattern as ADK, OpenAI, and CrewAI.
+        Note: LangGraph tools must be passed to the graph BEFORE compile(); these wrapped
+        tools are stored on the agent wrapper for inspection and future use.
+        :param tool: Raw Python callable or already-wrapped LangChain StructuredTool.
+        """
+        # Delegate to the tool builder to handle binding
+        wrapped = LangGraphToolBuilder.bind([tool])
+        for w in wrapped:
+            if w not in self._tools:
+                self._tools.append(w)
+
+    def override_system_prompt(self, prompt: str) -> None:
+        """
+        Stores the system prompt suffix on the agent wrapper.
+        Follows the same pattern as ADK, OpenAI, and CrewAI.
+        Note: LangGraph compiled graphs do not expose a mutable system prompt field;
+        the value is stored here for inspection. Set the prompt in your graph nodes
+        before compile() for it to take effect at inference time.
+        """
+        if prompt not in self._system_prompt:
+            self._system_prompt += ("\n" if self._system_prompt else "") + prompt
+


LangGraphAgent.attach_tool() and override_system_prompt() only store values on the wrapper (self._tools, self._system_prompt), but LangGraphRunner.run() never uses them when invoking the compiled graph. This means system tools like analyze_attachments are effectively not attached for LangGraph (despite being auto-attached by the base Agent), and the injected system prompt suffix is never applied. Either wire these into the graph before compile / invocation (e.g., ensure create_react_agent(..., tools=[...]) includes the system tools) or explicitly disable auto-attachment for LangGraph and document that users must add the tool/prompt manually.

examples/api/multimodal/openai/config.yaml

examples/api/multimodal/langgraph/config.yaml

Copilot · 2026-03-04T19:50:38Z

examples/api/multimodal/redis/get-docker.sh

+#!/bin/sh
+set -e
+# Docker Engine for Linux installation script.
+#
+# This script is intended as a convenient way to configure docker's package
+# repositories and to install Docker Engine, This script is not recommended
+# for production environments. Before running this script, make yourself familiar
+# with potential risks and limitations, and refer to the installation manual
+# at https://docs.docker.com/engine/install/ for alternative installation methods.
+#


This repo now vendors the full upstream get.docker.com installer script (~760 lines). That makes it hard to keep in sync with upstream security/compatibility updates and encourages running a privileged installer from the repo. Consider removing this file and linking to the official Docker installation docs (or documenting a minimal docker run redis:alpine ... alternative) instead of vendoring the whole installer.

Copilot · 2026-03-04T19:50:39Z

examples/api/multimodal/crewai/config.yaml

+multimodal:
+  enabled: true
+  storage_type: inmemory


storage_type: inmemory doesn’t match the allowed values in the config schema (in_memory, redis, dynamodb, session_cache). With the current schema, this YAML will fail validation / won’t load as intended. Use storage_type: in_memory (or omit it, since in-memory is the default).

Copilot · 2026-03-04T19:50:39Z

examples/api/multimodal/adk/config.yaml

+multimodal:
+  enabled: true
+  storage_type: inmemory


storage_type: inmemory doesn’t match the allowed values in the config schema (in_memory, redis, dynamodb, session_cache). With the current schema, this YAML will fail validation / won’t load as intended. Use storage_type: in_memory (or omit it, since in-memory is the default).

Copilot · 2026-03-04T19:50:39Z

ak-py/src/agentkernel/integration/telegram/telegram_chat.py


 import httpx
-from fastapi import APIRouter, HTTPException, Request
+from fastapi import APIRouter, BackgroundTasks, HTTPException, Request


HTTPException is imported but no longer used (webhook processing now returns early instead of raising). Consider removing the unused import to avoid lint warnings.

Suggested change

from fastapi import APIRouter, BackgroundTasks, HTTPException, Request

from fastapi import APIRouter, BackgroundTasks, Request

pulinduvidmal added 3 commits December 23, 2025 21:13

feat: add file support for messenger integration

9014f6b

feat: add multimodal file/image support for telegram

5222577

feat: add multimodal file/image support for telegram

54def41

pulinduvidmal force-pushed the feature/add_file_support_for_telegram branch from 09b0b41 to 54def41 Compare December 23, 2025 15:47

pulinduvidmal added 3 commits December 23, 2025 21:25

fix: resolve messenger integration conflicts from rebase

d0199b6

docs: update doc

698747d

fix: resolve messenger integration conflicts from rebase

bb4c1b3

tharindud requested a review from Copilot December 26, 2025 06:35

Copilot started reviewing on behalf of tharindud December 26, 2025 06:35 View session

Copilot AI reviewed Dec 26, 2025

View reviewed changes

feat: Add multimodal memory support

a906b91

pulinduvidmal force-pushed the feature/add_file_support_for_telegram branch from 7b6ec6a to a906b91 Compare December 28, 2025 21:02

pulinduvidmal and others added 10 commits December 29, 2025 09:15

feat: Add multimodal memory support

1cc6d53

Merge remote-tracking branch 'origin/develop' into feature/add_file_s…

5cb693f

…upport_for_telegram

fix: Add multimodal memory support

d99ae92

fix: Add multimodal memory support

d1f68e5

fix: telegram_chat.py

7c6d40c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

docs: Add multimodal memory support

84330c1

fix telegram_chat.py

7e08a7c

fix telegram_chat.py

d9e601b

fix telegram_chat.py

8f7733c

feat: add file/image attachment support with two-step LLM

94a350f

Copilot AI review requested due to automatic review settings January 12, 2026 19:19

Copilot started reviewing on behalf of pulinduvidmal January 12, 2026 19:19 View session

Copilot AI reviewed Jan 12, 2026

View reviewed changes

pulinduvidmal added 2 commits January 13, 2026 01:17

feat: add file/image attachment support with two-step LLM

94cf7a9

refactor: split multimodal into package

e6d9ba1

Copilot AI review requested due to automatic review settings January 12, 2026 20:16

Copilot started reviewing on behalf of pulinduvidmal January 12, 2026 20:16 View session

refactor: split multimodal into package

b529b66

Copilot started reviewing on behalf of pulinduvidmal March 3, 2026 16:19 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

pulinduvidmal added 2 commits March 3, 2026 23:19

refactor: address review comments on base.py and tool.py

612127b

refactor: address review comments of copilot

27eb3ef

Copilot AI review requested due to automatic review settings March 3, 2026 18:21

Copilot started reviewing on behalf of pulinduvidmal March 3, 2026 18:22 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

pulinduvidmal added 2 commits March 4, 2026 00:25

refactor: address review comments of copilot

5579e37

refactor: address review comments of copilot

1ac3cef

Copilot AI review requested due to automatic review settings March 3, 2026 19:10

Copilot started reviewing on behalf of pulinduvidmal March 3, 2026 19:10 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

docs/docs/advanced/multimodal.md Outdated Show resolved Hide resolved

docs/docs/advanced/multimodal.md Show resolved Hide resolved

fix: docs

744cac5

amithad reviewed Mar 4, 2026

View reviewed changes

amithad requested a review from Copilot March 4, 2026 06:39

Copilot started reviewing on behalf of amithad March 4, 2026 06:39 View session

amithad requested changes Mar 4, 2026

View reviewed changes

Copilot AI reviewed Mar 4, 2026

View reviewed changes

pulinduvidmal added 3 commits March 4, 2026 21:46

feat: add DynamoDB multimodal storage example

1a5995b

feat: add radis and unit tests

3880503

fix: adding isort and blck to examples

3aedd61

Copilot AI review requested due to automatic review settings March 4, 2026 19:42

Copilot started reviewing on behalf of pulinduvidmal March 4, 2026 19:43 View session

Copilot AI reviewed Mar 4, 2026

View reviewed changes

fix: config

9405a30

		_log = logging.getLogger("ak.core.multimodal.storage.redis")


		class RedisStorageDriver(AttachmentStorageDriver):

		``{prefix}{session_id}:{attachment_id}``. An additional sorted-set
		``{prefix}{session_id}:_index`` tracks attachment order for pruning.

	from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
	from fastapi import APIRouter, BackgroundTasks, Request

Conversation

pulinduvidmal commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Testing

Checklist

Screenshots (if applicable)

Additional Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

amithad Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

amithad Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

amithad Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

amithad Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

amithad Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

amithad Mar 4, 2026

pulinduvidmal commented Dec 23, 2025 •

edited

Loading