Use SharePoint Data Source with new Cloud Indexing Feature

[jack-vinitsky]

The documentation around the new Cloud Indexing option specifies the use of an Azure BLOB store for document storage. If one were to replace the BLOB data source with a SharePoint one, (putting aside some changes to metadata properties) would the rest of the ingestion pipeline still work as intended?

[pamelafox]

Hm, to use indexed Sharepoint ingestion? These days we would recommend using the new remote Sharepoint ingestion, supported by the Agentic Retrieval feature. Have you considered that?

[jack-vinitsky]

Hi Pamela. Thanks for the quick response.

While I think the agentic retrieval feature with SharePoint is interesting, it does not solve for the following:

1. Using SharePoint Document Libraries to store a curated set of documents that in turn can easily be managed by key business users (ex: research papers, corporate policies).
2. Use SharePoint library columns for custom metadata. These columns can be used for Semantic Search configuration and Boosting within Azure AI Search.
3. The agentic retrieval feature requires a paid Copilot license and there is no guarantee that SharePoint document content that is redundant, obsolete, or trivial is not brought in.

I know we can build an integration between SharePoint and Azure BLOB storage, but it seems like an extra step that does not provide much additional value aside from compatibility with your new ingestion strategy (which once again I think is great).

We have been using the SharePoint Indexer/Data Source in AI Search with the https://github.com/microsoft/sample-app-aoai-chatGPT project for over a year despite it still being in preview mode and none of the other skills themselves seem to have any issue operating with documents brought in this way.

[jack-vinitsky]

It appears Microsoft just added ingestion of Document ACLs from SharePoint.

https://learn.microsoft.com/en-us/azure/search/search-indexer-sharepoint-access-control-lists

From my vantage point supporting this feature is not a huge necessity, in all our cases if you have access to the chatbot, you get read access to the underlying documents by adding the same Azure Domain group assigned to the App Registration to the SharePoint library.

I can see other organizations valuing this differently though.

[jack-vinitsky]

I suppose displaying PDF documents that are stored in a SharePoint library within the current document viewer experience may also need to be adjusted. IFRAMING a PDF document from SharePoint is supported from what I understand.

[pamelafox]

@jack-vinitsky Can you add a feature request in the issue tracker for this? @mattgotteiner thinks we could add it as an option to CloudIngestionStrategy.

I don't think we're iframing the Sharepoint documents yet, that might need to be a separate issue if you prefer that. Right now I'm opening them in a new page, when we get them from the remote SharePoint source.

[jack-vinitsky]

Will do!

Opening SharePoint files in a new tab/page I think would be fine. Framing them in the app would be nice, but like you said we should treat that as a separate feature.

[jack-vinitsky]

#2857 As requested