Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions assemblies/plugins/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1839,6 +1839,12 @@
<version>${project.version}</version>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.apache.hop</groupId>
<artifactId>hop-tech-microsoft365</artifactId>
<version>${project.version}</version>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.apache.hop</groupId>
<artifactId>hop-tech-minio</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ public enum HopMetadataPropertyType {
VFS_FTP_CONNECTION,
VFS_HDFS_CONNECTION,
VFS_SMB_CONNECTION,
VFS_SHAREPOINT_CONNECTION,
VFS_GIT_CONNECTION,

// DATA STREAM
Expand Down
2 changes: 2 additions & 0 deletions docs/hop-user-manual/modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -476,6 +476,7 @@ under the License.
* xref:vfs.adoc[Virtual File System]
** xref:vfs/aws-s3-vfs.adoc[Amazon Web Services S3]
** xref:vfs/azure-blob-storage-vfs.adoc[Microsoft Azure Storage]
** xref:vfs/sharepoint-vfs.adoc[Microsoft SharePoint]
** xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace]
** xref:vfs/dropbox-vfs.adoc[Dropbox]
** xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage]
Expand Down Expand Up @@ -564,6 +565,7 @@ under the License.
** xref:metadata-types/spark-catalog.adoc[Lakehouse catalog]
** xref:metadata-types/mail-server-connection.adoc[Mail Server Connection]
** xref:metadata-types/masking-pattern.adoc[Masking Pattern]
** xref:metadata-types/sharepoint-connection.adoc[SharePoint Connection]
** xref:metadata-types/minio-connection.adoc[Minio (S3) Connection]
** xref:metadata-types/mongodb-connection.adoc[MongoDB Connection]
** xref:metadata-types/graphs/neo4j-connection.adoc[Neo4j Connection]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ xref:metadata-types/data-stream/data-stream.adoc[Data Stream]: A Data Stream can
* xref:metadata-types/mongodb-connection.adoc[MongoDB Connection]: Describes a MongoDB connection
* xref:metadata-types/mail-server-connection.adoc[Mail Server Connection]: Describes a mail server connection
* xref:metadata-types/masking-pattern.adoc[Masking Pattern]: A reusable rule for replacing personal data in a field
* xref:metadata-types/sharepoint-connection.adoc[SharePoint Connection]: Named VFS access to the document libraries of the SharePoint sites of a Microsoft 365
* xref:metadata-types/minio-connection.adoc[Minio (S3) Connection]: Connect to one or more S3 endpoints using a Minio client
* xref:metadata-types/naming-scheme.adoc[Naming Scheme]: Reusable rules to normalize Hop field names, database identifiers, and file or folder names
* xref:metadata-types/webdav-connection.adoc[WebDAV Connection]: Named WebDAV or WebDAV-over-HTTPS endpoint used as a VFS scheme in file paths
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
:imagesdir: ../../assets/images/
:page-pagination:
:description: A named file system on the document libraries of Microsoft SharePoint sites, through the Microsoft Graph API

= SharePoint Connection

== Description

This metadata type adds the SharePoint sites of a Microsoft 365 tenant to your project as a file system, and describes how Apache Hop signs in to that tenant through the Microsoft Graph API.
The name of the connection is also a file system scheme: the SharePoint sites of the tenant, their document libraries and the files in them can be read and written with paths such as `+name:///site/library/folder/file.csv+`.
See xref:vfs/sharepoint-vfs.adoc[Microsoft SharePoint VFS] for the layout of these paths.

Hop signs in as an application, not as a person.
Register an application in Microsoft Entra ID, grant it Microsoft Graph *application* permissions and have an administrator consent to them:

* `Sites.Read.All` to read files
* `Sites.ReadWrite.All` to also create, replace, rename and delete files and folders
* `Sites.Selected` to only reach the sites the application was granted. These sites can not be listed: use the full path of the site, `+name:///site/library+`, and set the SharePoint host on the connection if the site is not found.

== Options

[options="header"]
|===
|Option |Description
|Name| Name of the connection, this is the scheme used in file paths: `name://`
|Description| Description field for additional information
|Authentication type a| How Hop signs in to Microsoft Entra ID:

* *Client secret authentication*: the client ID and a client secret of the registered application
* *Client certificate authentication*: the client ID and a certificate of the registered application
* *Managed identity*: the identity of the environment Hop runs in, found through the default Azure credential chain (managed identity, environment variables, Azure CLI, and so on)
|Tenant ID| The Directory (tenant) ID of your Microsoft Entra tenant
|SharePoint host| The SharePoint host of the tenant, for example `contoso.sharepoint.com`. Optional in most cases, see <<_sharepoint_host>>.
|Client ID| The Application (client) ID of the registered application. Not used with a managed identity.
|Client Secret| The client secret of the registered application, for client secret authentication
|PEM Certificate| The certificate and its private key in PEM format, for client certificate authentication
|===

All the options except the name, the description and the authentication type accept variables.

=== SharePoint host

The host is only used to find a site that is not in the list of the sites of the tenant, typically because the application is not allowed to list them.
Such a site is read by its address, `<host>/sites/<site>` or `<host>/teams/<site>`.

When the option is empty, Hop reads the root site of the tenant to find the host, so the connection usually works without it.
Set it when:

* The application can not read the root site, for example with `Sites.Selected` and no grant on the root site.
The log then says `The SharePoint host of the tenant can not be read, set it on the connection`.
* The sites are on another host than the root site of the tenant.
* You want to save the request that reads the root site.

When the application can list the sites (`Sites.Read.All` or `Sites.ReadWrite.All`), the sites are found by their name and the host is not used.

**Test** on this editor requests a Microsoft Graph token with the values in the form.

**Explore** on this editor opens the xref:hop-gui/vfs-file-explorer.adoc[VFS File Explorer] at `+name:///+`.
2 changes: 2 additions & 0 deletions docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ Click the File system name to access more detailed file system documentation.
|xref:vfs/google-drive-vfs.adoc[Google Drive]|Provides access to Google Drive folders|`googledrive://`
|xref:vfs/git-vfs.adoc[Git]|Named connection to a git repository checked out at a branch, tag or commit (marketplace plugin `hop-tech-git-vfs`)|`+<connectionName>:///path+`
|xref:vfs/hdfs-vfs.adoc[HDFS]|Named connection to HDFS via HttpFS, Knox or WebHDFS (marketplace plugin `hop-tech-hadoop`)|`+<connectionName>:///path+`
|xref:vfs/sharepoint-vfs.adoc[Microsoft SharePoint]|Named xref:metadata-types/sharepoint-connection.adoc[SharePoint Connection] to the document libraries of SharePoint sites|`+<connectionName>:///site/library/path+`
|xref:vfs/smb-vfs.adoc[SMB]|Named connection to an SMB2/SMB3 file share|`+<connectionName>:///path+`
|xref:metadata-types/minio-connection.adoc[Minio connection]|Provides access to S3-compatible endpoints via a named Minio connection (metadata)|`+<connectionName>://+`
|xref:metadata-types/webdav-connection.adoc[WebDAV Connection]|Provides access to WebDAV servers, either via the static schemes or a named connection (metadata)|`webdav4://`, `webdav4s://`, or `+<connectionName>://+`
Expand Down Expand Up @@ -330,6 +331,7 @@ Server-side restrictions (read-only mounts, bucket policies, account permissions
|`+<databricks-conn>+` |✓|✓|✗|✓|✓|✓|✗|✗|read
|`+<git-conn>+` (see below) |✓|✓|✓|✓|✓|✓|✗|✗|read
|`+<hdfs-conn>+` |✓|✓|✗|✓|✓|✓|✗|✗|read
|`+<sharepoint-conn>+` |✓|✓|✗|✓|✓|✓|✗|✗|read
|`+<smb-conn>+` |✓|✓|✓|✓|✓|✓|✓|✓|read
|`webdav4`, `webdav4s`, named WebDAV connection |✓|✓|✗|✓|✓|✓|✓|✗|read

Expand Down
88 changes: 88 additions & 0 deletions docs/hop-user-manual/modules/ROOT/pages/vfs/sharepoint-vfs.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////

:documentationPath: /vfs/
:language: en_US
:description: Apache Hop supports reading from and writing to Microsoft SharePoint document libraries from almost anywhere in the platform through Apache VFS

= Microsoft SharePoint VFS

== Scheme

There is no fixed scheme.
Every xref:metadata-types/sharepoint-connection.adoc[SharePoint Connection] registers a scheme with its own name, so a connection named `m365` gives you:

`**m365://**`

The credentials are stored in the connection, never in the path.

== Configuration

Create a xref:metadata-types/sharepoint-connection.adoc[SharePoint Connection] in the metadata perspective.
The application it signs in with needs the Microsoft Graph application permission `Sites.Read.All` to read files, or `Sites.ReadWrite.All` to also write them.

== Usage and testing

The path has three parts: the site, the document library, and the path of the file or folder in that library.

`+m365:///<site>/<library>/<folder>/<file>+`

[options="header",cols="2,3"]
|===
|Path|What it is
|`+m365:///+`|The SharePoint sites of the tenant
|`+m365:///Finance+`|The document libraries of the site `Finance`
|`+m365:///Finance/Documents+`|The root folder of the library `Documents`
|`+m365:///Finance/Documents/reports/2026/daily.csv+`|A file in that library
|===

* The site is the name of the site as it appears in its URL (`Finance` for `+https://contoso.sharepoint.com/sites/Finance+`).
* The library is the name of the document library.
The default library of a site is named `Documents` on an English tenant, even though its URL says `Shared Documents`.
* Site and library names are not case-sensitive.
When two sites have the same name, only the first one is reachable and the other one is reported in the log.
* With the permission `Sites.Selected` the root `+m365:///+` is empty, because the application can not list the sites. Set the SharePoint host on the connection and enter the path of a site it was granted, such as `+m365:///Finance+`.

You can browse the sites, libraries and files in the xref:hop-gui/vfs-file-explorer.adoc[VFS File Explorer] or in any file dialog.

== Microsoft Teams channels

The files of a Microsoft Teams team are stored in a SharePoint site, so they are reached with the same paths.
Where the files of a channel are depends on the type of the channel.

[options="header",cols="1,2,3"]
|===
|Channel|Site|Path
|Standard|The site of the team|`+m365:///<team>/Documents/<channel>/<file>+`
|Private or shared|A site of its own, usually named `<team>-<channel>`|`+m365:///<team>-<channel>/Documents/<channel>/<file>+`
|===

* A channel is a folder of the library `Documents`, never a site or a library by itself.
The folder of the default channel is named `General`.
* The name of the site of a team is not always the name of the team.
To find the exact names, open the *Files* tab of the channel in Teams and choose *Open in SharePoint*: the site is what follows `/sites/` or `/teams/` in the URL, the folder is what follows `Shared Documents/`.
* Do not put `/sites` or `/teams` in the path, only the name of the site.
* With the permission `Sites.Selected`, the site of a private or shared channel has to be granted to the application separately from the site of the team.

== Limitations

* Sites and document libraries can be listed but not created, renamed or deleted.
* Appending to a file is not supported.
* A file or folder can only be renamed or moved inside its document library.
Moving it to another library or site copies the content and deletes the original.
* A file is buffered in a local temporary file while it is written and is sent to SharePoint when it is closed.
Files larger than 4 MB are sent in slices through an upload session.
Loading
Loading