ClassNotFoundException in remote api call

Hi,

We see new issue in remote api calling datastore in a transaction.

Java 25 / EE11 / Jetty 12.1
Appengine SDK 4.0.1

com.google.apphosting.datastore_bytes.proto2api.DatastoreV3Pb$PutRequestOrBuilder
java.lang.ClassNotFoundException: com.google.apphosting.datastore_bytes.proto2api.DatastoreV3Pb$PutRequestOrBuilder
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490)
	at com.google.appengine.tools.remoteapi.TransactionBuilder.makeCommitRequest(TransactionBuilder.java:140)
	at com.google.appengine.tools.remoteapi.RemoteDatastore.handleCommit(RemoteDatastore.java:251)
	at com.google.appengine.tools.remoteapi.RemoteDatastore.handleDatastoreCall(RemoteDatastore.java:96)
	at com.google.appengine.tools.remoteapi.RemoteApiDelegate.makeDefaultSyncCall(RemoteApiDelegate.java:71)
	at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate.makeSyncCall(StandaloneRemoteApiDelegate.java:60)
	at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate.lambda$makeAsyncCall$0(StandaloneRemoteApiDelegate.java:70)

Thanks
Magnus

Looking.
QQ: when you use a dep on the remote-apis, do you also put the same version number for the

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-api-1.0-sdk</artifactId>
            <version>${appengine.target.version}</version>.  <<<< must be same version as used in <artifactId>appengine-remote-api</artifactId>
        </dependency>

Thanks, if not, this is a real bug.

Thanks for the quick response. Yes we use the same dependency version for both the sdk and the remote-api

runtimeClasspath - Runtime classpath of source set 'main'.
+--- com.google.appengine:appengine-remote-api:4.0.1
|    \--- com.google.appengine:appengine-api-1.0-sdk:4.0.1
+--- jakarta.servlet:jakarta.servlet-api:6.1.0
\--- com.google.appengine:appengine-api-1.0-sdk:4.0.1

Enclose small isolated project to reproduce, hope this helps.
Unpack and run with ./gradlew test

remote-api-test

It only seems to happen when in a transaction scope.

Ack, Thanks!, i seems that the shading maven process is fragile for the corner case of inner classes.
So currently, your working version is with a 3.x.x version? Just to make sure you are not stuck while we look at the fix.
4.x.x has complete proto2 usage for client datastore, and my guess is generated inner classes in proto2 version do not have same name patterns than in proto1.

Working on a fix. While doing that, discovered that appengine:stop was not working anymore, so causing timeout delays in the full e2e test.
Filed GoogleCloudPlatform/appengine-plugins#1067 and proposed a fix. This is not critical, just annoying UX.

Fix shading of DatastoreV3Pb inner classes in remoteapi, fixing #475 …

While there is a potential fix, I am not sure about the support level of ancient remote API defined in old java7/8 runtimes. Do you really need them in your newer runtime environment? Doc for support here for newer runtimes is not clear.
I would be interested in your use case.

Still need to push to Maven Central before you test.

We use Remote API as a test-only bridge from the integration test JVM to a running App Engine service (or devserver) so the test can:

Seed/clean datastore data at the start/end of each test run.

Execute flows that require the app to run “for real” (HTTP + auth + task execution), while still being able to programmatically populate fixtures.

We are on 2.x branch in prod currently.

Thanks!

Great to know your use case is limited for UX ease of development locally only.

So we pushed a new 4.0.2 artifacts version on Maven central. Can you try again with this newer version?
the plugin stop feature will be in a newer plugin version whenever GoogleCloudPlatform/appengine-plugins#1069 is in and a new Maven release of the plugin is published.

Hi, yes we tested and now I see a new issue, not sure yet if this is related or not.


java.lang.NoClassDefFoundError: com/google/storage/onestore/v3_bytes/proto2api/OnestoreEntity$ReferenceOrBuilder
	at com.google.appengine.tools.remoteapi.TransactionBuilder.makeEntityNotFoundPrecondition(TransactionBuilder.java:180)
	at com.google.appengine.tools.remoteapi.TransactionBuilder.makeCommitRequest(TransactionBuilder.java:143)
	at com.google.appengine.tools.remoteapi.RemoteDatastore.handleCommit(RemoteDatastore.java:251)
	at com.google.appengine.tools.remoteapi.RemoteDatastore.handleDatastoreCall(RemoteDatastore.java:96)
	at com.google.appengine.tools.remoteapi.RemoteApiDelegate.makeDefaultSyncCall(RemoteApiDelegate.java:71)
	at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate.makeSyncCall(StandaloneRemoteApiDelegate.java:60)
	at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate.lambda$makeAsyncCall$0(StandaloneRemoteApiDelegate.java:70)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.lang.ClassNotFoundException: com.google.storage.onestore.v3_bytes.proto2api.OnestoreEntity$ReferenceOrBuilder
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490)
	... 11 more
Cause 1: java.lang.ClassNotFoundException: com.google.storage.onestore.v3_bytes.proto2api.OnestoreEntity$ReferenceOrBuilder
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490)
	at com.google.appengine.tools.remoteapi.TransactionBuilder.makeEntityNotFoundPrecondition(TransactionBuilder.java:180)
	at com.google.appengine.tools.remoteapi.TransactionBuilder.makeCommitRequest(TransactionBuilder.java:143)
	at com.google.appengine.tools.remoteapi.RemoteDatastore.handleCommit(RemoteDatastore.java:251)
	at com.google.appengine.tools.remoteapi.RemoteDatastore.handleDatastoreCall(RemoteDatastore.java:96)
	at com.google.appengine.tools.remoteapi.RemoteApiDelegate.makeDefaultSyncCall(RemoteApiDelegate.java:71)
	at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate.makeSyncCall(StandaloneRemoteApiDelegate.java:60)
	at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate.lambda$makeAsyncCall$0(StandaloneRemoteApiDelegate.java:70)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)

I see you closed this issue so let me know if I need to create a new one.

Thanks
Magnus

this is a new one, but keeping the context with this bug.

For some reasons, the repackaging to com.google.appengine.repackaged.com.google.storage.onestore.v3_bytes.proto2api is not functional, but it is there

appengine-java-standard/remoteapi/pom.xml

Line 152 in 74dfb85

<pattern>com.google.storage.onestore.v3_bytes.proto2api</pattern>

So i am on something, but do not have a test (yet?) to prove it might work or not, so if you could help me, that would be great.
Here is a jar file to test(renamed as zip since I cannot attach jar, only zip). Could you install it locally in your build env and run the test?

appengine-remote-api-4.0.3-SNAPSHOT.zip

I think it can be done via

./gradlew publishToMavenLocal \
  -PpubGroupId=com.google.appengine \
  -PpubArtifactId=appengine-remote-api \
  -PpubVersion=4.0.3-SNAPSHOT \
  -PpubRawFile=/path/to/appengine-remote-api-4.0.3-SNAPSHOT.jar

Ideally, your test project could be modified to do a repro?

Hi again and thanks for you help. I have updated my test project and added a new test that fails.

datastore-test.tar.zip

Tried locally with SNAPSHOT version and looks like issue is still there.

So I`m a Maven guy so moved your test to Maven but could not repro. Instead saw a XG transaction error.

But Your gradle test can repro.
I then deleted in your gradle build these 3 lines

    testImplementation 'com.google.appengine:appengine-testing:4.0.3-SNAPSHOT'
    testImplementation 'com.google.appengine:appengine-api-stubs:4.0.3-SNAPSHOT'
    testImplementation 'com.google.appengine:appengine-tools-sdk:4.0.3-SNAPSHOT'

(or 4.0.2) and I cannot repro the NoClassDefFoundError error, but a new 'user error' error:

com.google.apphosting.api.ApiProxy$ApplicationException: ApplicationError: 1: cross-group transaction need to be explicitly specified, see TransactionOptions.Builder.withXGfound both Element {
  type: "test-kind"
  id: 123
}
 and Element {
  type: "test-kind"
  id: 0x14000000000000
}

This error occurs because the Google App Engine Datastore requires all entities in a standard transaction to belong to the same entity group. Since both entities in your log are root entities (they have no parent/ancestor), they each belong to their own separate entity group.

By default, Datastore transactions are "Single-Group." To operate on multiple root entities at once, you must explicitly enable Cross-Group (XG) Transactions.

The Solution: Enable XG Transactions:You need to pass a TransactionOptions object when you start your transaction.

import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Transaction;
import com.google.appengine.api.datastore.TransactionOptions;
...
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();

// Enable Cross-Group (XG) support
TransactionOptions options = TransactionOptions.Builder.withXG(true);
Transaction txn = datastore.beginTransaction(options);

Would that be your next test error once the NoClassDefFoundError is not there anymore? Does it make sense for your test or your application?

Anyway, it seems that the 3 test jars added in the gradle dependencies are the one conflicting with the test environment and raising a NoClassDefFoundError.

I will continue tomorrow, trying 1 extra jar at a time to see which one is the cause.

Ah right true I forgot about the XG in the test, we use this normally so it was just in the haste of creating the test app this was missed by me. It didn't affect my test though.

Thanks for your help!

While I see it is documented under https://docs.cloud.google.com/appengine/docs/standard/java-gen2/using-gradle
I am puzled with the need for both appengine-api-stubs and appengine-tools-sdk in a test environment.
appengine-testing is also for unit tests, not e2e IT tests.

Would you mind letting me know if you can remove these 3 deps?

Ok, will try to remove these libraries from the IT test class path hopefully resolving our issue. Its not been a problem before so never thought it would interfere so thanks for the suggestion.

All test pass now for us so we can continue our move to java 25. Thanks for your support.

Thanks! Classloader mysteries are complex when multiple jars contain similar classes, especially with context classloaders.
There is definitely something fishy there, but very hard to pin point so I will stop investigating more this scenario.
Anyway, I think it is cleaner to separate unit tests (using only appengine-testing) and your e2e IT tests.
Also you should never rely on appengine-tools-sdk directly (local devappserver, using with the run/start commands) and appengine-api-stubs used by the api emulator used by also local devappserver. These dependencies are used for our internal tools, so not really consumeable by apps directly.
Glad you are on track on java25... In your test app, I noticed you used virtual threads setting for Jetty. It is not used for local devappserver. Let us know if you see differences in prod testings. The Jetty guys are not convinced yet... Also you might want to explore new jvm options specific for JDK25 via JAVA_USER_OPTS (see https://github.com/GoogleCloudPlatform/appengine-java-standard/blob/main/README.md)

Keep the comm channel open, thanks!

Ok, thanks good to know. We have turned on virtual threads in production since java 21.

Hi again, I have follow up questions. We investigated the possibility to improve app performance by turning on the appengine.use.HttpConnector setting in the appengine-web.xml after reading webtides blogpost and your own benchmark evaluation showing good result in both memory pressure and cpu utilization on appengine apps. This came after your comment on us using virtual threads. When testing this though we see unexpected response spikes we do not see otherwise. I just want to check if this flag is recommended use or not?

Also we have tested this flag in combination with turning on/off virtual threading but seems we have these spikes regardless of that.

Hi,
What is an unexpected spike?
We are pushing and experiment where this flag is on more and more transparently. If your jvm process is selected, you would see a http connector log at jvm boot.
Also, the user defined flag might have no effect, so let us know what is the skipe, but yes the idea is to move all apps to the new modern connector.

The experiment was up 100% a week again, then we re launched it slowly .
It was also up last summer...with 2 issues detected that took Q4 to fix.

Hi,
Ok, interesting wasn't aware of this.

We're a bit unsure what is going on at the moment, but we have observed a difference in behaviour between Java 25/EE11 and our current production version when turning on this flag according to the docs we read.

In our test/staging environment we see response latency spikes up to several minutes when using the configuration below. The spikes appear to happen on seemingly random requests.

The attached graph shows response latency across different percentiles.

During the periods without the high spikes, we were not using virtualthreads and httpConnector together. When they are not enabled in combination, the system seems to behave normally.

So far it looks like the spikes only occur when both virtualthreads and httpConnector are enabled simultaneously.

Hope this helps clarify what we’re seeing. Let me know if you need any additional details.

Checking production logs we do see the "HTTP Connector" in our logs, so we will go ahead and remove this setting.

Thanks

    <system-properties>
        <property name="appengine.use.EE11" value="true"/>
        <property name="appengine.use.virtualthreads" value="true"/>
        <property name="appengine.api.urlfetch.defaultDeadline" value="60"/>
        <property name="appengine.use.HttpConnector"
                  value="true"/> <!--https://webtide.com/google-app-engine-performance-improvements/-->
    </system-properties>

Java 25/EE11 version with above configurations (the period without high response we didn't have both http connector and virtualthreads == true)

Our producation configuration is


    <system-properties>
        <property name="appengine.use.EE8" value="false"/>
        <property name="appengine.use.EE10" value="true"/>
        <property name="appengine.use.virtualthreads" value="true"/>
        <property name="appengine.api.urlfetch.defaultDeadline" value="60"/>
    </system-properties>

Navigation Menu

Repository navigation

ClassNotFoundException in remote api call #475

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ClassNotFoundException in remote api call #475

Description

Activity

ludoch commented on Feb 10, 2026

masc71 commented on Feb 11, 2026

ludoch commented on Feb 11, 2026

ludoch commented on Feb 12, 2026

ludoch commented on Feb 12, 2026

masc71 commented on Feb 12, 2026

ludoch commented on Feb 12, 2026

ludoch commented on Feb 13, 2026

masc71 commented on Feb 17, 2026

ludoch commented on Feb 17, 2026

ludoch commented on Feb 18, 2026

ludoch commented on Feb 18, 2026

masc71 commented on Feb 18, 2026

masc71 commented on Feb 18, 2026

ludoch commented on Feb 19, 2026

masc71 commented on Feb 19, 2026

ludoch commented on Feb 19, 2026

masc71 commented on Feb 23, 2026

masc71 commented on Feb 23, 2026

ludoch commented on Feb 23, 2026

masc71 commented on Feb 25, 2026

masc71 commented on Mar 12, 2026

ludoch commented on Mar 12, 2026

masc71 commented on Mar 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions