RetroSearch Browse

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/googleapis/nodejs-datastore/issues/95 below:

Errors importing lots of entities to DataStore using the emulator · Issue #95 · googleapis/nodejs-datastore · GitHub

From @glenpike on May 23, 2018 10:26

[x] - Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
[x] - Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
[404] - Check our Troubleshooting guide: https://googlecloudplatform.github.io/google-cloud-node/#/docs/guides/troubleshooting
[404] - Check our FAQ: https://googlecloudplatform.github.io/google-cloud-node/#/docs/guides/faq

If you are still having issues, please be sure to include as much information as possible:

Environment details

gcloud SDK: 202.0.0
OS: OSX El Capitan (10.11.6) Using about 12GB / 16GB memory
Node.js version: v8.11.2
npm version: v5.6.0
google-cloud-node version:

├─┬ @google-cloud/datastore@1.4.0
│ ├─┬ @google-cloud/common@0.16.2
├─┬ @google-cloud/logging-bunyan@0.5.0
│ └─┬ @google-cloud/logging@1.1.1
│   ├─┬ @google-cloud/common@0.13.6
│   ├─┬ @google-cloud/common-grpc@0.4.3
├─┬ @google-cloud/storage@1.7.0
│ ├─┬ @google-cloud/common@0.17.0

Using DataStore via: gstore-node@4.2.1

Steps to reproduce

Looping through a list of data and creating a model for each one, then calling a function which
uses save

    const { body } = ctx;
    let promises = [];

    // Save new Object
    body.new.forEach((model) => {
        const createdEntity = FromModel.create(model, model.id);
        promises.push(createdEntity.upsert());
    });

    const response = await Promise.all(promises);

Trying to import about 2.5k models, we are getting lots of errors that look like they're coming from grpc maybe? Workaround is to split data into chunks, e.g. 1/4 works.

The log of errors looks like this ('...' is replacing several repeated events):

10:02:46.826Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.826Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.827Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
...
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.841Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.841Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
...
10:02:46.858Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.858Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.858Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.858Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.858Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.858Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.858Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.859Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.873Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.874Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
...
10:02:46.898Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.898Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.898Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.898Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.899Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.899Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:03:43.145Z ERROR import: 4 DEADLINE_EXCEEDED: Deadline Exceeded
10:03:43.145Z ERROR import: 4 DEADLINE_EXCEEDED: Deadline Exceeded

The DEADLINE_EXCEEDED error seems to correspond with this in the emulator:

datastore] May 23, 2018 11:03:16 AM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub$7 run
[datastore] INFO: Time to persist datastore: 198 ms
[datastore] Exception in thread "LocalDatastoreService-1" java.lang.OutOfMemoryError: unable to create new native thread
[datastore] 	at java.lang.Thread.start0(Native Method)
[datastore] 	at java.lang.Thread.start(Thread.java:714)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[datastore] 	at java.lang.Thread.run(Thread.java:745)
[datastore] Exception in thread "LocalDatastoreService-4" java.lang.OutOfMemoryError: unable to create new native thread
[datastore] 	at java.lang.Thread.start0(Native Method)
[datastore] 	at java.lang.Thread.start(Thread.java:714)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[datastore] 	at java.lang.Thread.run(Thread.java:745)

Copied from original issue: googleapis/google-cloud-node#2822

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4