This series of codelabs (self-paced, hands-on tutorials) aims to help Google App Engine (Standard) developers modernize their apps by guiding them through a series of migrations. The most significant step is to move away from original runtime bundled services because the next generation runtimes are more flexible, giving users a greater variety of service options. Moving to the newer generation runtime enables you to integrate with Google Cloud products more easily, use a wider range of supported services, and support current language releases.
This tutorial teaches you how to migrate from App Engine's built-in ndb (Next Database) client library to the Cloud NDB client library.
NOTE: If your app does not use ndb or Datastore, you can work through this codelab as an exercise to familiarize yourself with this migration.
ndb library (if you're unfamiliar with it)ndb to Cloud NDBIn Module 1, we migrated web frameworks from App Engine's built-in webapp2 to Flask. In this codelab, we continue to move away from App Engine's built-in services by switching from App Engine's ndb library to Google Cloud NDB.
Through completing this migration, you can then:
But, we aren't there yet. Finish this codelab before considering those next steps. This tutorial's migration features these primary steps:
Before we get going with the main part of the tutorial, let's setup our project, get the code, then deploy the baseline app so we know we started with working code.
1. Setup projectIf you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, you can create a brand new project or reuse another existing project. Ensure the project has an active billing account and that App Engine is enabled.
2. Get baseline sample appOne of the prerequisites is to have a working Module 1 sample app. Use your solution if you completed that tutorial. You can complete it now (link above), or if you wish to skip it, then copy the Module 1 repo (link below).
Whether you use yours or ours, the Module 1 code is where we'll START. This Module 2 codelab walks you through each step, and when complete, it should resemble code at the FINISH point (including an optional "bonus" port from Python 2 to 3):
Your STARTing Module 1 code folder should have the following contents:
$ ls
README.md appengine_config.py requirements.txt
app.yaml main.py templates
If you completed the Module 1 tutorial, you'll also have a lib folder with Flask and its dependencies. If you don't have a lib folder, create it with the pip install -t lib -r requirements.txt command so that we can deploy this baseline app in the next step. If you have both Python 2 and 3 installed, we recommend using pip2 instead of pip to avoid confusion with Python 3.
Your remaining prework steps to execute now:
gcloud command-line tool (if nec.)Once you've successfully executed those steps and confirm it's operational, we'll move ahead in this tutorial, starting with the configuration files.
4. Update configuration files (add Cloud NDB library)Many original App Engine built-in services have blossomed into their own products, and Datastore is one of them. Today non-App Engine apps can use Cloud Datastore. For long-time ndb users, the Google Cloud team has created the Cloud NDB client library to talk to Cloud Datastore. It is available for both Python 2 and 3.
Let's update the confirmation files to replace App Engine ndb with Cloud NDB then modify our application.
requirements.txt
In Module 1, the only external dependency for our app was Flask. Now we'll add Cloud NDB. Here is what your requirements.txt file looked like at the end of Module 1:
Flask==1.1.2
Migrating away from App Engine ndb requires the Cloud NDB library (google-cloud-ndb), so add its package to requirements.txt.
Flask==1.1.2
google-cloud-ndb==1.7.1
When this codelab was written, the latest recommended version is 1.7.1, but requirements.txt in the repo may have a newer version. We recommend the latest versions of each library, but if they don't work, you can roll back to an older release.
Delete your lib folder if you have one and didn't just create it above. Now (re)install the updated libraries with the pip install -t lib -r requirements.txt command, using pip2 instead of pip as necessary.
TROUBLESHOOTING: If your app experiences this error: ImportError: cannot import name collections_abc, ensure the Cloud NDB dependency, google-cloud-core is at least on version 1.4.4 in lib. The core issue causing that error exists only in the 1.4.2 and 1.4.3 releases.
app.yaml
Adding Google Cloud client libraries like google-cloud-ndb has a few requirements, all revolving around the inclusion of "built-in" libraries, 3rd-party packages already available on Google servers. You don't list them in requirements.txt nor do you copy them with pip install. The only requirements:
app.yamllib)Here is the STARTing app.yaml from Module 1:
runtime: python27
threadsafe: yes
api_version: 1
handlers:
- url: /.*
script: main.app
Now add the following lines to app.yaml to reference a pair of 3rd-party bundled packages: grpcio and setuptools in a new libraries section:
libraries:
- name: grpcio
version: 1.0.0
- name: setuptools
version: 36.6.0
Why use these built-in libraries? gRPC is an open RPC framework used by all Google Cloud client libraries, including google-cloud-ndb. The grpcio library is the Python gRPC adapter and thus required. The reasoning for including setuptools is coming up.
With the changes above, your updated app.yaml should now look like this:
runtime: python27
threadsafe: yes
api_version: 1
handlers:
- url: /.*
script: main.app
libraries:
- name: grpcio
version: 1.0.0
- name: setuptools
version: 36.6.0
appengine_config.py
The pkg_resources tool, part of the setuptools library, is used to let built-in 3rd-party libraries access the bundled ones. Update appengine_config.py to use pkg_resources to point them to the bundled libraries in lib. When you've completed this change, the entire file should look like this:
import pkg_resources
from google.appengine.ext import vendor
# Set PATH to your libraries folder.
PATH = 'lib'
# Add libraries installed in the PATH folder.
vendor.add(PATH)
# Add libraries to pkg_resources working set to find the distribution.
pkg_resources.working_set.add_entry(PATH)
With the configuration file formalities out of the way, you can now migrate from ndb to Cloud NDB. To complete the migration, update imported libraries and add use of context management in main.py.
Make the following import swap in main.py:
from google.appengine.ext import ndb
from google.cloud import ndb
The change from an App Engine library to a Google Cloud library is sometimes as subtle as this instance. For built-in services that have become full Google Cloud products, you'll be importing attributes from google.cloud instead of google.appengine.
To be able to use the Cloud NDB library, your app must use Python context managers. Their purpose is to "gate" access to resources such that they must be acquired before they can be used. Context managers are based on the computer science control technique known as Resource Allocation Is Initialization (or RAII). Context managers are used with Python files (which must be opened before they can be accessed) and concurrency, "spin locks" must be acquired before code in a "critical section" can be executed.
Similarly, Cloud NDB requires you acquire the context of a client to communicate with Datastore before any Datastore commands can execute. First, create a client (ndb.Client()) by adding ds_client = ndb.Client() in main.py right after Flask initialization:
app = Flask(__name__)
ds_client = ndb.Client()
The Pythonwith command is used solely to obtain an object's context. Wrap any code blocks accessing Datastore with with statements.
Below are the same functions from Module 1 for writing a new Entity to Datastore, and reading to display the most recently added Entities:
Here's the original code without context management:
def store_visit(remote_addr, user_agent):
'create new Visit entity in Datastore'
Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put()
def fetch_visits(limit):
'get most recent visits'
return (v.to_dict() for v in Visit.query().order(
-Visit.timestamp).fetch(limit))
Now add with ds_client.context(): and move your Datastore access code into the with block:
def store_visit(remote_addr, user_agent):
'create new Visit entity in Datastore'
with ds_client.context():
Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put()
def fetch_visits(limit):
'get most recent visits'
with ds_client.context():
return (v.to_dict() for v in Visit.query().order(
-Visit.timestamp).fetch(limit))
WARNING: Unlike ndb, when working with transactions and Cloud NDB, task queues work independently. For example, in your transactional code, task queue calls don't get rolled back if the transaction fails, only Datastore operations. While we don't have documentation about this limitation, it's listed on the "not available" with Cloud Tasks page.
The main driver application stays identical to what we had from Module 1 as there's no ndb (nor Cloud NDB) code here:
@app.route('/')
def root():
'main application (GET) handler'
store_visit(request.remote_addr, request.user_agent)
visits = fetch_visits(10)
return render_template('index.html', visits=visits)
A best practice is to ensure a clear distinction between application code and data access. This way, your main application code doesn't change when the underlying data storage mechanism is altered as we did with this migration.
6. Summary/Cleanup Deploy applicationRe-deploy your app with gcloud app deploy, and confirm the app works. Your code should now match what's in the Module 2 repo.
If you jumped into this series without doing any of the preceding codelabs, the app itself doesn't change; it registers all visits to the main web page (/) and looks like this once you've visited the site enough times:
Congrats for completing this Module 2 codelab. You've just crossed the finish line, since this is the last of the strongly recommended migrations in this series as far as Datastore goes.
Optional: Clean upWhat about cleaning up to avoid being billed until you're ready to move onto the next migration codelab? As existing developers, you're likely already up-to-speed on App Engine's pricing information.
IMPORTANT: The original App Engine runtimes did not require an active billing account, but that's no longer the case (beginning Nov 2019, enforced starting Apr 2020). However, rest assured that App Engine still has a free tier of access where you won't incur any billing unless those limits are exceeded.
Optional: Disable appIf you're not ready to go to the next tutorial yet, disable your app to avoid incurring charges. When you're ready to move onto the next codelab, you can re-enable it. While your app is disabled, it won't get any traffic to incur charges, however another thing you can get billed for is your Datastore usage if it exceeds the free quota, so delete enough to fall under that limit.
On the other hand, if you're not going to continue with migrations and want to delete everything completely, you can shutdown your project.
Next stepsFrom here, there's flexibility as to your next move. Choose any of these options:
taskqueue push tasks to Module 1 appDockerfilesTo access the latest App Engine runtime and features, we recommend that you migrate to Python 3. In our sample app, Datastore was the only built-in service we used, and since we've migrated from ndb to Cloud NDB, we can now port to App Engine's Python 3 runtime.
While porting to Python 3 is not within the scope of a Google Cloud tutorial, this part of the codelab gives developers an idea of how the Python 3 App Engine runtime differs. One outstanding feature of the next-gen runtime is simplified access to third-party packages; there's no need to specify built-in packages in app.yaml nor is it a requirement to copy or upload non-built-in libraries; they are implicitly installed from being listed in requirements.txt.
Because our sample is so basic and Cloud NDB is Python 2-3 compatible, no application code needs to be explicitly ported to 3.x; the app runs on 2.x and 3.x unmodified, meaning the only required changes are in configuration in this case:
app.yaml to reference Python 3 and remove 3rd-party libraries.appengine_config.py and the lib folder as they're no longer necessary.In addition to main.py, the requirements.txt and templates/index.html files remain unchanged.
app.yaml
BEFORE:
The only real change for this sample app is to significantly shorten app.yaml. As a reminder, here's what we had in app.yaml at the conclusion of Module 2:
runtime: python27
threadsafe: yes
api_version: 1
handlers:
- url: /.*
script: main.app
libraries:
- name: grpcio
version: 1.0.0
- name: setuptools
version: 36.6.0
AFTER:
In Python 3, the threadsafe, api_version, and libraries directives are all deprecated; all apps are presumed threadsafe and api_version isn't used in Python 3. There are no longer built-in third-party packages preinstalled on App Engine services, so libraries is also deprecated. Check the documentation on changes to app.yaml for more information on these changes. As a result, you should delete all three from app.yaml and update to a supported Python 3 version (see below).
handlers directive
In addition, the handlers directive, which directs traffic at App Engine applications has also been deprecated. Since the next-gen runtime expects web frameworks to manage app routing, all "handler scripts" must be changed to "auto". Combining the changes from above, you arrive at this app.yaml:
runtime: python38
handlers:
- url: /.*
script: auto
Learn more about script: auto from its documentation page.
handlers directive
Since handlers is deprecated, you can remove the entire section too, leaving a single-line app.yaml:
runtime: python38
By default, this will launch the Gunicorn WSGI web server which is available for all applications. If you're familiar with gunicorn, this is the command executed when it's started by default with the barebones app.yaml:
gunicorn main:app --workers 2 -c /config/gunicorn.py
entrypoint directive
If, however, your application requires a specific start-up command, that can be specified with an entrypoint directive where your app.yaml would look like this:
runtime: python38
entrypoint: python main.py
This example specifically requests the Flask development server be used instead of gunicorn. Code that starts the development server must also be added to your app to launch on the 0.0.0.0 interface on port 8080 by adding this small section to the bottom of main.py:
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8080, debug=True)
Learn more about entrypoint from its documentation page. More examples and best practices can be found in the App Engine Standard startup docs as well as the App Engine Flexible startup docs.
appengine_config.py and lib
Delete the appengine_config.py file and the lib folder. In migrating to Python 3, App Engine acquires and installs packages listed in requirements.txt.
The appengine_config.py config file is used to recognize third-party libraries/packages, whether you've copied them yourself or use ones already available on App Engine servers (built-in). When moving to Python 3, a summary of the big changes are:
requirements.txt)pip install into a lib folder, meaning no lib folder periodapp.yamlappengine_config.py fileListing all required third-party libraries in requirements.txt is all that's needed.
Re-deploy your app to ensure that it works. You can also confirm how close your solution is to the Module 2 sample Python 3 code. To visualize the differences with Python 2, compare the code with its Python 2 version.
Congrats on finishing the bonus step in Module 2! Visit the documentation on preparing configuration files for the Python 3 runtime. Finally, review the (earlier) Summary/Cleanup page for next steps and cleanup.
Preparing your applicationWhen it is time to migrate your application, you will have to port your main.py and other application files to 3.x, so a best practice is to try your best to make your 2.x application as "forward-compatible" as possible.
There are plenty of online resources to help you accomplish that, but some of the key tips:
six, Future, and/or ModernizeThe sample app was designed with all this in mind, hence why the app runs on 2.x and 3.x right out of the box so we can focus on showing you what needs to be changed in order to use the next-gen platform.
8. Additional resources App Engine migration module codelabs issues/feedbackIf you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:
Migration resourcesLinks to the repo folders for Module 1 (START) and Module 2 (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations which you can clone or download a ZIP file.
App Engine resourcesBelow are additional resources regarding this specific migration:
webapp to 2.7 and webapp2
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[],[]]
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4