A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://github.com/dataarts/tailbone below:

dataarts/tailbone: Automatic RESTful backend for AppEngine

Tailbone - Restful AppEngine

Install helper and dependencies

brew install google-app-engine go
go get github.com/doug/tailbone-generator/tailbone

Your new project

mkdir myproject
cd myproject
git init
tailbone init
tailbone serve
open localhost:8080
# <Ctrl-C> to stop local server
tailbone deploy master

A lot of the use cases I originally used tailbone for, a simple javascript based datastore, now have some great alternatives available. Wanted to give people information about https://www.firebase.com/ which is great and takes care of a lot of the things tailbone originally had to do.

Posting an alternative created some confusion, I'm still supporting Tailbone, but honestly at this point I don't really know how and if this library is being used. If you have used it and have stories of it being helpful even for something like prototyping an idea, please send them to tailbone.stories@gmail.com. That would really help me understand how this project is being used in the wild. Thanks!

App Engine is cheap, fast, and awesome. Using it for the first time is sometimes…well…different. There are tons of frameworks like Django or others out there that work with App Engine, but these days we write almost all our applications in JavaScript with AngularJS or Backbone.js, we just need a simple backend to do its part. The App Engine server side APIs are great and for more complex things we recommend you learn them and use them. All this hopes to do is ease that barrier of use and get people writing their apps faster without worrying about their backend code. That said, writing more code on your backend is great if you are up to it, we can’t recommend Go enough for doing that, it's a wonderful language.

Anyway, this was written in spare time to fill a need and hopefully others find it useful too. It provides a simple RESTful backend setup for App Engine so you can write your apps in JavaScript via frameworks like AngularJS, Backbone, etc. and not have to touch any App Engine code. Or just using plain JavaScript and your own xhr calls. All your static resources are automatically served from client/app. App Engine is great at static serving and if you turn on PageSpeed on App Engine you get automatic optimization of your images and scripts, as well as other goodies all for free. It even supports large file uploads and serving via the Google Blobstore.

Style Guide

This is a side project made out of past experiences. That being said there are a few rough edges. Also working on a Go branch with the same api. If you want to contribute please add a test for any fix or feature before you file a pull request.

Manual steps to get started:

N.B: For you javascript development we recommend two things yeoman for bootstrapping and installing js libraries and angularjs for your MVC javascript application framework.

POST /api/{modelname}/
  Creates an object.

PUT or POST /api/{modelname}/{id}
  Updates an object, does a complete overwrite of the properites. This does not do a partial patch.

GET /api/{modelname}/{id}
  Get a specific object.

GET /api/{modelname}/?filter={propertyname==somevalue}&order={propertyname}&projection={propertyname1,propertyname2}
  Query a type.

Any GET request can take an optional list of properties to return, the query will use those to make a projection query which will only return those properties from the model. The format of the projection is a comma seperated list of properties: projection=propertyname1,propertyname2,propertyname3

In appengine_config.py in your root directory copied from tailbone/appengine_config.template.py

from google.appengine.ext import ndb
from tailbone.restful import ScopedModel
class MyModel(ScopedModel):
  stuff = ndb.FloatProperty()

tailbone_restful_DEFINED_MODELS = {"mymodel": MyModel}

This will restrict it so that only /api/mymodel will work.

If you want some user defined models plus everything else to work with whatever you give it you can also specify.

tailbone_restful_RESTRICT_TO_DEFINED_MODELS = False

N.B:

All restful models have three special properties:

Special types include:

To extend the loading of a date strings into native javascript Date format try something like:

var reISO = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2}(?:\.\d*)?)Z$/;

JSON._parse = JSON.parse;
JSON.parse = function(json) {
  return JSON._parse(json, function(key, value) {
    if (typeof value === 'string') {
      if (reISO.exec(value)) {
        return new Date(value);
      }
    }
    return value;
  });
};

Note: By including /tailbone.js this is automatically added.

Public private exposure of properties on a model is controlled by capitalization of the first letter, similar to Go. All models except for users have a private owners list which is just a list of user ids that can access and change the private variables of a model. This is prepopulated with the person who first creates this model. Only the signed in user can edit information on their users model. We thought about owners vs. editors to grant access rights like many other systems, but thought it out of scope for this first pass. This is about rapid prototyping. Also, if you need to make something only modifable by an admin prefix the property with admin or Admin.

$.ajax({
  url: "/api/todos/",
  method: "POST",
  data: {
    Text: "some public text",
    secret: "some secret that only owners can see",
    adminSecret: "only editable by a user who is an admin."
  }
})

While you have to be authenticated, at the time of this writing you can still write anything to the datastore. This is fantastic for rapid development and changing schemas. However, you might want to be more strict once you deploy your application. In order to help, Tailbone does simple regex validation of all properties.

This is a map of a model name to a series of properties. Anything that is an empty string will effectively bypass validation. Everything else will be parsed as a regex and verified against you and your users requests.

validation.json should be created in your root project directory (one level above the tailbone submodule).

N.B: This is still experimental and not full vetted. Don’t hesitate to file any issues when you find bugs. Finally, as in the example below you will need to escape any '’s in your regular expression.

{
  "todos": {
    "skipvalidation": "",
    "anything": ".*",
    "shortstring": "^.{3,30}$",
    "integer": "^[-]?[0-9]+$",
    "float": "^[-]?([0-9]*\\.[0-9]+|[0-9]+)$",
    "timestamp": "^[0-9]+$",
    "object": "^\\{.*\\}$",
    "objectdeep": {
      "anything": ".*",
      "skipvalidation": "",
      "integer": "^[-]?[0-9]+$"
    },
    "list": "^\\[.*\\]$"
  },
  "documents_with_anything": ""
}

This validates a bunch of things on /api/todos/ and lets anything through on /api/documents_with_anything. No other models will be admitted to your database.

/api/users/
  Special restful model that can only be edited by the user, authentication via Google Account.

/api/users/me
  Returns the current users information.

/api/login
  Logs you in.

/api/logout
  Logs you out.

/api/search/?q=myquery
  Full text search of models.
  A special api call used for doing full text search of models.

To enable this experimental feature you need to create a searchable.json which lists which properties on which models are indexed and how they are indexed. Read more about search here. seachable.json should be created in your main project directory.

Example searchable.json

{
  "todos": {
    "_index": "optional_field_for_name_of_index_default_if_not_defined",
    "item": "TextField",
    "snippet": "HtmlField",
    "slug": "AtomField",
    "value": "NumberField",
    "dayof": "DateField",
    "place": "GeoField"
  }
}
GET /api/files/create
  Call prior to uploading files. Returns an object with an "upload_url" property. POST files there.

GET /api/files/
  List all blob info objects. This can only be accessed by administrators.

GET /api/files/{id}
  Returns the actual uploaded file. If file is an image there will be an "image_url" instead of the file.

POST {special url returned from GET /api/files/}
  Uploads form data to Blobstore. All files are public, but obscured. Returns the files names, info, and their ids.

DELETE /api/files/{id}
  Deletes a file from blobstore. This can only be done by administrators.

N.B: There is no PUT to update a file. You must always create a new one and delete the old one yourself.

Image Saving and Serving with /api/files

When the file you POSTed is an image that call will return a special url called image_url. This should be used as the url for any images, it will not only serve faster, but it can take additional parameters to automatically crop and produce thumbnail images. Do so by appending =sXX to the end of the url. E.G: =s200 will return a 200 sized image with the original aspect ratio. =s200-c will return a cropped 200 sized image.

These filetypes are considered images:

Relevant Information Attached to Large Files

When you POST files to the upload_url you’ve requested the response will contain a list of objects. Each object represents the file that you POSTed. Each object will have additional information outlined below:

[
  {
  "Id": file_id,
  "filename": filename,
  "content_type": content_type,
  "creation": creation-date,
  "size": file-size,
  "image_url": optional-image-url-if-content_type-is-image
  },
  ...
]

N.B: The image_url is only present if the file POSTed is an image. In addition the POST is currently the only scenario where you can receive meta-information about the file uploaded. So be sure to keep the file_id for retrieval later!

Upload an image of something drawn with canvas via JavaScript. 
function toBlob(data_url) {
  var d = atob(data_url.split(',')[1]);
  var b = new Uint8Array(d.length);
  for (var i = 0; i < d.length; i++) {
    b[i] = d.charCodeAt(i);
  }
  return new Blob([b], {type: 'image/png'});
}

asyncTest('Upload file', function() {
  var data = new FormData();
  var canvas = document.createElement('canvas');
  document.body.appendChild(canvas);
  var ctx = canvas.getContext('2d');
  ctx.fillRect(0, 0, 100, 100);
  var img = canvas.toDataURL();
  data.append('blob', toBlob(img), 'image_filename');
  document.body.removeChild(canvas);
  $.get('/api/files/create', function(d) {
    $.ajax({
      type: 'POST',
      url: d.upload_url,
      data: data,
      cache: false,
      contentType: false,
      processData: false,
      success: function(items) {
        var d = items[0];
        ok(d.Id != undefined, 'Id is ' + d.Id);
        ok(d.filename == 'image_filename', 'filename is ' + d.filename);
        ok(d.size == 1616, 'size is ' + d.size);
        ok(d.content_type == 'image/png', 'content type is ' + d.content_type);
        start();
      }
    });
  });
});
GET /api/cloudstore/path/to/your/file.webm
  Fetches files from cloudstore

Fetch files from cloudstore. Useful for serving large files off of cloudstorage without making things fully public add your applications service account as a reader of the file you uploaded to cloud storage.

Get the nearest geoip look up to the users ip address as well as return their remote address.

This module makes all paths be returned as app/index.html. Useful when creating an html5 history mode application, that does routing in javascript.

    GET /api/proxy?url=http://www.google.com

This module proxies a given url, useful for issues where CORS restricts access to a resource in javascript. You can restrict which domains are allowed by editing appengine_config.py with

    tailbone_proxy_RESTRICTED_DOMAINS = ["google.com"]

Syncronize clocks across javascript clients. See javascript code

This module helps create a mesh network for which will use websockets and try to upgrade to webrtc where possible. You will need to enable billing and the compute_engine api since this uses compute_engine to start and maintain TURN servers and Websocket servers.

<script src="/tailbone.js"></script>
  var mesh = new tailbone.Mesh();
  mesh.bind('connect', function() {
    console.log('mesh connected');
  });
  mesh.bind('test', function(x) {
    console.log('somone sent a test event');
    console.log(x, 'should be 7');
  });
  mesh.trigger('test', 7)

Compute engine is the lower level library for load balancing compute engine instances, see some of the examples in there for how to extend it.

Static content serving. You can change the authorization mechanism for the site in appengine_config.py. Defaults to public.

def my_auth_function(request): return True

tailbone_static_authorized = my_auth_function

For the testing you need to start the dev server by running dev_appserver.py --clear_datastore=yes . and browsing to http://localhost:8080/test/(testname) for example http://localhost:8080/test/restful. These are QUnit JavaScript tests and should be the same in either go, python or any future language to support consistency of any implementation of the api. Note, these tests modify the db, and will only run locally.

The tests are accessible at /api/test/{module_name} for example /api/test/restful

Tailbone can be extended by creating a appengine_config.py file in your root directory. Copy it from the template inside tailbone/appengine_config.template.py. Use this to extend tailbone with any hooks to modify the behavior of tailbone, such as a different authentication mechanism or various module constants. Examples are all commented out in the appengine_config.template.py. See Python Module Configuration

Additionally, you can turn on and off modules by changing what is included in the app.yaml. You can also add your own modules by adding additional incude.yaml paths the app.yaml file.

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4