Home Php C# Sql C C++ Javascript Python Java Go Android Git Linux Asp.net Django .net Node.js Ios Xcode Cocoa Iphone Mysql Tomcat Mongodb Bash Objective-c Scala Visual-studio Apache Elasticsearch Jar Eclipse Jquery Ruby-on-rails Ruby Rubygems Android-studio Spring Lua Sqlite Emacs Ubuntu Perl Docker Swift Amazon-web-services Svn Html Ajax Xml Java-ee Maven Intellij-idea Rvm Macos Unix Css Ipad Postgresql Css3 Json Windows-server Vue.js Typescript Oracle Hibernate Internet-explorer Github Tensorflow Laravel Symfony Redis Html5 Google-app-engine Nginx Firefox Sqlalchemy Lucene Erlang Flask Vim Solr Webview Facebook Zend-framework Virtualenv Nosql Ide Twitter Safari Flutter Bundle Phonegap Centos Sphinx Actionscript Tornado Register | Login | Edit Tags | New Questions | 繁体 | 简体


10 questions online user: 31

3
votes
answers
39 views
+10

ZeroMQ usage in web application: how frontend interacts with backend

I heard some facts about ZeroMQ, and I think it's very powerful thing. But now I try to imagine how it can be applied in web application.

Could you make an example of using ZeroMQ in web applications?

So, the first that strikes me - simple chat application. So, we need frontend and backend. I prefer using python+Tornado as backend. There is python lib for using ZeroMQ. It's clear. So, the next thing is frontend. In frontend I will use some javascript to interact with backend.

So, to do this I should use ajax calls, right? Are there some other ways to do it?

TIA!

沙发
+30
+50

The easiest way to do this is to map WebSockets to ZeroMQ sockets, which is quite simple with tornado and PyZMQ's ZMQStream objects. An example of such an app is the IPython Notebook. This approach has the downside of requiring websockets, which puts a limit on what browsers you can support. Of course, you could also map ajax calls with jQuery, etc. and handle the relay with async handlers in tornado.

A more sophisticated web:ZeroMQ app is the mongrel2 webserver.

The right choice for you is just going to depend on your communication patterns.

91
votes
answers
56 views
+10

using Flask and Tornado together?

I am a big fan of Flask - in part because it is simple and in part because has a lot of extensions. However, Flask is meant to be used in a WSGI environment, and WSGI is not a non-blocking, so (I believe) it doesn't scale as well as Tornado for certain kinds of applications.

Since each one has an URL dispatcher which will call a function, and both will use Python files (in Django you dont launch the python file but in flask or tornado you do) do does it make sense to have two seperate parts to your website - one part running the non-blocking jobs with Tornado, and the other part written with Flask?

If this is a good idea, how would you go about sharing cookies / sessions between Flask and Tornado? Will I run into issues, since Flask will use it own system and Tornado will use its own system?

up vote 87 down vote accepted favorite
沙发
+870
+50

i think i got 50% of the solution, the cookies are not tested yet, but now i can load Flask application using Tornado, and mixing Tornado + Flask together :)

first here is flasky.py the file where the flask application is:

from flask import Flask
app = Flask(__name__)

@app.route('/flask')
def hello_world():
  return 'This comes from Flask ^_^'

and then the cyclone.py the file which will load the flask application and the tornado server + a simple tornado application, hope there is no module called "cyclone" ^_^

from tornado.wsgi import WSGIContainer
from tornado.ioloop import IOLoop
from tornado.web import FallbackHandler, RequestHandler, Application
from flasky import app

class MainHandler(RequestHandler):
  def get(self):
    self.write("This message comes from Tornado ^_^")

tr = WSGIContainer(app)

application = Application([
(r"/tornado", MainHandler),
(r".*", FallbackHandler, dict(fallback=tr)),
])

if __name__ == "__main__":
  application.listen(8000)
  IOLoop.instance().start()

hope this will help someone that wants to mix them :)

如何在Tornado中添加異步方法?你能在燒瓶路線上使用async嗎? - Merlin 12年4月4日19:01

我知道Flask是一個微框架。但是Tornado並不是一個微框架,但它有一個強大的http服務器。假設您不需要關注WSGI“容器”的潛在交換,為什麼不開發Tornado提供的產品呢? - 傑森10月27日在18:40

Flask很好,因為它的擴展,所有東西都有Flask的擴展,所以不需要重新發明輪子。@Merlin我認為這不是一個簡單的方法,Flask使用WSGI並且WSGI不是異步的。 - Abdelouahab Pp 12年4月4日14:02

這真的可行嗎?你最終使用這個,還是只使用龍捲風? - Mittenchops 2014年4月7日20:08

@AbdelouahabPp,雖然Flask使用WSGI並且WSGI不是異步的,但是龍捲風可以接受異步連接然後我們每個請求都可以在不同的線程中運行(我不確定),然後我們可以讓異步龍捲風改為使用Gunicorn同步worker類就像Apache pre-fork模塊一樣。 - 安迪2015年10月15日3:48

+40

Based on 1 and 2, the combined and shorter answer is

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":

    from tornado.wsgi import WSGIContainer
    from tornado.httpserver import HTTPServer
    from tornado.ioloop import IOLoop

    http_server = HTTPServer(WSGIContainer(app))
    http_server.listen(8000)
    IOLoop.instance().start()

Please consider the warning about performance that has been mentioned on 2 , 3

最好單獨使用龍捲風而不是結合燒瓶和龍捲風。 - Ahmad Yoosofan 2017年3月25日15:25

22
votes
answers
47 views
+10

Difference between “yield” of Tornado and “yield from” of asyncio in mechanism?

In Tornado, we usually write the following code to call a function asynchronously:

class MainHandler(tornado.web.RequestHandler):

    @tornado.gen.coroutine
    def post(self):
        ...
        yield self.handleRequest(foo)
        ...

    @tornado.gen.coroutine
    def handleRequest(self, foo):
        ...

But in asyncio (will be shipped with Python 3.4, can be installed from pip for Python 3.3), we use yield from to achieve the same thing:

@asyncio.coroutine
def myPostHandler():
    ...
    yield from handleRequest(foo)
    ...


@asyncio.coroutine
def handleRequest(foo)
    ...

Seeing from the code, the difference is yield and yield from. However the former handleRequest(foo) returns a tornado.concurrent.Future object, the latter returns a generator object.

My question is, what is the difference between the two things in mechanism? How is the control flow? And who calls the actual handleRequest and retrieves its returning value?

Append: I have basic knowledge of Python generators and iterators. I wanted to understand what Tornado and asyncio achieved by using these, and what is the difference between those two mechanisms.

up vote 22 down vote accepted favorite
沙发
+220
+50

There is a huge difference between the two. yield from takes another generator and continues yielding from that generator instead (delegating responsibility, as it were). yield just yields one value.

In other words, yield from, in the simplest case, could be replaced by:

for value in self.handleRequest(foo):
    yield value

If you replaced a yield from <expression> line with yield <expression> you'd return the whole generator to the caller, not the values that generator produces.

The yield from syntax was only introduced in Python 3.3, see PEP 380: Syntax for Delegating to a Subgenerator. Tornado supports Python versions 2.6, 2.7 and 3.2 in addition to Python 3.3, so it cannot rely on the yield from syntax being available. asyncio, on the other hand, being a core Python library added in 3.4, can fully rely on the yield from generator delegation syntax being available.

As a result, Tornado will have to post-process values yielded from a @tornado.gen.coroutine generator to detect that a tornado.concurrent.Future object was yielded; the @asyncio.coroutine code handling can be much simpler. And indeed the Tornado Runner.run() method does explicit type checks to handle delegated tasks.

感謝您的快速答复。但我的問題與產量本身無關。我擔心Tornado和asyncio通過使用產量和產量來實現。 - Star Brilliant 2014年1月9日14:04

@StarBrilliant:他們將任務推遲到以後; 可以暫停生成器,以便事件循環可以將控制權傳遞給另一個生成器。 - Martijn Pieters♦2014年1月9日14:40

@StarBrilliant:協程運行器將循環通過活動協程並為每個人提供執行機會。然後,需要等待網絡資源的協程可以立即再次產生,以允許控制傳遞給其他可能不必等待的協同程序。 - Martijn Pieters♦2014年1月9日14:50

@StarBrilliant:但你的問題似乎是關於語法差異; 產量使得轉輪代碼更簡單; tornado必須使用顯式包裝器來處理委託,asyncio可以依賴語法來代替委託。 - Martijn Pieters♦2014年1月9日14:51

除了使轉子更簡單之外,yield from也更快,並且在出現錯誤時往往會產生更好的堆棧跟踪。另一方面,Tornado對yield的使用提供了協同程序和非協同程序之間更好的互操作性(包括具有顯式回調的異步邏輯和線程執行程序)。 - Ben Darnell 2014年1月9日18:02

34
votes
answers
30 views
+10

How to run functions outside websocket loop in python (tornado)

I'm trying to set up a small example of a public Twitter stream over websockets. This is my websocket.py, and it's working.

What I'm wondering is: how can I interact with the websocket from 'outside' the class WSHandler (ie. not only answer when receiving a message from websocket.js)? Say I want to run some other function within this same script that would post "hello!" every five seconds and send that to the websocket (browser) without any interaction from client-side. How could I do that?

So it's kind of a fundamental beginner's question, I suppose, about how to deal with classes as those below. Any pointers in any direction would be greatly appreciated!

import os.path
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web

# websocket
class FaviconHandler(tornado.web.RequestHandler):
    def get(self):
        self.redirect('/static/favicon.ico')

class WebHandler(tornado.web.RequestHandler):
    def get(self):
        self.render("websockets.html")

class WSHandler(tornado.websocket.WebSocketHandler):
    def open(self):
        print 'new connection'
        self.write_message("Hi, client: connection is made ...")

    def on_message(self, message):
        print 'message received: "%s"' % message
        self.write_message("Echo: "" + message + """)
        if (message == "green"):
            self.write_message("green!")

    def on_close(self):
        print 'connection closed'



handlers = [
    (r"/favicon.ico", FaviconHandler),
    (r'/static/(.*)', tornado.web.StaticFileHandler, {'path': 'static'}),
    (r'/', WebHandler),
    (r'/ws', WSHandler),
]

settings = dict(
    template_path=os.path.join(os.path.dirname(__file__), "static"),
)

application = tornado.web.Application(handlers, **settings)

if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()
up vote 19 down vote accepted favorite
沙发
+190
+50

You could call a

IOLoop.add_timeout(deadline, callback)

that calls the callback at specified deadline timeout (one shot, but you can reschedule), or use the

tornado.ioloop.PeriodicCallback if you have a more periodic task.

See: http://www.tornadoweb.org/en/stable/ioloop.html#tornado.ioloop.IOLoop.add_timeout

Update: some example

import datetime

def test():
    print "scheduled event fired"
...

if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    main_loop = tornado.ioloop.IOLoop.instance()
    # Schedule event (5 seconds from now)
    main_loop.add_timeout(datetime.timedelta(seconds=5), test)
    # Start main loop
    main_loop.start()

it calls test() after 5 seconds.

Update 2:

import os.path
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web

# websocket
class FaviconHandler(tornado.web.RequestHandler):
    def get(self):
        self.redirect('/static/favicon.ico')

class WebHandler(tornado.web.RequestHandler):
    def get(self):
        self.render("websockets.html")

class WSHandler(tornado.websocket.WebSocketHandler):
    def open(self):
        print 'new connection'
        self.write_message("Hi, client: connection is made ...")
        tornado.ioloop.IOLoop.instance().add_timeout(datetime.timedelta(seconds=5), self.test)

    def on_message(self, message):
        print 'message received: "%s"' % message
        self.write_message("Echo: "" + message + """)
        if (message == "green"):
            self.write_message("green!")

    def on_close(self):
        print 'connection closed'

    def test(self):
        self.write_message("scheduled!")

handlers = [
    (r"/favicon.ico", FaviconHandler),
    (r'/static/(.*)', tornado.web.StaticFileHandler, {'path': 'static'}),
    (r'/', WebHandler),
    (r'/ws', WSHandler),
]

settings = dict(
    template_path=os.path.join(os.path.dirname(__file__), "static"),
)

application = tornado.web.Application(handlers, **settings)

import datetime

if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

我使用add_timeout()添加了一些示例。 - pr0gg3d 12月18日在15:15

非常感謝!看起來不錯! - knutole 12年8月18日15:25

IOLoop.instance()是一個單例(在典型用法中)。你可以多次調用instance()並返回相同的單例。 - pr0gg3d 12月18日在15:33

請參閱第二次更新。它在連接5秒後發送預定的消息。 - pr0gg3d 12月18日在15:42

+140

I stumbled upon similar problem. Here is my solution. Hope this will be helpful to someone out there

wss = []
class wsHandler(tornado.websocket.WebSocketHandler):
    def open(self):
        print 'Online'
        if self not in wss:
            wss.append(self)

    def on_close(self):
        print 'Offline'
        if self in wss:
            wss.remove(self)

def wsSend(message):
    for ws in wss:
        ws.write_message(message)

To send message to your websockets simply use this:

wsSend(message)

wsSend update

I've been getting exceptions with wsSend once in a while. In order to fix it I've modified code a bit to following:

def wsSend(message):
    for ws in wss:
        if not ws.ws_connection.stream.socket:
            print "Web socket does not exist anymore!!!"
            wss.remove(ws)
        else:
            ws.write_message(message)

那太好了。 - 邁克2013年4月11日12:43

對Tornado來說是全新的.....所以你會想到為這個非常有前途的答案添加必要的樣板,因為我想看看它是如何工作的(如果我理解正確的話)不止一個websocket處理程序。那你可以給出一個if __name__ ==“__ main__”塊的例子來初始化IOLoop(s?)和處理程序(s?)。乍一看似乎非常優雅。 - Thomas Browne 2014年10月1日21:02

托馬斯,我很樂意幫助你,但它已經有一段時間了,我也不是龍捲風的專家。但據我記得,Tornado擁有出色的文檔,您可以隨時瀏覽源代碼。祝好運! - Barmaley 2014年10月1日22:26

+10

One way to also do this is to use a pub-sub module.

Meaning you have your connections subscribe to it, and rather than setting timeouts for every single connection, you just set one timeout to publish after said period of time.

Probably one of the most implemented is redis. There are also some modules specifically for tornado: toredis or brükva for example.

Of course this might not be necessary for just a simple page, but scales really well and is also very nice to maintain/extend once you've set it up.

49
votes
answers
39 views
+10

How do you *properly* query Redis from Tornado?

I'm curious what the recommended method of querying Redis (or any DB for that matter) is from Tornado.

I've seen some examples like https://gist.github.com/357306 but they all appear to be using blocking calls to redis.

My understanding is that to avoid grinding Tornado to a halt, I need to be using non-blocking DB libraries like the ones developed for Twisted.

Am I wrong? How is this supposed to be done?

up vote 40 down vote accepted favorite
沙发
+400
+50

When it comes to blocking commands like BLPOP or listening to a Pub/Sub channel you'll need an asynchronous client like tornado-redis. You may start with this demo to see how the tornado-redis client may be used to develope a simple public chat application.

But I would recommend using the synchronous redis-py client in conjunction with hiredis for most other cases.

The main advantage of asynchronous client is that your server can handle incoming requests while waiting for Redis server response. However, the Redis server is so fast that in most cases an overhead of setting up asynchronous callbacks in your Tornado application adds more to the total time of request processing then the time spent on waiting for Redis server response.

Using an asynchronous client you may try to send multiple requests to the Redis server at the same time, but the Redis server is a single-threaded one (just like Tornado server), so it will answer to these requests one-by-one and you'll gain almost nothing. And, in fact, you don't have to send multiple Redis commands at the same time to the same Redis server as long as there are pipelines and commands like MGET/MSET.

An asynchronous client has some advantages when you use several Redis server instances, but I suggest using a synchronous (redis-py) client and a proxy like twemproxy or this one (the latter supports pipelining and MGET/MSET commands).

Also I suggest not to use the connection pooling when using the redis-py client in Tornado applications. Just create a single Redis object instance for each Redis database your application connects to.

實際上,對於普通情況,使用同步客戶端聽起來不錯,但最糟糕的情況可能是不可接受的!示例1:如果Redis掛起或網絡到Redis很慢,您將掛起您的Tornado應用程序。示例2:在客戶端上序列化請求並在服務器上進行序列化僅等效於網絡延遲為0,這幾乎不是這種情況。 - AndréCaron2014年5月26日17:54

龍捲風 - 雷迪斯不再維護。但是存儲庫提出了一些替代方案:github.com/aio-libs/aioredis和github.com/mrjoes/toredis - matyas 3月18日19:51

+80

I would recommend to use brukva which is an "Asynchronous Redis client that works within Tornado IO loop".

為了防止任何人偶然發現這篇文章,我認為最新的異步redis庫現在已經被撕裂了.github.com/leporo/tornado-redis - Nick Jennings 2012年6月27日17:27

+10

One option is to use the port of Tornado to Twisted and then use the Twisted Redis API with that. Tornado itself doesn't seem to have arbitrary asynchronous operations as an objective (though if you wanted to rebuild all of the sorts of things that have been built for Twisted, you probably could build them from the low-level iostream APIs in Tornado, but I wouldn't recommend it).

或者在Tornado之上的扭曲反應堆實施:tornado.readthedocs.org/en/latest/twisted.html - Artur Gaspar 2015年7月4日9:36

48
votes
answers
38 views
+10

Python JSON encoder to support datetime?

is there any elegant way to make Python JSON encoder support datetime? some 3rd party module or easy hack?

I am using tornado's database wrapper to fetch some rows from db to generate a json. The query result includes a regular MySQL timestamp column.

It's quite annoying that Python's default json encoder doesn't support its own datetime type, which is so common in all kinds of database queries.

I don't want to modify Python's own json encoder. any good practice? Thanks a lot!

ps: I found a dirty hack by modifying the Python JSON encoder default method:

Change:

def default(self, o):
    raise TypeError(repr(o) + " is not JSON serializable")

To:

def default(self, o):
    from datetime import date
    from datetime import datetime
    if isinstance(o, datetime):
        return o.isoformat()
    elif isinstance(o, date):
        return o.isoformat()
    else:
        raise TypeError(repr(o) + " is not JSON serializable")

well, it will be a temporary solution just for dev environment.

But for long term solution or production environment, this is quite ugly, and I have to do the modification every time I deploy to a new server.

Is there a better way? I do not want to modify Python code itself, neither Tornado source code. Is there something I can do with my own project code to make this happen? preferably in one pace.

Thanks a lot!

up vote 19 down vote accepted favorite
沙发
+190
+50

The docs suggest subclassing JSONEncoder and implementing your own default method. Seems like you're basically there, and it's not a "dirty hack".

The reason dates aren't handled by the default encoder is there is no standard representation of a date in JSON. Some people are using the format /Date(1198908717056)/, but I prefer ISO format personally.

import datetime

class DateTimeEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime.datetime):
            return obj.isoformat()
        elif isinstance(obj, datetime.date):
            return obj.isoformat()
        elif isinstance(obj, datetime.timedelta):
            return (datetime.datetime.min + obj).time().isoformat()
        else:
            return super(DateTimeEncoder, self).default(obj)

DateTimeEncoder().encode(object)

非常感謝。這個isoformat()讓它看起來更好。:) - horacex 12年6月26日在9:33

你可以利用你為return obj.isoformat()和isinstance支持元組執行相同操作的事實來組合if和first elif:isinstance(obj,(datetime.datetime,datetime.date)) - Bailey帕克4月12日5:21

+170

json.dumps(thing, default=str)

如果您認為這符合問題中提到的“輕鬆黑客”的定義,請立即投票。 - 大衛瓊斯14年11月4日14:32

我想這應該是正確的答案,不知道這個答案有什麼問題? - titogeo 18年10月9日11:04

整潔的回答!不知道為什麼不接受這個! - mccbala 1月25日11:36

我覺得很有趣。5年後,這仍然是克服這個問題的最簡單和Pythonic方式,這個答案仍然被低估了。 - Fran Roura 5月3日3:37

+80

I made my own classes for my project:

import datetime
import decimal
import json
import sys

class EnhancedJSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime.datetime):
            ARGS = ('year', 'month', 'day', 'hour', 'minute',
                     'second', 'microsecond')
            return {'__type__': 'datetime.datetime',
                    'args': [getattr(obj, a) for a in ARGS]}
        elif isinstance(obj, datetime.date):
            ARGS = ('year', 'month', 'day')
            return {'__type__': 'datetime.date',
                    'args': [getattr(obj, a) for a in ARGS]}
        elif isinstance(obj, datetime.time):
            ARGS = ('hour', 'minute', 'second', 'microsecond')
            return {'__type__': 'datetime.time',
                    'args': [getattr(obj, a) for a in ARGS]}
        elif isinstance(obj, datetime.timedelta):
            ARGS = ('days', 'seconds', 'microseconds')
            return {'__type__': 'datetime.timedelta',
                    'args': [getattr(obj, a) for a in ARGS]}
        elif isinstance(obj, decimal.Decimal):
            return {'__type__': 'decimal.Decimal',
                    'args': [str(obj),]}
        else:
            return super().default(obj)


class EnhancedJSONDecoder(json.JSONDecoder):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, object_hook=self.object_hook,
                         **kwargs)

    def object_hook(self, d): 
        if '__type__' not in d:
            return d
        o = sys.modules[__name__]
        for e in d['__type__'].split('.'):
            o = getattr(o, e)
        args, kwargs = d.get('args', ()), d.get('kwargs', {})
        return o(*args, **kwargs)

if __name__ == '__main__':
    j1 = json.dumps({'now': datetime.datetime.now(),
        'val': decimal.Decimal('9.3456789098765434987654567')},
        cls=EnhancedJSONEncoder)
    print(j1)
    o1 = json.loads(j1, cls=EnhancedJSONDecoder)
    print(o1)

Result:

{"val": {"args": ["9.3456789098765434987654567"], "__type__": "decimal.Decimal"}, "now": {"args": [2014, 4, 29, 11, 44, 57, 971600], "__type__": "datetime.datetime"}}
{'val': Decimal('9.3456789098765434987654567'), 'now': datetime.datetime(2014, 4, 29, 11, 44, 57, 971600)}

References:

Note: It can be made more flexible by passing a custom dictionary with types as keys and args, kwargs as values to the encoder's __init__() and use that (or a default dictionary) in the default() method.

+30
json.dumps(r, default=lambda o: o.isoformat() if hasattr(o, 'isoformat') else o)

雖然此代碼段可以解決問題,但包括解釋確實有助於提高帖子的質量。請記住,您將來會回答讀者的問題,而這些人可能不知道您的代碼建議的原因。 - andreas 2016年10月14日19:51

+10

The Tryton project has a JSONEncoder implementation for datetime.datetime, datetime.date and datetime.time objects (with others). It is used for JSON RPC communication between the server and client.

See http://hg.tryton.org/2.4/trytond/file/ade5432ac476/trytond/protocols/jsonrpc.py#l53

0

Convert the datetime type into a unix timestamp, then encode the contents into a json.

e.g. : http://codepad.org/k3qF09Kr

你的意思是改變Python的默認日期時間類型?該怎麼辦?風險太大了嗎?它會破壞東西嗎? - horacex 2012年8月25日12:37

@horacex不,只需修改來自tornado數據庫包裝器的結果集。 - DhruvPathak 12年8月25日12:40

0

Create a custom decoder/encoder:

class CustomJSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime.datetime):
            return http_date(obj)
        if isinstance(obj, uuid.UUID):
            return str(obj)
        return json.JSONEncoder.default(self, obj)

class CustomJSONDecoder(json.JSONDecoder):
    def __init__(self, *args, **kwargs):
        json.JSONDecoder.__init__(self, object_hook=self.object_hook, *args, **kwargs)

    def object_hook(self, source):
        for k, v in source.items():
            if isinstance(v, str):
                try:
                    source[k] = datetime.datetime.strptime(str(v), '%a, %d %b %Y %H:%M:%S %Z')
                except:
                    pass
        return source
-10

Just create a custom encoder

(the small but important addition to Cole's answer is the handling of pd.NaT (or null/empty timestamp values), since without the addition you will get very weird timestamp conversions for NaT/missing timestamp data)

class CustomEncoder(json.JSONEncoder):
    def default(self, obj):
        if pd.isnull(obj):
            return None
        elif isinstance(obj, datetime):
            return obj.isoformat()
        elif isinstance(obj, date):
            return obj.isoformat()
        elif isinstance(obj, timedelta):
            return (datetime.min + obj).time().isoformat()
        else:
            return super(CustomEncoder, self).default(obj)

Then use it to encode a dataframe:

df_as_dict = df.to_dict(outtype = 'records')  # transform to dict

df_as_json = CustomEncoder().encode(df_as_dict) #transform to json

Since the encoder standardized the data, the regular decoder will act fine in transforming it back to a dataframe:

result_as_dict = json.JSONDecoder().decode(df_as_json) # decode back to dict

result_df = pd.DataFrame(result)  # transform dict back to dataframe

Of course this will also work if you put the dataframe into a larger dict before encoding, e.g

input_dict = {'key_1':val_1,'key_2':val_2,...,'df_as_dict':df_as_dict}
input_json = CustomEncoder().encode(input_dict)
input_json_back_as_dict = json.JSONDecoder().decode(input_json)
input_df_back_as_dict = input_json_back_as_dict['df_as_dict']
input_df_back_as_df = pd.DataFrame(input_df_back_as_dict)
34
votes
answers
39 views
+10

How do I stop Tornado web server?

I've been playing around a bit with the Tornado web server and have come to a point where I want to stop the web server (for example during unit testing). The following simple example exists on the Tornado web page:

import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world")

application = tornado.web.Application([
    (r"/", MainHandler),
])

if __name__ == "__main__":
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

Once tornado.ioloop.IOLoop.instance().start() is called, it blocks the program (or current thread). Reading the source code for the IOLoop object gives this example in the documentation for the stop function:

To use asynchronous methods from otherwise-synchronous code (such as
unit tests), you can start and stop the event loop like this:
  ioloop = IOLoop()
  async_method(ioloop=ioloop, callback=ioloop.stop)
  ioloop.start()
ioloop.start() will return after async_method has run its callback,
whether that callback was invoked before or after ioloop.start.

However, I have no idea how to integrate this into my program. I actually have a class that encapsulates the web server (having it's own start and stop functions), but as soon as I call start, the program (or tests) will of course block anyway.

I've tried to start the web server in another process (using the multiprocessing package). This is the class that is wrapping the web server:

class Server:
    def __init__(self, port=8888):
        self.application = tornado.web.Application([ (r"/", Handler) ])

        def server_thread(application, port):
            http_server = tornado.httpserver.HTTPServer(application)
            http_server.listen(port)
            tornado.ioloop.IOLoop.instance().start()

        self.process = Process(target=server_thread,
                               args=(self.application, port,))

    def start(self):
        self.process.start()

    def stop(self):
        ioloop = tornado.ioloop.IOLoop.instance()
        ioloop.add_callback(ioloop.stop)

However, stop does not seem to entirely stop the web server since it is still running in the next test, even with this test setup:

def setup_method(self, _function):
    self.server = Server()
    self.server.start()
    time.sleep(0.5)  # Wait for web server to start

def teardown_method(self, _function):
    self.kstore.stop()
    time.sleep(0.5)

How can I start and stop a Tornado web server from within a Python program?

沙发
+180

Here is the solution how to stop Torando from another thread. Schildmeijer provided a good hint, but it took me a while to actually figure the final example that works.

Please see below:

import threading
import tornado.ioloop
import tornado.web
import time


class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world!
")

def start_tornado(*args, **kwargs):
    application = tornado.web.Application([
        (r"/", MainHandler),
    ])
    application.listen(8888)
    print "Starting Torando"
    tornado.ioloop.IOLoop.instance().start()
    print "Tornado finished"

def stop_tornado():
    ioloop = tornado.ioloop.IOLoop.instance()
    ioloop.add_callback(ioloop.stop)
    print "Asked Tornado to exit"

def main():

    t = threading.Thread(target=start_tornado)  
    t.start()

    time.sleep(5)

    stop_tornado()
    t.join()

if __name__ == "__main__":
    main()

ioloop.add_callback(lambda x:x.stop(),ioloop)或更簡單:ioloop.add_callback(ioloop.stop) - ereOn 2015年12月24日15:04

這是否意在實際停止Web服務器?它似乎讓港口開放。 - rakslice 2016年6月14日1:38

不適用於Python 2.7 / Windows 10. - Dan H 18年5月21日10:53

板凳
+80

In case you do no want to bother with threads, you could catch a keyboard interrupt signal :

try:
    tornado.ioloop.IOLoop.instance().start()
# signal : CTRL + BREAK on windows or CTRL + C on linux
except KeyboardInterrupt:
    tornado.ioloop.IOLoop.instance().stop()

在Python 2.7 / Windows 10下:這很有用:CTRL-BREAK確實讓我脫離了tornado.ioloop。但是,它肯定不會引發KeyboardInterrupt。事實上,在CTRL-BREAK之後沒有達成任何聲明。無論它做什麼,它都在努力和快速地殺死這個過程。 - Dan H 18年5月21日10:55

我認為SIGINT(Ctrl + C)和SIGBREAK(Ctrl + Break)的處理方式不同。 - Ninjakannon 1月25日15:50

地板
+30

To stop the entire ioloop you simply invoke the ioloop.stop method when you have finished the unit test. (Remember that the only (documented) thread safe method is ioloop.add_callback, ie. if the unit tests is executed by another thread, you could wrap the stop invocation in a callback)

If its enough to stop the http web server you invoke the httpserver.stop() method

但是調用ioloop.start()會阻塞,所以我怎麼能調用ioloop.stop()?我應該在另一個線程中運行ioloop.start()嗎? - Adam Lindberg 2011年3月21日18:32

這是一個解決方案。(記得將ioloop.stop包裝在ioloop回調中以避免並發修改)。另一個解決方案是從ioloop本身停止ioloop。 - Schildmeijer 2011年3月23日8:16

從shell運行時,這似乎有效。謝謝! - Adam Lindberg 2011年3月28日10:11

我很抱歉打擾你,但我遇到了同樣的問題(我在線程中運行龍捲風,但我無法阻止它)。我讀了你的答案和評論,我真的不明白該怎麼做。您能否發布幾行代碼來說明您的方法?謝謝。 - Alik 2011年7月26日14:47

@Schildmeijer我在模塊“httpserver”中找不到方法“stop”,python報告同樣的事情 - jondinham 3月2日12:07

4楼
+20

If you need this behavior for unit testing, take a look at tornado.testing.AsyncTestCase.

By default, a new IOLoop is constructed for each test and is available as self.io_loop. This IOLoop should be used in the construction of HTTP clients/servers, etc. If the code being tested requires a global IOLoop, subclasses should override get_new_ioloop to return it.

If you need to start and stop an IOLoop for some other purpose and can't call ioloop.stop() from a callback for some reason, a multi-threaded implementation is possible. To avoid race conditions, however, you need to synchronize access to the ioloop, which is actually impossible. Something like the following will result in deadlock:

Thread 1:

with lock:
    ioloop.start()

Thread 2:

with lock:
    ioloop.stop()

because thread 1 will never release the lock (start() is blocking) and thread 2 will wait till the lock is released to stop the ioloop.

The only way to do it is for thread 2 to call ioloop.add_callback(ioloop.stop), which will call stop() on thread 1 in the event loop's next iteration. Rest assured, ioloop.add_callback() is thread-safe.

5楼
+20

There is a problem, with Zaar Hai's solution, namely that it leaves the socket open. The reason I was looking for a solution to stop Tornado is I'm running unit tests against my app server and I needed a way to start/stop the server between tests to have a clear state (empty session, etc.). By leaving the socket open, the second test always ran into an Address already in use error. So I came up with the following:

import logging as log
from time import sleep
from threading import Thread

import tornado
from tornado.httpserver import HTTPServer


server = None
thread = None


def start_app():
    def start():
        global server
        server = HTTPServer(create_app())
        server.listen(TEST_PORT, TEST_HOST)
        tornado.ioloop.IOLoop.instance().start()
    global thread
    thread = Thread(target=start)
    thread.start()
    # wait for the server to fully initialize
    sleep(0.5)


def stop_app():
    server.stop()
    # silence StreamClosedError Tornado is throwing after it is stopped
    log.getLogger().setLevel(log.FATAL)
    ioloop = tornado.ioloop.IOLoop.instance()
    ioloop.add_callback(ioloop.stop)
    thread.join()

So the main idea here is to keep a reference to the HTTPServer instance and call its stop() method. And create_app() just returns an Application instance configured with handlers. Now you can use these methods in your unit tests like this:

class FoobarTest(unittest.TestCase):

    def setUp(self):
        start_app()

    def tearDown(self):
        stop_app()

    def test_foobar(self):
        # here the server is up and running, so you can make requests to it
        pass
6楼
+10

Tornado's IOloop.instance() has trouble stopping from an external signal when run under multiprocessing.Process.

The only solution I came up with that works consistently, is by using Process.terminate():

import tornado.ioloop, tornado.web
import time
import multiprocessing

class Handler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world")

application = tornado.web.Application([ (r"/", Handler) ])

class TornadoStop(Exception):
    pass
def stop():
    raise TornadoStop
class worker(multiprocessing.Process):
    def __init__(self):
        multiprocessing.Process.__init__(self)
        application.listen(8888)
        self.ioloop = tornado.ioloop.IOLoop.instance()

    def run(self):
        self.ioloop.start()

    def stop(self, timeout = 0):
        self.ioloop.stop()
        time.sleep(timeout)
        self.terminate()



if __name__ == "__main__":

    w = worker()
    print 'starting server'
    w.start()
    t = 2
    print 'waiting {} seconds before stopping'.format(t)
    for i in range(t):
        time.sleep(1)
        print i
    print 'stopping'
    w.stop(1)
    print 'stopped'
7楼
0

Just add this before the start():

IOLoop.instance().add_timeout(10,IOLoop.instance().stop)

It will register the stop function as a callback in the loop and lauch it 10 second after the start

8楼
0

We want to use a multiprocessing.Process with a tornado.ioloop.IOLoop to work around the cPython GIL for performance and independency. To get access to the IOLoop we need to use Queue to pass the shutdown signal through.

Here is a minimalistic example:

class Server(BokehServer)

    def start(self, signal=None):
        logger.info('Starting server on http://localhost:%d'
                    % (self.port))

        if signal is not None:
            def shutdown():
                if not signal.empty():
                    self.stop()
            tornado.ioloop.PeriodicCallback(shutdown, 1000).start()

        BokehServer.start(self)
        self.ioloop.start()

    def stop(self, *args, **kwargs):  # args important for signals
        logger.info('Stopping server...')
        BokehServer.stop(self)
        self.ioloop.stop()

The Process

import multiprocessing as mp
import signal

from server import Server  # noqa

class ServerProcess(mp.Process):
    def __init__(self, *args, **kwargs):
        self.server = Server(*args, **kwargs)
        self.shutdown_signal = _mp.Queue(1)
        mp.Process.__init__(self)

        signal.signal(signal.SIGTERM, self.server.stop)
        signal.signal(signal.SIGINT, self.server.stop)

    def run(self):
        self.server.start(signal=self.shutdown_signal)

    def stop(self):
        self.shutdown_signal.put(True)

if __name__ == '__main__':
    p = ServerProcess()
    p.start()

Cheers!

10
votes
answers
36 views
+10

Tornado : support multiple Application on same IOLoop

I'm wondering if it is possible in the Tornado framework to register multiple Application on the same IOLoop ?

Something like

application1 = web.Application([
    (r"/", MainPageHandler),
])
http_server = httpserver.HTTPServer(application1)
http_server.listen(8080)

application2 = web.Application([
    (r"/appli2", MainPageHandler2),
])
http_server2 = httpserver.HTTPServer(application2)
http_server2.listen(8080)

ioloop.IOLoop.instance().start()

Basically I'm trying to structure my webapp so that:

  1. functional applications are separated
  2. multiple handlers with the same purpose (e.g. admin/monitoring/etc) are possible on each webapp
up vote 10 down vote accepted favorite
沙发
+100
+50

The simple thing is if you were to bind your applications to different ports:

...
http_server = httpserver.HTTPServer(application1)
http_server.listen(8080)    # NOTE - port 8080

...
http_server2 = httpserver.HTTPServer(application2)
http_server2.listen(8081)   # NOTE - port 8081

ioloop.IOLoop.instance().start()

This is the base case that Tornado makes easy. The challenge is that by routing to applications at the URI level you're crossing a design boundary which is that each application is responsible for all of the URIs that that are requested by it.

If they all really need to be serviced at the URI level not port, it would probably be best to host different applications on different ports and have Nginx/Apache do the URI routing - anything that involves messing with the Application/Request handling is going to be a world of hurt.

2
votes
answers
35 views
+10

how is cherrypy working? it handls requests well compared with tornado when concurrence is low

I carried out a test on cherrypy (using web.py as a framework) and tornado retrieving webpages from the internet.

I have three test cases using siege to send requests to server (-c means number of users; -t is testing time). Code is below the test results.

1. web.py (cherrpy)

  siege ip -c20 -t100s             server can handle 2747requests  
  siege ip -c200 -t30s             server can handle 1361requests
  siege ip -c500 -t30s             server can handle 170requests

2. tornado synchronous

  siege ip -c20 -t100s             server can handle 600requests  
  siege ip -c200 -t30s             server can handle 200requests
  siege ip -c500 -t30s             server can handle 116requests

3. tornado asynchronous

  siege ip -c20 -t100s             server can handle 3022requests  
  siege ip -c200 -t30s             server can handle 2259requests
  siege ip -c500 -t30s             server can handle 471requests

performance analysis:

tornado synchronous < web.py (cherrypy) < tornado asynchronous

Question 1:

I know, using an asynchronous architecture can improve the performance of a web server dramatically.

I'm curious about the difference between tornado asynchronous architecture and web.py (cherry).

I think tornado synchronous mode handles requests one by one, but how is cherrypy working, using multiple threads? But I didn't see a large increase of memory. Cherrypy might handle multiple requests concurrently. How does it solve the blocking of a program?

Question 2:

Can I improve the performance of tornado synchronous mode without using asynchronous techniques? I think tornado can do better.

Web.py code:

import web
import tornado.httpclient
urls = (
    '/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:
    def GET(self, name):
        client = tornado.httpclient.HTTPClient()
        response=client.fetch("http://www.baidu.com/")
        return response.body

if __name__ == "__main__":
    app.run()

Tornado synchronous:

import tornado.ioloop
import tornado.options
import tornado.web
import tornado.httpclient
from tornado.options import define, options
define("port", default=8000, help="run on the given port", type=int)
class IndexHandler(tornado.web.RequestHandler):
    def get(self):
        client = tornado.httpclient.HTTPClient()
        response = client.fetch("http://www.baidu.com/" )
        self.write(response.body)


if __name__=='__main__':
    tornado.options.parse_command_line()
    app=tornado.web.Application(handlers=[(r'/',IndexHandler)])
    http_server=tornado.httpserver.HTTPServer(app)
    http_server.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

Tornado asynchronous:

import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import tornado.httpclient
from tornado.options import define, options
define("port", default=8001, help="run on the given port", type=int)
class IndexHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        client = tornado.httpclient.AsyncHTTPClient()
        response = client.fetch("http://www.baidu.com/" ,callback=self.on_response)

    def on_response(self,response):
        self.write(response.body)
        self.finish()

if __name__=='__main__':
    tornado.options.parse_command_line()
    app=tornado.web.Application(handlers=[(r'/',IndexHandler)])
    http_server=tornado.httpserver.HTTPServer(app)
    http_server.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()
沙发
+20

To answer question 1...

Tornado is single threaded. If you block the main thread, as you do in your synchronous example, then that single thread cannot do anything until the blocking call returns. This limits the synchronous example to one request at a time.

I am not particularly familiar with web.py, but looking at the source for its HTTP server it appears to be using a threading mixin, which suggests that it is not limited to handling one request at a time. When the first request comes in, it is handled by a single thread. That thread will block until the HTTP client call returns, but other threads are free to handle further incoming requests. This allows for more requests to be processed at once.

I suspect if you emulated this with Tornado, eg, by handing off HTTP client requests to a thread pool, then you'd see similar throughput.

當我測試web.py時,我一直在觀察cherrpy佔用的內存變化。我知道,通常情況下,一個線程在linux中佔用8米。同時,在開始時,cherrypy需要12米才能處理請求,它只花了15米。我不認為它使用多線程。這就是為什麼我好奇櫻桃。 - 用戶1749075 12年12月4日11:42

板凳
0

Most of the handler time in your test code is by far spent in client.fetch(...) - effectively waiting for connection and for incoming data on a socket - while not blocking potential other Python threads.

So your "performance measure" is mostly determined by the max number of effective handler threads of the framework in question and by the max number of parallel connections which the "baidu" server allows from your IP.

wep.py's copy of CherryPyWSGIServer web.wsgiserver.CherryPyWSGIServer which is used by the default web.httpserver.runsimple() indeed uses threads - 10 by default.
Threads do not increase memory usage a lot here. Most memory is consumed by the libraries and Python interpreter itself here. And CherryPyWSGIServer's (10) handling worker threads are all started right at the beginning. The alternative web.httpserver.runbasic() also uses threads - via Python's builtin HTTPServer and SocketServer.ThreadingMixIn. This one starts a new thread for each request. Probably "unlimited" number of threads - but there is overhead for thread startup for each request.

tornado asynchronous mode may also use more/unlimited number of threads (?), which may explain the difference to web.py here.

Your test doesn't say much about the execution speed of the servers & handler frameworks themselves. You may simply increase the max number of threads in web.py's CherryPyWSGIServer. Parallel execution of your client.fetch(...)'s is somehow needed to get more "performance" here.

To test the mere server / framework speed (the overhead cost) simply return a string or a database query or a typical complete web page rendered from local contents.

A multithreaded CPython based web&app server in one process finally cannot use much more than one CPU core (of maybe 8 CPU cores typically today on server hardware) - because of the GIL in CPython, which is only released for some I/O. So if CPU-load becomes a factor (and not the network or database speed), Jython or a multi-processes approach could be considered.

95
votes
answers
32 views
+10

tornado 403 GET warning when opening websocket

I found this python script which should allow me to open a WebSocket. However, I receive the warning [W 1402720 14:44:35 web:1811] 403 GET / (192.168.0.102) 11.02 ms in my Linux terminal when trying to open the actual WebSocket (using Old WebSocket Terminal Chrome plugin). The messages "connection opened", "connection closed" and "message received" are never printed in the terminal window.

import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import tornado.websocket

class MyHandler(tornado.websocket.WebSocketHandler):
        def open(self):
                print "connection opened"
                self.write_message("connection opened")

        def on_close(self):
                print "connection closed"

        def on_message(self,message):
                print "Message received: {}".format(message)
                self.write_message("message received")

if __name__ == "__main__":
        tornado.options.parse_command_line()
        app = tornado.web.Application(handlers=[(r"/",MyHandler)])
        server = tornado.httpserver.HTTPServer(app)
        server.listen(8888)
        tornado.ioloop.IOLoop.instance().start()
up vote 93 down vote accepted favorite
沙发
+930
+50

please add

def check_origin(self, origin):
    return True

in class MyHandler like this

class MyHandler(tornado.websocket.WebSocketHandler):

    def check_origin(self, origin):
        return True

    def open(self):
        print "connection opened"
        self.write_message("connection opened")

    def on_close(self):
        print "connection closed"

    def on_message(self,message):
        print "Message received: {}".format(message)
        self.write_message("message received")

From the DOCs:

By default, [check_origin] rejects all requests with an origin on a host other than this one.

This is a security protection against cross site scripting attacks on browsers, since WebSockets are allowed to bypass the usual same-origin policies and don’t use CORS headers.

And again:

This is an important security measure; don’t disable it without understanding the security implications. In particular, if your authentication is cookie-based, you must either restrict the origins allowed by check_origin() or implement your own XSRF-like protection for websocket connections. See these articles for more.

Link.

文檔幾乎沒有提到它 - 非常感謝你從拉出我的頭髮中拯救出來。 - 匿名於2016年10月25日17:26

+10

Slightly modified @maxhawkdown's solution.

from tornado.util import PY3

if PY3:
    from urllib.parse import urlparse  # py2

    xrange = range
else:
    from urlparse import urlparse  # py3


class ChatHandler(tornado.websocket.WebSocketHandler):
    CORS_ORIGINS = ['localhost']

    def check_origin(self, origin):
        parsed_origin = urlparse(origin)
        # parsed_origin.netloc.lower() gives localhost:3333
        return parsed_origin.hostname in self.CORS_ORIGINS
+10

Don't just set return True on check_origin() because it's a security threat, use a list of allowed domains instead, i.e.:

def check_origin(self, origin):
    allowed = ["https://site1.tld", "https://site2.tld"]
    if origin in allowed:
        print("allowed", origin)
        return 1