== Practical Ubuntu One Files Integration - Instructors: Michael Terry == {{{#!irc === ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: App Developer Week - Current Session: Practical Ubuntu One Files Integration - Instructors: mterry [18:00] Logs for this session will be available at http://irclogs.ubuntu.com/2011/09/09/%23ubuntu-classroom.html following the conclusion of the session. [18:01] talking of which, next up mterry will be talking about connecting your apps to the cloud with ubuntu one :) [18:01] Hello everybody! Thanks njpatel! [18:01] So I'm Michael Terry, an Ubuntu developer as well as the developer of Deja Dup, a backup program [18:02] I recently added support for Ubuntu One to my program, and I thought I'd share how that went, and some simple examples of how to connect to Ubuntu One Files [18:02] I have lots of notes for this session here: https://wiki.ubuntu.com/mterry/UbuntuOneFilesNotes11.10 [18:02] Which you may be interested in going through as I talk [18:02] Please ask questions any old time [18:03] So, for this session (and for my purposes with Deja Dup), I only needed simple file functionality [18:03] get, put, list, delete basically [18:03] So we'll go through each of those basic ideas to help anyone else that's interested in integrating do so easily [18:03] We'll use Python, since that's most convenient [18:04] So let's create together a simple python script that can do basic file operations with U1 [18:04] You'll need an updated ubuntuone-couch package from Ubuntu 11.10 [18:04] I've backported it in my PPA [18:04] For this sessoin [18:04] So if you want to play along at home, but are stuck on Ubuntu 11.04, please do the following: [18:05] sudo add-apt-repository ppa:mterry/ppa2 [18:05] sudo apt-get update [18:05] sudo apt-get upgrade [18:05] That will give you a new ubuntuone-couch [18:05] So to start, let's write a super simple Python script that can just accept an argument [18:05] === [18:05] #!/usr/bin/python [18:05] import sys [18:05] if len(sys.argv) <= 1: [18:05] print "Need more arguments" [18:05] sys.exit(1) [18:05] print sys.argv[1:] [18:05] === [18:05] Very basic. We'll augment this with more in a second [18:06] Save that as u1file.py [18:06] And open a terminal in that same directory [18:06] The first thing you have to do when interacting with U1 is make sure the user is logged in [18:06] There is a helper library for that in ubuntuone.platform.credentials [18:06] It's designed to work with Twisted and be asynchronous [18:07] But we just want simple synchronous behavior for now [18:07] So I'll show you a function that will fake synchronousity by opening an event loop and waiting for login to finish [18:07] === [18:07] #!/usr/bin/python [18:07] import sys [18:07] _login_success = False [18:07] def login(): [18:07] from gobject import MainLoop [18:07] from dbus.mainloop.glib import DBusGMainLoop [18:07] from ubuntuone.platform.credentials import CredentialsManagementTool [18:07] global _login_success [18:07] _login_success = False [18:07] DBusGMainLoop(set_as_default=True) [18:07] loop = MainLoop() [18:07] def quit(result): [18:07] global _login_success [18:08] loop.quit() [18:08] if result: [18:08] _login_success = True [18:08] cd = CredentialsManagementTool() [18:08] d = cd.login() [18:08] d.addCallbacks(quit) [18:08] loop.run() [18:08] if not _login_success: [18:08] sys.exit(1) [18:08] if len(sys.argv) <= 1: [18:08] print "Need more arguments" [18:08] sys.exit(1) [18:08] if sys.argv[1] == "login": [18:08] login() [18:08] === [18:08] This may be easier to see on the wiki page https://wiki.ubuntu.com/mterry/UbuntuOneFilesNotes11.10#Logging_In [18:08] The important bit is from ubuntuone.platform.credentials import CredentialsManagementTool [18:08] followed by [18:08] cd = CredentialsManagementTool() [18:08] d = cd.login() [18:08] The rest is just wrappers to support the event loop [18:08] And to support calling "python u1file.py login" [18:09] So try running that now, and you should see a neat little U1 login screen [18:09] Unless... You've already used U1, in which case, nothing happens (because login() doesn't do anything in that case) [18:09] So let's add a logout function for testing purposes [18:09] === [18:09] def logout(): [18:09] from gobject import MainLoop [18:09] from dbus.mainloop.glib import DBusGMainLoop [18:09] from ubuntuone.platform.credentials import CredentialsManagementTool [18:09] DBusGMainLoop(set_as_default=True) [18:09] loop = MainLoop() [18:09] def quit(result): [18:09] loop.quit() [18:09] cd = CredentialsManagementTool() [18:09] d = cd.clear_credentials() [18:09] d.addCallbacks(quit) [18:09] loop.run() [18:09] if sys.argv[1] == "logout": [18:10] logout() [18:10] === [18:10] Now add that to your script and you can call "python u1file.py logout" to go back to a clean slate [18:10] OK. So we have a skeleton script that can talk to U1, but it doesn't do anything yet! [18:10] Let's upload a file [18:10] Oh, whoops [18:10] First, we have to make sure we create a volume [18:10] In U1-speak, a volume is a folder that can be synchronized between the user's computers [18:11] By default, new volumes are not synchronized anywhere [18:11] But let's create a testing volume so that we can upload files without screwing anything up [18:11] Note that the "Ubuntu One" volume always exists [18:11] Creating a volume is a simple enough call: [18:11] === [18:11] def create_volume(path): [18:11] import ubuntuone.couch.auth as auth [18:11] import urllib [18:11] base = "https://one.ubuntu.com/api/file_storage/v1/volumes/~/" [18:11] auth.request(base + urllib.quote(path), http_method="PUT") [18:11] if sys.argv[1] == "create-volume": [18:11] login() [18:12] create_volume(sys.argv[2]) [18:12] === [18:12] You'll see that we make a single PUT request to a specially crafted URL [18:12] There is no error handling in my snippets of code. I'll get into how to handle errors at the end [18:12] Add that to your u1file.py, and now you can call "python u1file.py create-volume testing" [18:12] If you open http://one.ubuntu.com/files/ you should be able to see the new volume [18:13] Congratulations if so! [18:13] You'll also note that I included a call to login() before creating the volume [18:13] This was to ensure that the user was logged in first [18:13] You'll also note that I made this weird auth.request call [18:14] This is a wrapper function provided by ubuntuone-couch that handles the OAuth signature required by U1 to securely identify the user [18:14] This is why you had to log in first [18:14] And the 11.10 version has some important fixes, which is why I backported it for this session [18:14] OK, *now* let's upload a file [18:14] (any questions?) [18:14] Uploading is a two-step process [18:15] First, we tell the server we want to create a new file [18:15] Then the server tells us a URL path to upload the contents to [18:15] I'll give you the code then we can talk about it [18:15] === [18:15] def put(local, remote): [18:15] import json [18:15] import ubuntuone.couch.auth as auth [18:15] import mimetypes [18:15] import urllib [18:15] # Create remote path (which contains volume path) [18:15] base = "https://one.ubuntu.com/api/file_storage/v1/~/" [18:15] answer = auth.request(base + urllib.quote(remote), [18:15] http_method="PUT", [18:15] request_body='{"kind":"file"}') [18:15] node = json.loads(answer[1]) [18:15] # Read info about local file [18:16] data = bytearray(open(local, 'rb').read()) [18:16] size = len(data) [18:16] content_type = mimetypes.guess_type(local)[0] [18:16] content_type = content_type or 'application/octet-stream' [18:16] headers = {"Content-Length": str(size), [18:16] "Content-Type": content_type} [18:16] # Upload content of local file to content_path from original response [18:16] base = "https://files.one.ubuntu.com" [18:16] url = base + urllib.quote(node.get('content_path'), safe="/~") [18:16] auth.request(url, http_method="PUT", [18:16] headers=headers, request_body=data) [18:16] if sys.argv[1] == "put": [18:16] login() [18:16] put(sys.argv[2], sys.argv[3]) [18:16] === [18:16] There are three parts to this [18:16] First is the request to create a file [18:16] We give a URL path and PUT a specially crafted message "{'kind':'file'}" [18:16] Then, we read the local content [18:16] And push it to where the server told us to [18:16] (this is the "content_path" bit) [18:16] The response from the server (and the specially crafted message we gave it) is called JSON [18:17] It's a special format for encoding data structures as strings [18:17] Looks very Python-y [18:17] The 'json' module has support for reading and writing it [18:17] As you can see [18:17] We also use a different base URL for uploading the content [18:17] We use "files.one.ubuntu.com" [18:18] So now, let's try this new code out: [18:18] "python u1file.py put u1file.py testing/u1file.py" [18:18] This will upload our script to the new testing volume we created [18:18] Again, you can visit the U1 page in your browser and refresh it to see if it was created [18:19] If so, congrats! [18:19] Also note that we had to specify the content length and content type [18:19] These are mandatory [18:19] I calculated both in my example (using the mimetypes module) [18:19] But if you already know the mimetype, you can skip that bit of course [18:20] OK, let's try downloading the script we just uploaded [18:20] This is very similar, but uses GET requests instead of PUT ones [18:20] Again, two step process [18:20] We first get the metadata about the file, which tells us the content_path [18:20] And then we get the content [18:20] === [18:20] def get(remote, local): [18:20] import json [18:20] import ubuntuone.couch.auth as auth [18:20] import urllib [18:20] # Request metadata [18:20] base = "https://one.ubuntu.com/api/file_storage/v1/~/" [18:20] answer = auth.request(base + urllib.quote(remote)) [18:20] node = json.loads(answer[1]) [18:20] # Request content [18:21] base = "https://files.one.ubuntu.com" [18:21] url = base + urllib.quote(node.get('content_path'), safe="/~") [18:21] answer = auth.request(url) [18:21] f = open(local, 'wb') [18:21] f.write(answer[1]) [18:21] if sys.argv[1] == "get": [18:21] login() [18:21] get(sys.argv[2], sys.argv[3]) [18:21] === [18:21] Nothing ground breaking there [18:21] Again, we hit files.one.ubuntu.com for the content [18:21] And again, there is no error checking here [18:21] We'll get to that later [18:21] Let's try to download that script we uploaded [18:21] "python u1file.py get testing/u1file.py /tmp/u1file.py" [18:22] This will put it in /tmp/u1file.py [18:22] Now let's see what we downloaded [18:22] "less /tmp/u1file.py" [18:22] It should look right [18:22] So we can create volumes, upload, and download files [18:23] Big things left to do are list files, query metadata, and delete files [18:23] Let's start with listing [18:23] === [18:23] def get_children(path): [18:23] import json [18:23] import ubuntuone.couch.auth as auth [18:23] import urllib [18:23] # Request children metadata [18:23] base = "https://one.ubuntu.com/api/file_storage/v1/~/" [18:23] url = base + urllib.quote(path) + "?include_children=true" [18:23] answer = auth.request(url) [18:23] # Create file list out of json data [18:23] filelist = [] [18:23] node = json.loads(answer[1]) [18:23] if node.get('has_children') == True: [18:23] for child in node.get('children'): [18:23] child_path = urllib.unquote(child.get('path')).lstrip('/') [18:23] filelist += [child_path] [18:23] print filelist [18:23] if sys.argv[1] == "list": [18:23] login() [18:23] get_children(sys.argv[2]) [18:23] === [18:23] This is very similar to downloading a file [18:23] But we add "?include_children=true" to the end of the request URL [18:24] Then we grab the list of children from the JSON data returned [18:25] black_puppydog has noted that my ubuntuone-couch backport has a bug preventing it from working right [18:25] I will prepare a new package [18:25] But you can fix it by doing the following [18:26] sudo gedit /usr/share/pyshared/ubuntuone-couch/ubuntuone/couch/auth.py [18:26] Search for ", disable_ssl_certificate_validation=True" near the bottom [18:26] And remove it [18:26] Sorry, I really thought I had tested with that [18:27] I've uploaded a fixed package, but it will take a few minutes to build [18:27] So to download the complete file we've got so far... [18:27] grab it here: https://wiki.ubuntu.com/mterry/UbuntuOneFilesNotes11.10?action=AttachFile&do=view&target=6.py [18:28] I'll give everyone a few seconds to catch up [18:28] Save that 6.py file as u1file.py [18:28] And do the following commands to get to the same state: [18:28] python u1file.py login [18:28] python u1file.py create-volume testing [18:29] python u1file.py put u1file.py testing/u1file.py [18:29] python u1file.py get testing/u1file.py /tmp/u1file.py [18:29] python u1file.py list testing [18:29] Really sorry about that [18:30] Note that if you are working on a project that needs to work in 11.04 but you still want this functionality [18:30] You can just locally make a copy of ubuntuone-couch's auth.py file and use it in your project (as long as the license is compatible of course) [18:31] OK, I'm going to wait just a moment longer to let people catch up and re-read the file now that it will actually work when they run it [18:31] So when you run "python u1file.py list testing" you should get a list of all the files you put there [18:32] Which I expect will just be the one u1file.py file [18:32] So now, let's see if we can't get a bit more info about that file [18:32] Sometimes you'll want to query file metadata [18:32] This is very much like downloading [18:32] But without getting the actual contents [18:32] === [18:32] def query(path): [18:32] import json [18:32] import ubuntuone.couch.auth as auth [18:32] import urllib [18:32] # Request metadata [18:32] base = "https://one.ubuntu.com/api/file_storage/v1/~/" [18:32] url = base + urllib.quote(path) [18:32] answer = auth.request(url) [18:33] node = json.loads(answer[1]) [18:33] # Print interesting info [18:33] print 'Size:', node.get('size') [18:33] if sys.argv[1] == "query": [18:33] login() [18:33] query(sys.argv[2]) [18:33] === [18:33] Adding that to your file will let you call "python u1file.py query testing/u1file.py" [18:33] You should see the size in bytes [18:33] There is a bit more metadata available (try inserting a "print node" in there to see it all) [18:33] And the last big file operation we'll cover is the easiest [18:34] Deleting files [18:34] === [18:34] def delete(path): [18:34] import ubuntuone.couch.auth as auth [18:34] import urllib [18:34] base = "https://one.ubuntu.com/api/file_storage/v1/~/" [18:34] auth.request(base + urllib.quote(path), http_method="DELETE") [18:34] if sys.argv[1] == "delete": [18:34] login() [18:34] delete(sys.argv[2]) [18:34] === [18:34] That's simple. Merely an HTTP DELETE request to the metadata URL [18:34] This covers the basic file operations you'd want to do [18:34] I promised I'd talk about error handling [18:34] So behind the scenes, this is all done using HTTP [18:35] And the responses you get back from the server are all in HTTP [18:35] So it makes sense that to check what kind of response you got, you'd use HTTP status codes [18:35] You may be familiar with these [18:35] To look at a status code, with the above examples, you'd do something like: [18:36] answer = auth.request(...) [18:36] status = int(answer[0].get('status')) [18:36] answer is a tuple of 2 [18:36] The first bit is HTTP headers [18:36] The second is the HTTP body [18:36] So we're asking for the 'status' HTTP header here [18:36] Any number in the 200s is an "operation succeeded" message [18:36] There are a few important status codes to be aware of [18:36] 400 is "permission denied" [18:37] 404 is "file not found" [18:37] 503 is "servers busy, please try again in a bit" [18:37] 507 is "out of space" [18:37] You may also just receive a boring old 500 status [18:37] This is like an "internal error" message [18:37] Which isn't very helpful, but usually you are also given an Oops ID to go with it [18:38] oops_id = answer[0].get('x-oops-id') [18:38] If you give this to the U1 server folks, they can tell you what happened and fix the bug [18:38] So if you're going to print a message for the user, include that so that when they report the bug, you'll have the Oops-ID to hand over [18:39] black_puppydog asked: how about checksums? this is needed for example in dejadup, right? [18:40] One piece of metadata is "hash" [18:40] That the server will give you [18:40] I actually have not used that, so I don't know what checksum algorithm it uses [18:41] But you can also just download the file and see (which is what Deja Dup does) [18:41] See https://one.ubuntu.com/developer/files/store_files/cloud/ [18:41] For a list of other metadata pieces you can get from the server [18:41] That also has other useful info. It's the official documentation for this stuff [18:42] If anyone is interested, the Deja Dup code is actually in duplicity, a command line tool that Deja Dup is a wrapper for [18:42] http://bazaar.launchpad.net/~duplicity-team/duplicity/0.6-series/view/head:/duplicity/backends/u1backend.py [18:42] That's real code in use right now [18:43] If you ever have a problem playing with this stuff, the folks in #ubuntuone are very helpful [18:43] With Oops that you run into or whatever [18:44] And that's all I have! I'll hang around for questions if there are any [18:46] black_puppydog asked: this file you used here, shouldn't that be some sort of library? [18:46] black_puppydog, yeah, it very well could be [18:46] You mean, some sort of library supported by the U1 folks to make this all easier? [18:46] Well... They've already provided a lot of the code around it. I think their intention is to focus on providing the best generic API (the web HTTP one) that all sorts of devices and languages can use. [18:47] I think they'd be happy to see an awesome Python wrapper library, but I don't think they want to maintain and promote one such library at the expense of others [18:47] This is close, it would just need much better error handling and such [18:47] But I also don't want to maintain it :) [18:48] But really, it's not *that* much code. A bit boiler plate, true [18:48] ubuntuone-couch takes care of most of the icky parts that are hard to do well (OAuth authentication) [18:49] Most languages have REST and OAuth libraries that can be used in conjunction to talk to the servers [18:50] There are 10 minutes remaining in the current session. [18:51] black_puppydog makes a good point in the chat channel. The duplicity code has better error handling than I've presented. So it may be a better jumping-off point to just steal wholesale than the script we've built here [18:52] Note that it is licensed GPL-2+ [18:52] So if that's not appropriate, maybe just whip something similar up yourself}}}