Webgateway api2 by will-moore · Pull Request #4708 · ome/openmicroscopy

will-moore · 2016-06-14T12:37:34Z

This is a subset of webgateway api methods explored in #4572.

User guide for the API which covers all urls added in this PR can be found at https://github.com/will-moore/design/blob/webgateway_json_api/webgateway_json_api.md

I have also added an example test script at
https://github.com/will-moore/openmicroscopy/blob/webgateway_api2/examples/Training/python/Json_Api/Login.py
which can be used for testing this PR.

Browse API at http://10.0.51.145/web/api/

This PR uses omero-marshal which is included in the build.
We also add /version/ into the url to allow versioning, E.g.

/api/v1/m/projects/

The supported versions are specified in settings API_VERSIONS = ['0.1']
Details of our versioning strategy are to be decided.
This PR splits the querying and marshalling into api_query.py and api_marshal.py respectively.

New tests are under OmeroWeb/test/integration/test_api_*
I have ported some integration tests from test_tree.py, and turned them into Django session-based tests to test the actual json api, not just the underlying querying and marshalling. These tests also use omero-marshal.

TODO / discuss

OmeroWeb Tests
Filter by owner & group
Single project /webgateway/api/v1/m/projects/:id/
Test page, limit, normalize, childCount & callback query params in test_api_projects.py
Test Project creation, update and delete in test_api_projects.py
Test 404 in test_api_projects.py
Complete test_api_login.py with success and error handling as in gist
IF we're dropping Django 1.6, can use 1.7 Http imports in omeroweb/http See 440ce90
Make sure GET, PUT & POST return correct status codes See https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.5

Future TODOs:

Investigate CORS instead of jsonp().
Register OMERO.webapi user agent with QA
Split 'api' out of webgateway into separate app
Handle 'normalize' by unloading groups & experimenters?
Test cases for _internal and _raw flags in @jsonp decorator
Handle inactive user login. Currently users don't get logged in and can't tell why.
Need omeroweb.webadmin.webadmin_utils.upgradeCheck expanded to allow for configuration of the agent

sbesson · 2016-06-14T12:53:13Z

components/tools/OmeroWeb/omeroweb/webgateway/api_marshal.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2019 University of Dundee & Open Microscopy Environment.


that's a futuristic PR :)

will-moore · 2016-06-14T14:16:27Z

@sbesson @chris-allan Travis is failing with

�[0K$ if [[ $BUILD == 'build-python' ]]; then pip install --user -r https://github.com/openmicroscopy/omero-marshal/tarball/master; fi
�[33mYou are using pip version 6.0.8, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.�[0m
�[31mException:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip-6.0.8-py2.7.egg/pip/basecommand.py", line 232, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip-6.0.8-py2.7.egg/pip/commands/install.py", line 321, in run
    finder=finder, options=options, session=session):
  File "/usr/local/lib/python2.7/dist-packages/pip-6.0.8-py2.7.egg/pip/req/req_file.py", line 135, in parse_requirements
    isolated=options.isolated_mode if options else False,
  File "/usr/local/lib/python2.7/dist-packages/pip-6.0.8-py2.7.egg/pip/req/req_install.py", line 137, in from_line
    elif (os.path.isdir(path)
  File "/usr/lib/python2.7/genericpath.py", line 41, in isdir
    st = os.stat(s)
TypeError: must be encoded string without NULL bytes, not str

Any ideas what I'm doing wrong?

knabar · 2016-06-14T14:24:31Z

@will-moore You don't need the -r since it's not a requirements file

atarkowska · 2016-06-14T14:24:35Z

-r is for requirement file, I think you need to create a file first
or

pip install https://github.com/openmicroscopy/omero-marshal/tarball/master
Collecting https://github.com/openmicroscopy/omero-marshal/tarball/master
  Cache entry deserialization failed, entry ignored
  Downloading https://github.com/openmicroscopy/omero-marshal/tarball/master
  Requirement already satisfied (use --upgrade to upgrade): omero-marshal==0.3.0 from https://github.com/openmicroscopy/omero-marshal/tarball/master in /Users/ola/.local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): importlib>=1.0.1 in /usr/local/lib/python2.7/site-packages (from omero-marshal==0.3.0)

will-moore · 2016-06-14T14:55:08Z

@aleksandra-tarkowska @knabar - thanks. pip install seems to be working (the build is failing elsewhere now)!

atarkowska · 2016-06-14T20:15:18Z

@will-moore could you create requirements file please as this PR broke devspace (because is green). And tomorrow web will fail on CI. Alternatively I would suggest to exclude and tomorrow I will help you with all builds?

jburel · 2016-06-14T20:24:06Z

excluding so it does not break tomorrow's build

jburel · 2016-06-14T20:26:28Z

Adding breaking too so it will be tested via breaking

will-moore · 2016-06-15T08:54:00Z

@aleksandra-tarkowska Where do you want me to create a requirements file?
Is there an existing file I can add omero-marshal to, or will this be a new file with only omero-marshal?

This provides a single point to specifiy additional requirements that is consumed by the other various requirements files.

This is now provided by OmeroWeb/requirements-common.txt

will-moore · 2016-06-15T09:48:02Z

@aleksandra-tarkowska As discussed: added omero-marshal to OmeroWeb requirements files and removed it from .travis.yml

atarkowska · 2016-06-15T09:48:10Z

just chat with @will-moore I propose to create requirements folder OmeroWeb/requiremenst and reorganize these files (rename then to py(26|27)-(apache|nginx).txt). Adjusting ant to match what is currently in the documentation should be easy.

 <target name="copy-web-requirements" depends="init">
        <mkdir dir="${dist.dir}/share/web"/>
        <copy todir="${dist.dir}/share/web">
            <fileset dir="components/tools/OmeroWeb/requirements">
                <include name="requirements*.txt"/>
            </fileset>
            <globmapper from="*" to="requirements-*"/>
        </copy>
    </target>

I will handle that in a separate PR for better review

will-moore · 2016-06-15T09:49:21Z

Working:

$ pip install -r requirements-py27-nginx.txt
Collecting https://github.com/openmicroscopy/omero-marshal/tarball/master (from -r requirements-common.txt (line 4))
  Downloading https://github.com/openmicroscopy/omero-marshal/tarball/master
     | 30kB 346kB/s
Requirement already satisfied (use --upgrade to upgrade): Django<1.9,>=1.8 in /usr/local/lib/python2.7/site-packages (from -r requirements-py27-nginx.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): gunicorn>=19.3 in /usr/local/lib/python2.7/site-packages (from -r requirements-py27-nginx.txt (line 7))
Collecting importlib>=1.0.1 (from omero-marshal==0.3.0->-r requirements-common.txt (line 4))
  Downloading importlib-1.0.3.tar.gz
Building wheels for collected packages: importlib
  Running setup.py bdist_wheel for importlib ... done
  Stored in directory: /Users/wmoore/Library/Caches/pip/wheels/2a/04/fb/873fc6754f3d84f4ad1ee05bd2e63bc52dc883fbe314c5801a
Successfully built importlib
Installing collected packages: importlib, omero-marshal
  Running setup.py install for omero-marshal ... done
Successfully installed importlib-1.0.3 omero-marshal-0.3.0

This allows behaviour to be customised by subclasses.

chris-allan · 2016-10-25T21:59:38Z

components/tools/OmeroWeb/omeroweb/webgateway/decorators.py

+    # To make django's method_decorator work, this is required until
+    # python/django sort out how argumented decorator wrapping should work
+    # https://github.com/openmicroscopy/openmicroscopy/pull/1820
+    def __getattr__(self, name):


Fix from @dpwrussell went into Django upstream in 1.7. Do we still need this?

Might be time, not in this PR, to roll back the hack added in #1820 as well.

/cc @aleksandra-tarkowska, @joshmoore

@chris-allan Seems we don't need this now since @dpwrussell's fix. I'll remove it.

See ome#1820

chris-allan

Provided that we have passing integration tests I'm happy for this to be merged.

Test run status to be confirmed by @sbesson.

snoopycrimecop · 2016-10-27T02:33:50Z

Conflicting PR. Removed from build OMERO-DEV-merge-push#487. See the console output for more details.
Possible conflicts:

PR Open with #4630 will-moore 'Open with'
- components/tools/OmeroWeb/omeroweb/webgateway/urls.py
- components/tools/OmeroWeb/omeroweb/webgateway/views.py
Upstream changes
- components/tools/OmeroWeb/omeroweb/webgateway/views.py

atarkowska · 2016-10-27T10:43:38Z

Where is follow up trello card for all the post merging issues?

jburel · 2016-10-27T14:38:00Z

@aleksandra-tarkowska: I have just created a card https://trello.com/c/IXaPj64B/212-web-api-round-2

snoopycrimecop · 2016-10-28T09:57:42Z

Conflicting PR. Removed from build OMERO-DEV-merge-push#488. See the console output for more details.
Possible conflicts:

Upstream changes
- components/tools/OmeroWeb/omeroweb/webgateway/urls.py
- components/tools/OmeroWeb/omeroweb/webgateway/views.py

atarkowska · 2016-10-28T18:08:25Z

OmeroWeb.test.integration.test_api_projects/TestProjects/test_project_update/ is broken

self = <test_api_projects.TestProjects object at 0x83d7510>
user1 = (<omero.clients.BaseClient object at 0x7839650>, object #0 (::omero::model::Experimenter)
{
    _id = object #1 (::omero::RLong...nnotationLinksLoaded = False
    _annotationLinksCountPerOwner = 
    {
    }
})

    def test_project_update(self, user1):
        conn = get_connection(user1)
        group = conn.getEventContext().groupId
        user_name = conn.getUser().getName()
        django_client = self.new_django_client(user_name, user_name)

        project = ProjectI()
        project.name = rstring('test_project_update')
        project.description = rstring('Test update')
        project = get_update_service(user1).saveAndReturnObject(project)

        # Update Project in 2 ways...
        version = settings.API_VERSIONS[-1]
        project_url = reverse('api_project', kwargs={'api_version': version,
                                                     'pid': project.id.val})
        save_url = reverse('api_save', kwargs={'api_version': version})
        # 1) Get Project, update and save back
        project_json = _get_response_json(django_client, project_url, {})
        assert project_json['Name'] == 'test_project_update'
        project_json['Name'] = 'new name'
        rsp = _csrf_put_json(django_client, save_url, project_json)
        assert rsp['@id'] == project.id.val
        assert rsp['Name'] == 'new name'    # Name has changed
        assert rsp['Description'] == 'Test update'  # No change

        # 2) Put from scratch (will delete empty fields, E.g. Description)
        save_url += '?group=' + str(group)
        payload = {'Name': 'updated name',
                   '@id': project.id.val}
        # Test error message if we don't pass @type:
        rsp = _csrf_put_json(django_client, save_url, payload, status_code=400)
        assert rsp['message'] == 'Need to specify @type attribute'
        # Add @type and try again
        payload['@type'] = project_json['@type']
        rsp = _csrf_put_json(django_client, save_url, payload)
        assert rsp['@id'] == project.id.val
        assert rsp['Name'] == 'updated name'
>       assert 'Description' not in rsp
E       assert 'Description' not in {'@id': 813, '@type': 'http://www.openmicroscopy.org/Schemas/OME/2016-06#Project', 'Description': '', 'Name': 'updated name', ...}

test/integration/test_api_projects.py:560: AssertionError
Standard Output

Vary: Cookie
Content-Type: application/json

{"omero:details": {"owner": {"UserName": "0ea9be08-1e0c-4236-92c8-86b0d0af0886", "FirstName": "0ea9be08-1e0c-4236-92c8-86b0d0af0886", "omero:details": {"@type": "TBD#Details", "permissions": {"canDelete": false, "perm": "rwra--", "canEdit": false, "canAnnotate": false, "canLink": true, "@type": "TBD#Permissions"}}, "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#Experimenter", "LastName": "0ea9be08-1e0c-4236-92c8-86b0d0af0886", "@id": 2487, "Email": ""}, "group": {"omero:details": {"@type": "TBD#Details", "permissions": {"canDelete": false, "perm": "rwra--", "canEdit": false, "canAnnotate": false, "canLink": true, "@type": "TBD#Permissions"}}, "@id": 2221, "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#ExperimenterGroup", "Name": "853a3fca-edfa-4a45-bf99-678dbe66897d"}, "@type": "TBD#Details", "permissions": {"canDelete": true, "perm": "rwra--", "canEdit": true, "canAnnotate": true, "canLink": true, "@type": "TBD#Permissions"}}, "Description": "Test update", "@id": 813, "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#Project", "Name": "new name"}
Vary: Cookie
Content-Type: application/json

{"message": "Need to specify @type attribute"}
Vary: Cookie
Content-Type: application/json

{"omero:details": {"owner": {"UserName": "0ea9be08-1e0c-4236-92c8-86b0d0af0886", "FirstName": "0ea9be08-1e0c-4236-92c8-86b0d0af0886", "omero:details": {"@type": "TBD#Details", "permissions": {"canDelete": false, "perm": "rwra--", "canEdit": false, "canAnnotate": false, "canLink": true, "@type": "TBD#Permissions"}}, "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#Experimenter", "LastName": "0ea9be08-1e0c-4236-92c8-86b0d0af0886", "@id": 2487, "Email": ""}, "group": {"omero:details": {"@type": "TBD#Details", "permissions": {"canDelete": false, "perm": "rwra--", "canEdit": false, "canAnnotate": false, "canLink": true, "@type": "TBD#Permissions"}}, "@id": 2221, "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#ExperimenterGroup", "Name": "853a3fca-edfa-4a45-bf99-678dbe66897d"}, "@type": "TBD#Details", "permissions": {"canDelete": true, "perm": "rwra--", "canEdit": true, "canAnnotate": true, "canLink": true, "@type": "TBD#Permissions"}}, "Description": "", "@id": 813, "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#Project", "Name": "updated name"}

will-moore · 2016-10-29T08:45:00Z

@aleksandra-tarkowska If this test is failing on the devspace it's probably because it doesn't have the latest server since that test now requires #4829. See #4708 (review)

atarkowska · 2016-10-29T09:16:15Z

@will-moore, I am afraid the problem must be somewhere else. It was tested against last night merge build snoopycrimecop@c2f9b13.

will-moore · 2016-10-31T12:45:16Z

Sorry, my mistake. The fix I need is actually from omero_marshal: ome/omero-marshal@e83f056
So I have just updated requirements to specify omero-marshal 0.4.1

sbesson · 2016-10-31T12:56:55Z

components/tools/OmeroWeb/requirements-common.txt

 Django>=1.8,<1.9
 django-pipeline==1.3.20
-git+git://github.com/openmicroscopy/omero-marshal.git@v0.4.0#egg=omero-marshal
+git+git://github.com/openmicroscopy/omero-marshal.git@v0.4.1#egg=omero-marshal


Any reason to use the Git form rather than omero-marshal==0.4.1 as above ?

Not that I know of.

will-moore · 2016-11-01T06:41:21Z

Last commit should fix https://ci.openmicroscopy.org/view/Failing/job/OMERO-DEV-merge-integration-Python27/lastCompletedBuild/testReport/OmeroWeb.test.integration.test_api_projects/TestProjects/test_project_update/

atarkowska · 2016-11-02T06:58:39Z

indeed, tests are green

jburel · 2016-11-02T08:45:12Z

Thanks all. Merging

will-moore added 3 commits June 14, 2016 13:17

Add /api/m/projects using new api_query.py and api_marshal.py

b8e8b3f

Add /version/ to url, ignored for now

180fddb

Add pip install omero-marshal to .travis.yml

fff45a1

sbesson reviewed Jun 14, 2016
View reviewed changes

Fix copyright date

cdabd97

Remove -r from pip install omero-marshall

061dd6a

will-moore added 2 commits June 14, 2016 16:34

Add support for ?owner=1&group=2 to /projects/

f68ddef

childCount false be default

8e8bd88

atarkowska mentioned this pull request Jun 14, 2016

Bugfix: mount nginx as /home/omero ome/devspace#46

Closed

jburel added exclude develop labels Jun 14, 2016

jburel added the breaking label Jun 14, 2016

will-moore added 2 commits June 15, 2016 10:43

Add omero-marshal to common requirements-common.txt file

e0ce1d8

This provides a single point to specifiy additional requirements that is consumed by the other various requirements files.

Remove omero-marshal from .travis.yml

c39eadc

This is now provided by OmeroWeb/requirements-common.txt

will-moore added 5 commits June 15, 2016 22:21

Use request.GET instead of request.REQUEST

54a3916

Restrict api urls to supported versions

0983701

Added api_base and api_versions with urls to browse

2cca1d9

Add /token /servers and /login urls

73be9c5

url renaming - /api/ is 'api_versions'

5ddfae1

will-moore added 2 commits October 25, 2016 15:53

json_response class handles success or error with separate methods

9c51b0e

This allows behaviour to be customised by subclasses.

Add docs for query_projects()

f31fb65

chris-allan reviewed Oct 25, 2016

View reviewed changes

will-moore added 3 commits October 26, 2016 13:09

Remove __getarrt__ workaround on @json_response decorator

bd6c4d3

See ome#1820

flake8 fix

2330774

Update test_project_update with PR ome#4829 fix

3d0e563

chris-allan approved these changes Oct 26, 2016

View reviewed changes

will-moore closed this Oct 26, 2016

will-moore reopened this Oct 26, 2016

will-moore mentioned this pull request Oct 27, 2016

Open with #4630

Merged

Merge remote-tracking branch 'origin/develop' into webgateway_api2

d4de580

Update to omero-marshal 0.4.1 in requirements-common.txt

e484d2e

sbesson reviewed Oct 31, 2016

View reviewed changes

Using 'omero-marshal==0.4.1' in requirements.txt

e4a6c6a

jburel merged commit 49109b0 into ome:develop Nov 2, 2016

will-moore mentioned this pull request Nov 3, 2016

web api app #4920

Merged

atarkowska mentioned this pull request Jan 16, 2017

Api urls in json #5035

Merged

jburel added this to the 5.3.0 milestone Mar 29, 2017

will-moore deleted the webgateway_api2 branch February 18, 2019 04:12

Conversation

will-moore commented Jun 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbesson Jun 14, 2016

Choose a reason for hiding this comment

Uh oh!

will-moore commented Jun 14, 2016

Uh oh!

knabar commented Jun 14, 2016

Uh oh!

atarkowska commented Jun 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

will-moore commented Jun 14, 2016

Uh oh!

atarkowska commented Jun 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jburel commented Jun 14, 2016

Uh oh!

jburel commented Jun 14, 2016

Uh oh!

will-moore commented Jun 15, 2016

Uh oh!

will-moore commented Jun 15, 2016

Uh oh!

atarkowska commented Jun 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

will-moore commented Jun 15, 2016

Uh oh!

chris-allan Oct 25, 2016

Choose a reason for hiding this comment

Uh oh!

will-moore Oct 26, 2016

Choose a reason for hiding this comment

Uh oh!

chris-allan left a comment

Choose a reason for hiding this comment

Uh oh!

snoopycrimecop commented Oct 27, 2016

Uh oh!

atarkowska commented Oct 27, 2016

Uh oh!

jburel commented Oct 27, 2016

Uh oh!

snoopycrimecop commented Oct 28, 2016

Uh oh!

atarkowska commented Oct 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

will-moore commented Oct 29, 2016

Uh oh!

atarkowska commented Oct 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

will-moore commented Oct 31, 2016

Uh oh!

sbesson Oct 31, 2016

Choose a reason for hiding this comment

Uh oh!

will-moore Oct 31, 2016

Choose a reason for hiding this comment

Uh oh!

will-moore commented Nov 1, 2016

Uh oh!

atarkowska commented Nov 2, 2016

Uh oh!

jburel commented Nov 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

will-moore commented Jun 14, 2016 •

edited

Loading

atarkowska commented Jun 14, 2016 •

edited

Loading

atarkowska commented Jun 14, 2016 •

edited

Loading

atarkowska commented Jun 15, 2016 •

edited

Loading

atarkowska commented Oct 28, 2016 •

edited

Loading

atarkowska commented Oct 29, 2016 •

edited

Loading