Expose stored procedures by begriffs · Pull Request #228 · PostgREST/postgrest

Conversation

Allow client to invoke stored procedures by POSTing to /rpc/proc_name with named arguments as keys in the JSON request body.

This is a work in progress. I created a pull request as a place for conversation as the feature evolves.

@begriffs

Major kudos for helping to bring PostgreSQL and Haskell to the masses. My Haskell experience consists of building this branch, so I cannot do much directly to progress this issue.

What do you need to make this happen (feedback/testing/use cases)?

Keep up the great work!

@jmealo thanks, I'd love to hear your ideas for use cases actually. I could then turn them into feature tests and get them passing on this branch.

@begriffs: I'm running this branch now and was wondering how to pass in parameters... I tried this:

POST /rpc/get_descendant_asn_ids?standard_asn_id=S11436A8 HTTP/1.1

{"standard_asn_id": "S11436A8"}

I'm fairly sure it doesn't look for parameters in the URL; but I tried it for good measure.

Will the RPC endpoints simply accept named parameters and cast them to the correct types using introspection?

Yeah it tries to match the json keys in the post body with named params in the stored procedure. What response are you getting when you make that post?

(And you're right, query string params are not passed to stored procs. They are used for filtering output)

The response for that request is:

{
    "message": "Cannot decode byte '\\x0': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream"
}

What return values are accepted? Is it just SET OF, or could it return any type? (I'm not sure how a boolean would handle filter values being passed to it)

It seems like this problem is happening before any of the information even makes it to the database. Can you tell me more about your development machine? What os are you using, and how are you making the post request? I'm wondering if there is a character encoding problem happening, like the post is being made with character encoding of something other than utf8.

My development machine is OSX. The request was used making the Postman app for Chrome.

Here is a curl command that issues an identical request:

curl 'http://<<redacted>>:3000/rpc/get_descendant_asn_ids?standard_asn_id=S11436A8' -H 'Origin: chrome-extension://fdmmgilgnpjigdojojpjoooidkmcomcm' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36' -H 'Content-Type: text/plain;charset=UTF-8' -H 'Accept: */*' -H 'Cache-Control: no-cache' -H 'Connection: keep-alive' --data-binary '{"standard_asn_id": "S11436A8"}' --compressed

The server is running Ubuntu 14.04 with Postgres 9.4.4. Here's my pg_config output:

BINDIR = /usr/lib/postgresql/9.4/bin
DOCDIR = /usr/share/doc/postgresql-doc-9.4
HTMLDIR = /usr/share/doc/postgresql-doc-9.4
INCLUDEDIR = /usr/include/postgresql
PKGINCLUDEDIR = /usr/include/postgresql
INCLUDEDIR-SERVER = /usr/include/postgresql/9.4/server
LIBDIR = /usr/lib/x86_64-linux-gnu
PKGLIBDIR = /usr/lib/postgresql/9.4/lib
LOCALEDIR = /usr/share/locale
MANDIR = /usr/share/postgresql/9.4/man
SHAREDIR = /usr/share/postgresql/9.4
SYSCONFDIR = /etc/postgresql-common
PGXS = /usr/lib/postgresql/9.4/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' '--with-tclconfig=/usr/lib/x86_64-linux-gnu/tcl8.6' '--with-includes=/usr/include/tcl8.6' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.4/man' '--docdir=/usr/share/doc/postgresql-doc-9.4' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.4' '--bindir=/usr/lib/postgresql/9.4/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/usr/lib/mit-krb5 -L/usr/lib/x86_64-linux-gnu/mit-krb5' '--with-krb5' '--with-gssapi' '--with-ldap' '--with-selinux' 'CPPFLAGS=-D_FORTIFY_SOURCE=2'
CC = gcc
CPPFLAGS = -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.6
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer
CFLAGS_SL = -fpic
LDFLAGS = -L../../../src/common -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/usr/lib/mit-krb5 -L/usr/lib/x86_64-linux-gnu/mit-krb5 -Wl,--as-needed
LDFLAGS_EX = 
LDFLAGS_SL = 
LIBS = -lpgcommon -lpgport -lselinux -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -ledit -lrt -lcrypt -ldl -lm 
VERSION = PostgreSQL 9.4.4

My locale output:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

 curl 'http://<redacted>:3000/rpc/get_descendant_asn_ids' -v -X POST -d '{"standard_asn_id": "S11437A4"}'

Gets me:

* Hostname was NOT found in DNS cache
*   Trying <redacted>...
* Connected to <redacted> (<redacted>) port 3000 (#0)
> POST /rpc/get_descendant_asn_ids HTTP/1.1
> User-Agent: curl/7.35.0
> Host: <redacted>:3000
> Accept: */*
> Content-Length: 31
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 31 out of 31 bytes
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Date: Thu, 06 Aug 2015 20:22:39 GMT
* Server postgrest/0.2.10.1 is not blacklisted
< Server: postgrest/0.2.10.1
< Content-Type: text/plain
< 
* Connection #0 to host <redacted> left intact
S1143ABS1143ABS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ACS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADS1143ADF

That seems to have fixed the error, however, the expected output is an array of char(8)'s. This isn't even the string representation of that, which I'd expect to be {item1,item2,item3}.

Interestingly enough, another similar function does not work and returns the same error as before:

$ curl 'http://<redacted>:3000/rpc/get_descendant_standards_nodes' -v -X POST -d '{"standard_asn_id": "S11437A4"}'

Returns:

* Hostname was NOT found in DNS cache
*   Trying <redacted>...
* Connected to <redacted> (<redacted>) port 3000 (#0)
> POST /rpc/get_descendant_standards_nodes HTTP/1.1
> User-Agent: curl/7.35.0
> Host: <redacted>:3000
> Accept: */*
> Content-Length: 31
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 31 out of 31 bytes
< HTTP/1.1 500 Internal Server Error
< Transfer-Encoding: chunked
< Date: Thu, 06 Aug 2015 20:28:22 GMT
* Server postgrest/0.2.10.1 is not blacklisted
< Server: postgrest/0.2.10.1
< Content-Type: application/json
< 
* Connection #0 to host spark0.slatepowered.net left intact
{"message":"Cannot decode byte '\\xff': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream"}

It might be worth noting there are two forms of each of these functions, one that takes a depth parameter as an integer and one that does not. I'm not sure if that's at play here. It seems unlikely though due to the fact that the working one has two forms as well.

All of the functions that return an array seem to be working fine, not of them that return a SET OF are.

Here is a simple example that fails with the UTF error:

CREATE OR REPLACE FUNCTION get_root_standards_nodes(standard_asn_id CHAR(8))
  RETURNS SETOF standards_nodes AS $$

  SELECT * FROM standards_nodes WHERE parent_asn_id LIKE 'D%';
$$
LANGUAGE SQL;

If I change the same function to output JSON it works as expected, however, the content-type of the response is text/plain.

CREATE OR REPLACE FUNCTION get_root_standards_nodes(standard_asn_id CHAR(8))
  RETURNS JSON AS $$

SELECT array_to_json(array_agg(row_to_json(t))) FROM (
    SELECT * FROM standards_nodes WHERE parent_asn_id LIKE 'D%'
) t
$$
LANGUAGE SQL;

I think I see what's going on. My first test (the current code in this branch) tries to read the result of the stored procedure as if it is a single column of text. For sets of multiple columns it is not happy.

@jmealo I think this branch is ready to merge now, would you mind pulling it and trying your examples again to check that it fixed things?

The output is too deeply nested however

begriffs added a commit that referenced this pull request

Aug 21, 2015

Conversation

3 participants