datacatalog.plugins.postgres package

Postgres storage and search plugin.

_etag_from_str(s)[source]
Return type:str
_execute_list_query(app, filterexpr, lang, sort_field_get)[source]
_execute_search_query(app, filterexpr, lang, q)[source]
_extract_values(elm, ptr_parts, ptr_idx=0)[source]

Recursive generator that yields all values that may live under the given JSON pointer.

_iso_639_1_code_to_pg(iso_639_1_code)[source]
Return type:str
_to_pg_json_filterexpression(filters)[source]
Return type:str
_to_pg_json_query(q)[source]
Return type:str
_to_pg_lang(iso_639_1_code)[source]
Return type:str
add_startup_action(app, name)[source]
check_startup_action(app, name)[source]
Return type:bool
deinitialize(app)[source]

Deinitialize the plugin.

get_old_identifiers(app)[source]
health_check(app)[source]

Health check.

Return type:Optional[str]
Returns:If unhealthy, a string describing the problem, otherwise None.
Raises:Exception – if that’s easier than returning a string.
initialize(app)[source]

Initialize the plugin.

This function validates the configuration and creates a connection pool. The pool is stored as a module-scoped singleton in app[‘pool’].

Search

See datacatalog.plugin_interfaces.search_search()

Return type:Asyncgenerator[Tuple[str, dict], None]
set_new_identifier(app, old_id, new_id)[source]
storage_all(app)[source]
Return type:Asyncgenerator[Tuple[str, str, dict], None]
storage_create(app, docid, doc, searchable_text, iso_639_1_code)[source]

Store a new document.

Parameters:
  • app (Mapping[~KT, +VT_co]) – the application
  • docid (str) – the ID under which to store this document. May or may not already exist in the data store.
  • doc (dict) – the document to store; a “JSON dictionary”.
  • searchable_text (str) – this will be indexed for free-text search.
  • iso_639_1_code (Optional[str]) – the language of the document.
Return type:

str
Returns:

new ETag
Raises:

KeyError if the docid already exists.
storage_delete(app, docid, etags)[source]

Delete document only if it has one of the provided Etags.

Parameters:
  • app (Mapping[~KT, +VT_co]) – the application
  • docid (str) – the ID of the document to delete.
  • etags (Set[str]) – the last known ETags of this document.
Raises:
  • ValueError – if none of the given etags match the stored etag.
  • KeyError – if a document with the given id doesn’t exist.
Return type:

None
storage_extract(app, ptr, distinct=False)[source]

Generator to extract values from the stored documents, optionally distinct.

Used to, for example, get a list of all tags or ids in the system. Or to get all documents stored in the system. If distinct=True then the generator will cache all values in a set, which may become prohibitively large.

Parameters:
  • app (Mapping[~KT, +VT_co]) – the application
  • ptr (str) – JSON pointer to the element.
  • distinct (bool) – Return only distinct values.
Raises:

ValueError if filter syntax is invalid.
Return type:

Generator[str, None, None]
storage_id()[source]

New unique identifier.

Returns a URL-safe random token with 80 bits of entropy, base64 encoded in ~13 characters. Given the birthday paradox this should be safe upto about 10 bilion (2^35) entries, when the probability of collisions is approximately 0.05% (p=0.0005).

Return type:str
storage_retrieve(app, docid, etags=None)[source]

Get document and corresponsing etag by id.

Parameters:
  • app (Mapping[~KT, +VT_co]) – the application
  • docid (str) – document id
  • etags (Optional[Set[str]]) – None, or a set of Etags
Return type:

Tuple[Optional[dict], str]
Returns:

A tuple. The first element is either the document or None if the document’s Etag corresponds to one of the given etags. The second element is the current etag.
Raises:

KeyError – if not found
storage_update(app, docid, doc, searchable_text, etags, iso_639_1_code)[source]

Update the document with the given ID only if it has one of the provided Etags.

Parameters:
  • app (Mapping[~KT, +VT_co]) – the application
  • docid (str) – the ID under which to store this document. May or may not already exist in the data store.
  • doc (dict) – the document to store; a “JSON dictionary”.
  • searchable_text (str) – this will be indexed for free-text search.
  • etags (Set[str]) – one or more Etags.
  • iso_639_1_code (Optional[str]) – the language of the document.
Return type:

str
Returns:

new ETag
Raises:

ValueError if none of the given etags match the stored etag.
Raises:

KeyError if the docid doesn’t exist.

Modules

languages