Commit a0d94a9c authored by Hugo Buddelmeijer's avatar Hugo Buddelmeijer
Browse files

Description of the ExternalTAP concept.

parent ff857cfd
The ExternalTAP is a SourceCollection to access IVOA TAP resources.
See astro/experimental/SourceCollection:
- class derived from SourceCollection
- demonstration to use the TAP functionality
* Retrieving Catalog Data:
The idea is that an ExternalTAP SC represents a table from a TAP resource it its entirety. FilterSources can use such an ExternalTAP as a parent to select a subset of the sources.
This is done through the load_data_tap() function, analogous to the load_data_sql() and load_data_python() functions. Ultimately this new function should be integrated with load_data(), as the others are.
The load_data_tap() function uses the get_tapurl_dict() function to determine how to retrieve the data. This function is currently only implemented in the ExternalTAP and FilterSources classes.
* Caching Catalog Data:
Downloaded rows (sources) can be stored in the sourcelist_data of an ExternalTAP. This sourcelist_data should be shared with a FilterSources that uses the ExternalTAP as parent. That way, the catalog data of the ExternalTAP can be cached partially, by storing the sources of the FilterSources.
This does not work yet, because (all) the SourceCollection classes assume that every source/row has a unique SLID/SID combination. This is not yet the case for rows retrieved over TAP.
* Using Cached Catalog Data:
There are two ways to use the cached catalog data. (Not yet implemented, because storing the catalog data is not working yet.)
- Set the 'all_data_stored' flag of a FilterSources that uses an ExternalTAP as parent.
- Instead of fetching the data, do a 'count()' query first on both the TAP resource and the sourcelist_data. If they match, all rows are fetched.
The optimization routines of the SourceCollectionTree can convert the ExternalTAP to a regular External once it has determined that all required rows are retrieved.
The ExternalTAP (or a SourceCollection like it) could be used in OMA, and some OMA functionality could be of use in the ExternalTAP. Some thoughts about this:
- OMA performs ConeSearches, the ExternalTAP currently only ADQL queries. Shouldn't be difficult to create a ConeSearch SourceCollection, or integrate that functionality with the FilterSources.
- Most of the OMATable class could be integrated with the TableConverter, I think. The evalFunction is similar to the AttributeCalculator.
- Storing retrieved OMA data requires a column-attribute mapping, because the attributes are stored in generic columns (e.g. 'USER_1_i'). The SourceCollections are designed to handle this (through the attribute_columns and attribute_names properties).
- An important goal of OMA is to associate sources. Although the SourceCollections are designed to facilitate this, it is not (yet?) possible to associate SourceCollections. This is not trivial, see the last sections of chapter 2/3 of my thesis.
- OMA handles the VOResource properly. E.g. with these classes: VOThread, AccessVOResource, AccessRegister, ResourceTree. This could perhaps be separated entirely? E.g. somewhere in astro.util?
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment