---
title: "development"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{development}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```


```{r}
library(openthl)
```

## About

This article desribes the phase of development at 2020/10. It describes several perspectives of the THL open data API and the openthl package.

For a description of the API, see [THL open data API docs](https://yhteistyotilat.fi/wiki08/x/RoCkAQ).


## API queries and general data retrieval

### API data types

The API can return the following type of data

- JSON (API meta information: hrefs and names of datasets)
- JSONP (dimensions of a dataset: data labels and classifications)
- JSONSTAT (data)


### General API queries

The main API interaction is coded in the file `api.R`. The function `getFromAPI()` 
is a general purpose data retrieval function. 
It takes as parameter the URL and the 'type' of the data, which is either 'meta' or 'data'. 
Depending on the type, the parsing of the response is handled by either 
jsonlite (meta) or rjstat (data) packages. String manipulation is utilised to handle the JSONP case after which jsonlite is used.

The general idea is that this main retrieval function is called by more specialised functions, 
which further parse the content into useful formats (eg. data.frames, specialised S3 classes). 

```{r}
url <- openthl:::api_data_url(path = "epirapo")
reslist <- openthl:::getFromAPI(url, type = "meta")
reslist
```


### API URL's

There are two public base URLs for the API: beta and prod. 

```{r}
openthl:::url_base(type = "prod") # default
openthl:::url_base("beta")
```

The URL to the API is given by

```{r}
url_api(type = "prod") # default
```

The function `api_data_url()` builds URLs which can be queried by `getFromAPI()`. 
The result is a character vector which has as an attribute the main API URL. 
It also has the S3 class "api-data-url".

The API can return csv or json type data, but I think it would make sense for the R package to only interact with JSON.

```{r}
openthl:::api_data_url("epirapo", format = "json")
```

Relevant source codes:

- api.R
- urls.R

*Development thoughts*:

- There is currently no clear plan on how to query the beta API.
- the scope of the package could maybe be restricted to JSON data and hence all function 
arguments related to the format may not be needed.



## API exploration and user starting point

### API terminology

The API terminology includes the following hierarchy

- aihealue (subject)
- hydra (hydra)
- kuutio (cube)

The subject is a bit like a schema. A single subject can include multiple hydras. 
A single hydra can include multiple cubes. A cube is a dataset with labels 
according to a single language. 
I believe that there are always as many cubes in the hydra as there are translations to the dataset. 

### User interaction with the terminology

`thlSubject()` lists all cubes belonging to a subject

```{r}
subject <- "epirapo"
x <- thlSubject(subject)
x
```

`thlDatasets()` takes the output of `thlSubject()` and presents the same information, 
but parses the hrefs into columns. There is probably no good reason why `thlDatasets()` could not simply accept the subject character name as input, 
but currently it only accepts the object returned by `thlSubject()`.

```{r}
thlDatasets(x)
```

`thlHydra()` takes a subject name and a hydra name and returns hrefs to the cubes (datasets) in that hydra. 
For example, the hydra covid19case is translated into fi, en and sv, so it has 3 cubes.

```{r}
thlHydra(subject, hydra = "covid19case")
```


*Development thoughts*:

- It is a bit unclear what should be the starting point for the user. Is it an url to a cube? Is it an URL to a hydra? Is it the name of the subject and hydra?
- thlDatasets should probably take as input the subject name (i.e. call thlSubject() internally first)

Main functionality source file location: `retrieve.R`



## Cube dimensions: labels and metadata

### Dimensions

A single dimension in the cube is a hierarchical structure with multiple stages. 
An example is Area with stages hospital district (stage 1) and municipality (stage 2). 

### thlCube object

the thlCube function now returns an object which includes

- url: the url of the cube (fact url), which is also the argument of thlCube
- dimensions: A list of data frames which include the complete hierarchy of each
of the dimensions in the dataset (including also measures).


```{r}
library(openthl)
urls <- thlHydra("toitu", "ennakko3")
url <- urls[1, 1] # fi href
cube <- openthl::thlCube(url)
names(cube$dimensions)
```


### Dimensions object

The hierarchical dimension information is presented as a wide format data frame. 
The prefix in the column names indicates the stage. 
There is a single row per a unique label in the highest hierarchy level. 
In the example below the highest hierarchy level is municipality, 
so there are 311 rows (number of municipalities) in the dimension data.frame.

```{r}
str(cube$dimensions[[1]]) # first dimension
```


### Dimension retrieval and parsing


The function `get_dimensions` queries the API for dimension information. 

```{r}
url <- "https://sampo.thl.fi/pivot/prod/en/epirapo/covid19case/fact_epirapo_covid19case.json"
dimensions <- openthl:::get_dimensions(url) # list
```

The function `parse_dimensions()` parses all dimensions as a list of data frames (each with S3 class 'hydra_dimension_df')

```{r}
# parse all dimensions as a list of data frames (each with S3 class 'hydra_dimension_df')
DF <- openthl:::parse_dimensions(dimensions)
names(DF)
```

`parse_dimensions()` uses `getHierarchy()`, which parses a single dimension as a data frame.

```{r}
# parse a single dimension as a data frame
df <- openthl:::getHierarchy(dimensions$children[[1]], parent_id = dimensions$id[[1]])
str(df)
```


## Data retrieval

### Backend

Data retriaval and parsing is largely unimplemented. The general data retriaval function with type 'data' should be tested and parsing implemented.


### User interface

Methods could be written which utilise the object returned by `openthl::thlCube`, 
which contains the dimension meta informantion, to build queries which

- retrieve data
- adds labels to the data

For example the following methods could be implemented:

- select() (builds a query which chooses dimensions/measures)
- filter() (adds a filter to the query)
- collect() (retrieves the data)

Some choices need to be made regarding how the user refers to the dimensions 
and measures, i.e. whether to use ID's or labels. 
It may make sense to use ID's when select():ing. 
A stage could be referred to by for example <dimension_id>.<stage_id>. 
This needs some consideration and exploration on what is most straight forward.