|Apr 24||Modding Day|
This document explains the tech-side status of the project at the end of phase 2a, possible questions, and what improvements could/will be undertaken in the future.
Project communication with developers using the API is envisioned as consisting of three components: API documentation, an interactive community forum, and for important alerts, an email bulletin.
The project provides static documentation and frequently asked questions at the project homepage, http://www.lmiforall.org.uk. Due to limited server resources, this documentation is currently static and bare-bones. The servers' capacity has now been extended, and the project website will be upgraded to a dynamic Content Management System (likely Wordpress) soon. In addition to the project website, the interactive API Explorer is available at http://api.lmiforall.org.uk. Developers can explore and try out functions of the API live on this website; this constitutes the main body of the developer documentation.
Furthermore, the project wiki at http://collab.lmiforall.org.uk provides in-depth knwledge about certain aspects of the project. However, not all of this information is pertinent (or even released as) developer information, so developers only have access to the “non-disclosive” portions of this wiki.
With the extended server capacity, it is now possible to set up a community forum for the LMI For All project, where developers, stakeholders and the project tech team can interact. We assume the following benefits to this approach:
For very important messages (planned server downtime, major API updates, major changes in the data), it seems prudent to alert developers via a bulletin system in advance. This could, for example, be realised via an email list. Either set up with a sign-up form on the main project website, or by requiring email addresses to sign up for an API key, this list would then be used to occasionally send out important information. Note that this is not planned as a regular 'newsletter' type of dissemination– developers tend to be annoyed by newsletters, so this should be a separate sign-up, if it is offered at all.
Another way of alerting developers about important project events could be setting up a Twitter account. Since Twitter works in both directions (unlike an email bulletin), this account would have to be monitored for any incoming questions once a day or so.
It is best here to first determine when API keys are actually necessary. Server resources have been scarce towards the end of phase 2a, so usage tracking by keys had not been implemented. However, due to the way the world wide web works, keys aren't actually necessary to track API usage on websites.
Every call or request made from a web browser includes a bit of data called the referrer. The referrer is wherever this request came from– when following a link, the referrer is the previous webpage. When making an API call, the referrer is the page the API was used on. Hence, we are already being provided with rough but useful statistics by referrers, and API keys are not necessary to track just where the API is used.
API keys become necessary in the following circumstances:
Note that blocking excessive usage can still be done by referrer-tracking only; it is the contact part that is addressed by the API key (the developer has to go to a website and possibly sign a contract to get it). Excessive native usage can also be blocked by device (IP address). Here, the API key is just necessary to identify the particular application (again, by having the developer contact and give this information). As such, the project team was comfortable relying on referrer-tracking only so far.
To curtail abuse of API resources and force developers to contact the project team if they anticipate great interest in their applications, several levels of usage rate limiting will be introduced as soon as API keys become available. Following is the set of rules we propose for this.
The project team is still debating the final version of the rate limiting rules. There is also the matter of designated app keys– those need to be cryptographically signed so other developers can't steal them, which introduces a fair amount of additional complexity for the developers and the tech team as well. Unsigned keys are much easier to manage, but don't guarantee that the tracking is correct. We welcome UKCES's input on what level of rate limiting and security is acceptable here.
An API key self-service interface is currently under construction. Here, developers will be able to manage their API keys, see their current usage, and create and revoke designated app keys (should they be offered). API keys are tied to developer email addresses and the website domains where their widgets/apps will be deployed.
The complexity of usage statistics is highly dependent on whether designated app keys will be used or not. In any case, an interface will be provided that generates a graphical report after API key tracking has been implemented. “Raw” referrer-based statistics are already available at http://api.lmiforall.org.uk/report.html. This page is periodically updated.
Several steps had already been taken to ensure as high as possible availability within the limitations of the project. After an unexpected outage caused (apparently) by the hosting provider restarting one of the servers, additional monitoring was layered on top to improve coverage.
The API has several service layers, which are interspersed with monitoring layers. Here is the current stack:
Project team alerts are routed to the project team's live tech chatroom as alerts, and via email to me (Philipp Rustemeier) on my mobile.
The API doesn't currently store any data on the server itself and instead reads everything from the database server. The API server currently has no critical information that needs to be backed up, and a backup plan is not in place. Non-critical information that could be backed up includes access logs and error logs. As soon as we begin tracking API keys, that information would also be a candidate for backup.
database server backups?
The server infrastructure is currently non-redundant; everything is a single point of failure due to budgetary constraints. Steps have been taken to ensure that the API server and its internal monitoring recover automatically from unplanned outages.