Differences

This shows you the differences between two versions of the page.

--- librematch:backend [2022-10-10 22:24] – [REST-API] simonsan
+++ librematch:backend [2022-10-23 21:19] (current) – removed nimda
@@ Line 1: / Line 1: @@
-====== LibreMatch Backend ======
-----
-===== Design =====
-==== Requirements ====
-=== User Stories ===
-Some things may be out of scope, we can still collect them here, to have an overview and include some of them in the design process.
-**User** stands for the user of the //LibreMatch// API.
-^As a/an ^I want to... ^So that.. ^
-| user | have a stable API | I can rely on it |
-| user | request win data of the games | I can easily determine the outcome of a match |
-| statistician | request database dumps for a time period | I can easily analyse a big amount of data without firing thousands of requests |
-| authenticated tournament organizer| delay matches showing up for a limited amount of time | I can take care that people are not being spoiled |
-| user | be able to get live updates about ongoing games | I can update my e.g. overlay immediately |
-| user | be able to log in with my Steam account | I can create lobbies from outside the game |
-| user | be able to query for matches of a player since a timestamp | I can update a list of matches of a certain user more easily |
-| user | be able to quickly start using the API | I am not spending much time reading the documentation or spend time implementing a client to use it |
-| user | be able to get all matches by a player from different games at once | I don't need to make several requests |
-| user | be notified of match state updates of all matches or a subset | I don't have to keep polling for updates and deriving state updates myself |
-| user | get notified when changes are made to a lobby | I can update my lobby tables and reduce the amount of requests |
-| recorded matches analyser | get notified when a new replay becomes available (e.g. in a leaderboard/from a player) | I can download and parse it in a timely fashion |
-| user | embed a player report function in my page | people can report suspicious activity (e.g. spoofing) to the official franchise |
-==== Additional Tooling ====
-== LibreMatch Proxy ==
-We will need a running instance of a Steam Client for each game. To circumvent limitations (as only holding one login to steam per account on a certain moment in time is needed) we use a proxy to forward our requests on behalf of a certain session.
-//Usage//
-Base URL: **https://rlink.api.tools.uber.space/relic**
-You need to get an //API_KEY// for the proxy from the team members.
-Then you can make calls to the proxy e.g. for endpoint `/game/news/getNews` call:
-<code>https://proxy.api.tools.uber.space/relic/game/news/getNews</code>
-You need to supply a header with the //API_KEY// e.g. <code>API_KEY=...</code>
-All query parameters that you add to the request will be forwarded to the Relic Link API on your behalf.
-== Insomnia  ==
-[[ https://insomnia.rest/ | Insomnia ]] is a API design platform.
-<WRAP center round download 60%>
-We have created some {{ :insomnia_librematchexamples.json | LibreMatch examples}} to use with their client.
-</WRAP>
-==== Architecture ====
-=== Introduction ===
-== Example (Bright Sky) ==
-[[ https://brightsky.dev/ | Bright Sky ]] pulls weather data from DWD and implements a JSON-API on top of it for its users.
-Their architecture looks like on the image on the right side. It's some kind of the same what we also want to achieve.
-{{:librematch:architecture_1_.png?400 |}}
-This is taken from the [[ https://github.com/jdemaeyer/brightsky/#architecture | Official repository ]]:
-  * The brightsky worker, which leverages the logic contained in the brightsky Python package to retrieve weather records from the DWD server, parse them, and store them in a database. It will periodically poll the DWD servers for new data.
-  * The brightsky webserver (API), which serves as gate to our database and processes all queries for weather records coming from the outside world.
-  * A PostgreSQL database consisting of two relevant tables:
-    *  sources contains information on the locations for which we hold weather records, and
-    * weather contains the history of actual meteorological measurements (or forecasts) for these locations.
-  * A Redis server, which is used as the backend of the worker's task queue.
-== LibreMatch (Draft) ==
-{{ ::librematch-arch-draft.png?400|}}
-Basically we should poll the Relic Link API endpoints for all data every so many seconds (this time is our minimum latency for our whole system).
-So the time frame in which we see certain information (Match infos, wins of games) and could deliver information to our API.
-Then we parse the results into a data structure of ourselves to make error handling easier and also deal with types of data and not handle everything as a huge JSON-Array that is being exposed by the Relic Link API. This data we send to the database which is our persistence layer. From our persistence layer we are able to request data for our endpoints that we expose to the users.
-Some endpoints may require a Pub/Sub API (e.g. live updates for match information), some are fine with the usual REST-API approach.
-{{:librematch:librematch_2022-10-09_3_.jpg?600|}}
-=== Data sources ===
-{{:librematch:data_sources.png?600 |}}
-There are a few available data sources (resource) we can query our data from.
-== Relic Link API ==
-  * Authenticated endpoints\\ Requests to these endpoints should be rather limited and only for really essential things.\\ Most of our data we should get from the /community-Endpoints.
-  * Community endpoints\\ Data here is relatively scarce, though most of our data we should fetch from here.
-== ageofempires.com API ==
-== Steamworks API ==
-=== Proxy ===
-{{:librematch:proxy.png?600|}}
-The proxy forwards our requests and takes over authentication.
-For the authenticated endpoints we should keep in mind:
-  * use a separate Steam account (or maybe even this Xbox stuff with a subscription for all 4 games for that (cheaper?))
-  * generally limit our access to the auth endpoints to a minimum (only what is really needed)
-    * even rate-limit our requests
-  * mimic in-game requests as close as possible
-Related: [[librematch:backend#librematch_proxy|LibreMatch Proxy]]
-=== Collector ===
-{{:librematch:collector.png?600| }}
-== Polling ==
-The polling stage checks for updated data frequently. [[https://huey.readthedocs.io/en/latest/ | huey]] as a task scheduler/queue takes over triggering our polling jobs.
-Data queried from the authenticated endpoints get a separate job.
-These endpoints need special treatment, as in only query what's essential.
-== Parsing ==
-The parsing stages does the decoding/decompression, normalization and prioritization.
-  * decoding/decompression \\ Some data, especially in 'slotinfo', is base64 encoded and further compressed with zlib.
-  * normalization \\ The data format is 'json-like', but only contains arrays. Probably for bandwidth reasons.\\ We need to transfer the data in our own schema.
-  * prioritization \\ Some data is the foundation for real-time information that we deliver to users of our API (state changes of lobbies, e.g. ongoing matches, and leaderboard rankings). \\ This data should fill 'sorted sets' or the Pub/Sub-Message bus from Redis.
-=== Persistence ===
-TODO: Create an overview over all data we can get from the RelicLink API.
-== Redis ==
-== Database ==
-= Schema =
-First step:
-[[ https://github.com/amacneil/dbmate | Version control our database schema with a lightweight, framework-agnostic database migration tool ]]
-== Postgres Statistics Extension ==
-To visualise long-running requests and loads we should use some statistics extension for Postgres.
-== Schema ==
-== Partitioning ==
-== Replication ==
-=== Backend ===
-== Authentication ==
-Sometimes resources will need authorisation to be accessed by privileged community members. Such members could be tournament organizers and admins that want to setup a schedule within the backend to delay the access to certain player information for a limited amount of time. This could be needed during tournaments, so people aren't able to find out results of matches before they are officially concluded (e.g. in case of casting recorded games) and spoil other people with the results.
-The Relic Link ID aka RLINK_ID is a unique identifier for every player on the Relic Link platform, no matter if the person bought a game on Steam or Microsoft Store. Authentication for a player on that platform could be achieved by utilising [[ https://partner.steamgames.com/doc/features/auth#website | web browser based authentication with Steam OpenID ]].
-== REST-API ==
-** OpenAPI **
-To make it easy to start using our API we should provide an [[ https://en.wikipedia.org/wiki/OpenAPI_Specification | OpenAPI file ]] so that the users are able to generate clients for our API for their favourite programming language. An [[ https://swagger.io/ | OpenAPI ]] file also works well as a complete documentation of the API. There are several [[ https://openapi.tools/ | Tools ]] that make dealing with an OpenAPI a breeze.
-Our OpenAPI file works as our specification for us. We design the API with it and generate a server stub that we implement for our use. The good thing about it is, that we can easily generate clients utilizing it and also testing of the endpoints is much easier. It comes at the cost of a longer design phase. But we assume that it's worth it to guarantee a stable interface.
-== Pub/Sub-API ==
-Some information for players might want to be constantly queried by the users of our API, for example match information. These information we should supply via an endpoint that can be subscribed to and communicates via a Websocket.
-==== Implementation ====
-==== Roadmap ====