speed-dreams-code/doc/download-manager

In-game download manager specification

Introduction

Speed Dreams features an in-game download manager to fetch assets, such as cars, tracks or drivers, from external providers. Assets are listed from a simple database on a known URL and fetched from a web server. The download manager can be accessed from the main screen by pressing the Downloads button.

This in-game download manager was loosely inspired by that from the FlightGear project.

This has the following benefits:

• Assets can be developed independently from Speed Dreams i.e., on separate repositories and possibly also by independent maintainers.
• As a free software project, Speed Dreams can only officially distribute assets under a free license, such as the CC BY-SA 4.0 or the Free Art License. However, users might still want to install assets under non-free licenses, such as non-commercial licenses.
• It allows Speed Dreams to contain only a minimum essential set of assets on its source tree, thus reducing the repository size significantly.
  • Existing assets can be split into their own repositories, too.

Even if Speed Dreams configures the official assets database by default, this download manager has been designed to remain decentralised. This means any web server compatible with the specification defined here can be added or removed from the list.

Asset database

In order for Speed Dreams to know which assets can be downloaded, a JSON database must be defined. A formal JSON schema and a practical example are defined. A database can be verified with the check-cjsonschema utility:

check-jsonschema --schemafile assets.schema.json <file>

The database defines arrays for the following asset types:

• Cars
• Tracks
• Drivers

In JSON syntax:

{
    "cars": [
    ],
    "tracks": [
    ],
    "drivers": [
    ],
}

Each one of the types listed above is optional. For example, tracks can be ommitted if a database does not define any:

{
    "cars": [
    ],
    "drivers": [
    ],
}

All entries inside cars, drivers or tracks follow the same schema:

{
    "name": <string>,
    "category": <string>,
    "author": <string>,
    "license": <string>,
    "url": <string>,
    "hashtype": <string>,
    "hash": <string>,
    "size": <string>,
    "thumbnail": <string>,
    "directory": <string>,
    "revision": <string>
},

Where:

• name: defines the human-readable name for an asset.
• category: defines which category an asset belongs to.
  • For cars, the value is ignored.
  • For tracks, the following values are accepted:
    • development
    • dirt
    • gprix
    • karting
    • road
    • speedway
  • For drivers, it must be one of the available robots, such as simplix or usr.
• author: defines the author(s) for an asset. Speed Dreams shall not perform any specific actions based on the value of this string.
• license: defines the license covering the asset. Speed Dreams shall not perform any specific actions based on the value of this string.
• url: defines the URL that contains the asset. http and https are currently the only supported protocols, but others might be added in the future. The URL must point to a ZIP file. The download manager shall not require any file extension to the ZIP file. The URL does not have to point to the same web server hosting the asset database.
• hashtype: defines the hash function used to calculate the hash for the asset. As of the time of this writing, only sha256 is supported, but new entries might be introduced in the future.
• hash: resulting hash as calculated by the hash function defined by hashtype. The download manager shall reject downloaded assets whose calculated hash does not match the expected hash defined here.
• size: asset size, in bytes. This value is expressed as a string so as to restrict the use JSON numbers, as those are defined as floating-point values. Subdivisions, such as KiB or MiB, are not allowed.
• thumbnail: defines the URL for the thumbnail that will be displayed on the download manager. The URL must point to an image file with an extension known to the GfTexReadImageFromFile function. The URL does not have to point to the same web server hosting the asset database.
• directory: defines the directory where the asset shall be stored. datadir depends on how Speed Dreams is configured during build and installation and matches the value returned by the GfDataDir function, whereas the localdir directory is calculated at run-time by Speed Dreams and matches the value returned by the GfLocalDir function.
  • Cars are stored into <datadir>/cars/models/<directory>.
  • Tracks are stored into <datadir>/tracks/<category>/<directory>.
  • Drivers are stored into <localdir>/drivers/<category>/<index>. Since Speed Dreams refers to drivers by their indexes, once extracted it will rename directory to the next integer index available in the directory.
• revision: an integer that defines the revision of an asset. This value is used as a basic mechanism to inform users about updates. The download manager shall write the currently downloaded revision into a file called .revision inside the asset directory.

TODO: future releases will also store cars and tracks into localdir instead of datadir.

Asset file structure

An asset must be defined as a ZIP file with the following tree structure:

<directory>/
├── <file 0>
├── <file 1>
├── ...
└── <file n>

Where directory is the directory name defined by the directory key from the assets database. Any files on the same level as the top-level directory shall be ignored.

When an asset is downloaded or updated, the original directory is recursively removed and replaced with the contents of the ZIP file, if it exists.

Design and implementation

Source files

The following files were created inside the src/modules/userinterface/legacymenu/mainscreens/ directory to implement the download manager:

• asset.cpp
• asset.h
• assets.cpp
• assets.h
• assets.h
• downloadservers.cpp
• downloadservers.h
• downloadsmenu.cpp
• downloadsmenu.h
• entry.cpp
• entry.h
• hash.cpp
• hash.h
• repomenu.cpp
• repomenu.h
• sha256.cpp
• sha256.h
• sink.cpp
• sink.h
• thumbnail.cpp
• thumbnail.h
• unzip.cpp
• unzip.h
• writebuf.cpp
• writebuf.h
• writefile.cpp
• writefile.h

Main classes

The following classes define the GUI menus related to the download manager:

DownloadsMenu

downloadsmenu.cpp and downloadsmenu.h implement the DownloadsMenu class, which holds the main logic for the download manager. This class relies on the rest of the files listed above to perform the following tasks:

• Download files to memory.
• Download files to the filesystem.
• Display thumbnails and other GUI elements.
• Unzipping downloaded assets.
• Calculate and verify hashes for downloaded assets.
• Fetch files from a web server.
• Fetch one or more assets databases and parse them.
• Keep track of the state for each asset (e.g.: fetching thumbnail, downloaded, updates available, etc.).
• Organize assets into pages.

DownloadsMenu::recompute is called by the event loop handler so that downloads can be updated. The GfEventLoop was modified so that the event loop handler would call the user-defined callback with an additional parameter: the maximum number of milliseconds the user-defined callback can spend before returning to the caller.

This allows to update ongoing downloads by calling curl_multi_poll without blocking the main thread, thus removing the need for a separate thread.

RepoMenu

repomenu.cpp and repomenu.h implement the RepoMenu class, which lists the repositories defined by users into a scroll list and allows them to add or remove repositories. This menu can be only entered from the Configure button inside the downloads menu.

When this menu is exited, the downloads menu shall fetch the assets databases again, based on the new configuration. However, assets still not completely downloaded shall not be removed from the downloads menu.

Auxiliary classes

Asset

asset.cpp and asset.h implement the Asset class, which is the binary representation of a parsed entry from the assets database.

Assets

assets.cpp and assets.h implement the Assets class, a thin abstraction layer of a std::vector. The Assets class shall read a JSON file and parse the different arrays inside it to parse each asset individually by calling Asset::parse. Then, a std::vector of Asset is constructed, that callers can retrieve via a read-only reference.

Thumbnail

thumbnail.cpp and thumbnail.h implement the thumbnail class, which defines one of the visible entries as shown on the downloads menu.

entry

entry.cpp and entry.h implement struct entry, a relatively simple data structure containing information about an asset, such as the location of its data and thumbnail in the filesystem, as well as its state.

Whereas the number of thumbnail objects is fixed and determined by the THUMBNAILS enumerator on downloadsmenu.cpp, every asset defines its own entry object, and might or might not be shown at a given moment, as this depend on the filter settings and the active page.

sink

sink.cpp and sink.h implement the sink class, which is meant to be derived to define how fetched files should be downloaded. downloadsmenu.cpp would make use of pointers to sink objects so as to remain agnostic to how files are actually downloaded, as this depends on the operation itself:

• Assets databases are downloaded to memory so that cJSON can parse them via the writebuf class.
• Thumbnails and assets are downloaded to files via the writefile class.

sink defines a maximum size for a given file, so that there is no risk for a download operation to cause resource exhaustion.

writebuf

writebuf.cpp and writebuf.h implement the writebuf class, derived from sink, which implements file downloads to memory.

writefile

writefile.cpp and writefile.h implement the writefile class, derived from sink, which implements file downloads to the filesystem.

unzip

unzip.cpp and unzip.h implement the unzip class, which is a thin, high-level abstraction layer on top of the minizip library. As its name suggests, the unzip class is responsible for taking a path to a ZIP file as input and a destination path where its contents shall be written.

hash

hash.cpp and hash.h implement the hash abstract class, which defines a common interface for all hash functions to implement.

sha256

sha256.cpp and sha256.h implement the sha256 class, derived from the hash abstract class, which implements the interface to run the SHA256 hash function.

Auxiliary source files

downloadservers.cpp

This file, together with downloadservers.h, defines functions to read and write the databases configuration file located on config/downloadservers.xml.

Future improvements

• Perform hash calculation asynchronously i.e., update every time a chunk is received from the network.
• Perform unzip operations asynchronously so as to avoid blocking the main thread.