Data Dumps

ListenBrainz provides data dumps that you can import into your own server or use for other purposes. The full data dumps are created twice a month and the incremental data dumps twice a week. Each dump contains a number of different files. Depending on your use cases, you may or may not require all of them.

We have a bunch of commands which may be useful in interacting with dumps during local development as well.

Dump mirrors

See the ListenBrainz data page for information about where to download the data dumps from.

File Descriptions

A ListenBrainz data dump consists of three archives:

  1. listenbrainz-public-dump.tar.xz

  2. listenbrainz-listens-dump.tar.xz

  3. listenbrainz-listens-dump-spark.tar.xz

listenbrainz-public-dump.tar.xz

This file contains information about ListenBrainz users and statistics derived from listens submitted to ListenBrainz calculated from users, artists, recordings etc.

listenbrainz-listens-dump.tar.xz

This is the core ListenBrainz data dump. This file contains all the listens submitted to ListenBrainz by its users.

listenbrainz-listens-dump-spark.tar.xz

This is also a dump of the core ListenBrainz listen data. These dumps are made for consumption by the ListenBrainz Apache Spark cluster, formatting all listens into monthly JSON files that can easily be loaded into dataframes.

Structure of the listens dump

The ListenBrainz listen dump consists of listens broken down by year and month. At the top level there are directories for each of the year for which we have data. Inside each year there are listens files with month number as its name:

  1. listenbrainz-listens-dump-183-20200727-001004-full/listens/2005/1.listens

  2. listenbrainz-listens-dump-183-20200727-001004-full/listens/2005/2.listens

  3. listenbrainz-listens-dump-183-20200727-001004-full/listens/2005/3.listens

  4. listenbrainz-listens-dump-183-20200727-001004-full/listens/2005/4.listens

  5. listenbrainz-listens-dump-183-20200727-001004-full/listens/2005/5.listens

Each of the .listens files contains one JSON document per line – each of the JSON documents is one listen, formatted in the standard listens format.

Incremental dumps

ListenBrainz provides incremental data dumps that you can use to keep up to date with the ListenBrainz dataset without needing to download the full dumps everytime. These dumps have the same structure as the corresponding full dumps, but only contain data that has been submitted since the creation of the previous dump. We create incremental data dumps daily.

The basic idea here is that dumps create a linear timeline of the dataset based on the time of submission of data. In order to use the incremental dumps, you must start with the latest full dump and then, applying all incremental dumps since will give you the latest data. The series is consistent, if you take a full dump and apply all incremental dumps since that full dump until the next full dump, you will have all data as the next full dump.

Deleted listens present a tricky problem in this setup, since they are not included in the incremental dumps. To get a fully accurate list of listens, with deleted listens removed, you’ll need to re-import a full dump.