Step 4: Uploading your own data

In the previous step, you created a new project by attaching an existing data set.

In this step of the tutorial, we will create a new project, and create a new data set by uploading your own data to the platform.

Free trial accounts do not support uploading your own data

This part of the tutorial illustrates how you can upload your own data using the REST API, but this is not supported in a free trial account.

Get in touch if you want to start uploading your own data, and you are using a free account.

Step 4.1: Obtaining the data

In this tutorial, you create a new time series data set with 5 months of taxi fare data in New York City.

A time series data set is defined using one or more GeoJSON files containing the areas the time series data is defined for, and CSV files with the temporal data records that reference the GeoJSON shapes.

If you are unfamiliar with adding own data to the platform, it is highly suggested to follow this tutorial first.

For this tutorial download the following 4 files:

Download these 4 files and place them in a single folder.

Step 4.2: Using the API to upload the data.

In this step, you use the generated TypeScript API to perform following steps:

  1. Obtain a JWT authentication token as before.

  2. Create a new project

  3. Create a new Mapbox background layer

  4. Attach the background layer to the project

  5. Create a new time series data set

  6. Define the GeoJSON properties for the data set

  7. Upload the (gzipped) GeoJSON file with the New York city taxi zones

  8. Define the time series data properties

  9. Upload the (gzipped) CSV file with the New York city taxi trip fares data

  10. Attach the new data set to the project

The below code is self-explanatory. Copy-paste and save it in a file src/App5.ts. As before, adjust the userName and password. In addition, also adjust the path referencing the folder where you saved the 4 files you downloaded in Step 4.1.

App5.ts
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
import {
  AuthenticationJWTTokenApi,
  BackgroundLayersAndAreasOfInterestMapdataApi,
  CreatePublicDataSetRequest,
  CreatePublicMapboxMapDataRequest,
  DataSetsApi,
  HttpBearerAuth,
  ProjectResourcesDataSetsApi,
  ProjectResourcesMapdataApi,
  ProjectsApi,
} from "./gen/api";
import { createReadStream } from "fs";
import TypeEnumMapboxTypes = CreatePublicMapboxMapDataRequest.TypeEnum;
import TypeEnumDataSet = CreatePublicDataSetRequest.TypeEnum;

const userName = "replace with your API user name";
const password = "replace with your API user password";
const path = "/change/this/path/to/where/you/saved/the/files/";

async function getJWTToken() {
  const authenticationApi = new AuthenticationJWTTokenApi();

  return await authenticationApi
    .obtainToken({
      userName: userName,
      password: password,
    })
    .then((r) => {
      return r.body.jwtToken;
    })
    .catch((reason) => {
      console.log(reason);
      return null;
    });
}

getJWTToken().then(async (token) => {
  console.log("Got access token : " + token);

  if (token == null) {
    return;
  }

  const authentication = new HttpBearerAuth();
  authentication.accessToken = token;

  // Creating a project
  console.log("Creating a new project");
  const projectsApi = new ProjectsApi();
  projectsApi.setDefaultAuthentication(authentication);
  const project = (
    await projectsApi.createProject({
      name: "New York Taxi",
      description: "Project created with the REST API",
    })
  ).body;

  // Creating a Mapbox background layer and adding it to the project
  const mapDataApi = new BackgroundLayersAndAreasOfInterestMapdataApi();
  mapDataApi.setDefaultAuthentication(authentication);
  console.log("Creating a Mapbox light background layer");
  const backgroundLayer = (
    await mapDataApi.createMapboxBackgroundLayer({
      type: TypeEnumMapboxTypes.Light,
    })
  ).body;

  console.log("Attaching the Mapbox background layer to the project");
  const projectMapDataApi = new ProjectResourcesMapdataApi();
  projectMapDataApi.setDefaultAuthentication(authentication);
  await projectMapDataApi.addMapDataToProject(project.id, {
    mapDataId: backgroundLayer.id,
  });

  // Creating and uploading a new time series data set and attaching it to the project
  console.log("Creating a new data set");
  const dataSetsApi = new DataSetsApi();
  dataSetsApi.setDefaultAuthentication(authentication);
  const dataSet = (
    await dataSetsApi.createDataSet({
      name: "New York Taxi",
      description: "Data set created with the API",
      type: TypeEnumDataSet.TimeSeriesData,
      assetsName: "zones",
    })
  ).body;

  console.log("Defining the GeoJSON structure using a properties file");
  await dataSetsApi.uploadTimeSeriesMetadataDefinitionFile(
    dataSet.id,
    createReadStream(path + "geojsonproperties.csv")
  );

  console.log("Uploading the GeoJSON file");
  await dataSetsApi.uploadGeoJSONShapeFile(
    dataSet.id,
    createReadStream(path + "NYC_Taxi_Zones.geojson.gz")
  );

  console.log("Defining the time series data structure");
  await dataSetsApi.uploadDataDefinitionFile(
    dataSet.id,
    createReadStream(path + "dataproperties.csv")
  );

  console.log("Uploading the time series data");
  await dataSetsApi.uploadData(
    dataSet.id,
    createReadStream(path + "taxidata.csv.gz")
  );

  console.log("Attaching the data set to the new project");
  let projectDataSetsAPI = new ProjectResourcesDataSetsApi();
  projectDataSetsAPI.setDefaultAuthentication(authentication);
  await projectDataSetsAPI.addDataSetToProject(project.id, {
    dataSetId: dataSet.id,
  });
});

Execute this script as before:

npx tsc
node dist/App5.js

Verify that a new project is created with one data set and one background layer attached. Click on the data set card and monitor the processing. Initially, the data set processing job is queued.

Once the card indicates that all data is processed, you can open the Visual analytics page for this project and verify that the taxi zones are shown on the map. You can now analyze the data, by changing for example the color map and the value range as usual.

Figure 1. Default visual analytics view of the project and data set created in Step 4.2.
Figure 2. Visual analytics view after changing the color map and adjusting the value range slider to focus on the main taxi fare range.

That’s a wrap!

Congrats! With just about 100 lines of code, you have used a script to automatically create a new project, background layer, and upload your own time series data.

In addition to project creation, data upload, and dashboard creation, like you have done in this tutorial, you can also use the REST API to:

  • Perform data queries to obtain for example a trend line or a property distribution.

  • Create bookmarks to configure the (default) state of the different analytics pages.

  • Create shared links that enable to share or embed your projects.

  • And much more.

Next part

Go to the next part: Next steps