Enable FileStore and DataStore for CKAN on CentOS6.8

This guide refers to the CKAN DataStore extension , and is based on my actual operation. If there is something wrong, I am happy to adopt your advice and correct it.

Tips:

The second section of enabling DataStore in this course is assumed that:

You install the CKAN6.8 from source on your CentOS system.

Your PostgreSQL is 9.0 or later.

You are able to use the

psql

command to connect to your database as a superuser

1 Enable FileStore and file uploads

When enabled, CKAN’s FileStore allows users to upload data files to CKAN resources, and to upload logo images for groups and organizations. Users will see an upload button when creating or updating a resource, group or organization.

New in version 2.2: Uploading logo images for groups and organizations was added in CKAN 2.2.

Changed in version 2.2: Previous versions of CKAN used to allow uploads to remote cloud hosting but we have simplified this to only allow local file uploads (see Migration from 2.1 to 2.2 for details on how to migrate). This is to give CKAN more control over the files and make access control possible.

Setup file uploads

To setup CKAN’s FileStore with local file storage:

（1）Create the directory where CKAN will store uploaded files:

# mkdir -p /var/lib/ckan/default

（2）Add the following line to your CKAN config file ( maybe

/etc/ckan/default/development.ini

) after the

[app:main]

line:

ckan.storage_path = /var/lib/ckan/default

（3）Set the permissions of your ckan.storage_path directory. For example if you’re running CKAN with Apache, then Apache’s user (apache on Centos) must have read, write and execute permissions for the ckan.storage_path:

# chown apache /var/lib/ckan/default
# chmod u+rwx /var/lib/ckan/default

（4）Restart your web server, for example to restart Apache:

# apachectl reload

2 Enable DataStore

Note：

The DataStore requires PostgreSQL 9.0 or later. This course is assumed that you have installed PostgreSQL 9.0 or later, and you are able to use the psql command to connect to your database as a superuser.

The CKAN DataStore extension provides an ad hoc database for storage of structured data from CKAN resources. Data can be pulled out of resource files and stored in the DataStore.

When a resource is added to the DataStore, you get:

Automatic data previews on the resource’s page, using the Data Explorer extension

The DataStore API: search, filter and update the data, without having to download and upload the entire data file

The DataStore is integrated into the CKAN API and authorization system.

Setting up the DataStore

（1） Enable the plugin

Add the datastore plugin to your CKAN config file( maybe

/etc/ckan/default/development.ini

) :

ckan.plugins = datastore

（2） Set-up the database

Warning：

Make sure that you follow the steps in Set Permissions below correctly. Wrong settings could lead to serious security issues.

The DataStore requires a separate PostgreSQL database to save the DataStore resources to.

1）List existing databases:

# su - postgres
-bash-4.1$ psql -l

Check that the encoding of databases is UTF8, if not internationalisation may be a problem. Since changing the encoding of PostgreSQL may mean deleting existing databases, it is suggested that this is fixed before continuing with the datastore setup.

2）Create users and databases

Tip：

If your CKAN database and DataStore databases are on different servers, then you need to create a new database user on the server where the DataStore database will be created. As in Installing CKAN from source we’ll name the database user

ckan\_default

:

-bash-4.1$ createuser -S -D -R -P -l ckan_default

Create a database_user called

datastore\_default

. This user will be given read-only access to your DataStore database in the Set Permissions step below:

-bash-4.1$ createuser -S -D -R -P -l datastore_default

Create the database (owned by

ckan_default

), which we’ll call

datastore\_default

:

-bash-4.1$ createdb -O ckan_default datastore_default -E utf-8

3）Set URLs

Now, uncomment the

ckan.datastore.write\_url

and

ckan.datastore.read\_url

lines in your CKAN config file ( maybe

/etc/ckan/default/development.ini

) and edit them if necessary, for example:

ckan.datastore.write_url = postgresql://ckan_default:pass@localhost/datastore_default
ckan.datastore.read_url = postgresql://datastore_default:pass@localhost/datastore_default

Replace pass with the passwords you created for your ckan_default and datastore_default database users.

4）Set permissions

Once the DataStore database and the users are created, the permissions on the DataStore and CKAN database have to be set. CKAN provides a paster command to help you correctly set these permissions.

Connect to your database server as the postgres superuser using:

# su - postgres
-bash-4.1$ psql

Then you can use this connection to set the permissions:

postgres=# paster --plugin=ckan datastore set-permissions -c /etc/ckan/default/development.ini

（3） Test the set-up

The DataStore is now set-up. To test the set-up, (re)start CKAN and run the following command to list all DataStore resources:

# service httpd restart
# curl -X GET "http://127.0.0.1:80/api/3/action/datastore_search?resource_id=_table_metadata"

This should return a JSON page without errors.

To test the whether the set-up allows writing, you can create a new DataStore resource. To do so, run the following command:

curl -X POST http://127.0.0.1:80/api/3/action/datastore_create -H "Authorization: {YOUR-API-KEY}" -d '{"resource": {"package_id": "{PACKAGE-ID}"}, "fields": [ {"id": "a"}, {"id": "b"} ], "records": [ { "a": 1, "b": "xyz"}, {"a": 2, "b": "zzz"} ]}'

Replace {YOUR-API-KEY} with a valid API key and {PACKAGE-ID} with the id of an existing CKAN dataset.

A table named after the resource id should have been created on your DataStore database. Visiting this URL should return a response from the DataStore with the records inserted above:

http://127.0.0.1:80/api/3/action/datastore_search?resource_id={RESOURCE_ID}

Replace {RESOURCE-ID} with the resource id that was returned as part of the response of the previous API call.

You can now delete the DataStore table with:

curl -X POST http://127.0.0.1:80/api/3/action/datastore_delete -H "Authorization: {YOUR-API-KEY}" -d '{"resource_id": "{RESOURCE-ID}"}'

To find out more about the DataStore API, see The DataStore API.