DocLite

A powerful PHP NoSQL document store built on top of SQLite.

Table of contents

About DocLite Why DocLite? Getting Started The Database Creating a memory database Creating a file database Error Handling & Logging Import and export data Importing Data Exporting data Advanced options Get DocLite version Optimize database Set synchronization mode Set rollback journal mode Collections About Collections Obtain a collection Create a document Save a document Retrieve a document Map a document to a custom class Delete a document Query a collection Find single document by values Find all matching documents by values Find all documents in collection Advanced queries Query operators Join collections Caching results Index a collection Unique indexes Delete a collection Collection transactions Full text search Documents About Documents Getting and setting document data Mapping document fields to objects Document Unique Id Saving a document Deleting a document Document validation Other info Symfony integration Licensing Bugs, issues Contact the author

About DocLite

DocLite is a powerful NoSQL document store for PHP built on top of SQLite. It uses the PHP PDO SQLite library to access a SQLite database and automatically manage documents organized in to named collections, which are stored as JSON.

DocLite takes advantage of the SQLite JSON1 extension (this is usually bundled in to the libsqlite included with your PHP distribution, so you probably already have it) to store, parse, index and query JSON documents - giving you the power and flexibility of a fully transactional and ACID compliant NoSQL solution, yet contained within the local file system. No need for more complex systems like Mongo, CouchDB or Elasticsearch when your requirements are slim. No need for any external dependencies, just PHP with PDO SQLite enabled.

DocLite provides a simple, intuitive, flexible and powerful PHP library that you can learn, install and start using in minutes.

Why DocLite?

DocLite lends itself well to a variety of use cases, including but not limited to:

Agile development and rapid prototyping while your requirements are evolving.

Powerful, self-contained NoSQL database for small to medium websites or applications, such as blogs, business website, CMS, CRM or forums.

A fast and reliable cache for data retrieved from remote databases, APIs or servers. Process your data in to documents, save in DocLite and easily query and filter your data as needed.

Robust, performant, ACID compliant replacement for weaker, slower, flat-file data stores utilizing JSON, XML or YAML.

Application database for web apps installed and run on a local environment.

Database for microservices and middleware.

Fast in-memory database for data processing or machine learning algorithms.

Broadly speaking, DocLite is suitable for the same uses cases as the underlying SQLite engine it is built on, but where you desire a NoSQL solution.

Getting Started

System requirements

PHP 7.4 or above

With PDO SQLite enabled, built against libsqlite ≥ 3.18.0 with JSON1 extension.

(on most systems, if you're running PHP 7.4 you probably already meet the second requirement)

Installation

Install with Composer

composer require dwgebler/doclite

Usage Overview

DocLite provides both a FileDatabase and MemoryDatabase implementation. To create or open an existing database, simply create a Database object, specifying the file path if using a FileDatabase.

If your FileDatabase does not exist, it will be created (ensure your script has the appropriate write permissions). This will include creating any parent directories as required.

If you specify an existing directory without a filename, a default filename data.db will be used.

use GeblerDoclite{FileDatabase, MemoryDatabase}; // To create or open an existing file database. $db = new FileDatabase('/path/to/db'); // To open an existing file database in read-only mode. $db = new FileDatabase('/path/to/existing/db', true); // To create a new in-memory database. $db = new MemoryDatabase();

Once you have opened a database, you can obtain a document Collection which will be automatically created if it does not exist.

$users = $db->collection("user");

The Collection object can then be used to retrieve, create and manipulate documents.

// Create a new User in the collection $user = $users->get(); // Get the automatically generated document ID $id = $user->getId(); // Set properties by magic set* methods $user->setUsername("dwgebler"); $user->setRole("admin"); $user->setPassword(password_hash("admin", PASSWORD_DEFAULT)); $user->setCreated(new DateTimeImmutable); // Update the user in the collection $user->save(); // Retrieve this user later on by ID $user = $users->get($id); // Or search for a user by any field $user = $users->findOneBy(["username" => "dwgebler"]);

In the example above, $user is an instance of a DocLite Document, but you can also hydrate objects of your own custom classes from a collection.

class CustomUser { private $id; private $username; private $password; public function getId() {...} public function setId($id) {...} public function getUsername() {...} public function setUsername($username) {...} } // Retrieve a previously created user and map the result on to a CustomUser object. // You can also pass a null ID as the first parameter to create a new CustomUser. $user = $users->get($id, CustomUser::class); // $user is now an instance of CustomUser and can be saved through the Collection. $users->save($user);

To learn more about the Collection object including how to query a document store, please read the full documentation below.

The Database

DocLite is built on top of SQLite 3 and supports two types of database; file and memory. The corresponding classes are FileDatabase and MemoryDatabase.

Creating a memory database

MemoryDatabase is stored in volatile memory and is therefore ephemeral for the lifetime of your application scripts. Its constructor takes optional parameters:

a boolean flag indicating whether to enable full text search features (defaults to false) - this feature requires SQLite to have been compiled with the FTS5 extension. an integer representing the maximum connection timeout in seconds (defaults to 1) which is how long the connection should wait if the underlying SQLite database is locked. A PSR-3 compatible logger instance (defaults to null).

```php use GeblerDocliteMemoryDatabase; $db = new MemoryDatabase(); // With full text search enabled and a 2-second connection timeout $logger = new MonologLogger('my-logger'); $db = new MemoryDatabase(true, 2, $logger);

Creating a file database

FileDatabase constructor takes one mandatory and then some optional parameters; only the file or directory path to a new or existing database is required.

Optional parameters are:

a boolean flag indicating whether the database should be opened in read-only mode, which defaults to false. a boolean flag indicating whether to enable full text search features (defaults to false) - this feature requires SQLite to have been compiled with the FTS5 extension. an integer representing the maximum connection timeout in seconds (defaults to 1) which is how long the connection should wait if the underlying SQLite database is locked. A PSR-3 Logger instance to use for logging database events.

The path supplied to FileDatabase can be a relative or absolute path which is any of:

An existing directory with read and write access. A non-existent file in a directory with read-write access. An existing database in a directory with read-write or read-only access (read-only mode). A non-existing directory path which your script has permission to create.

If no file name is specified, a default file name data.db will be used for the underlying database.

use GeblerDocliteFileDatabase; // Open a new database $db = new FileDatabase('./data/mydb.db'); // Open an existing database in read-only mode $db = new FileDatabase('./data/mydb.db', true); // Open a new database called data.db in existing directory /home/data $db = new FileDatabase('/home/data'); // All options - path, read-only mode, full text search, connection timeout and logger $logger = new MonologLogger('mylogger'); $db = new FileDatabase('./data/mydb.db', false, true, 1, $logger); // Or, in PHP 8, named parameters: $db = new FileDatabase(path: './data/mydb.db', readOnly: true, ftsEnabled: true);

If you open a database in read-only mode, you will be able to retrieve documents from a collection, but you will not be able to save them or create new documents or collections. Attempting to do so will trigger an error.

It is good practice wrapping FileDatabase creation in a try-catch block. Initializing a FileDatabase may throw either an IOException (for errors relating to the file system) or a DatabaseException (for errors establishing the DB connection).

use GeblerDocliteExceptionIOException; use GeblerDocliteExceptionDatabaseException; use GeblerDocliteFileDatabase; try { $db = new FileDatabase('/path/to/db'); } catch (IOException $e) { var_dump($e->getMessage()); } catch (DatabaseException $e) { var_dump($e->getMessage()); }

Error Handling & Logging

To enable logging queries and parameters in the full and final SQL sent to the database, pass a PSR-3 logger instance in to you FileDatabase or MemoryDatabase either via the constructor or the $database->setLogger(LoggerInterface $logger) method at any time.

Then call $database->enableQueryLogging() to enable logging of all queries. These will be logged at the debug level.

Or call $database->enableSlowQueryLogging() to enable logging of queries that take longer than 500ms. These will be logged with a warning level.

As long as a LoggerInterface instance is set on a database, any exceptions will also be logged at the error level.

You can disable logging by calling $database->disableQueryLogging() or $database->disableSlowQueryLogging().

DocLite primarily throws a DatabaseException when any error occurs. This is true across the Database, Collection and Document types. A Database exception will include a message, an error code (see below), any underlying system exception if there was one (so normal Exception behaviour up to this point), plus any SQL query which was being executed (DocLite hides these from you during normal operation, of course, as it is a NoSQL solution, but they are useful for filing bug reports!), and an array of any relevant parameters - these may be things like a document ID, the document data, etc.

use GeblerDocliteExceptionDatabaseException; ... try { $user->setUsername("dwgebler"); $user->save(); } catch (DatabaseException $e) { var_dump($e->getMessage(), $e->getCode(), $e->getQuery(), $e->getParams()); }

A DatabaseException can occur on any Database, Collection or Document method which interacts with the underlying database.

Error codes are represented by public constants in the DatabaseException class.
The full list of error codes are as follows:

Constant	Meaning
ERR_COLLECTION_IN_TRANSACTION	Attempted to begin, rollback or commit on a collection while a transaction on a different collection was already in progress.
ERR_CONNECTION	Unable to connect to database
ERR_NO_SQLITE	PDO SQLite extension is not installed
ERR_NO_JSON1	SQLite does not have the JSON1 extension installed
ERR_NO_FTS5	FTS5 extension not installed
ERR_INVALID_COLLECTION	Invalid collection name
ERR_MISSING_ID_FIELD	Custom class for mapping a document does not have an ID field
ERR_INVALID_FIND_CRITERIA	Attempted to find a document by non-scalar value
ERR_INVALID_ID_FIELD	Specified unique ID field for custom class does not exist, nor does default
ERR_ID_CONFLICT	Multiple documents in the same collection have the same ID
ERR_CLASS_NOT_FOUND	Custom class name being used for a document does not exist
ERR_INVALID_UUID	Attempted to get the timestamp from an invalid UUID
ERR_QUERY	Error executing SQL query
ERR_READ_ONLY_MODE	Attempted a write operation on a read only database
ERR_INVALID_JSON_SCHEMA	Attempted to import an invalid JSON schema
ERR_INVALID_DATA	Data does not match loaded JSON schema
ERR_MAPPING_DATA	Unable to map document to class
ERR_IMPORT_DATA	Error importing data
ERR_IN_TRANSACTION	Attempting locking operation while in a transaction
ERR_INVALID_TABLE	Attempting to access invalid table
ERR_UNIQUE_CONSTRAINT	Attempting to insert a document with a unique field that already exists

Import and export data

DocLite can import data from and export data to JSON, YAML, XML and CSV files. For this purpose, the Database object provides two methods, import() and export().

Import or export operations on very large collections may exhaust memory. This feature will (probably) be improved and made more efficient for working with large data sets in future.

It is recommended to use JSON for exports you intend to reload in to a DocLite database. Support for other formats is experimental.

Importing Data

The data you want to import can be organized either in to files, where each file represents a collection of multiple documents, or a directory where each sub-directory represents a collection and contains a number of files each representing a single document.

import(string $path, string $format, int $mode)

format can be any of json, yaml, xml, or csv. This should also match the extension of the filename(s) containing your data.

When using the csv format, the first line of a CSV file is assumed to be a header line containing field names.

mode can be either of the constants Database::MODE_IMPORT_COLLECTIONS or Database::MODE_IMPORT_DOCUMENTS.

Collection names are inferred from the subdirectory or file names. For example, /path/to/collections/users.json will import to the users collection, as will a sub-directory /path/to/collections/users/ when importing a collection from multiple files.

// Create a new, empty database $db = new FileDatabase('/path/to/data.db'); // Import the contents of a directory where each file is a collection $db->import('/path/to/collections', 'json', Database::IMPORT_COLLECTIONS); // Import the contents of a directory where each sub directory is a collection // of files representing single documents. $db->import('/path/to/collections', 'json', Database::IMPORT_DOCUMENTS);

When you import documents in to a collection, any documents which have a unique ID matching an existing document in the database will overwrite that document. Otherwise, new documents will be created for any documents with an unmatched or missing ID.