Zola Surreal DB site cache

• Publishes all of your site's contents and relevant metadata into a SurrealDB SQL file.
• Serves this file from your site.
• Loads an in-memory SurrealDB database on the page.
• Enables full-text search over all pages.
• Supports the taxonomy structure and metadata defined in the front matter.

---

Overview

1. Understanding SurrealDB and Browser Compatibility
2. Generating a SurrealDB SQL File During Site Build
3. Serving the SQL File from Your Site
4. Loading SurrealDB in the Browser
5. Implementing Full-Text Search and Taxonomy Support
6. Putting It All Together

---

1. Understanding SurrealDB and Browser Compatibility

What is SurrealDB?

SurrealDB is a scalable, distributed database designed to be flexible and efficient. It's capable of handling traditional relational data, document data, and graph data, and it provides an SQL-like query language.

Browser Compatibility as of January 2025

As of January 2025, SurrealDB has made significant strides in terms of client-side capabilities. With the advancements in WebAssembly (WASM) and web technologies, it's now possible to run SurrealDB in-memory in the browser using WASM.

Key Points:

• WebAssembly Support: SurrealDB provides a WASM build that can be executed in modern browsers, allowing you to run an in-memory instance of the database client-side.
• Client-Side Operations: This enables complex data querying and manipulation directly within the user's browser without the need for server-side operations.

---

2. Generating a SurrealDB SQL File During Site Build

To populate the in-memory database in the browser, we'll generate a SurrealDB SQL file containing all the site content and metadata.

2.1. Extracting Site Content and Metadata

We'll use Zola's data directory and global variables to access all pages, sections, taxonomies, and their metadata.

2.2. Creating a Template to Generate the SQL File

We'll create a template that iterates over all the pages and constructs SQL CREATE and INSERT statements.

Create a Template File

• Path: templates/surrealdb_export.sql
• Content:

{% raw %}
-- SurrealDB SQL Export
-- Generated on {{ now() | date(format="%Y-%m-%d %H:%M:%S") }}

-- Drop existing tables (if any)
REMOVE TABLE page;
REMOVE TABLE section;
REMOVE TABLE taxonomy_term;
REMOVE TABLE taxonomy_item;

-- Create tables
DEFINE TABLE page SCHEMAFULL
    PERMISSIONS FOR ALL NONE;

DEFINE TABLE section SCHEMAFULL
    PERMISSIONS FOR ALL NONE;

DEFINE TABLE taxonomy_term SCHEMAFULL
    PERMISSIONS FOR ALL NONE;

DEFINE TABLE taxonomy_item SCHEMAFULL
    PERMISSIONS FOR ALL NONE;

-- Insert pages
{% for page in pages %}
CREATE page:{{ page.slug }} CONTENT {
    title: {{ page.title | json_encode() }},
    slug: {{ page.slug | json_encode() }},
    path: {{ page.permalink | json_encode() }},
    content: {{ page.content | json_encode() }},
    summary: {{ page.summary | json_encode() }},
    date: {{ page.date | date(format="%Y-%m-%dT%H:%M:%SZ") | json_encode() }},
    taxonomies: [
        {% for key, terms in page.taxonomies %}
            { name: {{ key | json_encode() }}, terms: [ {{ terms | map(attribute="name") | join(", ") | json_encode() }} ] },
        {% endfor %}
    ],
    metadata: {{ page.extra | json_encode() }}
};

{% endfor %}

-- Insert sections
{% for section in sections %}
CREATE section:{{ section.slug }} CONTENT {
    title: {{ section.title | json_encode() }},
    slug: {{ section.slug | json_encode() }},
    path: {{ section.permalink | json_encode() }},
    content: {{ section.content | json_encode() }},
    summary: {{ section.summary | json_encode() }},
    date: {{ section.date | date(format="%Y-%m-%dT%H:%M:%SZ") | json_encode() }},
    metadata: {{ section.extra | json_encode() }}
};

{% endfor %}

-- Insert taxonomy terms
{% for kind, terms in taxonomies %}
    {% for term in terms %}
CREATE taxonomy_term:{{ term.name | slugify }} CONTENT {
    kind: {{ kind | json_encode() }},
    name: {{ term.name | json_encode() }},
    slug: {{ term.slug | json_encode() }},
    path: {{ term.permalink | json_encode() }},
    pages: [ {% for p in term.pages %} page:{{ p.slug }}, {% endfor %} ]
};
    {% endfor %}
{% endfor %}

-- Relationships between pages and taxonomies can be defined here if needed

{% endraw %}

Explanation:

• We're using Tera templating within a .sql file to generate the SQL statements.
• We loop over all pages, sections, and taxonomies to create SQL CREATE statements.
• JSON data is properly encoded using json_encode() to ensure compatibility.

2.3. Configuring the Template for Output

We need to ensure Zola processes our template and outputs the generated SQL file.

Adjust config.toml

Add an entry to extra to indicate where to output the SQL file.

[extra]
surrealdb_sql_output = "static/surrealdb_export.sql"

Alternatively, you can hardcode the output path in your build process.

2.4. Create a Build Script to Generate the SQL File

We'll use a build script to render the template.

Create generate_surrealdb_sql.sh

#!/bin/bash

# Ensure the script exits if any command fails
set -e

# Render the SQL template
zola build -o /tmp/zola_build_sql --template-only
# Copy the rendered file to the desired location
cp /tmp/zola_build_sql/surrealdb_export.sql static/
# Clean up temporary build directory
rm -rf /tmp/zola_build_sql

echo "SurrealDB SQL export generated successfully."

Explanation:

• We use zola build with --template-only to render only the templates without generating the full site.
• The output is temporarily stored in /tmp/zola_build_sql.
• We copy the rendered surrealdb_export.sql to the static/ directory so it gets served by the site.
• We clean up the temporary build directory afterward.

Make the Script Executable

chmod +x generate_surrealdb_sql.sh

2.5. Update Your Build Process

Ensure that the generate_surrealdb_sql.sh script runs before the site is built.

Locally

./generate_surrealdb_sql.sh && zola build

On Netlify

Update your netlify.toml or build command to include the script.

[build]
  publish = "public"
  command = "./generate_surrealdb_sql.sh && zola build"

---

3. Serving the SQL File from Your Site

By placing surrealdb_export.sql in the static/ directory, Zola will serve it at the root of your site.

• URL: https://yourdomain.com/surrealdb_export.sql

Ensure that it's accessible and not blocked by any server rules.

---

4. Loading SurrealDB in the Browser

4.1. Using the SurrealDB WebAssembly Build

As of January 2025, SurrealDB provides a WebAssembly (WASM) build that can be run in the browser.

Including SurrealDB WASM in Your Site

• Download the SurrealDB WASM and JavaScript bindings:

  • surrealdb.wasm
  • surrealdb.js

• Place them in your static/js/ directory.

4.2. Initializing SurrealDB in the Browser

<script>
  // Load the SurrealDB WASM module
  Surreal.initWasm({
    path: '/js/surrealdb.wasm'
  }).then(async () => {
    // Create a new in-memory SurrealDB instance
    const db = new Surreal();

    // Sign in as a root user (authentication can be bypassed in-memory)
    await db.signin({ user: 'root', pass: 'root' });

    // Select a namespace and database
    await db.use({ ns: 'myapp', db: 'mydb' });

    // Fetch the SQL file
    const response = await fetch('/surrealdb_export.sql');
    const sql = await response.text();

    // Execute the SQL statements to populate the database
    await db.query(sql);

    // Now the database is ready for use
    // You can perform queries, full-text search, etc.

    // Example query
    const pages = await db.select('page');

    console.log('Pages:', pages);

    // Implement search functionality and other features as needed

  }).catch((e) => {
    console.error('Error initializing SurrealDB:', e);
  });
</script>

Explanation:

• Surreal.initWasm: Initializes the WASM module.
• db.signin: Signs into the database (for in-memory, the credentials can be default).
• db.use: Selects the namespace and database.
• Fetching and Executing SQL:
  • We fetch the SQL file generated during the build.
  • Execute it using db.query(sql), which runs all the CREATE and INSERT statements to populate the database.

---

5. Implementing Full-Text Search and Taxonomy Support

With the database populated, you can perform complex queries, including full-text search and navigating taxonomies.

5.1. Full-Text Search

SurrealDB supports full-text search indexes.

Define Full-Text Search Indexes in the SQL

Modify your SQL template to include index definitions.

-- Define full-text search index on page content
DEFINE INDEX page_content_ft ON TABLE page FIELDS content SEARCH ANALYZER bm25 TOKENIZERS blank;

-- Similarly, you can define indexes on other fields as needed

Note: Ensure that your version of SurrealDB supports the specific index definitions. Syntax may vary; refer to the latest documentation.

Performing a Full-Text Search in JavaScript

// Function to perform a full-text search
async function searchPages(query) {
  // Use the 'search' function in your SurrealQL query
  const results = await db.query(`
    SELECT * FROM page WHERE search(content, $terms)
  `, { terms: query });

  return results[0].result;
}

// Example usage
const searchResults = await searchPages('your search terms');
console.log('Search Results:', searchResults);

5.2. Taxonomy Structure and Metadata

You can query pages by taxonomy terms.

Querying Pages by Taxonomy Term

async function getPagesByTaxonomy(termName) {
  const results = await db.query(`
    SELECT * FROM page WHERE $term INSIDE taxonomies[*].terms
  `, { term: termName });

  return results[0].result;
}

// Example usage
const pages = await getPagesByTaxonomy('tech');
console.log('Pages in Tech taxonomy:', pages);

Explanation:

• taxonomies[*].terms: Accesses all terms in the taxonomies array within each page.
• $term INSIDE terms: Checks if the term is present.

5.3. Accessing Metadata Defined in Front Matter

The metadata field in each page or section contains the extra data from the front matter.

async function getPageMetadata(slug) {
  const page = await db.select(`page:${slug}`);
  return page.metadata;
}

// Example usage
const metadata = await getPageMetadata('my-page-slug');
console.log('Page Metadata:', metadata);

---

6. Putting It All Together

6.1. Search Interface

Create a search input field in your HTML.

<input type="text" id="search-input" placeholder="Search...">
<div id="search-results"></div>

6.2. JavaScript to Handle Search

document.getElementById('search-input').addEventListener('input', async (event) => {
  const query = event.target.value;
  
  if (query.length > 2) { // Start searching after 3 characters
    const results = await searchPages(query);
    displaySearchResults(results);
  } else {
    clearSearchResults();
  }
});

function displaySearchResults(results) {
  const resultsDiv = document.getElementById('search-results');
  resultsDiv.innerHTML = '';

  results.forEach((page) => {
    const pageLink = document.createElement('a');
    pageLink.href = page.path;
    pageLink.textContent = page.title;
    resultsDiv.appendChild(pageLink);
  });
}

function clearSearchResults() {
  const resultsDiv = document.getElementById('search-results');
  resultsDiv.innerHTML = '';
}

6.3. Handling Taxonomy Navigation

// Example function to list all taxonomy terms
async function listTaxonomyTerms(kind) {
  const results = await db.query(`
    SELECT * FROM taxonomy_term WHERE kind = $kind
  `, { kind });

  return results[0].result;
}

// Display taxonomy terms
const terms = await listTaxonomyTerms('tags');
terms.forEach((term) => {
  console.log('Term:', term.name);
});

---

Considerations and Best Practices

Data Size and Performance

• Data Size: Loading all site content into the browser can be heavy, especially for large sites.
• Optimization: Consider limiting the amount of data, or implement lazy loading strategies.

Security

• Client-Side Data Exposure: All data loaded into the client is accessible to users. Avoid including sensitive information.
• Data Sanitization: Ensure all data is properly sanitized to prevent XSS attacks.

Browser Compatibility

• WASM Support: Ensure your target audience uses browsers that support WebAssembly.
• Fallbacks: Provide fallback mechanisms or messages for unsupported browsers.

Licensing and Permissions

• SurrealDB License: Ensure compliance with SurrealDB's licensing terms when using it client-side.
• Attribution: Provide necessary attributions if required.

---

Alternative Solutions

If using SurrealDB client-side becomes challenging due to data size, performance, or compatibility, consider these alternatives:

Lunr.js

• A lightweight full-text search library for the browser.
• Indexes can be generated during the build process.
• Suitable for static sites.

Elasticlunr.js

• Similar to Lunr.js but with additional features.

Algolia

• Provides hosted search solutions.
• Can be integrated with static sites via APIs.

Stork Search

• A WASM-based full-text search engine designed for static sites.
• Can generate search indexes at build time.

---

Conclusion

By generating a SurrealDB SQL file during your site's build process and leveraging the in-browser capabilities of SurrealDB with WebAssembly, you can create a powerful and dynamic search experience on your static site.

• Benefits:
  • Leverages the power of SurrealDB's querying and indexing.
  • Provides a rich, interactive experience without server-side dependencies.

---

Next Steps

• Implement the steps above in your project.
• Test thoroughly across different browsers and devices.
• Monitor performance and optimize as needed.

---

If you need further assistance with any of these steps or have additional questions, feel free to ask! I'm here to help you create the best possible experience for your site.