The anatomy of smart search in Joomla 5 Part 2: Creating a plugin I.

In the previous article, we got acquainted with the capabilities of the Joomla smart search component, talked about the parameters and configuration of scheduled indexing using CRON. Let's start creating the code for our own plugin.

List of resources

Before starting the technical part, I will mention some articles that directly address the main topic. As well as articles that generally cover the creation and/or updating of a plugin for the modern architecture of Joomla 4 / Joomla 5. Next, I will assume that the reader has read them and generally has an idea of how to make a working plugin for Joomla:

• Creating a Smart Search plugin - official Joomla documentation. It is for Joomla 3, but most of the provisions remained true for Joomla 4 / Joomla 5
• Developing a Smart Search Plugin an article from Joomla Community Magazine in 2012.
• The book Joomla Extensions Development by Nicholas Dionysopoulos that covers the development of Joomla! extensions under Joomla versions 4 and 5.
• The Database section on the new documentation portal manual.joomla.org - for Joomla 4 and Joomla 5. ## The technical part. Development of the Joomla 5 smart search plugin The smart search component works with data provider plugins, whose main task remains the same - to select data and give it to the component for indexing. But over time, reindexing tasks also fell into the plugin's area of responsibility. In the article, we will assume that we run content indexing manually from the admin panel. The work from the CLI is visually different, but its essence remains the same.

For experienced developers, I will say that the search plugin extends the \Joomla\Component\Finder\Administrator\Indexer\Adapter class, the class file is located in administrator/components/com_finder/src/Indexer/Adapter.php. Well, then they will figure it out for themselves. Also, as a sample, you can study the Joomla core smart search plugins - for articles, categories, contacts, tags, etc. - in the plugins/finder folder. I worked on a smart search plugin for JoomShopping (Joomla e-commerce component) and SW JProjects (your own Joomla extensions directory component with update server) components, so the class names and some nuances will be associated with them. I will show most of it using the example of JoomShopping. The solution to the issue of multilingualism is based on the example of SW JProjects.

The file structure of the smart search plugin

The file structure of the smart search plugin for Joomshopping does not differ from the typical one:

Joomla 5 smart searh plugin file structure

File services/provider.php

The file provider.php allows you to register a plugin in a Joomla DI container and allows you to access plugin methods from the outside using MVCFactory.

<?php

/**
 * @package     Joomla.Plugin
 * @subpackage  Finder.Wtjoomshoppingfinder
 *
 * @copyright   (C) 2023 Open Source Matters, Inc. <https://www.joomla.org>
 * @license     GNU General Public License version 2 or later; see LICENSE.txt
 */

\defined('_JEXEC') or die;

use Joomla\CMS\Extension\PluginInterface;
use Joomla\CMS\Factory;
use Joomla\CMS\Plugin\PluginHelper;
use Joomla\Database\DatabaseInterface;
use Joomla\DI\Container;
use Joomla\DI\ServiceProviderInterface;
use Joomla\Event\DispatcherInterface;
use Joomla\Plugin\Finder\Wtjoomshoppingfinder\Extension\Wtjoomshoppingfinder;

return new class () implements ServiceProviderInterface {
    /**
     * Registers the service provider with a DI container.
     *
     * @param   Container  $container  The DI container.
     *
     * @return  void
     *
     * @since   4.3.0
     */
    public function register(Container $container)
    {
        $container->set(
            PluginInterface::class,
            function (Container $container) {
                $plugin     = new Wtjoomshoppingfinder(
                    $container->get(DispatcherInterface::class),
                    (array) PluginHelper::getPlugin('finder', 'wtjoomshoppingfinder')
                );
                $plugin->setApplication(Factory::getApplication());

                // Our plugin uses DatabaseTrait, so the setDatabase() method appeared 
                // If it is not present, then we use only setApplication().
                $plugin->setDatabase($container->get(DatabaseInterface::class));

                return $plugin;
            }
        );
    }
};

Plugin class file

This is the file that contains the main working code of your plugin. It should be located in the src/Extension folder. In my case, the plugin class \Joomla\Plugin\Finder\Wtjoomshoppingfinder\Extension\Wtjoomshoppingfinder is in the file plugins/finder/wtjoomshoppingfinder/src/Extension/Wtjoomshoppingfinder.php. The namespace of the plugin is Joomla\Plugin\Finder\Wtjoomshoppingfinder\Extension.

There is a minimal set of class properties and methods required for operation (they are accessed, including by the parent Adapter class).

The minimum required properties of the class

• $extension - is the name of your component, which defines the type of your content. For example, com_content. In my case, this is com_jshopping.
• $context - is a unique identifier for the plugin, it sets the context of indexing, in which the plugin will be accessed. In fact, this is the name of the plugin class (element). In our case, Wtjoomshoppingfinder.
• $layout - is the name of the output layout for the search results element. This layout is used when displaying search results. For example, if the $layout parameter is set to article, then the default view mode will search for a layout file named default_article.php when you need to display a search result of this type. If such a file is not found, then a layout file with the name default_result.php will be used instead. The output layouts with HTML layout are located in components/com_finder/tmpl/search. However, we should place our layouts as overrides - in the html template folder - templates/YOUR_TEMPLATE/html/com_finder/search. In my case, I named the layout product, and the file is called default_product.php.
• $table - is the name of the table in the database that we are accessing to get data, for example, #__content. In my case, the main table with JoomShopping products is called #__jshopping_products.
• $state_field - is the name of the field in the database table that is responsible for whether the indexed element is published or not. By default, this field is called state. However, in the case of JoomShopping, this field is called product_publish.

<?php
// So far, only the namespaces used in the example are listed here.

use Joomla\Component\Finder\Administrator\Indexer\Adapter;
use Joomla\Event\SubscriberInterface;
use Joomla\Database\DatabaseAwareTrait;

\defined('_JEXEC') or die;

final class Wtjoomshoppingfinder extends Adapter implements SubscriberInterface
{
    // We want to use a setDatabase() method in provider.php and getDatabase() method in our plugin.
    // So let's...
    use DatabaseAwareTrait;

    /**
     * The unique identifier of the plugin. You can specify the name of the class.
     *
     * @var    string
     * @since  2.5
     */
    protected $context = 'Wtjoomshoppingfinder';

    /**
     * For which component are we indexing the content
     *
     * @var    string
     * @since  2.5
     */
    protected $extension = 'com_jshopping';

    /**
     *
     * The name of the suffix for the search results output sublayout.
     * If it is an "article", then the file name will be "default_article.php "
     *
     * @var    string
     * @since  2.5
     */
    protected $layout = 'product';

    /**
    * The type of content being indexed. The user can
    * search only among products, only among tags, only
     * among articles, etc.
     *
     * @var    string
     * @since  2.5
     */
    protected $type_title = 'Product';

    /**
     * A field in the database that stores the flag whether the item is published or not.
     * Default is "state"
     *
     * @var string
     * @since 1.0.0
     */
    protected $state_field = 'product_publish';

    /**
     * The name of the database table.
     *
     * @var    string
     * @since  2.5
     */
    protected $table = '#__jshopping_products';

    /**
     * Whether to load the plugin's language files when initializing the class.
     *
     * @var    boolean
     * @since  3.1
     */
    protected $autoloadLanguage = true;

    /**
     * Language tag for JoomShopping products. 
     * Non-standard class property, only we need
     * and only for JoomShopping.
     *
     * @var    string
     * @since  3.1
     */
    protected string $languageTag = '';

}

The minimum required methods of the class

• setup() : bool - is a method for pre-configuring the plugin, connecting libraries, etc. The method is called during reindexing (the reindex() method), on the onBeforeIndex event. The method must return true, otherwise indexing will be interrupted.
• index() : void - is the method to start indexing itself. It collects an object of the desired structure from raw SQL query data, which is then passed to the \Joomla\Component\Finder\Administrator\Indexer\Indexer class for indexing. The method is run for each indexed element. The method argument is $item - the result of a query to the database, formatted in the \Joomla\Component\Finder\Administrator\Indexer\Result class.
• getListQuery() : Joomla\Database\DatabaseQuery - is a method for getting a list of indexed items…

... and here we start to dive into the details, since the getListQuery() method is not really mandatory, despite the fact that both the documentation and most articles talk about it.

Any picture on the topic of "complex scheme" will do here.

Dive into the details. The data structure of the indexed element.

It's amazing how many times some information or idea sometimes passes by us in a circle before we notice and realize it! Many things, being in front of our eyes for more than one year, still do not reach awareness, and our attention focuses on them only after years of experience.

In connection with Joomla, for some reason, the vision does not immediately come that its components assume some kind of common architecture characteristic of Joomla (although this is an obvious fact). Including at the level of the database table structure. Let's look at some fields of the Joomla content table. I will make a reservation that specific column names are not so important to us (you can always query SELECT name AS title), how much is the data structure for one indexed element:

• id - autoincrement
• asset_id - the id of the entry in the #__assets table, where the access rights of groups and users are stored for each element of the site: articles, products, menus, modules, plugins, and everything else. Joomla uses the Access Control List (ACL) pattern.
• title - the element title
• language - the element language
• introtext - introductory text or a brief visible description of the element
• fulltext - the full text of the item, the full description of the product, etc.
• state - the logical flag responsible for the publication status: whether the item is published or not.
• catid - the ID of the item category. Joomla doesn't just have "site pages" like in other CMS. There are content entities (articles, contacts, products, etc.) that must belong to some categories.
• created- the date the item was created.
• access - access rights group id (unauthorized site users (guests), all, registered, etc.)
• metakey - meta keywords for the element. Yes, since 2009 they are not used by Google. But in Joomla, they historically remain, since this field is used in the similar articles module to search for actually similar articles using specified keywords.
• metadesc - the element meta description
• publish_up and publish_down - the date of the start of publication and de-publication of the element. This is more of an option, but it is found in many components.

If we compare the tables #__content (Joomla articles), #__contact_details (contact component), #__tags (Joomla tags), #__categories (Joomla category component), then we will find almost all the listed data types everywhere.

If the component for which smart search plugins are created followed the "Joomla way" and inherits its architecture, then you can do with a minimum of methods in the plugin class. If the developers decide not to look for easy ways and go their own way, then you will have to go the hard way, redefining almost all the methods of the Adapter class.

getListQuery() method

This method is called in 3 cases:

1. The getContentCount() method of the Adapter class is to get the number of indexed items (how many articles in total, how many products in total, etc.). Joomla Smart searh indexing process You can see the number of indexed items in debug mode.
2. The getItem($id) method of the Adapter class is to get a specific indexed element by its id. The getItem() method, in turn, is called in the reindex($id) method - during reindexing.
3. The getItems($offset, $limit, $query = null) method of the Adapter class is a method for getting a list of indexed elements. Offset and limit are set based on the component settings - how many indexed elements should be included in the "bundle". Joomla 5 smart search settings indexer batch size

Let's look at an example of implementation in Joomla core plugins:

<?php

// This is an example code from the Joomla smart search plugin for Joomla Articles.
use Joomla\Database\DatabaseQuery;

/**
     * Method to get the SQL query used to retrieve the list of content items.
     *
     * @param   mixed  $query  A DatabaseQuery object or null.
     *
     * @return  DatabaseQuery  A database object.
     *
     * @since   2.5
     */
    protected function getListQuery($query = null)
    {
        $db = $this->getDatabase();

        // Check if we can use the supplied SQL query.
        $query = $query instanceof DatabaseQuery ? $query : $db->getQuery(true)
            ->select('a.id, a.title, a.alias, a.introtext AS summary, a.fulltext AS body')
            ->select('a.images')
            ->select('a.state, a.catid, a.created AS start_date, a.created_by')
            ->select('a.created_by_alias, a.modified, a.modified_by, a.attribs AS params')
            ->select('a.metakey, a.metadesc, a.metadata, a.language, a.access, a.version, a.ordering')
            ->select('a.publish_up AS publish_start_date, a.publish_down AS publish_end_date')
            ->select('c.title AS category, c.published AS cat_state, c.access AS cat_access');

        // Handle the alias CASE WHEN portion of the query
        $case_when_item_alias = ' CASE WHEN ';
        $case_when_item_alias .= $query->charLength('a.alias', '!=', '0');
        $case_when_item_alias .= ' THEN ';
        $a_id = $query->castAsChar('a.id');
        $case_when_item_alias .= $query->concatenate([$a_id, 'a.alias'], ':');
        $case_when_item_alias .= ' ELSE ';
        $case_when_item_alias .= $a_id . ' END as slug';
        $query->select($case_when_item_alias);

        $case_when_category_alias = ' CASE WHEN ';
        $case_when_category_alias .= $query->charLength('c.alias', '!=', '0');
        $case_when_category_alias .= ' THEN ';
        $c_id = $query->castAsChar('c.id');
        $case_when_category_alias .= $query->concatenate([$c_id, 'c.alias'], ':');
        $case_when_category_alias .= ' ELSE ';
        $case_when_category_alias .= $c_id . ' END as catslug';
        $query->select($case_when_category_alias)

            ->select('u.name AS author')
            ->from('#__content AS a')
            ->join('LEFT', '#__categories AS c ON c.id = a.catid')
            ->join('LEFT', '#__users AS u ON u.id = a.created_by');

        return $query;
    }

The getListQuery() method returns a DatabaseQuery object, an object of the query constructor, where the name of the table and fields for selection are already specified. Work with it continues in the methods that call it.

If getListQuery() is called from getContentCount() in the DatabaseQuery $query object, the set values for select are replaced with COUNT(*).

If getListQuery() is called from getItem($id), the condition $query->where('a.id = ' . (int) $id) and only a specific element is selected. And already here we see that the parent Adapter class contains the table name in the query as a.*. This means that we should also use these prefixes in our implementation of the getListQuery() method.

In the case of calling getListQuery() from getItems(), $offset and $limit are added to the query that we have constructed in order to move through the list of elements for indexing.
Summary: getListQuery() - must contain a "work piece" for three different SQL queries. And there is nothing particularly difficult about implementing Joomla here. But, if necessary, you can implement 3 methods yourself without creating getListQuery().

Non Joomla way: In the case of JoomShopping, I came across the fact that a product can have several categories and historically the category id (catid) component for the product was stored in a separate table. At the same time, for many years it was not possible to specify the main category for the product. Upon receipt of the product category, a query was sent to the table with categories, where just the first query result was taken, sorted by default category id - i.e. ascending. If we changed categories when editing a product, then the main product category was the one with the lower id number. The URL of the product was based on it and the product could jump from one category to another.

But, almost 2 years ago, this JoomShopping behavior was fixed. Since the component has a long history, a large audience and cannot just break backward compatibility, the fix was made optional. The ability to specify the main category for the product must be enabled in the component settings. Then the main_category_id will be filled in the table with the products.

But this functionality is disabled by default. And in the smart search plugin, we need to get the parameters of the JoomShopping component, see if the option to specify the main product category is enabled (and it may be enabled recently and the main category for some products is not specified - also a nuance...) and generate an SQL query to receive the product(s) based on the component parameters: either a simple query where we add the main_category_id field, or a JOIN request to get the category id in the old wrong way.

Immediately, the nuance of multilingualism comes to the fore in this request. According to the Joomla way, a separate element is created for each language of the site and associations are set up between them. So, for the Russian language - one article. The same article in English is being created separately. Then we connect them with each other using language associations and when switching the language on the Joomla frontend, we will be redirected from one article to another.

This is not how it is done in JoomShopping: data for all languages is stored in the same table with products (Ok). Adding data for other languages is done by adding columns with the suffix of these languages (hmm...). That is, we do not have just a title or name field in the database. But there are fields name_ru-RU, name_en-GB, etc.

Joomla JoomShopping product table structure fragment
At the same time, we need to design a universal SQL query so that it can be indexed from both the admin panel and the CLI. At the same time, choosing the indexing language when launching the CLI using CRON is also a task. I admit, at the time of writing this article, I have postponed a full-fledged solution to this problem for the time being. The language is selected using our own getLangTag() method, where we either take the main language from the JoomShopping parameters, or the default language of the site. That is, so far this solution is only for a monolingual site. The search in different languages will not work yet.

However, 3 months later I solved this problem, but already in the smart search plugin for SW JProjects component. I will tell you about the solution further.

In the meantime, let's look at what happened for JoomShopping

<?php
use Joomla\Database\DatabaseQuery;

    /**
     * Method to get the SQL query used to retrieve the list of content items.
     *
     * @param   mixed  $query  A DatabaseQuery object or null.
     *
     * @return  DatabaseQuery  A database object.
     *
     * @since   2.5
     */
    protected function getListQuery($query = null): DatabaseQuery
    {

        $db  = $this->db;
        $tag = $this->getLangTag();
        // Check if we can use the supplied SQL query.
        $query = ($query instanceof DatabaseQuery) ? $query : $db->getQuery(true);
        $query->select(
            [
                'prod.product_ean',
                'prod.manufacturer_code',
                'prod.product_old_price',
                'prod.product_price',
                'prod.product_buy_price',
                'prod.min_price',
                'prod.product_weight',
            ]);
        // Columns with ... AS
        $query->select(
            $db->quoteName(
                [
                    'prod.product_id',
                    'prod.name_' . $tag,
                    'prod.alias_' . $tag,
                    'prod.description_' . $tag,
                    'prod.short_description_' . $tag,
                    'prod.product_date_added',
                    'prod.product_publish',
                    'prod.image',
                    'cat.name_' . $tag,
                ],
                [ // ... AS ...
                    'slug',
                    'title',
                    'alias',
                    'body',
                    'summary',
                    'created',
                    'state',
                    'image',
                    'category',
                ]
            )
        );

        $query->from($db->quoteName('#__jshopping_products', 'prod'))
            ->where($db->quoteName('prod.product_publish') . ' = ' . $db->quote(1))
            ->where($db->quoteName('cat.category_publish') . ' = ' . $db->quote(1));


        /**
         * If the JoomShopping option "Use the main category for the product" is available and enabled,
         * then the product has the main_category_id field.
         * If not, use the old JoomShopping approach - take the 1st category id from the table #__jshopping_products_to_categories
         * to do this, we will make a subquery, since category_id should only be 1.
         */
        if (property_exists($this, 'jshopConfig')
            && !empty($this->jshopConfig)
            && $this->jshopConfig->product_use_main_category_id == 1)
        {

            $query->select($db->quoteName('prod.main_category_id', 'catslug'));
            $query->join('LEFT', $db->quoteName('#__jshopping_categories', 'cat') . ' ON ' . $db->quoteName('cat.category_id') . ' = ' . $db->quoteName('prod.main_category_id'));
        }
        else
        {
            $query->select($db->quoteName('cat.category_id', 'catslug'));

            // Create a subquery for the sub-items list
            $subQuery = $db->getQuery(true)
                ->select($db->quoteName('pr_cat.product_id'))
                ->select('MIN(' . $db->quoteName('pr_cat.category_id') . ') AS ' . $db->quoteName('catslug'))
                ->from($db->quoteName('#__jshopping_products_to_categories', 'pr_cat'))
                ->group($db->quoteName('product_id'));

            $query->join('LEFT', '(' . $subQuery . ') AS ' . $db->quoteName('subquery') . ' ON ' . $db->quoteName('subquery.product_id') . ' = ' . $db->quoteName('prod.product_id'));
            $query->join('LEFT', $db->quoteName('#__jshopping_categories', 'cat'), $db->quoteName('cat.category_id') . ' = ' . $db->quoteName('subquery.catslug'));
        }

        return $query;
}

Check point

We created a method to query the database from Joomla and learned a lot about how the smart search plugin works.

In the next article, we will create a method for indexing content and complete the creation of plugin. We will also get acquainted with how indexed items are stored in the database and understand why this is important and solve the problem of indexing content for multilingual components with a non-standard implementation of multilingualism.

Joomla Community resources

• https://joomla.org/
• This article in Joomla Community Magazine
• Joomla Community chat in Mattermost (read more)