Merge branch 'smart-plugins-enhancements' into develop

docs/source/resources/SmartPlugins.rst

@@ -1,9 +1,9 @@
 Smart Plugins
 =============
 
-Pyrogram embeds a **smart** (automatic) and lightweight plugin system that is meant to further simplify the organization
+Pyrogram embeds a **smart**, lightweight yet powerful plugin system that is meant to further simplify the organization
-of large projects and to provide a way for creating pluggable components that can be **easily shared** across different
+of large projects and to provide a way for creating pluggable (modular) components that can be **easily shared** across
-Pyrogram applications with **minimal boilerplate code**.
+different Pyrogram applications with **minimal boilerplate code**.
 
 .. tip::
 
@@ -13,7 +13,8 @@ Introduction
 ------------
 
 Prior to the Smart Plugin system, pluggable handlers were already possible. For example, if you wanted to modularize
-your applications, you had to do something like this...
+your applications, you had to put your function definitions in separate files and register them inside your main script,
+like this:
 
 .. note::
 
@@ -63,19 +64,19 @@ your applications, you had to do something like this...
 
         app.run()
 
-...which is already nice and doesn't add *too much* boilerplate code, but things can get boring still; you have to
+This is already nice and doesn't add *too much* boilerplate code, but things can get boring still; you have to
 manually ``import``, manually :meth:`add_handler <pyrogram.Client.add_handler>` and manually instantiate each
 :obj:`MessageHandler <pyrogram.MessageHandler>` object because **you can't use those cool decorators** for your
-functions. So... What if you could?
+functions. So, what if you could? Smart Plugins solve this issue by taking care of handlers registration automatically.
 
 Using Smart Plugins
 -------------------
 
-Setting up your Pyrogram project to accommodate Smart Plugins is pretty straightforward:
+Setting up your Pyrogram project to accommodate Smart Plugins is straightforward:
 
-#. Create a new folder to store all the plugins (e.g.: "plugins").
+#. Create a new folder to store all the plugins (e.g.: "plugins", "handlers", ...).
-#. Put your files full of plugins inside.
+#. Put your python files full of plugins inside. Organize them as you wish.
-#. Enable plugins in your Client.
+#. Enable plugins in your Client or via the *config.ini* file.
 
 .. note::
 
@@ -107,20 +108,252 @@ Setting up your Pyrogram project to accommodate Smart Plugins is pretty straight
         def echo_reversed(client, message):
             message.reply(message.text[::-1])
 
+- ``config.ini``
+
+    .. code-block:: ini
+
+        [plugins]
+        root = plugins
+
 - ``main.py``
 
     .. code-block:: python
 
         from pyrogram import Client
 
-        Client("my_account", plugins_dir="plugins").run()
+        Client("my_account").run()
 
-The first important thing to note is the new ``plugins`` folder, whose name is passed to the the ``plugins_dir``
+    Alternatively, without using the *config.ini* file:
-parameter when creating a :obj:`Client <pyrogram.Client>` in the ``main.py`` file — you can put *any python file* in
+
-there and each file can contain *any decorated function* (handlers) with only one limitation: within a single plugin
+    .. code-block:: python
-file you must use different names for each decorated function. Your Pyrogram Client instance will **automatically**
+
-scan the folder upon creation to search for valid handlers and register them for you.
+        from pyrogram import Client
+
+        plugins = dict(
+            root="plugins"
+        )
+
+        Client("my_account", plugins=plugins).run()
+
+The first important thing to note is the new ``plugins`` folder. You can put *any python file* in *any subfolder* and
+each file can contain *any decorated function* (handlers) with one limitation: within a single module (file) you must
+use different names for each decorated function.
+
+The second thing is telling Pyrogram where to look for your plugins: you can either use the *config.ini* file or
+the Client parameter "plugins"; the *root* value must match the name of your plugins folder. Your Pyrogram Client
+instance will **automatically** scan the folder upon starting to search for valid handlers and register them for you.
 
 Then you'll notice you can now use decorators. That's right, you can apply the usual decorators to your callback
 functions in a static way, i.e. **without having the Client instance around**: simply use ``@Client`` (Client class)
-instead of the usual ``@app`` (Client instance) namespace and things will work just the same.
+instead of the usual ``@app`` (Client instance) and things will work just the same.
+
+Specifying the Plugins to include
+---------------------------------
+
+By default, if you don't explicitly supply a list of plugins, every valid one found inside your plugins root folder will
+be included by following the alphabetical order of the directory structure (files and subfolders); the single handlers
+found inside each module will be, instead, loaded in the order they are defined, from top to bottom.
+
+.. note::
+
+    Remember: there can be at most one handler, within a group, dealing with a specific update. Plugins with overlapping
+    filters included a second time will not work. Learn more at `More on Updates <MoreOnUpdates.html>`_.
+
+This default loading behaviour is usually enough, but sometimes you want to have more control on what to include (or
+exclude) and in which exact order to load plugins. The way to do this is to make use of ``include`` and ``exclude``
+keys, either in the *config.ini* file or in the dictionary passed as Client argument. Here's how they work:
+
+- If both ``include`` and ``exclude`` are omitted, all plugins are loaded as described above.
+- If ``include`` is given, only the specified plugins will be loaded, in the order they are passed.
+- If ``exclude`` is given, the plugins specified here will be unloaded.
+
+The ``include`` and ``exclude`` value is a **list of strings**. Each string containing the path of the module relative
+to the plugins root folder, in Python notation (dots instead of slashes).
+
+    E.g.: ``subfolder.module`` refers to ``plugins/subfolder/module.py``, with ``root="plugins"`.
+
+You can also choose the order in which the single handlers inside a module are loaded, thus overriding the default
+top-to-bottom loading policy. You can do this by appending the name of the functions to the module path, each one
+separated by a blank space.
+
+    E.g.: ``subfolder.module fn2 fn1 fn3`` will load *fn2*, *fn1* and *fn3* from *subfolder.module*, in this order.
+
+Examples
+^^^^^^^^
+
+Given this plugins folder structure with three modules, each containing their own handlers (fn1, fn2, etc...), which are
+also organized in subfolders:
+
+.. code-block:: text
+
+    myproject/
+        plugins/
+            subfolder1/
+                plugins1.py
+                    - fn1
+                    - fn2
+                    - fn3
+            subfolder2/
+                plugins2.py
+                    ...
+            plugins0.py
+                ...
+        ...
+
+- Load every handler from every module, namely *plugins0.py*, *plugins1.py* and *plugins2.py* in alphabetical order
+  (files) and definition order (handlers inside files):
+
+    Using *config.ini* file:
+
+    .. code-block:: ini
+
+        [plugins]
+        root = plugins
+
+    Using *Client*'s parameter:
+
+    .. code-block:: python
+
+        plugins = dict(
+            root="plugins"
+        )
+
+        Client("my_account", plugins=plugins).run()
+
+- Load only handlers defined inside *plugins2.py* and *plugins0.py*, in this order:
+
+    Using *config.ini* file:
+
+    .. code-block:: ini
+
+        [plugins]
+        root = plugins
+        include =
+            subfolder2.plugins2
+            plugins0
+
+    Using *Client*'s parameter:
+
+    .. code-block:: python
+
+        plugins = dict(
+            root="plugins",
+            include=[
+                "subfolder2.plugins2",
+                "plugins0"
+            ]
+        )
+
+        Client("my_account", plugins=plugins).run()
+
+- Load everything except the handlers inside *plugins2.py*:
+
+    Using *config.ini* file:
+
+    .. code-block:: ini
+
+        [plugins]
+        root = plugins
+        exclude = subfolder2.plugins2
+
+    Using *Client*'s parameter:
+
+    .. code-block:: python
+
+        plugins = dict(
+            root="plugins",
+            exclude=["subfolder2.plugins2"]
+        )
+
+        Client("my_account", plugins=plugins).run()
+
+- Load only *fn3*, *fn1* and *fn2* (in this order) from *plugins1.py*:
+
+    Using *config.ini* file:
+
+    .. code-block:: ini
+
+        [plugins]
+        root = plugins
+        include = subfolder1.plugins1 fn3 fn1 fn2
+
+    Using *Client*'s parameter:
+
+    .. code-block:: python
+
+        plugins = dict(
+            root="plugins",
+            include=["subfolder1.plugins1 fn3 fn1 fn2"]
+        )
+
+        Client("my_account", plugins=plugins).run()
+
+Load/Unload Plugins at Runtime
+------------------------------
+
+In the `previous section <#specifying-the-plugins-to-include>`_ we've explained how to specify which plugins to load and
+which to ignore before your Client starts. Here we'll show, instead, how to unload and load again a previously
+registered plugins at runtime.
+
+Each function decorated with the usual ``on_message`` decorator (or any other decorator that deals with Telegram updates
+) will be modified in such a way that, when you reference them later on, they will be actually pointing to a tuple of
+*(handler: Handler, group: int)*. The actual callback function is therefore stored inside the handler's *callback*
+attribute. Here's an example:
+
+- ``plugins/handlers.py``
+
+    .. code-block:: python
+        :emphasize-lines: 5, 6
+
+        @Client.on_message(Filters.text & Filters.private)
+        def echo(client, message):
+            message.reply(message.text)
+
+        print(echo)
+        print(echo[0].callback)
+
+-   Printing ``echo`` will show something like ``(<MessageHandler object at 0x10e3abc50>, 0)``.
+
+-   Printing ``echo[0].callback``, that is, the *callback* attribute of the first eleent of the tuple, which is an
+    Handler, will reveal the actual callback ``<function echo at 0x10e3b6598>``.
+
+Unloading
+^^^^^^^^^
+
+In order to unload a plugin, or any other handler, all you need to do is obtain a reference to it (by importing the
+relevant module) and call :meth:`remove_handler <pyrogram.Client.remove_handler>` Client's method with your function
+name preceded by the star ``*`` operator as argument. Example:
+
+- ``main.py``
+
+    .. code-block:: python
+
+        from plugins.handlers import echo
+
+        ...
+
+        app.remove_handler(*echo)
+
+The star ``*`` operator is used to unpack the tuple into positional arguments so that *remove_handler* will receive
+exactly what is needed. The same could have been achieved with:
+
+.. code-block:: python
+
+    handler, group = echo
+    app.remove_handler(handler, group)
+
+Loading
+^^^^^^^
+
+Similarly to the unloading process, in order to load again a previously unloaded plugin you do the same, but this time
+using :meth:`add_handler <pyrogram.Client.add_handler>` instead. Example:
+
+- ``main.py``
+
+    .. code-block:: python
+
+        from plugins.handlers import echo
+
+        ...
+
+        app.add_handler(*echo)

pyrogram/client/client.py

@@ -157,10 +157,8 @@ class Client(Methods, BaseClient):
         config_file (``str``, *optional*):
             Path of the configuration file. Defaults to ./config.ini
 
-        plugins_dir (``str``, *optional*):
+        plugins (``dict``, *optional*):
-            Define a custom directory for your plugins. The plugins directory is the location in your
+            TODO: doctrings
-            filesystem where Pyrogram will automatically load your update handlers.
-            Defaults to None (plugins disabled).
 
         no_updates (``bool``, *optional*):
             Pass True to completely disable incoming updates for the current session.
@@ -197,7 +195,7 @@ class Client(Methods, BaseClient):
                  workers: int = BaseClient.WORKERS,
                  workdir: str = BaseClient.WORKDIR,
                  config_file: str = BaseClient.CONFIG_FILE,
-                 plugins_dir: str = None,
+                 plugins: dict = None,
                  no_updates: bool = None,
                  takeout: bool = None):
         super().__init__()
@@ -223,7 +221,7 @@ class Client(Methods, BaseClient):
         self.workers = workers
         self.workdir = workdir
         self.config_file = config_file
-        self.plugins_dir = plugins_dir
+        self.plugins = plugins
         self.no_updates = no_updates
         self.takeout = takeout
 
@@ -1074,6 +1072,30 @@ class Client(Methods, BaseClient):
                 self._proxy["username"] = parser.get("proxy", "username", fallback=None) or None
                 self._proxy["password"] = parser.get("proxy", "password", fallback=None) or None
 
+        if self.plugins:
+            self.plugins["enabled"] = bool(self.plugins.get("enabled", True))
+            self.plugins["include"] = "\n".join(self.plugins.get("include", [])) or None
+            self.plugins["exclude"] = "\n".join(self.plugins.get("exclude", [])) or None
+        else:
+            try:
+                section = parser["plugins"]
+
+                self.plugins = {
+                    "enabled": section.getboolean("enabled", True),
+                    "root": section.get("root"),
+                    "include": section.get("include") or None,
+                    "exclude": section.get("exclude") or None
+                }
+            except KeyError:
+                pass
+
+        for option in ["include", "exclude"]:
+            if self.plugins[option] is not None:
+                self.plugins[option] = [
+                    (i.split()[0], i.split()[1:] or None)
+                    for i in self.plugins[option].strip().split("\n")
+                ]
+
     def load_session(self):
         try:
             with open(os.path.join(self.workdir, "{}.session".format(self.session_name)), encoding="utf-8") as f:
@@ -1105,43 +1127,108 @@ class Client(Methods, BaseClient):
                     self.peers_by_phone[k] = peer
 
     def load_plugins(self):
-        if self.plugins_dir is not None:
+        if self.plugins.get("enabled", False):
-            plugins_count = 0
+            root = self.plugins["root"]
+            include = self.plugins["include"]
+            exclude = self.plugins["exclude"]
 
-            for path in Path(self.plugins_dir).rglob("*.py"):
+            count = 0
-                file_path = os.path.splitext(str(path))[0]
-                import_path = []
 
-                while file_path:
+            if include is None:
-                    file_path, tail = os.path.split(file_path)
+                for path in sorted(Path(root).rglob("*.py")):
-                    import_path.insert(0, tail)
+                    module_path = os.path.splitext(str(path))[0].replace("/", ".")
+                    module = import_module(module_path)
 
-                import_path = ".".join(import_path)
+                    for name in vars(module).keys():
-                module = import_module(import_path)
+                        # noinspection PyBroadException
+                        try:
+                            handler, group = getattr(module, name)
 
-                for name in dir(module):
+                            if isinstance(handler, Handler) and isinstance(group, int):
-                    # noinspection PyBroadException
+                                self.add_handler(handler, group)
-                    try:
-                        handler, group = getattr(module, name)
 
-                        if isinstance(handler, Handler) and isinstance(group, int):
+                                log.info('[LOAD] {}("{}") in group {} from "{}"'.format(
-                            self.add_handler(handler, group)
+                                    type(handler).__name__, name, group, module_path))
 
-                            log.info('{}("{}") from "{}" loaded in group {}'.format(
+                                count += 1
-                                type(handler).__name__, name, import_path, group))
+                        except Exception:
-
+                            pass
-                            plugins_count += 1
-                    except Exception:
-                        pass
-
-            if plugins_count > 0:
-                log.warning('Successfully loaded {} plugin{} from "{}"'.format(
-                    plugins_count,
-                    "s" if plugins_count > 1 else "",
-                    self.plugins_dir
-                ))
             else:
-                log.warning('No plugin loaded: "{}" doesn\'t contain any valid plugin'.format(self.plugins_dir))
+                for path, handlers in include:
+                    module_path = root + "." + path
+                    warn_non_existent_functions = True
+
+                    try:
+                        module = import_module(module_path)
+                    except ModuleNotFoundError:
+                        log.warning('[LOAD] Ignoring non-existent module "{}"'.format(module_path))
+                        continue
+
+                    if "__path__" in dir(module):
+                        log.warning('[LOAD] Ignoring namespace "{}"'.format(module_path))
+                        continue
+
+                    if handlers is None:
+                        handlers = vars(module).keys()
+                        warn_non_existent_functions = False
+
+                    for name in handlers:
+                        # noinspection PyBroadException
+                        try:
+                            handler, group = getattr(module, name)
+
+                            if isinstance(handler, Handler) and isinstance(group, int):
+                                self.add_handler(handler, group)
+
+                                log.info('[LOAD] {}("{}") in group {} from "{}"'.format(
+                                    type(handler).__name__, name, group, module_path))
+
+                                count += 1
+                        except Exception:
+                            if warn_non_existent_functions:
+                                log.warning('[LOAD] Ignoring non-existent function "{}" from "{}"'.format(
+                                    name, module_path))
+
+            if exclude is not None:
+                for path, handlers in exclude:
+                    module_path = root + "." + path
+                    warn_non_existent_functions = True
+
+                    try:
+                        module = import_module(module_path)
+                    except ModuleNotFoundError:
+                        log.warning('[UNLOAD] Ignoring non-existent module "{}"'.format(module_path))
+                        continue
+
+                    if "__path__" in dir(module):
+                        log.warning('[UNLOAD] Ignoring namespace "{}"'.format(module_path))
+                        continue
+
+                    if handlers is None:
+                        handlers = vars(module).keys()
+                        warn_non_existent_functions = False
+
+                    for name in handlers:
+                        # noinspection PyBroadException
+                        try:
+                            handler, group = getattr(module, name)
+
+                            if isinstance(handler, Handler) and isinstance(group, int):
+                                self.remove_handler(handler, group)
+
+                                log.info('[UNLOAD] {}("{}") from group {} in "{}"'.format(
+                                    type(handler).__name__, name, group, module_path))
+
+                                count -= 1
+                        except Exception:
+                            if warn_non_existent_functions:
+                                log.warning('[UNLOAD] Ignoring non-existent function "{}" from "{}"'.format(
+                                    name, module_path))
+
+            if count > 0:
+                log.warning('Successfully loaded {} plugin{} from "{}"'.format(count, "s" if count > 1 else "", root))
+            else:
+                log.warning('No plugin loaded from "{}"'.format(root))
 
     def save_session(self):
         auth_key = base64.b64encode(self.auth_key).decode()