Architecture upgrade

This is nothing more but an outline for the time being. It’s not anything sophisticated, I just write comments that keep me on track and help me stay focused for when I go to re-architect OSIB.

Ok, I need to update the architecture of OSINTBuddy because

1. its currently kinda crappy

2. to be continued…

Backend

to be continued…

Frontend client

to be continued…

Plugin System

We need to be able to run multiple untrusted plugins concurrently.

Security and Sandboxing

I’m most likely going to use Firecracker for this. Firecracker is for creating and managing secure, muli-tenant container/function based services. Think secure and fast microVMs. A bonus of this is I could potentially provide a REPL for in-browser plugin (read entity) development. Imagine a fully-fledged bash shell with neovim for development, well, unlikely I’ll do that, but I can dream. I found this rust library that was created around 5 months ago by a Junior student at Peking University. I’m thinking screw it, no issues, 8 stars (now 9 from me), it’s good enough for this project.

Existing plugin system issues

My thoughts on how to improve the plugin system…

• We can currently only return entities directly attached to the source entity transformed from, in the instance we want to return a subgraph of relationships and entities we are unable to do so
• There’s no plugin settings/config system yet that can be loaded from the users settings
• We aren’t able to store metadata/general information on a plugin for the market UI (e.g. a README for a set of plugins and or a PLUGIN_FILENAME.md for info on a specific plugin)
• Can’t include external files with plugins, e.g. in the CSE Search plugin we depend on a Python request to a gist of JSON data, this is annoying and instead we should be able to store this JSON data in a plugins repo
• Can’t include external dependencies with a plugin, since we’re going to be running in a Firecracker VM we should be able to install any new requirements.txt dependencies or package.json (node deps) into the Firecracker environment
• Plugins are Python based, since we’re in a Firecracker VM we should perhaps load it up with some default scripts (e.g. node, nmap) and support multiple languages. For this we need to write docs on the expected structure of JSON returned from plugins
• Add a ‘hidden’ element type to plugins to store additional properties, all the visible properties of a plugin are rendered to the UI (including Empty which is used for element positioning), a hidden property would only show up when the user clicks a selected edge or selected entity’s button for showing all properties in a panel

• Plugins should be able to be loaded from a list of strings (Python strings for plugins, JSON for resources, .md for metadata, basically we need to note the filetypes loaded from a plugin repository) instead of only from filepaths which is what is currently used in development mode
• In dev mode any plugins edited on the UI should automatically write any changes to the filesystem
• In development mode the filepath to all plugin resources should be a setting on the settings page
• See if we can add types for when we get a plugin, Registry.get_plugin('type the possible plugin names/strings here?'), not sure how to do this but it would make development a lot nicer
• Add support for getting plugins by both the snake_case_format and the actual name e.g. Registry.get_plugin('Google Search')
• If a plugin has a syntax error the entire plugin system fails in a spectacular fashion, we should instead notify the user of the issue and correctly load the rest of the working plugins

• Finish implementing support for generators so we can provide progress updates on a plugin transform. e.g.if we are searching 20 pages of google and storing those results we should be able to send back the results as we pull them (on the ui.. “10 Google result entities returned! Loading more…“)

to be continued…

Footnotes

1. https://users.rust-lang.org/t/build-an-actix-web-endpoint-to-see-live-changes-of-my-redis-cache/109333

2. https://designpatternsmastery.com/1/13/3/1/

3. https://designpatternsmastery.com/1/10/4/

4. https://designpatternsmastery.com/1/14/1/1/#concurrency-patterns

5. https://stackoverflow.com/questions/42471870/publish-subscribe-vs-producer-consumer

6. https://github.com/dezashibi-c/a-prod_cons_vs_pub_sub_in_c

7. https://dev.to/aaravjoshi/6-essential-websocket-patterns-for-real-time-applications-39gf

8. https://docs.rs/actix/latest/actix/trait.AsyncContext.html

9. https://meta.discourse.org/t/introduction-to-discourse-development/349939

10. https://users.rust-lang.org/t/async-queue-with-concurrent-batch/75138

11. https://peerdh.com/blogs/programming-insights/implementing-a-rust-based-sandbox-for-isolated-command-execution

12. https://codezup.com/building-high-performance-python-extensions-with-rust-guide/

13. https://nullderef.com/blog/plugin-tech/

14. https://medium.com/devsphere/implementing-rust-based-plugins-for-existing-software-ecosystems-264054028c59

15. https://github.com/wax911/plugin-architecture

16. https://github.com/topics/plugin-system?l=rust

17. https://www.reddit.com/r/rust/comments/6v29z0/plugin_system_with_api/

18. https://mathieularose.com/plugin-architecture-in-python

19. https://pyquesthub.com/creating-a-dynamic-plugin-system-in-python

20. https://colliery.io/blog/rust-python-pattern/

21. https://medium.com/@kudryavtsev_ia/how-i-design-and-develop-real-world-python-extensions-in-rust-2abfe2377182

22. https://reorchestrate.com/posts/plugins-for-rust/

23. https://medium.com/rustaceans/a-plugin-system-for-rust-but-not-only-using-webassembly-26bb3d327c10

24. https://blog.herecura.eu/blog/2020-05-21-toying-around-with-firecracker/

25. https://lib.rs/crates/firecracker-rs-sdk