Watchdog 1.0.2

NO LONGER FOR SALE !

Need help with watching other players? Automatically? On demand?

Hello, I present you Watchdog.

What is Watchdog ?

•     A plugin/mod for your server, whose whole purpose is to record players on demand
•     An essential tool for server admins

Why to use Watchdog ?
    > One of the main problem with Rust and any game in general is always those pesky cheaters
    > But catching cheaters requires constant watching and making sure that the person being reported is indeed cheating
    > Watchdog solves this problem!
    > You as an admin may never even join the server if you wanted, and yet, you still can find and ban the cheaters, almost entirely from Discord!

Features of Watchdog

•     24/7 Demo recording
•     Very customizable triggers for starting a recording & managing that recording
•     Very customizable combatlog saving
•     Notifications of triggers & events entirely in Discord
•     Compression! Watchdog can compress the demo files and save on avarage 70% of your machine's disk space 
•     Memory usage control
•     Disk usage control (with Hard Limit for when your VPS/VM is limited)

How to use Watchdog ?
    > After a recording of a player is saved
    > Just get the .dem file and place it in your 'demos' folder which is located in Rust's game folder
    > Go in game and type in the console 'demo.browser'
    > Thats it, you can now watch the demo

Section 1: Hardware

Recommended server/VPS/VM hardware:
    - The more RAM, the more demos can be recorded simultaneously without triggering the memory control, also better performance when compressing demos.
    - At least 4 virtual cores are recommended. Rust's server is well known to not utilize multithreading technologies, Watchdog is heavily optimized for multithreading.
    - Disk space. I really mean it, A LOT is needed, the more the better! 
        Watchdog is able to produce over 20gb/hour of data for 300 player server (even with compression turned on).
        The more disk space you have the more demos can be stored on your server.
        Of course Watchdog is optimized for limited systems as well, so it has a Hard Limit option which will limit the data production with demos.
    - Fairly powerful CPU. While this may not be needed for non-compressed demos, the compression of demos is pretty computationally intensive,
        as the algorithm used for compressing is aggressive (LZMA)

All of the recommendations are based on the testing I've done on my machine, you need to do your own testing to see what you need.
The recommended hardware is just that, recommendation. My recommendations might even be incorrect.

Section 2: Software

Because of the insane capabilities of Watchdog, it can generate a really big amounts of data.
So for this reason Watchdog should be used with other custom software for ease of use. This is not required, but recommended.

This includes:
    - A discord bot that can read the combatlog file format and the compressed demos. This can be found here: https://codefling.com/discord-bots/watchdog-discord-bot

Section 3: Installation

This is a Harmony mod, its not an oxide plugin. Watchdog works with full-vanilla servers and with Oxide servers!

On Windows:
    - To install Watchdog you need to place the 'Watchdog.dll' file in your HarmonyMods folder,
        and either restart the server or type 'harmony.load Watchdog' which will dynamically load Watchdog while the server is running.

On Linux:
    - To install Watchdog you first need to set your terminal emulator to 'xterm', this is done with the following linux command: 'export TERM=xterm'.
        IMPORTANT: If the terminal emulator is not xterm Watchdog WILL FAIL to load, this is because of a bug with Mono
    - After that place the 'Watchdog.dll' file in your HarmonyMods folder,
        and either restart the server or type 'harmony.load Watchdog' (NOTE: On linux this name is case-sensitive) which will dynamically load Watchdog while the server is running.

After watchdog runs for the first time a folder with the name 'Watchdog' will be created in the root folder of the server.
Inside this folder there will be 2 folders and 1 file named 'config.json', don't forget to change the default config to your taste,
    and if you want you can change the localization file with your custom messages
More on this in sections 4 and 5 ...

Section 4: Configuration

Before we start with the configuration, remember that if you change the config file,
    in order to see the changes you need to either reload Watchdog with: 'harmony.load Watchdog' or restart the server.
NOTE: There is a bug with the Harmony loader in Rust's Dedicated Server and sometimes (unpredictable) when you load Watchdog
    it will not be loaded fully and might have strange bugs, this is fixed with a server restart.

If you are familiar with JSON every setting is a string. (even if its a number)
If you are not familiar with JSON, just dont delete the quotes ("") at the most right & left of the setting and type only in between them, always!

Everything is non case-sensitive, except webhook URLs in the Feedback settings.

The configuration file is split up into sections for easier presentation, these sections are the following:
    - General - Main settings of Watchdog
    - Specific - Settings that are specific to what main setting was chosen
    - Optimization - Settings for optimization
    - Limit - Limiters for Memory and Disk usage
    - Privacy - Privacy related settings
    - Feedback - Watchdog feedback info, mostly discord information

In general settings section:
    - recording_mode
        When and how to record players, you might want to record players 24/7 or only when they kill something, this is the setting you use for that
        This setting should be a number and can be one or combination of the follwing:
            0 : Always - Players will be recorded 24/7 while they are in the server
            1 : On Kill - Players will be recorded only when they have killed X amount of players/scientist. If this setting is selected, 'include_scientists' and 'kill_threshold' options from specific settings will be used
            2 : On Report - Players will be recorded only when they have been reported X amount of times. If this setting is selected, 'report_threshold' setting from specific settings will be used
            4 : On HitArea - Players will be recorded only when they have hit a player/scientist in X area of their body, X amount of times. If this setting is selected, 'include_scientists', 'hit_areas' and 'hit_count_threshold' options from specific settings will be used
            8 : On Stash - Players will be recorded only when they have uncovered X amount of stashes. If this setting is selected, 'stashes_threshold' and 'remember_stash_owner' options from specific settings will be used
        You can use these triggers individually or combine them, if you want to combine them you just add the numbers together.
        For example if you want to record OnKill and OnReport you will add them up, 1 + 2 = 3, so you type 3 in the config. If you want to record on every trigger its all of them added up, which is 15
    
    - recording_timout
        When to stop the recording after being triggered, this is applicable only when 'recording_mode' is not Always
        This setting should be a time, this is in the format <number>s<number>m<number>h, set to 0 to be infinite (until user disconnects)
        Examples:
            If you want to record someone for 30 minutes on recording trigger
                "recording_timout": "30m"
            If you want to record someone for 90 minutes on recording trigger
                "recording_timout": "1h30m" or "recording_timout": "90m"
            If you want to record someone for 2 hours, 30 minutes and 20 seconds on recording trigger
                "recording_timout": "2h30m20s" the order can also be different "recording_timout": "30m20s2h"
            If you want to record someone until they disconnect on recording trigger
                "recording_timout": "0"

    - combatlog_saving
        If you want to save the combatlog of players
        This setting can be one of the following:
            - never : This setting is off and will not save anything to disk
            - on_humanoid_combat - Saves combatlog of any combat to a player or scientist
            - on_player_combat - Saves combatlog of any combat to a player
            - on_humanoid_kill - Saves combatlog of killing a player or scientist
            - on_player_kill - Saves combatlog of killing a player
            - always - Saves every combatlog of every player to disk, this includes someone hitting a stone wall for example
        When any of the kill settings are set, the setting 'combatlog_history' from specific settings will be used to determine how many logs (including body hits) in the past are to be saved when someone kills a target
        This is for example if 'combatlog_history' is set to "10" and 'combatlog_saving' is set to "on_humanoid_kill", when a player kills a scientist or a player, the combatlog of that entire fight will be saved with up to 10 older hits, so the last 9 combat logs (anything, including stone wall hits) and the last kill hit log

In specific settings section:
    - include_scientists
        Should scientists be included in the trigger filter, this is applicable to 'recording_mode' setting with OnKill and OnHitArea
        This should be a yes/no answer
        For example: "include_scientists": "yes"
        You can use a short form of 'y','n' or '+','-' or 'true','false' or 't','f' and there is more

    - kill_threshold
        How many kills to make to trigger a recording
        This should be a number
        For example: "kill_threshold": "5"

    - report_threshold
        How many reports to receive to trigger a recording
        This should be a number
        For example: "report_threshold": "2"

    - hit_areas
        Which HitAreas to watch for. This is applicable to 'recording_mode' setting with OnHitArea
        This should be a number and can be a combination
             1 : Head
             2 : Chest
             4 : Stomach
             8 : Arms
            16 : Hands
            32 : Legs
            64 : Feet
        To select multiple again as in recording_mode setting you just add up the numbers
        For example if you want to watch for headshots, chest and stomach hits you will do: 1 + 2 + 4 = 7

    - hit_count_threshold
        How many times a player needs to hit someone in 'hit_areas' areas in order to trigger a recording
        This should be a number
        For example: "hit_count_threshold": "2"
        So with this example and the one before if someone hits a player/scientist (depends on setting) in the head,chest or stomach 2 times or more they will start being recorded

    - stashes_threshold
        How many stashes should a player uncover to trigger a recording
        This should be a number
        For example: "stashes_threshold": "2"

    - remember_stash_owner
        If a player uncovers their own potentially forgotten stash, should they trigger a recording
        This should be a yes/no answer
        For example: "remember_stash_owner": "yes"
        I recommend setting this to 'yes' always

    - combatlog_history
        How many combat logs to save when a kill event is triggered in 'combatlog_saving' setting
        This should be a number
        For example: "combatlog_history": "10"
        For example: Player A kills Player B, but in the process of their fight, Player A hits another player behind Player B, you can see it, with this setting you can see up to X amount of combatlog hits before their fight finished

In optimization settings section:
    - compress_recorded_demos
        Should the saved demos be compressed. Compression is saving ~70% of space but uses more CPU & RAM
        This should be a yes/no answer
        For example: "compress_recorded_demos": "yes"
        
    - compress_recorded_demos_thread_model
        If 'compress_recorded_demos' is turned on, should the demo be compressed on the main server thread or on a separate thread
        This should be one of the following values
            - threaded
            - default
        For example: "compress_recorded_demos": "threaded"
        I recommend setting to threaded if the machine hosting the server has spare CPU cores. Be careful with fewer core machines, the compression operation can take up to 200ms per demo (maybe even more on larger demos)

    - stop_demo_for_afk_players
        Should the demo recording be paused if a player goes AFK
        This should be a yes/no answer
        For example: "stop_demo_for_afk_players": "yes"
        I recommend setting this to yes as it will save you disk space on players who are AFK and being recorded. A small lagspike for the player might occur when they come back from being afk (for example move their mouse), the demo might be choppy at the start if its a transition between AFK and non AFK

    - afk_time_threshold
        Should the demo recording be paused if a player goes AFK
        This should be time, this is in the format <number>s<number>m<number>h (this is the same format as in setting 'recording_timout')
        For example, at 1 minute AFK user's recording will get paused: "afk_time_threshold": "1m"

    - suppress_console_demo_messages
        When a player is being recorded or their recording is stopped you get a message in the console, this setting will remove that message if turned on
        This should be a yes/no answer
        For example: "suppress_console_demo_messages": "no"

    - combatlog_buffer_size_threshold
        When this RAM threshold is reached the buffer of the global combatlog will be saved to disk
        This should be a size (in B, KB, MB or GB)
        For example: "combatlog_buffer_size_threshold": "10MB"
        I recommend leaving this to default if you are not experiencing issues. Might need to be increased with more player activity for optimal performance

    - combatlog_buffer_time_threshold
        When this time threshold is reached the buffer of the global combatlog will be saved to disk.
        This should be a time, this is in the format <number>s<number>m<number>h (this is the same format as in setting 'recording_timout')
        For example: "combatlog_buffer_size_threshold": "10s"
        I recommend leaving this to default if you are not experiencing issues. Might need to be increased with more player activity for optimal performance

    NOTE: If both combatlog_buffer_thresholds are set to 0, the combatlog will be only saved to disk when Watchdog is unloaded (aka on server restart/stop). If one of them is 0 only the other one is active, if both are not 0 then whichever threshold first gets reached

In limit settings section:
    - max_total_gb_ram
        Memory limit for all currently recording demos, if this limit is reached all of them are stopped
        This should be a size (in B, KB, MB or GB)
        For example: "max_total_gb_ram": "4GB"

    - max_total_gb_disk_hardlimit
        Disk limit for all saved demos and combatlog file. When this limit is reached no further demos/combatlogs will be saved. This hardlimit is useful if your VPS/Machine has limited storage or you pay for every GB.
        IMPORTANT: Do not set the limit to the exact size of your disk size or free space, leave some wiggle room
        This should be a size (in B, KB, MB or GB)
        For example: "max_total_gb_disk_hardlimit": "300GB"

In privacy settings section:
    - notify_users_when_recording
        Should the user get notified in chat & their console that they are being currently recorded
        This should be a yes/no answer
        For example: "notify_users_when_recording": "no"

    - notify_users_when_joining
        Should the user get notified in chat & their console that the server is using Watchdog
        This should be a yes/no answer
        For example: "notify_users_when_joining": "yes"

    - send_diagnostics_information
        Should Watchdog send diagnostic information that will be used to further optimize and improve watchdog
        This should be a yes/no answer
        For example: "send_diagnostics_information": "yes"
        I recommend leaving this turned on because it will greatly help me to see what needs to get optimized and/or changed
        The data being collected is the following:
            - Internal performance & error data
            - Server identity (server ip, port, player count, server uptime, server average fps)
            - Plugins/Mods being used

    - send_diagnostics_interval
        How often to send the diagnostics information
        This should be a time. this is in the format <number>s<number>m<number>h (this is the same format as in setting 'recording_timout')
        For example: "send_diagnostics_interval": "5m"
        I recommend leaving this on the default. Increase if you notice more than comfortable network usage. The higher this interval is the higher the network impact will be when the data is being uploaded as its being uploaded as a big chunk

In privacy settings section:
    - notify_recording_triggered_webhook
        Discord Webhook that will be used to send messages to when a recording is started
        This should be a URL, leave empty to disable
        For example: "notify_recording_triggered_webhook": "https://discord.com/api/webhooks/1095317841874187282/LHjjksahd_tokengoeshere_d182u9182"
    
    - player_reported_webhook
        Discord Webhook that will be used to send messages to when a player is F7 reported
        This should be a URL, leave empty to disable
        For example: "player_reported_webhook": "https://discord.com/api/webhooks/1095317841874187282/LHjjksahd_tokengoeshere_d182u9182"

    - recording_about_to_finish_webhook
        Discord Webhook that will be used to send messages to when a recording is about to finish
        This should be a URL, leave empty to disable
        For example: "recording_about_to_finish_webhook": "https://discord.com/api/webhooks/1095317841874187282/LHjjksahd_tokengoeshere_d182u9182"

    - recording_about_to_finish_threshold
        When the recording time hits this percentage threshold a message will be sent in discord informing you that the recording is about to finish
        This should be a percentage
        For example: "recording_about_to_finish_threshold": "90%"
        If this is set to 100% you will get a message when the recording stopped (roughly, timing is not exact millisecond precision)

NOTE: You can always see what the default generated config is by deleting or renaming the file 'config.json', if you want to use it as a reference

Ingame config settings:
    These settings are built-in ConVars and can be changed via the console like any other ConVar

    - demo.splitmegabytes : When a demo reaches this amount of megabytes in size it gets stopped and another is started, this is to split them
    -   demo.splitseconds : When a demo reaches this amount of time recording it gets stopped and another is started, this is to split them
    -     demo.recordlist : Add player steamids in here to record them automatically on join
    - demo.recordlistmode : 0 for whitelist, 1 for blacklist. Whitelist means all players in 'demo.recordlist' will be recorded on server join, blacklist means all player NOT in 'demo.recordlist' will be recorded on server join

Section 5: Localization

Watchdog supports localization (although currently only 1 type of language is supported)

To edit the localization, go to the 'Watchdog' folder which is at the root of your server folder,
    from there go into the directory 'localization', a file 'english.txt' will contain the localization strings
Just open that file with any text editor and edit the messages to your liking

The format of the localization file is "some locale = some text"
After the equals sign is the message, you edit that, do not touch the text that is at the left of the equals sign

Whenever you see a percentage in front and back of some text that means this is a variable and is replaced with the actual value when that event happens
For example %steamid% means the steamid64 of the person who initiated this message
Here are the meanings of all the variables:
    %steamid% - steamid64 of player
    %username% - in-game name of player
    %recording_trigger% - what triggered the recording, this can tell you for example if it was someone uncovering a stash or killing a player or any other thing
    %reported_times% - how many times this person was reported since the last recording
    %progress% - current recording progress at the time of sending this message

Section 6: Commands

Watchdog has some built-in commands and modifies another, the commands are:
    - watchdog.version - Prints the version of Watchdog
    - watchdog.webhooktest - Sends test messages to all enabled webhooks. This is used if you want to see if the link you provided is working
    - demo.record - Starts a manual demo recording, this will not stop automatically until you stop with manually again, even if the user disconnects, this is useful if you want to record someone 24/7
    - demo.stop - Stops a manual demo recording, if a demo is started with 'demo.record' this is the only way to stop that demo

Section 7: Contact

If you have any questions, suggestions or want to report a bug
you can contact me on discord: turner#7777