Daemon & Client Overview
Architecture
Client <--> Daemon --> Control Plane
The Gremlin daemon, gremlind is a binary installed on the operating system or available inside the Gremlin container. It heartbeats with the Gremlin Control Plane to let Gremlin know that the host is active and able to receive attack orders. It only communicates outbound with the Gremlin Control Plane. All traffic is encrypted.
The Gremlin client, gremlin, refers to the Gremlin command line interface that is responsible for creating the local impact within the host.
The daemon bundled with the command line interface as a unit is referred to as a targetable Client to the platform.
Client lifecycle
Gremlin clients (infrastructure and application) that have been authenticated to the Gremlin Control Plane appear in the infrastructure clients and application clients lists. You can only run attacks on "active" clients. A client goes into an "idle" state if there is no activity for the past 5 minutes. You cannot run or schedule attacks on idle clients. If Gremlin does not hear from these idle clients for a period of 24 hours, the clients are removed from the list. However, if a client starts communicating with Gremlin again while still within the 24 hour idle window, the client is reactivated and returned to the "active" state.
Logs
Logs can be found under the /var/log/gremlin directory.
Daemon log entries can be found in the daemon.log file. Log entries in this file may indicate events where the daemon is not able to communicate with the Control Plane and eventually trigger the Dead Man Switch.
Each attack on the host is logged under /var/log/gremlin/executions using its unique attack execution ID.
Log size
To see how much disk space is being used by logs, run the du utility on the /var/log/gremlin directory:
1du -sh /var/log/gremlin
Bandwidth usage
Idle state
The daemon uses very little bandwidth in its idle state. In testing over a 5 minute period, the daemon sent a total of 11.3KB and received 24.8KB—an average combined bandwidth of 0.12KB/s.
Attack state
There is a slight increase in overall bandwidth consumption during attacks. While attacks are being executed, the daemon stays in constant communication with the control plane as it checks for the abort condition to be executed. The bandwidth used is not affected by the type of attack being run. In testing over a 5 minute period, the daemon sent a total of 112.3KB and received 114.0KB—an average combined bandwidth of 0.75KB/s.
Services Discovery
When Services Discovery is enabled the Gremlin daemon will send additional data and the bandwidth consumed will depend on how many services are discovered. The information is gzip compressed in order to minimize network consumption. To measure the actual bandwidth consumed by Gremlin for your particular installation, we recommend using a tool such as iptraf or nethogs.